BULLSHIT QUANTIFICATION

What do the happiness index, effective altruism and subprime mortgage lending have in common?

Our world has a funny relationship with numbers. We rely on them, we obsess over them, as witness the current flap over Trump’s firing the head of the Bureau of Labor Statistics for producing job data he didn’t like. Though the more telling part of that story wasn’t the outrage over Trump’s outrageous action, but the panic over its implications; how will we be able to function, people were asking, if we can’t trust the data? As a data wonk, I share that fear up to a point, but I remind myself that (1) the data is always at best imperfect; and (2) as the great English economist Ronald Coase famously said, “if you torture the data long enough, it will confess to anything.”

You don’t want to look too closely under the hood at a lot of the so-called quantitative research that shows up in scholarly journals, let alone the numbers that appear in the media – where one is constantly being subjected to articles that start out with something like “new research shows that 73% of all heavy smokers suffered from bullying as children.” If you go to the trouble to track down the source, you usually find out that either (1) the report misunderstood or otherwise got the research wrong; (2) the research itself was flawed at best, for any number of reasons; or (3) the entire question didn’t make sense. Or all of the above.

But most of that stuff, in the final analysis, doesn’t matter a whole lot. Nobody’s going to lose their home or see their children go hungry if it turns out that there really wasn’t any link between heavy smoking and schoolyard bullying. If one drew parallels with criminal law, most data abuse is like shoplifting; it is bad for society and should be stopped, but the individual impacts of each case are minimal. But there are certain individual cases of data abuse which, in my opinion, rise to an entirely higher level of statistical criminality. These are the Bernie Madoffs or Enrons of the world of numbers. And, interestingly, they all seem to start with the best intentions.

My three examples are the happiness index, effective altruism and the subprime mortgage business. They seem to be wildly different, yet at root they are all about the same thing. Playing with numbers, or bullshit quantification, but for bigger stakes.

Lets start with the subprime mortgage business. Many things came together to plunge the world into the greatest financial crisis since the Great Depression, which we still haven’t fully gotten over, but an important strand, particularly in the United States, was the rise of the so-called ‘subprime mortgage’. Given how much ink has been spilled over the harm it did to the economy, let alone the millions of families who lost their homes and saw their savings disappear, it’s hard to believe that it all started with a misuse of data conducted with at least partly good intentions.

At some point in the early 1990s, a number of supposedly number-savvy people loosely known in the world of finance as “quants” had what they considered to be a brilliant idea. Up to that point, the customary approach to mortgage making was that if the borrower was considered a “prime” borrower, meaning that she met certain basic tests of creditworthiness, she got a mortgage to buy or refinance a home at the going interest rate. If she didn’t meet those tests, she didn’t get a mortgage. Simple.

The brilliant idea was that that model left out a lot of people who were “subprime” but might under the right conditions be good candidates for homeownership. Since these people posed a higher risk of default than the “prime” borrowers, the solution was straightforward: calculate how much higher the risk, and raise the interest rate to reflect the higher risk. That was a simple arithmetical relationship, which was easy to turn into a financial model.

For some people in the industry, this was a wonderful way to give struggling families a shot at homeownership. For them and others, it was also an equally wonderful way for lenders to generate more business and build their portfolios. It didn’t hurt that at the same time mortgage interest rates were coming down and there was a glut of global money floating around looking for higher-interest-rate investments.

Of course, the quants were modeling something which up to that point had been entirely hypothetical, which meant that they had no actual data to use as the basis for their models. In other words, even if their assumption – that higher interest rates or other changes in mortgage terms could offset higher risk – was valid in theory, they had no idea how much greater the risk actually was, or what changes in terms would offset it, or for that matter, what kind of feedback systems they were creating; namely, how much increasing interest rates would itself increase the risk of default. The models, basically, were pure bullshit quantification.

We all know what happened. In 2008, the leaders of the Group of 20 (G20) nations issued a statement explaining what happened, which I admire for its stately bureaucratic prose:

During a period of strong global growth, growing capital flows, and prolonged stability earlier this decade, market participants sought higher yields without an adequate appreciation of the risks and failed to exercise proper due diligence. At the same time, weak underwriting standards, unsound risk management practices, increasingly complex and opaque financial products, and consequent excessive leverage combined to create vulnerabilities in the system. Policy-makers, regulators and supervisors, in some advanced countries, did not adequately appreciate and address the risks building up in financial markets, keep pace with financial innovation, or take into account the systemic ramifications of domestic regulatory actions.[1]

Translated into simple English, it means, boy, did we ever fuck up.

Nobody has ever accused the happiness index of setting off a global financial crisis. The idea of a happiness index emerged from many people’s not-unreasonable dissatisfaction with the use of gross domestic (or national) product as a way of measuring the quality of life or well-being in a nation, as distinct from its gross level of economic activity. Every year since 2012, a World Happiness Report has been published, initially by the Earth Institute at Columbia and currently by the Wellbeing Research Centre at the University of Oxford. The report, which currently ranks 147 of the world’s countries by their supposed relative happiness, receives considerable media attention when released every March 20, which has been dubbed World Happiness Day. Reporters and pundits speculate on the significance of the year’s rankings, which are presented to three decimal places, as the screenshot – which shows a handful of countries in the middle of the pack – illustrates.[2] The right hand column shows the change in the country’s score since the initial report in 2012.

But what exactly is happiness? The most recent report is 260 pages long and includes fascinating essays summarizing the research on a variety of factors related to well-being and quality of life. But in the end, none of those factors have anything to do with the index. The world happiness index comes down to a single factor, the country’s score on a life evaluation question called the Cantril Ladder, which goes as follows:

Suppose we say that the top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you. If the top step is 10 and the bottom step is 0, on which step of the ladder do you feel you personally stand at the present time?

The World Happiness Index people justify this by arguing that:

This question is both democratic and universal. Rather than constructing an index from multiple metrics, the Cantril Ladder empowers people to make their own judgements about what matters most, regardless of their culture and background.

This justification is nothing more than self-serving rhetoric. It is patently obvious that culture and background will define “what matters most”, or what the steps on the ladder mean, whether it has to do with health, family, wealth, power, status, the ability to kill one’s rival with impunity or whatever. I won’t go into this further, because Yasha Mounk has already done an excellent job in a post earlier this year on his Persuasion substack, which I urge everyone to read.[3] But in the end it doesn’t really matter whether you use a single factor, or construct a complicated index, plug in multiple variables, give them different weights, etc., etc. Ultimately constructing a numerical happiness index amounts to nothing more than a mirror of the subjective values of the people doing the ostensible quantifying.

Once again, bullshit quantification.

Except that it uses up funding that might go to more productive purposes, the World Happiness Index can be considered, in the immortal words of The Hitchhiker’s Guide to the Galaxy, mostly harmless. My third example, so-called effective altruism, may be less so. Effective altruism, largely the brainchild of Oxford philosopher William McAskill, has been summarized as a movement that seeks to “use reason and evidence to assess how to do as much good as possible, and take action on this basis.”[4]

While that statement is broad enough (or fuzzy enough) to be hard to differ with (though “reason” and “evidence” are both heavily value-laden terms), its proponents, McAskill in particular, take it a step further, into a concept that they call “longtermism,” as he writes:

Even setting aside the possibility that our distant descendants will settle the stars, Earth itself could host around 10¹⁶ (10 million billion) people over the next billion years. If their lives would go well were all these people to exist, then ensuring their existence may do an astronomical amount of good. One might thus argue that engaging in an activity that even slightly reduces some particular existential risk does more good in expectation than engaging in an activity that will nearly certainly save a thousand lives today (emphasis added).

As a thought experiment in an idiosyncratic philosophical offshoot of utilitarianism, it may appeal to some. But does it offer any “evidence” that is remotely useful in helping rational people decide how to spend their money? While one can assume that McAskill did some kind of calculation to come up with the figure of 10¹⁶ itself, it is gross nonsense given the innumerable factors that could influence the size of future populations. Indeed, if one does a similar calculation extrapolating from the declining birth rates that I’ve written about in the past, one could come up with an equally plausible conclusion that there will no longer be any human beings on the planet in four or five hundred years. Or for that matter, any number in between zero and 10¹⁶. I don’t think either one is particularly likely, but that’s just my opinion.

But it is precisely the magnitude of his numbers – and 10¹⁶ is a very big number – that leads McAskill and his colleagues to conclude that any short-term action to ameliorate misery or prevent dire events which fall short of extinction in McAskill’s judgment, such as nuclear war or climate change, should be avoided in order to focus on reducing the existential risk of an extinction event in the future. As he argues, as paraphrased by MIT philosopher Kieran Setiya, “if you could save a million lives today or shave 0.0001 percent off the probability of premature human extinction—a one in a million chance of saving at least 8 trillion lives—you should do the latter, allowing a million people to die.” [5]

But in fact his numbers are no more than the sort of guesswork associated with alcohol-fueled college bull sessions, and, as was true for the quants who came up with the subprime lending models, McAskill appears to be indifferent to feedback effects; namely, how the outcome of (relatively) short-term events like climate change will affect the long-term trajectory of the human species. Even though humanity will probably survive in either case, the difference in both the numbers of people and their well-being in a world where global warming is capped at 2°C versus one where it reaches 4°C is literally incalculable.

Again, others have written about the broader pitfalls of longtermism, including Setiya and others.[6] Longtermism is unlikely ever to bring down global financial institutions, although by distorting many people’s philanthropic priorities, however, it could do real harm. It has also propped up some classic bad actors like Sam Blankman-Fried, for whom it served as a way to give his scams an altruistic gloss. But the entire edifice of longtermism, the central pillar of effective altruism, is based on pretending to quantify something that neither they nor anyone else can credibly quantify. Once again, bullshit quantification.

I’m not arguing that we should give up trying to quantify complex issues. There’s a big difference between what is very difficult to quantify, and what is simply not susceptible to being quantified at all. It is very difficult to quantify the potential effects of climate change, but those effects are quantifiable, albeit with a respectable margin of uncertainty. And whenever one is trying to quantify something that difficult, it is critically important to avoid introducing one’s biases and values. But that is an order of magnitude different from pretending to quantify what is unquantifiable. That is bullshit quantification, and even if it appears to be a harmless exercise, like the World Happiness Index, it is harmful, because it debases the meaning of data and fosters misinformation, however seemingly benign it may appear at first.

---

[1] https://georgewbush-whitehouse.archives.gov/news/releases/2008/11/20081115-1.html

[2] https://data.worldhappiness.report/table. The table also shows scores for a variety of factors related to well-being which are not part of the index, and which are not explained anywhere, as far as I could determine. I find it intriguing that Libya, much of which is subject to constant violence by dueling warlords, would have a higher ranking than Greece, and that Venezuela – from which roughly a quarter of the nation’s population have fled in desperation – would score better than many relatively peaceful countries.

[3]

The World Happiness Report Is a Sham

[4] William McAskill, “Effective Altruism”, in Hugh LaFollette, ed. International Encyclopedia of Ethics. John Wiley & Sons Ltd (2020).

[5] Kieran Setiya, “The new moral mathematics”, Boston Review, August 15, 2022. https://www.bostonreview.net/articles/the-new-moral-mathematics/

[6] See also Rebecca Ackermann, “Inside effective altruism, where the far future counts a lot more than the present”, MIT Technology Review, October 17, 2022, https://www.technologyreview.com/2022/10/17/1060967/effective-altruism-growth/