Think of noise and what comes to mind? Intrusion. Disturbance. Traffic. A dripping tap at 3am. Neighbours from hell. Increasingly, though, a different noise is in the air—a metaphorical noise meaning something that distracts us from important signals we really should be paying attention to.
Statisticians and engineers refer to the way electrical interference makes a radio crackle, or to the meaningless data that makes it harder to pick out what matters on, say, a graph. In 1983, this variety of noise seems to have been obscure enough for my two-volume, four-kilo Oxford English Dictionary to ignore. But by 2012, Nate Silver’s The Signal and the Noise showed that it could be an intellectually fashionable subject and, today, metaphorical noise has become standard clever-person jargon in both business and politics. “We need less noise,” says a colleague when his board papers are too thick.
In Noise, a new book by Daniel Kahneman, Olivier Sibony and Cass Sunstein, the word stands alone, alongside the claim by one of the authors to have “discovered a new continent.” (Tell that to the statisticians.) But hype and fashion aside, the problem of noise is genuinely fascinating. Here the authors find it running riot in the way our judgments vary.
They offer several killer examples: first, the alarming variation in sentencing by judges in the US for similar cases. This is not the kind caused by bias—such as harsher punishment if you’re poor or black—it’s variation across the board. “Justice is a lottery,” they say. Or take the two psychiatrists who independently reviewed 426 patients in hospital to decide which mental illness they had and agreed only half the time.
You might once have called this the problem of subjectivity. But that’s old school now. If you’re not soon calling it “noise” you might become noise yourself. Nobel Prize winner Kahneman wrote the bestseller Thinking, Fast and Slow. Sunstein and Sibony are in similar smart thinking territory. The star endorsements on the cover are of the “most important book in a decade” variety. Books like this are supposed to change how we talk.
But it’s worth taking a moment to ask why we should think of variation in judgment as “noise.” In this case, what exactly is the “signal”? Their answer is that it is we who get in the way of unvarnished truth. You can tell this, they say, because we disagree and, since we disagree, we can’t all be right. That means where there’s variation there’s error. Our varied judgments must be resting on all sorts of wrong-headed, often irrelevant or random influences—noise, in other words—which the research they quote suggests is very much the case. Those dripping taps and neighbours from hell are generated in our own heads.
“In public conversations about human error and in organisations all over the world, noise is rarely recognised,” say Kahneman and co. And yet, from sentencing in court to the quality of a wine or a student essay, the right price for insurance, which applicant should get the job or decisions in medicine, variation in judgment is everywhere even among experts.
But the problem is not just different judgments between different people. Asked on separate occasions to rate a student essay, you would typically be inconsistent. “You are not the same person at all times,” they write.
You might be influenced by a passing mood, the weather, the time of day, hunger, or watching a football match on TV—all forms of “noise.” In fact, the authors sometimes sound ready to write off human judgment altogether, pointing out how often machines or simple rules perform better, and how experts who think they’re adding sophistication to their judgments are often just adding more noise. Their goal is not to have us all replaced, they say. But they certainly think people could be more machine-like.
Noise has been hailed as a radical exposé of our failings and a revelatory guide to doing better, offering us new tools called “noise audits” and “decision hygiene.” Should we embrace all this, or is there reason for caution?
Both. Variation in judgment is a rich and perplexing problem that deserves more attention. The cost and injustice can be huge. Noise is engaging, full of ideas and interesting examples, and gives many well-deserved kicks to the vanity and overconfidence of much human judgment.
I hesitate first because some studies the book cites uncritically have been challenged by other experts as unreliable—including one about judges. Furthermore, there are several paragraphs of serious qualification that read a bit like the small print on a contract. For example, they introduce us to what they call “the error equation,” something so vital to the argument that they’re prepared to take readers unfamiliar with statistics through the concept of mean squared error. This equation shows that persuading all the actuaries in an insurance firm to move their varied judgments of the right price for a policy towards the average price—and thus to reduce noisy variation—produces less error overall, even if that average turns out in time to be wrong because the price they all now settle on is too high or low.
It’s counterintuitive: how can forcing everyone to converge on the same, unknown degree of wrongness be better than having a spread of judgment that probably includes what is right? But the maths seems to make sense because large errors have far more weight in their equation than small ones. Dragging everyone towards the average—even if it’s wrong—can pay off. Maybe the difficulty of this part of the book is why it has received little attention, but it’s critical, as it’s their justification for reducing variation in judgment almost regardless.
Then comes the small print: “The error equation does not apply to evaluative judgments, however, because the concept of error, which depends on the existence of a true value, is far more difficult to apply.” Erm… what? Because now I’m confused. Aren’t judges exercising precisely this kind of evaluative judgment when they disagree about sentencing? Isn’t this exactly a case where there is no true value and so the error equation—the intellectual foundation of the book—can’t apply? We’d still be right to call varied sentencing unfair, but how can there be noise in the sense that the true signal is obstructed when there is no objectively true signal? Maybe the old ways of describing varied human judgment as subjective or inconsistent have something to be said for them.
An illustration in the book shows a target with shots all to one side of the bullseye. This is bias, they say. Then it shows another target with the shots all over the place. This is noise. What it doesn’t show, but the small print acknowledges, is the picture where there is no true bullseye or target. So, what’s a good shot then?
Their answer is that you should still average everyone’s best guess.
But there’s also more small print: “Even if errors could be specified, their costs would rarely be symmetrical.” The meaning of this is soon spelled out: “for a company that makes elevators, for example, the consequences of errors in estimating the maximum load of an elevator are obviously asymmetrical: underestimation is costly, but overestimation would be catastrophic.” Likewise, when an insurance company prices policies, “errors in both directions are costly but there is no reason to assume that their costs are equivalent.”
Again, the implications are perplexing: one is that, contrary to their arguments elsewhere, it could be a mistake to converge on a price that could be wrong, rather than allow a spread of judgment that includes both right and wrong, as not all wrongs are equally harmful. Since at the time you might not know what kind of wrong your convergence point might be, you don’t know if convergence is better than remaining noisy, or if it could put you out of business. That’s why people sometimes hedge their bets, and why some outlying bets pay off royally.
But with all this qualification, tackling noise becomes rather harder than some of the authors’ more sweeping statements suggest: if studies that purport to identify sources of noise are unreliable, the diagnostic problem is evidently not straightforward; if subjectivity is often inevitable and there is no true signal, we’ll struggle to apply the book’s ideas to many areas of life where judgment varies; if error is asymmetrical, variation might be a lesser evil than convergence. Maybe even the example of conflicting diagnoses of mental illness doesn’t have a right resolution—given what they refer to as our “objective ignorance”—and so forcing uniformity on psychiatrists could be harmful to patients if that uniformity turns out to be uniformly wrong.
Simpler but less catchy than noise, objective ignorance has a lot to be said for it as the underrated reason so much human judgement and decision-making is poor and hard to improve. The somewhat obvious point that we vary wildly in our judgments because often we don’t know—and maybe can’t know—the reality is quite enough to make the case for more humility about our ability to get our judgments right. The further point that much of the reasoning we invoke to support these judgments is biased and random is largely the subject of Kahneman’s previous work. All that he has really done by calling this “noise” is to point out how our individual cognitive foibles add up to a collective mess. But often the problem is not that our foibles put us in the way of the true signal, it’s that there isn’t one—and so the foibles fill the gap.
In saying this, of course, I could easily be fooled by the noise in my own head. Anyway, none of it negates a word about how erratic our judgments are. Read this book but also take in the small print—it seems like a big signal to me.
Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony and Cass Sunstein (William Collins, £25)
