“Two thirds of people hospitalised with Covid-19 in New York have been socially distancing.”
Maybe we’re thrown by his nugget of news, maybe we’re delighted, but we're all paying attention:
"Is this true?’’
It is. The mental gears turn…
"OMG, how could that be happening?"
…to a terminal conclusion:
"Social distancing doesn’t work!"
Naturally, there’s a "but." In truth, you knew there would be. Whether that "but" is welcome will depend on your prior views. If you instinctively bristle against social distancing, any "but" is awkward, muddying your newfound "proof" that it doesn’t work. If you approve of social distancing, you’ll be relieved to find that the claim isn’t all it seems.
The difficulty in a raucous environment is how to make sure the cautious "but" is heard over the OMG conclusion. If we’ve all become armchair epidemiologists now, this is the vital, but often overlooked part of our education: to learn how downright uncomfortable the armchair ought to be. If you’re not struggling, you’re doing it wrong. It’s a lesson that real epidemiology—all expertise in fact—must also continually relearn.
We want to know everything, urgently. We want the newest and latest. We want it especially if it proves us right. And—here’s the twist—so do they! Ego and hope are plentiful in science too. Questioning the evidence always takes patience. Questioning evidence that we like takes steel.
Research circles wrestle with exactly this dilemma: how to make sure the "buts" don’t get elbowed aside in the urgency to tell the world the latest finding, especially if it’s a finding they like, such as: "have you seen how effective our new drug is?"
Daniël Lakens, a statistician, is a case in point. He recently began the experiment of actually paying a red team to challenge his own work, positively inviting "buts" at every stage. As he wrote in the journal Nature earlier this week about Covid-19 research: "Finding ways to prove ourselves wrong is a scientific ideal, but it is rarely scientific practice. Openness to critiques is nowhere near as widespread as researchers like to think." The best time for scrutiny, he adds, is before you have fallen in love with your results.
In similar vein, John Krakauer, professor of Neuroscience at Johns Hopkins University School of Medicine, says of the work in his lab: "I encourage an increasing sense of unease."
In a crisis, the pressure is on. But go too fast and we might cause harm. Conclude too readily and we often conclude wrongly. The onus on the epidemiologist—whether of the qualified or the armchair variety—ought to be to remain humble about how far from full our understanding is. "But…," said sincerely, is more than ever an ethical and intellectual duty, however inconvenient that may be.
Let’s start with the easier cases, the anecdote that most of us can see right through, but desperation or vested interest could lead some to believe:
99 PER CENT NONSENSE
"Hey, someone who ate 16 oranges* recovered overnight!"
(*insert your preferred miracle cure, here).
Or the pitiful extensions of the same idea from people (or indeed Presidents) who should know better:
"Hey, we treated six people with hydroxychloroquine (without proper controls) and they got better."
The "but" in these cases is, of course, that 99 per cent or more get better anyway. That shouldn’t need spelling out, although sadly the fact that interventions of all kinds—medical, psychological, social—can also cause harm does seem to need to be repeated again and again.
Next, though, we move up a level to those tempting observations that might need a little more reflection, as they sometimes contain a partial truth:
"Hey, have you seen the people in intensive care? They’re not that old. It’s terrifying.
Must be much riskier than they say for the middle aged!"
The partial truth is that everyone is at some level of risk, and the evidence from hospital ICUs brings that home, but… you don't have to think too long to suspect that the sample of people in intensive care might exclude a great many of the elderly who are dying without even getting there. So, maybe don’t look here for a true measure of what the risks are.
As the "buts" get more complicated, they become more interesting, but they can easily slip past if we’re not alert. Like the one we started with.
"Have you seen that two thirds of the people now coming into hospital have been practising social distancing? (reported in New York). OMG social distancing doesn’t work!"
Here we have to think hypotheticals and baselines—and that takes imagination. Let’s take an extreme hypothetical to make the point. Imagine that in every 100 people, 99 are socially distancing, one isn’t. The one who isn’t finishes up in hospital, along with two who are (2 out of 3 … two thirds of the total are socially distancing! OMG!). But… the risk of being hospitalised if you’re distancing is now revealed as just over 2 per cent (2 in 99) compared with, if you’re not distancing, 100 per cent (1 out of the 1 in every 100).
The real world won’t be that extreme, but we see from the example how the numbers work. Well, we do now, but it took effort. And the attraction of such grabby newsbites (or even initial scientific findings) is that they can lure us into thinking that no effort is required, that things which appear interesting can be relied on to be revelatory. And the required effort only ramps up as the task grows trickier again.
ENGLAND EXPECTS
"Have you seen how high the mortality rate is in young people, but only in England?! OMG, what are we doing wrong in England that kills young people?"
The claim is not only grabby but may prompt instinctive worries about whether our own government is for some reason failing to do something for young people that is taken care of elsewhere. But take the time to examine a whole pile of data (or, as I usually do, find someone else who’s already done it), and we see that most of the English Covid-19 cases are in urban areas (London especially), and people in urban areas tend to be younger. There are relatively more young deaths than we might have expected only because the virus has spread most where the young people are, not because English young people are either intrinsically (or by virtue of how we are treating them) somehow unusually vulnerable.
And then a final case, where we have to peer into the methodology of a study ("help!") and appreciate subtleties around causation that often escape us.
"Have you seen this study that says the Covid-19 risk for smoking is lower than normal? OMG! Take up smoking!"
It turns out that this study, a piece of technical brilliance, set out to calculate from a mountain of data every risk relationship independent of every other. Sounds like a good thing. But… this meant calculating the risk of smoking after separating other risks like heart and lung diseases—precisely the kind of big things caused by smoking that make it so unhealthy in the first place. At least, that’s the latest "but." There might be more, on both sides of the argument—another reminder never to grow too comfortable in the armchair.
A BIAS FOR EVERY TASTE
There are dozens more examples we could cite. Most of these (not the oranges) have caught out the experts and armchair experts alike. Researchers themselves often overlook an already-known variety of "but" (among the more recent I finally caught up with—you never stop—were "collider bias" and "immortal time bias." Check them out).
There is, in sum, many a dangerous slip between evidence and inference, but what on earth to do about it?
Social media, where the armchair argument takes place, produces plenty of cleverness at high speed. On the other hand, it has also been a naive OMG machine. Increasingly quick-fire academic research covers the same range. Some is genius. Some has wiser critics pulling their hair out.
The bottom line is that a critically-informed data literacy has become as important as any other kind. Number-based claims are now the language of public argument, and without at least some skill in how to sleuth our way through the evidence, we’re stuffed. One of the first skills, for the armchair, the lab, or the dataset, is how to exercise responsible doubt, especially when it clashes with our own interests or beliefs.
Eleanor Murray, an epidemiologist with a particular interest in causal inference (@Epiellie), had this advice recently: “We should never just accept the numeric results at face value... think of all the ways we could have gotten that number by accident or mistake… and see what we learn along the way.”
"All the ways we could have gotten that result" covers a lot of ground. Learning how to doubt, scientifically—regardless of whether doubting an answer suits us—is an education that never ends and is best started young.
I’ll offer one small contribution to the curriculum: beware those who are certain. Pseudoscience and its friends tend to admit to no uncertainty, no buts. Real science doubts, every step of the way. And one of the best ways to start practising that is with our own convictions.
Michael Blastland is the author of The Hidden Half: How the World Conceals its Secrets (Atlantic)
