The incoherence of “evidence-based policy-making”

“Our aim is to pursue policies that are evidence-based,” proclaimed the Independent Group in its founding statement. It’s not exactly the part of the declaration that got the most attention, for two good reasons. First, isn’t everyone in favour of evidence-based policy? What’s the alternative? Government by gut feeling? Administration by intuition?

Second, we’ve heard all this before. In the early days of New Labour’s government, for example, the 1999 Modernising government white paper declared that “policy decisions should be based on sound evidence.”

It’s telling that the Blair administration made the most noise about Evidence-Based Policy-Making (EBPM). For New Labour, the emphasis on evidence was supposed to signal a break with ideology. “This government has given a clear commitment that we will be guided not by dogma but by an open-minded approach to understanding what works and why,” as David Blunkett said in a 2002 speech.

The Independent Group is taking the same line, saying its aim is to pursue policies “not led by ideology.” The trouble is that this aim is incoherent. Of course policy should be informed by evidence, but evidence alone can never determine what the right policy should be. As David Hume pointed out nearly 300 years ago, you cannot leap from facts to values. No amount of information about how things are can tell you how they ought to be.

To take an obvious example, the evidence could tell you the probable effects on the government’s tax take of various different fiscal changes. But once the evidence is in, only “ideology”—or to give it a better name, political judgment—can decide which policy is best. Whether the priority is to reduce government spending, reduce inequality or soak the rich is a political choice, not a technocratic one.

In that sense values, not data, are the basis of policy. The job of evidence is to inform policy-making to maximise the chances that any given reform achieves the ideological goal. It cannot tell us what that goal should be.

That is not the only problem with declaring that policy has to be evidence-led. The truth is that when making any genuinely innovative reform, the evidence is rarely clear anyway. Even when a policy’s been implemented elsewhere, you can never be sure that what works in one socio-political ecosystem (Singaporean maths teaching, for example) will work elsewhere. The principle that a government should never do what evidence strongly suggests won’t work is a good one, but if it only did what was proven to work, it could do little new at all.

Pilots, although sensible, are even less reliable than looking abroad. Experimenter bias means that even the most sincere researcher is likely to skew the findings unintentionally if they are invested in a positive result, which most people running pilots are. Then there is the famed Hawthorne Effect which means many changes bring a short-term benefit simply because change shakes people up.

Perhaps the biggest single reason not to place too much faith in evidence is the replication crisis which has rocked all of science, most visibly psychology and medicine. EBPM models itself on science, but if even in controlled laboratory conditions, apparently conclusive experiments can turn out to be fatally flawed, it would be foolish to put too much faith in real-world policy experiments which are conducted in anything but controlled conditions.

Even when the evidence is robust, when you ask the question “which evidence?” matters quickly become unclear again. What you are measuring might show an improvement but who knows the effects on what you don’t measure? There are policies that will reduce unemployment but decrease the quality of employment. Some benefits that produced an immediate and measurable improvement on claimants’ lives created a welfare trap that caused great harm in the long term. And while you might be able to measure the economic impacts of immigration, there are social impacts, negative and positive, that defy quantification. There is always a danger of finding the evidence you’re looking for and missing the evidence you're not even aware might be there.

All of these problems assume that the search for objective evidence is sincere. Sadly, that is rarely the case. What we often see is what Rebecca Boden and Debbie Epstein have called “policy-based evidence.” In other words, the government or a minister is determined to push a certain policy and they go looking for any evidence that will support it, ignoring any that doesn’t. Even when there is no intentional cherry-picking, no one is free from the unconscious influence of confirmation bias.

Of course it would be a very good thing if governments paid more attention to evidence. The prison system and drugs policy would both be much better if only expertise had trumped the knee-jerk moralistic attitudes of politicians and voters. But to imagine there could ever be a form of evidence-based policy-making that could rise above ideology shows naive faith in the power of evidence and too little understanding of how facts differ from values.

The final flaw with EBPM is that it sounds good to everyone until the evidence suggests something they don’t like. It’s telling that the only branch of government to embrace the principle of EBPM with gusto has been the Department for International Development, delivering projects thousands of miles away from UK constituencies. DfID includes “evidence based” as one of the ten principles governing its “smart rules” for better programme delivery.

The Independent Group will soon discover that while everyone applauds EBPM in theory, when any policy directly affects the electorate, sound evidence always loses out in the battle with what sounds best.