Why is democracy so surprising?

Electoral politics has been rife with surprises recently, most notably in the UK and the US. From the 2015 general election (which the Tories weren't supposed to win but did), through Brexit and the election of Donald Trump in 2016, and on to the general election of 2017 (which the Tories were supposed to win but didn't), event after event has confounded expectations.

Why were we so surprised, and what does that tell us about the current state of democracy? Is being consistently surprised a good thing or a bad thing for democratic politics?

At one level, democracy is meant to be surprising. It is designed to allow the voters regular opportunities to thumb their noses at their masters, by returning them answers they did not expect. It picks up on buried signals of popular discontent. Democracies may not always come up with the correct responses to the challenges that they face. But by being open to sudden changes of direction, they are good at avoiding getting stuck with the wrong ones.

That said, modern representative democracy also requires some degree of consistency and predictability in order to function effectively. Our bureaucratic and administrative machinery has to plan for the future. That is very difficult if the future keeps changing overnight.

Of course, not everyone has been taken by surprise recently. My mother repeatedly told me before the May 2015 general election that the Tories would win an overall majority, regardless of what the polls were saying, and she was right. I should listen to her more often. But almost no one saw all these events coming, and those who did were ignored by just about everyone else.

Why everyone got it wrong

In each case, six distinct categories of participants were taken by surprise by the outcome—let us call them the six “Ps.” The public, pundits, political scientists, pollsters, prediction markets and politicians all got it wrong.

Perhaps we should not be so surprised by the failure to foresee the outcome in the first four cases. The public decides the result of elections, but that does not mean the public knows what it is going to do before it does it. Journalists and commentators are in the business of reporting and commenting on news, not of predicting it. Political scientists are also not really in the prediction business. Their job is to make sense of what has happened. And polls have been getting elections wrong as well as right ever since modern polling was invented.

Prediction markets, however, should be different. In prediction markets, there is no other incentive than getting it right.

Nonetheless, on the day of each of the four votes in question, the actual result (a Tory majority, a Brexit win, a Trump win, no overall majority) was priced at about 15 per cent on the prediction markets, and in each case significant sums of money had been wagered on the outcome (nearly £120m was gambled on the EU referendum, the largest sum ever staked on a nonsporting event in the UK).

This means that while some people made large sums, equally large sums were lost by people who ought to know better. When something with a 15 per cent probability happens, it is surprising but not all that surprising: after all, it will come to pass roughly one in every seven times. But when four such events happen in a row, that is very rare indeed.

The combined probability of these four results, as priced on the day of each vote, was cumulatively 0.05 per cent, or 1 in 2,000. It could be argued that this is a false statistic as it treats each event as independent when in fact there was some correlation between them: Trump did not call himself Mr Brexit for nothing. Yet what is extraordinary is that these were the market prices when the markets had the information about the earlier results to draw on, so any correlation was already factored in.

What explains this failure? It seems likely that the real correlation was not between the events themselves but between the people betting on the outcome. Feedback loops and groupthink were at work. Punters were spending more time watching what other punters were doing than looking for outside sources of evidence. At the same time, political betting—unlike say betting on a major sporting event—remains a relatively niche activity closely intertwined with the financial markets: prices in each tended to track the other.

Let’s move on to why the politicians got it wrong. The fact that professional politicians were as dumbfounded by these results as professional gamblers is striking because their careers, unlike in the case of journalists and academics, depend on their being able to predict what the voters are likely to do, or at least to anticipate it. Yet nowadays, politicians are increasingly cut off from many of the wider currents of public opinion.

Education is one of the most significant lines of division in Britain; educational attainment turned out to be among the best indicators of likely support for Brexit, for Trump, or for Corbyn. While more than 90 per cent of MPs in the current House of Commons are university graduates, the majority of their constituents did not go to university (this is still true even of the 18–24 age group). Earlier routes into parliament for those who didn't go to university—trade unions for wouldbe politicians on the left, or the army for those on the right—are now relatively closed off.

Parallel worlds

However, there is still a puzzle here. We have more information than ever before about voter preferences and behaviours. So why does more information not lead to more predictability? Because with the greater availability of informational choice comes the greater extent of informational bias. The key issue is not so much the amount of new information available but, given the sheer volume, how it is filtered.

Digital technology is helping to create parallel worlds of information. We hear a lot about “echo chambers.” But the divide may be more basic than that. What for some people is news is for other people not even happening.

A comprehensive study by BuzzFeed showed that during the 2017 British general election the most shared stories on Facebook coming from the Labour side were about fox hunting and the ivory trade. The most shared image was a picture of Theresa May juxtaposed with a dead elephant. People who were not sharing these stories and images were simply oblivious to them. The Tories did not respond to this line of attack because they did not know the extent to which it was happening. Out of sight was out of mind.

Fake news and “microtargeting,” as an explanation for why politics is so surprising, do not offer much. That would take too much agency away from voters. We choose the information that suits us just as much as we have it chosen for us.

The unpredictability of tribes

If politics is tribal, with the sense that conveys of how deeprooted many of our allegiances are, why is it currently so surprising? Shouldn't tribalism make politics easier to predict?

In fact, tribalism feeds unpredictability in contemporary politics. In an age of tribal politics, the key question is not who switches allegiance but who shows up on the day. Microtargeting in the US presidential election appears to have been particularly effective in persuading some Democrats from the Obama coalition—particularly young African–American men and young white women—not to turn out for Hillary Clinton, because she was not one of them.

Also, we are less sure than we used to be who the tribes in our politics actually are. In the 2017 UK general election, the tribalism that motivated the vote was not purely party based. The age divide—primarily between those aged under and those over 45—was a large factor in shaping the outcome of the election, and some of that was tribal too: no one seeing the queues of young people turning out for Corbyn in university towns could be in any doubt that their sense of belonging was generational as much as it was party political.

Tribalism is also harder to read because it may be more transient than it was in the past. Digital technology can create powerful new kinds of identity that exist for a relatively short period of time.

A danger to democracy

So, is the “surprisingness” of contemporary democratic politics a good thing or a bad thing? Well, it’s positive that political elites cannot take their power for granted. What's more, it means that politics is not boring, and that keeps people engaged.

However, this run of surprises, especially if it continues, does seem to be a danger sign for democracy. It is not only the voters who have ceased asking for permission to do something surprising. Politicians are feeling increasingly liberated too, including Trump and the more iconoclastic supporters of a hard Brexit. Frustration with “the system” can lead democracy alarmingly close to authoritarianism.

Surprisingness, or randomness, are never enough on their own. Surprising politics needs robust institutions in order to thrive. We need to recognise that for every surprise that democracy throws up, there has to be a shared democratic framework to make those surprises workable. At present, it is not clear we have one.

 

Adapted from a chapter in Rethinking Democracy, edited by Andrew Gamble and Tony Wright (Political Quarterly Monograph Series, 2019). There is a series of free launch events throughout the year.