Once again, the iron law of opinion polls has asserted itself. At election time, they are judged not by how close they came to getting the numbers right, but whether they called the right winner. Last week, most polls showed Kamala Harris just ahead in the popular vote. In the event, Donald Trump came out ahead.
Worse (for the pollsters) their virtually unanimous prediction that the race was too close to call seems to have been undermined by Trump’s comfortable 312-226 lead in electoral college votes.
In fact, taking an average of the huge numbers of polls that were conducted at both national and state level, they did pretty well. These are the figures for Trump and Harris, excluding the small numbers who voted for minor parties, and those recorded by the pollsters as “don’t know”. (The figures below include my estimates for the votes yet to be counted, mainly in strongly Democratic California.)
National vote shares:
Polling average: Trump 49.4 per cent, Harris 50.6 per cent
Result: Trump 50.9 per cent, Harris 49.1 per cent
Error per candidate: 1.5 points
Vote shares in the seven swing states:
Polling average: Trump 50.2 per cent, Harris 49.8 per cent
Result: Trump 51.5 per cent, Harris 48.5 per cent
Error per candidate: 1.3 points
The error in the national share is typical of polls since the mid-1950s. This time they have been more accurate than eight, less accurate than four and done about as well in the other five. (Before the mid-1950s their errors tended to be far larger.) As for polls in the swing states, an analysis by ABC News has found that the polling errors were the smallest this century.
No doubt America’s pollsters will conduct one of their standard post-mortems into their performance—and so they should. Going by what has happened in the past, we can expect a thorough report sometime in 2026. Meanwhile, some early observations:
The polls did better than in 2020. Then they understated Trump’s support by two points, both nationally and in the swing states.
Had there been just one national poll last week, a 1.5 per cent error could be ascribed to chance. However, we had lots of polls. Had random error been the main problem, then we would expect roughly half the polls to understate Trump’s support and half to overstate it. Instead, virtually all the national polls, and the averages for polls in each of the seven swing states, were wrong in the same direction. This tells us that the industry’s bias was systematic, not just random.
However, last week’s polls could be as close as we will get. The days have long gone when polls could assemble high-quality, genuinely random samples. Response rates to face-to-face and telephone polls have collapsed in the past 30 years. Online panels are at the mercy of those who choose to join them. Pollsters have to extrapolate from the voters they can reach to those they can’t.
This means designing a model of the electorate, to match its characteristics by age, gender, education, geography, political affiliation and so on. The problem is that no model, however carefully designed, can be sure to capture every one of the quirks and wrinkles of any country, let alone one as varied as the US. The real puzzle is not why the polls got the election so wrong, but how they got it so close.
The exit poll seems to have erred in the same direction as the pre-election polls, and by roughly the same amount. Its voting numbers have not been disclosed, but we can glean clues from the information it has reported. On Tuesday evening, before any votes had been counted, we were told that Harris’s favourability rating was four points higher than Trump’s (46-42 per cent). By Wednesday afternoon, once it was clear that Trump had won, her lead was down to one point (44-43 per cent). Changes to other results told similar stories. The pollsters’ post-mortem should include the exit poll—not to castigate it but to compare its data to the campaign polls.
For example, the exit poll’s data might throw light on one of the main post-election suggestions of what went wrong in the campaign polls: that they overstated the determination of Harris’s supporters to turn out to vote. We know that turnout was down last week. Did some women who told the pollsters that they would vote, in fact not vote? If that were the case, then we would expect the exit poll to be more accurate, as its sample excluded non-voters. As it seems not to have been more accurate, the differential turnout theory may not explain the polls’ overstatement of Harris’s support. It’s worth trying to settle the matter.
A second law of polling is that the more surprising the result, the more news it makes—and the more likely it is to be wrong. Nine days ago, a Selzer poll in Iowa reported a three-point lead for Harris—a huge shift from Trump’s eight point margin of victory four years ago. Ann Selzer’s company has a good track record; if it was anywhere near right this time, then more conventional polls were seriously overstating Trump’s appeal. The Harris camp was cock-a-hoop. I saw more news coverage of this survey than any other single poll in the campaign. However, as I pointed out last Monday, Selzer does not adjust its samples to take account of education or political affiliation. I wrote that I wished its figures were right but doubted they were. In the event, Trump won the state by 13 points.
One significant outcome of the election is that the bias against the Democrats in the electoral college has disappeared, at least for now. Had Harris won Michigan, Pennsylvania and Wisconsin, she would now be president-elect. Instead she lost them by 1.4, 2.1 and 0.8 points respectively. Had Harris done 1.1 points better and Trump 1.1 points worse in every state, she would have won the national vote by 50.2-49.8 per cent, and the electoral college by 270-268. In 2016 Trump lost the nationwide popular vote by 2.1 per cent. The same performance last week would probably have deprived him of all the swing states except Arizona.
American polls are far less transparent than British polls. The British Polling Council, a trade organisation that I helped to set up 20 years ago, insists that its members make full data tables and weighting policies freely available. US polls are under no such obligation. Some are more open than others. A number sell their detailed information rather than providing it free. Some don’t even do that. This limits the opportunity for informed debate about their methods during the course of the campaign. A request to the American Association for Public Opinion Research: please adopt BPC transparency rules.
One final point. Nate Silver, who is arguably America’s most eminent poll analyst, caused a stir in the media, both traditional and social, when he accused the polling fraternity of “herding”. He said the swing state results produced by different companies were too close together. He argued that the probability of random samples producing such bunching was one in 9.5 trillion.
The problem with Silver’s argument is that, as noted above, today’s polls are models, not random samples. To illustrate the difference, imagine the following. Narnia is having an election. Half the population live in the north, and all vote for the Lion. The other half live in the south and all vote for the Witch. A pure random sample would be subject to the conventional margin of error (eg 3 per cent on a sample of 1,000).
But smart pollsters in Narnia wouldn’t conduct random samples. They would ensure that half their respondents live in the north, and half in the south. They would get the result spot on.
It’s not that simple in the US, but the dominant fact is the degree of Narnia-like polarisation. Where we have the breakdown, the polls agree that more than more than 90 per cent of those who backed Trump in 2020 have stayed with him, while more than 90 per cent of Joe Biden’s voters transferred their allegiance to Harris. The polls’ precise methods vary, but their models are designed to produce politically balanced samples, by past vote, party registration, or both. Done properly, this should reduce the risk of random error when the electorate is so polarised. In other words, we would expect a degree of bunching. To assume that today’s polling models obey the same statistical laws as purely random samples is like assuming that porridge flows as freely as water.
The real problem is not herding but factors that can cause systematic error: first-time voters that pollsters find hard to reach, specific demographic and geographic differences that they don’t account for in their models, differential turnout, out-of-date census data, respondents misremembering their past vote, and so on. But that’s a separate issue. If all the pollsters use similar methods (most do, to some extent), they are likely to end up with similar results (most do, to some extent). But Silver is accusing them of deliberate bunching, not poor methodology. Proof of jiggery-pokery by any pollster needs direct evidence. Silver hasn’t presented any.
