It should not have come to this. The Tories were serenely sailing along, oblivious to the need to connect with the public or outline much in the way of tangible policy. Just put your faith in Theresa May, they said, who is not Jeremy Corbyn, they reminded us, and all would be well with the world. That was just one week ago. A swathe of populist free-for-all policies and one U-turn later and, on Friday, YouGov struck fear into Tory souls with a 5-pointer.
Is it an outlier? Probably. Should we take serious note of it? Probably not. Does it indicate a pretty solid direction of travel? It certainly does. In anticipation of a numerous “Polls apart” headlines, I have to report that ICM’s poll in the Sun on Sunday had a 14-point Tory lead. Just to confirm, if your numeracy levels need some help, that’s not a 5-pointer: it’s 14 points, and it’s the same number that our previous poll had a week earlier. Big enough for a Tory majority of 126. But the polls are narrowing—we had a 20-point Tory lead no more than ten days ago—and suddenly can, justifiably, be said to be all over the place.
What on earth, I hear you ask, is going on?
At the conference, the numerous methodological changes enacted by all firms of pollsters in preparation for 2017 were outlined—not least by myself, on behalf of ICM.
Truly, as Sir Bob Worcester used to say about ICM—without as much vindication as he would have were he to say it now—we’ve thrown the baby out with the bathwater.
Here’s a quick walk-through of changes that have been made to the Guardian/ICM poll since 2015: we’ve switched from phone to online data collection; doubled our sample size; introduced a range of new quotas and weights; radically overhauled our turnout modelling; and introduced a post-data adjustment, that reallocates people who we know nothing about politically. It’s a pretty radical set of methods.
But there’s a catch; one which relates to that penultimate change. For were it not for the particularly strong turnout model employed, we too would be showing something very close to YouGov's 5-pointer, rather than the 14-pointer we have.
ICM’s view, which has been so long-held it pre-dates even my own 22 years in situ, is that polls intrinsically inflate Labour’s share—there’s more evidence of this than a stick can be shaken at—and finding ways to mitigate that problem is the responsibility of the polling agency. So, to summarise, YouGov are softer on turnout than ICM and have a 5-point Tory lead. ICM is probably the hardest polling firm on turnout and we have a 14-point Tory lead.
These differing numbers can be attributed to the philosophies and choices made by individual pollsters, rather than the raw data we collect. Of course, in many cases the combined effects of all tools at our disposal work to bring us together; today, we see that they have driven us apart.
The difficulty is that nobody really understands why the samples were unrepresentative, and if the problem is too complex to understand, you can bet the solution might be directed towards the wrong root cause. That’s a terrifying prospect in a close election; it’s not so much of one if a landslide is still the most likely outcome. This is why YouGov's poll is the proverbial cat amongst the pigeons. This was to be a cruise control election, in which our new methods would not be seriously tested, as there was only one winner in serious contention.
The idea of an unexpectedly close election, in which both May and Corbyn are predicted to be the next incumbent of Number 10, is just too concerning for pollsters to contemplate.
Is it an outlier? Probably. Should we take serious note of it? Probably not. Does it indicate a pretty solid direction of travel? It certainly does. In anticipation of a numerous “Polls apart” headlines, I have to report that ICM’s poll in the Sun on Sunday had a 14-point Tory lead. Just to confirm, if your numeracy levels need some help, that’s not a 5-pointer: it’s 14 points, and it’s the same number that our previous poll had a week earlier. Big enough for a Tory majority of 126. But the polls are narrowing—we had a 20-point Tory lead no more than ten days ago—and suddenly can, justifiably, be said to be all over the place.
What on earth, I hear you ask, is going on?
Changes since 2015
Polling models have changed radically in the aftermath of the 2015 General Election. In fact, that polling miss was mulled over once again at the British Polling Council/National Centre for Research Methods conference only last week.At the conference, the numerous methodological changes enacted by all firms of pollsters in preparation for 2017 were outlined—not least by myself, on behalf of ICM.
Truly, as Sir Bob Worcester used to say about ICM—without as much vindication as he would have were he to say it now—we’ve thrown the baby out with the bathwater.
Here’s a quick walk-through of changes that have been made to the Guardian/ICM poll since 2015: we’ve switched from phone to online data collection; doubled our sample size; introduced a range of new quotas and weights; radically overhauled our turnout modelling; and introduced a post-data adjustment, that reallocates people who we know nothing about politically. It’s a pretty radical set of methods.
But there’s a catch; one which relates to that penultimate change. For were it not for the particularly strong turnout model employed, we too would be showing something very close to YouGov's 5-pointer, rather than the 14-pointer we have.
Polls apart
It’s true that raw data can be moved materially from its base scores, and the strength and direction of the movement entirely depend on the intellectual and practical beliefs of the pollster in charge.ICM’s view, which has been so long-held it pre-dates even my own 22 years in situ, is that polls intrinsically inflate Labour’s share—there’s more evidence of this than a stick can be shaken at—and finding ways to mitigate that problem is the responsibility of the polling agency. So, to summarise, YouGov are softer on turnout than ICM and have a 5-point Tory lead. ICM is probably the hardest polling firm on turnout and we have a 14-point Tory lead.
These differing numbers can be attributed to the philosophies and choices made by individual pollsters, rather than the raw data we collect. Of course, in many cases the combined effects of all tools at our disposal work to bring us together; today, we see that they have driven us apart.
The cat among the pigeons
But the raw data we collect is, actually, the core problem. After 2015, the British Polling Council Inquiry identified “unrepresentative samples” as the cause of the polling miss, and all us pollsters have tried to address this problem in different ways.The difficulty is that nobody really understands why the samples were unrepresentative, and if the problem is too complex to understand, you can bet the solution might be directed towards the wrong root cause. That’s a terrifying prospect in a close election; it’s not so much of one if a landslide is still the most likely outcome. This is why YouGov's poll is the proverbial cat amongst the pigeons. This was to be a cruise control election, in which our new methods would not be seriously tested, as there was only one winner in serious contention.
The idea of an unexpectedly close election, in which both May and Corbyn are predicted to be the next incumbent of Number 10, is just too concerning for pollsters to contemplate.
