By Frank Graves
This article was published in the January-February 2017 issue of Policy Magazine.
In the wake of Donald Trump’s unexpected election victory, pollsters are again taking a beating for getting it wrong. In the Brexit referendum, the last U.K. election and recent Canadian elections, both federal and provincial, pollsters have been blamed for getting it wrong. EKOS President Frank Graves offers this explanation for what happened with the polls in the US election campaign.
[Ottawa – January 4, 2017] United States presidential elections are the Super Bowl and World Cup of polling. The stakes don’t get any higher and, in this historic election, the consensus predictions based on the polling were a near-certain Clinton win. In stunning defiance of the weight of “scientific” probabilities, Donald Trump fashioned a pretty clear victory in the Electoral College. Oopsie! Epic failure, once again on the part of the increasingly sketchy polling industry. But wait a minute; that may be a serious distortion of what really happened.
In this non-technical review I am going to try and sort out what actually worked and what didn’t. Although the detailed post-mortems have yet to be concluded, it is actually pretty clear what went wrong and why. Apart from the question of whether the polls blew it, we need to look at some broader questions of what the polls told us about where society and politics are headed.
We may be on the precipice of a new dark age but of the eruption of incredibly insightful and important analysis, much fuelled by the polls, has been of profound value. We can now sort out much more clearly what happened and why and it would never have been evident without polling.
1.) Polls can still accurately model populations
On the issue of the ostensible failure of the polls, we argue that the depiction of a massive pratfall on the part of the pollsters is egregiously overwrought. There were some spectacular prediction errors, as there often will be when turnout is not high and the preferences of non-voters are systematically different than those of those who showed up. We can still model a known population (e.g., all eligible voters) but we will continue to have difficulty guessing who will actually show up; the unknown population of actual voters. The aggregate polls suggested that Hillary Clinton would win the popular vote by about three points. She will win by two, so that’s pretty close.
2.) The real problem was one of prediction
The problem wasn’t the polls; it was the aggregators and predictors.
These aggregators include:
- The New York Times, which predicted an 85 per cent chance of a Clinton victory
- FiveThirtyEight, which predicted a 71 per cent chance of a Clinton victory
- The Huffington Post, which predicted a 98 per cent chance of a Clinton victory
- PredictWise, which predicted an 89 per cent chance of a Clinton victory
- Princeton Election Consortium, which predicted a 99 per cent chance of a Clinton victory
- Daily Kos, which predicted a 92 per cent chance of a Clinton victory
For years, I have watched the aggregators borrow our polls and put them into their aggregation models. They draw their own conclusions from others’ data without consulting – let alone paying – the firms which collected the data. Summarizing the averages and breakdowns of the polls is fine but many of the aggregators go beyond this and apply predictive models and seat forecasts, which are increasingly used by voters who may wish to vote strategically.
In Canada, our record of seat forecasts at EKOS has consistently been better than those of the aggregators despite the fact that they are drawn from a single polling company. In fact, at the federal level, we have never had the winner of the contest wrong (as all of the aggregators did in the US). Our worst error was calling a strong Conservative minority in 2011 (along with everybody else). The surprising majority reflected some of the same prediction problems that plagued the U.S. presidential forecasts. In a nutshell, when turnout is relatively low, and there are systematic differences between actual and non-voters, we are left with making conjecture. From my perspective, it is pretty clear that we aren’t very good at that when those conditions exist.
The U.S. election provides an even more vivid illustration of the hazards of putting too much confidence in the aggregators’ forecasts. I thought it only fair to aggregate the most influential aggregators and see what the overall forecasts were. Their logic (with some rationality) is that the averages across various polls will be more reliable than using a single source. So if the aggregation of polls is sound then the aggregation of aggregators should be even more powerful. What could possibly go wrong?
If we aggregate the average probabilities of the six major US aggregators that offered precise predictions we would find a range from 71 to 99 per cent with an aggregate average probability of around 90 per cent likelihood of a Clinton victory. The pollsters may have had some shortcomings but they pale against this epic failure. Clearly these predictions were egregiously wrong and the stated probabilities were fiction, not science. Worse (or better if you were a Trump supporter), it is highly likely that many weakly motivated Clinton voters stayed home on the spurious assumption that it didn’t matter whether they showed up.
So let’s cut the pollsters a break and pin this tail squarely on the donkey that deserves it. I don’t mind defending and correcting errors that stem from our work. But I am getting really tired of hearing about this huge polling failure which, in this case, was manufactured by aggregators. Here are a few suggestions for the future:
- Be more modest in prediction claims and perhaps focus on summarizing the data that you ‘borrow’ from those who actually design, collect, and analyze the data that you so artfully manipulate; and
- Let those voters who are deciding on whether and how to vote know that you often really don’t know what is going to happen.
3.) Polls measure voter intention, not voter behaviour
Let’s move on from the blame game and try and discern why the prediction errors and what the polling did reveal that was important and new. Some have speculated that the problem was, in large part, a measurement error. Another variation on the ‘shy Tory’ concept. It may be that the same institutional mistrust that was a driver of Trump vote also caused those voters to not participate in polls or park themselves in undecided. One of the firms that got it right found that asking whether your neighbour was going to vote for Trump more accurately reflected his real strength. One possible flaw in the shy Tory type explanations is there wasn’t much difference across live interviewer and IVR/Online. Social desirability (linked to shy Tory) should be much more evident with live interviewer and it really wasn’t.
Five Thirty Eight’s Nate Silver suggests it was a sampling error and there were simply not enough poorly educated males sampled. The IVR to landline surveys allowed Trafalgar to predict key swing states more accurately (possibly linked to this factor).
Personally, I believe the major flaw was one of prediction, not polling. As Pew has shown recently, we are actually doing a better job of modelling voters with probability samples than we were 10 years ago. The growing skepticism of the media and pundits in polling is not rooted in any scientific evidence that probability samples don’t continue to work, despite declining response rates and the proliferation of cellphone-only households.
The problem is that the population of actual voters is an unknown (until Election Day). Let me be blunt: we don’t really know who is going to show up on Election Day. When turnout is low to modest, and voters differ from non-voters, we are going to see errors. Asking people whether they are going to vote is useless. We might gain a little insight by asking if they know where their polling station. One of the best predictors is that past voting behaviour – serial voters – generally remain such as do serial non-voters; until they don’t. It turns out that a lot of lapsed Republican voters who haven’t voted in years actually showed up this time, which threw the likely voter models off.
We have no unified theory of voter turnout and it isn’t on the horizon. We face Hume’s problem of induction – the future doesn’t necessarily resemble the past. Or as Yogi Berra more pithily summarized, prediction is really hard, particularly about the future.
4.) This focus on prediction obscures the real value of polling
The exit polls (which solve the problem of the unknown population) provided some interesting answers to these questions. For example, we heard that various expressions of xenophobia and nativism, even white supremacism, were critical drivers. It appears that they were factors but almost certainly driven by more primordial economic and social class forces. Trump won many more states, but the total economic output of those states was roughly half that of the fewer states Clinton won. Trump also did better than Romney with blacks and Hispanics, which weakens the view that racism was a critical driver.
Intriguing analyses have connected both Trump and Brexit to growing resentment and anger due to middle class decline and economic stagnation; resentment to the kids at the front of the class who have fared fine while everyone else has stagnated or fallen backward. Joan Wilson notes the critical role of class consciousness in her excellent review. Consistently, research in the UK and U.S. is pointing to new dark forces that are linked to economic stagnation and inequality, but express themselves in a disturbing rise of authoritarianism. In the 1960s, Daniel Bell argued that the ordered versus open worldview was displacing traditional left-right tensions. That prediction may have finally come true, but with a new interdependence between these views and the left-right spectrum. This coalesces to produce a highly uncertain complex of unpredictable social and political forces.
Polling may not help us predict the future but it sure is helpful in understanding what is going on and how we got here.
There are four key conclusions to be drawn from this analysis:
- Polling can still accurately model populations (although there are new challenges and probably more sketchy polling today than in the past).
- The last election polling in the United States wasn’t great but it wasn’t terrible. Polls don’t measure electoral colleges they measure voters so perhaps that is one of the problems. The polling was closer than 2012 and overall within the margin of error.
- The real problem was one of prediction. Some pollsters made bad predictions but the truly terrible errors came from the aggregators, not the pollsters.
- Finally we should reduce our fixation on predicting the result (which everyone will know the day of) and try and understand what is going on with citizens. What were the rhythms and forces which drove the voters (and non-voters) and what do they tell us about how society and politics are evolving?
Please click here for a PDF copy of this article.