About EKOS Politics

We launched this website in order to showcase our election research, and our suite of polling technologies including Probit and IVR. We will be updating this site frequently with new polls, analysis and insight into Canadian politics. EKOS's experience, knowledge and sophisticated research designs have contributed positively to many previous elections.

Other EKOS Products

In addition to current political analysis, EKOS also makes available to the public general research of interest, including research in evaluation, general public domain research, as well as a full history of EKOS press releases.

An Unapologetic Analysis of the BC Polling Debacle

WHAT REALLY HAPPENED?

May 29, 2013

Click here for the full report: Post BC Election Review (May 29, 2013)


 

Introduction

The provincial election in British Columbia produced a major gap between the forecasts based on polls and the results of the actual election. While the polls had shown a narrowing race and we had an unweighted tie in our final poll, it appeared that the NDP still enjoyed a modest lead and that they were ticketed for some form of government. This was shockingly not the case and the BC Liberals went on to form a majority government. In fact, looking at the chart below which compares the recent election with the 2009 election we see that the previous election was a far better predictor of the final outcome than the polls. Which may lead the astute reader to wonder why bother with the polls at all – a question we are increasingly asking ourselves as well.

The election has variably been seen to show the triumph of negative advertising, the inability of all polls to actually measure anything reliably, the relative flaws of different polling methodologies, the presence of huge shifts in the electorate, the dire commercial consequences of this pratfall for the market research industry and a host of other claims.

It also has some pretty important implications for understanding the political prospects of apparently besieged incumbents in the current political and economic context.

In this article, we are going to use hard evidence and logic to try and sort through some of these often conflicting claims and to draw some conclusions about what all of this might mean for the future of political polling in Canada and the state of survey practice. We will also draw some conclusions about what this means about the current state of democracy in Canada. We will offer a practical solution which could simultaneously solve some of the biggest threats to the integrity of both polling and democracy in Canada.


 

Why We Shouldn’t Have Been Surprised
The risk aversion premium for incumbents

Let’s begin with a few observations about why the polling outcome shouldn’t have been that shocking to astute observers and why the gap between our poll and election result occurred. Consider the situations in the most recent elections in Canada’s four largest provinces. In each of those provinces, polls showed a picture of tired incumbents reeling from regime fatigue and other issues. As a rough summary, the incumbents all appeared to suffer virtually insurmountable deficits in the polls, some as high as 20 points. Come Election Day, however, those huge gaps were virtually eliminated and in two of these three cases, the incumbent actually won. So it is somewhat puzzling why we would be so astounded that exactly the same pattern happened again in the case of the BC election.

In the current political and economic landscape of Canada, it appears that maintaining economic stability is producing a major incumbent premium which is largely occluded from the pollster. It expresses itself as a consistent narrowing of the large incumbent deficits throughout recent campaigns and producing a significant turnout advantage. In the case of BC, the desire to keep the economy on track eventually eclipsed a desire for change, particularly among over 45 voters and men.

So it may be that the negative ads were the decisive factor in BC, but we must reconcile that with the eerily familiar dynamics in Alberta, Ontario, and Quebec where this apparent explanation would not apply. What appears to be truly common is risk aversion in an economically anxious public. Our survey will show that keeping the economy on track eclipsed the issues of ethics and social programs. Overwhelmingly this is why the final Liberal constituency voted for the party.

If we turn to the federal scene, this dynamic would seem to augur well for an apparently reeling federal government. It might also suggest that the buoyant hopes of rejuvenated Liberal supporters be tempered with a prudence factor. It might be reasonable to add 10 points to the polling target for electoral success. If anything, this challenge of success for the Liberals and NDP in the next election is further deepened by the (for them) inauspicious political arithmetic of three centre-left parties confronting a unified right. Those seeking to write Steven Harper’s epitaph might want to reflect on these points.


 

Disentangling Modeling the Population and Forecasting the Outcome
Forecasts an Increasingly sketchy yardstick of polling quality

Let’s return to the polls now. It appears pretty cheeky to title this discussion as “unapologetic” given the pratfall of the polls in predicting the election outcome and the fact that our final poll was not close to the final result (especially in the case of the Liberal outcome – we actually had the correct NDP number). And let me clarify that our final poll was a crappy guide to the outcome of the election and we are not dodging accountability for that. This forecast error deserves careful attention and correction if we are going to continue to enter the realm of election polling – something we will consider very carefully in the future.

This issue of forecast error is being conflated with the issue of survey error and the degree to which a methodology that accurately measures a known population can be constructed. Many have erroneously claimed that this is the true cause of the failure. I have heard some frankly fatiguing and pious pontificating about what types of polls work and do not work. We went back to the field and conducted a basic study which pretty definitively tells us what went on and what really explains the gap between our poll and the election outcome. I will not comment on the other polls in any depth as I do not have similar retrospective data.

A little historical context is in order here. The practice of assessing pollsters against election outcomes is a longstanding one and generally considered the blue ribbon standard for judging polling quality. The pollster sweepstakes that surrounded the nearness of final poll to election outcome once made a lot of sense. In the past, most people voted and the relatively small minority of those who did not vote were not systematically different. So the yardstick made sense. Neither of those conditions applies today.

In British Columbia, the electorate can be divided into two roughly equally sized groups – those who voted and those who did not. This is fairly typical of recent provincial elections. If the non-voters are very different than the voters, then a poll can be an accurate measure of the population of all eligible voters but a flawed predictor of the election outcome. In fact, it can become the case that more careful attention to the task of modelling all voters can reduce the appearance of quality for this very reason.

The confusion of forecasting and modeling the population is pervasive to the discussion of modern polling. It is a dangerous error that confuses two increasingly different tasks – modeling an overall population with scientific accuracy and forecasting an election outcome. It is unclear to me how the latter has become the much more important issue – particularly since everyone is going to know the real answer the next day, but let’s go along with this game.

Some of our colleagues more familiar with the internal dynamics of campaigns do not agree that forecasting is merely a vanity exercise for pollsters. They feel it provides crucial feedback on campaign performance. Similarly these practitioners are not as concerned with the vagaries of the non-voter who they consider anecdotally curious but not central to their role.

Even acknowledging that all eligible voters might have resembled our final poll: (1) how do we know that; and (2) why don’t we just focus on the voters? I will present our data as to why we know the answer to the first momentarily. I will also answer the second question with the modest but honest answer that in circumstances like the BC election we do not know how to do that; and frankly, despite claims to the contrary neither does anyone else.

We can get closer to answering the second question with more resources (larger and better samples, more diagnostic data, ongoing monitoring) but we really can not know this with much certainty. I will explain why but note that if anyone in my field could really predict next day decision making with any certainty they would not be wasting their time polling. They would be clipping coupons and living large from making a killing on the stock market (I note that all of the assembled supercomputers and Nobel Prize winners have never reliably solved the problem of what investors are going to do tomorrow with any certainty). In the case of predicting human behaviour based on complex decision-making where there is co-agency across the subjects being studied and the pollster’s research, it is very difficult. As Nassim Taleb has noted, much of success of this sort is based on chance. A simpler aphorism would be Yogi Berra’s note that prediction is really hard, particularly when it is about the future.

Fortunately, hindsight is easy and here is why we believe that our poll accurately measured the voting intentions of all voters. First of all, it was drawn carefully using random selection with known probabilities (which in our experimental work produces the same results across IVR and live CATI). We utilised a minimum of three call backs and weighted according to best practices. We included benchmark measures of representatives such as incidence of possession of valid Canadian passport. We randomised response categories to guard against straightlining. We included two fictitious rating questions and eliminated those who provided incorrect answers. We carefully sample both those with and without landlines and cellphones and we included both the offline and online populations. Looking at the demographics and other indicators things looked pretty good. So how did we get such an underestimate of the Liberal vote?

It does not appear that it was due to sampling or measurement errors. Two days after the election we went back into the field using the same methodology and guess what? We found that our sample gave us an accurate measure of the election result.

There was no trick weighting and we have made both our weighted and unweighted data publicly available. The methodology described above is scientific. It does not work one day and not the next. There are random errors but they are pretty minor. The fact that the exact same methodology captured the election result suggests that the method meets the basic criteria we seek – reliability (intersubjective repeatability) and validity (it measures what it purports to measure). In this case, while not perfect the methodology meets sound standards of reliability and validity. The myriad of claims about how IVR cannot be linked to sound survey methodology are in a word wrong. Our experimental testing shows that, as does the excellent work of properly applied IVR in the United States political world. IVR can be linked to sloppy or flawed survey methods as can mail, in person or live telephone. Properly applied, it works very well and for short surveys we prefer it to live interviewer telephone (which we also use).

There were no massive last minute shifts. Rather, the final results were a product of two factors. First, there was a steady and highly significant drift away from the NDP and Conservative supporters throughout the campaign. Second, the Liberals had a major turnout advantage while a lot of Green Party and Conservative Party supporters stayed. The NDP was evenly represented in non-voters and voters.

Given the large variations in vote intention between voters and non-voters, one may wonder why we simply do not focus on the voters. With great respect, I have received all kinds of advice on this issue. For example, “why don’t you just ask people if they are going to vote?” To which I respond, “Duh! Why didn’t I think of that?” and then quickly add that we have tried all kinds of variations on how likely one is to vote (as have others). What someone tells me about their likelihood of voting is of absolutely no value in predicting who actually votes. Intention to vote and voting behaviour generate an overall statistically orthogonal relationship – it has no predictive power.

Without boring the reader, we note that there is huge literature on this topic and we have tested extensively as well. Some things are helpful. For example, do you know where your polling station is? If you do not know 48 hours beforehand, you are not likely to vote. Excluding past non-voters is also useful, but this is a modest aid and can introduce errors.

Others have said in the past that ‘enthusiasm’ is a good predictor of voting. We tested this theory in the BC election, however, and found that NDP supporters were 30 per cent more enthusiastic than Liberal supporters. In Quebec, we found that enthusiasm magnified erroneous prediction of the election result by 50 per cent.

And herein lies the rub. The future need not resemble the past – Hume’s problem of induction. Incumbent premiums shift to throw out the incumbent premiums. Sometime positive emotions like hope dominate and other times it is fear and anger. In the United States presidential elections, the likely voters were understood well in the last election. With over a thousand polls, a large literature on prior linkages between turnout and other factors, a two-party system (not four), and a remarkably stable pattern of partisan voting the polls often get it right. In British Columbia, we had no ongoing tracking, relatively few polls, none of the resources required to do the in-depth diagnostics and tracking available to American pollsters. There is no “British Columbia theory” or model of voter turnout and the degree to which the polls can predict the outcome is based on a large degree of chance.

One of the party pollsters – Dimitri Pantazopoulos – claims to have accurately predicted the election outcome (and we have no reason to doubt the claim) and has generously provided feedback. His model involved using statistics from Elections BC to weight the data by the demographic composition of actual voters. We attempted to retroactively apply a similar weighting scheme to our own data but we saw only minor improvements.

We believe that a better approximation of likely voters is possible, but it will require more resources such as better tracking and better diagnostic tools. At the federal level, for example, there is more polling and more prior knowledge so the forecast side should be better. But I am still unconvinced that it is a good measure of polling quality. Electoral stock markets and “wisdom of crowds’ approaches have shown good success without any polling.

Polling should focus on measuring overall populations and separately focus on how to improve forecasts, which when they occur is largely just a cosmetic vanity game for the pollsters anyway. If we are going to be measured against prediction accuracy, perhaps there should be adequate resources to do so and the media should work harder to understand the difference between modeling the population of all eligible voters and making a reasoned conjecture at turnout and election results.

The current conflation may be having a corrosive impact on the quality of survey methodology and the soundness of market research. I have no doubt my polls would make better forecasts if I dropped everyone under the age of 45 from my sample, did not bother with the expense of cellphone-only households, and never spoke to first-time voters. Can you imagine how well such a methodology would work in the retail world, let alone the world of public policy?

If large portions of the population are systematically opting out of democracy (for example, younger voters are increasingly falling out of the fray), is it advisable to remove or under-represent them in polling? If their voices are not heard in elections, then is not it even more important that polling provide some voice for this growing zone of democratic silence? Can polling be more democratic than elections which are tilting toward oligarchy?


 

Conclusions

In a nutshell here is the story of the BC election and the pollsters. Overall pollsters failed to recognize a Liberal majority outcome. This is despite the fact that the result looked almost identical to the 2009 election and that the dynamics of surprising resilience for the incumbent had been seen in three earlier provincial elections —the two most recent which had produced similar pratfalls for pollsters. In our case we went back to the electorate to find out what happened. Our post-election poll was unique in that it talked to the entire population of all eligible voters. We discovered the following crucial findings:

  • The overall methodology generated the election result within the margin of error. Interestingly, this is true whether the results are weighted or not which suggests sound validity and reliability.
  • Although some may have had likely voter models which predicted the election we believe that it is exceedingly difficult to know who the voter is beforehand. In circumstances like BC it is virtually unknowable with the resources currently available.
  • The media and public are conflating the issues of accurate modeling of a population with predictive accuracy. We know who the population of eligible voters are, we don’t know who actual voters are till after the vote. We have found no likely voter model that correctly moves our last poll to the election result (including weighting by the demographics of turnout in the last election).
  • There are some things that can reduce prediction errors but they are imperfect and inadequate for dealing with large variations in vote intentions between voters and non-voters.
  • In the current political and economic landscape of Canada it does appear that maintaining economic stability is producing a major incumbent premium which is largely occluded from the pollster. It expresses itself as a consistent narrowing of the large incumbent deficits throughout recent campaigns and producing a significant turnout advantage. A desire to keep the economy on track eventually eclipsed a desire for change, particularly amongst over 45 voters, particularly males. This pattern is clearly evident in BC and present in most recent elections in Canada.
  • In British Columbia, the final result, which differed profoundly from pre-campaign polls was a product of significant defection of NDP and Conservative Party of BC votes to the Liberals. This occurred steadily throughout the campaign. While there were some late shifts these did not explain the gap between our final poll (they didn’t favour the Liberals). Rather the Liberals had a major turnout advantage and many GP and Conservative Party supporters stayed home. The risk aversion factor focused in motivated higher voting demographic segments was the key to this result.
  • These forces are clear in hindsight but the degree to which they can produce a sound likely voter model is questionable, The future need not resemble the past and we have nothing approaching a sound theory of voter turnout.
  • A better approximation of likely voters is possible but it will require more resources such as better tracking, better diagnostic tools. We once again recommend some form of consortium within the industry and media to provide a benchmark service to all those seeking sound evidence during campaigns
  • Younger voters are increasingly falling out of the fray in elections as senior ranks swell and young vote less.


 

A Closing Note on the Case for Mandatory Voting

Fortunately, I have an immediate fix for these linked scourges for polling and democracy. Mandatory voting as practiced in many countries would refocus both democracy and polling on where it should be focused – everybody. Pollsters would poll everyone and political parties would market policies and leaders to all. Vote suppression and the underbelly of the permanent campaign might be temporarily replaced with a renewed focus on developing more resonant policies for all parts of society.

I am not Pollyannaish enough to believe that this would fix these problems, but it seems like a timely and promising step in the right direction. In fact as one of my colleagues who reviewed an earlier draft suggested this was a cop out in light of the fact that there was zero probability of a mandatory vote in Canada any time soon The case for mandatory voting would be mixed in a normal and healthy democratic environment. According to the public our current democratic health is very poor. I therefore think that mandatory voting deserves to be on the agenda for the future. Saving polling would only be a trivial unintended impact. Ending vote suppression, re- engaging young Canadians and moving back from the ledge of effective oligarchy to fuller democracy would be the more profound consequences.

Click here for the full report: Post BC Election Review (May 29, 2013)

Leave a Reply