About EKOS Politics

We launched this website in order to showcase our election research, and our suite of polling technologies including Probit and IVR. We will be updating this site frequently with new polls, analysis and insight into Canadian politics. EKOS's experience, knowledge and sophisticated research designs have contributed positively to many previous elections.

Other EKOS Products

In addition to current political analysis, EKOS also makes available to the public general research of interest, including research in evaluation, general public domain research, as well as a full history of EKOS press releases.

Media Inquires

For media inquires, please contact: Frank Graves President EKOS Research Associates t: 613.235-7215 [email protected]

NOTES ON INTERPRETING POLL RESULTS – December 8, 2010

COMMENTS AND OBSERVATIONS BY FRANK GRAVES

[Ottawa – December 8, 2010] – In reporting on our past couple of polls, our favourite polling blogster, Kady O’Malley, has raised some legitimate questions about the credibility of our next to last poll (released November 11th). The basic argument was that the statistical tie wasn’t credible and that our most recent poll results (November 25th) showing a six point lead was “more like it”. We would like to respectfully respond to the question of whether or not this was a fair assessment.

The main evidence Kady offered was that large and “wonky” shifts in regional numbers brought into question the overall validity of the poll. Not only were the regional numbers suspicious, but this was used to question whether the overall results of a statistical tie were accurate. In short, the six point lead of last week was plausible but the putative statistical tie was not.

I would like to address some of these concerns. In so doing, I would like to point to some of the logical and statistical issues associated with reading the polls. Upfront, I will concede that these aren’t the most exciting or accessible points around polling, but they are worth considering.

First, it is important to remember that all surveys (even ours!) have errors. Errors are basically the difference between what we have estimated in the survey and what we would have found if we had measured the entire population with perfect measures. Errors come from sampling, measurement, and other method effects. Good surveys try and minimize controllable errors, but there will always be both systematic and random errors in every poll.

Our question here is whether the weight of evidence and logic favours the view that the Liberals and Conservatives were statistically tied versus the alternate view that the results of the following poll were, in fact, more reflective of what the previous poll should have found.

Let’s consider the presence of odd poll to poll regional shifts as a proxy for a wonky overall national result. There are some problems with repeatedly looking at poll to poll variations across regions. These problems relate to disentangling “real” changes from random variations and for using weird regional shifts as a yardstick to judge overall poll reliability. Finding counterintuitive departures from the real population means in small samples is an inevitable by-product of this approach.

To illustrate this point, let’s use a very simple example: flipping a coin. If you flip the same coin over and over, every once in a while, you will flip 15 heads in a row. It’s literally the luck of the draw. And finding such an improbable result is no indication that the broader series of trial is off, or that the overall number of heads and tails wouldn’t eventually converge at 50/50.

Most of the limitations of the week to week regional change tests relate to the problems of the relatively small (and, in some cases, very small) sample sizes available at the regional level. This can produce two problems. First is the risk of spurious claims of effects which are not based on real changes. This is fine at the level of reasoned conjecture because we can always look the next time to see if the hypothesis remains plausible. But it is the second issue that is of more concern here.

Inevitably (and this is a statistical law) there will always be odd departures where the statistics do not model the broader population parameters. This is simply part of the nature of the law of large numbers and the central limit theorem.

Think about the issue of testing the difference between one week and the next.

Each week, you are basically “testing” six regional change hypotheses. In some cases, these regions are modeled with very small sample sizes, often with a confidence interval (at the.05 or 95% level) of plus or minus 10%; that is a twenty point range. At the smaller sub-regional level, this will inevitably produce variations within the range of random variation which are quite large. Many of these will reflect real changes while others will simply be artefacts of random play. This problem is much less important at the national or large provincial level (e.g., Ontario). But to point to predictable random fluctuations and say that these are confusing and hence questionable (or even faulty) is unfair. It overburdens the data and unfairly brings into question the validity of the overall survey.

Of course, there will be odd movements in small sub-national samples from week to week. They are simply the naturally occurring fluctuations in the stochastic process of sampling. More reasonable is to consider the shifts in the bigger subpopulations and to look at those patterns over several recent iterations. To routinely pillory expected fluctuations in the Prairies and Atlantic Canada (which we rarely comment on for exactly this reasons) is to submit the polling to a level of scrutiny which is unreasonable and will be used unfairly by critics to question the overall results.

Occasionally, however, we will find a number which falls outside of the margin of error. Take the Atlantic sample where there was a huge and frankly implausible shift in NDP support. We have little doubt that the NDP went up in Atlantic Canada over the polling period. We are also reasonably certain that the increase was not 28 points. But here is the problem. If we are testing six regional hypotheses each poll period (BC, Alberta, the Prairies, Ontario, Quebec, and Atlantic Canada), we would pretty well expect by chance alone to find one outside the 95% level after just three weekly iterations. In fact, one in twenty will fall outside of this band by chance alone.

So going back and testing these hypotheses this way each poll is going to produce a “clanger” every three or four iterations. If it did not, we should be even more suspicious. A good safeguard would be to look at the overall patterns in the preceding polling periods to see what the natural oscillations are and to see if there are any reliable longer term patterns evident in the fuller time series. Better still would be to randomly split the sample in half and see if the results are evident in each random subsample.

So finding a predictable outlier should not be used to trumpet the “wonkiness” of the overall poll. In the cases of movement within the margin of error, there are reasonable explanations and examining the longer trend line and being mindful of the role of random variations would provide a fairer basis for assessing whether the poll is trustworthy.

So yes, the regional innards of polls will often appear odd but this is by no means a signal of broader integrity issues in the case of the overall national results. In the case of our penultimate to last poll comparison, all of the empirical and logical evidence suggests it is highly likely that we went from a near tie to a significant lead. To further reinforce this claim (which is what the statistical tests suggest) we offer two final observations. There was only one other national poll using probability sampling which was concurrent with our tied poll. This was a Harris Decima poll (conducted November 11th-14th) which also showed a statistical tie. So all of our evidence from our poll suggests a near tie and the only other cotemporaneous poll shows the same thing. Moreover, it is apparent on closer inspection that this modest volatility is restricted largely to less attached portions of the electorate (younger voters for example). The only contrary evidence is the presence of what are frankly totally expected variations across small sample size comparisons and the inevitable presence of type 1 error every one in twenty times.

The methods employed from week to week are intersubjectively repeatable scientific methods. There is little reason to believe that they worked one week but that the same methods misfired the next week. So the hard evidence and logic says it is far more likely that we had a virtual tie followed by an approximate six point lead.

3 comments to NOTES ON INTERPRETING POLL RESULTS – December 8, 2010

  • Wascally Wabbit

    Thank you Mr. Graves. I suspect that Kady wasn’t doubting you (or your numbers and/ or methodology) but simply trying to make sense – even knowing some of the theory – of those particular ones.
    From my perspective – I look at the general and broader trends – and try to put anomalies into context – much as Kady tried to do.
    Where I may do something that you may not do / should not do – is to compare your results with those of other pollsters – looking for patterns of bias – either because of methodology – the questions themselves (if the poll was not an independent one – i.e. paid for by a politically biased lobby group / party or media outlet.
    I’m enough of a groupie that there are polls – especially one for example that is run by a former political backroom boy – that tends to favour Conservatives by 2-3% EVERY TIME.
    I’m happy to say that I believe your poll results – and those of Nanos Research – and these are the ones I put most faith in!

  • Chris Chmelyk

    I am always astounded by the level of ignorance displayed by media types with respect to anything remotely having to do with statistics. Ms. O’Malley’s statements would surely strike anyone with even a basic grasp of statistical principles as absurd. Alas, far too many of the public are as oblivious as Ms. O’Malley seems to be about basic concepts that I remember having learned in high school.

    Thank you very much for your clear and succinct lesson, and I hope that Ms. O’Malley and her colleagues take the time to read it (assuming that literacy is not another skill somehow missed in their educations).

  • The use of the Nov 11-14 Harris Decima poll to validate the Nov 3-9 Ekos poll might have held up without the assertion that this was “the only other cotemporaneous poll.” There were actually two other national polls that were in reality contemporaneous with the Ekos poll, although neither covered the full period of the Ekos interviews. Nanos conducted a poll Nov 1-5 and found a 5 point spread between the Conservatives and Liberals. Ipsos conducted a poll Nov 2-4 and found a 6 point spread. These companies conducted interviews on days that overlapped with the Ekos poll – not several days afterwards, as in the case of the HD poll.

    That said, differences in the wording of questions and the treatment of leaners, etc will produce differences from one polling company to another. These discrepancies were vividly seen during the 2008 election, when many companies were polling daily. The results were as dramatic as 9 or 10 points of difference between companies in reported support for individual parties. One can see a scatterplot at this period at: http://www.sfu.ca/~aheard/elections/polls-scatter-plot-2008.html