FRANK GRAVES RESPONDS TO EVALUATING THE POLLS: AN OPEN LETTER TO ONTARIO’S JOURNALISTS BY DARRELL BRICKER AND JOHN WRIGHT
[Ottawa – September 16, 2011] – Darrell Bricker and John Wright have recently taken a pretty harsh scattergun to many of their media polling competitors. While they score a number of direct hits on polling, and the media, they also say some things which are simply unfair. Given the failure of public opinion polls to accurately predict the 2011 election results, the frustration expressed by Mr. Bricker and Mr. Wright is understandable. Some of the claims that have been made regarding Interactive Voice Response (IVR) technology, however, are wildly inconsistent with the facts.
It is important to remember that not all polls are equal. While there are specious illustrations of almost all types of polls it is unfair and inaccurate to label all IVR polls as unreliable. IVR is no more a polling method than is internet or live operator. There are bad versions of all types of polls: there are bad phone surveys, bad mail-out surveys, and yes, bad IVR surveys (though we tend to agree with Mr. Wright and Mr. Bricker that there some currently formidable problems with opt-in online polls). The question, however, lays not with the method, but with how well the method is implemented. Good polling should be based on random probability sampling which covers the entire target population with equal probability of each member of the population appearing in the sample. Declining response rates, rising incidence of difficult to reach subpopulations such as those who have cut their land lines in favour of a cell phone, and the conspicuous absence of the nearly one in three who never use internet from all online polls are just some of the challenges. On top of this both the resources devoted to polling by the media have dramatically declined, as has the methodological fluency of major media outlets.
Despite these challenges it is still possible to do good polling and the search for new methodological innovations to deal with these challenges should not be dismissed without careful assessment. We at EKOS were quite sceptical about IVR polling which is now standard fare in the USA, and we remain unconvinced about opt in online polling. We have, however, have good success in adapting traditional polling methods to the economic and coverage challenges posed today.
At EKOS, we always provide all weighted data and we have made our unweighted data available when requested. In our hard testing our IVR methods are as good as or better than our live CATI methods (we use both and have rigorously tested in parallel for years). This is true both in terms of data quality and that is the conclusion is shared by the no less than the American Association of Public Opinion Research (AAPOR) . Obviously, these conclusions only apply when sound sampling and measurement approaches are applied. We didn’t have ‘massive’ errors in our last polls, although we did underestimate the final actual vote because there were highly unusual correlations between propensity to turn out and vote and support for the Conservative Party.
For example, we randomly covered cell phone only households. Good sampling covers all members of the population and does so in a random method. It turns out that cell only respondents are both more likely to not vote and to not support the Conservative Party. This may have been an error in forecasting, but it was based on better modelling of the overall population of eligible voters.
In conclusion, the sweeping depiction of all IVR is egregiously unfair and inaccurate in our case. We did parallel tests and refinements to the methods for an entire election before bringing it out in public. We are well aware of some specious IVR applications but we are scrupulous in our methodology and no one provides more disclosure than us. Apart from a final forecast error last time which we have clearly acknowledged (and which was not based on bad polling data but bad understanding of turnout dynamics), we have an excellent record of getting elections right. Indeed, one need only look to the 2010 Toronto mayoral election where CATI polls were calling a statistical tie between Rob Ford and George Smitherman. Indeed, EKOS’ IVR poll stood alone in accurately predicting Ford’s landslide victory. A number of our own background reports and references to the expert literature are appended to this note.
Click here for the full report: press_release_september_16_2011
A NOTE ON IVR POLLS
FRANK GRAVES RESPONDS TO EVALUATING THE POLLS: AN OPEN LETTER TO ONTARIO’S JOURNALISTS BY DARRELL BRICKER AND JOHN WRIGHT
[Ottawa – September 16, 2011] – Darrell Bricker and John Wright have recently taken a pretty harsh scattergun to many of their media polling competitors. While they score a number of direct hits on polling, and the media, they also say some things which are simply unfair. Given the failure of public opinion polls to accurately predict the 2011 election results, the frustration expressed by Mr. Bricker and Mr. Wright is understandable. Some of the claims that have been made regarding Interactive Voice Response (IVR) technology, however, are wildly inconsistent with the facts.
It is important to remember that not all polls are equal. While there are specious illustrations of almost all types of polls it is unfair and inaccurate to label all IVR polls as unreliable. IVR is no more a polling method than is internet or live operator. There are bad versions of all types of polls: there are bad phone surveys, bad mail-out surveys, and yes, bad IVR surveys (though we tend to agree with Mr. Wright and Mr. Bricker that there some currently formidable problems with opt-in online polls). The question, however, lays not with the method, but with how well the method is implemented. Good polling should be based on random probability sampling which covers the entire target population with equal probability of each member of the population appearing in the sample. Declining response rates, rising incidence of difficult to reach subpopulations such as those who have cut their land lines in favour of a cell phone, and the conspicuous absence of the nearly one in three who never use internet from all online polls are just some of the challenges. On top of this both the resources devoted to polling by the media have dramatically declined, as has the methodological fluency of major media outlets.
Despite these challenges it is still possible to do good polling and the search for new methodological innovations to deal with these challenges should not be dismissed without careful assessment. We at EKOS were quite sceptical about IVR polling which is now standard fare in the USA, and we remain unconvinced about opt in online polling. We have, however, have good success in adapting traditional polling methods to the economic and coverage challenges posed today.
At EKOS, we always provide all weighted data and we have made our unweighted data available when requested. In our hard testing our IVR methods are as good as or better than our live CATI methods (we use both and have rigorously tested in parallel for years). This is true both in terms of data quality and that is the conclusion is shared by the no less than the American Association of Public Opinion Research (AAPOR) . Obviously, these conclusions only apply when sound sampling and measurement approaches are applied. We didn’t have ‘massive’ errors in our last polls, although we did underestimate the final actual vote because there were highly unusual correlations between propensity to turn out and vote and support for the Conservative Party.
For example, we randomly covered cell phone only households. Good sampling covers all members of the population and does so in a random method. It turns out that cell only respondents are both more likely to not vote and to not support the Conservative Party. This may have been an error in forecasting, but it was based on better modelling of the overall population of eligible voters.
In conclusion, the sweeping depiction of all IVR is egregiously unfair and inaccurate in our case. We did parallel tests and refinements to the methods for an entire election before bringing it out in public. We are well aware of some specious IVR applications but we are scrupulous in our methodology and no one provides more disclosure than us. Apart from a final forecast error last time which we have clearly acknowledged (and which was not based on bad polling data but bad understanding of turnout dynamics), we have an excellent record of getting elections right. Indeed, one need only look to the 2010 Toronto mayoral election where CATI polls were calling a statistical tie between Rob Ford and George Smitherman. Indeed, EKOS’ IVR poll stood alone in accurately predicting Ford’s landslide victory. A number of our own background reports and references to the expert literature are appended to this note.
Click here for the full report: press_release_september_16_2011
Tweet
Share this: