About EKOS Politics

We launched this website in order to showcase our election research, and our suite of polling technologies including Probit and IVR. We will be updating this site frequently with new polls, analysis and insight into Canadian politics. EKOS's experience, knowledge and sophisticated research designs have contributed positively to many previous elections.

Other EKOS Products

In addition to current political analysis, EKOS also makes available to the public general research of interest, including research in evaluation, general public domain research, as well as a full history of EKOS press releases.

Media Inquires

For media inquires, please contact: Frank Graves President EKOS Research Associates t: 613.235-7215 [email protected]

An Analysis of the Effects of Weighting by Education

[Ottawa – October 31, 2013] Over the last few days, a handful of pundits have meticulously reviewed the results to our latest poll and have raised concerns over the apparent over-representation of university graduates in our sample. Typically, survey results are weighted by age, gender, and region, and now some have suggested that we should also be weighting for education. We would like to take this opportunity to respond to some of these concerns.

In total, 1,377 Canadian adults responded to our survey. After the results were weighted for age, region, and gender, we were left with the following educational breakdown:

At first glance, these figures compare quite unfavourably to Census data. Those with a university education are over-represented by a factor of three, while high school educated adults are under-represented by a factor of four. The story behind these numbers, however, is far more complex.

The first issue to look at is measurement error. Measurement error occurs when a respondent’s answers are inaccurate. This is particularly relevant when it comes to measuring education. There is a certain social stigma attached to low educational attainment and respondents have a tendency to over-report the highest level of education they have achieved. For example, some respondents who have attended community college will simply categorize their education has “university” while others will count their life experiences as “college”.

Next, there is also a certain level of coverage error (coverage error occurs when certain segments of the population have no chance of responding to the survey). In all survey samples, there is a certain portion of the population that is simply unreachable (for example, those who do not speak either Charter language, those who are currently institutionalized, those who do not have telephone access, etc.). These groups all have significantly lower incidences of university education.

For both these reasons (as well as others), it is unfair to compare the educational composition of a survey sample to Census data. Indeed, we would expect that even in the case of perfect sampling, there would still be two-to-one over-representation of university graduates (as well as a corresponding under-representation of those limited to a high school education or less).

Nevertheless, measurement errors and coverage errors do not seem to account for the discrepancy between the Census data and our survey results. Indeed, the proportion of self-described university educated adults appears overrepresented by a factor of three (rather than our “acceptable limit” of two). Unfortunately, we have no way of disentangling the effects of sampling error from those of the measurement and coverage errors we described above. For the sake of discussion, we have re-weighted our data so that the university educated make up 45 per cent of the sample, college educated make up 30 per cent, and high school educated make up 25 per cent.

Not surprisingly, the results have changed somewhat. The case for the current government is a little less negative, but the claims we made are still accurate: confidence in direction of the government is near an all-time low, Mr. Harper’s approval rating is quite low and, by a wide margin, Canadians believe Mike Duffy’s version of events over those of Stephen Harper. Indeed, while it is possible that the over-representation of university educated slightly biased the results, there is nothing that would qualitatively alter any of our conclusions.

In the case of vote intention, where a single point of variation can mean the difference between victory and defeat, the changes are not inconsequential. Nevertheless, attempting to restore educational attainment to the Census composition could end up causing more problems that it solves, as doing so will begin to over-represent those who will not show up at a polling booth (those without post-secondary education are consistently less likely to vote on Election Day). In any case, we are two years away from an election and with the attention of most Canadians focused elsewhere, we are only interested in creating an “overall picture”.

In short, weighting by gender, age, and region can usually correct survey results to a manageable level. But sometimes, when response rates appear correlated to another demographic variable, it might be defensible to apply additional weighting. In retrospect, yes – a weighting adjustment for education would have been appropriate in this instance. However, there is a myriad of other factors that might have similar effects – household size, labour force status, etc. – and we could go through this exercise infinitely.

Finally, it should be noted that we contacted some respondents by phone in order to ensure full coverage of the Canadian population (those who do not have Internet access are less likely to have pursued university studies). This procedure is not undertaken by non-probability, online-only panels, which now make up the majority of political polling conducted in Canada.

Click here for a PDF version of this report: Methodological Note (October 31, 2013)

Comments are closed.