Vote Compass survey during federal election campaign
CBC News' 2011 Federal Election online presence at CBC.ca included a survey, called Vote Compass, that it said calculated users' positions in the political landscape and showed them the respective positions of the five principal parties (Conservatives, Liberals, NDP, Bloc Quebecois and Greens).
The survey was modeled after the Kieskompas, developed in The Netherlands in 2007 and presented in various forms in a handful of other elections in, among other places, the European Union, Sweden, Portugal, Turkey, Israel (since offline), The Netherlands and the United States.
The Canadian version of the survey featured 30 statements on a variety of issues. Users were asked whether they agreed strongly or somewhat, disagreed strongly or somewhat, or neither agreed nor disagreed with the statements.
Once the questionnaire was completed, users were asked three questions about the party leaders and two questions about the parties.
The survey then produced a result that plotted a user's position on a compass-like display. At the “north” end of the compass was “social liberalism” and at the “south” end was “social conservatism.” At the “west” end was the “economic left” and at the “east” end was the “economic right.”
Users were also offered an opportunity to refine the result by selecting specific issues of importance to them and taking an abridged survey.
As part of the Canadian application of Vote Compass, an academic panel advised the Vote Compass organization of researchers on the composition of the survey and the phrasing of the statements.
The five major political parties were canvassed for their views on the issues and asked to self- assess. They did not have a role in the survey design. Researchers also reviewed their public statements. In cases in which there were differences between the self-assessments and the public statements, an advisory board reconciled the differences.
Vote Compass placed the political parties on the compass axis according to how they answered the questions. Users received results based on which party's answers most resembled their own.
On this basis, the Conservatives were positioned in the southeast quadrant of the compass and the other four parties were positioned in the northwest quadrant. Of those four, the Liberals were closest to the geographic centre of the compass, and that stationing produced results that prompted some emails to this Office and media coverage questioning the impartiality of the tool.
Vote Compass was used more than 3.9 million times by some 1.9 million registered users. The Office of the Ombudsman drew 76 emails shortly after Vote Compass was introduced. The gist of most of the complaints was that the result stationed users closest to the Liberals, even though those users professed to have neither supported nor voted for them. Many of them hypothesized motives.
One complainant, Avery Burdett, wrote March 30, 2011, to assert Vote Compass had “built-in biases” because it stationed the Liberals toward the centre.
A second complainant, Kim Deimert, wrote April 22, 2011, that Vote Compass “provided inaccurate, biased, results and that it should in no way be used to gauge your political alignment.”
The executive editor for CBC News, Esther Enkin, wrote back that Vote Compass was neither a scientific poll nor a device “to provide voting advice.” Rather, it was primarily an educational tool. She asserted there was no bias in the survey and that its design and operation were conducted independently.
Burdett and Deimert each asked for reviews of their complaints.
Burdett asked that the Ombudsman pull the feature from CBC.ca because it was “a gross interference in the democratic process.”
Deimert noted that it was possible to answer closest to the Conservative position on 20 of 30 questions and still be deemed closest to the Liberals. Deimert asked: “Is the tool inaccurate? Yes. Is the tool deceiving people? Yes. Does the tool have extreme bias?
Yes.” CBC Journalistic Standards and Practices include the principle of serving the public interest “to contribute to the understanding of issues of public interest and to encourage citizens to participate in our free and democratic society.”
On the matter of integrity, the policy states: “The trust of the public is our most valued asset.”
During elections, “Canadians expect us to provide a wide range of information and context so that they can make decisions during election and referendum campaigns.” Such information needs to be equitable over the course of campaigns.
The complaints and other emails to this Office reflect a broad concern about CBC journalism, but it isn't clear how applicable journalistic policy is to Vote Compass. It is new territory with several unique factors. For example:
- Vote Compass wasn't a poll. CBC called it an educational tool and a survey. Journalistic policy on polls was not applicable. Given that CBC has not reported on the results of the survey, it hasn't subjected the findings to any journalistic scrutiny or review.
- Vote Compass was not a CBC creation. It was a product of an independent academic group outside CBC News. CBC associated its brand with Vote Compass, publicized it across its platforms, and carried it on the website, but it didn't hold the equivalent of the patent.
- Its presence as a tool and survey on CBC.ca suggests it was within the boundaries of news and information content, but even there it had a curious position. Vote Compass produced information personally vested in the user; that survey-to-user relationship was not mediated by CBC News. I concluded that unless CBC chose to integrate that information into its content and report on it, it wasn't actually CBC journalism.
The complainants asserted the methodology of the survey was unsound. But independent public opinion researchers I questioned did not find methodological flaws in the work of the academic advisory team or the research unit in devising and implementing the survey. (The algorithm to produce the result from the survey remains proprietary.) On the basis of what could be seen, I concluded there was no foundation in the assertion about bias.
It was clear many people were taking the survey more than once, and some were “gaming” it at times to see if they could produce a particular result. Thus, even with a form of registration to know who was taking the survey, it is challenging to interpret which uses were authentic and which might have been contrived. (CBC has so far not employed any of the data in its reporting.) To reiterate: the professional researchers said the source material did not appear problematic.
Part of my review was an evaluation of the 30 statements in the survey and whether they reflected party positions. I felt that evaluation intersected with journalistic policies involving accuracy and fairness because CBC was “reporting” on those positions in publishing them. I could not find any that digressed from party positions and I note that no party has quarreled with the survey statements or their perceived relationship to them during or since the election campaign.
I also did not find any exaggerative or misleading publicity about Vote Compass that might give rise to concerns about fairness, accuracy or balance.
I have concluded that CBC fulfilled its policies on accuracy and fairness.
A challenge in a complicated survey of this nature is to help people understand its limitations. In this case, it was important to tell users what Vote Compass was and wasn't. In some instances, users didn't take note of that cautionary material and inferred this was an advice tool or a scientific poll — it was neither, as CBC made clear in its background information.
Judging by email to this Office, some users did not understand that the three questions on party and leader preferences did not count toward the final survey results. I can understand how it might have been baffling to answer those questions about party and leader preference and not be plotted congruently on the compass. Clearer language to signal that those questions would not contribute to the survey results would have been helpful. Also beneficial would have been the opportunity at the outset to employ the option of narrowing the survey to deal only with those issues about which users cared; that option was only available after the survey was completed.
I took note of the extensive background information CBC News provided on Vote Compass, including an explanation of the statements and each party's position on them, a Frequently- Asked-Questions feature prepared for CBC by the researchers, as well as substantial information on how to use Vote Compass. I cannot recall so much background information on any feature. It was a commendable approach, all there for the taking, and a model of transparency for a news organization.
There was limited television reporting about the presence of Vote Compass once it was launched. CBC Radio involved one of the researchers in discussions and the material about its origins remained online throughout and since. For such a significant feature, it would have been helpful for CBC Television to sustain reporting on the involvement of Canadians in the survey and to discuss any questions about it.
It might also be useful for CBC News to examine how such surveys can be assessed within CBC's Journalistic Standards and Practices, given the popularity of Vote Compass suggests it is the first of several.