Polling Pitfalls: Getting it right means attention to every detail - not all polls are created equal

The complainant, Barry Kiefl, who runs Canadian Media Research and used to work here at CBC, had some concerns about some features on Power and Politics. He thought other programs were not living up to CBC’s polling policies either. He was right in at least two cases. Power and Politics is not living up to the rigors of CBC policy on polling in a couple of their regular features. I suggested CBC lay out some standard processes for polling approval and presentation.

COMPLAINT

The basis of your complaint was published on your blog. You asked CBC News to respond to the issues you raised in the posting. You were concerned that CBC News’s standards were slipping because you thought that some program segments on the CBC News Network program Power and Politics were violating CBC’s own policies on polling. You felt the material being presented in these segments was being presented as if it met the criteria of bona fide polls, and could therefore be characterized as representative of public opinion. And in the case where polling techniques were used, you thought the policy was not being followed because there was inadequate explanation of the methodology behind the results being presented.

One of the features you were concerned about is entitled “Political Traction.” This segment features Jaime Watt, the Executive Chairman of Navigator, a communications firm. The purpose of this weekly presentation is to compare the issues that are most discussed in Ottawa with those that are prominent in the rest of the country. The Power and Politics website describes the purpose of the feature as a chance to take “a closer look at the top public affairs issues being discussed in Ottawa to evaluate how well these issues resonate among everyday Canadians.”

Watt’s team at Navigator assesses what Members of Parliament, including questions asked during Question Period, editorialists and political and public affairs programs are talking about each week. Through various analyses, these are assigned a numeric value. The top issues are given a ranking and then compared to the frequency of “conversation” in a broader Canadian context. The broader picture is measured via key word search in social media, as well as a scan of public affairs and political programs and national editorials. The program host, Evan Solomon, and Jaime Watt spend some time outlining the issues and measuring how the preoccupations of Ottawa are mirrored, or not, in the rest of the country.

You felt that the nature of the discussion led viewers to believe the data is representative of Canadians, and it cannot be because it does not use conventional polling methodologies to make that claim. You point out that the “tracking” percentages are actually based on analyses of blogs, online comments and twitter. You thought the methods used should be much more prominently featured:

“There is nothing wrong with someone analyzing tweets and online comments but CBC should not be implying that this is representative of Canadians and therefore possibly leading viewers into thinking that Navigator has conducted representative polling. We know that relatively few people tweet and research, including some done by CBC, determined that those who bother to write newspapers or call talk shows, etc. are generally not representative of either the audience or the population. Power and Politics should be more forthright about the methodology used by Navigator.”

The second weekly Power and Politics feature that concerned you was the “Nanos Number.” This program segment is devoted to interpretation of public opinion research and its meaning and impact on the political scene. The “Nanos Number” is given at the conclusion of the segment as a way of highlighting the significant development or trend of the week. Your concern about this segment is that the methodology is not presented prominently enough. You thought that CBC should be a lot more transparent about the Nanos surveys and provide information about the methodologies used in the polls, both on air and on the Power and Politics website. You also had your doubts about some of the methods used and questioned whether CBC’s own Research Department had approved its use:

“Survey results on some occasions appear to come from a conventional telephone poll, while on other occasions, results are based on what Nanos calls an “RDD Crowdsource random survey of 1,000 Canadians… recruited by telephone through the proprietary Nanos Crowdsource sample and administered a survey online.” Crowdsourcing has been used to fund entrepreneurs but it is very unclear how it has been used here. The Nanos Number segment, featuring the amiable and professional Nik Nanos, usually provides little or no information about the methodologies, other than an illegible footnote partially obscured by program graphics. One must go to the Nanos website to learn how the surveys have (presumably) been conducted. So, once again, this seems to be in violation of the CBC’s journalistic policy.”

The third feature that troubled you was the daily “Ballot Box” question. Before the show even goes to air, the programmers post a question on the website and then promote it throughout the show. The question is topical and people are asked to vote, generally between two choices and a “don’t know” category. They can participate by going to the website, through the program’s Facebook page or by using a QR code displayed on the TV screen. The results are referenced throughout the program. The numbers are expressed both as a percentage and the actual number of votes for each position.

You see this as a violation of CBC policy because it is a completely non-scientific poll and in no way represents Canadian public opinion, but you think it is being presented as such. You point out that CBC’s own policy on the matter clearly states that “Since it does not fulfill any of the criteria set out in polling policy, the questions and the results are not characterized as polls. We report the results by giving the number of votes cast for each option. We do not give the results as a percentage, as we normally do with bona fide polls.”

The “Ballot Box” does express the results as percentages both on air and on the web site. Your concern is that this presents the results as a bona fide poll: In my view this lends an air of polling legitimacy to The Ballot Box, which is not warranted, as any professional researcher and experienced journalist knows.”

In subsequent correspondence you mentioned the misuse of these online surveys on other radio and television programs as well.

MANAGEMENT RESPONSE

The Executive Producer of Power and Politics, Amy Castle, responded to your concerns. She rejected the contention that Power and Politics’ three features did not live up to CBC’s own standards.

Specifically, she disagreed with you that Mr. Solomon and Mr. Watt left the impression that the data they reference during the “Political Traction” segment implied or left the impression they were discussing public opinion based on polling results:

“Rather, we are clear on the show and on our website that Jaime Watt is tracking the political conversation in Ottawa and across Canada. The goal of the segment is to discuss which political issues are trending in the Canadian conversation, and to highlight the differences - if any - in the conversation being held in Ottawa compared to the national conversation”.

She pointed out that the methodology for the Traction feature was available on the CBC website and that in the interests of transparency she would ensure that a mention of where to find the details behind the data would be given on air.

She stated that the Nanos Number feature also abided by CBC journalistic policy. She explained “pollster Nik Nanos draws on a number of polls from reputable sources and we are always clear where the polls come from.” She added that all polls referenced in the feature meet CBC criteria for bona fide polls. Information on methodology, sample size and margin of error are available on the website as well as on a graphic on air. Ms. Castle informed you that she is working with the program’s graphic designers to find an on-air format that is easier to read.

She also addressed your concern that Mr. Nanos used crowdsourcing in the creation of his polls. She explained that “in fact the term crowdsource is a Nanos tradename and should not be confused with a process. The Nanos polling methodology includes a sample of random land and cell lines where people are randomly selected to do a study online. His methodology is robust and meets our strict CBC standards.”

She believed that the “Ballot Box” feature also met the criteria of CBC policy. She pointed out that the results are never referred to as a poll, and that while it is against the policy to express the results as a percentage, the absolute numbers appear on the screen as well. She undertook to revise the presentation online so that it too gave both percentages and numbers.

REVIEW

CBC Journalistic Standards and Practices has three policies on polling which apply here. They are all similar in demanding rigorous standards and a review of methodology by the corporation’s research department. For polls that CBC itself commissions, the policy states:

CBC commissions polls to reflect Canadian opinion on important issues of the day. The polls are commissioned, designed and the results are interpreted with the oversight of the CBC's research department.

To help our audience place a poll in context, we provide relevant information about the size of the sample and methodology along with the results. Where appropriate we report the margin of error.

It goes on to say because of the impact public opinion polling can have on public policy debates or elections, “the commissioning of polls requires the approval of the head of research and the Director. The final design requires sign off from the head of research or delegate.”

The policy also states that CBC staff can report on polls they have not commissioned, as long as they can verify the methods meet CBC standards. In these cases too it calls for a review by the Research department and for providing the audience with some basic information, i.e. sample size, method used and the margin of error where applicable.

As I have mentioned in other reviews I have written, in the interests of full disclosure, I was involved along with some colleagues in the creation of this version of CBC policies and standards. These policies built on the principles of earlier polling policies. They were written after consultation with experts. The entire Journalistic Standards and Practices was externally reviewed and found to be equal to or better than other existing journalistic codes.

I will review separately each of the features you questioned.

Political Traction

You are right in your assertion that the Traction feature does not meet the criteria of a poll. I disagree that it is being presented that way. After watching multiple segments, it is clear to me that this is a conversation about trends and issues in Ottawa and beyond the Rideau bubble. The program hosts and Mr. Watt carefully talk about the “conversation,” as in “Clearly the robocall conversation is running out of steam…” Another phrase is “What we are seeing in the Canadian conversation is …” There is never any claim for being a reading of public opinion but rather a reading of the issues that are prominent in the world of people preoccupied by politics and public policy issues. I note as well that Ms. Castle has followed through on adding a mention at the end of the segment which points people to the online detailed explanation of how the issues and rankings are derived.

Nanos Number

The Nanos Number feature raises some flags. Ms. Castle states that the Nanos Number methodology fulfills the rigors of CBC research standards. Without passing any judgment on the methodology, it is not clear that it does because it is not regularly or properly reviewed. There appears to be no rigorous or regular process for news staff to submit and get approval for polls.

While Ms. Castle’s predecessor had approval for some of the polls used in the Nanos Number, Debra McLaughlin, Senior Director of CBC Research, pointed out to me that “I can understand why someone might think that approval of several polls from the same person might suggest a broader endorsement but in research this cannot be the case. There are too many variables.” Those variables might be sample size, variation in the phrasing of the question, or even in the description of the methodology of some polls.

There is a lack of rigor in the process to ensure a consistent adherence to CBC’s policies and high standards. The Research department has not reviewed one of the Nanos Number segments in quite a long time, and while it may be the result of a misunderstanding of the requirements, it is a violation of policy. It also undermines CBC News’s value of credibility.

The whole public opinion research field is in flux, and has had some hard knocks recently. Practitioners are faced with challenges in obtaining representative samples at a time when the way we communicate is profoundly changing. And like everyone else, they are under more scrutiny and demands for openness about their methods. It is important for CBC News credibility and its commitment to transparency to ensure a rigorous process of review for all polls it chooses to publish on any of its platforms. It might be worthwhile for News management to consider a standardized process to approve the use of any public opinion research in its content, and to provide guidelines and templates for sharing the critical aspects of the methodologies with its audience.

While you are critical of Power and Politics for the way it presents the methodology, it is present on the screen. It would be worthwhile to consider at least a standard template to provide context, and as in the case of the Traction feature, to point people to the website for more complete information. While you feel that to fulfill policy there should be consistent on air mention, I think it is reasonable to leave it up to the programmers to ensure audience members know where to get the information and provide it in context in a form that complements the medium and fulfills the policy requirement. As News management has stated it is committed to transparency, it is up to them to ensure that methodology is shared and understood by news users.

Ballot Box

The Ballot Box feature also presents some challenges. It falls under the policy on online surveys which states:

Online surveys are a tool of audience engagement.

Since it does not fulfill any of the criteria set out in polling policy, the questions and the results are not characterized as polls.

We report the results by giving the number of votes cast for each option. We do not give the results as a percentage, as we normally do with bona fide polls.

If programs refer to online questions, the results are reported in a way that clearly indicates it has no scientific validity and are not meant to represent the accurate range of either public opinion nor the opinion of our audience.

It is true that, as Ms. Castle states, the Ballot Box feature does provide numbers as well as percentages. But the percentages are much more prominent, and in the online iteration, the graphic does not provide raw numbers. The survey is never referred to as a poll, and the language used encourages people to be “part of the conversation.” The producer of the program provided me with the standard language program host Evan Solomon uses when referring to the ballot box results:

Now let's check in on our Ballot Box question.

Today we're asking you (question of the day).

Some of you have already voted on this.

If you haven't just go to our website cbcnews.ca, or click on the ballot box section of our facebook page, or you can scan the QR code at the bottom of your screen and you'll go instantly to our Ballot Box.

And we're also tracking your comments.

You can see the comment board on your screen, with all the ways you can get in touch with us.

Be part of the conversation on Power and Politics.

But Mr. Solomon also says “x % of you think that…” and adds one of the options given. I believe it is disingenuous to say it does not leave the impression that the numbers somehow represent what Canadians are thinking. It does not even represent what watchers of the broadcast and readers of the website of Power and Politics are thinking. It represents those who have clicked on a box. This does not live up to the intent of the policy. It might be worthwhile to add a disclaimer when the results are given so that it is very clear that this is a way of engaging the audience and in no way measures public opinion.

And since you mentioned the widespread use of these kinds of surveys on air and on the website, the same practice might be considered any time this POV survey tool, as it is labelled elsewhere on the website, is used. The fact that the tool so prominently displays percentages of votes contradicts the policy which states the results should not be given in this way.

Overall, there seems to be inconsistent application of the policies on polling. I urge CBC News management to review its practices and provide practical guidelines for abiding by those policies. Alternatively, it might want to review the policies themselves. I believe they do represent best practices but, in the end, policy is the purview of management.

Esther Enkin
CBC Ombudsman