This is a story about the alarming lack of transparency and disclosure by some public pollsters.
But to tell it, I need to tell another story that involves an innocent victim and a subject all too familiar to regular readers: surveys conducted using an automated, recorded-voice methodology rather than live interviewers.
The first story starts on Thursday morning, March 11, when our colleagues at NBC's First Read posted news of two new "robo-polls" in Colorado showing very different results. One had the Democratic candidate for governor leading by 11 points, the other had the Republican candidate leading by 6.
This discrepancy, they explained, is "why we have little confidence in these types of polls." Yes, they wrote, "many number starved folks desperately want to look at even bad polls just to scratch an itch. But trust us: It's like 'Raiders of the Lost Ark' -- if you look, eventually you'll burn your eyes out, and maybe even melt or have your entire brain turn to mush."
But the very next paragraph of the First Read e-mail that morning included news of another recent survey: "In Florida, Marco Rubio is leading Charlie Crist by 34 points (!!!), according to an Insider/Advantage poll."
Unbeknownst to NBC, the Florida poll was also automated. "As soon as the e-mail went out," NBC's Mark Murray tells me, "I received e-mails from political colleagues that InsiderAdvantage was now a robo-poll, which we don't report on by policy at NBC." So he cut references to the poll from their online version before it was published.
Now, regular readers will recall that I disagree with those who judge all automated polls (sometimes referred to by the acronym IVR for "interactive voice response") as inherently unreliable solely because they use a recorded voice rather than a live interviewer to ask questions. But I will grant that I may represent a minority in the media polling establishment. When it comes to setting standards for the reporting of polls, NBC is certainly in good company: ABC News, the Washington Post, the New York Times and the AP also refuse to publish or air results from automated surveys.
So we can set aside, for today, the argument about whether news organizations should report the results of automated surveys. The issue here is why the mode of interview for that Florida poll was such a mystery. We can forgive Murray and NBC for their confusion: InsiderAdvantage -- the polling unit of a Georgia-based public relations firm -- had conducted polls using live interviewers for years, but changed its methods at some point in the 2008 election cycle. Moreover, this particular poll had been sponsored by a major newspaper, the Florida Times-Union, but neither its article nor the results posted on the InsiderAdvantage Web site make any mention of the mode of interview used for the survey.
That omission was not unusual. During the 2008 election cycle, InsiderAdvantage rarely if ever disclosed the mode of interview used for its surveys in its published reports. The company's current Web site does little to clarify its methods, aside from listing "interactive surveys" as one of many types of polls and research it conducts. Aside from the occasional blog post or column by yours truly, which reported on specific instances of the company's use of automated polling, it is hard to imagine how anyone would have known that InsiderAdvantage had shifted to an automated methodology.
And that's a problem.
When I e-mailed InsiderAdvantage CEO Matt Towery last week, he did quickly respond: "Every poll we conduct is IVR," he wrote, including "the [Florida] poll we conducted for Politico right before the 2008 presidential race that showed Obama winning by 1%." Suffice it to say, the Politico report in October 2008 made no mention of the fact that the poll had been conducted using an automated methodology.
For more than 30 years, the National Council on Public Polls (NCPP) has promoted transparency for polls released into the public domain. The Council's Principles of Disclosure -- a code designed by news media pollsters intended for news media pollsters -- mandates the disclosure of certain information in "all reports of survey findings issued for public release" by its members. These include things like sample size, survey dates and the margin of error as well as the "survey mode." And in case the meaning is unclear, the Council includes specific examples such as "telephone/interviewer" and "telephone/automated."
No, neither InsiderAdvantage nor the Florida Times-Union are members of NCPP. And yes, other pollsters often fail to meet the standards they set. But when good reporters like our colleagues at NBC are fooled because a pollster is being coy about basic disclosure, we have a problem. Unless news media sites, including Pollster.com, start holding pollsters to the minimal disclosure standards like those set forth by NCPP, we will see more of this sort of confusion.