I have written many times in this space and elsewhere about the ethical obligations of pollsters to disclose the details of surveys they release into the public domain. But what obligation do bloggers or journalists have to disclose details of their methods when rating the performance of pollsters?
That question came up this week when Nate Silver (whose website, Fivethirtyeight.com, will soon be hosted by the New York Times) released an updated set of pollster ratings and a lengthy description of the procedures he used to calculate them.
Three days later, Markos Moulitsas announced that his website, Daily Kos, was firing its pollster, Research 2000. "The decision," Moulitsas explained, "was made, in part, on the results of Nate Silver's new pollster rankings." Research 2000 ranks as one of the least accurate on Silver's list of the 75 most prolific pollsters.
Later that day, Political Wire blogger Taegan Goddard questioned the transparency of Silver's scorecard. In explaining his ratings, Silver had noted that "polling firms which have made a public commitment to disclosure and transparency" have been more accurate. As such, the scoring system essentially awards bonus accuracy points to pollsters who made such a public commitment, especially those who put a relatively small number of polls into the public domain.
Goddard noted the irony: "When I talk with pollsters about the latest scorecard," he wrote, "they're universally puzzled as to why Silver doesn't hold himself to the same level of transparency and release his database of polls. In fact, some even claim he's using faulty data in putting together his rankings."
Silver quickly responded by pointing to his initial description: "Where's the transparency? Well, it's here, in an article that contains 4,807 words and 18 footnotes." He also noted what he called a similar irony of the criticism of his disclosure coming in the form of "an anonymously-sourced item in Taegan's newsletter."
He conceded, however, that he had not released "the complete polling database" used to calculate the ratings. "The principal reason," Silver wrote, is "because I don't know that I'm legally entitled to do so," explaining that the data had been culled from 10 different sources, including some subscription-based services and others, like my site Pollster.com, that are "direct competitors" of his own. He nonetheless posted the list of 293 polls conducted by Research 2000 in his database from 2000 through 2010.
Not surprisingly, Research 2000 President Del Ali pored over the list and this morning sent both Goddard and me a list of 13 Research 2000 polls that Silver should have included and two that had errors. When I spoke with him by telephone today, Ali was livid; still, he insisted that he was "not trying to get back at Nate" but rather "trying to redeem my body of work" and "constructively help Nate do better research." He also hoped that other pollsters would want "to take a look at their list," too, "because after all, we want to get this right."
We checked the list and found that the omissions and errors alone may not do much to redeem Research 2000's ranking. While we cannot replicate Silver's regression-based scoring, we found that taking Ali's corrections into account barely changes the simple average of error scores for Research 2000. It declines from 6.0 percentage points of error on the winner's margin to 5.9.
Let me stop at this point and say, as the editor of a website that has been collecting and publishing public-domain polling data for nearly four years: Mistakes happen. While we work hard to find every poll and record its data accurately, I am confident, unfortunately, that some mistakes and omissions remain. In fact, I found at least five of the 13 polls missing from Silver's database were also missing from our compilations at Pollster.com. Many of the shortcomings in his database reflect errors or omissions by his sources.
As such, it is both impressive that Silver has endeavored to collect 10 years' worth of data and unsurprising that errors exist within it. But I agree with Goddard and others who argue that there ought to be a higher standard for accuracy when the reputations and careers of those being scored are on the line.
This morning, I put out a question via e-mail to pollsters I know asking for on-the-record reactions to Goddard's criticism of Silver's ratings. A small handful agreed that Silver should release his data, although most either did not respond or expressed no opinion on this aspect of Silver's ratings. The most constructive suggestion came from a former employer of mine, Democratic pollster Harrison Hickman.
Hickman sees "several reasons why Nate might not release the entire database" but believes "a procedure should be in place to validate the data used in his calculations. For example, it should be relatively simple to provide each polling firm with a list of the polls used to calculate their score, [and] ask them [to] verify the results being used." He also suggests asking for additional information about methodology "that might even provide him more (and more pertinent) information [than] he currently possesses and uses for his calculations."
Either way, I agree with Goddard: "When decisions are made to hire and fire pollsters based on these ratings, it's even more important to show people" -- or at least the pollsters whose data is being evaluated -- "the data that's being used."
Want to stay ahead of the curve? Sign up for National Journal’s AM & PM Must Reads. News and analysis to ensure you don’t miss a thing.
Join the Discussion