It's not often that news breaks at the annual conference of the American Association for Public Opinion Research. With sessions on topics like "Non-Response, Paradata and Data Quality" and "Methodological Advances in Address Based Sampling," it's not exactly surprising that interest is usually confined to a small circle of researchers and pollsters like me (I am an active member and served on AAPOR's executive committee from 2006 to 2008).
But last week, when outgoing AAPOR President Peter V. Miller addressed the meeting, he shared news that could have a profound impact on our ability to make better sense of the flood of polling data we confront every day.
Specifically, Miller announced the names of 35 media, academic and survey research organizations that have pledged to support a new "Transparency Initiative." Each has pledged, in principle, to routinely disclose basic information about the methodology of surveys that it releases into the public domain -- as required by AAPOR's newly revised code of ethics -- and deposit that information into a public archive that AAPOR will create.
Most impressive is the breadth of pollsters on the list, including ABC News, the Associated Press, CBS News, the Gallup Organization, GfK Roper Public Affairs, the Kaiser Family Foundation, Knowledge Networks, the Marist Institute, the New York Times, the Pew Research Center, SurveyUSA and the Washington Post. Each pledges to routinely and publicly disclose information not typically released, including information about sample frames, weighting, sample coverage and response rates. And Miller emphasized that the list is "just the beginning" because he expects more organizations "will join this who have yet to."
The initiative is AAPOR's response to the changes roiling the survey profession. "We are in a sea of undocumented data," Miller said as he addressed the AAPOR membership, "data which hits us everywhere and every way and data which is probably, in many cases, untrustworthy."
He also noted the rise of "do-it-yourself" automated surveys -- like Precision Polling and Scott Rasmussen's Pulse Opinion Research -- and lamented a decline in professionalism. "The barriers to entry in our field are very low. Anybody can say they did a survey."
In this new environment, Miller said, "People can simply fabricate data and put it out there and claim that it's real." He recounted the allegations of fraud leveled at Strategic Vision LLC after AAPOR issued a statement of censure of the polling company for failing to disclose methodological information about survey data released during the 2008 presidential primaries.
Miller has advocated this new initiative, however, because he believes AAPOR's punitive actions, such as those aimed at Strategic Vision, have "largely failed" because they have been too limited and ineffective. Instead, AAPOR now intends to "turn this around" and adopt an approach that is "less stick and more carrot." If an organization pledges to routinely disclose methodological information about publicly released polls, Miller said, AAPOR will "recognize it and honor it" and give it AAPOR's "seal of approval."
In his address, Miller spoke frankly about the many reasons why the initiative might not work.
Some research organizations might not consider AAPOR's "imprimatur" useful or prestigious enough to matter to clients or end users of their data. Some might balk at the effort and cost involved. Some may be barred contractually from disclosing client data. Some will fear that "somebody is going to look through the information and look at response rates, and they're going to say, 'Look, so-and-so got a response rate of 12 percent, can you believe that!'" Some will question whether AAPOR can carry out "something this ambitious."
And some will worry that increasing disclosure will not address "the broader issue of quality." What if a pollster is perfectly transparent about the details of a badly done survey? Should they be rewarded with a seal of approval?
Most of the specifics remain undefined. The revised AAPOR code, for example, now calls on researchers to provide "a description of the sample design," although it is unclear what specific information that provision requires. Miller promised that AAPOR would not "require researchers to do things that are impossible" and pledged to "work together to define what we can do and what we can't."
But Miller also shared reasons for hope: Contracts can be rewritten and many organizations have already pledged their support. AAPOR has beefed up its management structure. The "information environment" has also changed and now includes "virtual communities who are very interested in the details of surveys," such as the bloggers who took up the cause against Strategic Vision following AAPOR's censure last year.
Miller also argued, in effect, that transparency is a path toward quality. The creation of what he described as a "repository of data" on how pollsters do their work will allow the profession to develop and refine its best practices. More important, greater transparency will allow the rest of us to better assess the quality of the data we consume.
Get the latest news and analysis delivered to your inbox. Sign up for National Journal's morning alert, Wake-Up Call, and afternoon newsletter, The Edge. Subscribe here.
Leave A Comment