Last week, I got a generous dare via Twitter from Markos Moulitsas, founder and publisher of the liberal Web site Daily Kos: "I'll do this for you, @MysteryPollster -- design a 'wingnut' poll that would be valid in your eyes, and I will run it."
That's a hard offer to pass up, but it needs some explanation.
Last week, Harris Interactive released an online survey showing that "large minorities" of Americans hold "some remarkable opinions" of President Obama. Some 40 percent of adults are said to believe "he is a socialist." More than 30 percent think "he wants to take away Americans' right to own guns" and "he is a Muslim." And 20 percent or more are willing to agree that Obama is "racist," "anti-American" and "doing many of the things that Hitler did."
"Hot words, those," wrote ABC News Polling Director Gary Langer, who took to the Internet within hours to attack the survey for taking a "highly manipulative approach to questionnaire design." While many Americans might believe "profoundly negative things" about Obama, Langer wrote, the Harris poll "demonstrates splendidly how not to measure them."
Harris' online methodology came in for some criticism too. Harris solicits respondents from a panel of individuals who have agreed to participate in its surveys. The panel is not representative of the U.S population, so Harris weights the completed interviews to "bring them into line" demographically. The American Association for Public Opinion Research questioned the representativeness of such "non-probability online panels" last week in a task force report (sure to stir up controversy in its own right), and I reviewed the online polling controversy in two columns last fall.
Even so, Langer is right to imply that measurement is a bigger worry in this case than sampling. The survey consisted of 15 negative statements about Obama that respondents were asked to evaluate as true or false. Langer's complaints are that:
• Introducing the statements as "things people have said about Obama" gives them added credibility.
• True-false questions take an "unbalanced" approach that academic research has shown to "overstate agreement with whatever's been posited."
• The statements were all "unrelentingly negative," rather than a mix of positive and negative.
• Respondents may have been using the questions to "to express their general antipathy toward Obama" rather than endorsing the truthfulness of each statement.
I e-mailed Humphrey Taylor, the chairman of the Harris Poll, for comment. "I feel good about our data," he replied, and pointed me to a blog post in which he expresses pride in having "stimulated a great deal of interesting discussion and made a valuable contribution to the debate." However, even Taylor concedes that their questions may have exaggerated agreement. "It is possible," he writes, "if we had included both positive and negative statements, fewer people would have said the negative statements were true."
What does all of this have to do with Markos Moulitsas? Shortly after Harris released its survey, Republican pollster Alex Lundry warned his Twitter followers that the survey shared some of the "same problems" as a similar poll sponsored by Daily Kos in February. I tweeted my agreement, pointing to a blog post that summarized some of the early academic research on the problems of unbalanced questions. Moulitsas responded with the dare at the top of this column.
So, how would I do such a poll?
First and foremost, it would be important to ask about both positive and negative statements on Obama, and even better, to try to measure reactions to "extreme" statements emanating from both the left and right wings of American politics.
One valid concern about the Harris survey is that many Obama supporters may have taken offense to the statements and stopped completing the survey. Keeping a politically representative sample on the phone would require both balance in the questions and a careful introduction to explain the purpose of the unusual questions to follow.
Second, it would be far better to ask respondents to choose between balanced alternatives, rather than rating each statement as true or false. The best example, flagged by Time's Michael Scherer, has already been asked by the Pew Research Center in surveys conducted in 2008 and 2009: "Now, thinking about Barack Obama's religious beliefs, do you happen to know what Barack Obama's religion is? Is he Christian, Muslim, Buddhist, Hindu, atheist, agnostic or something else?"
When Pew Research asked that question in March 2009, 11 percent of respondents -- and just 17 percent of Republicans -- identified Obama's religion as Muslim. Compare that to 32 percent of adults and 57 percent of Republicans on the Harris survey. As Scherer notes, it is hard to explain why belief that Obama is a Muslim would have tripled in the last year.
That same format could be adapted to some of the other extreme beliefs tested by Harris. For example: "As far as you know, was Barack Obama born in Hawaii, Illinois or Kenya?"
Third, open-ended questions might provide an even better measure in some cases. For example, if we are looking for belief that Obama is not a natural-born citizen, it might be better to simply ask, "To the best of your knowledge, where was Barack Obama born?" and record those who volunteer a location outside the United States.
Finally, we all need to take seriously Langer's admonition about "over-literalism" in interpreting this sort of survey. Consider the context of labels tested on the Harris poll like "socialism" or doing "many things that are unconstitutional": During the 2008 campaign, John McCain and prominent Republicans used the word "socialist" to describe Obama's policies; and right now, more than a dozen state attorneys general are challenging the constitutionality of the health care law. So how extreme is it for many ordinary Americans to apply these labels to Obama?