Ideally, we poll to measure what we don't know about public opinion, not to confirm the things we already know. Yet when good pollsters obtain surprising results, their first instincts are usually to question their own findings: Could the unexpected finding be an artifact of methodology? What new questions can we ask in future surveys to check if the findings are real and broaden our understanding of this new wrinkle in public opinion?
Those instincts have been evident in the questions pollsters have been asking about the Tea Party movement in recent weeks, both of respondents and of each other. The issues they have raised -- and, more importantly, the results from two new national surveys -- tell us a great deal about what Americans know about the movement, and why many are sympathetic if not hard-core supporters.
Over the last few months, surveys conducted in Illinois by the Chicago Tribune, in California by the Field Poll, in Iowa by the Des Moines Register and nationally by Rasmussen Reports have shown varying degrees of agreement, identification and support with the Tea Party movement. Yet in most cases, pollsters expressed some surprise at the extent of that support.
Those results inspired an unusually active discussion on the members-only e-mail listserv of the American Association for Public Opinion Research, the professional organization that is home to many media pollsters. In several lengthy exchanges, many expressed skepticism:
• Several researchers argued that the "Tea Party" movement is ambiguous, with a meaning "in the eye of the beholder," so questions about it are analogous to asking Americans whether they support "health care reform." What do supporters think the movement stands for?
• One reminded his colleagues of the experiments showing that as many as one-third of respondents will offer opinions on subjects they know nothing about.
• Others wondered about the effects of question order or whether Tea Party supporters might be inherently more likely to respond to a telephone survey (the theory being that they would be more eager to express their anger to the pollster).
In short, they were asking the kinds of questions we should ask about any survey: Can we trust these results? Are there methodological issues that might cause a skew? What are the limitations of the findings?
Yet for all the doubts raised, this Tea Party example shows how, with careful thought and analysis, pollsters can help us sort out much of what Americans know and believe about the Tea Party movement.
Two surveys released last week by ABC News and the Washington Post and by CBS News and the New York Times give different impressions of how many Americans have a positive impression of the fledgling Tea Party movement, largely because one survey pushes harder for an answer than the other. The CBS/Times poll finds 17 percent of adults rate the movement favorably and 13 percent unfavorably, but their question offers respondents the option of saying they are undecided (12 percent) or don't know enough to have an opinion (24 percent) even after they take out the 34 percent who say on a previous question that they have heard "nothing" about the Tea Party movement. The ABC/Post poll, on the other hand, finds far more who rate the Tea Party movement favorably (35 prcent) or unfavorably (40 percent) because they present all adults with just those two answer choices.
The two surveys are more consistent with each other -- and with many of the doubts expressed in the AAPOR discussion -- in showing that relatively few Americans report hearing much about the Tea Party movement. On the ABC/Post poll, only about a third say they know "a great deal" (13 percent) or "a good amount" (22 percent) about "what the Tea Party stands for." On the CBS/Times survey, slightly more say they have heard something about the movement and believe they know "a lot" (13 percent) or "some" (27 percent) about what it stands for.
Thus, support for the Tea Party movement looks smaller on measures of intensity: The CBS/Times survey finds only 18 percent who describe themselves as "supporters" of the Tea Party movement, and the ABC/Post survey finds just 14 percent who say they "strongly agree" with the movement's issue positions.
The two surveys provide a consistent profile of the hard-core Tea Party supporters: overwhelmingly white, conservative and Republican, nearly all expressing disapproval of President Obama and either anger or dissatisfaction with the direction of government in Washington.
Equally interesting, however, is the way the general impression of the Tea Party movement -- conservative, anti-Obama and anti-government -- appears to extend to those who are sympathetic but not hard-core supporters. The ABC/Post poll found that nearly half of Americans (45 percent) say they at least "somewhat" agree with Tea Party positions on issues. That support was greatest among conservatives (63 percent), strong Republicans (67 percent), those who disapprove of Obama (65 percent), and those who say they are angry about the way the federal government works (69 percent).
So while these two new surveys cannot resolve every question or doubt, they do get us a much clearer picture of public opinion on the emerging Tea Party movement. While relatively few know "a lot" about it, the movement has made a strong general impression.