Last week, Time columnist Joe Klein had seen one issue poll too many.
"Polling on issues is next to useless," he blogged, "especially on issues as emotionally complicated as wars and as technically complicated as health care reform." Surprisingly, many pollsters, including yours truly, see some merit in Klein's argument about the way the news media typically report poll questions on public policy issues, even if we might not be ready to cast away such polls altogether.
Let's start with where we agree. Klein was talking about two new survey questions, one from CNN and the Opinion Research Corp. showing plurality opposition (49 percent to 46 percent) to the health care bill just passed by the House, and another from ABC News and the Washington Post showing a majority (52 percent) concluding that the war in Afghanistan has not been worth fighting.
Klein then speculated, reasonably, that different questions asked about the same subjects would likely tell a different story. "The only safe conclusion from these particular polls," Klein concludes, is that "the public has mixed feelings on Afghanistan and health care reform. Brilliant! I have mixed feelings, too."
His objection then, is less about specific survey results than the screaming headlines they typically produce ("Public Opposes Health Bill. Public Opposes War") and the false sense of certainty they convey. Fair enough. I have offered similar caution about the limitations of issue questions.
When I posted Klein's comments at Pollster.com, I heard from a handful of campaign pollsters via Twitter who quarreled with his blanket dismissal of issue polling but generally agreed with his critique of the way the news media often reports on survey results.
Democratic pollster Jason Boxt, for example, finds it "infuriating when polls are misrepresented (this am on Morning Joe, for example)." Republican micro-targeter Alex Lundry chimed in to say that any agreement with Klein "would be limited to media polls."
"Klein reminds us why most campaign pollsters don't ask [questions] like media pollsters do," writes Republican pollster Matt Dabrowski. "Speculative policy is not our deal. Rather, it is how policy might move elections."
Having spent much of my career conducting internal polling for political campaigns, I tend to agree. Klein is also right that poll questions are flawed measures of attitudes on public policy issues.
But we all still see value in looking at polling data (even Klein ends his blog post confessing that he, too, is addicted). The danger is labeling any one result to any one question as "The Truth." To get closer to the reality of public opinion on an issue like health care, we need look at a lot of different questions asked by many different pollsters.
Consider, for example, the CNN poll that stoked Klein's fire. It included quite a few health care questions -- roughly 20 in all (released in two batches by CNN) -- "asked in a different way" about health care reform. These various measures, while undoubtedly imperfect, collectively provide a deeper picture of the rich, sometimes conflicting attitudes and values that make up "public opinion" on health care reform.
Now consider one specific result from that poll in the context of the "Public Opposes Health Bill" headline that Klein imagines and that many recent poll results support. Immediately after their favor-or-oppose question about the recently passed health reform bill, CNN's pollsters asked a new follow-up question that others have not. They found that while a third of all adults (34 percent) say they oppose the bill because "its approach toward health care is too liberal," 10 percent oppose it because it is "not liberal enough."
As Texas Tech University professor Alan Reifman points out, those results mean 56 percent of Americans "favor either the House-passed version of health care reform or something further to the left."
A check of the Nexis news search shows that CNN broadcast results of the follow-up question as part of their poll stories several times on the day they were released. And the story that ran on CNN's online Political Ticker featured the same numbers prominently, including this characterization from CNN Polling Director Keating Holland: "That may indicate that a majority opposes the details in the bill, but also that a majority may approve of the overall approach taken by House Democrats and President Obama."
Ironically, the story featured a different headline than what Klein had imagined: "Americans split on health care reform."