This week I participated in a panel at the Personal Democracy Forum in New York, and my assigned topic was "How Not to Be Fooled by Polls." My No. 1 suggestion: Look at many polls, not just one.
At about the same time as my talk, Los Angeles Times/Bloomberg released a new poll [PDF] showing Barack Obama leading John McCain by a 12-point margin (49 percent to 37 percent). Immediately, reporters and political junkies were asking, "Is this result real?" Once again, we started focusing intently on just one poll, overlooking other recent surveys showing a closer race.
On Wednesday, the McCain campaign distributed a memo from its pollsters, Bill McInturff, Liz Harrington and David Kanevsky, arguing that the Arizona Republican's "double digit deficit," as measured by the poll, "is not a reflection of reality."
The culprit, they argue, is the poll's partisan composition. Thirty-nine percent of the voters surveyed identified themselves as Democrats, 22 percent as Republicans and 27 percent as independents; 12 percent were either unsure or identified with another party. The McCain pollsters pointed to other recent polls showing higher percentages of Republicans and argued that if the LAT/Bloomberg survey had been weighted to a "more normalized range for party identification," Obama's lead would have been narrower.
Welcome to the 2008 edition of an election-year perennial: the party ID wars. Pollsters have been asking respondents the party ID question ("Generally speaking, do you consider yourself a Republican, a Democrat, an independent or what?") for more than 50 years. They differ, sometimes heatedly, about whether it is ever appropriate to weight or "normalize" their samples based on it.
The reason is that party ID is attitude, unlike purely demographic traits such as gender, age or race. Attitudes can change. It's true that most Americans tend to form attachments to a particular party, or to a sense of independence from parties, that persist for a lifetime. But some individuals will change (typically shifting between one party and the independent category) in response to the larger political climate.
Over the last four years, for example, surveys by the Pew Research Center show [PDF] the Democratic advantage in party ID among adults growing from 3 percentage points in 2004 (33 percent Democrat to 30 percent Republican) to an average of 9 points so far in 2008 (36 percent Democrat to 25 percent Republican).
We might also remember that in 2004, just after the Republican convention, five national survey organizations released polls showing net shifts in party identification of 4 to 10 points in the Republican direction. Democratic partisans protested that the result could not possibly reflect reality, and the Republican shift did abate slightly in October. However, when the votes were cast, the national exit poll indicated both parties were tied at 37 percent in identification, the strongest Republican showing (as the McCain memo demonstrates) in at least 30 years.
So were the LAT/Bloomberg pollsters right in putting out a survey showing a bigger Democratic advantage than many other recent surveys? The answer is not as obvious as the McCain pollsters' memo implies.
The memo lists party identification results in 10 polls from May and June that average out to 38 percent Democrat and 29 percent Republican. LAT/Bloomberg had roughly the same proportion of Democrats (39 percent) but far fewer Republicans (22 percent).
The problem with that comparison is that four of the most recent polls -- by Newsweek, AP/Ipsos, ABC News/Washington Post and NBC News/Wall Street Journal -- all reported a Republican identification in the range of 23 percent to 24 percent and Democratic ID ranging from 33 percent to 37 percent. While the Democratic advantage on the LAT/Bloomberg poll is bigger, it is not wildly different.
Also, most of these pollsters ask party identification at the end of their interviews, and experimental research has shown that questions asked in the middle of the survey may lead some respondents to change or withhold their partisan leanings by the end of the survey.
Consider that this particular LAT/Bloomberg survey included a series of questions describing the candidates' positions on health care, tax cuts and home foreclosures, and ended with President Bush's job rating. It also had a double-digit "don't know or refused" result for party ID, which the McCain polling memo described as "unusual." It is possible that the lower-than-average Republican identification on this poll was partly an artifact of the content of the interview and not necessarily an indicator of a skewed sample.
I agree completely with McInturff and his colleagues on their bottom line. "It is important," they write, "that both the campaign, as well as reporters covering the campaign not over-react to every single survey that is released."
So let's follow that advice and look at the chart of all the national polls on Pollster.com. It displays the usual wide band of variation in individual polls, but we also see Obama's percentage rising steadily in recent weeks and McCain's falling. Our trend estimates show Obama now holding a 6-point lead over McCain (48 percent to 42 percent), the largest so far for the Illinois Democrat in 2008.