Friday night, assuming John McCain and Barack Obama meet to debate in Oxford, Miss., they will experience something rare in American politics: A huge television audience, perhaps a majority of the likely voters, will tune in to hear them discuss the issues for nearly 90 minutes. Can instant reaction polls tell us who won?
With less than six weeks remaining in the race for president, the Diageo/Hotline poll [PDF] tells us that more than one-fifth of voters are either totally undecided (8 percent) or say they could still change their minds about their choice (13 percent). Many are conflicted, harboring doubts about both candidates. Those uncertain voters express doubts about the readiness of Barack Obama and the ability of John McCain to understand the needs of ordinary people.
The potential for the debates to reach and influence those uncertain voters is vast. Four years ago, according to Nielsen Media Research, 62.5 million Americans watched the first debate between John Kerry and George W. Bush. That fell short of the record 80.6 million that saw Ronald Reagan debate Jimmy Carter in 1980, but it was an enormous audience nonetheless.
Despite the huge audience for debates, they rarely produce big swings in voter preferences. Tom Holbrook, a political scientist at the University of Wisconsin-Milwaukee, examined the impact of presidential debates since 1988 and found "relatively limited effects." Across 13 debates in five elections, "the average absolute change in candidate support was 1 percentage point."
What we really want to know is whether minds were changed. However, even the modest shifts of 1992 and 2004 would have fallen well within the margin of sampling error of most instant reaction polls.
The reason is that debates usually serve to reinforce existing opinions. Voters tend to be most impressed by the candidate they already support, though there have been exceptions. Holbrook found that George H.W. Bush lost 2 percentage points of his margin against Bill Clinton following the second debate in 1992, and George W. Bush's lead over John Kerry narrowed by roughly the same number following the first debate in 2004. In a race as close as the Obama-McCain matchup, even a small shift can be crucial.
So after Friday night's debate, what can surveys tell us about who won?
One of the challenges of surveys conducted immediately after the debate is that they cannot possibly provide a true representative sampling of all voters; pollsters are able to reach only the somewhat skewed audience that happens to be home near the phone after the debate ends.
Timing provides another challenge: Friday's 90-minute debate will not end until 10:30 p.m. Eastern time, and no pollster that I know of is willing to place cold calls to households at that hour. Instead, they will rely on various efforts to call back previously interviewed respondents who said they planned to watch the debate and expressed a willingness to participate in a second survey.
Even then we still confront the issue of what to measure. What we really want to know is whether minds were changed, so the ideal measure would look at shifting vote preference. However, even the modest shifts of 1992 and 2004 would have fallen well within the margin of sampling error of most instant reaction polls. So what typically gets the most attention is the nebulous question of which candidate did the best job or was the best debater.
One critical tip is to ignore the overall results and pay closer attention to tabulations by pre-debate candidate preference. How do the reactions of those who came into the debate supporting Barack Obama compare to those who initially supported John McCain?
Those results can be telling, but not always. For example, the instant polls conducted after the first debate four years ago showed that John Kerry was clearly perceived as the better debater, although the same surveys failed to pick up the small shift to Kerry that more rigorous national polls picked up in the days that followed.
Another popular tool used by many networks is the live, televised focus group -- a free-flowing discussion with 10 to 30 preselected "swing voters" conducted immediately after the debate. While these discussions make for far more compelling television than dry survey statistics, they have important limitations: Every group is a small, non-random sample, and it is hard to know the degree to which the views of participants may be influenced by the atmospherics of the telecast, the probes of the moderator or the opinions expressed by others in the group.
Mostly, the small size of the focus group "sample" combined with the vagaries of focus group recruiting sometimes combine to produce odd, conflicting results. Often, over the last year, televised focus groups on one network would paint a very different picture of a primary debate than those on another network.
Yes, the stakes in the upcoming debates are as high as they get, but the instant response polls are likely to produce inconclusive results. Sometimes the coverage in the days following the debate can have more of an impact than the debate itself. So to get a more definitive read on the impact of Friday's debate, I offer three words of advice: Wait a week.