Polling conducted in the lead-up to Republican Scott Brown's victory over Democrat Martha Coakley in Massachusetts resulted in an intriguing contrast in results. Two Democratic-affiliated pollsters produced spot-on estimates of the final margin, yet barely a week before, the same pollsters had produced widely variant pictures of the race.
The poll conducted by Democratic pollster Mark Mellman on behalf of the Democratic Senatorial Campaign Committee over the final weekend showed Brown leading by 5 percentage points (48 percent to 43 percent). Similarly, the final poll conducted at the same time by Public Policy Polling, a Democratic firm that did no work on behalf of the Coakley campaign, also gave Brown a 5-point margin (51 percent to 46 percent). Both firms accurately forecast Browns' 4.8 percentage point margin of victory.
Just a week earlier, however, the two firms produced numbers that told a very different story. PPP fielded a survey from Jan. 7 to Jan. 9 that showed the candidates deadlocked (Brown 48 percent, Coakley 47 percent), while a Mellman poll fielded Jan. 8 to Jan. 10 showed Brown trailing by 14 points (50 percent to 36 percent).
In a column published in The Hill earlier this week before the results were known, Mellman explained why he believed the first automated PPP poll, along with an automated Rasmussen Reports survey conducted on Jan. 11 showing a similarly close race, had "significantly overstated" support for Brown.
An earlier survey by the Boston Globe and the University of New Hampshire had shown Coakley with a 17-point lead. Mellman argued that by Jan. 4, Brown had aired just enough television advertising "to be barely noticed, despite a clever ad."
In the wake of the release of the first PPP poll, Brown raised $1.3 million on a single day. Thus, Mellman argues, the PPP and Rasmussen polls were ultimately "a false prophecy that may have been self-fulfilling." Without those polls, he speculates, "Brown would have been unable to deliver his message." The combination of the polls and new money brought him "positive press coverage while leaving Coakley to wallow in a flood of 'what's wrong with her' stories." The polls were thus the "spark" that "ignited the dry kindling on the forest floor" leading to the "conflagration" that ultimately elected Brown.
The real impact of the automated polls may have been slightly less dramatic. Republican sources tell me that for the five days before the Jan. 11 "money bomb," the Brown campaign was taking in approximately $250,000 per day. Moreover, according to Evan Tracey, founder of the Campaign Media Analysis Group, Brown and Coakley had spent a half million dollars each on television advertising by Jan. 9. Thus, Brown had significantly increased his advertising buy in the days immediately before PPP fielded its survey and while it was in the field.
Finally, just to add one more shade of gray, Republicans tell me that between Jan. 7 and Jan. 10, Brown's internal tracking polls showed him trailing by an average of just 4 points (47 percent to 43 percent), a result that falls in between the similarly timed PPP and Mellman polls.
Mellman's question about the differences in the earlier numbers is still important, however, because as he points out, the two sets of results tell a very different story. PPP's numbers imply that "hardly anything happened in this race during the last week," while his own surveys show the race changing "dramatically" over the same period.
I can offer two likely explanations: One possibility is that the two polls ultimately selected different kinds of people. The automated methodology, some argue, ultimately interviews only the most interested and enthusiastic voters, while live interviewers are better at keeping a broader sample of voters on the phone. Since all surveys seem to identify a huge enthusiasm gap favoring Brown from the outset, PPP and Rasmussen would have shown a closer race sooner.
A second possibility is that both polls selected the same kinds of people -- both equally representative of the likely electorate -- but the automated polls produce a different measurement of their vote preferences. Voters may be reluctant to express a preference to a stranger on the phone, but some argue that by simulating the anonymity of the voting booth, automated surveys effectively push voters harder to reveal their true intentions. Mellman's survey also offered independent Joe L. Kennedy as an option, while PPP did not. Either way, this theory would argue, PPP's measurement pushed wavering voters harder in a way that revealed a latent preference for Brown.
Of course, both explanations may be at work, but this is not an academic distinction. A bias toward the most-likely-to-vote may be helpful in the days just before an election, but a skew to the most interested or informed respondents raises important questions about representativeness for polls on public policy issues conducted year-round.
The key point here is that pollsters like PPP and Mellman are in a position to provide empirical evidence to help resolve this question. The four surveys discussed in this column were drawn from samples of registered voters. Once Massachusetts updates its public records to show who voted on Jan. 19 and who did not, both pollsters could theoretically obtain the actual turnout history for the households they sampled (matching demographics to individual voters if necessary). Such an exercise could help tell us just how representative their surveys -- early and late -- were of the actual electorate.
Pollsters, what say you?
Want to stay ahead of the curve? Sign up for National Journal’s AM & PM Must Reads. News and analysis to ensure you don’t miss a thing.
Join the Discussion