A statistics convention is not exactly the place you would expect to see a dramatic presentation. But if not drama, one presentation at last week's Joint Statistical Meeting in Washington, billed as the world's largest annual gathering of statisticians, did at least offer something of a surprise: A pollster willing to say that the telephone polling he conducts for a living is "doomed" by 2012.
The pollster was Jay Leve, editor and founder of SurveyUSA, and he spoke as part of a panel discussion organized by the American Association for Public Opinion Research.
At the outset, his pitch was consistent with what one would expect from someone selling wares to a potentially skeptical audience. "I'm not a statistician," Leve demurred, before beginning a rapid-fire defense of the "recorded voice polling" that his firm has conducted for the last 18 years.
SurveyUSA counts more than 60 local television stations as clients, so Leve is no stranger to selling his automated methodology, and it shows. His crisp, data-heavy presentation reflected the scars of many years of slings and arrows from the traditional polling establishment. The polling standards adopted by the New York Times, the Washington Post, the Associated Press and ABC News still forbid the reporting of surveys conducted using an automated, "interactive voice response" methodology, but Leve is undaunted.
"If you came into the room thinking that recorded voice pollsters were inferior," he said, "there's no data to support" that contention, from 2008 or prior years. Leve rifled through a half-dozen slides on independent analyses from the National Council on Public Polls, AAPOR's Ad Hoc Committee on 2008 Primary Polling, the Pew Research Center, the Wall Street Journal and Nate Silver's FiveThirtyEight.com. All show automated, recorded voice surveys performing "better or at least comparable" to conventional, live interviewer surveys.
But then the presentation took an unexpected turn. Leve began to outline a set of challenges facing not just his company, but all telephone pollsters. He noted the havoc wreaked by the dramatic rise in the percentage of younger people with cell phones but no landline telephone service -- a near tripling in four years among those aged 18 to 24, to over 40 percent, according to statistics produced by the National Center for Health Statistics.
"For every 100 young people, age 18 to 24, that we should have had [last year] in our [unweighted] samples," Leve said, "we had only 24." His chart showed a complementary over-representation of college-educated and upper-income respondents.
While SurveyUSA, like all pollsters, attempts to weight demographically to correct these patterns, Leve admitted that the problem of declining coverage is "a problem I don't know how to correct." Interviewing by cell phone represents a "dramatic" increase in costs for all pollsters, but especially for the recorded voice polling he conducts.
To conclude his talk, Leve summed up the problem. All phone polling, he said, depends on a set of assumptions:
You're at home; you have a [home] phone; your phone has a hard-coded area code and exchange which means I know where you are; ... you're waiting for your phone to ring; when it rings you'll answer it; it's OK for me to interrupt you; you're happy to talk to me; whatever you're doing is less important than talking to me; and I won't take no for an answer -- I'm going to keep calling back until you talk to me.
The current reality, he said, is often much different:
In fact, you don't have a home phone; your number can ring anywhere in the world; you're not waiting for your phone to ring; nobody calls you on the phone anyway they text you or IM you; when your phone rings you don't answer it -- your time is precious, you have competing interests, you resent calls from strangers, you're on one or more do-not-call lists, and 20 minutes [the length of many pollsters' interviews] is an eternity.
All of this brought Leve to a somewhat stunning bottom line: "If you look at where we are here in 2009," for phone polling, he said, "it's over... this is the end. Something else has got to come along."
Now I can hear some heads shaking. We have long heard many such "sky is falling" prognostications from polling's critics or from those selling the next big thing, but consider the source: Leve's current business model depends entirely on telephone polling. If he is to profit in the future from "something else," his company has yet to develop or sell that something.
I asked Leve what form he thinks the "next big thing" in survey methodology might be, but he declined to speculate except to say that the key is the "new paradigm" in which Americans have "two-way communication in their homes, on their belts, in their briefcase 24 hours a day."
And for those who might ask, he adds that he "doesn't look to the future with despair but with wonder" at the opportunities for the polling profession. I wonder how many other pollsters will agree.