Mark Twain famously denounced “lies, damned lies, and statistics.” So you can imagine what his reaction would have been to the controversial survey released earlier this month by McKinsey & Co., the prominent management consulting group.
McKinsey wasn’t lying, of course, when it proclaimed that a survey it had commissioned “points to a radical restructuring of employer-sponsored health benefits” in the years following implementation of the Affordable Care Act. The real eyepopper was the assertion that “Overall, 30 percent of employers will definitely or probably stop offering (employer-sponsored insurance) in the years after 2014.” It was true that 30 percent of those responding to the firm’s survey said they would probably stop offering health insurance once the health care law took full effect. But how likely were they to actually do that? Even McKinsey (after getting beaten badly about the head and shoulders by backers of the law) later conceded that its survey “was not intended as a predictive economic analysis of the impact of the Affordable Care Act.”
So exactly what, then, was it? Well, McKinsey says, the survey was more of a point-in-time reading of employer opinion. “As noted, the survey only captured current attitudes,” the firm explained.
“Employers’ future actions will be determined by many considerations. Among them: medical-cost inflation; the details of new state health insurance exchanges; employee attitudes toward compensation and benefits; a company’s ability to attract and retain talent; actions taken by competitors; and the state of the economy.” All of which leads to one really obvious question: Why didn’t McKinsey say that in the first place?
To back up a bit, for those who missed the entire tempest, the study got a lot of straightforward, uncritical play in the media (mostly because it was from, well, McKinsey, which isn’t known for doing slipshod or partisan work). That led Republicans and other opponents of the health care law to pounce on the “30 percent” finding, saying it gave the lie to President Obama’s promise that people who liked their insurance would be able to keep it. “We are now, to our horror, finding out how harmful this measure is,” Karl Rove wrote in The Wall Street Journal, just one of many Republicans to cite the McKinsey work.
But questions about the survey were raised almost as fast, some in the blogosphere and some by the White House and Democrats on Capitol Hill. Why were the McKinsey findings so out of line with studies done by the Congressional Budget Office, the Urban Institute, and Rand, all of which predicted far fewer employers would stop offering health insurance under the new law and choose to pay the penalty instead? And what about Massachusetts, which has had a similar insurance structure in place for four years and where the availability of employer-provided insurance has actually increased?
Most troubling, however, was the fact that McKinsey refused to release the survey’s methodology. The American Association for Public Opinion Research has a “transparency initiative” to “encourage the routine disclosure of methodological information from polls and surveys whose findings are released to the public.” But nowhere in the lengthy (and even footnoted) article in the McKinsey Quarterly did it mention anything about how the survey was conducted or even who did it. In fact, it took almost two weeks and an increasing number of demands, largely from Senate Finance Committee Chairman Max Baucus, D-Mont., for McKinsey to cough up its survey questions.
Once the firm did, there was little doubt about why it was so reluctant. It turns out that employers were asked leading questions that made it seem logical for them to stop offering insurance. Respondents were told that the new health insurance exchanges would become “an easy, affordable way for individuals to obtain health insurance.” Then they were given examples of how little their low- and moderate-income workers might have to pay for insurance, thanks to new federal subsidies. Only then were they asked how likely they would be to stop offering health insurance.
There is also a big question about the survey’s actual methods. Scott Keeter, director of survey research for the Pew Research Center and president of the American Association for Public Opinion Research, noted that the survey is, in fact, not even a random sample of business owners. Instead, he said, it’s a collection of responses from an amorphous group of people who may not be business managers responsible for decisions about health benefits. “Unlike the questionnaire, which they disclosed, we know relatively little about the sample, what part of the relevant population of businesses it covers, and even whether the people in the sample are really the key decision makers regarding health benefits,” Keeter said. “Many of the respondents expressed uncertainty about basic facts about the benefits in their companies.”
Despite all of that, McKinsey says that it “stands behind the integrity and methodology of its survey.”
So, what are the real lessons here? First, in the highly charged partisan atmosphere that surrounds all aspects of the new health law, Republicans and Democrats will spin any piece of information to their best advantage. And second, it might behoove more than a couple of journalists to review what they were taught early in their careers about reporting on “surveys” that don’t include those boring pages on how they were conducted.
This article appears in the June 30, 2011, edition of National Journal Daily.