Babes on the beach: a survey told tall tales

RALEIGH -- How can we tell when a public opinion survey is scientific -- as opposed to just spin? One survey put forward last month by the American Medical Association on the subject of college women and spring break shows just how difficult that question is.

Before getting to the AMA's survey, consider the way courts deal with surveys. Surveys are offered as evidence in a variety of legal cases. A trademark owner may offer a survey to show that consumers are confused by a similar trademark. A criminal defendant may use one to show that he can't get a fair trial in a particular jurisdiction.

Courts, though, won't accept survey evidence unless it clears a rigorous set of hurdles. The survey must select and examine a proper "universe." In other words, if the issue is confused consumers, that universe should be potential buyers of the product in question.

Second, a random sample must drawn from that universe. Courts also require that the data gathered be accurately reported and that the questionnaire and the interviewing be conducted by experts in accordance with generally accepted standards of procedure and statistics.

Those criteria can make it hard for a lawyer to get a survey before a jury. Just this month, the North Carolina Court of Appeals rejected surveys offered by an attorney who was himself accused of misleading advertising. The surveys sought to show that the public was not misled. (One survey was conducted by a Wake Forest political science professor, the other by a Duke anthropology professor.) The court brushed the surveys aside without a second thought and proceeded to affirm the disciplinary action against the lawyer.

• • •

Contrast that skeptical treatment with the AMA's survey. It was released just as colleges were setting their students free to party in Florida and other warmer climes. The findings were disturbing. Over a third of those surveyed said they went on spring break partly because it would involve drinking alcohol. A whopping 70 percent of those who went on a spring break trip said they knew friends who had more than one sex partner during the trip; 13 percent said they did so themselves. The balance of the findings were equally ominous. The survey's margin of error was reported as +/- 4 percentage points.

Needless to say, the survey was reported as fact in hundreds of media outlets. Katie Couric referred to it on "The Today Show" and USA Today ran a big story. (The N&O carried an Associated Press report on March 13.) Few could resist the temptation to report a "girls gone wild" study from a reputable organization, the AMA.

The only problem was that the AMA's survey wasn't reputable. For starters, the sample surveyed, some 644 women, wasn't a random sample but was instead drawn from a panel of persons who agreed to be polled on the Internet. Such a sample -- 100 percent of statisticians would agree -- cannot be generalized to the "universe" of female spring breakers.

Equally problematic was the presentation of the data. Remember that 13 percent who reported multiple partners during spring break? It turns out that number was drawn from the percentage of those surveyed who actually went on spring break, only 27 percent of the total. When you do the math, that leaves only 3.5 percent of the sample engaging in the risky behavior. And remember, even that number can't be generalized to the universe of college-aged women because of the sampling problem.

• • •

Challenged by the president of an association of public opinion researchers, the AMA defended itself by claiming that its survey was never intended to be "scientific" but instead was a "standard media advocacy tool." The AMA later eliminated references to the more egregious claims from its Web site, but the damage was already done. As far as I can tell, no media outlet has issued a correction or reported on how the AMA -- an organization that ought to stand for science -- played us all for chumps in a standard media advocacy sort of way.

In fact, the AMA's shoddy practices were outed in, of all things, an Internet blog, MysteryPollster.com. (Its editor, no longer a mystery, is Mark Blumenthal, a longtime political pollster who cares about his profession.) It's more than a little ironic that the only real reporting on this scandal occurred in the blogosphere, often viewed as irresponsible, while the old-line media took the AMA's bait hook, line and sinker.

Like the courts, we all have to be more educated and ultimately more skeptical about surveys pretending to be scientific. Let's resist manipulation.

(Press Millen is a trial lawyer in a Raleigh firm.)