Polling Science, or Framing Matters

Image by Colin Purrington via Flickr

A nice bit from a Daily Kos regular. (Of course, you should read the whole thing, inspired by a Gallup Poll question about continental drift.)

The question is also framed in terms of individual "belief" in a scientific theory. This gives the impression that data are of greater worth when they're more popular. It's the kind of wording that not only garners bad responses, it encourages bad interpretation of the results. If 90% of Americans believe in the photoelectric effect, but only 20% believe in quantum theory, what effect does that have on electronic performance? Absolutely none. Asking the question as a belief question encourages a complete misinterpretation of what science is about. What you get when a question is asked this way confuses "is this true" with "do you like it." In short, a belief question is totally worthless in measuring American's knowledge on the subject or the value of the theory. It's a measure of a theory's popularity. And science is not a popularity contest. Worse, this kind of question intractably mingles favorability and knowledge -- it's not even a good test of popularity.

Finally, the presentation of the results is particularly egregious. "Only 4 in 10 Americans..." may reflect the 42% who voted "yes" in the poll, but it completely ignores the fact that this is the most popular answer in the survey, and by a broad margin. It would be a much better description of the results to say that "far more Americans believe that Africa and America were merged in the past" than it is to misuse the raw numbers in a form that makes Americans look, well, ignorant. The results show that only a quarter of Americans don't believe this well-supported theory, while a third of the population admits that they don't understand well enough to have an opinion -- a result that should shock no one. It is, in fact, the kind of number you should expect when asking about any scientific theory.