Outsourced largely to a e-mailer to Andrew Sullivan’s blog.
The back story: A study in the journal Obesity (press release here) extrapolates from current data to suggest an enormous increase in the percentage of Americans who become obese (defined as possessing a Body Mass Index over 30). If this comes to pass, it would evoke a huge amount of spending to deal with health consequences of such American expansion.
The only problem: no one in the expanding circles of puffing this very slender piece of work took note of the key phrase which, to the original study author’s credit, did make it into the press release that otherwise over-hyped its subject. The release said: “Their projections illustrate the potential burden of the U.S. obesity epidemic if current trends continue.” (Italics added.)
Here’s the comment that — also to his credit — Patrick Appel (subbing for Andrew) then published:
It never fails to impress me the fact that people see a journal article and then turn their critical reasoning skills off. Looking through the actual paper in question, it’ll be figure 1 that’s giving the headliner quote of 86% fat by 2030. Except that this is wrong….
…the kicker: these are *linear* extrapolations, taken out well beyond where they actually tell us anything. The tell-tale hint? Take those projections out another 15 years and they say the overweight plus obesity fraction will be 100% before 2045. Yes, that’s right. Not a single healthy person left alive in the US. Marathon runners? Triathletes? Starving supermodels? Richard Simmons? All of them obese. Presumably from the fresh vegetable blight of 2040, forcing every last one of us to subsist entirely on Chicken McNuggets and Spam.Oh, and that trend they’re talking about is extrapolated from 3 data points. Sure, it’s suggestive, but I wouldn’t scream bloody murder from these stats.
….Yes. Chalk this one up there with, “According to current trends, housing prices will keep rising, allowing us to take on LOADS of bad debt!”
The moral of this story is one I and my colleagues at the MIT science writing grad program try to drum into our students very early. Just because a press release or a paper says something doesn’t mean you can suspend your bull-shit sensor. Science writing is a specialized beat because claims are asserted in technical language, and in many cases, in forms that require at least a bit of statistical due-diligence to assess.
Simply glossing a press release with a hip-ish reference to Wall-E (Wired), and then passing on the news as fact (Appel-for-Sullivan) ain’t close to good enough; in fact, I would say, this kind of slapdash reporting (or transcribing) that does a fair amount of damage to the public’s willingness to pay attention to scientific results — not as much as the overtly fraudulent kind of stuff that comes out of the Discovery Institute or climate change denialists — but still, this kind of stuff doesn’t help matters.
Now — professional or credentialed science writers are hardly immune to all kinds of flaws of their own, ranging from the cheer-leading problem (in which science writers only tell the “good” stories – and miss, for example stuff like this. (Abstract only — full article costs $).
Then there is the context problem – it’s possible, for example, to get so absorbed in the particular fashion in a field that it becomes hard to remember — and report, that there is more to physics than string theory, for example, or that the identification of the gene “for” something is only a tiny part of the biological knowledge needed to comprehend most of what’s going on in an organism.
And certainly, plenty of science writers don’t possess in themselves enough specialized knowledge to smell out dicey stories in much or most of what they cover. I could not do any of the science I have covered over the last quarter of a century. What I have learned (with some hard lessons, to be sure) is to check not just the facts of any story I want to write — but its meaning as well.
In this case, the facts were fine. A study does exist that says what the Wired item and the Appel post say it does. But it was the interpretation of those facts that was off. In this case, as the commenter above points out, the issue was simple — any trend line that suggests incidences exceeding 100 percent coming soon ought to raise a couple of alarm bells.
Ideally, this kind of first-order BS test should not require specialized beat-centered training. Anyone writing for the public about more or less anything ought to know enough about numbers to get that one; it is or ought to be as much a part of a liberal arts intellectual arsenal as is the skill of writing a clear sentence.
To that end, I wish I could publish here the guide to mathematical reasoning my colleague Alan Lightman has written to introduce the science writing grad students at MIT to the tools they can use to make sense of the hype factor in science news. He”s getting ready to turn that material into a short book, I believe, and it can’t come to soon.
In the meantime, this concise and funny book is a good place to start.
Image: Cornelis de Vos, “The Triumph of Bacchus,” 17th century. Source: Wikimedia Commons.