It’s not that McArdle can’t read…it’s that she can’t (won’t) think: part three
In part two, I noted that serial offender Megan McArdle was trying to defend a claim about how health care reform will kill grandpa by asserting that the scientific literature supported that view.
The literature she cited began and mostly ended with a long paragraph quoted from a study by the Rand corporation…and in the previous post I noted that one of the problems in making the claim that McArdle’s argument was based on a rigorous review of the literature was that this paper was essentially research for hire, where the client was the world’s largest drug company.
While it is not true that just because Pfizer paid for a study that showed cutting Big Pharma revenues would result in a decline in pharma innovation that would lead to a loss in life expectancy*…it does mean that you can’t just do what McArdle did here: say “look — some folks with initials after their names confirm my unexamined conclusions. Therefore I win! Yippee.”
Rather, what you have to do with any piece of research, and especially one that is both making a major claim and is doing so from a clear position of interest in the outcome of the research, is to check. You gotta interrogate the paper, its methods, its claims, its interpretations, its conclusions, the lot.
You know — basic reporting, the basic lesson we make sure each science journalism student we encounter at my shop (and every other good science writing/journalism program too) learns in the first weeks of study.
This McArdle clearly did not do. How do I know? I’m not (I promise) going to fisk the Rand paper top to bottom, but there are several issues with it that don’t pass the smell test right off.
The first is that the authors present their results as the output of a complicated model, itself derived from several other models for the behavior of the large variety of inputs needed to understand whether or not a cut in drug company revenue will have an impact on innovation.
A first plausible question is how the model actually works, and to what extent it has been tested. Not to get too wonky — and not to claim expertise I certainly don’t have — but if this were a serious paper for the professional literature, you’d expect at least some discussion about the underlying logical and mathematical structure and strategy of the model. It’s not there, at least in the publicly released form of the paper.
Next: check out the authors’ rhetoric . It doesn’t read like scientific writing…and there’s a good reason for that.
To see what I mean, look to the paper both the Rand folks and McArdle cite as supportive of their arguments, , written by two MIT economists. There you can scroll down to the final section and you see a set of graphics supplied to support the discussion above. and Linn
Some are labelled “tables” and they contain accounts of the data collected to support the model, complete with explanatory captions to allow a reader to follow the reasoning that led the authors to gather that particular slice of reality and not some other.
Some are called “figures,” and they come in the form of graphs which show what happens to that data when run through a model calculation.
Now go look at the Rand document. It presents six graphics. Each presents some feature of the argument they seek to make — how a given approach to pharmaceutical cost control affects innovation and or longevity. They are easy to read, striking, even, with graphs or bar charts to show the devastating consequence of reducing producer payments to big drug companies. They should scare anyone who wants to live their fully alloted span — as they appear to have terrified the young and impressionable McArdle.
But if you want to figure out if the graphs represent much of anything beyond conclusions expressing the assumptions with which their creators began, you can’t. Each has the identical caption:
Source: Authors’ calculations based on the Global Pharmaceutical Policy Model [the authors’ rather modest signfiier for their black box of an analytical engine].
Just in case you were wondering — that’s the language of advocacy, not research.
The authors are saying “Trust me,” and anyone with even a passing knowledge of the movie business knows that this is the punch line to the old joke:
How does a Hollywood executive say “F*ck you?
And if you needed a yet more obvious clue, check out the label put on each graphic. It’s not “Figure,” or “Table,” or even “Results.” Oh no. This is no mere milque-toast publication of data and the logic that lies behind the authors’ inferences. That kind of thing is for the intellectually conservative, or those committed to an attempt at disinterested investigation.
The Rand team, hired by Pfizer, knows what it is doing. It is making a case for a particular policy outcome, and hence its graphics are labeled — and I’m not kidding — “Exhibits.”
Not to belabor this — I’m after McArdle and the approach to argument she embodies, not the well-known habits of the bespoke policy research game — but one of the first and most basic lessons we try to teach our students in the Graduate Program in Science Writing in MIT is that just because some document looks like a real scientific paper, or that some result gets published somewhere that looks impressive, you cannot then safely conclude that what it says is true.
Rather, we tell our students, you have to read it not just for the results, but for the degree to which the paper itself does what a serious piece of research should. Does it at a minimum provide you with enough information to ask intelligent questions about what it purports to show. If, as here, you see such a broad tell as the word “exhibit,” then you have to know that this demands a lot more digging before you can accept its claims. The say-so of the paper and its authors isn’t enough; they’ve told you so themselves.
It is tempting simply to ignore any paper like this one — anytime someone tells you that they’ve come up with some complicated model that gives a magic answer, a long life in science writing tells you that they are blowing smoke. Remember: big claims require big justification.
Over time, with experience in the business (either that of science or science writing) you learn when to get revved up about something, and when to sit back and let shoddy work slide by without close examination. Life is too short to spend one’s time doing what xkcd so famously documented.
But let’s give the Rand paper, and McArdle yet more benefit of the doubt. All that I’ve said above suggests that the Rand paper itself is telling you that you need to dig deeper before you rely on it. Who knows? Maybe its conclusions are true, even if it is impossible to determine that from the evidence presented.
Well, I haven’t done anything like a proper job of reporting to that depth. But what I got in a morning’s reading and calling is strong hint that the Rand paper is, as expected, propaganda, nicely garbed in Rand blue.
For the details….look to part four.
Images: Rube Goldberg cartoon.
xkcd “Duty Calls“