It’s not that McArdle can’t read…it’s that she can’t (won’t) think: part three

This is the third part of a ridiculously oversized tome on one example of what I see as a systematic failure on the right to engage science in any meaningful way. [Part one is here; part two, here]

In part two, I noted that serial offender Megan McArdle was trying to defend a claim about how health care reform will kill grandpa by asserting that the scientific literature supported that view.

The literature she cited began and mostly ended with a long paragraph quoted from a study by the Rand corporation…and in the previous post I noted that one of the problems in making the claim that McArdle’s argument was based on a rigorous review of the literature was that this paper was essentially research for hire, where the client was the world’s largest drug company.

While it is not true that just because Pfizer paid for a study that showed cutting Big Pharma revenues would result in a decline in pharma innovation that would lead to a loss in life expectancy*…it does mean that you can’t just do what McArdle did here: say “look — some folks with initials after their names confirm my unexamined conclusions.  Therefore I win!  Yippee.”

Rather, what you have to do with any piece of research, and especially one that is both making a major claim and is doing so from a clear position of interest in the outcome of the research, is to check.  You gotta interrogate the paper, its methods, its claims, its interpretations, its conclusions, the lot.

You know — basic reporting, the basic lesson we make sure each science journalism student we encounter at my shop (and every other good science writing/journalism program too) learns in the first weeks of study.

This McArdle clearly did not do.  How do I know?  I’m not (I promise) going to fisk the Rand paper top to bottom, but there are several issues with it that don’t pass the smell test right off.

The first is that the authors present their results as the output of a complicated model, itself derived from several other models for the behavior of the large variety of inputs needed to understand whether or not a cut in drug company revenue will have an impact on innovation.

A first plausible question is how the model actually works, and to what extent it has been tested.  Not to get too wonky — and not to claim expertise I certainly don’t have — but if this were a serious paper for the professional literature, you’d expect at least some discussion about the underlying logical and mathematical structure and strategy of the model.  It’s not there, at least in the publicly released form of the paper.

Next: check out the authors’ rhetoric .  It doesn’t read like scientific writing…and there’s a good reason for that.

To see what I mean, look to the paper both the Rand folks and McArdle cite as supportive of their arguments, Acemoglu and Linn, written by two MIT economists.  There you can scroll down to the final section and you see a set of graphics supplied to support the discussion above.

Some are labelled “tables” and they contain accounts of the data collected to support the model, complete with explanatory captions to allow a reader to follow the reasoning that led the authors to gather that particular slice of reality and not some other.

Some are called “figures,” and they come in the form of graphs which show what happens to that data when run through a model calculation.

Now go look at the Rand document.  It presents six graphics.  Each presents some feature of the argument they seek to make — how a given approach to pharmaceutical cost control affects innovation and or longevity.  They are easy to read, striking, even, with graphs or bar charts to show the devastating consequence of reducing producer payments to big drug companies.  They should scare anyone who wants to live their fully alloted span — as they appear to have terrified the young and impressionable McArdle.

But if you want to figure out if the graphs represent much of anything beyond conclusions expressing the assumptions with which their creators began, you can’t.  Each has the identical caption:

Source:  Authors’ calculations based on the Global Pharmaceutical Policy Model [the authors' rather modest signfiier for their black box of an analytical engine].

Just in case you were wondering — that’s the language of advocacy, not research.

The authors are saying “Trust me,” and anyone with even a passing knowledge of the movie business knows that this is the punch line to the old joke:

How does a Hollywood executive say “F*ck you?

And if you needed a yet more obvious clue, check out the label put on each graphic.  It’s not “Figure,” or “Table,” or even “Results.”  Oh no.  This is no mere milque-toast publication of data and the logic that lies behind the authors’ inferences.  That kind of thing is for the intellectually conservative, or those committed to an attempt at disinterested investigation.

The Rand team, hired by Pfizer, knows what it is doing.  It is making a case for a particular policy outcome, and hence its graphics are labeled — and I’m not kidding — “Exhibits.”

Not to belabor this — I’m after McArdle and the approach to argument she embodies, not the well-known habits of the bespoke policy research game — but one of the first and most basic lessons we try to teach our students in the Graduate Program in Science Writing in MIT is that  just because some document looks like a real scientific paper, or that  some result gets published somewhere that looks impressive, you cannot then safely conclude that what it says is true.

Rather, we tell our students, you have to read it not just for the results, but for the degree to which the paper itself does what a serious piece of research should.  Does it at a minimum provide you with enough information to ask intelligent questions about what it purports to show.  If, as here, you see such a broad tell as the word “exhibit,” then you have to know that this demands a lot more digging before you can accept its claims.  The say-so of the paper and its authors isn’t enough; they’ve told you so themselves.

It is tempting simply to ignore any paper like this one — anytime someone tells you that they’ve come up with some complicated model that gives a magic answer, a long life in science writing tells you that they are blowing smoke.  Remember: big claims require big justification.

Over time, with experience in the business (either that of science or science writing) you learn when to get revved up about something, and when to sit back and let shoddy work slide by without close examination.  Life is too short to spend one’s time doing what xkcd so famously documented.

But let’s give the Rand paper, and McArdle yet more benefit of the doubt.  All that I’ve said above suggests that the Rand paper itself is telling you that you need to dig deeper before you rely on it.  Who knows?   Maybe its conclusions are true, even if it is impossible to determine that from the evidence presented.

Well, I haven’t done anything like a proper job of reporting to that depth.  But what I got in a morning’s reading and calling is strong hint that the Rand paper is, as expected,  propaganda, nicely garbed in Rand blue.

For the details….look to part four.

Images:  Rube Goldberg cartoon.

xkcd “Duty Calls

About these ads
Explore posts in the same categories: bad writing, Economic follies, Journalism and its discontents, Medicine, Stupidity, Uncategorized, words mattter

Tags: , , , , ,

You can comment below, or link to this permanent URL from your own site.

5 Comments on “It’s not that McArdle can’t read…it’s that she can’t (won’t) think: part three”

  1. Myron Says:

    I’m automatically wary of anyone who repeatedly quotes Paul Krugman and the rest of the Keynesian cronies. It’s a sure sign of someone who believes what they are told.

    Then again I’m not a reporter.

    • Tom Says:

      It is clear that you are not a reporter. If you were, you might have searched “Krugman” on this blog, and found on the order of 13 posts (out of 668) that mention him. Of those, I count three as engaging with a specific economic claim made by PK.

      Now, it is true that I admire him, and do not admire those you list as your favorite economists, whose work, however theoretically elegant they may seem to some, has not survived rude contact with reality.

      But you have to do a lot better than this to gain credit for what I surmise is your dislike of the post above.

      • Myron Says:

        “your favorite economists, whose work, however theoretically elegant they may seem to some, has not survived rude contact with reality.”

        Where is this reality? How many free markets do you actually believe exist in the world today? I personally see zero. The central bankers have made sure Keynesian economics rule the world and look where it has gotten us.

        There are faults with any economic theory, including certain ones in Austrian economics (I’m not buying 100% gold backed currency as the answer to our problems). Having said that, I don’t see how you can insult McArdle and then turn around and support some of the nonsense that Krugman says which has been consistently refuted by almost every non-Keynesian economist.

        To clarify, however, I was in agreement with your post and was speaking of McArdle… not you. I thought this was a good piece.

  2. [...] The Inverse Square Blog science and the public square — by thomas levenson « It’s not that McArdle can’t read…it’s that she can’t (won’t) thi… [...]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 7,801 other followers

%d bloggers like this: