Archive for the ‘science process’ category

The Land Of Broken Links

February 6, 2013

Copyright is broken. Intellectual property in general has become a troll’s playground. Jump to that link to see how easy it is for extortionists to invert every rationale for a patent/copyright regime, transforming the support of innovation into simple theft.

But we knew that.

Still:  the latest outrage to hit my twitter feed gave me real pause.  It’s an example of the ease with which private censorship can manipulate the IP legal regime to disappear uncomfortable speech.  I don’t know how many of you know of the excellent site Retraction Watch, founded and run by Adam Marcus and Ivan Oransky (full disclosure — Ivan’s a friend of mine).

The site monitors the scientific and medical press to identify and discuss withdrawn research papers — its motto is “Tracking retractions as a window into the scientific process.”  The fact of retraction is occasionally a sign of genuine error, but as Ivan and Adam have documented, betting on misconduct is never a foolish option.

Georges_de_La_Tour_-_Cheater_with_the_Ace_of_Diamonds_-_WGA12334

Yesterday Ivan put up a post that differed from the usual fare of discredited research and queries about peer review or the editorial processes involved. Titled “WordPress removes Anil Potti posts from Retraction Watch in error after false DMCA copyright claim”*, the piece documents how an obscure Indian website managed to persuade Word Press to remove ten posts about former Duke University researcher Anil Potti.  As described in an Ars Technica article,

Potti first fell under scrutiny for embellishing his resume, but the investigation quickly expanded as broader questions were raised about his research. As the investigation continued, a number of Potti’s papers ended up being retracted as accusations of falsified data were raised. Eventually, three clinical trials that were started based on Potti’s data were stopped entirely. Although federal investigations of Potti’s conduct are still in progress, he eventually resigned from Duke.

By Ars Technica’s count, Retraction Watch has so far published 22 posts on the Potti case.  Ten of those have now gone missing.  Here’s Ivan:

If you went looking for ten of our posts about Anil Potti today, you would have seen error messages instead. That’s because someone claiming to be from a news site in India alleged we violated their copyright with those ten posts about the former Duke University cancer researcher who has had 19 papers retracted, corrected, or partially retracted.

The claim, as Ivan documents, is bullshit (a term of art, of course, but mine, not deployed by Retraction Watch):

If you click on any of the NewsBulet.In URLs provided in the takedown notice, you will indeed find the text — and images — from ten of our posts about Anil Potti. But as will be abundantly clear to anyone who does so that our text was placed on NewsBulet.In, not the other way around.

In other words, NewsBulet.In is violating our copyright; we are not violating theirs. That’s driven home by the fact that the site did not exist until October 2012, according to a WhoIs search. All but one of the Retraction Watch posts they cite appeared before they even existed.

Retraction Watch is on the case — neither of its two authors fell off a turnip truck recently.  But even if — when — the material gets restored to the site, the chilling power available to those who would use copyright for evil is obvious.  The assertion of bluntly false claims is hassle enough — and if it distorts or simply constrains folks’ ability to cover controversy, then the damage is obvious.

Of course, the problems with copyright (and the patent system) extend far beyond overt nonsense like that which Retraction Watch confronts today.  For a historically grounded insight into our troubles, I highly recommend Lewis Hyde’s Common as Air, in which Hyde examins what our founding fathers actually meant by the intellectual property system they advanced at the birth of the American republic. In the here and now, the problem — or at least one big one — is that it is just too damn easy to disrupt the free exchange of ideas with spurious claims, acts for which there are no consequences sufficient to offer an incentive to play nice.

I have no good idea how to dig out from this mess (though any solution that makes Mickey Mouse cry would be a move in the right direction, I think).  You?

*OK.  I’ll admit that the headline isn’t quite in the “Headless Body Found In Topless Bar” league

Image:  Georges de la Tour, Cheater with the Ace of Diamonds, 1635.

Conservative Science: Yur Doon It Rong

February 11, 2011

DougJ and Tim F. have both weighed in on the John TierneyMegan McCardle “Why are universities so mean to conservatives?” whimper. DougJ is at once gobsmacked and confirmed in his view of the cluelessness of the “argument” advanced, while Tim F. sees much less here than meets his perhaps-jaundiced eye.

Me?  I can’t quite agree with Tim’s sense of the pithlessness of this latest attempt to demand equal treatment for principled young earthers amongst the ravings of all those Foucoultvian mathmeticians.*

I’m more with DougJ, as to me, Tierney and McArdle are firing one more shot at the whole idea of authoritative knowledge as a source of influence in civil society.  Partly, this is just self interest: the more folks like McArdle can devalue the status of expert knowledge, the less they have to fear of correction by those who, in fact, both know and understand more than they do.

But beyond the goal of reducing personal embarrassment, the more that the independent authority of scholars and scientists can be diminished, the easier it becomes for ever more risible statements to take on the status of holy writ.  After all, we all know that progressive taxes infallibly drive away the rich…Right?

That said (and there’s more to be done with a detailed fisking of both Tiernery and McCardle, which I may yet attempt) here I just want to point out that Tierney undercuts the entire farrago with one injudicious anecdote.

His source, Jonathan Haidt, a U. VA social psychologist, made the central claim that Tierney takes up and McArdle then amplifies.  Haidt claims that  the political orientation of the members of his field is so overwhelmingly liberal that only discrimination can account for that distribution.  His proof?  A show of hands at a conference.**  Other than that, the only other Haidt evidence Tierney references comes from an email from an allegedly victimized student:

“I consider myself very middle-of-the-road politically: a social liberal but fiscal conservative. Nonetheless, I avoid the topic of politics around work,” one student wrote. “Given what I’ve read of the literature, I am certain any research I conducted in political psychology would provide contrary findings and, therefore, go unpublished. Although I think I could make a substantial contribution to the knowledge base, and would be excited to do so, I will not.”

O. K. class.  What does this complainant get wrong?

I’ll give you a hint.  Look again at this sentence:

Given what I’ve read of the literature, I am certain any research I conducted in political psychology would provide contrary findings and, therefore, go unpublished.

That is: this student says that he or she is “certain” that her/his results would break consensus, and hence, inevitably, would languish in conspiratorially enforced obscurity.

Uh, kid.  Listen up:  When you already know what your research will reveal, what does that tell you?

It ain’t research.

You have no knowledge to “contribute to the knowledge base” if the conclusions you propose to add to our collective store of human wisdom is what you already know by some process other than the “research” you propose.

Note that Haidt’s anonymous disappointed  ideologue tells us of his/her intention to respond to her/his field’s stunning lack of awe at this proposal by picking up his/her marbles and going home.

Which is another way of saying that this student found it impossible to do the actual hard work of science:  construct testable hypotheses, and experiments in which the results may in fact confound your expectations.   If you won’t do that, you can’t make it science … and Hey, Presto! another conservative is discriminated against.

Errr…

This is, of course, exactly the problem we face in trying to get our political discourse to respond to what we do actually know about the world.

So, to cycle back to the beginning of this post, I agree with Tim that the overt attack on the “liberal” academy has faded a bit since the haute crazoid days of 2002 and 2003.

But this drip, drip, drip of suggestion that somehow everything we know or discover about the world is tinged by partisan contingency does enormous damage, more so by far, IMHO, than any transparent direct assault on the academy

All of which is why it remains vital to remind folks over and over again that one big reason modern American conservatives have such trouble in so much of the academy is because reality possesses that well known liberal bias.

*No, really.  Here’s McCardle, verbatim: “No, I’m not saying you have to hire a Young Earth Creationist to be a biology professor, but I don’t see why it should matter in a professor of Mathematics or Sociology.”  That she doesn’t see the problem here is a precise representation of why conservatives of the McArdle stripe have a hard time in the academy.  The notion that bodies of knowledge contain worldviews doesn’t seem to penetrate her consciousness.  In plain language:  it’s really hard to do empirical research or construct complicated proofs in a wide range of fields if you have a deep commitment to something that denies a mountain of physical evidence and logical argument.  By way of analogy:  you slouch your whole life (towards Bethlehem?) it becomes increasingly difficult to stand up straight.  Same things go with habits of mind.

The shorter:  you can’t hide the crazy forever, and when it emerges, it makes your colleagues (justifiably) nervous about anything you say.

**Yup, really.  I might guess that Haidt has done some real research on this point, but Tierney doesn’t let one know.   All he draws upon is Haidt’s account of his own speech.  Which, in journalistic terms, is the tell.  Tierney misleading (one might say, actually deceptively) cites some studies, but at no point does anyone but Haidt speak, and  no time does either Tierney or his subject offer anything but assertions.  Which is to say –this isn’t journalism; as a bit of advocacy (that’s the polite word) it would be properly situated at the Corner, and not The New York Times.

Images:  Raphael, School of Athens, 1505.

Paul Cézanne, Harlequin, 1888-1890

Quickie Must-Read Link …

August 5, 2009

This, from The New York Times. (h/t  DougJ at Balloon Juice)

In brief:  court documents reveal that big Pharma co. Wyeth paid a medical communication firm to ghostwrite 28 review articles slanted in support of hormone replacement therapy for menopausal women for seven years between 1998 and 2005.  That effort supported a boom in the sales of Wyeth’s products in that area, up to the point when this happened:

But the seeming consensus fell apart in 2002 when a huge federal study on hormone therapy was stopped after researchers found that menopausal women who took certain hormones had an increased risk of invasive breast cancer, heart disease and stroke. A later study found that hormones increased the risk of dementia in older patients.

Read the article as a whole.  It is important as a public policy issue, and it is perhaps even more so as a deep challenge to science as a civic enterprise.

A couple of thoughts there, very quickly, as this is a big area and I actually want to think and research a bit before charging in.

Mostly, I’d agree with one comment, I think from the Balloon Juice thread, that this is the science community’s answer to the steroid scandal in baseball.

It makes it almost impossible not to question any result published in even the most seemingly prestigious journal; certainly any research report and especially any review article on an area in which major financial interests are at play has to be read with a “who benefits” filter on high.

There’s no way to pretend that the myth of science as a disinterested truth community is an accurate description of the world we live in any more.  (If ever it was).  Of course, individual by individual and lab by lab — and lots of people I know personally — would not countenance the kind of deception in which Wyeth and its enablers indulged.  But science is a big country, and the amount of cash flowing through some of its provinces is enough to compel not just the ordinary skepticism that is part of the scientist’s toolkit, but that worldly reality check that tells us to follow the money.

The other thought, and its really half a thought, bears on an argument I’ve had running with Bora over at PLoS — one of the parties whose legal action brought this scandal to light — about the question of story telling and science communication.  He’s written repeatedly of his view that the construction of a story structure around a body of information distorts and even can smother the actual scientific result that should be what is being communicated.

I’ve told Bora that this is, to me, nonsense.  Information without context and data presented without some vector to carry it into the mind of its intended audience will simply disappear.

Part of our dispute lies in the very different sense of audiences we have.  Bora is concerned both with communication between researchers and the communication of research to a broader audience.  I’m interested in the former, but am really concerned with the latter.

In communication between scientists working within fields, the data really is the key.  I was speaking to a friend recently who works in a senior position at a major drug company, directing one of the major areas in which basic science and medical applications ram right into each other.  That researcher/manager told me that all that matters to the decision making process is the data.  The discussion, the interpretation, the “spin” a scientist might put on their data is secondary, or even worse, special pleading. What matters is the pure info about what was done, with what methods and instruments, to produce what measurements.

But in communicating to lay audiences — and even, as the Wyeth story suggests to somewhat more broadly constructed scientific ones — that doesn’t seem to me to be a plausible approach.  You have to tell a tale that a reader not necessarily either interested or informed about the context of the work will be able to follow, and that will hold her or his interest sufficiently to keep them reading graf to graf, until the story’s end.

But as I understand him, for Bora the problem — and it’s a real one  — is that it’s terribly easy to construct of essentially false narratives that distort the meaning of the science from the start.  See, e.g. all the writing that has floated the autism-vaccine woo for so long.  For me the issue lies with finding a way to express as narrative the key ideas to be communicated without distorting them — and thereby hangs a much longer tale than I’m going to write here.  Doing that is, in essence, what we try to teach our students at the Graduate Program in Science Writing at MIT, and I deeply believe that this is an essential civic-society endeavor.

That’s all by-the-by to what made my ears perk up in this story of Wyeth’s and their tame “authors” misdeeds .  Apparently, most of the astroturfed articles were review articles — summaries-and-interpretations of the state of research in hormone replacement therapy.  Review articles are, of course, a genre of  scientific literature wrapped up in storytelling.  By its nature, it demands the collation of a range of incidents — individual research reports — into a sequence logically and narratively designed to lead the reader to the interpretation of the state of the field that its author wants to advance.

There is nothing wrong with such a genre; quite the reverse.  It exists in part to provide gateways into bodies of work and ideas, and it is all the more necessary in fields in which sub specialties throw up information useful to practitioners within the field but beyond the speciality…which pretty much describes all of molecular biology, for one.

But as the Wyeth story reconfirms, the writing of review articles is prone to precisely the kind of abuse that Bora and many others have decried in popular writing about science: distortion based on constrained or disputed points of view, misinterpretation or misunderstanding (ignorance of the subtleties) of works under discussion, mis-emphasis on one point or another … and outright corruption, as above.

I’m not trying to defend popular science writing and its discontents here by saying that similar problems exist within the scientific literature.  What I am saying is that the Wyeth case is an extreme version, a morally bankrupt one, of two facts of life.

One is that money talks.  The other is that the way human beings tell each other important things contains within it real vulnerabilities.  But any response that says don’t communicate in that way doesn’t make sense; the issue is not how to stop humans from organizing their knowledge into stories; it is how to build institutional and personal bullshit detectors that sniff out the crap amongst the good stuff.

As I said — only half thoughts for now.  And rather meta at that.  The real story is, of course, that drug companies really, always, do have our best interests at heart.  Right!?  (And I won’t link to the latest Megan McArdle foolishness on this score, noting only that she has qualified another of my favorite real fitness reports for British military officers:  “Since my last report, (s)he has hit bottom and started to dig.”

Image:  Sir William Fettes Douglas, The Alchemist, before 1891

Science Bloggers vs. Science Writers Round 2: It’s Just A Theory dept.

December 28, 2008

This may be a true blogospheric case of a day late and a dollar short, but I’d like to pursue a few more threads drawn from the kerfluffle Bora set off with this post.  I want to get into the meat of what Bora wrote — especially in two areas, but before I get there I wanted to take a swipe at what I see as a dangerously mistaken notion put forward by one of Bora’s sciblings in the intitial wave of responses to the initial post.

This is what Ed Yong had to say in an otherwise smart piece that offered some good advice to scientists confronting the media:

…the majority of journalists are not seekers of the truth; they serve at the all-important altar of “The Story” and the ultimate goal of The Story is to keep the reader/listener/viewer entranced with it from opening word to final syllable. It’s entertainment…

There is a critical error in this passage, and it is one that I have seen repeated again and again by scientists (and others, to be sure) for whom the complexity of their own work blinds them to the technical hurdles faced in other fields.  (That this happens within science before it even makes its way out to confound science/rest-of-the-world interactions is the point made by Greg Laden in the post on which I blogged below.)

The error lies with the claim that the goal of communicating in story-form is “entertainment.”  This has essentially the same affect for a writer as saying that evolution is “only a theory” has for anyone who knows anything about that the science of life.

Entertainment occurs in the presence of a well-made story, certainly, but that pleasure derives from success at the primary goal of story:  engagement.  Ed has it right in the sentence before:  the function of story form is to enable the author to hold a reader’s attention to the end of what he or she is trying to say — and to do so in a way that will enable that audience to understand and remember whatever it was the writer was trying to communicate.

Not to go all evolutionary psychology on y’all (though that is a tale-telling discipline as ever was) but story structure is something that human beings seem to rely on to frame meaning and to construct memory.  I’m not versed enough in the neuroscience to pull up chapter and verse here, (but see, e.g., some of Jonah Lehrer’s writing for a variety approaches to connect brain function and human culture)  but it is clear anthropologically that stories are the ways human beings have organized their knowledge for a very long time.

None of this is remotely new, nor I suspect, any surprise to scientists reading this; science is, pace Ed, (and perhaps Bora?) a culture deeply steeped in story telling from the informal  level of conjecture in the lab or seminar up to and including (some but not all) of its most formal communication.  A couple of examples:

Albert Einstein’s first relativity paper, “On the Electrodynamics of Moving Objects,” begins with a little story: consider this, Albert writes, this little mystery.  According to what we now say we know, if a magnet moves through a conductor, an electric field is formed that produces a current in the wire.  If, on the other hand, the conductor moves while the magnet remains at rest, no field is formed, “while in the conductor an electromotive force will arise, to which in itself there does not correspond any energy, but which ….gives rise to electrical currents.” (Collected Papers of Albert Einstein, vol. 2, English translation, p. 140.)

This little anecdote is a story in itself, of the “Let me tell you something strange” variety, almost a tall-tale.  It is also what the screenwriting types call the inciting incident — a mystery or a problem to be solved through a series of narrative incidents, the sequence of mathematical derivations that Einstein pursues to reach his extraordinary narrative (and physical) conclusions.

Charles Darwin explicitly framed his great work, The Origin of Species, as an argument and nothing like  a novel, but it is an essay permeated with stories from the descent of fancy pigeons from the rock dove, to the narrative of sedimentation that underpins the assertion that the geological record is imperfect, to the hypothetical narrative here, just a few pages before the fabled “tangled bank” scene:

When we no longer look at an organic being as a savage looks at a ship, as at something wholly beyond his comprehension; when we regard every production of nature as one which has had a history; when we contemplate every complex structure and instinct as the summing up of many contrivances, each useful to the possessor, nearly in the same way as when we look at any great mechanical invention as the summing up of the labour, the experience, the reason, and even the blunders of numerous workmen; when we thus view each organic being, how far more interesting, Ispeak from experience, will the study of natural history become!

Last, I promise):  I’ll be publishing this June my account of Isaac Newton’s work at the Royal Mint, chasing counterfeiters and helping to create the modern financial world (thanks, Isaac).  Along the way, I read Principia, and while no one has ever accused Newton of ripping prose style, I found that when you read that book as a book, and not as a series of demonstrations, Book Three, “The System of the World,” has a narrative structure that is integral to the argument Newton was trying to make:  that his new mechanics extended through the entire universe, to its infinite, and to human senses inaccessible, extent.  He did so by the way he organized that last section — which takes on the recognizable form of an epic journey.

But all that, of course, was then.  What about now?  Modern scientific communication is a highly formalized and artificial genre, of course.  No one reading this has to be told that. But story still creeps in, as it must, given the way people tell themselves stories about what they do as the ideas frozen into papers take shape.  The issue is not the data, but, as in the dispute with which Bora led off his original post, in the interpretation of whatever has been measured or observed.  For interpretation, read story, as in what story does my experiment tell me?; as in, who has the better story here?  Darwin again:

Authors of the highest eminence seem to be fully satisfied with the view that each species has been independently created. To my mind it accords better with what we know of the laws impressed on matter by the Creator, that the production and extinction of the past and present inhabitants of the world should have been due to secondary causes, like those determining the birth and death of the individual.

Is Darwin an entertainer?  Well, yes he is, if you have a certain cast of mind.  But is that pleasure, the thrill at a well turned thought, leading you just where the writer wants you to go, his primary ambition and accomplishment?

One last thought.  The real conflict between science writers and their scientist-sources does not seem to me to be the question of accuracy.  It really lies in the fact that writers and their subjects disagree on who owns the story being told.   For scientists acting as sources, it ain’t theirs.

That gets to the meat of what Bora was arguing, that the emergence of the blogosphere and of the communications technology behind it in principle will eliminate much or all of the need for intermediaries like science writers.  I think he’s wrong, mostly, and I’ll take that up in another piece.  But in the meantime, I’ll leave you with an anecdote that illustrates the underlying tension.  A few years ago I wrote a piece that became a cover story for Discover on some of the issues raised in the race to construct the next generation of extremely large optical telescopes.

In that piece I focused on one instrument, the Giant Magellan Telescope, for two reasons.  The first was that the GMT group had decided to start building their hardware long before they had the full sum in hand to construct the entire observatory — and I could use the casting of the first of seven mirror segments as my path into the subject.  The second was that my story was not simply about building big ‘scopes, but focused instead on the questions raised around the choices of dozens of people working on that and other similar projects to commit enormous chunks of their careers to such an uncertain goal.  Think LHC, think the James Webb Space Telescope, think, even, of the human genome project at its inception, think of the GMT’s competitor project, the Thirty Meter Telescope or TMT.

And that’s where the problem emerged.  I conveyed that theme through the stories of a half dozen different people working on the GMT, and those mini-stories occupied most of my account.  I did interview both Richard Ellis, of Caltech, and Jerry Nelson of the University of California, Santa Cruz, the two leaders of the TMT project.  I told them both up front that the emphasis of my article would lie with their rivals, but that I wanted to make sure to include enough of what they had to say so that my readers would know that there was more than one project striving after the same goal.

And that’s what I wrote.  And Richard and Jerry were both upset, and for good reason from their perspective, and they told me so very clearly.

That good reason:  there was and remains a public and private competition between the two projects for funding, in which a sense of inevitability, of unchecked progress, was very important.  An article that featured one project much more than the other was an advantage to one side, (and the TMT people were already aggrieved after Dennis Overbye published his New York Times account of the GMT mirror casting without any significant mention of the TMT folks.)

But I wrote back that, in effect, they had no cause for complaint — for two reasons.  One was that I had a story I was trying to tell, and it was about one aspect of life in science, not simply about one machine or another; I may or may not have succeeded in telling that story, but that I didn’t tell a different story hardly seemed (or seems) to me an adequate critique.

The other was that I am interested and have been for a long time in the interplay between instruments and discovery, especially in astronomy, and I planned to keep on covering what is a truly remarkable story of transformational technology unwrapping the universe.  I had every intention of doing a TMT story as soon as there was (a) enough time passed to allow the market for big telescope stories by Tom Levenson to recover, and (b) there was some kind of a hook to hang on which to hang the piece.

I haven’t written that piece yet.  Partly, I’ve been busy; new job, new book, kids, life, twelve inches of snow to shovel last weekend, all of the above. And partly, this brutal fact:  there are many more good stories to tell in the world than I or anyone has time to write.  If I or any writer get a rocket from some source about the wreckage they’ve made of some story, I don’t say I’ll never write about that person’s work again.  But I’m human enough to hesitate to pick up the phone to call up such an aggrieved soul.  Other things come along, other pebbles on the shore catch the light and grab my magpie’s interest.

The moral:  if you are upset that the story you would have told was not, and you do not choose to write it yourself, then think about how you might want to convey your disappointments and your hopes to the offending story-teller.  And as for me, this new-year’s resolution.  I’ll give Jerry a call this January and find out how goes the TMT.  (Better than the GMT, I think, given that Gordon Moore’s foundation decided to give the California-based project a ton of bucks that the GMT consortium has yet to match.)

Images:  Pierre-Auguste Renoir, “Portrait of Jean and Geneviève Caillebotte,” 1895. Source:  The Yorck Project: 10.000 Meisterwerke der Malerei. DVD-ROM, 2002. ISBN 3936122202. Distributed by DIRECTMEDIA Publishing GmbH.

Jan Matejko, “The Astronomer Copernicus in Conversation with God,” 1872.

And Spare A Thought For…

December 25, 2008

The Beagle 2 Lander — lost. presumed wrecked on this day 2003.

Not only is it appropriate to remember this one-among-many-failed space missions on the eve of the Darwin year, but it serves as a more general reminder of how hard it is to do science.

If the stuff we wanted to know (is there/was there life on Mars?; what underlies the remarkable order we observe in the universe?; what explains the odd fact that the object typing these letters is aware of itself typing these letters?; and so on) was easy, then everyone would do it and/or we would know all there is to be known.

Ain’t happened yet; doesn’t seem likely that it will.  The little Beagle, silent this last half a decade gives one minor insight into why.  So raise a glass to it, and to those who thought the gamble worth the risk of sending it off in the first place.

Happy Newton day all, again.

Image: Chasma Boreale, a feature of Mars’ north polar ice cap.  NASA Mars as Art gallery.

Quote for the Day: Stephen Pinker/Albert Einstein edition

November 20, 2008

It could be just me, but I ain’t so sure about this:

Some people raise an eyebrow at linguists’ practice of treating their own sentence judgments as objecitve empirical data.  The danger is that linguist’s pet theory could unconsiously warp his or her judgments.  It’s a legiimtate worry, but in practice linguistic judgments can go a long way.  One of the perquisites of research on basic cognitive processes is that you always have easy access to a specimin of the species you study, namely, yourself.  When I was a student in a perception lab I asked my advisor when we sould stop generting tones to listen to and start doing the research. He correcte me:  listening to the tones was research, as far as he was concerned, since he wasconfident that if a sequence sounded a certain way to him, it would sound that way to every other normal member of the species.  As a sanity check (and to satisfy journal referees) we would eventualy pay students to listen to the sounds and press buttons according to what they heard, but the results always ratified what we could hear with our own ears.  I’ve followed the same strategy in psycholinguistics, and in dozens of studies I’ve found that the average ratings from volunteers have alsways lined up with the original subjective judgments of the linguists.  (Stephen Pinker, The Stuff of Thought, 2007, p. 34)

I know (or I think I do) what Pinker is trying to say here.  You can’t even begin to formulate an idea without having some idea of what you’re looking at or for.  Professional experience and a depth of knowledge of other work in the field do count.  One’s own perceptions are real, and can (must) guide experimental design and interpretation.

But at the same time, I fear Pinker’s diminishment of the possibility of observer bias, of the fact that people have commitments both conscious and unconscious to a given idea or expected outcome.

That such expectations can deeply affect one’s ability to understand what your measurements actually are saying to you is a matter of historical fact — and this kind of observer bias can strike even the brightest of investigators, even in fields seemingly safely far removed from the subjectivity and noise that accompanies any attempt to penetrate human mental life.  Peter Galison has dissected the famous (among a certain crowd) case of Albert Einstein’s misplaced confidence in the interpretation of his collaboration with W. de Haas on an experiment to explore properties of what became known as the Einstein-de Haas effect.

The experiments the two conducted did advance the understanding of the magnetic behavior of electrons, though a proper interpretation of what was going on had to wait (in a familiar trope for early 20th century physics) for quantum mechanical intervention.  But the point here is that Einstein had made a theoretical calculation to determine the expected value of the ratio of the magnetic moment to the angular momentum of electrons travelling in their closed orbits around atomic nuclei.  In his calculation, he derived a value of one.

Then he and de Haas performed the measurement, using a delicate and complicated experimental set up. Sure enough, they were able to extract data that produced a value for the quantity to be confirmed of 1.02.   Einstein was aware that this looked almost too sweet — he wrote that the “good agreement may be due to chance” — but the coincidence of expectation and result was too much for him to ignore.

Unfortunately, subsequent experiments, and then the theoretical description in quantum mechanical terms showed the correct value to be two.

The moral?  Pace Pinker, while judgments by practitioners immersed in their fields do and should go a long way, past (and future) performance is no guarantee that observer bias ain’t about to bite you in the ass right now.   (Say I, ex cathedra — that is, someone whose last lab experience involved hideous acts performed on a frog — see E. M. Fogarty, “Anatomy of a Frog,” Journal of Irreproducable Results, 1963, 11, 65.)

That said — I’m well stuck into The Stuff of Thought and am enjoying it greatly.  I just got stuck for a moment on what might be the scientist-popularizer’s equivalent of an episode of irrational exuberance.


Follow

Get every new post delivered to your Inbox.

Join 8,662 other followers