Posted tagged ‘Experimental bias’

Quote for the Day: Stephen Pinker/Albert Einstein edition

November 20, 2008

It could be just me, but I ain’t so sure about this:

Some people raise an eyebrow at linguists’ practice of treating their own sentence judgments as objecitve empirical data.  The danger is that linguist’s pet theory could unconsiously warp his or her judgments.  It’s a legiimtate worry, but in practice linguistic judgments can go a long way.  One of the perquisites of research on basic cognitive processes is that you always have easy access to a specimin of the species you study, namely, yourself.  When I was a student in a perception lab I asked my advisor when we sould stop generting tones to listen to and start doing the research. He correcte me:  listening to the tones was research, as far as he was concerned, since he wasconfident that if a sequence sounded a certain way to him, it would sound that way to every other normal member of the species.  As a sanity check (and to satisfy journal referees) we would eventualy pay students to listen to the sounds and press buttons according to what they heard, but the results always ratified what we could hear with our own ears.  I’ve followed the same strategy in psycholinguistics, and in dozens of studies I’ve found that the average ratings from volunteers have alsways lined up with the original subjective judgments of the linguists.  (Stephen Pinker, The Stuff of Thought, 2007, p. 34)

I know (or I think I do) what Pinker is trying to say here.  You can’t even begin to formulate an idea without having some idea of what you’re looking at or for.  Professional experience and a depth of knowledge of other work in the field do count.  One’s own perceptions are real, and can (must) guide experimental design and interpretation.

But at the same time, I fear Pinker’s diminishment of the possibility of observer bias, of the fact that people have commitments both conscious and unconscious to a given idea or expected outcome.

That such expectations can deeply affect one’s ability to understand what your measurements actually are saying to you is a matter of historical fact — and this kind of observer bias can strike even the brightest of investigators, even in fields seemingly safely far removed from the subjectivity and noise that accompanies any attempt to penetrate human mental life.  Peter Galison has dissected the famous (among a certain crowd) case of Albert Einstein’s misplaced confidence in the interpretation of his collaboration with W. de Haas on an experiment to explore properties of what became known as the Einstein-de Haas effect.

The experiments the two conducted did advance the understanding of the magnetic behavior of electrons, though a proper interpretation of what was going on had to wait (in a familiar trope for early 20th century physics) for quantum mechanical intervention.  But the point here is that Einstein had made a theoretical calculation to determine the expected value of the ratio of the magnetic moment to the angular momentum of electrons travelling in their closed orbits around atomic nuclei.  In his calculation, he derived a value of one.

Then he and de Haas performed the measurement, using a delicate and complicated experimental set up. Sure enough, they were able to extract data that produced a value for the quantity to be confirmed of 1.02.   Einstein was aware that this looked almost too sweet — he wrote that the “good agreement may be due to chance” — but the coincidence of expectation and result was too much for him to ignore.

Unfortunately, subsequent experiments, and then the theoretical description in quantum mechanical terms showed the correct value to be two.

The moral?  Pace Pinker, while judgments by practitioners immersed in their fields do and should go a long way, past (and future) performance is no guarantee that observer bias ain’t about to bite you in the ass right now.   (Say I, ex cathedra — that is, someone whose last lab experience involved hideous acts performed on a frog — see E. M. Fogarty, “Anatomy of a Frog,” Journal of Irreproducable Results, 1963, 11, 65.)

That said — I’m well stuck into The Stuff of Thought and am enjoying it greatly.  I just got stuck for a moment on what might be the scientist-popularizer’s equivalent of an episode of irrational exuberance.