Evidence is the raw fuel of science: We collect data, it pushes forward our understanding. But there is also anti-evidence: observations that have the effect of holding back our understanding. The clearest example I know comes from experiments that supposedly “tested” mathematical learning theories in the 1950s and later. The observation was that the theory could fit the data. Theorists wrote papers to report this observation. In fact, the theory was so flexible it could fit any plausible results. The papers, which were taken seriously, retarded the study of learning because they wasted everyone’s time. They gave the illusion of progress. Hal Pashler and I wrote about this.
Another example of anti-evidence, I think, is the sort of data that linguistic theorists have been fond of: Observations that this or that sentence or sentence fragment strikes the theorist as grammatical, i.e., possible. Not studies of how people actually talk; the observation that a speaker of English or whatever could say this or that. The theorist’s judgment based on introspection. I’m not saying that this isn’t actual data of some sort; I just suspect that the value of these sorts of observations has been overrated and the net effect has been to keep linguists from collecting data that would push theorizing forward.
Months ago I blogged about how I found that when I made playing a game contingent upon clearing off my kitchen table, I was able to clear off the table. Which had been messy for quite a while. My question: is this evidence or anti-evidence? If I think about this, and try to understand it, will I be deluding myself, as the mathematical learning theorists and the linguistic theorists deluded themselves? On its face, it seems like a very ordinary, very narrow observation, much like the observation that “George played with the game Dave brought over” is a possible English sentence. On the other hand, it is something unusual and helpful that actually happened, unlike an observation that this or that is a possible English sentence.
When someone says “the plural of anecdote is not data,” you can be sure their grasp of scientific method is weak; lots of important discoveries have begun with accidental single observations. But those productive single observations are always surprising. My table-clearing observation was slightly surprising…