Seth's Blog

The very first comment on my preventive stupidity post said this:

“Absence of evidence is not evidence of absence.” This was a very useful concept to point out to those people who believed that because nominal national US housing prices had never dropped before that they wouldn’t drop in the future.

At 39:59 in this excellent podcast, Barry Ritholtz says, “It’s just not true [that US housing prices have never declined]. Obviously in the post-Depression era home prices really collapsed.” The saying absence of evidence is not evidence of absence kept those who said it from looking into the facts of the matter.

I learned something from the comments on my preventive stupidity post (also here). The best comment was from Kim Oyhus, whose earlier comment had started it all. Were I to discuss the subject from scratch, here’s what I’d add (mostly elaborating what Kim said):

Scientific discussions are usually about data and theory. From Data A, someone has “concluded” — more precisely, raised the plausibility of — Theory X. At this point, preventive stupidity often begins: Someone says “correlation doesn’t equal causation” or “the plural of anecdote is not data” or something similar.

Here’s what I think. More data are better. Two data sets are better than one. To go from one data set (A) to two (A and B) is a step forward. Less data are worse. To go from one data set (A) to none is a step backward. If you respond to the assertion that A supports X by mentioning more data that bears on the truth of X, that’s a step forward. The more convincing the new data (in either direction, pro or con), the bigger the improvement.

Likewise, more ideas are better. Two plausible explanations of A are better than one. To go from one idea (X) to two (X and Y) is a step forward. Fewer ideas are worse. To go from one explanation (X) to none is a step backward. If you respond to the assertion that A supports X by mentioning another plausible explanation of A, that’s a step forward. The more plausible the new explanation, the bigger the step forward.

The sayings I wrote about (e.g., “absence of evidence doesn’t equal evidence of absence”) make their users stupider because they push them from thinking about one data set to thinking about none (they dismiss Data A) or from considering one idea to considering none (they dismiss Theory X). They make the rest of us stupider because they prevent their users from making useful contributions. They really are preventive stupidity, as Orwell said.

If these sayings were used as transitions, as throat-clearing, fine. If somebody wrote, “Look, correlation doesn’t equal causation. Here’s another plausible explanation for what you observed . . . ” that would be okay. In my experience, that’s not what happens. They’re used to support an overall dismissiveness. Several months ago I wrote about my observations that connected socks with foot fungus. Some of the comments provided new relevant data — steps forward. A few comments, however, made this or that preventive-stupidity point (“Sample size of 1, no control . . . . you can’t seriously think you’ve proved anything here“, “your post is post hoc ergo propter hoc reasoning“). The comments didn’t go on to make a step forward. They were steps backward.

• Jane’s [Jacobs] Walks
• 9 years of sleep data
• overview of Jane Jacobs’s work
• God is in every leaf of every tree. A great summing up of what science is about — taking seriously little things (such as leaves) and drawing inferences from the very small (“leaf”) to the very large (“God”). Someday I am going to use this saying . . . I would have used it already, had I known of it.

Thanks to Anne Weiss, Tom George, and JR Minkel

Professional scientists mostly ignore the slogans (e.g., “absence of evidence isn’t evidence of absence”) I discussed in my previous post. For example, the professional-scientific conclusion that smoking causes lung cancer came mostly from correlations. This conclusion was criticized, sure, but not by saying “correlation does not equal causation”.

Professional scientists have a much worse problem, which is that they criticize much more easily and fluently than they praise. (Marginal Revolution is an excellent blog partly because it doesn’t suffer from this.) This can be depressing (lots of work is underappreciated), exciting (anyone who sees this has a big advantage), or merely amusing, as in this example to which Stephen Marsh drew my attention:

I just returned from the MS4 conference. It is the fourth year that a group of philosophers of science have gathered to try to tease apart the implications of computer simulation in science. . . .Several presentations gave harsh criticism of climate science models. Bayesian tools (a statistical technique) were given some especially harsh criticisms. Everyone agreed the models were problematic in some sense or another. That the results were subject to all kinds of errors and suspicions, and there were substantially difficult difficulties to sort out. . . . Despite this, everyone concurs the models are robust . . . No one disagreed that the planet was warming.

The poor ability of professional scientists to praise means that comparison of A and B (two theories, say, or two experiments) mainly consists of comparing how much A and B have been criticized. How much A and B would have been praised, had scientists been better at praise, is unknown. This is a very poor way to compare stuff. Inability to praise also means that there is too much criticism. In my experience, scientists have trouble separating serious criticisms from trivial ones. For example, that climate-change models haven’t been shown to predict correctly is a serious criticism not emphasized enough (e.g., at the MS4 conference).

In the world of Orwell’s 1984,

To the end of suppressing any unorthodoxy, the [ruling] Party inculcates self-deceptive habits of mind to the inner and outer members, thus crimestop (“preventive stupidity”) halts thinking at the threshold of politically-dangerous thought.

Three sayings popular in scientific discussions show that in our world, preventive stupidity exists — and works. In a comment, Kim Ayhus has brought my attention to this.

1. Absence of evidence is not evidence of absence. Ayhus explains why this is wrong. That such an Orwellian saying is popular in discussions of data suggests there are many ways we push away inconvenient data.

2. Correlation does not equal causation. In practice, this is used to mean that correlation is not evidence for causation. At UC Berkeley, a job candidate for a faculty position in psychology said this to me. I said, “Isn’t zero correlation evidence against causation?” She looked puzzled.

3. The plural of anecdote is not data. How dare you try to learn from stories you are told or what you yourself observe!

Orwell was right. People use these sayings — especially #1 and #3 — to push away data that contradicts this or that approved view of the world. Without any data at all, the world would be simpler: We would simply believe what authorities tell us. Data complicates things. These sayings help those who say them ignore data, thus restoring comforting certainty.

Maybe there should be a term (antiscientific method?) to describe the many ways people push away data. Or maybe preventive stupidity will do.