Seth's Blog

The Nobel Prizes awarded each year resemble a kind of report card where each prize-worthy discipline (Physics, Chemistry, etc.) gets a grade that depends on the prize-winning research. If the prize-winning research is useful and surprising, the grade is high. If not the grade is low. More generally, at least to me, the intellectual history of the prize winners sheds light on the whole profession. Perhaps some biologists were unaware of the behavior of Eric Kandel described in Explorers of the Black Box when he was awarded the biology prize. Kandel, I hasten to add, is an unusual case.

Thomas Sargent is one of the winners of this year’s Economics prize. In 2007, he gave a graduation speech at Berkeley to economics majors (via Marginal Revolution). In the speech, Sargent called economics “organized common sense”. He went on to list 12 common-sense ideas, such as “Individuals and communities face trade-offs” and “governments and voters respond to incentives” that economists believe. The reasons for their belief weren’t stated.

When I started as a professor (at Berkeley) I did many experiments with rats and, to my annoyance, discovered an inconvenient truth: I understood rats less well than I thought. Even in a heavily-controlled heavily-studied situation (Skinner box), my rats often did not do what I expected. My common sense was often wrong, in other words. This experience made me considerably more skeptical of other people’s “common sense”.

To me, and I think to most scientists, science begins with common sense. Experimental psychology certainly does. I used common sense to design my experiments. Had I not done those experiments, I would not have learned that my common sense was wrong. So relying on common sense was helpful — as a place to start. As a way to begin to understand. You begin with common-sense ideas and you test them. That common sense is often wrong is a theme of Freakonomics, in agreement with my experience. Yet Sargent seemed content (he called economics “our beautiful subject”) to end with common sense, perhaps tidied up.

This is really unfortunate because economics, beautiful or not, is so important. If you ignore data, the answer to every hard question is the same: the most powerful people are right. That way lies stagnation (problems build up unsolved because powerful people prefer the status quo) and collapse (when the problems become overwhelming). Alan Greenspan’s faith-based belief in free markets and the 2008 financial crisis — after Sargent’s speech — is an example. In 2009, Sargent’s speech might have been less well-received.

A new series on Showtime called Homeland is about a CIA agent (played by Claire Danes) who believes that a newly-released American prisoner of war may have been “turned” during his years in Iraqi captivity. In the first episode, she tries to find evidence to support her belief. Judging by that episode, it is very good.

I told Edward Jay Epstein about it — his book on James Angleton centers on CIA infiltration by “moles”. He commented: (more…)

Edward Jay Epstein‘s newest Kindle book is James Jesus Angleton: Was He Right?. Angleton worked at the CIA most of his career, which spanned the Cold War. He struck some of his colleagues as paranoid: He believed that the CIA could easily contain Russian spies. Colleagues said Oh, no, that couldn’t happen. After his death, it turned out he was right (e.g., Aldrich Ames). At one point he warned the CIA director, “an intelligence [agency] is most vulnerable to deception when it considers itself invulnerable to deception.”

What interests me is the asymmetry of the mistakes. When it really matters, we overestimate far more than underestimate our understanding. CIA employees’ overestimation of their ability to detect deception is a big example. There are innumerable small examples. When people are asked to guess everyday facts (e.g., height of the Empire State Building) and provide 95% confidence intervals for their guesses, their intervals are too short, usually much too short (e.g., the correct answer is outside the intervals 20% of the time). People arrive at destinations more often later than expected than earlier than expected. Projects large and small take longer than expected far more often than shorter than expected. For any one example, there are many possible explanations. But the diversity of examples suggests the common thread is true: We are too sure of what we know.

There are several plausible explanations. One is that it helps groups work together. If people work together toward a single goal, they are more likely to reach that goal and at least learn what happens than if they squabble. Another is the same idea at an individual level. Overconfidence in our beliefs helps us act on them. By acting on them, we learn. Doing nothing teaches less. A third is a mismatch idea: We are overconfident because modern life is more complicated than the Stone-Age world to which evolution adjusted our brains. No one asked Stone-Age people How tall is the Empire State Building? A fourth is that we assume what physicists assume: the distant world follows the same rules as the world close to us. This is a natural assumption, but it’s wrong.

Early in Angleton’s career, he had a very unpleasant shock: He realized he had been fooled by the Russians in a big way for a long time. This led him to try to understand why he’d been fooled. Early in my scientific career, I too was shocked: Rats in Skinner boxes did not act as expected far more often than I would have thought. I overestimated my understanding of them. In a heavily-controlled heavily-studied situation! I generalized from this. If I couldn’t predict the behavior of rats in a Skinner box, I couldn’t predict human behavior in ordinary life. My conclusion was data is more precious than we think. In other words, data is underpriced. If a stock is underpriced, you buy as much of it as possible. I tried to collect as much data as possible. Personal science — studying my sleep, my weight, and so on — was a way to gather data at essentially zero cost. And, indeed, the results surprised me far more than I expected. I could act based on the overconfidence effect but I could not remove it from my expectations.

In Cities and the Wealth of Nations, Jane Jacobs tells how, in the 1920s, one of her aunts moved to an isolated North Carolina village to, among other things, have a church built. The aunt suggested to the villagers that the church be built out of the large stones in a nearby river. The villagers scoffed: Impossible. They had not just forgotten how to build with stone, they had forgotten it was possible.

A similar forgetting has taken place among influential Western intellectuals — the people whose words you read every day. (more…)

In a paper (and blog post), Andrew Gelman writes:

As a statistician, I was trained to think of randomized experimentation as representing the gold standard of knowledge in the social sciences, and, despite having seen occasional arguments to the contrary, I still hold that view, expressed pithily by Box, Hunter, and Hunter (1978) that “To find out what happens when you change something, it is necessary to change it.”

Box, Hunter, and Hunter (1978) (a book called Statistics for Experimenters) is well-regarded by statisticians. Perhaps Box, Hunter, and Hunter, and Andrew, were/are unfamiliar with another quote (modified from Beveridge): “Everyone believes an experiment except the experimenter; no one believes a theory except the theorist.” (more…)