Seth's Blog

One hundred years ago (January, 1912), at the annual meeting of the Geological Association in Frankfurt, Germany, Alfred Wegener, a meteorologist, presented his theory of continental drift for the first time. It was almost uniformly dismissed by geologists. One of them called it “mere geopoetry”. Much later, he was proved right.

To me, this is a classic example of the power of what I call insider/outsiders. Wegener had a great deal of scientific training, including a Ph.D. in astronomy. Unlike professional geologists, however, (a) he had the freedom to say whatever he wanted about geology without endangering his job (as a meteorologist) or prospects for advancement and (b) was under no pressure to publish. He could spend as much time on his theory as he wanted.

Peter A. Lawrence is a British biologist who has written several papers about problems with the way biology and other areas of science are now done. In this interview a year ago he summarizes his complaints:

• Scientific publication “has become a system of collecting counters for particular purposes – to get grants, to get tenure, etc. – rather than to communicate and illuminate findings to other people. The literature is, by and large, unreadable.” There is far too much counting of papers.
• “There’s a reward system for building up a large group, if you can, and it doesn’t really matter how many of your group fail, as long as one or two succeed. You can build your career on their success.” If you do something on your own it is viewed with suspicion.
• There is too much emphasis on counting citations. “If you work in a big crowded field, you’ll get many more citations. . . . This is independent of the quality of the work or whether you’ve contributed anything. [There is] enormous pressure on the journals to accept papers that will be cited a lot. And this is also having a corrupting effect. Journals will tend to take papers in medically-related disciplines, for example, that mention or relate to common genetic diseases. Journals from, say, the Cell group, will favor such papers when they’re submitted.”
• Grant writing takes too much time — e.g., 30-40% of your time. “There is an enormous increase in bureaucracy – form
  filling, targeting, assessment, evaluations. This has gone right through society, like the Black Death!”
• “Science is not like some kind of an army, with a large number of people who make the main steps forward together. You need to have individually creative people who are making breakthroughs – who make things different. But how do you find those people? I don’t think you want to have a situation in which only those who are competitive and tough can
  get to the top, and those who are reflective and retiring would be cast aside.” I’ve said something similar: Science is like single ants wandering around looking for food, not like a trail of ants to and from a food source. The trail of ants is engineering.

I agree. I would add that I think modern biology is far too invested in the idea that genes cause disease and that studying genes will help reduce human suffering. I think the historical record (the last 30 years) shows that this is not a promising line of work — but modern biologists cannot switch course.

What explains the depressing facts Lawrence points out? I think it is something deep and impossible to change: Science and job don’t mix well. The demands of any job and the demands of science are not very compatible. Jobs are about repetition. Science is the opposite. Jobs demand regular output. Science is unpredictable. However, jobs and science  overlap in terms of training: Both benefit from specialized knowledge. They also overlap in terms of resources: More resources (e.g., better tools) will usually help you do your job better, likewise with science. So we have two groups (insiders — professional scientists — and outsiders — everyone else). Both groups have big advantages and big disadvantages relative to the other. In the last 50 years, the insiders have been “winning” in the sense of doing better work. Their advantages of training and resources far outweighed the problems caused by the need for repetition and predictability. But now — as I try to show on this blog — outsiders are catching up and going ahead because the necessary training and tools have become much more widely available (e.g., tools have become much cheaper). And, as Lawrence emphasizes, professional science has gotten worse.

Epidemiology has lots of critics. In this article, for example, it is called “lying on a grand scale.” Every critique I have read has ignored history. Epidemiologists have been right about two major issues: 1. Heavy smoking causes lung cancer. 2. Folate deficiency causes birth defects. In both cases, the first evidence was epidemiological. Another example is John Snow’s conclusion about the value of clean water. In my experience, epidemiologists often overstate the strength of their evidence (as do most of us) but overstatement is quite different from having nothing worth saying.

Let’s look at an example. Many people think osteoporosis is due to lack of calcium. Bones are made of calcium, right? The epidemiology of hip fractures is clear. In spite of the conventional idea, the rate of hip fracture has been highest in places where people eat a lot of calcium, such as Sweden, and lowest in places where they eat little, such as Hong Kong. (For example.) In other words, the epidemiology flatly contradicted the conventional idea. This was apparently ignored by nutrition experts (everyone knows correlation does not equal causation) who advised millions of people, especially women, to take calcium supplements  to avoid osteoporosis. Millions of people followed (and follow) that advice.

Thanks to a recent meta-analysis we now know that experiments and better data firmly support the earlier epidemiology, which suggested that calcium supplements are dangerous. Here are its main conclusions:

In meta-analyses of placebo controlled trials of calcium or calcium and vitamin D, complete trial-level data were available for 28,072 participants from eight trials of calcium supplements and the WHI CaD participants not taking personal calcium supplements. . . .Calcium or calcium and vitamin D increased the risk of myocardial infarction (relative risk 1.24 (1.07 to 1.45), P = 0.004) and the composite of myocardial infarction or stroke (1.15 (1.03 to 1.27), P = 0.009). . . . A reassessment of the role of calcium supplements in osteoporosis management is warranted.

If the epidemiology had been taken more seriously, many heart attacks might have been avoided.

Is this an “anecdote” — a single example — proving nothing? Here’s how you can check. Randomly select a meta-analysis of epidemiological studies. Thousands have been done. Then ask if the results summarized in the meta-analysis appear random. Better yet, randomly pick two meta-analyses. Suppose the first summarizes 5 studies and the second summarizes 6. If the 11 results were shuffled together, how well could you assign them correctly?

At the Quantified Self blog, in response to a video of me talking about QS and the Ancestral Health Symposium (paleo), someone named Colin made the following comment:

Very interesting talk. I am just curious how someone can claim a study conducted with a sample size of one is “100 times better” than someone else’s study. I do not know anything about the other study mentioned, but I do know that a study based on n=1 cannot be considered scientific proof. And sure, he hears from people who have lost weight drinking the sugar water he prescribed, but it is quite possible there are 100 times as many people who didn’t email him because they didn’t see any positive results and decided to try something else. I think the QS stuff is very interesting and helpful on a personal level, but it seems like a stretch to generalize your results to others.

I responded:

I have two responses.

1. Sample size isn’t everything. Sure, a study with n=1 isn’t “scientific proof”. Nor is any other study, in my experience. “Scientific proof” has always required many studies. New scientific ideas have very often started with n = 1 experiments or observations. Later, larger experiments or observations were done. Both — the initial n=1 observation and the later n = many observations — were necessary for the new idea to be discovered and confirmed.

2. The history of biology teaches there are few exceptions to general rules. See any biology textbook. For example, a textbook might say “lymphocytes fight germs”. This means no serious exceptions have ever been found to that rule. So, as matter of biological history, the person who managed to figure out what one particular lymphocyte does turned out to have figured out what they all do. Biology textbooks have thousands of statements like “lymphocytes fight infection” meaning that this sequence of events (you can generalize from one to all, or nearly all) has happened thousands of times. There is no shadow hidden history of biology that teaches otherwise.

In the latest issue of American Scientist, Andrew Gelman (an old friend) and Kaiser Fung criticize Freakonomics and Superfreakonomics by Steve Levitt and Stephen Dubner (who wrote about my work). Although the article is titled “Freakonomics: What Went Wrong?” none of the supposed errors are in Freakonomics. You can get an idea of the conclusions from the title and this sentence: “How could an experienced journalist and a widely respected researcher slip up in so many ways?”

Gelman and Fung examine a series (“so many ways”) of what they consider mistakes. I will comment on each of them.

1. The case of the missing girls. I agree with Gelman and Fung: Levitt and Dubner accepted Emily Oster’s research too uncritically.

2. The risk of driving a car. I think Gelman and Fung miss the point. Yes, the claim (driving drunk is safer than walking drunk) was not well-supported by the evidence provided because the comparison was so confounded. However, I read the whole example differently. I didn’t think that Levitt and Dubner thought drunk people should drive. I thought their point was more subtle — that comparisons are difficult (“look how we can reach a crazy conclusion”).

3. Stars are made not born. I think Gelman and Fung fail to see the big picture. The birth-month effect in professional sports, which Gelman and Fung dismiss as “very small,” is of great interest to many people, if not to Gelman and Fung.  It suggests what Levitt and Dubner and Gladwell and others say: Early success matters. That’s not obvious at all. There are lots of similar associations in epidemiology. They have been the first evidence for many important conclusions, such as smoking causes lung cancer. Are professional sports important? Maybe. But epidemiology and epidemiological methods are surely important. By learning about this effect, we learn about them. Lots of smart people fail to take epidemiology seriously enough (e.g., “correlation does not equal causation”).

4. Making the majors and hitting a curve ball. Gelman and Fung point out that one sentence is misleading. One sentence. This is called praising with faint damn.

5. Predicting terrorists. Gelman and Fung say that the terrorist prediction algorithm of a man named Ian Horsley, which Levitt and Dubner seem to take seriously, is not practical. But their review fails to convince me it was presented as practical. Since there are no data about how well the algorithm works, and Levitt and Dubner are all about data….

6. The climate change dust-up. I agree with Gelman and Fung that Nathan Myrvold’s geoengineering ideas are unimportant. (My view of Myrvold’s patent trolling.)  But in this case, I’d say both sides — Gelman and Fung and Levitt and Dubner — miss what’s really important, namely that the usual claims that humans are dangerously warming the planet are held far too strongly. The advocates of this view are far too sure of themselves. I have blogged about this many times. In a nutshell, the climate models that we are supposed to trust have never been shown to persuasively predict the climate ten or twenty years from now (or even one year from now). There is no good reason to believe them. That Levitt and Dubner seem to take that stuff seriously is the only big criticism I have of their work . At least in that geoengineering stuff Levitt and Dubner were dissenting from conventional wisdom. Gelman and Fung do not. They fail to realize that something we’ve been told thousands of times is nonsense (in the sense of being wildly overstated). It was Levitt and Dubner’s comments about this that led me to look closely at all that climate-change scare stuff. I was surprised how poor the evidence was.

The biggest problem with Gelman and Fung’s critique is that they say nothing about the great contribution of Steve Levitt to economics. They fail to grasp that he has made economics considerably more of a science, if by science you mean a data-driven enterprise as opposed to an ideologically-driven or prestige-driven one (mathematics is prestigious, the more difficult, the more prestigious). He did so by pioneering a new way to use data to learn interesting things. His method is essentially epidemiological, except his methods are considerably better (better matching, less formulaic) and his topics much more diverse (e.g., sumo wrestling) than mainstream epidemiology. A large fraction of prestige economics is math, divorced from empirical tests. This stuff wins Nobel Prizes, but, in my and many other people’s opinion, contributes very little to understanding. (Psychology has had the same too much math, too little data problem — minus the Nobel Prizes, of course.) To persuade a big chunk of an entire discipline to pay more attention to data is a huge accomplishment.

Levitt’s methodological innovation makes Freakonomics far from what Gelman and Fung call “pop statistics”. It is actually an amusing and well-written record of something close to a revolution. In the 1980s, a friend of mine at UC Berkeley took an introductory economics class. She told me a little of what the teacher said in class. All theory. What about data? I said. It’s a strange science that doesn’t care about data. My friend went to office hours. She asked the instructor (a Berkeley economics professor): What about data? Don’t worry about data, he replied. Gelman and Fung fail to appreciate what economics used to be like. The ratio of strongly-asserted ideas to persuasive data used to be very large. Now it is less.

Thanks to Ashish Mukharji.