Seth's Blog

Recently the Guardian ran an article by David Colquhoun, a professor of pharmacology at University College London, complaining about peer review. His complaints were innocuous; what was interesting was his example. How bad is peer review? he said. Look what gets published! He pointed to a study of the efficacy of acupuncture and included graphs of the results. “It’s obvious at a glance that acupuncture has at best a tiny and erratic effect on any of the outcomes that were measured,” he wrote.

Except it wasn’t. There were four graphs. Each had two lines — one labelled “acupuncture,” the other labelled “control”. You might think to assess the effect of acupuncture you compare the two lines. That wasn’t true. The labels were misleading. The “acupuncture” group got acupuncture early in the experiment; the “control” group got acupuncture late in the experiment. Better names would have been early treatment and late treatment. You could not allow for this “at a glance”. It was too complicated. With this design, if acupuncture were effective the difference between the two lines should be “erratic”.

The paper’s data analysis is poor. To judge the efficacy of acupuncture, their main comparison used only the data from the first 26 weeks. They could have used data from all 52 weeks. That is, they ignored half of their data when trying to answer their main question. Colquhoun could have criticized that, but he didn’t.

Colquhoun’s criticism was so harsh and shallow, apparently he is biased against acupuncture. But there are two big things few pharmacology professors appear to know. One is how to stimulate the immune system. This should be central in pharmacology, but it isn’t. Half of why I think fermented foods are so important is that I think they stimulate the immune system. (The other half is they improve digestion.) There are plenty of less common ways to do this. The phenomenon of hormesis suggests that small doses of all sorts of poisons, including radiation, stimulate repair systems. The evidence behind the hygiene hypothesis suggests that dirt improves the immune systems of children. Bee stings have been used to treat arthritis. And so on. In this context, sticking needles into someone, which puts a small amount of bacteria into their blood, is not absurd. Acupuncture also allowed patients to share their symptoms, the value of which Jon Cousins has emphasized.

The other big thing Colquhoun doesn’t seem to know is the absurdity of the chemical imbalance theory of depression. Speaking of ridiculous, that’s ridiculous. Which plays a larger role in modern medicine — antidepressants or acupuncture? If you want criticize peer review, criticize the chemical imbalance theory. It is as if peer reviewers have been saying, yes, the earth really is flat for fifty years. Perhaps this is ending. During a talk that Robert Whitaker gave at the Massachusetts General Hospital in January, he was told by doctors there that the chemical-imbalance theory was an “outdated model”.

Thanks to Dave Lull and Gary Wolf.

With the ability to measure individual genes has come interest in learning what they do. Perhaps Person X is depressed and Person Y is not depressed because Person X’s genes differ from Person Y’s. A whole generation of psychiatry researchers now believes this is plausible. There are “general reasons to expect that GxEs [gene by environment interactions] are common,” says a new review paper in the American Journal of Psychiatry. By “common” they mean large enough and common enough to do research about. (more…)

Yesterday I wrote how Alexandra Carmichael’s headache story illustrated a large and awful truth about modern healthcare: It happily provides expensive relief of symptoms while ignoring investigation of underlying causes. If we understood underlying causes (e.g., causes of migraines), prevention would be easy. Let people get sick so that we can make money from them. There should be a name for this scam. In law enforcement, it’s called entrapment.

Sensible prevention research would start small. Not by trying to prevent breast cancer, or heart disease, or something like that: They take many years to develop and therefore are hard to study. Sensible prevention research would focus on things that are easy to measure and happen soon after their causative agents. One example is migraines. Migraines happen hours after exposure. The fact that Chemical X causes migraines means it is likely that Chemical X is bad for us, even if it doesn’t cause migraines in everyone. This is the canary-in-a-coal-mine idea. Migraines are the canary.

Acne is another canary. Acne is easy to measure. Figuring out how to prevent it would be a good way to begin prevention research. To prevent acne would be to take the first steps toward preventing many more diseases. A high-school student could do ground-breaking research — research that would improve the lives of hundreds of millions of people — about how to prevent acne but somehow this never happens. In spite of this possibility, grand-prize-winning high-school science projects, from the most brilliant students in the whole country, are always about trivia.

A just-published review in The Lancet reveals once again the unfortunate perspective of medical school professors. The abstract ends with this:

New research is needed into the therapeutic comparative effectiveness and safety of the many products available, and to better understand the natural history, subtypes, and triggers of acne.

Actually, finding out what causes acne is all that’s needed.

To figure out what causes acne (and thereby how to prevent it) three things are necessary: (a) study of environmental causes, such as diet, (b) starting with n=1, and (c) willingness to test many ideas that might be wrong (because it’s far from obvious how to prevent acne). All three of these things are exactly what the current healthcare research system opposes. It opposes prevention research because drug companies don’t fund it. It opposes n=1 studies because they are small and cheap, which is low-status. To do such a study would be like driving a Corolla. It opposes studies that could take indefinitely long because such studies are bad for a researcher’s career. Researchers need a steady stream of publications.

High school students, who aren’t worried about status or number of publications, could make a real contribution here. You don’t need fancy equipment to measure acne.

Thanks to Michael Constans.

Last night I slept unusually well, waking up more rested and with more energy than usual.  I slept longer than usual: 7.0 hours versus my usual 5.1 hours (median of the previous 20 days).  My rating of how rested I felt was 99.2% (that is, 99.2% of fully rested); the median of the previous 20 days is 98.8%. Because the maximum is 100%, this is really a comparison of 0.8% (this morning) with 1.2% (previous mornings); and the comparison is not adjusted for the number of times I stood on one leg to exhaustion, which improves this rating. During the previous 20 days I often stood on one leg to exhaustion six times; yesterday I only did it four times. Above all, I felt more energy in the morning. This was obvious. I have just started to measure this.  At 8 am and 9 am, I rate my energy on a 0-100 scale where 50 = neither sluggish nor energetic/energized, 60 = slightly energetic/energized, 70 = somewhat energetic/energized, and 75 = energetic/energized. My ratings this morning were 73 (8 am) and 74 (9 am). The median of my 9 previous ratings is 62. The energy improvement (73/74 vs 62) is why I am curious. I would like to feel this way every morning.

What caused it? I had not exercised the previous day. My room was no darker than usual. My flaxseed oil intake was no different than usual. I had not eaten more pork fat than usual. However, four things had been different than usual:

1. 2 tablespoons of butter at lunch. In addition to my usual 4 tablespoons per day.

2. 0.5-1 tablespoons of butter at bedtime. Again, in addition the usual 4.

3. 1 tablespoon (15 g) coconut butter at bedtime. Part of a longer study of the effect of coconut butter. Gary Taubes suggested this. I had eaten 1 T coconut butter at bedtime 13 previous days. On the first of those 13 days, I had felt a lot more energetic than usual in the morning. On the remaining days, however, the improvement was less clear. I started measuring how energetic I felt in the morning to study this further. Last night was Friday night. On the previous two nights (Wednesday and Thursday) I had not eaten the coconut butter. Maybe absence of coconut butter followed by resumption of coconut butter is the cause.

4. Fresh air and ambient noise. Following a friend’s suggestion, I opened one of my bedroom windows.

My first question is whether the improvement is repeatable. If so, I will start to vary these four factors.

The most important thing I learned in graduate school — or ever — about research is: Better to do than to think. By do I mean collect data. It is better to do an experiment than to think about doing an experiment, in the sense that you will learn more from an hour spent doing (e.g., doing an experiment) than from an hour thinking about what to do. Because 99% of what goes on in university classrooms and homework assignments is much closer to thinking than doing, and because professors often say they teach “thinking” (“I teach my students how to think”) but never say they teach “doing”, you can see this goes against prevailing norms. I first came across this idea in an article by Paul Halmos about teaching mathematics. Halmos put it like this: “The best way to learn is to do.” When I put it into practice, it was soon clear he was right. (more…)

In a paper (and blog post), Andrew Gelman writes:

As a statistician, I was trained to think of randomized experimentation as representing the gold standard of knowledge in the social sciences, and, despite having seen occasional arguments to the contrary, I still hold that view, expressed pithily by Box, Hunter, and Hunter (1978) that “To find out what happens when you change something, it is necessary to change it.”

Box, Hunter, and Hunter (1978) (a book called Statistics for Experimenters) is well-regarded by statisticians. Perhaps Box, Hunter, and Hunter, and Andrew, were/are unfamiliar with another quote (modified from Beveridge): “Everyone believes an experiment except the experimenter; no one believes a theory except the theorist.” (more…)

For several years I have been doing simple daily tests to measure my brain function. I got the idea when I noticed that a few capsules of flaxseed oil improved my balance. Flaxseed oil also improved other measures of brain function, such as digit span. I wasn’t surprised I could do better; what was surprising was how easy it was. It revealed a big gap in our understanding of nutrition. I do the daily tests not only to improve brain function but also to improve the rest of my body. I think the brain is like a canary in a coal mine — especially sensitive to bad environments. Learning what environment was best for the brain would suggest what environment is best for the rest of the body. When I started taking an optimal amount of flaxseed oil, my gums turned from red (inflamed) to pink (not inflamed), supporting this assumption.

I tried six or seven mental tests and eventually settled on a test of arithmetic (how fast I could do simple problems such as 5-3). I hoped that now and then my score would change (in either direction, faster or slower) and that these changes would point to new things that control brain function. No one had/has done such a thing. I had no idea if unexpected changes would show up or, if they did, how often. I didn’t know what the score changes would look like (their size and shape) nor, of course, what would cause them. Would all of them involve diet? Would all of them make sense in terms of what we already know? (Flaxseed oil makes sense because the brain contains lots of omega-3.)

The first two surprises were these: 1. My score suddenly improved a few days after switching from Chinese flaxseed oil to American flaxseed oil. This made sense: It is easy to destroy omega-3 if flaxseed oil is kept at room temperature. 2. My score suddenly improved when I switched from pig fat to butter. This was counter-intuitive: pig fat is more paleo than butter.

Last fall, there was another surprise: My score greatly improved since the summer. I was much faster than ever before. At first I thought the improvement was due to moving to Beijing. I had moved from Berkeley to Beijing in early September.  My Beijing life differed in a thousand ways from my Berkeley life. I had three ideas about which differences might matter. 1. Walnuts. Perhaps I ate more walnuts in Beijing. Walnuts are supposed to be good for brain function. 2. Heat. It was much hotter in Beijing than Berkeley. Maybe that improved brain function. 3. Vitamins. I took less vitamin supplements in Beijing. Maybe they harmed brain function.

I tested these possibilities. 1. I stopped eating walnuts. My arithmetic score did not clearly change. 2. Winter came, it got much colder. The improvement did not go away. 3. I took the same amount of vitamins I’d taken in Berkeley. My arithmetic score didn’t change. So all of these ideas were wrong.

Because they were wrong, I considered a fourth possibility: The improvement was due to removal of two mercury amalgam fillings on July 28, 2010. They were replaced with non-amalgam fillings. I’d had them removed for precautionary reasons. I wasn’t suffering from any signs of mercury poisoning. Hair tests had repeatedly shown mildly high amounts of mercury in my hair (75th percentile of an unspecified sample). Measurements of the mercury in my breath had come out higher than usual but it was hard to be sure the machine was working correctly.

I looked again at my data. It showed something I hadn’t noticed: the improvement started before I went to Beijing. It started very close to July 28. That was good evidence that the mercury explanation was correct. Now the evidence is even stronger. I’ve returned to Berkeley and thereby made my life quite similar to the situation when my scores were much higher than now. The improvement has remained.

The evidence for causality — removal of mercury amalgam fillings improved my arithmetic score — rests on three things: 1. Four other explanations made incorrect predictions. 2. The improvement, which lasted months, started within a few days of the removal. Long-term improvements (not due to practice) are rare — this is the only one I’ve noticed. 3. Mercury is known to harm neural function (“mad as a hatter”). As far as I’m concerned, that’s plenty.

A long Wikipedia article describes evidence on both sides of the question of whether mercury amalgam fillings cause damage. In 2009, the American Dental Association stated in a press release “the overwhelming weight of scientific evidence supports the safety and efficacy of dental [mercury-containing] amalgam.”  As recently as 1991, Consumer Reports told readers “if a dentist wants to remove your fillings because they contain mercury, watch your wallet.” (Dental insurance will pay most of the cost of removing my remaining amalgam fillings.) In an essay last revised in 2006, Stephen “Quackwatch” Barrett explained at length why mercury toxicity is a “scam”. According to Barrett, “there is overwhelming evidence that amalgam fillings are safe.”

Ask your dentist some pointed questions.

Martin Wolf relays what passes for wisdom:

Albert Einstein is reported to have said that insanity consists of doing the same thing over and over again and expecting different results.

Which, if true, shows that Einstein was a theorist.

Call me insane. Based on many years of data collection, I believe scientific progress has a power-law distribution. You sample from this distribution when you collect data. You collect data again and again — “doing the same thing over and over again”. Almost all the data you collect produces little progress; a tiny fraction produces great progress. The secret to scientific progress is doing the same thing over and over — and being wise enough to grasp that the results will vary greatly. (Nassim Taleb understands this.) In the short term, it seems like you are getting nowhere.

I learned this lesson from my sleep research. For ten years I tried various solutions to my problem of early awakening. Nothing worked. All my ideas were wrong. Eventually I got “lucky” but actually I made my own luck by persisting so long.

Once you realize the distribution of progress, you grasp that the secret of success is making the cost per sample as low as possible. Few scientists, in my experience, have figured this out. They prefer expensive experiments because larger grants signal higher status. Won’t fancy equipment tell me more? they rationalize. Grant givers, also failing to understand the basic point, are happy to oblige the status-seekers: Much easier to administer one $200,000 grant than 10 $20,000 grants. And progress slows to a crawl.

More Rita Mae Brown is a more likely source of this saying than Albert Einstein.

A new study in JAMA found higher salt consumption strongly associated with less death from heart disease. The association with total mortality (more salt, less death) was almost significant. To grasp the strength of the evidence, see this. Yes, it’s a correlation, but I don’t know of any examples of such a strong correlation reversing (so that more salt is now correlated with more death) when now-unknown confounders are taken into account. In 1998, Gary Taubes argued that the benefits of salt reduction were greatly overstated. The new study did find more salt correlated with higher systolic blood pressure but in the big picture (mortality) that didn’t matter. If all those warnings about salt had any effect, the new study suggests their effect was negative.

Perhaps people who eat less salt are more credulous (they believed the experts) — and this damages them in other ways? Perhaps they rely on doctors more, for example. It is hard to interpret this finding in a way that makes mainstream health care look good. A New York Times article about the study points out that “the new study is not the only one to find adverse effects of low-sodium diets.” And it reports what someone at the Centers for Disease Control said:

Dr. Peter Briss, a medical director at the centers, said that the study was small; that its subjects were relatively young, with an average age of 40 at the start; and that with few cardiovascular events, it was hard to draw conclusions.

Dr. Briss fails to understand statistics. Ordinary statistical calculations take sample size and number of events into consideration when indicating the strength of the evidence. That’s the one of the main purposes of those calculations. As for “relatively young,” I know of nothing to suggest that the effects of sodium reverse with age — so it is irrelevant that the subjects were relatively young. That someone at the CDC is so clueless is remarkable.

Conway’s Law is the observation that the structure of a product will reflect the structure of the organization that designed it. If the organization has three parts, so will the product. In the original paper (1968), Conway put it like this:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Here is an example:

A contract research organization had eight people who were to produce a COBOL and an ALGOL compiler. After some initial estimates of difficulty and time, five people were assigned to the COBOL job and three to the ALGOL job. The resulting COBOL compiler ran in five phases, the ALG0L compiler ran in three.

A consumer — someone outside the organization who uses the product — wants the best design. Conway’s Law implies they are unlikely to get it.

I generalize Conway’s Law like this: It is hard for people with jobs to innovate — for reasons that outsiders know nothing about. Whereas persons without jobs have total freedom. An example is a politician who promises change but fails to deliver. The promises of change are plausible to outsiders (voters) so they elect the politician. However, being outsiders, they barely understand how government works. When the promised changes don’t happen, the voters are “disillusioned”.

To me, the most interesting application of the generalized law is to science. In my experience, people who complain about “bad science”, such as John Ioannides and Ben Goldacre, have the same incomplete view of the world as the “disillusioned” voters. They fail to grasp the constraints involved. They fail to consider that the science they are criticizing may be the best those professional scientists can produce, given the system within which they work. Better critiques would look at the constraints the professional scientists are under, the reasons for those constraints, and how those constraints might be overcome.

“Much research is conducted for reasons other than the pursuit of truth,” writes Ioannidis. Well, yes — people with jobs want to keep them and get promoted. They want to appear high status. That’s not going to change. It’s absolutely true that drug company scientists slant the evidence to favor their company’s drug, as Irving Kirsch explains in The Emperor’s New Drugs. But if you don’t understand what causes depression and you’re trying to produce a new anti-depressant and you want to keep your job . . . things get difficult. The core problem is lack of understanding. Lack of understanding makes innovation difficult. Completely failing to understand this, Ioannidis recommends something that would discourage new ideas: “We must routinely demand robust and extensive external validation—in the form of additional studies—for any report that claims to have found something new.”

Truly “bad science” has little to do with what Ioannides or Goldacre or any quackbuster talks about. Truly bad science is derivative science, science that fails to find new answers to major questions, such as the cause of obesity. Failure of innovation isn’t shown by any one study. Given the rarity of innovation, it is unwise to expect much of any one study. To see lack of innovation clearly you need to look at the whole distribution of innovation. Whether the system is working well or poorly, I think the distribution of innovation resembles a power law: most studies produce little progress, a tiny number produce large progress. The slope of the distribution is what matters. Bad science = steep downward slope. With bad science, even the most fruitful studies produce only small amounts of innovation.

Just as outsiders expect too much from professionals, they fail to grasp the innovative power of non-professionals. Mendel was not a professional scientist. Darwin was not a professional scientist. Einstein did his best work while a patent clerk. John Snow, the first person to use data (a graph) to learn the cause of an infection, was a doctor. His job had nothing to do with preventing infection. To improve innovation about health (or anything else), we should give more power to non-professionals, as I argued in my talk at the First Quantified Self Conference.

Thanks to Robin Barooah.