In a recent post I said that scientists are often much too dismissive. They are “evidence snobs,” Alex Tabarrok might say. A letter in the current issue of the American Journal of Clinical Nutrition criticizes a important example of just such dismissiveness:
In conclusion, whereas we agree that policy decisions should be evidence-based and not hasty, we do not agree that the evidence base [used to make those decisions] should be constrained to one type of study [long-term randomized controlled trials]â€”in particular, not to a study design that is inherently limited. Do we really want to wait perhaps decades for results of long-term RCTs, which almost certainly will not provide definitive evidence, while ignoring other relevant evidence involving shorter-term endpoints? An example is provided in the panel’s own summary statement (2). In lauding RCTs as the “gold standard for evidence-based decision making,” the panel proudly points to the fact that, even though folate was well known to decrease the risk of neural tube defects in animal studies, policy recommendations for folate supplementation to prevent neural tube defects were delayed while authorities waited some years for confirmation from RCTs. One can only wonder how many infants were born with neural tube defects while authorities waited.
“Proudly,” huh? Inclusion of that word shows how pissed the authors of the letter are — and rightly so. One author is Bruce Ames, a neighbor of mine, for whom I have great respect; another is Walter Willett, the Harvard epidemiologist. In 1998, Willett wrote a smart article challenging the popular belief that a low-fat diet is a good way to lose weight.
Here is part of the reply from the authors of the report that Ames et al. criticized:
It is important to note that our panel was not charged with asking whether vitamins and minerals play a role in human disease –a topic that occupies much of the letter by Ames et al, and for which observational evidence is indeed central — but, as a State-of-the Science Panel, was charged to reflect on the state of the available evidence for a treatment recommendation on the use of vitamins and minerals in the general population. For treatment decisions, the RCT is the established standard. No better proof of this principle can be found than in the RCTs reviewed in our report, which showed serious harm from vitamin ingestion in certain circumstances.
A less-than-reassuring answer. A commentator on my earlier post thought I should address the strongest arguments on the other side. I had trouble thinking of any. It’s hard to argue that less evidence is better. You can see that those who wrote this paragraph — some of the most prominent nutrition scientists in the country — were equally baffled.
I will revise my “common mistakes” article to mention the Ames et al. letter.