Seth's Blog

Drug companies, in a few cases, have recently started to release much more data from drug trials. Unsurprisingly, analysis of the new data by outsiders — people who have nothing to gain from positive results — has often contradicted the drug company analysis of the same data.

One example involves the flu drug Tamiflu. The new analysis suggested that “Tamiflu falls short of claims—not just that it ameliorates flu complications, but also that the drug reduces the transmission of influenza.” Another example involved Prozac. The new analysis “ended up bucking much of the published literature on antidepressants. . . . [It]found no link between Prozac and suicide risk among children and young adults . . . Prozac appeared to be more effective in youth, and antidepressants far less efficacious in the elderly, than previously thought.”

Another reason to believe in the value of this new data is the work of Lisa Bero at UCSF. She looked at the efficacy of nine drugs using unpublished FDA data. “Nineteen of the redone analyses showed a drug to be more efficacious, while 19 found a drug to be less efficacious. The one harm analysis that was reanalyzed showed more harm from the drug than had been reported.”

I hope that the FDA will eventually require that all raw data from drug trials be publicly available as a condition of approval. (The same should also be true of journal articles, as a condition of publication.) It is abundantly clear that drug company analyses are often misleading — which harms the public.

• The corruption of science by research grants. This reminds me of a BBC documentary called something like Science Under Attack. It was hosted by a Nobel Prize winner (Biology) named Paul Nurse. Part of it was about “climate change denialism”.  If you don’t believe that humans are dangerously warming the planet, Nurse implied, you are somehow attacking science. When people who win Nobel Prizes cannot see that AGW is a crock, something curious has happened.
• Edward Jay Epstein interviews DSK. “”Thank you so much for your interest in this case,” he says.”
• Researcher discovers new treatment for her own vertigo. ” A University of Colorado School of Medicine researcher who suffers from benign paroxysmal positional vertigo (BPPV) and had to “fix it” before she could go to work one day was using a maneuver to treat herself [the usual treatment] that only made her sicker. “So I sat down and thought about it and figured out an alternate way to do it. Then I fixed myself and went in to work” and [thereby] discovered a new treatment for this type of vertigo.”

Thanks to Melissa Francis.

• ALS patients test home-brewed drug themselves
• Vitamin K in American foods. I was surprised there is a lot more Vitamin K in chicken than other meats.
• Value of early exposure to microbes.
• 40+ years of mislabelled cell lines in cancer research. “Nearly 40 years later, Dr. Masters, in a study of scientific papers published between 2000 and 2004, found nearly 1,000 citations of the same contaminated cancer lines revealed in Dr. Gartler’s 1966 findings, which have since been replicated many times using more advanced techniques. “They are either crooks or stupid,” said Dr. Masters.”

Thanks to Peter Spero and Hal Pashler.

This is me a few days ago. I did a choice reaction time task many times. Each dot is a session with enough trials to supply 32 correct answers.The y axis is in “percentile” units, meaning speed relative to recent performance. If my speed was at the average of recent performance, the percentile would be 50, for example. Higher percentiles = better performance = faster (shorter reaction time). Each point is a mean; the vertical bars are standard errors. The dotted line is the median of the means.

The graph shows that Friday afternoon I was suddenly unusually slow. After dinner, I returned to normal. A change from 60%ile to 20%ile to 60%ile resembles an IQ change from 105 to 87 to 105 (an 18-point change).

At the same time accuracy was roughly constant:

Because accuracy was roughly constant, the change in speed was not due to a shift on a speed-accuracy tradeoff function.

There are two puzzles here. 1. Why were my scores low Friday afternoon? 2. Why did they recover after dinner? On Friday I didn’t feel well. As a result, I didn’t eat much. Maybe my blood sugar was lower than usual. I usually eat 30 g butter twice/day. On Friday I didn’t have any. At dinner I did have moderate amounts of pork fat (but not butter) and sugar (in lemon citron tea). Friday 6 pm I had a cup of black tea. Although I haven’t noticed effects of tea on these scores, there’s a first time for everything.

Here is a clue to what makes my brain work well (= fast), I conclude. Butter causes sudden improvement, I have found; which makes it plausible that lack of butter (and other animal fat) could cause sudden degradation. Another possibility was that my blood sugar was low Friday afternoon. (I didn’t think of this at the time, and didn’t measure it.)  I’m surprised that something as important as brain function would be as fragile as these results imply. When various nutrient deficiencies are studied with conventional measures, it generally takes weeks or months without the nutrient for the bad effects to become apparent. It takes many weeks without Vitamin C to get scurvy, for example.

These results raise the intriguing possibility that everyone has sudden ups and downs in brain function and that these ups and downs can be detected at high signal/noise ratios. If so, we can use these ups and downs to learn how to make our brains work well. These results also imply — because my choice reaction time test required only a laptop — that anyone can detect them, study them, and learn what causes them. No experts needed. What a change that would be.

In a recent editorial in Nature (gated), the research head of a drug company complained that scientists working for him could not repeat almost all of the “landmark” findings in cancer research that they tried to repeat. They wanted to use these findings as a basis for new drugs.  An article in Reuters summarized it like this:

During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 “landmark” publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development. Result: 47 of the 53 could not be replicated.

Yet these findings were cited, on average, about 200 times. The editorial goes on to make reasonable suggestions for improvement based on differences between the findings that could be repeated and those that could not. The Reuters article goes on to describe other examples of lack of reproducibility and includes a story about why this is happening:

Part way through his project to reproduce promising studies, Begley met for breakfast at a cancer conference with the lead scientist of one of the problematic studies. “We went through the paper line by line, figure by figure,” said Begley. “I explained that we re-did their experiment 50 times and never got their result. He said they’d done it six times and got this result once, but put it in the paper because it made the best story.

Okay, cancer research is less trustworthy than someone just barely outside it (Begley) ever guessed. Apparently careerism is one reason why. What is unexplained in both the Nature editorial and the Reuters summary is how research can ever succeed if things aren’t reproducible. Science has been compared to a game of Twenty Questions. Suppose you play Twenty Questions and 25% of the answers are wrong. It’s hopeless. In experimental research, you generally build on previous experimental results. The editorial points out that the non-reproducible results had been cited 200 times but what about how often they had been reproduced in other labs? The editorial says nothing about this.

I can think of several possibilities: (a) Current lab research is based on experimental findings of thirty years ago when (for unknown reasons) careerism was less of a problem. Standards were higher, there was less pressure to publish, whatever. (b) There is a silent invisible “survival of the reproducible”: Findings that can be reproduced live on because people do lab work based on them. The other findings are cited but are not the basis of new work. (c) There is lots of redundancy — different people approach the same question in different ways. Although each individual answer is not very trustworthy their average is considerably more trustworthy.

Leaving aside the mystery (how can science make any progress if so many results are not reproducible?), the lack of reproducibility interests me because it suggests that the pressure to publish faced by professional scientists has serious (bad) consequences. In contrast, personal scientists are under zero pressure to publish.

Thanks to Bryan Castañeda.

A reader asks:

I haven’t found much on your blog commenting on tools you use to track your data. Any recommendations? Have you tried smart phones? For example, I have tried tracking fifteen variables daily via the iPhone app Moodtracker, the only one I found that can track and graph multiple variables and also give you automated reminders to submit data. There are other variants (Data Logger, Daytum) that will graph one variable (say, miles run per day), but Moodtracker is the only app I’ve found that lets you analyze multiple variables.

I use R on a laptop to track and analyze my data.  I write functions for doing this — they are not built-in. This particular reader hadn’t heard of R. It is free and the most popular software among statisticians. It has lots of built-in functions (although not for data collection — apparently statisticians rarely collect data) and provides lots of control over the graphs you make, which is very important. R also has several programs for fitting loess curves to your data. Loess is a kind of curve-fitting. There is a vast amount of R-related material, including introductory stuff, here.

To give an example, after I weigh myself each morning (I have three scales), I enter the three weights into R, which stores them and makes a graph. That’s on the simple side. At the other extreme are the various mental tests I’ve written (e.g., arithmetic) to measure how well my brain is working. The programs for doing the test are in R, the data is stored in R, and analyzed with R.

The analysis possibilities (e.g., the graphs you can make, your control over those graphs) I’ve seen on smart phone apps are hopelessly primitive for what I want to do. The people who write the analysis software seem to know almost nothing about data analysis. For example, I use a website called RankTracer to track the Amazon ranking of The Shangri-La Diet. Whoever wrote the software is so clueless the rank versus time graphs don’t even show log ranks.

I don’t know what the future holds. In academic psychology, there is near-total reliance on statistical packages (e.g., SPSS) that are so limited perhaps they can extract only half of the information in the usual data. There are many graphs you’d like to make that it is impossible to make. SPSS may not even have loess, for example. Yet I see no sign of this changing. Will personal scientists want to learn more from their data than psychology professors (and therefore be motivated to go beyond pre-packaged analyses)? I don’t know.

• Older eyes block blue light more than young eyes, which may cause sleep problems.
• Caroline Rodgers makes a case (29 slides) that autism is often caused by prenatal ultrasound. This link is from the comments on my previous post about the danger of prenatal ultrasound.
• Why did Deep Throat leak the secrets that brought down President Nixon?
• Do bitters make a difference? A taste test. A better test would have been simpler: to ask, many times in paired comparisons, which is better: Drink X with bitters or drink X without bitters?

Thanks to David Cramer, Jahed Momand and Nancy Evans.

An area of science is an ecosystem in the sense that research builds on other research. In an ordinary ecosystem the animals and plants need each other. Different organisms add different things. Their contributions fit together. In a healthy scientific ecosystem, different types of research add different things and fit together. (more…)

One hundred years ago (January, 1912), at the annual meeting of the Geological Association in Frankfurt, Germany, Alfred Wegener, a meteorologist, presented his theory of continental drift for the first time. It was almost uniformly dismissed by geologists. One of them called it “mere geopoetry”. Much later, he was proved right.

To me, this is a classic example of the power of what I call insider/outsiders. Wegener had a great deal of scientific training, including a Ph.D. in astronomy. Unlike professional geologists, however, (a) he had the freedom to say whatever he wanted about geology without endangering his job (as a meteorologist) or prospects for advancement and (b) was under no pressure to publish. He could spend as much time on his theory as he wanted.

Peter A. Lawrence is a British biologist who has written several papers about problems with the way biology and other areas of science are now done. In this interview a year ago he summarizes his complaints:

• Scientific publication “has become a system of collecting counters for particular purposes – to get grants, to get tenure, etc. – rather than to communicate and illuminate findings to other people. The literature is, by and large, unreadable.” There is far too much counting of papers.
• “There’s a reward system for building up a large group, if you can, and it doesn’t really matter how many of your group fail, as long as one or two succeed. You can build your career on their success.” If you do something on your own it is viewed with suspicion.
• There is too much emphasis on counting citations. “If you work in a big crowded field, you’ll get many more citations. . . . This is independent of the quality of the work or whether you’ve contributed anything. [There is] enormous pressure on the journals to accept papers that will be cited a lot. And this is also having a corrupting effect. Journals will tend to take papers in medically-related disciplines, for example, that mention or relate to common genetic diseases. Journals from, say, the Cell group, will favor such papers when they’re submitted.”
• Grant writing takes too much time — e.g., 30-40% of your time. “There is an enormous increase in bureaucracy – form
  filling, targeting, assessment, evaluations. This has gone right through society, like the Black Death!”
• “Science is not like some kind of an army, with a large number of people who make the main steps forward together. You need to have individually creative people who are making breakthroughs – who make things different. But how do you find those people? I don’t think you want to have a situation in which only those who are competitive and tough can
  get to the top, and those who are reflective and retiring would be cast aside.” I’ve said something similar: Science is like single ants wandering around looking for food, not like a trail of ants to and from a food source. The trail of ants is engineering.

I agree. I would add that I think modern biology is far too invested in the idea that genes cause disease and that studying genes will help reduce human suffering. I think the historical record (the last 30 years) shows that this is not a promising line of work — but modern biologists cannot switch course.

What explains the depressing facts Lawrence points out? I think it is something deep and impossible to change: Science and job don’t mix well. The demands of any job and the demands of science are not very compatible. Jobs are about repetition. Science is the opposite. Jobs demand regular output. Science is unpredictable. However, jobs and science  overlap in terms of training: Both benefit from specialized knowledge. They also overlap in terms of resources: More resources (e.g., better tools) will usually help you do your job better, likewise with science. So we have two groups (insiders — professional scientists — and outsiders — everyone else). Both groups have big advantages and big disadvantages relative to the other. In the last 50 years, the insiders have been “winning” in the sense of doing better work. Their advantages of training and resources far outweighed the problems caused by the need for repetition and predictability. But now — as I try to show on this blog — outsiders are catching up and going ahead because the necessary training and tools have become much more widely available (e.g., tools have become much cheaper). And, as Lawrence emphasizes, professional science has gotten worse.