The Parable of the Wii

For exercise (Dance Dance Revolution) and self-tracking, I decided to buy a Wii. My first attempt, I was scammed. It arrived in August. With difficulty, I took it and accessories unopened to China. That was hard. It was even harder — for no obvious reason — to install it in China. The box sat unopened next to my TV, easily visible, for two months.

Finally I opened the box, took out the parts, put them together, added batteries, plugged it into the TV in my apartment. And nothing happened! Was my TV at fault? Or the Wii? Wii’s aren’t sold in China. I imagined bringing it back to America to get the problem fixed. After a few days, I tested my TV using video output from a neighbor’s Apple computer. My TV worked. After the test, my Wii also worked. When I replaced the Apple input with the Wii input I saw the Wii input for the first time. I don’t understand it, but that’s what happened.

In my experience, this is how science works. It is much harder than expected, then it pays off in ways that defy understanding. The concept of self-experimentation is simple: I will measure X (sleep, productivity) about myself. I will test different ways to improve X, learn what works, and thereby improve X. The reality is different. For years I measured my sleep and tried to improve it. It was hard to deal with the data. Even worse, every idea I had was wrong. That seemed like a huge obstacle — like my Wii needing repair. But I kept plugging away, because it was better than doing nothing, and . . . got somewhere. Out of nowhere and nothing. Not only did I improve my sleep, I arrived at a broader idea about health that turned out to be very helpful (that our bodies are designed for Stone-Age conditions and self-experimentation can help determine those conditions, which aren’t obvious). Just as we overvalue big steps (e.g., well-funded prestigious research), we undervalue small ones (e.g., cheap research with no prestige).

Science is basically a bunch of little steps. Many little experiments that explore cause-effect space. If you find a new example of cause and effect, the payoff is unpredictably large. Scientists don’t like thinking of themselves as wandering ants. But that’s how they are most effective. This goes against human psychology because wandering (Nassim Taleb calls it “tinkering”) is low status and lonely. The payoff is too rare and too unclear. It isn’t supported by powerful institutions, such as research universities and medical schools. Imagine an ant who says “I know where food is!” This is a way to get many ants to follow him, to feel important, to have high status, to get support from his employer. That’s why he does it. But he doesn’t know. The effect on the rest of us, the potential beneficiaries of progress, is that instead of having a thousand ants wandering everywhere, we have a thousand ants following one ant who doesn’t know what he’s doing.

9 Responses to “The Parable of the Wii”

  1. NotInChinanow Says:

    I was and still am unprepared for how unsuccessful my experiments are. Although I don’t have a true tally, the number is certainly less than 20%, and I think probably a fair and accurate count would probably put that number at around 5%. It seems like that number is similar to your success rate with self-experiments. While it’s personally hard to deal with that level of failure, I think it’s even harder at the institutional level. Most institutions couldn’t tolerate employees who get such a low success rate for their experiments even if the resulting success brings much more valuable information. So on a personal level thats why I try to spend time on things that are much more likely to work because it’s easier to defend your work that way. That’s why I like your self-experiments because they allow for more risk that would probably kill the career of a normal experimenter.

  2. Ashish Says:

    If you enjoy Dance Dance Revolution, definitely check out Dance Dance Immolation at Burning Man.

  3. Walter Says:

    When I worked in R/D the upper management wanted predictability in “findings” so we went our way in testing ideas so we ended up with a pipeline of hidden successes and then we metered it out in a “predictable” manner. We ended up predictable but slow in comparison to the rest of the competitor. Upper management patted themselves on the back for being able to manage and to be predictable.
    There is a lot of randomness in R/D and the quicker one built up a pile of “failures” the quicker one found the nugget(s).
    The philosophy I found to work the best is TREE-test randomly, evaluate and elect-and the team should consist of a bunch of people with cognitive diversity and experience diversity.

  4. Andrew Gelman Says:

    Seth: As we’ve discussed many times (and I’ve blogged about too), I think I’ve followed a low-risk, low-return model for science, doing a lot of small projects, each of which is a sure thing (or, at least, something like 50% chance of success, where “success” means advancing the field in some way and publication in a top journal). In contrast, you’ve followed a high-risk, high-return model by spending 15 years doing self-experimentation. (Also, rather than writing 6 big books as I did, you wrote one little book–but your one little book was a bestseller.)

    I guess what I’m saying is that your statement about science, “It is much harder than expected, then it pays off in ways that defy understanding,” describes how you do science, but not necessarily how others do science. Perhaps both types are necessary: we need the bold thinkers like yourself and also the more methodical people like me to fill in the gaps.

  5. seth Says:

    Andrew, by science I meant empirical science, where the main goal is gathering data from which you learn how the world works. This isn’t the main goal of statistics professors. They don’t gather data nor focus on substantive issues. But I agree, you make a good point. Even within this subset of science I agree that one can pursue a path where progress is more predictable. In terms of the scientist = ant analogy, an ant has a choice of joining a trail of many ants to a food source or wandering around by itself to find a new food source. Likewise a scientist who gathers data has the choice of exploiting a known cause-effect relationship (e.g. doing variations on it, trying to explain it) or trying to find new cause-effect relationships. I don’t know what a similar choice would be for statistics profs.

  6. Andrew Gelman Says:

    Seth: Psychologists are good at gathering data. But you don’t need to gather data to be a scientist. You can analyze or build theories based on others’ data. For example, Einstein, Feynmann, etc. were scientists even though they were not experimentalists. And they were empirical scientists too–they explained empirical facts and made testable predictions.

    Also, I completely disagree with your statement that statistics professors don’t “focus on substantive issues.” Take a look at my research articles! It’s possible to be a statistics professor and do applied statistics.

  7. seth Says:

    Andrew, yes, there are exceptions to the broad statements I made. But most scientists (95%?) gather data in one way or another. I think physics and astronomy are the only sciences where there are a lot of pure theorists (such as Feynman), and even in those fields I think the data collectors outnumber them. I think you’re an exception, too — as far as I can tell, most statistics profs spend little of their time trying to answer substantive questions. Most statistics profs are most interested in developing new methods. Only a few statistics profs are also profs in a substantive area, such as political science. I say all this partly because I wonder what would be a “high-risk” line of research for a statistics professor. Maybe developing methods to do something unconventional, such as generate ideas?

  8. seth roberts on scientists as wandering ants « the pulchrifex papers Says:

    […] seth roberts on scientists as wandering ants 2009 November 16 by Matt I’m competing for postdoc funding from the National Institutes of Health; my career path, if I follow it to its logical culmination, will be regularly punctuated with similar competition. So I’m keenly interested in claims like this: Scientists don’t like thinking of themselves as wandering ants. But that’s how they are most effective. This goes against human psychology because wandering (Nassim Taleb calls it “tinkering”) is low status and lonely. The payoff is too rare and too unclear. It isn’t supported by powerful institutions, such as research universities and medical schools. Imagine an ant who says “I know where food is!” This is a way to get many ants to follow him, to feel important, to have high status, to get support from his employer. That’s why he does it. But he doesn’t know. The effect on the rest of us, the potential beneficiaries of progress, is that instead of having a thousand ants wandering everywhere, we have a thousand ants following one ant who doesn’t know what he’s doing. (Full post.) […]

  9. Dennis Whittle: Good Scientists are Like Wandering Ants | News from: The Huffington Post - Breaking News and Opinion Says:

    […] That is from the iconclastic Seth Roberts, formerly a professor of psychology at Berkeley who now teaches at in Beijing at Tsinghua University. His overarching theme is that for science to advance it requires people to come up with and test novel hypotheses rather than tinkering at the margins of the currently accepted wisdom. In short, orthodoxy is often unproductive for scientists, and sometimes dangerous. His blog is full of unexpected hypotheses about how the world works, and he often tests these hypotheses on himself, enlisting his own readers as co-experimenters. […]