Seth's Blog

The New York Times reported a study in Science that found that testing yourself on material you have learned is a good way to improve retention. Then they published a set of letters about it.

John Taylor Gatto, whose books I like, wrote:

Real learning is measured only by utility, by application. In the case of this research, the success claimed for practice testing is being measured by further testing — not by any real-world application.So what? Nobody should care whether memorization is enhanced by practice testing or not.

I disagree. Every day I study Chinese. A lot of that study is memorization, such as what characters mean. Learning what the characters mean while studying in my apartment really does help me understand what they mean out in the world. I care a lot how to memorize better.

A Pace University professor and “director of learning assessment” wrote:

Studying, not test taking, is the key to learning. . . .Testing, particularly standardized testing, does nothing to enhance knowledge and hinders the development of an appreciation for learning that should begin in school and last a lifetime.

I couldn’t disagree more. After I have studied Chinese, frequently testing myself on what I’ve learned  turns out to be essential to long-term retention. Without those tests — say, daily for a week, and less often after that — I forget what I’ve learned.

Standardized testing is especially helpful because it helps me see what works and what doesn’t work. It makes it easier to compare various conditions, in other words.

“Development of an appreciation for learning that should begin in school . . . ” Should? I enjoyed learning long before I started school.

It has taken me a few years to figure out how to learn Chinese. Now I think I am on the right track but these letters illustrate what my self-experimentation also taught me: Experts say the darndest things.

After referring to Jane Jacobs (“successful city neighborhoods need a mixture of old and new buildings”), which I liked, Tim Harford wrote this, which I didn’t like:

Many medical treatments (and a few social policies) have been tested by randomized trials. It is hard to imagine a more clear-cut practice of denying treatment to some and giving it to others. Yet such lotteries — proper lotteries, too — are the foundation of much medical progress.

The notion of evidence-based medicine was a step forward in that it recognized that evidence mattered. It was only a small step forward, however, because its valuation of evidence — on a single dimension, with double-blind randomized trials at the top — was naive. Different sorts of decisions need different sorts of evidence, just as Jacobs said different sorts of businesses need different sorts of buildings. In particular, new ideas need cheap tests, just as new businesses need cheap rent. As an idea becomes more plausible, it makes sense to test it in more expensive ways. That is one reason a monoculture of evidence is a poor idea.

Another is that you should learn from the past. Sometimes a placebo effect is plausible; sometimes it isn’t. To ignore this and insist everything should be placebo-controlled is to fail to learn a lot you could have learned.

A third reason a monoculture of evidence is a poor idea is that it ignores mechanistic understandings — understanding of what causes this or that problem. In some cases, you may think that the disorder you are studying has a single cause (e.g., scurvy). In other cases, you may think the problem probably has several causes (e.g., depression, often divided into endogenous and exogenous). In the latter case, it is plausible that a treatment will help only some of those with the problem. So you should design your study and analyze your data taking into account that possibility. You may want to decide for each subject whether or not the treatment helped rather than lump all subjects together. And the “best” designs will be those that best allow you to do this.

I love this show! It is to comedy what Mad Men is to drama.

• According to Dani Rodrik, a Harvard economics prof, in November 2010 Tunisia was an “unsung development miracle.” According to a Tunisian, “for years, [President] Ben Ali and his cronies have literally stolen from his people and businesses i.e. stealing money from banks, government reserves, gold, etc.”
• Amazingly stupid legal threat against BoingBoing. Maybe a paralegal bet a co-worker that her boss will sign anything.
• No more new computers.

A month ago I posted this graph, which shows how long I needed to type the answer to simple arithmetic problems (7-5, 4*1, 9+0). I tested myself with about 40 problems once or twice per day. Because I’d been doing this for a long time, I no longer improved due to practice. Then, at the end of July 2010, I started improving again.

In September I moved from Berkeley to Beijing. I was worried that in Beijing my scores would get worse. Perhaps I couldn’t get good flaxseed oil or butter. Maybe I would suffer from the air pollution. Maybe I would eat contaminated food. But my scores got better in Beijing.

When I eventually noticed the improvement, I wondered what I was doing differently. Obviously my diet and my life were a lot different in Beijing than Berkeley. Was I eating more walnuts in Beijing? I stopped eating walnuts and my scores didn’t get worse. So it wasn’t walnuts. The most plausible differences I could think of were: 1. Less aerobic exercise in Beijing. 2. Less vitamins in Beijing. 3. Warmer in Beijing. I collected data that implied that shower temperature matters — and I can take warmer showers in Beijing than in Berkeley.

All of these proposed explanations implied that the crucial difference was Berkeley versus Beijing. But the improvement started in Berkeley — around the end of July. That was a problem. Recently I realized there was another possible explanation. In Berkeley I had had an amalgam mercury-containing filling replaced with a non-metallic filling. Not because I had symptoms of mercury poisoning, but because it seemed prudent.

I checked my records to see when I had the filling replaced. It was July 28 — right when the improvement started. To my shock, reduction in mercury exposure is now the most plausible explanation of the improvement. Two tests of this explanation are coming up: 1. When I return to Berkeley, will my reaction times go up? 2. When I have more amalgam fillings replaced, will my reaction times go down?

If it turns out that reduction in mercury exposure is the correct explanation, this will be important. I have an average number of fillings. I’d guess that half of Americans have as many amalgam fillings as I did. And — if the mercury explanation is correct — this arithmetic test is a sensitive measure of mercury poisoning. Over the last few years, before the filling was removed, I’d had six hair tests done, all from the same reputable lab. They showed that my mercury level was moderately high, perhaps 75th percentile. Not very worrisome.

I changed dentists because my old dentist made a terrible mistake: he put a gold filling next to an amalgam one. Putting one metal next to a different one is an elementary mistake. Contact of different metals creates an electric current (as Galvani discovered) and releases mercury. (So although I have a normal number of fillings perhaps I have more mercury exposure.) I stopped going to him for any dental work. The last time I went there for a cleaning, I was given a booklet (“we must give you this”) about the many sorts of dental materials — mercury amalgams plus several new ones. The purpose seems to be to tell people mercury amalgams aren’t dangerous (this was stressed) yet get them to choose other materials in the future — mercury amalgams are just one of several possible choices. The controversy about the safety of mercury amalgams is covered here. Sweden, Denmark, and Norway have banned mercury amalgams. The ban began 2008.