Seth's Blog

Alex Tabarrok discusses a proposal to make referee reports and associated material publicly available. I think it would be a good thing because it would make writing a self-serving review (e.g., a retaliatory review) more dangerous. If Reviewer X writes an unreasonable review, the author is likely to complain to the editor. If the paper gets published, the unreasonableness will be highlighted — and nominal anonymity may not be enough to hide who wrote it. On the other side, as a reader, it would be extremely educational. You could learn a lot from studying these reports and the replies they generated, especially if you’re a grad student. I would like to know why some papers got accepted. For example, my Tsinghua students pointed out serious flaws in published papers. Were the problems noted by reviewers and ignored, or what?

My experience is that about 80% of reviews are reasonable. Many of those are ignorant, but that’s no crime. (A lot of reviewers know more than me.) The remaining 20% seem to go off the rails somehow. For example, Hal Pashler and I wrote a paper criticizing the use of good fits to support quantitative models. The first two reviewers seemed to have been people who did just that. Their reviews were ridiculous. Apparently they thought the paper shouldn’t be published because it might call their work into question. A few reviews have appeared to be retaliation. In the 1990s, I complained to the Office of Research Integrity that a certain set of papers appeared to contain made-up data. (ORI sent the case to the institution where the research was done. A committee to investigate did the shallowest possible review and decided I was wrong. I learned my lesson — don’t trust ORI — which I applied to the Chandra case.) After that allegation, I got stunningly unfair reviews from time to time, presumably from the people I accused. A small fraction of reviews (5%?) are so lazy they’re worthless. One reviewer of my long self-experimentation paper said it shouldn’t be published because it wasn’t science. The author (me) should go do some real science.

The main things I’ve learned about how to respond are: 1. When resubmitting the paper (revised in light of the reviews), go over every objection and how it was dealt with or why it was ignored. Making such a list isn’t very hard, it makes ignoring a criticism much easier (because you are explicit about it), and editors like it. This has become common. 2. When a review is unreasonable, complain. The theory-testing paper I wrote with Hal is one of my favorite papers and it wouldn’t have been published where it was if we hadn’t complained. Another paper of mine said that some data failed a chi-square test many times — suggesting that something was wrong. One of the reviewers seemed to not understand what a chi-square test was. I complained and got a new reviewer.

I’m curious: What have you learned about responding to reviewers?

In an excellent talk last week about North Korea — linked to his book The Cleanest Race — Brian Myers, a professor in South Korea, said that people don’t fear dying, they fear dying without significance. Without their life having meant something. Life in North Korea is far more attractive than Americans realize, he said. The border between North Korea and China is easy to cross, and about half of the North Koreans who go to China later return, in spite of North Korea’s poverty. How does the North Korean government do such a good job under such difficult circumstances? Partly by playing up external threats (U.S. troops in South Korea), the obvious way politicians win support, but also by telling the North Korean people they are special. Maybe it plays this card because it has to — they can’t afford a police state — but there is no denying how well it works. In contrast, Myers said, the South Korean government offers its citizens no more than consumerism. That doesn’t work well, and South Korea, in spite of high per capita income, has high rates of depression and suicide.

I think the attractiveness of North Korean life has a lot to do with why Penn State students like Penn State so much. This American Life did a show about Penn State a few months ago. Life at the nation’s top party school said the description. Sounds boring, I thought, so I waited to listen to it until I’d run out of stuff to listen to. It turned out to be one of their best shows ever. Mostly it’s about the large amount of drinking — this is why they did the show — but at the very end is a short segment about how much Penn State students love their school. Not much detail but I was convinced. The attractive school cheer (“We Are Penn State”) comes up in conversation! A few people reading this won’t know that Penn State has an extremely successful football team. A large fraction of the students attend its games. After graduation, a lot of them continue to attend the games.

Here is a powerful and neglected force in human life. The bland technical term is group identity.  As the South Korea comparison indicates, governments don’t routinely use it to govern. As Penn State exceptionalism indicates, colleges don’t routinely use it either. Faculty routinely disparage football. Beer and Circus: How Big-Time College Sports Has Crippled Undergraduate Education was written by a professor — of course. The Penn State chancellor seemed mystified that his students were so proud and supportive of their school. (They’re just that way, he seemed to say.) A lot of my self-experimentation has been about discovering what we need to be healthy, such as morning faces. I can’t self-experiment about this but I would if I could. It’s yet another thing that people must have routinely gotten in Stone-Age life but don’t get any more — unless you happen to be a rabid sports fan or an alumnus of a college with a sufficiently successful football team. Or live in North Korea.

• Paleohacks. Crowd-sourced Q & A.
• Cable cars are undervalued. No mention of elevators, which strengthen the case.
• Vancouver = world’s most livable city? Silly Economist!
• blood glucose self-experimentation

In 1977, John Tukey published a book called Exploratory Data Analysis. It introduced many new ways of analyzing data, all relatively simple. Most of the new ways involved plotting your data. A few involved transforming your data. Tukey’s broad point was that statisticians (taught by statistics professors) were missing a lot: Conventional statistics focussed too much on confirmatory data analysis (testing hypotheses) to the omission of exploratory data analysis — data analysis that might show you something new. Here are some tools to help you explore your data, Tukey was saying.

No question the new tools are useful. I have found great benefits from plotting and transforming my data. No question that conventional statistics textbooks place far too little emphasis on graphs and transformations. But I no longer agree with Tukey’s exploratory versus confirmatory distinction. The distinction that matters — at least to historians, if not to data analysts — is between low-status and high-status. A more accurate title of Tukey’s book would have been Low-Status Data Analysis. Exploratory data analysis already had a derogatory name: Descriptive data analysis. As in mere description. Graphs and transformations are low-status. They are low-status because graphs are common and transformations are easy. Anyone can make a graph or transform their data. I believe they were neglected for that reason. To show their high status, statistics professors focused their research and teaching on more difficult and esoteric stuff — like complicated regression. That the new stuff wasn’t terribly useful (compared to graphs and transformations) mattered little. Like all academics — like everyone — they cared enormously about showing high status. It was far more important to be impressive than to be useful. As Veblen showed, it might have helped that the new stuff wasn’t very useful. “Applied” science is lower status than “pure” science.

That most of what statistics professors have developed (and taught) is less useful than graphs and transformations strikes me as utterly clear. My explanation is that in statistics, just as in every other academic area I know about, desire to display status led to a lot of useless highly-visible work. (What Veblen called conspicuous waste.) Less visibly, it led to the best tools being neglected. Tukey saw the neglect –  underdevelopment and underteaching of graphs, for example — but perhaps misdiagnosed the cause. Here’s why Tukey’s exploratory versus confirmatory distinction was misleading: Because the tools that Tukey promoted for exploration also improve confirmation. They are neglected everywhere. For example:

1. Graphs improve confirmatory data analysis. If you do a t test (or compute a p value in any way) but don’t make an associated graph, there is room for improvement. A graph will show whether the assumptions of the computation are reasonable. Often they aren’t.

2. Transformations improve confirmatory data analysis. That a good transformation will make the assumptions of the test more reasonable many people know. What few people seem to know is that a good transformation will make the statistical test more sensitive. If a difference exists, the test will be more likely to detect it. This is like increasing your sample size at no extra cost.

3. Exploratory data analysis is sometimes thought of as going beyond the question you started with to find other structure in the data — to explore your data. (Tukey saw it this way.) But to answer the question you started with as well as possible you should find all the structure in the data. Suppose my question is whether X has an effect.  I should care whether Y and Z have an effect in order to (a) make my test of X more sensitive (by removing the effects of Y and Z) and (b) assess the generality of the effect of X (does it interact with Y or Z?).

Most statistics professors and their textbooks have neglected all uses of graphs and transformations, not just their exploratory uses. I used to think exploratory data analysis (and exploratory science more generally) needed different tools than confirmatory data analysis and confirmatory science. Now I don’t. A big simplification.

Exploration (generating new ideas) and confirmation (testing old ideas) are outputs of data analysis, not inputs. To explore your data and to test ideas you already have you should do exactly the same analysis. What’s good for one is good for the other.

Likewise, Freakonomics could have been titled Low-status Economics. That’s essentially what it was, the common theme. Levitt studied all sorts of things other economists thought were beneath them to study. That was Levitt’s real innovation — showing that these questions were neglected. Unsurprisingly, the general public, uninterested in the status of economists, found the work more interesting than high-status economics. I’m sensitive to this because my self-experimentation was extremely low-status. It was useful (low-status), cheap (low-status), small (low-status), and anyone could do it (extremely low status).

More Andrew Gelman comments. Robin Hanson comments.

I blogged earlier about an Italian med school professor named Paulo Zamboni who, studying his wife, came up with an entirely new theory about multiple sclerosis (MS): It’s caused by restricted blood outflow from the brain. Almost all MS patients had this condition, Zamboni found. The great value of this theory is that blood outflow can often be improved with surgery. In at least some cases, this surgery has reduced MS symptoms.

Now, a study done in Buffalo has found results that support Zamboni’s idea. MS patients were twice as likely as healthy people to have restricted blood flow. This is a weaker correlation than Zamboni found but I make nothing of it — there are lots of ways to mess things up, so that you get noisier results. (And there are lots of ways to push results in a preferred direction.)

Zamboni wasn’t an MS expert. He made this breakthrough because his wife had MS and he had technical skills (including surgical skills — his specialty is surgery).

Thanks to Anne Weiss.

More A more detailed description.