Seth's Blog

Nassim Taleb recently criticized the Nobel Prize in Economics:

According to Taleb, there are a number of mistaken ideas about forecasting and measuring risk, which all contribute to events like the 2008 global crisis. The Nobel prize, he says, has given them a stamp of approval, allowing them to propagate.

It isn’t just economics. As I’ve said before, the Nobel Prize in medicine was not given for the discovery that smoking causes lung cancer. It was not given for the discovery that lack of folate causes birth defects. Both enormously useful. It has been given for several discoveries, such as the connection between teleomeres and aging, with (so far) little or no practical value.

This is no mystery. The Nobel Prize must be prestigious, therefore must honor high-prestige research. Veblen argued long ago that in academia high prestige correlates with low practical value. Just today I told a friend Veblen’s idea that professors use jargon for the same reason men wear ties — to show off how useless they are. The economics research (“Harry Markowitz, William Sharpe, Robert Merton, Myron Scholes, Robert Engle, Franco Modigliani and Merton Miller”) that Taleb is criticizing was high prestige. The so-far-useless biology that has received a Nobel Prize was high prestige; the highly-useful epidemiology that didn’t receive the prize was low prestige.

Thanks to Dave Lull.

After posting about Doug Lemov, I ordered Teach Like a Champion. It arrived yesterday. Leafing through it, I came across a section titled “The Irony of What Works,” which begins:

One of the biggest ironies I hope you will take away from reading this book is that many of the tools likely to yield the strongest classroom results remain essentially beneath the notice of our theories and theorists of education.

Lemov continues with an example: Teaching students how to distribute classroom materials, such as handouts. This can save a lot of time. Then he adds:

Unfortunately this dizzyingly efficient technique — so efficient it is all but a moral imperative for teachers to use it — remains beneath the notice of our avatars of educational theory. There isn’t a school of education that would stoop to teach its aspiring teachers how to train their students to pass out papers.

The last chapter of Veblen’s  Theory of the Leisure Class is about just this — the importance that professors (like everyone else) place on status display and how this interferes with their effectiveness. The connection with self-experimentation is that no matter how effective it is, no psychology department would stoop to teach it. Or, at least, that’s the current state of affairs.

The book’s index doesn’t include Veblen, although it does include Richard Thaler.

I’ve blogged many times about the desire of professors to show off and how it interferes with being useful. It doesn’t just make them bad teachers, it makes them bad scientists. Here’s an example from economics (via Marginal Revolution):

“The mainstream of academic research in macroeconomics puts theoretical coherence and elegance first, and investigating the data second,” says Mr. Rogoff. For that reason, he says, much of the profession’s celebrated work “was not terribly useful in either predicting the financial crisis, or in assessing how it would it play out once it happened.”

“[Academic economists] almost pride themselves on not paying attention to current events,” he says.

Pure Veblen, who in Theory of the Leisure Class provided many examples of people, including professors, priding themselves on being useless. Men wear ties, he said, to show they don’t do manual labor (which is clearly useful).

My research is closer to biology, where you can say the same thing: much of the profession’s celebrated work has not been terribly useful. Yesterday I gave an example (the oncogene theory of cancer).

Modern Veblen: Flight From Data.

A new Jeffrey Eugenides short story quotes Derrida. Quote 1:

In that sense it is the Aufhebung of other writings, particularly of hieroglyphic script and of the Leibnizian characteristic that had been criticized previously through one and the same gesture.

Quote 2:

What writing itself, in its nonphonetic moment, betrays, is life. It menaces at once the breath, the spirit, and history as the spirit’s relationship with itself. It is their end, their finitude, their paralysis.

“A little Derrida goes a long way and a lot of Derrida goes a little way,” said a friend of mine who was a graduate student in English. These quotes show why. In Theory of the Leisure Class, Veblen argued that professors write like this (and assign such stuff to their students) to show status. I have yet to hear a convincing refutation of this explanation nor a plausible alternative. Is there a plausible alternative?

Veblen was saying that professors are like everyone else. Think of English professors as a model system. Their showing-off is especially clear. It’s pretty harmless, too, but when a biology professor (say) pursues a high-status line of research about some disease rather than a low-status but more effective one, it does — if it happens a lot — hurt the rest of us. Sleep researchers, for example, could do lots of self-experimentation but don’t, presumably because it’s low-status. And poor sleep is a real problem. Throughout medical school labs, researchers are studying the biochemical mechanism and genetic basis of this or that disorder. I’m sure this is likely to be less effective in helping people avoid that disorder than studying its environmental roots, but such lines of research allow the researchers to request expensive equipment and work in clean isolated laboratories — higher status than cheap equipment and getting your hands dirty. I don’t mean high-status research shouldn’t happen; we need diversity of research. But, like the thinking illustrated by the Derrida quotes, there’s too much of it. A little biochemical-mechanism research goes a long way and lot of biochemical-mechanism research goes a little way.

An assistant manager at Marshall Fields, the Chicago department store, told Gawker the following story:

I was walking through the floor, and I hear a voice call my name. . . . Once she started speaking to me, I realized it was Oprah. Honestly, she is unrecognizable without the spackle/wig. Anyway, she was very nice, and asked me if I would offer my opinion on a china pattern she was looking at for her house. It was Villeroy and Boch (German, middle-range) “Petite Fleur.” Very cute, kind of French-country, with a small, scattered floral design. I said, “What’s not to like?” Oprah responded, “Well, it’s not that expensive, and I don’t want people who come to my house to think I’m cheap.”

Andrew Gelman writes:

If I had to come up with one statistical tip that would be most useful to you–that is, good advice that’s easy to apply and which you might not already know–it would be to use transformations. Log, square-root, etc.–yes, all that, but more! I’m talking about transforming a continuous variable into several discrete variables (to model nonlinear patterns such as voting by age) and combining several discrete variables to make something [more] continuous (those “total scores” that we all love). And not doing dumb transformations such as the use of a threshold to break up a perfectly useful continuous variable into something binary. I don’t care if the threshold is “clinically relevant” or whatever–just don’t do it. If you gotta discretize, for Christ’s sake break the variable into 3 categories.

I agree (and wrote an article about it). Transforming data is so important that intro stats texts should have a whole chapter on it — but instead barely mention it. A good discussion of transformation would also include use of principal components to boil down many variables into a much smaller number. (You should do this twice — once with your independent variables, once with your dependent variables.) Many researchers measure many things (e.g., a questionnaire with 50 questions, a blood test that measures 10 components) and then foolishly correlate all independent variables with all dependent variables. They end up testing dozens of likely-to-be-zero correlations for significance. Thereby effectively throwing all their data away — when you do dozens of such tests, none can be trusted.

My explanation why this isn’t taught differs from Andrew’s. I think it’s pure Veblen: professors dislike appearing useful and like showing off. Statistics professors, like engineering professors, do less useful research than you might expect, so they are less aware than you might expect of how useful transformations are. And because most transformations don’t involve esoteric math, writing about them doesn’t allow you to show off.

In my experience, not transforming your data is at least as bad as throwing half of it away, in the sense that your tests will be that much less sensitive.

Describing those who made money in the subprime mortgage market, Michael Lewis said this:

They were outsiders to the market that they were betting on. And in addition, they were, in many cases, personally curious people, not clubbable members of the group. And I think that was a key to the success. I think that the fact that they didn’t feel compelled in any way, on any level, to think like other people gave them an advantage.

This is what Thorstein Veblen said about Jews in a 1917 essay titled “The intellectual pre-eminence of Jews in modern Europe.” Being outsiders gave them freedom of thought. Lewis may have read that essay. A few years ago, he compiled an anthology of economic classics, one of which was Veblen’s Theory of the Leisure Class. I mentioned this essay earlier.

A few days ago I quoted an editor who works for Rupert Murdoch as saying that journalists care too much about impressing their colleagues and winning prizes and not enough about helping readers. Here is Walter Pincus, a Washington Post reporter, saying the same thing:

Editors have paid more attention to what gains them prestige among their journalistic peers than on subjects more related to the everyday lives of readers. For example, education affects everyone, yet I cannot name an outstanding American journalist on this subject.

I quote this to support the Veblenian view I’ve expressed many times on this blog — that scientists would rather do what gains them prestige among their peers than what helps the rest of us, who support most science. I think it’s hard to understand the success of my self-experimentation (e.g., new ways of losing weight) until you understand this aspect of science. I was successful partly because my motivation was different.

According to a recent New York article about Rupert Murdoch, Robert Thomson, one of Murdoch’s top editors,

thinks most [journalists] are liberals overly concerned with writing stories that will impress other liberal journalists and win prizes in journalism competitions.

Well, yes. Not everyone is a liberal, of course, but basically everyone wants to impress their colleagues. Scientists have an amusing spin on this: They call it “peer review.” The amusing part is that somehow no one else’s opinion should matter. (E.g., all journals must be peer-reviewed.) Scientists get away with this bizarre view of economics (thinking someone should pay you and get nothing in return) perhaps because it is indeed difficult to assess the quality of this or that bit of science if you’re not in the field and because science has produced huge benefits for the rest of us in the past.

As I said, this is just human nature. As far as I can tell, professors act this way — try to impress colleagues — in every academic department. In schools of education, the result is this:

Amy Treadwell . . . received her master’s degree in education from DePaul University, a small private university in Chicago. . . . But when she walked into her first job, teaching first graders on the city’s South Side, she discovered a major shortcoming: She had no idea how to teach children to read. “I was certified and stamped with a mark of approval, and I couldn’t teach them the one thing they most needed to know how to do,” she told me.

It’s no secret that many schools of education do a poor job of training their students to teach — which is nominally one of their main goals. I am just repeating what Veblen said long ago.

What’s new is this: One man, Doug Lemov, working mostly alone, has figured out how to make people better teachers. One man. Not a professor. Did he build on the work of others? No, he started from scratch. He’s made a list of about 50 techniques. They are teachable. He gives workshops about them. As far as I can tell from this magazine article, Lemov has done a better job of figuring out how to train teachers than all the education professors in the world put together. If you arrived on earth from outer space, and didn’t understand human nature, you’d think this couldn’t possibly be true, but apparently it is. It’s like something out of a comic book.

In 1977, John Tukey published a book called Exploratory Data Analysis. It introduced many new ways of analyzing data, all relatively simple. Most of the new ways involved plotting your data. A few involved transforming your data. Tukey’s broad point was that statisticians (taught by statistics professors) were missing a lot: Conventional statistics focussed too much on confirmatory data analysis (testing hypotheses) to the omission of exploratory data analysis — data analysis that might show you something new. Here are some tools to help you explore your data, Tukey was saying.

No question the new tools are useful. I have found great benefits from plotting and transforming my data. No question that conventional statistics textbooks place far too little emphasis on graphs and transformations. But I no longer agree with Tukey’s exploratory versus confirmatory distinction. The distinction that matters — at least to historians, if not to data analysts — is between low-status and high-status. A more accurate title of Tukey’s book would have been Low-Status Data Analysis. Exploratory data analysis already had a derogatory name: Descriptive data analysis. As in mere description. Graphs and transformations are low-status. They are low-status because graphs are common and transformations are easy. Anyone can make a graph or transform their data. I believe they were neglected for that reason. To show their high status, statistics professors focused their research and teaching on more difficult and esoteric stuff — like complicated regression. That the new stuff wasn’t terribly useful (compared to graphs and transformations) mattered little. Like all academics — like everyone — they cared enormously about showing high status. It was far more important to be impressive than to be useful. As Veblen showed, it might have helped that the new stuff wasn’t very useful. “Applied” science is lower status than “pure” science.

That most of what statistics professors have developed (and taught) is less useful than graphs and transformations strikes me as utterly clear. My explanation is that in statistics, just as in every other academic area I know about, desire to display status led to a lot of useless highly-visible work. (What Veblen called conspicuous waste.) Less visibly, it led to the best tools being neglected. Tukey saw the neglect –  underdevelopment and underteaching of graphs, for example — but perhaps misdiagnosed the cause. Here’s why Tukey’s exploratory versus confirmatory distinction was misleading: Because the tools that Tukey promoted for exploration also improve confirmation. They are neglected everywhere. For example:

1. Graphs improve confirmatory data analysis. If you do a t test (or compute a p value in any way) but don’t make an associated graph, there is room for improvement. A graph will show whether the assumptions of the computation are reasonable. Often they aren’t.

2. Transformations improve confirmatory data analysis. That a good transformation will make the assumptions of the test more reasonable many people know. What few people seem to know is that a good transformation will make the statistical test more sensitive. If a difference exists, the test will be more likely to detect it. This is like increasing your sample size at no extra cost.

3. Exploratory data analysis is sometimes thought of as going beyond the question you started with to find other structure in the data — to explore your data. (Tukey saw it this way.) But to answer the question you started with as well as possible you should find all the structure in the data. Suppose my question is whether X has an effect.  I should care whether Y and Z have an effect in order to (a) make my test of X more sensitive (by removing the effects of Y and Z) and (b) assess the generality of the effect of X (does it interact with Y or Z?).

Most statistics professors and their textbooks have neglected all uses of graphs and transformations, not just their exploratory uses. I used to think exploratory data analysis (and exploratory science more generally) needed different tools than confirmatory data analysis and confirmatory science. Now I don’t. A big simplification.

Exploration (generating new ideas) and confirmation (testing old ideas) are outputs of data analysis, not inputs. To explore your data and to test ideas you already have you should do exactly the same analysis. What’s good for one is good for the other.

Likewise, Freakonomics could have been titled Low-status Economics. That’s essentially what it was, the common theme. Levitt studied all sorts of things other economists thought were beneath them to study. That was Levitt’s real innovation — showing that these questions were neglected. Unsurprisingly, the general public, uninterested in the status of economists, found the work more interesting than high-status economics. I’m sensitive to this because my self-experimentation was extremely low-status. It was useful (low-status), cheap (low-status), small (low-status), and anyone could do it (extremely low status).

More Andrew Gelman comments. Robin Hanson comments.