The Twilight of Expertise (part 12: Super Crunchers)

Ian Ayres’ interesting new book, Super Crunchers, has a chapter about expert prediction versus predictions from math models. Almost always, the math models do better than the experts. I learned about this in graduate school when I read stuff by Paul Meehl, a psychology professor who compared the predictions of clinicians and regression equations in the 1950s. The idea has gathered strength since then and now the persons in some jobs — such as loan officers — are required to follow an algorithm for making decisions. Their expertise is ignored. Obviously they no longer derive as much self-worth from their job, Ayres points out.

It’s like the beginning of agriculture. Lots has been written about the physical problems caused by the change to agriculture. Stature decreased, tooth decay increased, and so on. I’ve never read about the mental problems it must have caused. I can only speculate, of course, but here’s an possible example: Hunters derived self-worth from bringing meat to their families. Taking that away caused problems. (Watching Once Were Warriors, a terrific movie, should make this more plausible.)

I have never read anything about how to reintroduce into everyday jobs crucial mental elements that hunting had and farming lacked. Nutrition education, vitamin supplements, dietary fortification, and other nutrition programs push us toward a pre-agricultural diet, which was far more diverse and better balanced. There is no similar set of things that move us closer to pre-agricultural ways of making a living. My self-experimental research is all about the value stuff that ancient life had but modern life lacks — such as seeing lots of faces in the morning — but I have never figured out how to simulate elements of hunting, beyond being on one’s feet a lot.

7 Responses to “The Twilight of Expertise (part 12: Super Crunchers)”

  1. Andrew Gelman Says:

    Sure, but what you call “the math models” (I would call them “statistical models” since I think when you fit a mathematical model to data, it’s a statistical model) need experts to run them well. A good algorithm doesn’t come from nowhere.

  2. michael vassar Says:

    You get some of the elements of hunting-gathering, at least of the gathering part, by shopping, hence, presumably, its popularity. Other elements can come from frantically running around to do small one-shot freelance jobs and tasks, such as being a taxi driver searching for fares. Video games presumably satisfy other elements of the faux-hunter lifestyle, or certain physical recreations such as hide-and-seek.

  3. Timothy Beneke Says:

    Philip Tetlock’s book on expert political prediction “Expert Political Judgment: How Good Is It? How Can We Know?” found
    (http://www.newyorker.com/archive/2005/12/05/051205crbo_books1)
    it to be largely bogus, as I understand it. The New Yorker summarized his findings. Even rats made better predictions than Yale students at a particular task.
    From Louis Menand’s New Yorker article:
    “It is the somewhat gratifying lesson of Philip Tetlock’s new book, “Expert Political Judgment: How Good Is It? How Can We Know?” (Princeton; $35),that people who make prediction their business—people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables—are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones.”
    Why am I not surprised?

  4. Tom Myers Says:

    You’re linking two transitions: (A) from hunter-gatherer to agricultural economics, and (B) from expert to math-model decision-making. Okay, each involves a replacement of creativity/expertise by a routine. We might throw in (C) the industrial revolution, where Adam Smith’s pin factory replaces craftmanship by assembly-line production. However, the “Super-Cruncher” transition (B) seems to me to be fundamentally different in that the routine is to be carried out by a computer. The expert is upset at losing status, but the resulting society is not one of increased drudgery; it’s simply a society in which people (patients, sports teams, investors, customers, governments) depend less on human experts and more on their “own” (computer) resources, as fed by their own intuitions and web-collected data from all over the world. See page 124: “The most important thing that is left to humans is to use our minds and our intuitions to guess at what variables should and should not be included in statistical analysis.” This may be intrinsically closer to a hunter-gatherer structure after all; there’s no hierarchy of experts, perhaps no hierarchy of authority at all.
    Of course we will need Gelmans for a while, but in the end perhaps we can breed good modeling algorithms via genetic algorithms; if experts are replaced by algorithms, meta-experts may be replaced by meta-algorithms. And then the Singularity? Well, maybe. It is a step in that direction, whether the journey continues or not.

  5. Seth’s blog » Blog Archive » The Twilight of Expertise (directory) Says:

    [...] The Twilight of Expertise (part 12: Super Crunchers) [...]

  6. Seth’s blog » Blog Archive » Once Were Warriors Says:

    [...] The Twilight of Expertise (part 12: Super Crunchers) [...]

  7. Heather Says:

    I don’t know what’s keeping Dr. Consorti so busy. She obviously doesn’t spend her time making sure her patients are receiving the quality care they deserve. If she would’ve taken the other 30 minutes required for my surgery, maybe the skin from both sides of my incision would match up. Thanks to her, I’ll probably be having another surgery down the road to remove scar tissue from her horrific attempt at stitching. Maybe she should’ve taken HomeEc before going to med school.