Seth's Blog

Yesterday I commented about Give and Take by Adam Grant, a professor at Wharton who teaches organizational psychology.

When Grant was a graduate student (at the University of Michigan), he was asked to help people at the university’s fund-raising call center raise more money. They call alumni, asking for money. The person who ran the center had tried the usual motivational tactics, such as offering bonuses. They hadn’t worked.

Grant noticed that most of the money being raised went for scholarships. He tried various ways of making the call center employees aware that the money they raised helped students directly. The most effective way turned out to be a 5-minute meeting with a scholarship recipient. This had a staggering effect:

The average caller doubled in calls per hour and minutes on the phone per week . . . Revenue quintipled: callers averaged $412 [per week] before meeting the scholarship recipient and more than $2000 afterward.

A huge effect — and a useful huge effect. And one that is not even hinted at in countless introductory psychology books. Notice that physical conditions of the job and the “physical” payoff (the salary) didn’t change. All that changed was employees’s mental models of their job.

I conclude that people are far more motivated by a desire to help others than you would ever guess from reading psychology textbooks — and, even more, from reading economics textbooks. Grant says nothing about this, at least in the book, but I’d guess that the employees were considerably happier at their jobs as well. You might think that there has been so much research on job design that there were no big effects left to be discovered. You’d be wrong.

The publisher sent me a copy of Give and Take by Adam Grant after I sent several emails asking for a review copy. I expected it to be the best book about psychology in many years and it is.

The book’s main theme is the non-obvious advantages of being a “giver” (someone who helps others without concern about payback). Grant teaches at Wharton, whose students apparently enter Wharton believing (or are taught there?) that this is a poor strategy. With dozens of studies and stories, Grant argues that the truth is more complicated — that a giver, properly focussed, does better than others. Whether this reflects cause and effect (Grant seems to say it does) I have no idea. Perhaps “givers” are psychologically unusually sophisticated in many ways, not just a relaxed attitude toward payback, and that is why some of them do very well. (more…)

• Acting classes improve memory of older adults
• Baltimore hairdresser proposes interesting theory about ancient Rome (via John Hawks weblog). The WSJ article fails to mention what a commenter points out, that the hairdresser’s husband is a professor of Italian.
• Dried plum reverses bone loss in mice. More than other dried fruit. Impressive study.
• US life expectancy stalled.

Thanks to Greg Pomerantz and Casey Manion.

A post on the New Yorker website called “Brain Games are Bogus” provides considerable evidence for that conclusion. The evidence is about the use of brain games to raise the IQ of children and young adults, whereas Posit Science’s training program — which I raised questions about — is aimed at older people. However, it would be surprising if brain games have no effect until you reach a certain age. More plausible is that they never provide substantial benefits — at least, benefits broad enough and strong enough and long-lasting enough to be worth the training time (one hour/day for many weeks).

I read a Posit Science paper, with older subjects, that seemed to me to show that its training had little lasting benefit. The stated conclusions of the paper were more positive. Too bad the head of Posit Science didn’t answer most of my questions.

Thanks to Alex Chernavsky.

In an earlier post I asked 15 questions about Zelinski et al. (2011) (“Improvement in memory with plasticity-based adaptive cognitive training: results of the 3-month follow-up”), a study done to measure the efficacy of the brain training sold by Posit Science. The study asked if the effects of training were detectable three months after it stopped. Henry Mahncke, the head of Posit Science, recently sent me answers to a few of my questions.

Most of my questions he declined to answer. He didn’t answer them, he said, because they contained “innuendo”. My questions were ordinary tough (or “critical”) questions. Their negative slant was not at all hidden (in contrast to  innuendo). For the questions he didn’t answer, he substituted less critical questions. I give a few examples below.  Unwillingness to answer tough questions about a study raises doubts about it.

His answers raised more doubts. (more…)

At Berkeley, Andrew Gelman and I taught a freshman seminar about left-handedness. Half the students were left-handed. We did two fascinating studies with them that found that left-handers tend to have left-handed friends. I kick myself for not publishing those results, which I bring up in conversation again and again.

After the class ended I got a call from a journalist who was writing an article about ridiculous classes. I told him the left-handedness class had value as a way of introducing methodological issues but all I cared about was that his article be accurate. He decided not to include our class in his examples.

Stephen Christman, who got his Ph.D. from Berkeley (and did quirky interesting stuff even as a graduate student), and two colleagues have now published a paper that is a considerable step forward in the understanding of handedness. They argue that what really matters is not direction of handedness but the consistency of it. The terms left-handed and right-handed hide a confounding. Right-handers almost all have very consistent handedness (they do everything with the right hand). In contrast, left-handers much more often have inconsistent handedness: they do some things with the left hand, some with the right. I am a good example. I write with my right hand, bat and throw left-handed, play tennis left-handed, ping-pong right-handed. In fact, I am right-wristed and left-armed. When something involves wrist movement (writing, ping-pong) I use my right hand. When something involves arm movement (batting, throwing a ball, tennis), I use my left hand. Right-handers are much more similar to each other than left-handers.

Christman and his co-authors point to two things: 1. When you can get enough subjects to unconfound the two variables, it turns out that consistency of handedness is what makes the difference. Consistent left-handers resemble consistent right-handers.  2. Consistency of handedness predicts many things. Inconsistent-handers are less authoritarian than consistent-handers. They show more of a placebo effect. They have better memory for paragraphs. And on and on — about 20 differences. It isn’t easy to say what all these differences have in common but maybe inconsistent-handers are more flexible in their beliefs. (Which would explain the friendship findings in our handedness class.)

I think about these differences as another example of how every economy needs diversity and our brains have been shaped to provide it, one idea underlying my theory of human evolution. Presidents of the United States are left-handed much more than the general population. For example, Obama is left-handed. The difference between Presidents and everyone else is overwhelming and must mean something. Yet left-handers die younger. I would say that in any group of people you need a certain fraction, not necessarily large, to be open-minded and realistic. That describes inconsistent-handers (who are usually left-handed). These people make good leaders because they will respond to changing conditions. People who are not open-minded make good followers. Just as important as realism is cooperation, ability to work together toward a common goal.

Posit Science is a San Francisco company, started by Michael Merzenich (UCSF) and others, that sells access to brain-training exercises aimed at older adults. Their training program, they say, will make you “remember more”, “focus better”, and “think faster”. A friend recently sent me a 2011 paper (“Improvement in memory with plasticity-based adaptive cognitive training: results of the 3-month follow-up” by Elizabeth Zelinski and others, published in the Journal of the American Geriatrics Society) that describes a study about Posit Science training. The study asked if the improvements due to training are detectable three months after training stops. The training takes long enough (1 hour/day in the study) that you wouldn’t want to do it forever. The study appears to have been entirely funded by Posit Science.

I found the paper puzzling in several ways. I sent the corresponding author and the head of Posit Science a list of questions:

1. Isn’t it correct that after three months there was no longer reliable improvement due to training according to the main measure that was chosen by you (the investigators) in advance? If so, shouldn’t that have been the main conclusion (e.g., in the abstract and final paragraph)?

2. The training is barely described. The entire description is this: “a brain plasticity-based computer program designed to improve the speed and accuracy of auditory information processing and to engage neuromodulatory systems.” To learn more, readers are referred to a paper that is not easily available — in particular, I could not find it on the Posit Science website. Because the training is so briefly described, I was unable to judge how much the outcome tests differ from the training tasks. This made it impossible for me to judge how much the training generalizes to other tasks — which is the whole point. Why wasn’t the training better described?

3. What was the “ET [experimental treatment] processing speed exercise”? It sounds like a reaction-time task. People will get faster at any reaction-time task if given extensive practice on that task. How is such improvement relevant to daily life? If it is irrelevant, why is it given considerable attention (one of the paper’s four graphs)?

4. According to Table 2, the CSRQ (Cognitive Self-Report Questionnaire) questions showed no significant improvement in trainees’ perceptions of their own daily cognitive functioning, although the p value was close to 0.05. Given the large sample size (~500), this failure to find significant improvement suggests the self-report improvements were small or zero. Why wasn’t this discussed? Is the amount of improvement suggested by Posit Science’s marketing consistent with these results?

5. Is it possible that the improvement subjects experienced was due to the acquisition of strategies for dealing with rapidly presented auditory material, and especially for focusing on the literal words (rather than on their meaning, as may be the usual approach taken in daily life)? If so, is it possible that the skills being improved have little value in daily life, explaining the lack of effect on the CSRQ?

6. In the Methods section, you write “In the a priori data analysis plan for the IMPACT Study, it was hypothesized that the tests constituting the secondary outcome measure would be more sensitive than the RBANS given their larger raw score ranges and sensitivity to cognitive aging effects.” Do the initial post-training tests (measurements of the training effect soon after training ended) support this hypothesis? Why aren’t the initial post-training results described so that readers can see for themselves if this hypothesis is plausible? If you thought the “secondary outcome measure would be more sensitive than the RBANS” why wasn’t the secondary outcome measure the primary measure?

7. The primary outcome measure was some of the RBANS (Repeatable Battery for the Assessment of Neuropsychological Status). Did subjects take the whole RBANS or only part of it? If they took the whole RBANS, what were the results with the rest of the RBANS (the subtests not included in the primary outcome measure)?

8. The data analysis refers to a “secondary composite measure”. Why that particular composite and not any of the many other possible composite measures? Were other secondary composite measures considered? If so, were p values corrected for this?

9. If Test A resembles training more closely than Test B, Test A should show more effect of training (at any retention interval) than Test B. In this case Test A = the RBANS auditory subtests and Test B = the secondary composite measure.  In contrast to this prediction, you found that Test B showed a clearer training effect (in terms of p value) than Test A. Why wasn’t this anomaly discussed (beyond what was said in the Methods section)?

10. Were any tests given the subjects not described in this report? If there were other tests, why were their results not described?

11. The secondary composite measure is composed of several memory tests and called “Overall Memory”. The Posit Science website says their training will not only help you “remember more” but also “think faster” and “focus better”. Why weren’t tests of thinking speed (different from the training tasks) and focus included in the assessment?

12. Do the results support the idea that the training causes trainees to “focus better”?

13. The Posit Science homepage suggests that their training increases “intelligence”. Was intelligence measured in this study? If not, why not?

14. Do the results support the idea that the training causes trainees to become more intelligent?

15. The only test of thinking speed included in the assessment appears to be a reaction-time task that was part of the training. Are you saying that getting faster on one reaction-time task after lots of practice with that task shows that your training causes trainees to “think faster”?

Update: Henry Mahncke, the head of Posit Science, said that he would be happy to answer these questions by phone. I replied that I was sure many people were curious about the answers and written answers would be much easier to share.

Further update: Mahncke replied that he would prefer a phone call and that some of the questions seemed to him hard to answer in writing. He said nothing about the sharing problem. I repeated my belief that many people are interested in the answers and that a phone call would be hard to share. I offered to rewrite any questions that seemed hard to answer in writing.

Earlier questions for Posit Science.

• Has the impact of Helicobacter pylori therapy on ulcer recurrence in the United States been overstated? “20% of patients in these studies had ulcer recurrence within 6 months.”
• Over-sold flu vaccine. Carl Zimmer in the NY Times, however, says “The vaccines usually provide strong protection against the virus.”
• Incredibly fast mental addition.

Thanks to Paul Nash, Grace Liu and Anne Weiss.

Here at Tsinghua University, the Second Annual Chinese International Conference on Positive Psychology has just begun. The first speaker was Martin Seligman, a professor at the University of Pennsylvania and former president of the American Psychological Association (the main professional group of American psychologists). Seligman is more responsible for the Positive Psychology movement than anyone else. Here are some things I liked and disliked about his talk.

Likes:

1. Countries, such as England, have started to measure well-being in big frequent surveys (e.g., 2000 people every month) and some politicians, such as David Cameron, have vowed to increase well-being as measured by these surveys. This is a vast improvement over trying to increase how much money people make. The more common and popular and publicized this assessment becomes — this went unsaid — the more powerful psychologists will become, at the expense of economists. Seligman showed a measure of well-being for several European countries. Denmark was highest, Portugal lowest. His next slide showed the overall result of the same survey for China: 11.83%. However, by then I had forgotten the numerical scores on the preceding graph so I couldn’t say where this score put China.

2. Work by Angela Duckworth, another Penn professor, shows that “GRIT” (which means something like perseverance) is a much better predictor of school success than IQ. This work was mentioned in only one slide so I can’t elaborate. I had already heard about this work from Paul Tough in a talk about his new book.

3. Teaching school children something about positive psychology (it was unclear what) raised their grades a bit.

Dislikes:

1. Three years ago, Seligman got $125 million from the US Army to reduce suicides, depression, etc. (At the birth of the positive psychology movement, Seligman proclaimed that psychologists spent too much time studying suicide, depression, etc.) I don’t mind the grant. What bothered me was a slide used to illustrate the results of an experiment. I couldn’t understand it. The experiment seems to have had two groups. The results from each group appeared to be on different graphs (making comparison difficult, of course).

2. Why does a measure of well-being not include health? This wasn’t explained.

3. Seligman said that a person’s level of happiness was “genetically determined” and therefore was difficult or impossible to change. (He put his own happiness in “the bottom 50%”.) Good grief. I’ve blogged several times about how the fact that something is “genetically-determined” doesn’t mean it cannot be profoundly changed by the environment. Quite a misunderstanding by an APA president and Penn professor.

4. He mentioned a few studies that showed optimism (or lack of it) was a risk factor for heart disease after you adjust for the traditional risk factors (smoking, exercise, etc.). There is a whole school of “social epidemiology” that has shown the importance of stuff like where you are in the social hierarchy for heart disease. It’s at least 30 years old. Seligman appeared unaware of this. If you’re going to talk about heart disease epidemiology and claim to find new risk factors, at least know the basics.

5. Seligman said that China had “a good safety net.” People in China save a large fraction of their income at least partly because they are afraid of catastrophic medical costs. Poor people in China, when they get seriously sick, come to Beijing or Shanghai for treatment, perhaps because they don’t trust their local doctor (or the local doctor’s treatment failed). In Beijing or Shanghai, they are forced to pay enormous sums (e.g., half their life’s savings) for treatment. That’s the opposite of a good safety net.

6. Given the attention and resources and age of the Positive Psychology movement, the talk seemed short on new ways to make people better off. There was an experiment with school children where the main point appeared to be their grades improved a bit. A measure of how they treat each other also improved a bit. (Marilyn Watson, the wife of a Berkeley psychology professor, was doing a study about getting school kids to treat each other better long before the Positive Psychology movement.) There was an experiment with the U.S. Army I couldn’t understand. That’s it, in a 90-minute talk. At the beginning of his talk Seligman said he was going to tell us things “your grandmother didn’t know.” I can’t say he did that.

Tim Lundeen pointed me to the website of Posit Science, which sells ($10/month) access to a bunch of exercises that supposedly improve various brain functions, such as memory, attention, and navigation. I first encountered Posit Science at a booth at a convention for psychologists about five years ago. They had reprints available. I looked at a study published in the Proceedings of the National Academy of Sciences. I was surprised how weak was the evidence that their exercises helped.

Maybe the evidence has improved. Under the heading “world class science” the Posit Science website emphasizes a few of the 20-odd published studies. First on their list of “peer-reviewed research” is “the IMPACT study”, which has its own web page.

With 524 participants, the IMPACT study is the largest clinical trial ever to examine whether a specially designed, widely available cognitive training program significantly improves cognitive abilities in adults. Led by distinguished scientists from Mayo Clinic and the University of Southern California, the IMPACT study proves that people can make statistically significant gains in memory and processing speed if they do the right kind of scientifically designed cognitive exercises.

The study compared a few hundred people who got the Posit Science exercises with a few hundred people who got an “active control” treatment that is poorly described. It is called “computer-based learning”. I couldn’t care less that people who spend an enormous amount of time doing laboratory brain tests (1 hour/day, 5 days/week, 8-10 weeks) thereby do better on other laboratory brain tests. I wanted to know if the laboratory training produced improvement in everyday life. This is what most people want to know, I’m sure. The study designers seem to agree. The procedure description says “to be of real value to users, improvement on a training program must generalize to improvement on real-world activities”. (more…)