Science Critics Are Human: Cautionary Tale

One reason personal science is a good idea is it is simple and immediate (in the sense of near). You study one person, you do experiments (easier to interpret than surveys), you can easily repeat the experiment (so you are not confused by secular trends — big changes over time — and implausible statistical assumptions), you are aware of unusual events during the experiment (so you are less confused by anomalous results and outliers), you are close to the data collection (so you understand the limits and error rates of the measurements). These elements make good interpretation of your data much easier. Professional science generally lacks some of these elements. For example, the person who writes the paper may not have collected the data. This makes it harder to understand what the data mean.

I hear criticism of (professional) science more now than ten years ago. Lack of replicability, for example. What I rarely hear — actually, never — is how often science critics make big blunders. As far as I can tell, as often as those they criticize. This is not to say they are wrong — who knows. Just overstated.
An example is a critique of salt and blood pressure studies I read recently. Many people say salt raises blood pressure. The critique, by Michael Alderman, a professor of epidemiology at Albert Einstein College of Medicine, said, not so fast. The title is: “Salt, blood pressure and health: a cautionary tale.” It’s a good review, with lots of interesting data, but the reviewer, at the same time he is criticizing others, makes a major blunder.

He describes a study in which people were placed on a low-salt diet. Their blood pressure was measured twice, before the diet (Time 1) and after they had been on the diet for quite a while (Time 2). Comparison of the two readings showed a wide range of changes. Some people’s blood pressure went up, some people’s blood pressure stayed the same, and some people’s blood pressure went down. Alderman called this result “enormous variation between individuals on the effect of salt on pressure”. Oh no! He assumes that if your blood pressure is different at Time 2 than Time 1, it was because of the change in dietary salt. There are dozens of possible reasons a person’s blood pressure might differ at the two times (leaving aside measurement error, another possibility). Dozens of things that affect blood pressure were not kept constant.

Had there been a second group that did not change their diet and was also measured at Time 1 and Time 2 — and had the subjects given the low-salt diet showed a larger spread of Time 2/Time 1 difference scores than the no-change group, then you could reasonably conclude that there was variation in the response to the low-salt diet. To conclude “enormous variation” you’d want to see an enormous increase in difference-score variability. But there was no second group.

This is not some small detail. Alderman actually believes there is great variation in response to salt reduction. It is the main point of his article. Spy magazine had a great column called Review of Reviewers. Such as book and movie reviewers. Unfortunately there is no such thing in science.

6 Responses to “Science Critics Are Human: Cautionary Tale”

  1. George Says:

    Right, so he commits a very basic and fundamental purely logical error that anyone with a good training in philosophy or science or who has taken the trouble to consciously develop good intellectual habits would easily avoid.

    What strikes me is

    1) How relatively basic the error is yet serves as the basis for his entire position.

    2) How utterly characteristic this kind of thing is across the nutrition field even with researchers and writers on nutrition.

    The entire field needs to be massively cleaned up and high intellectual standards imposed. Will this happen? The problem here seems to be that nutrition and diet have some kind of connection with our need for myth and the search for human meaning in a way that a more abstract science like physics simply doesn’t have. The result is that nutrition and diet frequently stimulate the emotions and the myth-making faculty in ways that physics does not. This emotion doesn’t just distort thinking but seems

  2. B D McCullough Says:

    Reason #1,113,412 to remember to ask: “Where’s the control group?”

  3. Nancy Lebovitz Says:

    Off-topic: I just ran across this, and it seems like the kind of thing you and/or your readers might want to look into. I drink very little, so I’m won’t be helping with the research.

    “A large social group that I am involved in, which has been known to drink heavily, has started taking N-acetyl-cysteine (500-1000mg) and Source Naturals Hangover Formula (which is primarily a C & B complex), and the effect it has on hangovers is not in any way subtle.”

  4. Gina Says:

    Nancy Lebovitz,

    I drink a lot, and have used this combination for years. It works! It is a bit of a pain, because you need to pop an NAC, C and B1 with almost every drink, but it’s worth it. I bring my pill bottles to the bar and hand them out to friends.

    Another thing that I think all drinkers should have on hand is Evening Primrose Oil. It prevents liver damage and tolerance.

  5. kxmoore Says:

    Gina i just started brewing beer. I wonder how adding the anti hangover formula directly to the brew would work. Gonna experiment with my next brew. The question is whether the formula will compromise the flavor of the beer and if the beer affects the formula unfavorably.

  6. John Smith Says:

    Seth, here is my take on ‘science.’

    It is a new word brought about by the need to quit using the word ‘truth.’
    The priests of our religions just about murdered the word truth, to the point where it was no longer viable. So the ‘scientific method’ was designed to differentiate between ‘truths’ as the establishments (religions?) issued them, and workable facts that were in keeping with current understanding.

    But what we really did at that point was establish a new religion. We created a body of thought more up to date, yes, and far more practical, workable and predictable. It followed that it also made more and better results, which are expressed as money.

    Yet we only changed what we believed to be true. Never did the scientific method guarantee that the conclusions drawn from the data would be correct or long lasting. New science contradicts old science every day. The science of today will be obsolete tomorrow, so to speak, at least in many instances.

    Once we as individuals ‘believe’ something to be true, it becomes truth to us. Unfortunately we are usually loathe to back away from that position, and when the masses believe untruths, the ‘church’ becomes the haunt of dogma.

    Individualized science is the only good solution. If I believe it, it is true for me. I can learn better tomorrow, maybe, but for today it is true. Others may find my truth nothing more than lies. Bully for them……….. as I see their truth as lies all too often. But it is only truth if the individual believes it really and truly…………. like deep down, man. And if I am not prepared to act on that belief, probably I don’t really believe it.

    Unfortunately the ‘science’ of today has morphed into a religion, duly presided over by High Priests who maintain yesterday’s truth for the protection of the power and influence of established methods and organizations.

    The www is doing to ‘science’ and the power bases it has spawned what Christianity did to the Roman Empire and the printing press did to the Catholic Church (which the Roman Empire morphed into).

    Individualism has taken a back seat to collectivism in the last hundred years, but the www has ushered in a whole new epoch.

    Seth: I agree. The habits of mind, the built-in tendencies, that led to religion didn’t change just because science was invented. I think of personal science as similar to literacy, and mass literacy rested on the printing press.