One reason personal science is a good idea is it is simple and immediate (in the sense of near). You study one person, you do experiments (easier to interpret than surveys), you can easily repeat the experiment (so you are not confused by secular trends — big changes over time — and implausible statistical assumptions), you are aware of unusual events during the experiment (so you are less confused by anomalous results and outliers), you are close to the data collection (so you understand the limits and error rates of the measurements). These elements make good interpretation of your data much easier. Professional science generally lacks some of these elements. For example, the person who writes the paper may not have collected the data. This makes it harder to understand what the data mean.
I hear criticism of (professional) science more now than ten years ago. Lack of replicability, for example. What I rarely hear — actually, never — is how often science critics make big blunders. As far as I can tell, as often as those they criticize. This is not to say they are wrong — who knows. Just overstated.
An example is a critique of salt and blood pressure studies I read recently. Many people say salt raises blood pressure. The critique, by Michael Alderman, a professor of epidemiology at Albert Einstein College of Medicine, said, not so fast. The title is: “Salt, blood pressure and health: a cautionary tale.” It’s a good review, with lots of interesting data, but the reviewer, at the same time he is criticizing others, makes a major blunder.
He describes a study in which people were placed on a low-salt diet. Their blood pressure was measured twice, before the diet (Time 1) and after they had been on the diet for quite a while (Time 2). Comparison of the two readings showed a wide range of changes. Some people’s blood pressure went up, some people’s blood pressure stayed the same, and some people’s blood pressure went down. Alderman called this result “enormous variation between individuals on the effect of salt on pressure”. Oh no! He assumes that if your blood pressure is different at Time 2 than Time 1, it was because of the change in dietary salt. There are dozens of possible reasons a person’s blood pressure might differ at the two times (leaving aside measurement error, another possibility). Dozens of things that affect blood pressure were not kept constant.
Had there been a second group that did not change their diet and was also measured at Time 1 and Time 2 — and had the subjects given the low-salt diet showed a larger spread of Time 2/Time 1 difference scores than the no-change group, then you could reasonably conclude that there was variation in the response to the low-salt diet. To conclude “enormous variation” you’d want to see an enormous increase in difference-score variability. But there was no second group.
This is not some small detail. Alderman actually believes there is great variation in response to salt reduction. It is the main point of his article. Spy magazine had a great column called Review of Reviewers. Such as book and movie reviewers. Unfortunately there is no such thing in science.