The researchers then ran a bunch of additional statistical tests on this same data (tests to include other factors, such as age, and then breaking the data down into specific regions and re-running the tests—a controversial statistical process known as p-hacking because it increases the likelihood of finding a statistically significant result by chance). Even after all this, the researchers found that the depression group and the healthy control group continued to have serotonin levels which were no different, except for one slight average difference in one brain region (the temporal cortex). Even in this area, the data shows an almost complete overlap between the two groups.

The researchers, as they reported their results, ignored this fact. Instead, they calculated the mean change in serotonin release scores for the 20 healthy controls and 16 depressed patients, and concluded that there was a slight “statistically significant” difference (p value = .041). Without the one outlier, this statistically significant finding would have vanished.

Study Biases

The study was published in Biological Psychiatry and was led by David Erritzoe at Imperial College, London. Several other researchers unaffiliated with the study quickly pointed out several flaws in the study, starting with the fact that it was quite tiny, which increases the likelihood that any finding is due to chance. In a Twitter thread, researcher Eiko Fried compared the study to a presidential poll. Would you trust a poll of 37 people (31 male) to estimate who is most likely to win a presidential election? There is a reason polls try to reach a quorum of several thousand and a representative sample of the whole population.

In addition, 14 of the 17 in the depression group—and 17 of the 20 in the health control group—were male. The inclusion of only three women in the depression group (though women are more likely to be diagnosed with depression), the inclusion of five people with Parkinson’s disease (which could create a different neurobiological effect), the inclusion of two people taking a drug for Parkinson’s (which could create larger brain chemistry changes), and the fact that antidepressants had been used by six people in the past (which could have led to long-lasting brain adaptations) are all significant confounding factors. It is also worth noting that study was paid for by Imanova Ltd (now Invicro), whose profit is dependent on demonstrating the success of imaging techniques, such as PET, for “drug and diagnostic development.” In an acknowledgment, the researchers express appreciation for the work of the company’s employees “for their excellent technical support.” It is unclear how much input the company had in developing the study, conducting the analysis, and writing the paper.

Bad science at its finest folks.