What a dead salmon can teach you about statistics

In 2009, Bennett, Braid, Miller & Wolford presented the study “Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument For Proper Multiple Comparisons Correction”, also known as the dead salmon study. In this study, Bennett et al. placed a dead salmon in a functional magnetic resonance imaging (fMRI) machine. Yes, you did read that right: PhD’s ran an fMRI on a dead fish. The subject of the study was described as follows:

“One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning.  It is not known if the salmon was male or female, but given the post-mortem state of the subject this was not thought to be a critical variable.”

The experiment
The researchers placed the fish in the fMRI and showed pictures of a human in various social situations. Next, they asked the salmon what emotion the individual in the picture must have been experiencing. And guess what? After using statistical methods to analyse the collected data, their analysis showed evidence of brain activity. The results of the FMRI scan are shown in the image below.

That is scary! A dead fish that thinks.. Or is it not really about the salmon? No it is not, it is about the statistical analysis. According to the researchers the precise point of the study was this:

“Can we conclude from this data that the salmon is engaging in the perspective-taking task? Certainly not. What we can determine is that random noise in the EPI timeseries may yield spurious results if multiple comparisons are not controlled for. Adaptive methods for controlling the FDR and FWER are excellent options and are widely available in all major fMRI analysis packages. We argue that relying on standard statistical thresholds (p < 0.001) and low minimum cluster sizes (k > 8) is an ineffective control for multiple comparisons. We further argue that the vast majority of fMRI studies should be utilizing multiple comparisons correction as standard practice in the computation of their statistics.”

Multi comparisons problem
Bennett et al. wanted to raise awareness for the “multiple comparisons problem”. If you do a lot of tests, at least some of them will come out positive. These are called false positives. There are several methods to correct the multiple comparisons, but as a result researchers lose statistical power. When they get rid of false positives, it might be that they do not see things that are really there and they might find false negatives instead. In the fMRI field, there is a debate going on whether false positives or false negatives are more threatening.

Opportunistic bias
According to DeCoster, Sparks, Sparks, Sparks & Sparks researchers commonly explore their data in multiple ways before deciding which analysis they will include in the final version of the paper. This can also be done with the false positives and false negatives. If researchers do not accurately solve the multiple comparisons problem, this can improve the chances of researcher’s finding publishable results. DeCoster et al. state that this introduces an “opportunistic bias”. In this, the reported relations are stronger or otherwise more supportive of the researcher’s theories than they would be without the exploratory process.

Decline effect
The opportunistic bias raises questions about the reliability of research(ers) and so does the decline effect. The decline effect may occur when scientific claims receive decreasing support over time. In the 2010 article of Jonah Lehrer, various examples of the occurrence of the decline effect are given. To continue in the medical profession, he also had an example about anti-psychotic drugs. The development of second generation anti-psychotic drugs, reveals that the first tests had a dramatic decrease in the subject’s psychiatric symptoms. However, repeated tests had a declined effect and at a certain point, it was not possible to report that these drugs had a better effect than the first generation anti-psychotics. The decline effect is troubling because it reminds the researchers of how difficult it is to prove anything. Possible explanations of the decline effect are “regression to the mean” and “publication bias”.

Importance of significant results 
For researchers there is a high pressure to find significant results. Fanelli states in her article that the growing competition and “publish or perish” culture in academia might conflict with the objectivity and integrity of research. It forces scientists to produce “publishable” results. Statistical tricks make it possible to reach significance but make the published results unreliable. Why are significant results this important to researchers? Nuijten shows this through the following exaggerated but clear example:

• “You will only get published if you find a significant effect.”

↓

• “You will only get tenure if you find a significant effect.”

↓

• “You will only get a mortgage if you find a significant effect.”

Impact dead salmon study
In order to make this blog post a little more positive, I want to show you that the dead salmon study had great impact. In 2012 the researchers won the IG Noble prize for neuroscience, “for demonstrating that brain researchers, by using complicated instruments and simple statistics, can see meaningful brain activity anywhere – even in a dead salmon.” In their acceptance speech, they mentioned the following:

“We found that up to 40% of papers were using an incorrect statistical approach. And many other people had argued that they should be doing it correctly, but it wasn’t sticking. So we decided: can we use the tools of humor and absurdity to change a scientific field? Nay, to put a dent in the universe. And the truth is that you can. Through a dead salmon, you can get the number of people who use the incorrect statistic under 10%. So we’ve had a real impact.”

Conclusion
To conclude, there are a lot of improvements possible with regard to the objectivity and integrity of research. But it is hard to reach this, because journals are unwilling to publish null results, which brings a certain pressure for scientist to find significant results. Although researchers, such as Greenwald and Walster and Cleary, provided several suggestions for both researchers and editors to change attitudes toward the null hypothesis, not much has changed and, according to Lee the publish or perish culture has increasingly caused anxiety and induced stress among scientists. In my opinion the publish or perish culture is a major threat in science and especially in the medical field. As mentioned before, there is a debate running in the fMRI field whether false positive or false negatives are more dangerous. The researchers of the dead salmon study argue that false positives are more likely to get overblown and lead to problems down the line and I agree with them. We should be really careful with medical research, in worst cases it could be about life or death.