What a dead salmon can teach you about statistics

In 2009, Bennett, Braid, Miller & Wolford presented the study “Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument For Proper Multiple Comparisons Correction”, also known as the dead salmon study. In this study, Bennett et al. placed a dead salmon in a functional magnetic resonance imaging (fMRI) machine. Yes, you did read that right: PhD’s ran an fMRI on a dead fish. The subject of the study was described as follows:

One mature Atlantic Salmon (Salmo salar) participated in the fMRI study. The salmon was approximately 18 inches long, weighed 3.8 lbs, and was not alive at the time of scanning.  It is not known if the salmon was male or female, but given the post-mortem state of the subject this was not thought to be a critical variable.”

The experiment
The researchers placed the fish in the fMRI and showed pictures of a human in various social situations. Next, they asked the salmon what emotion the individual in the picture must have been experiencing. And guess what? After using statistical methods to analyse the collected data, their analysis showed evidence of brain activity. The results of the FMRI scan are shown in the image below.

deadsalmon

That is scary! A dead fish that thinks.. Or is it not really about the salmon? No it is not, it is about the statistical analysis. According to the researchers the precise point of the study was this:

“Can we conclude from this data that the salmon is engaging in the perspective-taking task? Certainly not. What we can determine is that random noise in the EPI timeseries may yield spurious results if multiple comparisons are not controlled for. Adaptive methods for controlling the FDR and FWER are excellent options and are widely available in all major fMRI analysis packages. We argue that relying on standard statistical thresholds (p < 0.001) and low minimum cluster sizes (k > 8) is an ineffective control for multiple comparisons. We further argue that the vast majority of fMRI studies should be utilizing multiple comparisons correction as standard practice in the computation of their statistics.”

Multi comparisons problem
Bennett et al. wanted to raise awareness for the “multiple comparisons problem”. If you do a lot of tests, at least some of them will come out positive. These are called false positives. There are several methods to correct the multiple comparisons, but as a result researchers lose statistical power. When they get rid of false positives, it might be that they do not see things that are really there and they might find false negatives instead. In the fMRI field, there is a debate going on whether false positives or false negatives are more threatening.

Opportunistic bias
According to DeCoster, Sparks, Sparks, Sparks & Sparks researchers commonly explore their data in multiple ways before deciding which analysis they will include in the final version of the paper. This can also be done with the false positives and false negatives. If researchers do not accurately solve the multiple comparisons problem, this can improve the chances of researcher’s finding publishable results. DeCoster et al. state that this introduces an “opportunistic bias”. In this, the reported relations are stronger or otherwise more supportive of the researcher’s theories than they would be without the exploratory process.

Decline effect
The opportunistic bias raises questions about the reliability of research(ers) and so does the decline effect. The decline effect may occur when scientific claims receive decreasing support over time. In the 2010 article of Jonah Lehrer, various examples of the occurrence of the decline effect are given. To continue in the medical profession, he also had an example about anti-psychotic drugs. The development of second generation anti-psychotic drugs, reveals that the first tests had a dramatic decrease in the subject’s psychiatric symptoms. However, repeated tests had a declined effect and at a certain point, it was not possible to report that these drugs had a better effect than the first generation anti-psychotics. The decline effect is troubling because it reminds the researchers of how difficult it is to prove anything. Possible explanations of the decline effect are “regression to the mean” and “publication bias”.

Importance of significant results 
For researchers there is a high pressure to find significant results. Fanelli states in her article that the growing competition and “publish or perish” culture in academia might conflict with the objectivity and integrity of research. It forces scientists to produce “publishable” results. Statistical tricks make it possible to reach significance but make the published results unreliable. Why are significant results this important to researchers? Nuijten shows this through the following exaggerated but clear example:

  • “You will only get published if you find a significant effect.”

  • “You will only get tenure if you find a significant effect.”

  • “You will only get a mortgage if you find a significant effect.”

Impact dead salmon study
In order to make this blog post a little more positive, I want to show you that the dead salmon study had great impact. In 2012 the researchers won the IG Noble prize for neuroscience, “for demonstrating that brain researchers, by using complicated instruments and simple statistics, can see meaningful brain activity anywhere – even in a dead salmon.” In their acceptance speech, they mentioned the following:

We found that up to 40% of papers were using an incorrect statistical approach. And many other people had argued that they should be doing it correctly, but it wasn’t sticking. So we decided: can we use the tools of humor and absurdity to change a scientific field? Nay, to put a dent in the universe. And the truth is that you can. Through a dead salmon, you can get the number of people who use the incorrect statistic under 10%. So we’ve had a real impact.”

Conclusion
To conclude, there are a lot of improvements possible with regard to the objectivity and integrity of research. But it is hard to reach this, because journals are unwilling to publish null results, which brings a certain pressure for scientist to find significant results. Although researchers, such as Greenwald and Walster and Cleary, provided several suggestions for both researchers and editors to change attitudes toward the null hypothesis, not much has changed and, according to Lee the publish or perish culture has increasingly caused anxiety and induced stress among scientists. In my opinion the publish or perish culture is a major threat in science and especially in the medical field. As mentioned before, there is a debate running in the fMRI field whether false positive or false negatives are more dangerous. The researchers of the dead salmon study argue that false positives are more likely to get overblown and lead to problems down the line and I agree with them. We should be really careful with medical research, in worst cases it could be about life or death.

Advertenties

5 gedachtes over “What a dead salmon can teach you about statistics

  1. Haha, I loved your title once again! Either false positives or false negatives, we are talking about results that are FALSE. And that’s enough! As you underline, in the medical field it is even more important to be accurate and true. But, to my point of view, it’s not a matter of field. Journalists and scientists should be careful with everything they write and analyse! The power they have in their hands, the power of being the eyes, ears and mouth of the people can result being negative once over-used.

    Like

  2. Hey, nice salmon example! I wrote this weeks blog on especially the publish or perish culture and this really affects the scientific world in a negative way.
    You said: “We should be really careful with medical research, in worst cases it could be about life or death.”
    I totally agree. In some cases you can give for example the terminally ill hope, even when there is none because the medicine actually doesn’t work. But I really hope that researchers don’t play with statistics and in this way “try to cure cancer” because that would be heartless.

    Like

  3. haha, really nice example and a good start of the blog. It engaged me very quickly, which is always a good sign. I think you picked a good example to prove your point “We should be really careful with medical research, in worst cases it could be about life or death”. It is very sad that actually nowadays we can not trust anybody. In the age of information there is big doubts whether the information is correct. I really hope that researchers will stop playing with data because it can not only damage their reputations but it can also change attitudes towards science and lead to the situation when science will no longer be perceived as a reliable source of knowledge.

    Like

  4. I am especially fond of the research article that you included in your article. It made the topic more easy to grasp and it helped with indicating it importance (it’s also nice that they had a big impact).

    I agree with your perspective that the publish or perish culture is major threat in science and that is is dangerous in the field of medicine. I also agree with your statements that journals should publish null results (I wrote something about that as well), but I do wonder how can we really change this culture? I think you have to approach it by different means; you cannot only change it by changing the researchers method design, but this could be a start.

    Like

  5. I liked reading your blog :). To my opinion, this part of your blog is the biggest issue:

    “For researchers there is a high pressure to find significant results. Fanelli states in her article that the growing competition and “publish or perish” culture in academia might conflict with the objectivity and integrity of research. It forces scientists to produce “publishable” results.”

    The pressure scientists feel to produce publishable results is really unhealthy. It is almost something from the business world that sneaked in to the academic world and that is something to worry about. I think it is hard to change it, but it certainly isn’t impossible to accomplish. In order to fix this, the academic world shoud apply pressure on researchers to be transparant, showing every bit of information and data to get an objective view on the research, and not a filtered one.

    Like

Geef een reactie

Vul je gegevens in of klik op een icoon om in te loggen.

WordPress.com logo

Je reageert onder je WordPress.com account. Log uit /  Bijwerken )

Google+ photo

Je reageert onder je Google+ account. Log uit /  Bijwerken )

Twitter-afbeelding

Je reageert onder je Twitter account. Log uit /  Bijwerken )

Facebook foto

Je reageert onder je Facebook account. Log uit /  Bijwerken )

Verbinden met %s