“Probably not random” is not good enough

In 1925 Ronald A. Fisher published “Statistical Methods for Research Workers” in which he described an approach for rejecting, or nullifying, a hypothesis by calculating how often it would be likely to produce an observed effect size (Fisher 1925). Over the last hundred years, this approach has been inappropriately mixed with other approaches, simplified, and misinterpreted to the point where calculations of p-values now do more harm than good.

see full manuscript for “Probably not random is not good enough” in progress

P-values are used to support the claim that “X is important for Y”. They don’t. The calculation that is being done is: “If we assume there is no relationship between X and Y, how often would we expect to see the observed result or a more extreme result”. The inconsistencies between the calculation and the claim are numerous and include:

  1. The zero-effect null hypothesis being tested and rejected is not plausible in experimental biology. Everything is connected and there is always a possibility of sampling bias. Rejecting the zero-effect null hypothesis does not advance knowledge.
  2. Rejecting a null hypothesis is only evidence for an alternative hypothesis if the prior probability of the alternative hypothesis is understood (inverse probability illusion and Bayesian statistics)
  3. The hypothesis testing is not being performed in a context that controls for false positives and false negatives (p-hacking).

The simplest solution is to replace p-values and significance testing with confidence intervals that estimate the relationships between groups. However, the wide acceptance and low bar of rejecting the zero-effect hypothesis puts researchers using meaningful statistics at a competitive disadvantage.

I am proposing that the biological community organize to ask all life science journals to implement policies that recognize that rejecting the zero-effect hypothesis is biologically uninteresting. My hope is that, in 2023, we will develop a policy statement on transitioning from p-values to confidence intervals in cell biology. In 2024 we will collect signatures for the policy statement. In 2025, we will submit a petition to life science journals to implement the policy.

If you would like to be involved in developing a policy statement for rejecting p-values, please contact me at jlmorgan@wustl.edu.