Over the years Major League Baseball has
tweaked the dimensions of the field, specifically the distance and height of the pitcher’s mound and the area of the strike zone. They did this in order to adjust the balance between pitchers and hitters, mostly to shift the balance toward hitters to make games more exciting for the fans.
Scientists are debating similar tweaks to statistical significance, to adjust the balance between false positives and false negatives. As with pitchers and batters, some changes are a zero-sum game – if you lower false positives, you increase false negatives, and vice versa. Where the perfect balance lies is a complicated question and the increasing subject of debate.
A recent paper (
available in preprint) by a long list of authors, including some heavy hitters like
John P.A. Ioannidis, suggests that the p-value that is typically used for the threshold of statistical significance, be changes in the psychology and biomedical fields from 0.05 to 0.005. They write:
For fields where the threshold for defining statistical significance for new discoveries is P < 0.05, we propose a change to P < 0.005. This simple step would immediately improve the reproducibility of scientific research in many fields. Results that would currently be called “significant” but do not meet the new threshold should instead be called “suggestive.” While statisticians have known the relative weakness of using P≈0.05 as a threshold for discovery and the proposal to lower it to 0.005 is not new, a critical mass of researchers now endorse this change.
