• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

0.05 or 0.005? P-value Wars Continue

Got to admit I didn't read this, other than the first couple of paragraphs, but thought it might interest others here.

Over the years Major League Baseball has tweaked the dimensions of the field, specifically the distance and height of the pitcher’s mound and the area of the strike zone. They did this in order to adjust the balance between pitchers and hitters, mostly to shift the balance toward hitters to make games more exciting for the fans.

Scientists are debating similar tweaks to statistical significance, to adjust the balance between false positives and false negatives. As with pitchers and batters, some changes are a zero-sum game – if you lower false positives, you increase false negatives, and vice versa. Where the perfect balance lies is a complicated question and the increasing subject of debate.

A recent paper (available in preprint) by a long list of authors, including some heavy hitters like John P.A. Ioannidis, suggests that the p-value that is typically used for the threshold of statistical significance, be changes in the psychology and biomedical fields from 0.05 to 0.005. They write:

For fields where the threshold for defining statistical significance for new discoveries is P < 0.05, we propose a change to P < 0.005. This simple step would immediately improve the reproducibility of scientific research in many fields. Results that would currently be called “significant” but do not meet the new threshold should instead be called “suggestive.” While statisticians have known the relative weakness of using P≈0.05 as a threshold for discovery and the proposal to lower it to 0.005 is not new, a critical mass of researchers now endorse this change.
https://sciencebasedmedicine.org/0-05-or-0-005-p-value-wars-continue/
 

Forbin

Senior Member
Messages
966
Well, as an exercise, in the Stanford study the following met the p < 0.005 threshold:

FDR-adjusted p-values:

ME.CFS patients vs. Controls: Resistin, TGF-β [Both p-values were actually 0.0052, or 1 in 192 vs. 1 in 200]

Unadjusted p-values:

ME/CFS patients vs. Controls: Resistin, TGF-β

Mild vs. Controls: Leptin, Resistin
Moderate vs. Controls: ICAM1
Severe vs. Controls: IL-7, IL-13, NGF, Resistin

Linear Trend Analysis: CCL11, CXCL10, G-CSF, GM-CSF, IFN-γ, IL-4, IL-5, IL-7, IL-12p70, IL-13, IL-17F, Leptin, LIF, NGF
Non-Linear Trend Analysis: ICAM1, Resistin
 
Last edited: