• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Lancet editorial: "poor methods get results", lack of action from research community

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK

Offline: What is medicine's 5 sigma? - The Lancet


“A lot of what is published is incorrect.” I'm not allowed to say who made this remark because we were asked to observe Chatham House rules....
A recent editorial from The Lancet, reporting on a meeting of the big scientific cheeses including the Wellcome Trust, the MRC and journal editors about the crisis in life science research standards.

The piece says on of the best contributions came from Tony Weidberg, a professor in particle physics who said the particle physics community responded to a similar crisis over quality in reproducibility in the past by focusing on research standards. In particular, they introduced the '5 sigma' test for significance, which translates to a p value of <=0.0000003, as opposed to the p<0.05 used regularly in life science. That's a lot harder to achieve. [this may not be right for life science, but it does give a feel for the lengths they went to in particle physics]

The conclusion of the symposium was that something must be done... But as to precisely what to do or how to do it, there were no firm answers. Those who have the power to act seem to think somebody else should act first. And every positive action (eg, funding well-powered replications) has a counterargument (science will become less creative). The good news is that science is beginning to take some of its worst failings very seriously. The bad news is that nobody is ready to take the first step to clean up the system.

Read the full text
 
Last edited:

Sidereal

Senior Member
Messages
4,856
In particular, they introduced the '5 sigma' test for significance, which translates to a p value of <=0.0000003, as opposed to the p<0.05 used regularly in life science. That's a lot harder to achieve. [this may not be right for life science, but it does give a feel for the lengths they went to in particle physics]

Such a thing will never happen in medicine because medicine isn't physics and such p values cannot be achieved. If journals introduced such a ludicrously high bar for statistical significance there would never be another medical treatment approved again. Mr Horton's rag would better serve the medical community if they stopped publishing extremely implausible, novel, never-to-be-replicated-again studies and instead started focussing on solid albeit not very sexy science.
 

Jonathan Edwards

"Gibberish"
Messages
5,256
Indeed. The statement 'Those who have the power to act seem to think somebody else should act first.' might well apply to the author. Mr Horton could get the ball rolling by politely bowing out and letting someone more astute take over.

These people only have themselves to blame. They are the great and good who referee (and indeed write) the papers. They are the ones who insist that promotion should be based on citation scores. But at least they seem to realise there is a problem. Jo Cambridge and I have been saying most of immunology is rubbish for years and getting funny looks. Maybe the penny has dropped.
 

Marco

Grrrrrrr!
Messages
2,386
Location
Near Cognac, France
I always thought that 5 (6?) Sigma was a quality standard applied to manufacturing and other processes promoted by management consultancy types and usually involving continuous tedious form filling.

I'm at a loss as to how this would or should apply to the scientific method. Apart from anything else particle physicists tend not to be short of particles to study compared to patients. By all means tighten up significance levels to p=0.01 but '5 Sigma'?

There's a major difference between the number of faults per million of widgets produced and determing whether or not a trial result is due to chance. In the latter more depends on the trial design and analysis than on confidence levels.
 

Jonathan Edwards

"Gibberish"
Messages
5,256
I always thought that 5 (6?) Sigma was a quality standard applied to manufacturing and other processes promoted by management consultancy types and usually involving continuous tedious form filling.

I'm at a loss as to how this would or should apply to the scientific method. Apart from anything else particle physicists tend not to be short of particles to study compared to patients. By all means tighten up significance levels to p=0.01 but '5 Sigma'?

There's a major difference between the number of faults per million of widgets produced and determing whether or not a trial result is due to chance. In the latter more depends on the trial design and analysis than on confidence levels.

Quite so. Maybe they had to have Chatham House Rules so that nobody could recognise which dunderheads were in charge of this rather primitive naval gazing - and could be dispensed with. Maybe we should suggest that all papers should be submitted to Phoenix Rising before publication - that should sort a few things out.
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
I always thought that 5 (6?) Sigma was a quality standard applied to manufacturing and other processes promoted by management consultancy types and usually involving continuous tedious form filling.

I'm at a loss as to how this would or should apply to the scientific method.
Yes, it is used in manufacturing, but also in the discovery of the Higgs Bosun so qualifies pretty-comfortably as the scientific method too.
By all means tighten up significance levels to p=0.01 but '5 Sigma'?
I agree 5 sigma is too extreme.
[this may not be right for life science, but it does give a feel for the lengths they went to in particle physics]
Apart from anything else, as the extreme p value reduces the chance of false positive, it increases the chance of false negatives, ie of missing real effects. P values alone won't solve the problems in life science but tightening them up would help, maybe to 0.01 as you suggest or even 0.005.

The issue I have with the current standard p<0.05 is that it's way too easy to hit, especially as nobody ever seems to correct for multiple comparisons [what are the peer reviewers doing?]. (With a 20-item questionnaire, there is 64% chance of one item falling below p=0.05 just by chance; it's 92% chance of a false positive on a 50-cytokine panel, though check out the impressive p values on the recent Hornig paper). Add in sloppy design and researcher bias (eg iffy analyses) and it's way too easy to get a result for publication.

Replication is another handy tool for checking the result is real, without resorting to extreme p values (false positives are unlikely to replicate). There's no panacea fix but I do think p values have a part to play.
 

worldbackwards

Senior Member
Messages
2,051
Maybe we should suggest that all papers should be submitted to Phoenix Rising before publication - that should sort a few things out.
To be fair, we have got the patients' armed insurgence to organise. Those polite but firm Twitter responses don't write themselves, y'know.:)
 

Dolphin

Senior Member
Messages
17,567
The heading is "What is medicine's 5 sigma?" I didn't get the impression that they were looking for the same thing in medicine and biomedical research, just something to tighten up/improve standards.

Sample:
Can bad scientific practices be fixed? Part of the problem is that no-one is incentivised to be right. Instead, scientists are incentivised to be productive and innovative. Would a Hippocratic Oath for science help? Certainly don't add more layers of research red-tape. Instead of changing incentives, perhaps one could remove incentives altogether. Or insist on replicability statements in grant applications and research papers. Or emphasise collaboration, not competition. Or insist on preregistration of protocols. Or reward better pre and post publication peer review. Or improve research training and mentorship.