Severe ME Day of Understanding and Remembrance: Aug. 8, 2017
Determined to paper the Internet with articles about ME, Jody Smith brings some additional focus to Severe Myalgic Encephalomyelitis Day of Understanding and Remembrance on Aug. 8, 2017 ...
Discuss the article on the Forums.

Lancet editorial: "poor methods get results", lack of action from research community

Discussion in 'Other Health News and Research' started by Simon, Apr 18, 2015.

  1. Simon

    Simon

    Messages:
    1,927
    Likes:
    14,578
    Monmouth, UK

    Offline: What is medicine's 5 sigma? - The Lancet


    A recent editorial from The Lancet, reporting on a meeting of the big scientific cheeses including the Wellcome Trust, the MRC and journal editors about the crisis in life science research standards.

    The piece says on of the best contributions came from Tony Weidberg, a professor in particle physics who said the particle physics community responded to a similar crisis over quality in reproducibility in the past by focusing on research standards. In particular, they introduced the '5 sigma' test for significance, which translates to a p value of <=0.0000003, as opposed to the p<0.05 used regularly in life science. That's a lot harder to achieve. [this may not be right for life science, but it does give a feel for the lengths they went to in particle physics]

    Read the full text
     
    Last edited: Apr 18, 2015
    wdb, alex3619, SOC and 5 others like this.
  2. Sidereal

    Sidereal Senior Member

    Messages:
    3,097
    Likes:
    17,189
    Such a thing will never happen in medicine because medicine isn't physics and such p values cannot be achieved. If journals introduced such a ludicrously high bar for statistical significance there would never be another medical treatment approved again. Mr Horton's rag would better serve the medical community if they stopped publishing extremely implausible, novel, never-to-be-replicated-again studies and instead started focussing on solid albeit not very sexy science.
     
    cigana, adreno, Sean and 4 others like this.
  3. Jonathan Edwards

    Jonathan Edwards "Gibberish"

    Messages:
    5,250
    Likes:
    32,015
    Indeed. The statement 'Those who have the power to act seem to think somebody else should act first.' might well apply to the author. Mr Horton could get the ball rolling by politely bowing out and letting someone more astute take over.

    These people only have themselves to blame. They are the great and good who referee (and indeed write) the papers. They are the ones who insist that promotion should be based on citation scores. But at least they seem to realise there is a problem. Jo Cambridge and I have been saying most of immunology is rubbish for years and getting funny looks. Maybe the penny has dropped.
     
    wdb, beaker, aimossy and 8 others like this.
  4. Marco

    Marco Grrrrrrr!

    Messages:
    2,379
    Likes:
    3,193
    Near Cognac, France
    I always thought that 5 (6?) Sigma was a quality standard applied to manufacturing and other processes promoted by management consultancy types and usually involving continuous tedious form filling.

    I'm at a loss as to how this would or should apply to the scientific method. Apart from anything else particle physicists tend not to be short of particles to study compared to patients. By all means tighten up significance levels to p=0.01 but '5 Sigma'?

    There's a major difference between the number of faults per million of widgets produced and determing whether or not a trial result is due to chance. In the latter more depends on the trial design and analysis than on confidence levels.
     
    Sean and Valentijn like this.
  5. Jonathan Edwards

    Jonathan Edwards "Gibberish"

    Messages:
    5,250
    Likes:
    32,015
    Quite so. Maybe they had to have Chatham House Rules so that nobody could recognise which dunderheads were in charge of this rather primitive naval gazing - and could be dispensed with. Maybe we should suggest that all papers should be submitted to Phoenix Rising before publication - that should sort a few things out.
     
    wdb, beaker, aimossy and 11 others like this.
  6. worldbackwards

    worldbackwards A unique snowflake

    Messages:
    2,091
    Likes:
    10,358
    Earth
    *Prof. White wakes up screaming*
     
    beaker, Cheshire, Sean and 6 others like this.
  7. Simon

    Simon

    Messages:
    1,927
    Likes:
    14,578
    Monmouth, UK
    Yes, it is used in manufacturing, but also in the discovery of the Higgs Bosun so qualifies pretty-comfortably as the scientific method too.
    I agree 5 sigma is too extreme.
    Apart from anything else, as the extreme p value reduces the chance of false positive, it increases the chance of false negatives, ie of missing real effects. P values alone won't solve the problems in life science but tightening them up would help, maybe to 0.01 as you suggest or even 0.005.

    The issue I have with the current standard p<0.05 is that it's way too easy to hit, especially as nobody ever seems to correct for multiple comparisons [what are the peer reviewers doing?]. (With a 20-item questionnaire, there is 64% chance of one item falling below p=0.05 just by chance; it's 92% chance of a false positive on a 50-cytokine panel, though check out the impressive p values on the recent Hornig paper). Add in sloppy design and researcher bias (eg iffy analyses) and it's way too easy to get a result for publication.

    Replication is another handy tool for checking the result is real, without resorting to extreme p values (false positives are unlikely to replicate). There's no panacea fix but I do think p values have a part to play.
     
    SOC, wdb, Valentijn and 5 others like this.
  8. worldbackwards

    worldbackwards A unique snowflake

    Messages:
    2,091
    Likes:
    10,358
    Earth
    To be fair, we have got the patients' armed insurgence to organise. Those polite but firm Twitter responses don't write themselves, y'know.:)
     
  9. Snow Leopard

    Snow Leopard Hibernating

    Messages:
    4,616
    Likes:
    12,462
    South Australia
    It's all divination. Choosing 5 sigma arbitrarily is just as irrational a 0.05.
     
    Valentijn likes this.
  10. Dolphin

    Dolphin Senior Member

    Messages:
    10,708
    Likes:
    28,266
    The heading is "What is medicine's 5 sigma?" I didn't get the impression that they were looking for the same thing in medicine and biomedical research, just something to tighten up/improve standards.

    Sample:
     
    Valentijn likes this.

See more popular forum discussions.

Share This Page