Many Psychology Papers Fail Replication Test.

[barbc56]

My question is how do these results compare with studies in the physical sciences.

I'm not sure how Ioannidis's conclusions fit with this study. In other words, is Ioannides speaking only about replication or faulty study design? Both, neither or somewhere in between and is one determined by the other?

Does Ioannidis address the different rates of false conclusions as determined by the type of science?

CONCLUSION

No single indicator sufficiently describes replication success, and the five indicators examined here are not the only ways to evaluate reproducibility. Nonetheless, collectively these results offer a clear conclusion: A large portion of replications produced weaker evidence for the original findings despite using materials provided by the original authors, review in advance for methodological fidelity, and high statistical power to detect the original effect sizes

My bold.

http://m.sciencemag.org/content/349/6251/aac4716

 

[Woolie]

@barbc56, thanks for sharing that with us. An earlier paper by Ioannidis, Why most published research findings are false, is super famous. If you click on the title, there's a link.

I don't know much about the physical sciences, but my guess is that failure to replicate grows exponentially with the complexity of the subject matter, because there's just a whole lot more measurement noise. So yea, the problem could be more acute in the behavioural sciences.

Maybe outright fraud is more common in the physical/biomed sciences, but this isn't what Ioannidis is talking about. Its mainly honest work done by people who use good designs and take care. Its just that the research culture is set up to reward new, significant findings, it doesn't sufficiently support attempts to replicate, and you need that to ensure an effect is really reliable. Plus, papers that get significant results are more likely to get published, so you don't know how many people failed to find the effect, that information is simply hidden from view. So what's out there in print isn't the full story. Its the tip of the iceberg, as it were.

Ioannidis also talks about researcher interest being part of the problem (for example, researchers being invested in the idea being demonstrated, either ideologically, financially, or whatever). But he notes that this can occur across a wide range of disciplines- so for example, biomedical scientists can be just as wedded to their (wrong) ideas as behavioural scientists, and are probably more likely to have a financial interest in them as well.