Psychologists are facing up to problems with replication.

[wdb]

Interesting article about psychological research
http://www.nature.com/news/replication-studies-bad-copy-1.10634

Pervasive bias
Psychology is not alone in facing these problems. In a now-famous paper2, John Ioannidis, an epidemiologist currently at Stanford School of Medicine in California argued that “most published research findings are false”, according to statistical logic. In a survey of 4,600 studies from across the sciences, Daniele Fanelli, a social scientist at the University of Edinburgh, UK, found that the proportion of positive results rose by more than 22% between 1990 and 2007 (ref. 3). Psychology and psychiatry, according to other work by Fanelli4, are the worst offenders: they are five times more likely to report a positive result than are the space sciences, which are at the other end of the spectrum (see 'Accentuate the positive'). The situation is not improving. In 1959, statistician Theodore Sterling found that 97% of the studies in four major psychology journals had reported statistically significant positive results5. When he repeated the analysis in 1995, nothing had changed6.

One reason for the excess in positive results for psychology is an emphasis on “slightly freak-show-ish” results, says Chris Chambers, an experimental psychologist at Cardiff University, UK. “High-impact journals often regard psychology as a sort of parlour-trick area,” he says. Results need to be exciting, eye-catching, even implausible. Simmons says that the blame lies partly in the review process. “When we review papers, we're often making authors prove that their findings are novel or interesting,” he says. “We're not often making them prove that their findings are true.”

Simmons should know. He recently published a tongue-in-cheek paper in Psychological Science 'showing' that listening to the song When I'm Sixty-four by the Beatles can actually reduce a listener's age by 1.5 years7. Simmons designed the experiments to show how “unacceptably easy” it can be to find statistically significant results to support a hypothesis. Many psychologists make on-the-fly decisions about key aspects of their studies, including how many volunteers to recruit, which variables to measure and how to analyse the results. These choices could be innocently made, but they give researchers the freedom to torture experiments and data until they produce positive results.

 

[Sean]

Many psychologists make on-the-fly decisions about key aspects of their studies, including how many volunteers to recruit, which variables to measure and how to analyse the results. These choices could be innocently made, but they give researchers the freedom to torture experiments and data until they produce positive results.

Sounds familiar. Except maybe not so innocent.

 

[Esther12]

I've not read the whole thing, but I want too... I can re-use a reply I just made to one of Dolphin's threads:

Thanks. Too much interesting stuff to read at the mo.

It does seem like exactly the sorts of problems we've complained about in CFS, and had dismissed as a reflection of patients failing to understand how mind and body interact, or a stigmatisation of mental health issues, are now beginning to be recognised as a serious problem within much of psychology. They just couldn't believe it when it came from CFS patients.

 

[Sean]

They just couldn't believe it when it came from CFS patients.

The cognitive dissonance generated by acknowledging that we were right would rent the psychiatric heavens asunder. Gods would be falling all over the place.

Can't have that.