Researchers overturn landmark study on the replicability of psychological science

[Sean]

Just gets more and more confusing, doesn't it.

https://www.sciencedaily.com/releases/2016/03/160303145733.htm

An in-depth examination of a 2015 landmark study showing that more than half of all psychology studies cannot be replicated has revealed serious mistakes that make its pessimistic conclusion completely unwarranted.

 

[Chrisb]

I am sure that any possible conflicts of interest will have been declared.

 

[TiredSam]

And a reply here:

http://andrewgelman.com/2016/03/05/29195/

 

[A.B.]

The funny thing is that replicability doesn't even mean the results are true. It may just mean that the same biases leading to the false appearance of a result are present.

I recall one study where a placebo was as good as asthma medication if one only looked at subjective measures. Only looking at objective measures revealed that the placebo actually made no difference. Studies of psychological interventions rarely look at objective measures. When they do, they may simply ignore inconvenient null results on objective measures (PACE being a good example)

Yes, we cannot compare psychotherapy to placebo, but that is no excuse for not even trying to minimize bias, because there are other ways as well.

It's a junk science factory.

 

[Jonathan Edwards]

I think these harvard boys and girls should be promoted to milk monitors and given a badge for doing what Sir likes. Of course the original paper was totally subjective and hyped - but that would make up for all the fiddled replications wouldn't it.

 

[Gijs]

Where is the psyche? In what place? Can someone show me on mri? The psyche does not exist objectively. Conclusion: psychological illnesses are nonsense. We are our brains.

 

[Snow Leopard]

When we compared the replication rates of the endorsed and unendorsed protocols, we discovered that the endorsed protocols were nearly four times as likely to produce a successful replication (59.7%) as were the unendorsed protocols (15.4%).

This strongly suggests that the infidelities did not just introduce random error but instead biased the replication studies toward failure. If OSC had limited their analyses to endorsed studies, they would have found that 59.7% [95% confidence interval (CI): 47.5%, 70.9%] were replicated successfully.

In fact, we estimate that if all the replication studies had been high enough in fidelity to earn the endorsement of the original authors, then the rate of successful replication would have been 58.6% (95% CI: 47.0%, 69.5%) when controlling for relevant covariates.

Remarkably, the CIs of these estimates actually overlap the 65.5% replication rate that one would expect if every one of the original studies had reported a true effect.

Although that seems rather unlikely, OSC’s data clearly provide no evidence for a “replication crisis” in psychological science.

60%? Gee, I'm so impressed! Crisis over!

 

[*GG*]

When we compared the replication rates of the endorsed and unendorsed protocols, we discovered that the endorsed protocols were nearly four times as likely to produce a successful replication (59.7%) as were the unendorsed protocols (15.4%).

This strongly suggests that the infidelities did not just introduce random error but instead biased the replication studies toward failure. If OSC had limited their analyses to endorsed studies, they would have found that 59.7% [95% confidence interval (CI): 47.5%, 70.9%] were replicated successfully.

In fact, we estimate that if all the replication studies had been high enough in fidelity to earn the endorsement of the original authors, then the rate of successful replication would have been 58.6% (95% CI: 47.0%, 69.5%) when controlling for relevant covariates.

Remarkably, the CIs of these estimates actually overlap the 65.5% replication rate that one would expect if every one of the original studies had reported a true effect.

Although that seems rather unlikely, OSC’s data clearly provide no evidence for a “replication crisis” in psychological science.

60%? Gee, I'm so impressed! Crisis over!

Breaking this up, to much freaking text in a large block!

GG

 

[alex3619]

The funny thing is that replicability doesn't even mean the results are true. It may just mean that the same biases leading to the false appearance of a result are present.

Yes. Also that a significant result can be significant bias. Significance is not truth.