• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

PACE study and the Lancet: Journal reputation is a two-way street

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
Columbia Uni prof and Good Science advocate Andrew Gelman wonders how the Lancet could get things so wrong, and how when they did, everyone believed them.

PACE study and the Lancet: Journal reputation is a two-way street - Statistical Modeling, Causal Inference, and Social Science

One think that struck me about this PACE scandal: if this study was so bad as all that, how did it taken so seriously by policymakers and the press?

There’s been a lot of discussion about serious flaws in the published papers, and even more discussion about the unforgivable refusal of the research team to share their data. But the question I want to address here is, how did they get into the position where this research got taken seriously in the first place?

As Dan Kahan might say, what do you call a flawed paper that was published in a journal with impact factor 50 after endless rounds of peer review? A flawed paper.

Read the full blog
 

Marco

Grrrrrrr!
Messages
2,386
Location
Near Cognac, France
Worth preserving his edit of the Wiki page for The lancet (under controversies) for posterity :

PACE study (2011)[edit]

A study by the PACE trial management group reported success of a combination of exercise and talk therapy for chronic fatigue syndrome.[41] The study has been highly controversial; for example, biostatistician Bruce Levin of Columbia University described the study as "the height of clinical trial amateurism," and Ronald Davis of Stanford University wrote, “I’m shocked that the Lancet published it…The PACE study has so many flaws and there are so many questions you’d want to ask about it that I don’t understand how it got through any kind of peer review.”[42]

Controversy has arisen not just from the published papers but also with the authors' and the Lancet's refusal to share data from the study: "Starting in 2011, patients analyzing the study filed Freedom of Information Act requests to learn what the trial’s results would have been under the original protocol. Those were denied along with many other requests about the trial, some on the grounds that the requests were 'vexatious.' The investigators said they considered the requests to be harassment. . . . Richard Horton, the editor of the Lancet, aggressively defended the trial. In a radio interview, he called the critics 'a fairly small, but highly organized, very vocal and very damaging group of individuals who have, I would say, actually hijacked this agenda and distorted the debate so that it actually harms the overwhelming majority of patients.'"[43]

Imagine having that on your CV :nervous:
 

Jonathan Edwards

"Gibberish"
Messages
5,256
Good to see. But why is he asking the question? I told him the answer in the comments so maybe that will come up after moderation. Surely he can see that the study is flawed? As someone else said, I think reviewers ARE there to pick up flaws and the fact that they did not is not simply a matter of bad luck.
 

Sasha

Fine, thank you
Messages
17,863
Location
UK

Roy S

former DC ME/CFS lobbyist
Messages
1,376
Location
Illinois, USA
Andrew Gelman wrote:

"I got interested because a friend has been sick for 20+ years and he nudged me to look into PACE."

He is another person interested in this disease because of having a friend with it for a very long time. I wish we had many more.
 

Cheshire

Senior Member
Messages
1,129
Andrew Gelman wrote:

"I got interested because a friend has been sick for 20+ years and he nudged me to look into PACE."

He is another person interested in this disease because of having a friend with it for a very long time. I wish we had many more.

He is talking about David Tuller:
As David Tuller, a public health and journalism lecturer at UC Berkeley, put it:

The thing about PACE that has astonished me is how papers with such obvious flaws were accepted immediately by the entire UK academic/psychiatric/medical establishment, including the Lancet. It’s a case of mass confirmation bias. The emperor really has no clothes, and the patients have known it for years and have been screaming about it. But they have been dismissed as irrational and dangerous. I got interested because a friend has been sick for 20+ years and he nudged me to look into PACE.
 

Snowdrop

Rebel without a biscuit
Messages
2,933

adreno

PR activist
Messages
4,841
He is another person interested in this disease because of having a friend with it for a very long time. I wish we had many more.
I get your point. I am always a little saddened that most researchers only take an interest when they have someone personal involved. But I guess this is just how the world works. As ME becomes more respectable and maybe even prestigious to research, I suspect many researchers will take an interest for career reasons.
 

Simon

Senior Member
Messages
3,789
Location
Monmouth, UK
[to @Jonathan Edwards]I think this is your comment (and he replies):

details here:
Jonathan said:
January 5, 2016 at 12:45 pm
Andrew: As I see it the problem with the peer review process was simply that psychiatry has inadequate standards for trials. Any clinical pharmacologist sent this paper for review would have said that it was not providing reliable information: period. This is not an issue of missing subtle problems. It is a barn door issue of a trial that is methodologically a non-starter. Unfortunately, it seems that in psychiatry people simply do not understand how bias has to be dealt with in order to produce a useful result. And I have had emails from eminent psychiatrists, who may well have reviewed this, indicating just that – they do not seem to understand. So the answer to your question is probably fairly simple. Psychiatry needs to get its act together.

reply:
Andrew Gelman said:
January 5, 2016 at 1:52 pm
Jonathan:

This is related to the problem we discussed a couple years ago with that ovulation-and-clothing study that had been published in Psychological Science. The estimates effects were ridiculously implausible, and the measurement of peak fertility was both biased and noisy. Even setting aside the garden of forking paths, the study, and others like it, were dead on arrival. But the researchers didn’t seem to care about how bad the measurements were. They’d been trained to believe that if you have statistical significance, you’re good to go. I found the whole interaction very frustrating in that they didn’t even want to engage with concerns about bias and variance of measurements. They were holding on to that statistical significance and not letting go.
====

One other comment caught my eye
In reply to Leveleller January 5, 2016 at 2:31 pm
Andrew Gelman said:
January 5, 2016 at 2:38 pm
Leveller:

“Vexatious” lobbying, huh? I think “vexatious” is going to be one of our new catchphrases, along with “evilicious,” “replication bullies,” and, of course, “gremlins.”

People questioning a study published in a top journal—what could be more vexatious than that?!
 
Messages
3,263
Worth preserving his edit of the Wiki page for The lancet (under controversies) for posterity :
Marco, I don't see that change on the page, perhaps its been reverted. Do you know what username Gelman used to edit? Also, there are a few pages that discuss PACE, I might be looking at the wrong one.