Is Trial Registratn Working to Prevent Publicatn of Unregistered Trials & selective outcome reportng

[Dolphin]

(A minority interest)
(Had to abbreviate title: that's all the characters I had)

POMs=Primary Outcome Measures

Free full text: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0133718

Is Mandatory Prospective Trial Registration Working to Prevent Publication of Unregistered Trials and Selective Outcome Reporting? An Observational Study of Five Psychiatry Journals That Mandate Prospective Clinical Trial Registration

• Amelia Scott,
• Julia J. Rucklidge ,
• Roger T. Mulder

• Published: August 19, 2015

Abstract

Objective

To address the bias occurring in the medical literature associated with selective outcome reporting, in 2005, the International Committee of Medical Journal Editors (ICMJE) introduced mandatory trial registration guidelines and member journals required prospective registration of trials prior to patient enrolment as a condition of publication.

No research has examined whether these guidelines are impacting psychiatry publications.

Our objectives were to determine the extent to which articles published in psychiatry journals adhering to ICMJE guidelines were correctly prospectively registered, whether there was evidence of selective outcome reporting and changes to participant numbers, and whether there was a relationship between registration status and source of funding.

Materials and Methods

Any clinical trial (as defined by ICMJE) published between 1 January 2009 and 31 July 2013 in the top five psychiatry journals adhering to ICMJE guidelines (The American Journal of Psychiatry, Archives of General Psychiatry/JAMA Psychiatry, Biological Psychiatry, Journal of the American Academy of Child and Adolescent Psychiatry, and The Journal of Clinical Psychiatry) and conducted after July 2005 (or 2007 for two journals) was included.

For each identified trial, where possible we extracted trial registration information, changes to POMs between publication and registry to assess selective outcome reporting, changes to participant numbers, and funding type.

Results

Out of 3305 articles, 181 studies were identified as clinical trials requiring registration: 21 (11.6%) were deemed unregistered, 61 (33.7%) were retrospectively registered, 37 (20.4%) had unclear POMs either in the article or the registry and 2 (1.1%) were registered in an inaccessible trial registry.

Only 60 (33.1%) studies were prospectively registered with clearly defined POMs; 17 of these 60 (28.3%) showed evidence of selective outcome reporting and 16 (26.7%) demonstrated a change in participant numbers of 20% or more; only 26 (14.4%) of the 181 the trials were prospectively registered and did not alter their POMs or the time frames at which they were measured.

Prospective registration with no changes in POMs occurred more frequently with pharmaceutical funding.

Discussion

Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs.

Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.

Authors, journal editors and reviewers need to further efforts to highlight the value of prospective trial registration.

 

[SOC]

Why am I not surprised?

 

[alex3619]

Why am I not surprised?

(I realize this is rhetorical, but ... ) Because we saw it in the PACE trial and almost nobody cared? PACE was before this initiative, but they did not use their proposed plan, selectively reported data, and abandoned an effective and objective measure with actometers. The real issue is they were largely not held to account, or appropriately criticized, for these issues.

 

[SOC]

(I realize this is rhetorical, but ... ) Because we saw it in the PACE trial and almost nobody cared? PACE was before this initiative, but they did not use their proposed plan, selectively reported data, and abandoned an effective and objective measure with actometers. The real issue is they were largely not held to account, or appropriately criticized, for these issues.

Oh yeah, that's why.

 

[Simon]

To summarise, then:

• Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs (Published Primary Outcome Measures).
• Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.
• Prospective registration with no changes in POMs occurred more frequently with pharmaceutical funding.

Marvellous

 

[Sasha]

To summarise, then:

• Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs (Published Outcome Measures?).
• Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.
• Prospective registration with no changes in POMs occurred more frequently with pharmaceutical funding.

Marvellous

I remember that White (somewhere) defended changing the planned outcome measures in PACE that had been registered in the protocol, by saying that it was very common to make such changes. I was astonished by this claim at the time (particularly in relation to primary outcome analyses and particularly in a 641-patient, £5m trial) but I wonder if it's true in psychiatry? Are there comparative figures for non-psychiatric medicine?

 

[worldbackwards]

I remember that White (somewhere) defended changing the planned outcome measures in PACE that had been registered in the protocol, by saying that it was very common to make such changes.

It's a classic - It's OK, we're all frauds. The whole thing's a fraud. What's up with that?

 

[Bob]

I remember that White (somewhere) defended changing the planned outcome measures in PACE that had been registered in the protocol, by saying that it was very common to make such changes.

Yes, that was rather mind-boggling. I think it's OK to decide that your primary outcome measures aren't helpful or appropriate after the trial is complete, but they still need to be published. Any post-hoc outcome measures can be published alongside the primary measures that are set out in the protocol or trial registry, and they should be clearly labelled as secondary and post-hoc. A discussion can be included about why the primary outcome measures proved to be unhelpful.

 

[Sasha]

Yes, that was rather mind-boggling. I think it's OK to decide that your primary outcome measures aren't helpful or appropriate after the trial is complete, but they still need to be published. Any post-hoc outcome measures can be published alongside the primary measures that are set out in the protocol or trial registry, and they should be clearly labelled as secondary and post-hoc. A discussion can be included about why the primary outcome measures proved to be unhelpful.

I think it's more complicated than that. White says (as I understand it) that the new analyses weren't post hoc because they hadn't seen the data (although they had seen the failure of the same analyses in the FINE trial). I don't think that one needs to assume post-hockery to find a shedload of reasons not to trust his interpretation of the data.

But such wholesale disregard for the per-protocol analysis (especially in the "recovery" paper) has undermined trust in the trial and analysis according to the original results should now be produced.

 

[Bob]

I think it's more complicated than that. White says (as I understand it) that the new analyses weren't post hoc because they hadn't seen the data...

Yes, he claimed that about the primary outcomes, but his disregard of rigorous process (i.e. by not publishing the proposed primary outcomes) negates the purpose of the protocol and trial registry. If we take his claims at face value, it's still problematic, and they should still publish the primary outcomes as proposed in the protocol. Otherwise there's no point in protocols. The whole point in the pre trial publications is to safe-guard against dubious practice, and to ensure rigour. It gives the public confidence that proper process has been followed. As it is, we have to accept the word of those involved in the trial that proper process was followed. We might have no reason to doubt them, but the public needs to have confidence. So it's complicated, but actually not that complicated. Good practice is to publish the outcomes as set out in the trial protocol. Changes are sometimes necessary, but the original primary outcome measures should still be published alongside any new outcome measures so that the public can judge the merit and value of the changes.

That's my interpretation of good practice anyway. But perhaps others don't agree?

 

[Sasha]

The protocol has more than one purpose. One is as part of the submission of the trial to a review committee to see if it should be funded. If people want to make protocol changes later, they need to run it past another committee to get the OK.

The weird thing about PACE, though, is that (in my experience, anyway) changes are only made to analyses when the data are in and the stattos find that the actual data violate the statistical assumptions of the planned analysis (the usual problem is that they're not normally distributed) - but I think the change is generally then to normalise the data by transforming it (log scale, etc.). It doesn't require that the original analyses be done too (I think? Unless that's done as a sensitivity analysis? I'm rusty).

This isn't what happened in PACE, though - especially in the "recovery" paper, which took a wrecking ball to the planned analyses.

It would be interesting to know what the establishment view is of what should happen when a protocol has been published and the analyses change. Cochrane would be a good source for best practice.

 

[Dolphin]

To summarise, then:

• Although standards are in place to improve prospective registration and transparency in clinical trials, less than 15% of psychiatry trials were prospectively registered with no changes in POMs (Published Outcome Measures?).
• Most trials were either not prospectively registered, changed POMs or the timeframes at some point after registration or changed participant numbers.
• Prospective registration with no changes in POMs occurred more frequently with pharmaceutical funding.

Marvellous

I think it's more likely POMs=Primary Outcome Measures. That's what is usually looked at in papers on outcome reporting bias.