Why the ‘gold standard’ of medical research is no longer enough - Opinon piece by Tom Frieden

[AndyPR]

Another one that I thought some here might find interesting. Tom Frieden was director of the CDC until recently.

Randomized controlled trials have long been held up as the “gold standard” of clinical research. There’s no doubt that well-designed trials are effective tools for testing a new drug, device, or other intervention. Yet much of modern medical care — perhaps most of it — is not based on randomized controlled trials and likely never will be. In this “dark matter” of clinical medicine, past practices and anecdotes all too often rule. We need to look beyond trials to improve medical care in these areas.

In a randomized controlled trial (RCT), participants are randomly assigned to receive either the treatment under investigation or, as a control, a placebo or the current standard treatment. The randomization process helps ensure that the various groups in the study are virtually identical in age, gender, socioeconomic status, and other variables. This minimizes the potential for bias and the influence of confounding factors.

Despite their strengths, RCTs have substantial limitations. They can be very expensive to run. They can take many years to complete, and even then may not last long enough to assess the long-term effect of an intervention such as vaccine immunity, or to detect rare or long-term adverse effects. Findings from RCTs may not be valid beyond the study population — a trial that included a high-risk population in order to maximize the possibility of detecting an effect, for example, may not be relevant to a low-risk population. RCTs may not be practical for population-wide interventions and often aren’t relevant for urgent health issues such as infectious disease outbreaks, for which public health decisions must be made quickly.

https://www.statnews.com/2017/08/02/randomized-controlled-trials-medical-research/

 

[duncan]

RCTs are also subject to manipulation or inadvertent error. Get one variable wrong - say, treatment duration or dosage - and the results could be fatally flawed.

RCTs may be the best we have, but they can be had.

 

[Jonathan Edwards]

Another one that I thought some here might find interesting. Tom Frieden was director of the CDC until recently.

https://www.statnews.com/2017/08/02/randomized-controlled-trials-medical-research/

This is interesting but the arguments in the quoted passage look muddled to me. He says there are dark areas where RCTs are not applied - so maybe RCTs should be applied there.

My impression is that the valid message is that any old RCT is not enough. You need follow up safety checks AS WELL and further studies on different populations AS WELL. And in the case of ME/CFS you need studies that are actually RCTs, i.e. actually controlled, which PACE is not. And you need DBRCTs (double blind) most of the time.

I don't get it if the argument is that RCTs are no longer the gold standard. They remain that for simple logic reasons. They may fall short, but a Rolls Royce with a scratched bumper is still better than a Fiat 500.

 

[sb4]

I too think RCT need to be taken with a grain of salt.

If you are testing a drug or diet and in one study the rats/humans are under blue light, in high nn-emf, and a warm room vs under window light, low nn-emf, in a cool room, you could get very differen't results.

Ideally these things are controlled for but if the conventional wisdom is that these things have no effect or the researches don't know about it then you would have big trouble getting reliable results.

There is also a problem with researches going in with biases and maybe tinkering with results to get a more favourable outcome. Then their is the danger that if you get the result that doesn't align with the established opinion, your name could get dragged through the mud and your reputation ruined.

 

[Hutan]

It's a nice article, clearly making the point that there are a number of types of evidence that should be considered when evaluating a treatment or public health program. Cochrane, take note.

For some public health issues, it isn’t ethical to conduct an RCT.
...
For the several thousand rare diseases, RCTs are unlikely to be conducted due to the small number of people who have them and other logistical constraints.
...
The emerging use of “big data,” including information from electronic health records and expanded patient registries, presents new opportunities to conduct large-scale studies with many of the benefits of RCTs but without the expense. One such study used data from the Veterans Health Administration and Medicare to examine outcomes of treatment for type 2 diabetes. This study was many times larger, with much longer follow-up and lower cost, than previous RCTs comparing the effectiveness of different diabetes drugs. It clearly showed that one class of drug, the thiazolidinediones, was much more effective than another class, the sulfonylureas, in reducing hospitalization and death.
...
Clinical and public health decisions are almost always made with imperfect data. There is no single, best approach to obtain better information about health interventions. Evidence grading systems, policy makers, and researchers must embrace other study types in addition to RCTs. Essential steps in interpreting findings and identifying data for action include promoting transparency in study methods, ensuring standardized data collection for key outcomes, and using new approaches to improve data synthesis.
...
Glorifying RCTs above other approaches, even when these other approaches may be either superior or the only practical way to get an answer, relegates patients to receiving treatments that aren’t based on the best available evidence.

 

[TenuousGrip]

My brother used to deal blackjack at a casino. He could tell you the odds of the next card being a Jack, Queen, King, or Ace.

But smart people didn't bet their rent money on it ;-)

This is how I've long felt about the RCT. It uses sample size in an effort to compensate for infinitely variable individual physiology.

But, against infinitely variable individual physiology, the sample size in your average RCT is mighty weak sauce.

Too many MDs, IMHO, don't understand the statistical limitations of an RCT -- what you can and cannot definitively say based on one.

Which also brings us to the meta-study -- the study OF the aggregate of RCTs on a particular topic -- again, trying to use sample size to compensate for the variability.

Personalized medicine ... should be a game changer ... if the companies who stand to profit from it can beat out the companies who stand to lose money from it ... in the marketplace ;-)

 

[Learner1]

Random controlled trials have their place. However, especially on a site like this, we have to be aware that there are pitfalls.

We are all genetically unique with individual environmental exposures and unknown comorbidities, which are typically not factored into most clinical trials.

I've endeavored to put together a set of articles that explore some of these issues, and look at the advent of personalized medicine, not only from a genetic perspective, but also taking into account lifestyle and environmental factors.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1761670/#!po=2.27273

https://www.bloomberg.com/view/articles/2017-07-28/big-data-shows-big-promise-in-medicine

https://fivethirtyeight.com/features/medicine-is-getting-more-precise-for-white-people/

https://www.systemsbiology.org/research/p4-medicine/

http://mayoresearch.mayo.edu/center-for-individualized-medicine/personalized-medicine.asp

https://health.ucsd.edu/news/releas...medicine-leads-to-better-cancer-outcomes.aspx

https://academic.oup.com/hmg/articl...cogenomics-of-adverse-drug-reactions#11385711

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4196974/

https://www.sciencedaily.com/releases/2017/03/170309093206.htm

http://www.aafp.org/afp/2007/0801/p391.html

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1963417/

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3312765/

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3813354/

http://www.cpnhelp.org/b_12_deficiency_in_cpn_in

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2949235/#__ffn_sectitle

 

[RogerBlack]

We are all genetically unique with individual environmental exposures and unknown comorbidities, which are typically not factored into most clinical trials.

I'm not!

 

[Learner1]

I'm not!

Ah, then you must be female, like me...

 

[RogerBlack]

Ah, then you must be female, like me...

It was both a quote from "Life of Brian" (

) and a commentary that RCTs have a place in a human body we can't hope to understand all of the mechanisms in that would enable 'precision medicine'.
Unfortunately, fixing RCTs is hard.
(well, fixing them the right way )
Some of the 'easy' fixes (include women, and a representative patient population with comorbidities) will generally mean much more expensive, larger trials.

The only real improvement that could be 'cheap' would be requiring all clinical trial data to be published, along with the supporting data, positive or negative, with the penalty for concealing data being something like twice global worldwide sales of the drug.

 

[Learner1]

Thanks for clarifying.... I think we're on the same page (...though you certainly seem to know more about the Life of Brian ).

As a patient, I'm finding it more and more skeptical of any RCT... and finding it far more helpful for my doctors and me to understand the various biological mechanisms and figure out what will work for me...as there are too many complexities to apply anything directly from a drug trial.

We're a long way from truly personalized medicine, but for many if us, it'll be the only path to wellness... The devil is in the details...

 

[RogerBlack]

As a patient, I'm finding it more and more skeptical of any RCT... and finding it far more helpful for my doctors and me to understand the various biological mechanisms and figure out what will work for me...as there are too many complexities to apply anything directly from a drug trial.

This is unfortunately impossible for all but the most basic and simple of interactions, and makes the opposite wrong assumption at times. (that nothing can be learned from the group).

If, for example, the body has three ways of making X, and science has found one of them, and lack of X will kill you at age 40.
If you are found to be deficient in one way of making X, this may have little or no clinical impact, but you'd be treated aggressively because the other two mechanisms that compensate are unknown.

To take a more concrete example, the mechanistic link from high levels of cholesterol to heart disease seems simple. The reasonable assumption was made that lowering cholesterol would cause lower levels of heart disease, and statins were developed on this basis.
It's unfortunately proved not to be so simple and the effect if any of statins is unclear.

There is no good reason unfortunately to expect all 'personal medicine' that naively finds a marker for a disease and treats to reduce that marker will reliably help the person.

 

[Jonathan Edwards]

There is no good reason unfortunately to expect all 'personal medicine' that naively finds a marker for a disease and treats to reduce that marker will reliably help the person.

I agree. Medicine has always been personalised as best we can. You adjust dose to size. You give Japanese less sulphasalazine, and so on, but calling it 'personalised medicine' is just an empty buzzphrase. There is not the remotest chance that we will be able to tailor drug treatments to genes in more than a very few examples (like the Japanese). When physicians talk about tailoring to patients most of the time they are flanelling. I used to flannel like that but gave up around 1990.

And unless you have the results of a decent controlled trial to start from there is nothing to personalise. If you do not know something works at all then you are not going to know who it works best for!! Controlled trials are not perfect but the whole point of them is that they are better than the alternative.

 

[alex3619]

And you need DBRCTs (double blind) most of the time.

Yes, wherever possible.

 

[Barry53]

Ideally these things are controlled for but if the conventional wisdom is that these things have no effect or the researches don't know about it then you would have big trouble getting reliable results.

Presumably the idea is to ensure the only difference between groups is the treatment under test? But I guess you are saying that in reality that can be tricky.

 

[sb4]

Presumably the idea is to ensure the only difference between groups is the treatment under test? But I guess you are saying that in reality that can be tricky.

Yea and it's especially tricky when you are dealing with variables you have assumed / been told are not variables. Imagine putting one rat cage next to a wifi router, right under the cool white lab lights, and the other at the other side of the lab near a window, away from router. These things could significantly effect your experiment without you knowing why.

Now imagine someone trying to repeat that experiment in another country.

Now imagine say trying a diet where all the participants have similair mitochondrial haplotype and SNPs, yet you have not measured for this. You get a certain result. Someone tries to replicated this with people who have different haplotype and SNPs and gets different result. Confusion ensues.

This is one of the reasons, I think, that there are so many contradictory studies out there.