Most research spending is wasted on poor study design. Someone wants to change that

[Snowdrop]

Sorry don't know where to put this.

http://www.vox.com/2015/10/4/9440931/arnold-foundation-meta-research

Scientific research is often conducted in a highly unscientific manner. About $200 billion — or about 85 percent of global spending on research — is routinely wasted on poorly designed and redundant studies. As much as 30 percent of the most influential original medical research papers later turn out to be wrong or exaggerated.

 

[PatJ]

Nice find Snowdrop. This article provides more information about the Laura and John Arnold foundation and the billionaires behind it.

I think that CFS/ME and other difficult health conditions aren't dealt with properly due to a health care and funding system that just doesn't work properly for many reasons. Someone without brainfog, who has the skills, knowledge, and ability might do well by creating a document that outlines the many ways that CFS/ME funding, research, and treatment reflects a broken health-care system. This document could then be presented to the Laura and John Arnold foundation to see if they would be interested in taking on an interesting and very difficult problem where solutions could potentially benefit many conditions all at once.

Laura and John Arnold have an interest in working on high-level problems like this. They directly fund projects on a personal level but the foundation focuses on using data and analysis to fix problems at system and policy levels. They aren't afraid of very big, very long-term projects.

Does the idea for this sort of proposal make sense? Is there a CFS/ME organization that might be suitable and interested in this suggestion?

 

[Simon]

Wow, this is fascinating stuff, and I'd love to blog about it if I had more energy .

It's about Laura and John Arnold, a couple who made their billions in their thirties and now want to do something different. Laura Arnold was interviewed:

Most research spending is wasted on bad studies. These billionaires want to change that. - Vox

Research affects everything we do, from medicine and pharmaceuticals to basic science. So if we don’t have a good core of rigorous research, and as a research community have discipline as to what constitutes rigorous research, almost by definition our work will not be accomplished. We will not be able to move the conversation toward logical good answers.

Asked why they didn't focus on the more obvious targets for philanthropy, such as breakthrough research projects and buildings they could literally put their name on, Arnold said:

We’re less attracted to the breakthroughs, the most traditional analysis for this kind of investment. We’re much more attracted to what is a systemic problem that we can fix in the long term and will have a greater impact on this piece of research and all others...

The end goal is to help somebody with a disease make the right decision about a treatment, to give a doctor who is taking care of a child the ability and information to make the best treatment decision for that child, to help a person who is at risk of dying or of contracting diseases from obesity make the right lifestyle choices so he or she isn’t at risk.
So it is ultimately — and very prominently — about making society better and helping people.

They've already funded Brian Nosek's famous reproducibility project, which found that most of 100 chosen psychology research studies didn't reproduce, and Stanford Center for Reproducible Neuroscience because Brain imaging research is often wrong (that almost certainly includes some of the fMRI work on mecfs too).

 

[Snowdrop]

Yes, I thought it was an exciting development as well.

And I have also believed from early on that developing links with other groups for advocacy or even just to support research being done for other issues would be to our benefit as well. Finding and highlighting research that's not ME specific so that the ME research community is aware of other developments--but there's always been the problem of any study being crap science.

Even something like raising awareness of poverty and how this group is perceived by others as marginal people who deserve their fate can impact our situation which is often similar.

I hope they are successful.

 

[Woolie]

@Snowdrop, thanks for the links, very interesting!

Plus, on one of those links you gave I found this interesting article:

Why we've probably overestimated the benefits of psychotherapy for major depression

This article addresses the problem of publication bias, which is the fact that studies finding significant effects are more likely to get published than those that don't find significant effects. So in other words, what you see out there in print isn't a true representative sample of the research that's been done, but rather a selective sample of the most "flattering" results.

They tried to estimate just how big this bias is, and what would happen if we took into account all the hidden, unpublished studies. The result: the beneficial effect of therapy is probably much smaller than we've been led to believe, at least for major depression.

Bear in mind that the article considers only publication bias. There are all sorts of other problems with psychotherapy research that might have also led to inflated claims of its effectiveness. It doesn't even start to consider these (for example, use of subjective measures, poor control conditions, selective reporting of only the "best" outcome measures, etc.).

 

[Snowdrop]

Yes, I came across the depression one too on twitter and I agree with your post. It boggles the mind how people can acquire an advanced degree and not have a clue about how to design a study to yield useful data. And while I'm thinking about it...who the heck is funding all this crap?

 

[Snowdrop]

And more in the same vein : https://mbnuijten.files.wordpress.c...ingerrorspsychology_acceptedmanuscriptbrm.pdf

I haven't read it all and there are lots of graphs etc discussing various aspects of reporting data. Beyond me.

And this: http://www.nature.com/news/how-scientists-fool-themselves-and-how-they-can-stop-1.18517

The title pretty much sums it up. There seems to be an increasing number of these articles and studies.
I'm going to call it the James C Coyne effect.

 

[Woolie]

Great stuff, @Snowdrop.
Wish they'd analyse doctors' biases in making clinical diagnoses more.

 

[barbc56]

While I don't have a doubt that a lot of studies aren't valid, I wonder what kind of studies are taken into consideration?

A preliminary study that shows a positive result may not bear out when you go to the next phase of testing. That's just part of the process of proving a hypothesis.This is why we often see in the news that x is healthy only to be replaced or negated later. The media picks up the results of these studies but don't necessarily realize or state that the resuls are just preliminary. In that case the money isn't wasted.

When we see studies where the actual design or interpretation is flawed, it's a different story.

I need to read more of the links in this thread plus review Ionides as I just don't remember at the moment.

Great thread @Snowdrop. This looks hopeful.

Barb

 

[Snowdrop]

Nice find Snowdrop. This article provides more information about the Laura and John Arnold foundation and the billionaires behind it.

I think that CFS/ME and other difficult health conditions aren't dealt with properly due to a health care and funding system that just doesn't work properly for many reasons. Someone without brainfog, who has the skills, knowledge, and ability might do well by creating a document that outlines the many ways that CFS/ME funding, research, and treatment reflects a broken health-care system. This document could then be presented to the Laura and John Arnold foundation to see if they would be interested in taking on an interesting and very difficult problem where solutions could potentially benefit many conditions all at once.

Laura and John Arnold have an interest in working on high-level problems like this. They directly fund projects on a personal level but the foundation focuses on using data and analysis to fix problems at system and policy levels. They aren't afraid of very big, very long-term projects.

Does the idea for this sort of proposal make sense? Is there a CFS/ME organization that might be suitable and interested in this suggestion?

Such a document exists if I understand what you're getting at. In my signature is a link to a doc by Mary Dimmock. It's long and thorough and is about the history and politics of ME.

 

[SOC]

It boggles the mind how people can acquire an advanced degree and not have a clue about how to design a study to yield useful data.

Amen. I find myself frequently saying (often to an empty room ), "Where the heck did these people go to college? We had courses in design of experiments and statistical analysis as undergraduates. How could they get through graduate school without a clue about designing rigorous, useful, logical research studies?"

It simply makes no sense to me that any self-respecting university would graduate science students without a clue how to design a scientific study and correctly analyze and report the data. And yet they do... in droves, apparently.