Radio programme reveals fundamental flaws in research

[MeSci]

This programme on BBC Radio 4 last night showed that the BBC can still make intelligent, important, incisive, groundbreaking, eye-opening programmes.

Here is the synopsis:

Every day the newspapers carry stories of new scientific findings. There are 15 million scientists worldwide all trying to get their research published. But a disturbing fact appears if you look closely: as time goes by, many scientific findings seem to become less true than we thought. It's called the "decline effect" - and some findings even dwindle away to zero.

A highly influential paper by Dr John Ioannidis at Stanford University called "Why most published research findings are false" argues that fewer than half of scientific papers can be believed, and that the hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true. He even showed that of the 49 most highly cited medical papers, only 34 had been retested and of them 41 per cent had been convincingly shown to be wrong. And yet they were still being cited.

Again and again, researchers are finding the same things, whether it's with observational studies, or even the "gold standard" Randomised Controlled Studies, whether it's medicine or economics. Nobody bothers to try to replicate most studies, and when they do try, the majority of findings don't stack up. The awkward truth is that, taken as a whole, the scientific literature is full of falsehoods.

Jolyon Jenkins reports on the factors that lie behind this. How researchers who are obliged for career reasons to produce studies that have "impact"; of small teams who produce headline-grabbing studies that are too statistically underpowered to produce meaningful results; of the way that scientists are under pressure to spin their findings and pretend that things they discovered by chance are what they were looking for in the first place. It's not exactly fraud, but it's not completely honest either. And he reports on new initiatives to go through the literature systematically trying to reproduce published findings, and of the bitter and personalised battles that can occur as a result.

I am now wondering what research I should take seriously! It's good to know what's wrong, but very disheartening when we are so desperate to figure out how to get at least some of our health back.

 

[alex3619]

Replication, and independent replication is key. Recently I discovered a term called Citation Circle, where a small area of research has people who constantly cite each other (and I suspect they constantly review each others papers for publication). This results in high citations, high publication etc.

High citation counts of pathetic science ... its a warning sign that something is wrong.

Its like EBM versus EBP ... evidence based medicine versus evidence based practice. EBM has become a rubber stamp, another way to put an authoritarian stamp on medical dogma. EB practice requires practitioners read the research, evaluate it, and make up their own mind. Very few doctors do that, instead they cite some other claim to evidence base without any concept as to whether or not its valid.

In any case no science can be considered proven. If its proven then its an abstract mathematical concept, proven from a given set of axioms. We can at most do two things in the real world, provide good evidence and test an hypothesis for robustness. Psychobabble gets away with providing vague suggestive evidence and studiously avoiding testing.

 

[user9876]

Replication, and independent replication is key.

I worry about what people mean by replication. At a simple level it can be to repeat an experiment which seems important.

But an experiment may get results due to its bad design - For example different treatments may get different levels of placebo effect with self reports and if better measurement is in place then no change would be recorded.

Thus I think when we talk of replication we should also mean that the hypothesis is stated and other groups try to find other experiments that would test the hypothesis in other ways.

 

[alex3619]

Agreed @user9876, but also that larger studies or even just unmodified replication often get different results. This wont fix bias issues, but it may address issues of chance. One of the problems with a p value of 0.05 is that the research fills up with false negatives.

Studies also have to be examined carefully for methodological failings. Statistitical p values only give an indication of possibility that a result is due to chance ... and a biased study may not have much due to chance. So statistical significance cannot sort out bias or even fraud.

We should perhaps start a research methodology thread. These questions come up so very often.

 

[Wayne]

It seems a vast amount of medical research is predicated on trying to produce a blockbuster drug that will somehow provide a solution to their discoveries. What I often think of is the research that is NEVER done, because there's no economic payoff that would result anyway.

I don't know how to quantify this, but in my own mind (not reliable ), the amount of valid research that doesn't get done because there would be no monetary benefit is many times more than the research done that might provide a large payoff. With such a large exclusion, this casts doubt for me on how really accurate it is to call the research that is done "scientific".

 

[alex3619]

Warning: philosophical ranting slightly off topic

@Wayne this has been claimed about the US research base. Back when the US was doing very well indeed, the government was funding lots of research with no obvious payoff. They then downsized and moved to an economic payoff model. The result is most research is private now (iirc) and most research, even government research, is target funded. As a result chance discoveries might even get ignored and go unpublished if they don't have an obvious payoff. Science is waning in the US, as is economic growth. There are other factors of course, but the really big discoveries tend to be random, and by focusing on technological widgets or targeted science a huge amount is missed.

I suspect that much of this is due to a sea change in economics in the 1970s. The economic ideology shifted. With a little lag things went into decline. Efficiency saves money but it destroys opportunity and robustness. It makes more money now by selling off the future. Things are, in my view, going to get worse. What we need is to turn back the clock to smaller business, and higher taxes, and governments who actually use those taxes to create environments that boost growth. Growth instead of profit is the ideal, as future profits are larger.

We also need to decrease or put a ceiling on resource utilization. Of course it will be a challenge to foster growth without consuming more resources. I have strong suspicion that critical resources are going to become prohibitively expensive for business about mid-century. I do not mean expensive as in costing money, nor that they will run out, but that the resource cost to obtain resources will go negative on net gain as the resources become harder to get.

So we need goal based research. However we cannot ignore research that improves our understanding of the world. The work of applied scientists, even just widget makers, requires that we have a good understanding of the world. Its the chance discoveries and developments that change economies, very rarely is it the latest widget advancement. Enhanced widgets make a little money, and can do that on a large scale, but suck resources.

For the record I am pro-democracy, pro-small business, anti-monopoly, pro free speech, and believe that governments have a role in providing services, including to business, and that these need to be paid for.

In my view the biggest current economic problem in much of the world, and definitely the US, is government and private debt. It creates severe vulnerability. A big economic shock might collapse even an economic giant.

 

[Wayne]

Great post Alex. I could have written it myself, as it reflects so closely my own perspectives. Well, maybe I couldn't have written it; not sure I have the cognition at this time. --- I agree, debt is a grave threat to our future. It more than anything else could bring on an economic collapse, one we only recently narrowly averted.

 

[*GG*]

Warning: philosophical ranting slightly off topic

@Wayne
I suspect that much of this is due to a sea change in economics in the 1970s. The economic ideology shifted. With a little lag things went into decline. Efficiency saves money but it destroys opportunity and robustness. It makes more money now by selling off the future. Things are, in my view, going to get worse. What we need is to turn back the clock to smaller business, and higher taxes, and governments who actually use those taxes to create environments that boost growth. Growth instead of profit is the ideal, as future profits are larger.

What do you mean by higher taxes? People in some states pay nearly 60% of what they earn in taxes. Is that not enough? Do you not believe in the Laffer Curve?

GG

 

[alex3619]

Hmmm, the Laffer curve is a heuristic, and quite irrelevant to the topic anyway. When the US was at its fastest growth it had 90% marginal tax rate, very high income growth, lots of research on non-commercial oriented science. Its not just about taxation rate, and more importantly marginal tax rate, its about what the taxes are spent on. You can spend them in ways that don't boost the economy, or do. If it boosts the economy then over time you actually make more money - compound growth beats money under the mattress. Its also the case that though the more wealthy paid more in tax on income, unrealized capital gains were HUGE. Wealth went up.

When business lacks essential infrastructure, because governments have consistently failed, then their competitiveness goes down. Profits and growth go down. They go offshore. Wealth moves out of the country. These are all generalizations of course, but the central points are correct I think.

Similar arguments usually apply, but do not always apply, to privatization of government function. Costs to the public nearly universally go up. The government gets a short term cash windfall, and a long term loss. I think of the analyzed privatizations only three were good, one in Brazil and two in Australia. The Brazil one was a dud government steel making business, which got privatized and made highly profitable. The other two I forget, except they were made during boom not gloom. Privatize in boom, never in gloom. In the UK they have had to renationalize at least one privatization involving the railways. Its a mistake to privatize business during economic gloom.

I think its possible for the US to lower spending, increasingly lower national debt, and shift spending to areas that boost growth. That is the key, boosting growth in ways that are sustainable long term. This can take money. Spending money on other things can have other gains, but do not help future finances.

Cutting back government spending without care destroys growth. Spending more without care wastes money. How its used is critical. Basic research is very profitable, but benefits are often not seen for many years.

 

[*GG*]

Hmmm, the Laffer curve is a heuristic, and quite irrelevant to the topic anyway. When the US was at its fastest growth it had 90% marginal tax rate, very high income growth, lots of research on non-commercial oriented science. Its not just about taxation rate, and more importantly marginal tax rate, its about what the taxes are spent on. You can spend them in ways that don't boost the economy, or do. If it boosts the economy then over time you actually make more money - compound growth beats money under the mattress. Its also the case that though the more wealthy paid more in tax on income, unrealized capital gains were HUGE. Wealth went up.

Cutting back government spending without care destroys growth. Spending more without care wastes money. How its used is critical. Basic research is very profitable, but benefits are often not seen for many years.

Have heard that income growth was high with 90% tax rate, but isn't this because we did not have a global economy then and America was one of the few countries that could produce the goods needed after WW2?

Need to look up what heuristic is. I am not an economist, but why would I want to work if the gov't took 90%! of my income (never mind near half)? Think I would rather subsist off the land and let them tax people who are willing to work, for the sake of working! Not with this illness!

GG

 

[alex3619]

Warning: yet more economic philosophy

People at the high end who paid 90% marginal tax were not taxed at 90%. That is a misconception. The tax is only on profit over a very high figure. Further, given that this was in conjunction with large income rises, and large capital wealth increases, the net wealth was on average growing. Its confusing cash with wealth.

Nobody really knows all the factors that were involved. Some of that was a big increase in research, creating huge economic opportunities. We understand pieces of the issues, but not the whole.

In any case the single biggest thing the world needs in economics is a shift to sustainable growth. Number two, and actually number one for the US in the short term in my view, is a reduction in debt. If things continue then at some point the US will go into catastrophic economic meltdown. Things will become dire.

Some might wonder why I am concerned about US issues. There are several reasons. First, the world is now connected. The US is a quarter of the world economy. If the US has a major economic meltdown, I don't mean the little one we are starting to recover from, then the whole world will have an economic crash. The second big factor is the US is only one example. We are addicted to debt. Small debt for business launches might be desirable, but I think in general debt creates economic instability. It creates risk. Every country in the world is exposed to that. Its not just a US issue.

Nobody has a good model for how to plot a good course for economies in the next century. What we do know is that current projections and models are wrong. Yet we have nothing to replace them, so we cling to them.

Research is a big factor in facing the future. As times become more dire we will see research retract. This is bad in the long term. I would like to see most advanced economies push a research agenda a lot more. I would like to see an end to the debt model. I would like to see failing companies fail, and not be bailed out. I would like to see a different reward model for bank and investment executives. I would like to see a reduction in what we consider a viable economic growth rate unless growth can occur without consumption growth.

All this requires oodles of research, especially a-dogmatic research. We need better ways, and that means we need to be looking for them.