• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

BBC Radio 4 - Ben Goldacre's - Bad Evidence

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
I don't suppose this will be of interest to everyone and it's not relevant to our condition. Just something that might be worth listening to:

Medic and author Ben Goldacre explores the idea of evidence-based policy and asks if it can ever become a reality in the UK.

In medicine, how do we know if a particular treatment works? The simple answer is to subject it to a fair test against other treatments or a placebo. So far the best example of a fair test in medicine is a randomised controlled trial or RCT.

Often to referred to as 'the gold standard' when it comes to determining what works, RCTs are now commonplace in business. But what about government? The idea of evidence-based policy is hardly new - it's what social scientists have been banging on about since the 1960s. But in practice, when evidence has been used to determine policy, it's often been anything but 'gold-standard'.

In this programme, the medic and author of Bad Science, Ben Goldacre, sets out to explore the potential for putting RCTs at the heart of the policy-making process, arguing that not only can they reveal if our existing policies are effective but RCTs have the potential to transform the way we create and implement social policy across the country, from education to health, from welfare to crime.

Of course not everyone agrees that all you need is hard data to make the best policy. Experience, values, ideology - these, say critics should never be abandoned in favour of cold statistics. And whilst the RCT may work well for pills and potions, it's too blunt an instrument to deal with the subtle and complex challenges of assessing how best to punish crime, treat drug users or teach children from impoverished background to read and write.

Just look at the recent fiasco over badger-culling. over a ten-year period, randomised experiments and pilot studies have resulted in no clear policy on how to prevent the spread of bovine TB. And then there's the ethical question - how for example could you allow randomisation to determine something as morally (and politically) sensitive as sentencing criminals, let alone teaching kids?

What is clear, is that bad policies cost us dear - both socially and economically. The challenges are many but the potential, argues Ben, could be truly transformational, both for society and for government.

8pm for 40 minutes on 1 January 2013: http://www.bbc.co.uk/programmes/b01phhb9
 

user9876

Senior Member
Messages
4,556
I find the whole notion rather nieve and worrying. I think it would be really challenging to design a trial that controls for many different factors and gave a view on long term effects. Its hard enough with a drug trial. EBM is failing in some ways in that drug trials are often too short and data about side effects (especially long term ones or side effects that could also be illness based) is lacking, To extend a troubled methodology to a much more complex situation seems like a huge mistake. That said I'm not against trials of social and political interventions but people should be careful about drawing strong or definative solutions.

To me calling a randomised controled trial a gold standard is farsical. We need to understand mechanism. A physicist would not just carry out an experiment comparing two things. They would try to form a theory around experimental results and explain it in terms of what other theories exist. This is an essential part of science. So to stop at the point of some experiment with bad control and inaccurate measurement systems just seems wrong.
 

PhoenixDown

Senior Member
Messages
456
Location
UK
...We need to understand mechanism. A physicist would not just carry out an experiment comparing two things. They would try to form a theory around experimental results and explain it in terms of what other theories exist. This is an essential part of science. So to stop at the point of some experiment with bad control and inaccurate measurement systems just seems wrong.
Adjusting policy based on poor studies which are later dogmatically tauted as "well conducted", bad science? bad evidence? - Bad idea.

PS: I'm still waiting on Ben to properly bash the PACE trial, he seems so eager to bash everything else yet he ignores the poor & misrepresented science that's right on his doorstep.
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
Well I'm interested in hearing where the discussion will lead. Take Welfare Reform in this country driven (in my opinion) by an assumed need to make savings and an overriding ideology that 'work pays' in terms of health and social growth.

Yet to what degree were the decisions to introduce new assessments and the whole 'migration' reassessment across several health related benefits actually tested objectively? There is a lot of ideology used to direct the decisions that were taken - to justify them and yet when a new Government comes in these measures could be repealed or altered - not because they are not working but because of a new ideology.

Labour have already declared they will repeal the Health and Social Care Bill without giving due consideration to assessing objectively how the introduction of e.g. GP commissioners might have impacted on the effective delivery of health care.

When the Welfare reforms were announced, there was a book, a study, that appeared to provide the rationale - the ideology - for these measures. It was about (paraphrasing with bias) ensuring that people who could work, worked, because work was the best way to ensure health and social growth. Work is good. Work is all.

As an ideology I tend to agree - though it said very little about personal choice or how e.g. bad health can rightly effect this drive. Indeed, the tome did not consider the effect on a e.g. disabled person who is driven to work unfairly. This tome was proclaimed as the rationale for change and whilst I don't think ideology, or opinion, should be excluded from any decision making in Government - if the opportunity for RCT is there then why not take it?

Why not undertake an RCT to explore the e.g. financial costs and personal costs involved in reassessing someone who has been on invalidity benefit for X number of years; in migrating them to a new benefit; in assessing the effectiveness and fairness of the process; of following former claimants who are found 'fit for work' to see if they can find a job and hold onto it and/or if they are forced to return to benefits due to their health; to see what effect all of this upheaval has on a person's life before during and after etc. etc.

These kind of studies are rarely undertaken. This kind of consideration rarely happens. One reason is probably, time. Each Government has (now) 5 years. Each Government wants to stamp it's mark and reward it's supporters by at least trying to follow through on what it proclaims in it's manifesto. And yet we have a civil service that exisits largely intact throughout the change in Government. So such studies are possible I would suggest.

Is there though a willingness to complete RCTs - assuming they are possible - to complement existing methods of assessment for Government policy? Hopefully the debate might explore these issues.

n.b. The tome I refer to above does exist - at this moment (and early morning time) I can't link you to it but I'll return and insert if I remember and can of course find the darn thing...
 

barbc56

Senior Member
Messages
3,657
I find the whole notion rather nieve and worrying. I think it would be really challenging to design a trial that controls for many different factors and gave a view on long term effects. Its hard enough with a drug trial. EBM is failing in some ways in that drug trials are often too short and data about side effects (especially long term ones or side effects that could also be illness based) is lacking, To extend a troubled methodology to a much more complex situation seems like a huge mistake. That said I'm not against trials of social and political interventions but people should be careful about drawing strong or definative solutions.

To me calling a randomised controled trial a gold standard is farsical. We need to understand mechanism. A physicist would not just carry out an experiment comparing two things. They would try to form a theory around experimental results and explain it in terms of what other theories exist. This is an essential part of science. So to stop at the point of some experiment with bad control and inaccurate measurement systems just seems wrong.

EBM fails in many ways. Science based medicine goes a step further and this is the philosophy that scientist like Ben Goldacre follow. SBM takes into account factors such as plausibility, the design of the study, ethical considerations as well as the present knowledge base of how the world works.

A humorous example of EBM is a "mock" published RTC experiment to see if parachutes prevent death when you jump out of a flying plane. Of course we know they do. Not just intuitively but also because we're taking into account the laws of physics. Plus throwing people out of a plane without a parachute would make this type of experiment unethical.
Good science is the best and only way to determine which treatments and products are truly safe and effective. That idea is already formalized in a movement known as evidence-based medicine (EBM). EBM is a vital and positive influence on the practice of medicine, but it has limitations and problems in practice: it often overemphasizes the value of evidence from clinical trials alone, with some unintended consequences, such as taxpayer dollars spent on “more research” of questionable value. The idea of SBM is not to compete with EBM, but a call to enhance it with a broader view: to answer the question “what works?” we must give more importance to our cumulative scientific knowledge from all relevant disciplines.
http://www.sciencebasedmedicine.org/index.php/about-science-based-medicine/

This is the short version of SBM as it goes a lot deeper and is much more complicated than what I have written above. It's a fascinating subject.

If you would like more information, I would recommend the following websites as a start.

http://norvig.com/experiment-design.html
http://www.sciencebasedmedicine.org/


Barb C.:>)
 

Marco

Grrrrrrr!
Messages
2,386
Location
Near Cognac, France
Surely Goldacre must be aware that evidence based policy making has been nominally embedded in government (and public service) policy making for decades now.

However 'evidence' is intended to 'inform' political decision making. At the end of the day political decisions are taken for 'political' reasons that must weigh up all sorts of intangibles that the 'scientific method' can never hope to capture. We may not like it but these 'intangibles' include considerations of what an individual politician/senior civil servant/health trust manager or whatever believes to be in their own best interests.

Taking a hypothetical example, a review of the evidence (assuming any review can ever claim to have fully captured all the relevant 'facts' and therefore represent the 'truth') clearly shows a growing epidemic of childhood obesity in western economies and that diet is a major contributory factor with the ready availability of 'fast food' of primary concern. Furthermore the estimated public costs associated with this 'epidemic' can be calculated and projecting forward 10-20 years suggests that the financial burden on national healthcare systems is likely to become unsustainable.

The evidence based policy implications are likely to include a recommendation that the government should endeavour to discourage the consumption of 'fast food' using methods already used to control substances such as tobacco e.g. through increased taxation; banning advertising; blank packaging or packaging containing graphic images of clogged arteries; age restrictions on the sale of 'fast food' etc.

Those are the 'realities' implied by the 'evidence'. The political realities are that politicians and political parties exist to retain power which works on roughly four year cycles. Politicians are less concerned with 'doing the right thing' with consideration to what is likely to happen in ten to twenty years time than with avoiding committing political suicide in the immediate future due to unpopular policies.

Unlike tobacco use, there is no popular consensus that 'junk food' is anything more than a harmless occasional treat and no politician is likely to introduce the measures suggested by the 'evidence base' and face the wrath of many parents who can only survive the weekly shopping trip by 'bribing' their kids with a trip to the local 'McBurgerKFC'.

Additionally, who would want to enter politics if your only function in the decision making process is to rubber stamp the recommendations flowing from a cadre of technocrats churning out endless RCTs that in all likelihood are so constrained that they fail to capture anything approaching the complexities of real life?

'Evidence based policy making' may sound great in theory (actually to me it doesn't) but its a more than a little naive philosophy that really doesn't bear any relationship to how things work in real life.

Warts and all I prefer the current imperfect system to the alternative :

And the meek shall inherit the earth...

We've taken care of everything
The words you hear, the songs you sing
The pictures that give pleasure to your eyes
It's one for all, all for one
We work together, common sons
Never need to wonder how or why

We are the priests
Of the temples of syrinx
Our great computers
Fill the hollowed halls
We are the priests
Of the temples of syrinx
All the gifts of life
Are held within our walls

Look around this world we made
Equality our stock in trade
Come and join the brotherhood of man
Oh what a wide contented world
Let the banners be unfurled
Hold the red star proudly high in hand

Sing along!!
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
It's the balance - the cumulative - that I think he will be talking about or rather how RCT's and Evidence Based could/should hold greater sway when it comes to policy direction and implementation; as well as in following up on policies that have been taken.

Anyway, will wait to hear how it all goes. It may be that as you said Marco, these practices are embedded in the system and we mere mortals do not get to hear about them as much. Indeed we might be victims of emotional bias embedded in the media or in that we only really see the 'discussions' that our politicians engage in in their relative parliaments.

I am sure there is more to decision-taking than meets the eye. I am not blind to the notion that what David Cameron says, goes. But I do think more can and should be done to better ensure outcomes. I think ideology among the political parties features too much in the actual policies themselves with little notice taken of the impact of those decisions. And if RCTs or more careful consideration or existing and proposed systems can be included as a matter of course - to inform the ill-informed; including the public - I'd be in favour of it.

All comes with a financial and time cost of course and these considerations might prove impractical but for instance, in the UK more and more is being heard of from 'think-tanks' and it may be that these are a suitable vehicle to carry out such studies (though of course they themselves are often biased).
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
Evidence based management has been a reality for some time. NICE is an example, though still within medical boundaries. EBM uses RCTs, but to be truly gold standard they must be double blinded.

There are so many problems with using evidence based approaches in politics that I am worried its subject to serious abuse. One highly possible issue is that it would be subject to severe manipulation and bias in anything really complex in real world politics/management/economics etc. It would also involve massive social experments of huge amounts of time, with huge embedded costs, to run even a few experiments.

I am not saying it cannot be done. I am saying its something that would require huge consideration and investigation before rushing into it.

The whole evidence based approach relies on being able to define and experiment with the situation in some way that can be reduced to numbers. This is massively problematic in its own right Taking this a step further to a political equivalent of science based medicine is verging on the absurd. Many, if not the vast majority, of the "rules" used in management, politics and economics, to name just several disciplines, are no better than heuristics and are themselves invalid. To proceed down this path would mean making these disciplines truly scientific.

Having said that I think we can make these things into more rigorous disciplines, with enough time and effort and will to do this. Do we have that kind of time though?

One of the biggest issues with RCTs and "science-based" approaches that rely on simple principles is that it all presumes that complex real world issues can be dealt with using simpe reductionistic approaches. The answer has long been known: its severely impractical to do so.That doesn't mean we can't improve existing systems.

A big problem in the UK, and I am fairly sure this is the case almost everywhere, is that accurate and reliable data on many of these issues is not produced and made available to the public. To run a scientific program on political issues would first require that democratic politics ceases to exist. You could do it with a totalitarian society to probably horrendous outcomes. George Orwell would have been horrified.

A simpler option, that is already crossing so many lines it would be very hard to manage, is to push for both transparency and accountability in government. It requires a reinvigoration of the fourth estate, or perhaps the fifth estate. It requires that every aspect of government decision making be available to the public. No more cover-ups. No more PR/spindoctoring. FOI requests? Not necessary, its all available for perusal on demand, right down to which bureaucrat made which decision on which day, minutes of every meeting, etc.

Treating political policy as scientifically testable entities for RCT purposes is a nonsense. Arguing that we need raw data freely available, and a knowledgeable and educated population to review that data, is doable. Even that however will be resisted bitterly. It would put a light on corruption, incompetance and scandal. Yet I regard this as just one necessary step to even begin to make political/managerial/economic issues scientific.

I could write much more, and probably will later, but the whole notion has so many problems that just to sort out what they are would take decades of debate and research, and probably the creation of many dedicated research institutes.

My best guess is that any program of evidence based policy would run into the same issues as EBM. They would become justification to rubber stamp government decisions and have little or no basis in fact, despite reams of impressive data.

We cannot even get transparency and accountability for the PACE trial, and that is at least nominally "scientific"! Where is that raw data? What are the real reasons for many of the dubious decisions made in the course of the study?

You cannot redefine society as a machine. I thought we gave up on such notions most of a century ago.

Bye, Alex
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
RCT's are better than anything else in my humble and layman's opine. But (as I said earlier on another thread) RCT's are not exempt criticism. If someone was to design and complete another (better) RCT for CBT, GET, and Pacing we might have something to directly compare to PACE. We don't. But that doesn't mean it cannot be critiqued or that those behind that study should not be made to produce all the data used (that is not patient sensitive) especially as it was funded by public money. Hell, this morning, I can't even recall if PACE was blinded!
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
You can't blind something linke PACE Firestormm, and double blinding is not even a consideration.

What number do you put on justice? Love? Liberty? Freedom? Fairness? Compassion? How about faith? Values are still outside the scope of issues that can be reduced to factors that can be covered in RCTs. When management tries to become "scientific" it runs huge risks. In order to even assess those risks, to make these issues accountable, everything has to be public and the public needs the resources to scrutinize these issues. Is it possible? I think so. Is it practical? I highly doubt it. Is it desirable ... I am nearly certain the answer is NO.

Can policy making be improved? Yes, indeed, starting with transparency. All the data needs to be available so everything can be checked by anyone. To have dedicated people doing that first requires fixing the fourth estate ... its dying.

Can policy be reduced to principles that can be tested using RCTs? I severely doubt it to the point of almost certainty. Whatever numbers you use for outcomes will severely affect the result, and the interpretation of the result. I can however see this being used to set society down the path to totalitarian rule, in our own good of course, the numbers say so.

The problem is the same as with statistics and even logic. Its not the math that is in doubt, its the translation of the real world into math, and the translation of results back into the real world.

In the interest of transparency I should note I am looking at a way to make EBM and BPS more valid as scientific disciplines. So I am starting from a position that is different to Ben Goldacre's. I will expand on this in time. I have years of analysis ahead of me.

I think any and every discipline can be made more rational. I do not agree this means wholesale adopting scientific methods to disciplines it doesn't fit. Shall we put religion to experimental testing. Bzzzzzzzz ... all fail. No proof. Goodbye. Yet its been obvious to philosophers of science that many aspects of religion are simply not scientifically testable. Its not a valid discipline for reductionist and traditional science. Nor is politics or economics or management.

Need I point out that social darwinism and the eugenics movement were based on scientific hypotheses? What kind of society would they have created?

Bye, Alex
 

Marco

Grrrrrrr!
Messages
2,386
Location
Near Cognac, France
There is also the tendency to assume that we know with any certainty more than we do whether it be in science, medical science or any of the myriad aspects of a complex society on which public policy impinges.

There are hundreds of thousands if not millions of economists worldwide who have been doing this sort of data collection, crunching and modelling for many decades.

Economic data collection is much more established, codified and rigorous than that found in most other fields (although still problematic) from which economists can measure, correlate and devise multi factor regression equations to 'model' what they expect to happen in the short, medium and long-term. All very scientific you would imagine despite the fact fact many of the terms entered into the regressions are often no more than a finger in the air gut feeling. Policy makers would though be deeply impressed by the pages of calculus.

Despite the complex mathematics of course the resulting projections can really only be stated in terms of statistical likelihood and any policy proposals do also have to consider issues such as deadweight, substitution, and 'unintended consequences'. The latter by definition can't be forecast.

Despite the long tradition and complex modelling techniques how many of these professional economists or 'think tanks' forecast the 'credit crunch' and current recession?

On the other hand any astute layman with any sense of history could have seen a classic unsustainable 'bubble' as clear as day.

I'm afraid life is much too complex to be accurately modeled. A single random event can have widespread ramifications that explode any notion of 'predictability'. Sometimes what's really needed is someone to take a little leadership and take a decision - for better or worse.

PS don't getting me started on targets and monitoring. There's an apocryphal tale that, under the old soviet union, the ministry of production (or whatever) was concerned about productivity coming out of the engineering works and decided to introduce rigorous and binding targets.

First they introduced targets that required a certain weight of ball bearings to be produced and the plants produced a few huge bearings that nobody could use but which enabled them to easily meet their target. The next target specified numbers so the plants produced millions of tiny bearings that again no-one could use. They then specified a certain number of bearing of a certain size and the plants complied but the quality was rubbish ...... etc. You can see where this is going?

Targets will always be manipulated despite all best efforts whether it be 'triage' in the NHS or any other attempt to manage complex systems on a 'rational' basis.
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
Ideology cannot be taken out of politics unless we first redesign humanity. The whole notion of society as a machine was rejected so long ago, for many good reasons, that I would despair if it resurfaced and became acceptable.

Nor should ideology be taken out of politics. Ideology reflects values, values are a the core of what kind of society we want, and what is acceptable to get there. If we were able to translate science to policy making, I doubt we would make politics more scientific by any substantial amount, but we would succeed in making science more political.

The trend to make science political is already here though, and I regard it as the bane of rationality, democracy and maybe even civilization itself. Science should not be political. Politics should not be scientific. Both can however be more rational.
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
You can't blind something linke PACE Firestormm, and double blinding is not even a consideration....

Wasn't it 'blinded' in the sense that SMC was the control? I still believe that it could have been blinded better. I dont' believe that blinding was beyond the scope of PACE or that PACE was not in the way I have said 'blinded'. A stretch? Yes but only in comparison to other trials. I think it a valid criticism of PACE that is wasn't better blinded.

As to the rest, 'love' etc. well we were talking about RCT's being not the be-all-and-end-all above. I think I shall wait to hear what the programme has to say about it all.

Figured it would be interesting is all.
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
Marco, I agree with you about economics. Its an example of math-based social policy making, with business as part of society. Its impossible to reduce all of this to numbers, but it is possible to use it to better inform decision makers.

Having said that economic crises keep coming and are often not forseen and predictable. With all that data, all those dedicated computers to analyzing this, institutes, businesses, government all spending huge sums of money and dedicating armies of specialists ... and they still can't get it right. Every government fails. The economists fail. Its not a perfect system accountable to perfect analysis, nor is it easily reduced to factors that can be manipulated by experiment in a really scientific fashion.

It would be easier to predict the weather, right down to how many drops of rain will fall on a given patch of rooftop on a given house in a given minute, than to make economics truly scientific. Yet we cannot make politics scientific if we can't get a good grip on economics.

As I have already stated though, if the argument is generalized to make government policies more rational, then things can be done.
 

alex3619

Senior Member
Messages
13,810
Location
Logan, Queensland, Australia
Wasn't it 'blinded' in the sense that SMC was the control? I still believe that it could have been blinded better. I dont' believe that blinding was beyond the scope of PACE or that PACE was not in the way I have said 'blinded'. A stretch? Yes but only in comparison to other trials. I think it a valid criticism of PACE that is wasn't better blinded.

http://en.wikipedia.org/wiki/Blind_experiment

"A blind or blinded experiment is a scientific experiment where some of the people involved are prevented from knowing certain information that might lead to conscious or subconscious bias on their part, thus invalidating the results."

If you know what treatment you are getting then its not blinded. Period. To be double blinded then even the researchers cannot know.

SMC was a control. Thats something else. So far as I recall the patients were randomly allocated too. So its a randomized controlled experiment, but not blinded. There is however doubt about the value of the control group chosen and other important aspects of the experimental design and implementation. At the very least PACE needed a control group with NO treatment.
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
Marco. What about after-the-fact trials? You know how the newly elected party takes great strides to de-construct what has gone before - education being one area I think regularly gets a makeover with each new Minister. How about employing RCT's to look at existing policy in order to see how effective it really is? Or at least taking a more objective and quantitative approach to such things?

Economics is a very large driving force especially in today's climate. It was after all the driver being the austerity measures that kicked off the welfare reforms. But. When this Government decided to reform the welfare system they began with a decision to slash £20bn from the expenditure (figure from memory). Then they determined how they'd achieve it. Then they determined 'work is good'. I couldn't help but feel the cart was before the horse.

Something has to kick off the policy change I agree. Ideology for example. Or an accounting measure. We now see more care (not enough IMO) being taken to at least try and involve members of the public and organisations in 'public consultations' although the effect that these views have on policy is not known.

What we don't see (might be there might not) is the 'model' behind the policy. The mechanics. Transparency - as I think Alex referred to above - might help us in this but whether or not it would lead to greater care being taken in the thinking before acting and acting with a better appreciation of the likely effects - is doubtful.

Greater regard to RCT's might do this but only of course if the results were published and the reasons for not following/following the results were explained. RCT's as I have said are not THE answer but they could be better used.