http://www.cabinetoffice.gov.uk/sites/default/files/resources/TLA-1906126.pdf
If you look at the optimization mentioned, such as with payment of fines in the court system, they got some positive results. However the outcome measure was court defined and specific. It is indeed possible to improve such measures, but what if we should be considering, for example, the disabled and their services and benefits? If you optimize for cash saved by that department, you will get results that look good on paper. However, the real question is whether the services better address the needs of BOTH government and the disabled.
The government works for society, and should be accountable to it. This is not what I see most of the time.
If disabled are forced off benefits, this might be a very bad thing not a good thing. For a start, are costs of constant appeals being taken into account? Appeals cost money.
Are medical costs being taken into account? A broke disabled person might not even be able to get to a doctor or hospital, or be otherwise unable to look after themselves adequately. Hence they might have more medical issues. On the one hand they might die, oh look, thats a big saving to government. On the other hand their health issues might worsen and take up more and more medical resources in an already overstretched system. Is that a saving?
What about legal issues? For example, a broke disabled person who has some physical capacity might turn to crime. This hurts society, and then court means the legal process is invoked ... at huge financial cost. Increase in policing and public relations costs also go with this.
Are secondary administrative costs being taken into account? How much extra paperwork is being generated at all levels of government including local councils?
What about secondary demands? They might put more pressure on other services, especially those involving food or accomodation. Are those costs taken into account?
A narrow outcome measure such as how much money is saved on disability payments has to be balanced against costs and benefits in other areas. Simply isolating that one figure and optimizing it might cost society and government MORE not LESS in the big picture. Is this being tracked?
How about intangible benefits? Public dissent is a factor in government policy. If government got things right more often they would not have to spend the huge fortune that is currently spent on public relations. The disabled would also be better served, and this would have a flow-on effect to their friends and family.
The paper also cites an example of RCTs saving lives, in this case the use of steroids in head injury. It had been presumed this saved lives, when in fact it increased the death rate. This is a valid use of an RCT, and is only one of several examples I am aware of. However the death rate is a clear issue, and the use of steroid therapy can be strictly defined.
They also stress that RCTs are a good way to show value for money. That this can save money I have no doubt, the question is about value. Saving money for identical or better results is great, saving a lot of money for slightly worse results might be justifiable, but outcomes often include intangibles that are not factored in, and other issues that are deliberately ignored.
Money is often at the heart of this. Money is easily represented as a number. How about social cohesion, justice or fairness? How do you enumerate those?
Step two of the test recommendations is where I have my major issues: "Determine the outcome that the policy is intended to influence and how it will be measured in the trial." A mistake here can lead to much worse results instead of better ones. For example, in an education policy they comment the results might be examination results. This then ties the test to an interpretation of what the exam results mean. Its not just about numbers. Better test scores in a more poorly educated student population are a risk of teaching to tests. The overal education strategy has to be more broad than just test results.
What RCTs can do is essentially the same as what pilot studies do, but offers a methodology to combine multiple pilot studies and compare results. Suppose there were three proposed small changes in education policy, all of which are optimized to exam results. An RCT with a control group would enable you to compare these policies. Its additional information to policy makers. All that information is useful, but the results might tend to be over-emphasized on the grounds the results are more "scientific" than other analyses. It could also be spun that way to push an agenda.
Step 7 discusses other issues, including secondary measure and qualitative data. Such measures can be useful in helping to interpret the study, and to some extent, for well designed and appropriate studies, will mitigate many of my concerns.
I assert again, to go down this road we first need accountable and transparent government.
The Test, Learn, Adapt strategy I have no problem with, in fact I will be promoting similar ideas. Its the implementation and how reliable such studies are considered to be that are the worry, particularly if the raw data is hidden and for issues driven by purely economic considerations.
Bye, Alex