New attempt to avoid releasing data on 'recovery' from PACE

There's also the problem with conflating what mathematicians would consider a fair 'normal range' for disability with what the lay media would understand when being told that treatments got patients 'back to normal'.

 

I think they shouldn't collapse it into a single value but have different thresholds based on people's different ages. Once they are defined, it does the same thing.
(Alternatively one could get a single value for an equivalent general/healthy population i.e. weighting it to make it like the study population).

 

As I've argued before, I have problems with people satisfying recovery if they are within a certain amount of the mean (or median): If they mostly have worse scores than the mean/median, it doesn't suggest they are a population like others in the general or healthy population.

So what I think needs to be done is to take people out of this recovered group, until one has a group where there isn't a statistical difference between them.

However, even then, one might still end up with groups that have lower means so continuing taking out individuals until the means match up with the general population might be a better way to go.

 

One way of summarising one of the points user9876 is making might be where one has a sort of cliff-effect where, say, there is a huge difference (in real terms) between a score of 70 and a score of 80 on a scale, with a much smaller difference between a score of 60 and a score of 70 (or 80 and 90).

My point on this: this might not be so important when comparing means of different intervention groups (but could be in some situations), but might be very important when trying to claim certain values represent recovery.

* These numbers were chosen at random

 

Not sure if I've said this before (in another thread), but this reminds of what I understand happened with a debate with TSH values, used to measure thyroid functioning. Normal values of up to around 5 were set. However these have had to be reduced as abnormal thyroid functioning is so common in the population that not all in the middle 95% have normal thyroid functioning.
Similarly one can't use 95% of the population in terms of functioning as so many have problems with health and functioning that couldn't be considered normal functioning.

 

One person wrote to me off list on this:

----------------------------------------------------------------------------------------

The person who sent me this added:

 

Yeah - I agree. This would matter less is the criteria for recovery were similar to what you mentioned in post #43, and it becomes increasingly likely to be a problem as looser criteria are used.

Ta for the other info too.

 

They do but the also quote the other measures. There are some interesting questions around the accuracy of the EQ-5d scale. But they are very different. Basically there are questions around the way they convert the different dimentions into a single utility value. A cost effective treatment in one country may not be cost effective in another because each country would have different weightings. Also I would argue that there is error in their estimate of the utility function which where there are only small benefits (such as reported with PACE) these errors may become significant. Its an argument that casts doubt on the reliability of the result.

There is an argument around it being in appropriate to calculate the mean and std form the general population distribution. Which in turn they use to calculate the normal levels.

As you say some statistitions complain but they fail to get traction but I think we should keep trying. To my mind it becomes particularly important as the gains are small.

Its very hard to design a good scale to me this suggests the need to quote individual values rather rather than lumping lots of stuff into an adhoc scale. The fatigue scale is much worse than SF-36.

I think the point about changes is that say you fill in the questionaire and come out at 60. Now you have some treatment and have a very small improvement. Now that very small improvement may allow you to change your answers to 3 questions hence getting a score of 75.This would be because a number of the particular set of questions chosen for the SF-36 scale fall around this region of physical activity.

Now a different person who came in on 50 may make the same improvement and only be able to change the answer to one question. Hence moving to 55. Again due to how the questions test different aspects of physical function.

So this means that the change is sensitive or to be accurate may be sensitive to the stating point.

Given a fair distribution of the groups over the scale it may not be important since these errors may cancel out. But I'm not sure about this. I seem to remember that the different groups in PACE also had differing starting means so the errors may become important.

The argument is though, one of accuracy rather than one suggesting their results are too high (I think this follows from an argument of some groups being giving a pychological framing that they better hence give better answers). The point about accuracy is that it should feed into significance tests although I have no clue as to how this would work.

 

I think there is a problem defining normal simply by taking a statistical model. Normal suggests that some mechanism is working correctly. If as Dolphin points out you just take an overall population you don't know if they all or most have a working mechanism.

The medical world seems not to take a great deal of care of how statistics are used

 

Sorry Esther. I don't think you'll like this answer (or non-answer). From the House of Lords:

 

Ta Fire. I do worry about the people who write those non-answers. Surely they have to hate themselves. I never would have expected anyone at the MRC or in government to be interested in making more public data so that patients could be better informed though.

 

Its not quite a non statement as it says that the MRC are aware that they are ignoring the original protocol. I don't know whether the lack of any reports of action of the MRC can be assumed to be agreement. Just the fact that the MRC knows should put political pressure on the MRC as debates happen about drug trial protocols. For example if the commons health select committee were to ask the MRC whether all the research they funded published the data as specified in the original protocol they would have to say no.

 

I think as a patient, I would want to see data for the alleged theory that 'Got ME? Get out and exercise!' and all the support that headline (I think it was a headline - or was very similar in tone) received from the authors. I think if I were to return to the answer afforded in the House of Lords - it would be a legitimate concern to express. If there is evidence that exercise can be as effective as it was sold to be - then we'd all like to see GET in place across the country. So 'put up or shut up'.

 

From January 2013, the BMJ says that it will require a commitment from trialists to make their data available on reasonable request.
Dr Fiona Godlee and Dr Trish Groves (BMJ Editor and Deputy Editor) have said that this decision has been made because "it is no longer possible to pretend that a report of a clinical trial is sufficient to allow full independent scrutiny of the results."

BMJ articles:
http://group.bmj.com/group/media/la...sts-believe-data-should-be-more-easily-shared
http://www.bmj.com/content/345/bmj.e7980


It seems that Ben Goldacre has been working with Dr Sarah Wollaston MP (Totnes) (Con), who has already raised this issue (access to trial data) in parliament, and has had a response (and an invitation to a meeting) from the govt. See the Hansard entry here:
http://www.publications.parliament....121023/debtext/121023-0001.htm#12102347000762

In the govt response to Dr Sarah Wollaston, the govt makes a reference to European regulations. The EU is said to be looking at some issues in relation to medical trials, but I'm not sure exactly what Europe is proposing, and they might not be proposing full public access to data. In any case, in a blog by Ben Goldacre, he points out that any future European regulations will not give us access to data from past trials, but it will only relate to future trials, so it doesn't help us for PACE:
http://www.badscience.net/2012/10/questions-in-parliament-and-a-briefing-note-on-missing-trials/

 

There is no reason not to release the data now, if they plan no further use of it themselves.

Failure to release all data after finishing an analysis is scientific fraud in my opinion, as claims arising from a single source data analysis cannot be independently tested.

Shitty, shabby, shonky stuff.

 

Queen Mary, University of London

Research Data Management Policy

http://www.arcs.qmul.ac.uk/policy_zone/research/QMUL_research_data_management_policy_2012.pdf

"Most grant applications for research
which will generate digital data sets require a data management plan that meets the
2011 Research Councils UK (RCUK) policy; this states that: ‘Publicly funded
research data are a public good, produced in the public interest, which should be
made openly available with as few restrictions as possible in a timely and responsible
manner that does not harm intellectual property:’..."
...
"RCUK now require all funded universities to have a data management policy and
road map that will meet their expectations for data sharing in place by 1 May 2012
with full implementation by 2015. This is to ensure we make it clear how publicly
funded data can be accessed for at least ten years after publication. Our policy
should deliver the following criteria:
• publicly funded research data should be made openly available in a timely
manner;
• data with acknowledged long term value should be made accessible;..."
...
"Policy
Where possible publicly funded research data should be made available for access and re-use."

 

A1 Sean.

If they don't release the entire (redacted) data set, the case for considering the entire trial to be scientifically inadmissable is open and shut as far as I'm concerned. Ignoring the parliamentary answers and all the rest of the spin, and the details of wording and timing of the MRC's policy, if they refuse to release the data then this 'study' has no claim whatsoever to call itself "Science". Period. "Appeal to authority" is rightly supposed to be a fallacy, ergo this kind of pseudo-scientific publication has no basis for its claim to be 'scientific' to any rational mind.

Here is the MRC's actual policy on open data:

http://www.mrc.ac.uk/Ourresearch/Ethicsresearchguidance/datasharing/Policy/index.htm

Note also the RCUK's Principles:

http://www.rcuk.ac.uk/research/Pages/DataPolicy.aspx

The original PACE funding approval was obtained just before the MRC's policy came in (conveniently enough). But PACE's funding has been renewed throughout the years since that policy was established 7 years ago. Queen Mary's are in possession of all the relevant raw data. Only their lack of political will prevents them from releasing that data. It's my opinion that the MRC would quite likely wish for them to release it - that would certainly be in accordance with the MRC's present policy. It's quite possible that public pressure might persuade the MRC to put pressure on Queen Mary's and the PACE authors to release this data. It's good that all this is now on the record in parliament, but a campaign to ask the MRC to demand the release of the data seems most appropriate to me.

We paid for the collection of this data: it does not legitimately belong to the authors or to Queen Mary's for them to spin it however they wish. It was publicly funded. The data rightly belongs to us, and only legal anachronisms have allowed them to keep it from us. If the PACE authors want to claim scientific authority, what do they have to fear from releasing their data? Why should they expect us to trust their biased interpretation of that data, and accept it as "science"? Do they not wish us to behave scientifically and follow the data? If not, why not? What legitimate scientist has anything to fear from the release of the raw data which they claim supports their published findings? If the PACE trial wishes to claim any kind of authority as "science" then the data must be released. If they insist on keeping the data secret, then let nobody argue that PACE has any legitimate claim to call itself "science". Biased commentary on secret data is not a bona fide scientific publication, however prestigious the journal that rubber-stamps it: without open data, in a case like this, I see nothing but a pseudo-scientific scam.

 

Can't go into detail but after some enquiries I have it that the recovery data is currently with an unnamed medical Journal, currently under peer review, and is then expected to be published. No doubt if it is then it'll recieve the same or more scrutiny as the data released so far.

I also have it that this situation was passed back to whoever made the FOI request for that same data, Hooper? Mar? Other??, as part of the reason for non disclosure but I haven't come accross any reference to that on this or other sites (at least as yet)

Can anyone corroberate or refute?

Thanks,

 

Holmsey: my understanding is that they have created a new criteria for 'remission' (details in first post of this thread), but that they do not want to release the 'recovery' data as it was laid out in the original protocol. It is this remission data which is now under peer review. This could have changed, but if so, I do not think anyone has been informed (there's a link in the first post to a web-page that should update info about about the FOI).

Hopefully they will have changed their mind following the FOI and publicity, and added the recovery data to their original paper, but I've not seen any evidence of this yet.

 

In their FoI response they state

http://www.whatdotheyknow.com/request/pace_trial_recovery_rates_and_po

So they do say they are publishing something they call recovery but not what they originally said recovery would be defined as. Of course the peer reviewers should insist that they publish rates as defined in the original protocol especially as the study was not blinded.

Expect a dodgy definition of recovery and some headline figures on the assumption people won't read beyond the headlines.