• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Freedom of Information Request for 6MWT data for "recovered" in PACE Trial turned down

Tom Kindlon

Senior Member
Messages
1,734
https://www.whatdotheyknow.com/request/6min_walking_test_results_for_re#comment-42596


6min Walking Test Results for Recovered Patients in PACE trial


Anna Sheridan made this Freedom of Information request to Queen Mary, University of London

Queen Mary, University of London did not have the information requested.

--------
From: Anna Sheridan

28 March 2013

Dear Queen Mary, University of London and the PACE Trial Management Group

I have been following the PACE trial with interest for a number of years, both as an ME sufferer, and as the relative of two clinical psychologists involved in administering CBT.

I am particularly interested in the results of the 6 min walking test,(as reported in the original PACE paper) and would be grateful if you could provide me with mean and standard deviation results for those who, in your latest paper: 'Recovery from chronic fatigue syndrome after treatments given in the PACE trial.' are considered recovered. Please break these down, if possible into therapy type.

Thank you in advance

Yours faithfully,

Anna Sheridan (Dr)

-------

Queen Mary, University of London

28 March 2013

We acknowledge receipt of your request and will respond as soon as we can.

--------

Queen Mary, University of London

29 April 2013

Dear Dr. Sheridan

Thank you for your email of 28th March requesting information about the PACE trial.

I can confirm that the information you have requested is not held by the College.

A correspondent to the journal 'Psychological Medicine' has recently made a similar point about these data in a letter that will be published and the Principal Investigators of the PACE trial have addressed this in a reply, which is in press.

If you are dissatisfied with this response, you may ask the College to conduct a review of this decision. To do this, please contact the College in writing (including by fax, letter or email), describe the original request, explain your grounds for dissatisfaction, and include an address for correspondence. You have 40 working days from receipt of this communication to submit a review request. When the review process has been completed, if you are still dissatisfied, you may ask the Information Commissioner to intervene. Please see [1]www.ico.gov.uk for details.

Yours sincerely

Paul Smallcombe
Records & Information Compliance Manager

References
Visible links 1. http://www.ico.gov.uk/

--------

Epson left an annotation (24 August 2013)
They may not already have the data in the form requested, however they do have access to data that would allow for the information requested to be provided. Their response seems somewhat misleading to me.
 

Tom Kindlon

Senior Member
Messages
1,734
I can't say I find this response very satisfactory: "I can confirm that the information you have requested is not held by the College."

Anna just said on Twitter it was frustrating as their letter wasn't out till after the 40-day appeal period was up (see: https://twitter.com/Annakwood/status/374585141508968448 ).

One can see the PACE Trial investigators' letter on the recovery paper here: http://www.meassociation.org.uk/?p=16209 - it doesn't give the data - no major reason (they may only have 6MWT data for 72% of those that did GET, but they still published that data in the Lancet paper).

People can comment underneath the FOI request if so inclined.
 
Messages
5,238
Location
Sofa, UK
I can't say I find this response very satisfactory: "I can confirm that the information you have requested is not held by the College."
I think this gets quite complicated quite quickly. One of the standard FoI exemptions is where the information is not held. However, in this case, the underlying data clearly is held, and using that data the analysis could certainly be performed to provide the information requested. But any such analysis would require work, so should FoIs be expected to cover all possible (or reasonable) statistical analyses of the raw data, or the raw data itself, or what? My answer: the question demonstrates that a credible concept of the scientific method demands open data, even if that requires planning for anonymisation of source data from the outset as a matter of course.

The raw data cannot all be released because the entire raw dataset contains information that could identify individuals. But despite protestations from MRC staff who I spoke to at an Open Data conference last year, I am not at all convinced that in a study like PACE it is not feasible to release an entire anonymised dataset. On the contrary, I think that the release of such anonymised datasets should be the minimum standard that should be required for scientific credibility.

PACE brings this issue into sharp relief for those who are paying close attention to it. It is no more than scientific spin to leave the entire dataset - publicly funded - in the hands of researchers with a clear vested interest in the outcome, and to allow them to determine the protocols for what analysis is published. They are not publishing this crucial data that has been requested, nor are they publishing any analysis of the data that would answer this very pertinent question. They are, quite simply, given free reign to spin the data however they choose. They can decide in great detail what questions they will - and won't - answer, and they are analysing it in ways that enable them to say the things they want to be able to say. They have obvious conflicts of interest and lifelong careers invested in the success of what they claim to be studying. This is not, by any stretch of imagination for any intelligent observer, worthy of the name 'Science'. It is simply an exercise in spin. That their conclusions are so pathetic even after this exercise in spin is a damning indictment of the alleged 'treatment' on which their careers are based.


Anna just said on Twitter it was frustrating as their letter wasn't out till after the 40-day appeal period was up (see: https://twitter.com/Annakwood/status/374585141508968448 ).
Very much so, and they seem to have played a disingenuous game to say the least. I would think there should be a basis for an appeal, or a fresh request, based on this timeline. It's a strong argument just in terms of raw facts of the timeline if it could be laid out clearly. Similarly with most of the other FOI requests: they keep on arguing that they are going to present the answers some time, some day...but they never do, and surely eventually when this is all pointed out to an FOI tribunal, an unbiased judge should be able to see that this is not on.
 

Tom Kindlon

Senior Member
Messages
1,734
I think this gets quite complicated quite quickly. One of the standard FoI exemptions is where the information is not held. However, in this case, the underlying data clearly is held, and using that data the analysis could certainly be performed to provide the information requested. But any such analysis would require work, so should FoIs be expected to cover all possible (or reasonable) statistical analyses of the raw data, or the raw data itself, or what? My answer: the question demonstrates that a credible concept of the scientific method demands open data, even if that requires planning for anonymisation of source data from the outset as a matter of course.
Ok, but one is setting a low standard to escape FOIs if a holder of data can claim that means (averages) aren't held when they hold the raw data. One doesn't need to be a mathematician or statistician to do this - somebody could do this by hand. Having to calculate a mean is only a tiny bit more than a request to ask "how many of X happened/whatever" (SDs can easily be computed also with a small bit of help from a computer as you know).

Anyway, hopefully somebody will request the data.
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Mark

There is legal guidance about when data should be released if it isn't stored in the exact format asked for in a FOI request.
I can't remember the exact details, but I think it's defined in the number of hours needed to convert the data.
If it can be converted within a certain number of person-hours, then not having the data in the exact format requested is not an adequate reason for denying the data.

As for making a fresh FOI request for the same data, that is always possible. If the other party refuses to release the data, that is only the beginning of the process. The next step of the process is to take the request (make a 'complaint') to the Information Commissioner's Office (ICO). The ICO will then decide, independently, whether the data should be released or not. After that decision is made, either party can appeal to a tribunal service, where a judge will make the decision.
 

Sean

Senior Member
Messages
7,378
PACE brings this issue into sharp relief for those who are paying close attention to it. It is no more than scientific spin to leave the entire dataset - publicly funded - in the hands of researchers with a clear vested interest in the outcome, and to allow them to determine the protocols for what analysis is published. They are not publishing this crucial data that has been requested, nor are they publishing any analysis of the data that would answer this very pertinent question. They are, quite simply, given free reign to spin the data however they choose. They can decide in great detail what questions they will - and won't - answer, and they are analysing it in ways that enable them to say the things they want to be able to say. They have obvious conflicts of interest and lifelong careers invested in the success of what they claim to be studying. This is not, by any stretch of imagination for any intelligent observer, worthy of the name 'Science'. It is simply an exercise in spin. That their conclusions are so pathetic even after this exercise in spin is a damning indictment of the alleged 'treatment' on which their careers are based.

It is a total sham. The most basic requirement of all in science is that your data and methods must be absolutely transparent for others to review. Anything less is either incompetence, laziness, or fraud.
 
Messages
5,238
Location
Sofa, UK
Thanks Bob, I didn't know that about the number of hours needed to do the processing. It did seem like a rather grey area to me and I never read anything about that question before. Defining it in terms of the number of hours required to process seems about as good a rule as I can imagine to address this issue.

So it's clearly important to be smart about this in formulating FOI requests - making each individual request as simple to calculate as possible should cut off this kind of excuse. For example, if it were possible to specifically request data that didn't require any intelligent processing in order to provide it, but which could be processed later to provide the information, maybe if this were done exactly right they would not be able to give the excuse that the data 'isn't held'. In effect, work out which columns in the existing database you can ask for to get the desired data, and don't ask them to do any calculations on them because they could argue it would take them too long. Having said that, I think this request was probably as simple as it could be: I guess the calculation they applied to determine who was supposedly 'recovered' would be way too complicated to determine for oneself. So asking for the 6MWT data for those specific individuals was a really good question IMO and I can't think of a way to improve on it.

It really is an absolute travesty of science that they are allowed to guard their data for years and massage it however they see fit, and nobody else is allowed to see that data and perform analysis on it. They can spin it however they like, and nobody can counter it because only they have the data. To me, it makes their publications null and void as supposed 'scientific' evidence. They should be inadmissible as 'science' for this reason alone. It would be bad enough with privately funded 'science'; for publicly funded science it is just inexcusable. The fact that it is possible for them to get away with this behaviour brings science into disrepute, and shame on the scientific establishment that it allows this sort of thing to go on.
 
Messages
5,238
Location
Sofa, UK
It is a total sham. The most basic requirement of all in science is that your data and methods must be absolutely transparent for others to review. Anything less is either incompetence, laziness, or fraud.
I agree completely Sean. However, modern science almost universally fails to meet these most basic of requirements. I often struggle to find the words to sum up this situation. The closest simple summary I can get is to say that science, as I understand it, is not really practiced at all any more - at least, not in many fields. I think the physicists generally do actual science, and anyone who releases their full datasets has a strong claim to be doing science, but the rest of it is little more than spin, when you think about it - if their data isn't open, all they really have is an appeal to authority and a polite request to trust them and treat them as if they were (mythical) impartial observers. I think the traditional emphasis on published papers has obscured the reality that the truly valuable output of scientific study is the data, and objective analysis of that data is only really possible if the data is open so that everyone can analyse it for themselves. Until it gets over that hurdle, much of academia (PACE included) will remain more like a religion than a science as far as I'm concerned.
 

user9876

Senior Member
Messages
4,556
Thanks Bob, I didn't know that about the number of hours needed to do the processing. It did seem like a rather grey area to me and I never read anything about that question before. Defining it in terms of the number of hours required to process seems about as good a rule as I can imagine to address this issue.

So it's clearly important to be smart about this in formulating FOI requests - making each individual request as simple to calculate as possible should cut off this kind of excuse. For example, if it were possible to specifically request data that didn't require any intelligent processing in order to provide it, but which could be processed later to provide the information, maybe if this were done exactly right they would not be able to give the excuse that the data 'isn't held'. In effect, work out which columns in the existing database you can ask for to get the desired data, and don't ask them to do any calculations on them because they could argue it would take them too long. Having said that, I think this request was probably as simple as it could be: I guess the calculation they applied to determine who was supposedly 'recovered' would be way too complicated to determine for oneself. So asking for the 6MWT data for those specific individuals was a really good question IMO and I can't think of a way to improve on it.

It really is an absolute travesty of science that they are allowed to guard their data for years and massage it however they see fit, and nobody else is allowed to see that data and perform analysis on it. They can spin it however they like, and nobody can counter it because only they have the data. To me, it makes their publications null and void as supposed 'scientific' evidence. They should be inadmissible as 'science' for this reason alone. It would be bad enough with privately funded 'science'; for publicly funded science it is just inexcusable. The fact that it is possible for them to get away with this behaviour brings science into disrepute, and shame on the scientific establishment that it allows this sort of thing to go on.

Maybe there should be a question about the form the data is held and its schema as this would give an insite into the time taken to write and process a query. I'm assuming that the data would be in a standard stats package and hence processing would be easy.

Personally I would be surprised if they hadn't already written a query to get the 6MWT data for those they judge as recovered. Not to would show a remarkable lack of curiosity for a researcher.
 

biophile

Places I'd rather be.
Messages
8,977
Re the second request for 6MWD data. This should be rather interesting if ever released. I would be just as interested in seeing the mean(SD) and median(IQR) scores on physical function SF-36 for the "recovered" participants. I bet those would be real eye-openers when compared to population data from healthy people of working age.

As an aside, compare the attitude of PACE, to the attitude of O'Dowd et al (2006). In their extensive 140 page report on a single RCT of group CBT for CFS which would take ages to read properly:

"We have presented all of our data, with explanations of our reasoning, so that the work is as transparent as possible."

O'Dowd H, Gladwell P, Rogers CA, Hollinghurst S, Gregory A. Cognitive behavioural therapy in chronic fatigue syndrome: a randomised controlled trial of an outpatient group programme. Health Technol Assess. 2006 Oct;10(37):iii-iv, ix-x, 1-121. PMID: 17014748. http://www.journalslibrary.nihr.ac.uk/hta/volume-10/issue-37
 

Tom Kindlon

Senior Member
Messages
1,734
Re the second request for 6MWD data. This should be rather interesting if ever released. I would be just as interested in seeing the mean(SD) and median(IQR) scores on physical function SF-36 for the "recovered" participants. I bet those would be real eye-openers when compared to population data from healthy people of working age.
Yes, would be a good one for somebody to make.
 

Tom Kindlon

Senior Member
Messages
1,734
Re the second request for 6MWD data. This should be rather interesting if ever released. I would be just as interested in seeing the mean(SD) and median(IQR) scores on physical function SF-36 for the "recovered" participants. I bet those would be real eye-openers when compared to population data from healthy people of working age.
Perhaps somebody should now request median and interquartile ranges for the 6 minute walking distance data for the recovered so that it can be dealt with at the same time?
 

Bob

Senior Member
Messages
16,455
Location
England (south coast)
Instructions to public bodies re cost of fulfilling FOI requests, from the ICO website:

http://www.ico.org.uk/for_organisat...hen-can-we-refuse-a-request-for-information-2

You are only required to estimate whether the limit would be exceeded. You do not have to do the work covered by the estimate before deciding to refuse the request. However, the estimate must be reasonable and must follow the rules in the Freedom of Information (Appropriate Limit and Fees) Regulations 2004.
When estimating the cost of compliance, you can only take into account the cost of the following activities:
  • determining whether you hold the information;
  • finding the requested information, or records containing the information;
  • retrieving the information or records; and
  • extracting the requested information from records.
The biggest cost is likely to be staff time. You should rate staff time at £25 per person per hour, regardless of who does the work, including external contractors. This means a limit of 18 or 24 staff hours, depending on whether the £450 or £600 limit applies to your public authority.
You cannot take into account the time you are likely to need to decide whether exemptions apply, to redact (edit out) exempt information, or to carry out the public interest test.
However, if the cost and resources required to review and remove any exempt information are likely to be so great as to place the organisation under a grossly obsessive burden then you may be able to consider the request under Section 14 instead. (vexatious requests).
 

Anna Wood

wood/sheridan
Messages
487
Hello Everyone

I don't normally have the energy to come on here very often, but will try to keep this up to date. Queen Mary Univiersity, unsurprisingly declined by FOI request - https://www.whatdotheyknow.com/request/pace_trial_recovery_rates_and_po
with the standard 'we don't hold the data' response.
I've submitted an internal review.
Interesting Bob, to see the details about time to calculate FOI requests - this one really should be only minutes work!

I still find it hard to believe that the walking results were so poor - most studies show PWME are no more deconditioned than sedentary people, so why were they unable to walk further on a one off test? I'm sure I could, although I would feel dreadful and not be able to repeat the next day.

The other issue is the test used 10m instead of the standard 30m for this types of test. Again , I really can't understand why they did this. But they will try to use it as an argument not to give out the data - as it won't be valid. But even supposing that the extra turns (and I estimate there were 25 extra turns), took an very generous extra 1 second, the GET 6mwt would only increase by about 50m - from 379, to 429m. That is still a lot below average.
 

Tom Kindlon

Senior Member
Messages
1,734
The other issue is the test used 10m instead of the standard 30m for this types of test. Again , I really can't understand why they did this. But they will try to use it as an argument not to give out the data - as it won't be valid. But even supposing that the extra turns (and I estimate there were 25 extra turns), took an very generous extra 1 second, the GET 6mwt would only increase by about 50m - from 379, to 429m. That is still a lot below average.
People should be free to decide for themselves what is or isn't valid in terms of interpreting the results of the 6 minute walking test distance once the data is released. I think there is some information out there on the effect of the length of the track or, even if there isn't now, there might be in future (but the data on the recovered individuals could be lost for ever if it's not released now). So I don't think it's a good reason not to release the data.

Related to this: if a pharmaceutical company was hyping claims that their drug could bring about recovery, a claim that the data on recovery might be misinterpreted would likely be seen for what it is: a way to continue to hype the effectiveness.

People misinterpret data all the time - one could make that a reason for not releasing most data. Also, the same data can often been seen in different ways.

ETA: Here's an abstract on how getting access to data from research tends to be lost with time:

http://www.peerreviewcongress.org/2013/Plenary-Session-Abstracts-9-9.pdf

How Does the Availability of Research Data Change With Time Since Publication?

Timothy H. Vines,1,2 Arianne Y. K. Albert,3 Rose L. Andrew,1 Florence Débarre,1 Dan G. Bock,1 Michelle T. Franklin,1,4 Kimberly J. Gilbert,1 Jean-Sébastien Moore,1,5 Sébastien Renaut,1 Diana J. Rennison1

Objective

To quantify how fast the availability of research data decreases with time since publication and to identify the main causes.

Design

As part of a parallel study on how the reproducibility of data sets changes through time, we identified 516 papers that conducted a discriminant function analysis (DFA) on morphological data from plants, animals, or other organisms.

These papers were published in the odd years between 2011 and 1991.

We obtained e-mail addresses for the first, last, and corresponding authors of the papers and by searching online.

We then requested the morphological data used in the DFA by e-mail. For papers where the data were not available, we also asked the authors to give a reason, such as “the data are stored on inaccessible hardware” or “the data are currently in use.”

Results

We received 101 data sets, and another 20 were reported extant but could not be shared.

We found that 37% of the data from papers published in 2011 still existed, but this fell to 18% for 2001 and 7% for 1991 (Figure 5).

The odds of receiving the data decreased by about 7% per year.

The proportion of papers with no functioning e-mails fell from 12 of 80 papers (15%) in 2011 to 10 of 26 (38%) in 1991.

Furthermore, for papers where we heard about the status of the data, the proportion of authors reporting it lost or on inaccessible hardware rose gradually from 0 of 30 in 2011 to 2 of 9 (22%) in 1997, and then increased to 7 of 8 in 1993 (87%) and 4 of 6 (66%) in 1991.

Other variables like the proportion of nonresponders or the proportion of datasets that could be shared showed no relationship with time.

Conclusions

Researchers should be able to obtain published data from the authors long after the study is complete, but we found that almost all research data was lost 10 to 15 years after publication.

The main causes of data loss appeared to be a lack of working e-mails for the authors and the data being stored on outdated hardware.

1Biodiversity Department, University of British Columbia, Vancouver, BC, Canada, vines@zoology.ubc.ca; 2Molecular Ecology Editorial Office, Vancouver, BC, Canada; 3Women’s Health Research Institute, Vancouver, BC, Canada; 4Department of Biological Sciences, Simon Fraser University, Burnaby, BC, Canada; 5Department of Biology, Université Laval, Laval, QC, Canada

Conflict of Interest Disclosures None reported.

Reproducible
 

Anna Wood

wood/sheridan
Messages
487
Interesting - I can see why this happens. Most research is done by PhD students and early career researchers who move often (I have worked for 4 institutions over an 11 year period). But also how data is stored changes - in theory I've got my PhD data, but it is on a zip disk, and I've not idea if I could access a zip drive to read it!

Totally agree with your previous point about the worry about what might happen to the data not being a valid reason not to give it. Still think they will try using it as an excuse!
 
Messages
13,774
Thanks for trying to get the release of this data. It's really important to let patients be as informed as possible about the treatments options available to them, and how they might affect them. Hopefully QMUL will see sense after a bit more prodding, and stop trying to keep patients in the dark.
 

MeSci

ME/CFS since 1995; activity level 6?
Messages
8,231
Location
Cornwall, UK
Instructions to public bodies re cost of fulfilling FOI requests, from the ICO website:


However, if the cost and resources required to review and remove any exempt information are likely to be so great as to place the organisation under a grossly obsessive burden then you may be able to consider the request under Section 14 instead.

(My bolding)

Do you think they mean excessive?!