Julie Rehmeyer's 'Through the Shadowlands'
Writer Never Give Up talks about Julie Rehmeyer's new book "Through the Shadowlands: A Science Writer's Odyssey into an Illness Science Doesn't Understand" and shares an interview with Julie ...
Discuss the article on the Forums.

Documents relating to a complaint about a Lancet editorial in 2011 on the PACE Trial & CFS

Discussion in 'General ME/CFS News' started by Dolphin, Sep 20, 2017.

  1. Dolphin

    Dolphin Senior Member

    Messages:
    10,672
    Likes:
    28,181
    I have been passed these documents, and been told it's fine to pass them on to others too or make them public.
     

    Attached Files:

  2. Dolphin

    Dolphin Senior Member

    Messages:
    10,672
    Likes:
    28,181
    This is rubbish. The SF-36 physical function data did not relate to healthy people.

    And indeed the PACE Trial authors were forced to admit it related to the full population, not just those of working age
    http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60651-X/fulltext

    The fatigue data didn't also relates just to people who were healthy either:
     
    Last edited: Sep 20, 2017
  3. Dolphin

    Dolphin Senior Member

    Messages:
    10,672
    Likes:
    28,181
  4. Dolphin

    Dolphin Senior Member

    Messages:
    10,672
    Likes:
    28,181
    I put 2 Astrid James of Lancet to PCC through OCR:

     
    Kati, Invisible Woman, BurnA and 4 others like this.
  5. snowathlete

    snowathlete

    Messages:
    3,312
    Likes:
    14,610
    UK
    Glad someone has been looking into this.

    Wow, that's really bad, to provide answers that are so obviously not true to the PCC. It's one of the most blatantly untrue things I've seen in relation to this. Problem is, with questions asked behind the scenes like this, there is no one to correct it.
     
  6. Esther12

    Esther12 Senior Member

    Messages:
    8,449
    Likes:
    28,522
    I left this page open, but then got distracted by SMILE.

    This does look closer to dishonesty than mere incompetence.

    It's amazing to me how badly the Lancet has behaved with this.... and been able to get away with it for so long!
     
    Last edited: Sep 30, 2017
  7. Snow Leopard

    Snow Leopard Hibernating

    Messages:
    4,613
    Likes:
    12,437
    South Australia
    These people need to learn about histograms and look at the SF-36 data. They wouldn't be making such incorrect claims if they understood statistics.
     
    Sean likes this.
  8. Barry53

    Barry53 Senior Member

    Messages:
    2,277
    Likes:
    13,704
    UK
    A little coy about what they said in the letter it would seem.
     
    Invisible Woman likes this.
  9. Barry53

    Barry53 Senior Member

    Messages:
    2,277
    Likes:
    13,704
    UK
    Deeply suspect of course.Validity and reliability are heavily dependant on the context in which the outcomes measures are applied; a decision had to be made of whether they were valid/reliable in the context of PACE. That very decision making process is itself, highly subjective unless very carefully managed and controlled. Given the trial was unavoidably unblinded, and the trial run by people totally wedded to the outcomes they aimed (and supposedly did) "prove", I think their trial execution was itself heavily loaded with considerable subjectivity from the start, let alone their subjective outcome measures.

    Did the peer reviewing really not question the issue that if a trial is fully unblinded (as some unavoidably are), then allowing it to be run by scientists with a 100% lock-in to the outcomes they desperately aspire to prove, is just so dangerous and unethical.
     
    Last edited: Sep 30, 2017
    Valentijn, Jan, HowToEscape? and 2 others like this.
  10. HowToEscape?

    HowToEscape? Senior Member

    Messages:
    465
    Likes:
    607
    What exactly is on that SF 36 list? Does it test for PEM? If the patients do not have significantly reduced or eliminated next day PEM, then they have no meaningful objective improvement.
     
    MeSci and Wonko like this.
  11. Esther12

    Esther12 Senior Member

    Messages:
    8,449
    Likes:
    28,522
    This video, along with some explanation in the details, should be useful:

     
    Valentijn, Kati, Orla and 4 others like this.
  12. user9876

    user9876 Senior Member

    Messages:
    2,583
    Likes:
    18,182
    They should be concerned about it. They say how recovery is defined in the protocol and then say that has been published in the PACE recovery paper which is was not. They then claim that this shows that people can recover hence the editorial was correct.

    This is of course misleading because the recovery paper doesn't cover the protocol and data was available for recovery to be calculated as defined in the protocol then there was no significant recovery.
     
  13. anciendaze

    anciendaze Senior Member

    Messages:
    1,806
    Likes:
    4,655
    Still wondering how they got that mean - one standard deviation when the mode was 95 and there was a hard cut-off at 100. A casual glance at the histogram suggests there might be many scores above 100, if you were to allow them. You are using mean and variance of a one-sided distribution as if it were not merely two-sided and symmetrical, but actually normal (Gaussian).

    "If you call a tail a leg, how many legs does a dog have?" "Four. Calling a tail a leg doesn't make it one."

    Second, we know for a fact that many of the people in the reference population they were using had unambiguous illnesses, some as serious as COPD or heart failure. Was there any attempt to separate these from the healthy reference group?

    What they demonstrated was that if you ignore the difference between people who are sick and well, you can decide that nobody can be seriously ill without a note from their doctor. This quality of reasoning, which commonly takes place in pubs near closing time, does not require or deserve research funding.
     
  14. Esther12

    Esther12 Senior Member

    Messages:
    8,449
    Likes:
    28,522
    No, to get a mean-1 sd of 60 they included all sick and elderly members of the population.
     
    Snow Leopard, anni66 and Dolphin like this.
  15. anciendaze

    anciendaze Senior Member

    Messages:
    1,806
    Likes:
    4,655
    @Esther12
    Of course, it might be 60 or it might be 65, depending on what the authors needed at the moment. By changing this they were tacitly arguing that a change of 5 points was not significant. Strangely, this did not apply to changes which might indicate recovery. This is an internal inconsistency in the authors reasoning.

    I've made more complicated arguments about that distribution. If it is an example of Lévy distribution, a stable distribution generally other than normal, the theoretical distribution has no finite variance. This is not a good basis for any of the techniques used in analysis of variance. In practice this means values for variance and standard deviation are entirely determined by the bounds on the sampling. If this is true then manipulating bounds can give you whatever answer you want.

    Now, there might be a normal distribution of healthy people, although this group made no effort to find it or separate it from the distribution of sick people. They effectively precluded any such attempt by using a measure with a hard cut-off at 100. I suspect the scale was designed so that healthy people scored at least close to 100. Some people, being healthier than average, would be expected to score above the mean, but we know nothing about them.

    The problem comes in modeling that "fat tail" of sick people. I've tried several combinations of different distributions to model the complete composite distribution. One thing I learned from the exercise is that a wide range of standard deviations might be used in modeling that tail. This is pragmatic evidence that little weight should be given to statistical inference based on analysis of variance in this situation.

    Lévy distributions turn up in many cases where ordinary statistical measures are likely to underestimate the chance of catastrophic failures. Everyone should now be aware that banking, insurance and stock-trading on the assumption that negative excursions are as rare as predicted by normal distributions is dangerous. Similar mistakes happen when you talk about "hundred-year floods", as we have seen this year. Assuming people are still healthy when their performance has been reduced to a fraction of that of a healthy person is equally unrealistic.

    Every doctor regularly sees people affected by a concatenation of biological insults, not all of which can be identified, and some of them die. Death is the cut-off that prevents people from being infinitely ill, as a mathematical model might allow. Assuming multiple problems are independent is a gross distortion of the truth, yet that is what assuming a normal distribution implies.

    Doesn't anyone out there have a minimal basic understanding of the mathematics of sickness? It should be at least as sophisticated as the mathematics of reliability of complicated machines, and that predicts that catastrophic failures are much more likely than similarly naive analyses imply. People didn't believe me when I said this about the Space Shuttles, which are now museum exhibits. I was neither the first nor the last to say this, but honest realism conflicted with the need to lobby for funding.
     
    CFS_for_19_years and Esther12 like this.
  16. Mithriel

    Mithriel Senior Member

    Messages:
    673
    Likes:
    770
    Scotland
    I may be missing the point, if so, sorry. The SF36 is just a list of questions about how people feel and what they can do. It is a man made scale that was devised so that doctors and patients could have a definite number to see if a treatment was helping or if a disease was progressing in things like cancer or Parkinson's disease as it is very hard to remember or quantify how you feel now as opposed to how you felt 6 months ago.

    The questions were worked out so that a healthy person would score 95 - 100 (most people can go out when they want to for instance) and split into domains so that a doctor could see if a heart patient, say was doing physically better while becoming depressed. It was never a normal distribution like height, say, which has a peak and then outliers.

    In the PACE trial it was simply used wrongly and strangled to give them the result they wanted in a completely blatant way. Michael Sharpe said that if they hadn't changed the outcome most healthy people would have fell within it which was either a lie or ignorance.

    At the time, some of the paper, or other things they wrote said they used 2 standard deviations.
     
    Snow Leopard and Barry53 like this.

See more popular forum discussions.

Share This Page