1. Patients launch $1.27 million crowdfunding campaign for ME/CFS gut microbiome study.
    Check out the website, Facebook and Twitter. Join in donate and spread the word!
A disease with two faces? Re-naming ME/CFS
Persuasion Smith covers the bases on the misleading and disreputable name for our disease we've all been saddled with ...
Discuss the article on the Forums.

Quality and acceptability of patient-reported outcome measures used in CFS/ME

Discussion in 'Latest ME/CFS Research' started by Dolphin, May 28, 2011.

  1. Dolphin

    Dolphin Senior Member

    Messages:
    6,825
    Likes:
    5,941
    I don't promise to say interesting at all in this thread. Add that to the fact that the topic itself might only have a minority appeal and people might find other threads of more interest...

     
  2. Dolphin

    Dolphin Senior Member

    Messages:
    6,825
    Likes:
    5,941
    I was hoping for more specific criticisms of questionnaires. In some ways, there are criticisms in the tables - but they are generally about more "factual" aspects. They generally don't talk about how particular items [i.e. questions] on particular questionnaires may be problematic e.g. the effect of confounding with measures of mood i.e. some symptoms that might be part of CFS rather than evidence of depression, etc. (I think on this specific point they make a vague reference to two papers but don't spell out the possible problem).

    Anyway, it's quite a long paper so perhaps some people may find the bits I've underlined (and am now posting) of more interest than reading it all. Or they might not find what I post of interest as I'm tired now so am not promising to editorialise or put explanatory notes.
     
    Esther12 likes this.
  3. Dolphin

    Dolphin Senior Member

    Messages:
    6,825
    Likes:
    5,941
    From abstract (more information explaining this is given at the end)
    Underlined bits are bits I'm interested in:
    They didn't search the Journal of Chronic Fatigue Syndrome which I think is a major omission.

    Underlined bit is bit I'm interested in:

     
    biophile, Simon and Esther12 like this.
  4. Dolphin

    Dolphin Senior Member

    Messages:
    6,825
    Likes:
    5,941
    Last two paragraphs of discussion section

    These are the last two paragraphs of discussion section - I have a lot underlined here so thought I'd include them all:

     
  5. wdb

    wdb Admin

    Messages:
    861
    Likes:
    656
    London
    Very interesting paper, lots of stuff to think over. I'm so glad some researchers at least are taking this issue seriously. I can't see how we can make any real progress until we have the means to actually measure it reliably.

    I think Malcolm Hooper hit the hail on the head with this paragraph:

     
    alex3619 likes this.
  6. Enid

    Enid Senior Member

    Messages:
    3,309
    Likes:
    840
    UK
    He always gets things right - Prof Hooper - definately "nail on the head" for me too wdb. Must add that my own personal observations/descriptions to my own medics were met with disbelief so often anyway despite obvious difficulties. Even at that stage much prone to Prof Hooper's "influences" - deference gratitude for investigations etc.
     
  7. Simon

    Simon

    Messages:
    1,464
    Likes:
    4,666
    Monmouth, UK
    Thanks for posting this Dolphin. Thought I'd summarise a few key points, which repeats some of yours

    1. Central importance of properly meauring outcomes that are important to patients
    This isn't some wishy-washy point about consultation, it's about good science: if you are going to use self-reported outcomes as a measure then the patient view is everything. So if your fatigue scale, developed without checking with patients, fails to meaningfully and accurately describes fatigue then its a lousy measure from a scientific as well as a patient perspective.

    Many scales are 'validated' basically on their 'psychometric' properties, which usually means checking things like their being plausible statistical relationships between individual questions on the questionnaires. But if the questions themselves are wrong, or unbalanced, the questionnaire can still be a poor measure of fatigue (or whatever) -however impressive its statistical qualities.

    2. Failure to show outcome scales actually measure what they claim to measure
    Linked to the first point

    One way to show as questionnaire does what it claims is to compare how it's results with an external measure that should measure the same underlying thing. Comparing, say, questionnaire results of reported activity against an objective measure of activity such as actomters is one way of doing this. Comparing reported activity levels against employment rates or benefit rates of the same people would be another way of doing this. This sort of thing is also covered by various types of 'validity'. And it hasn't been done for CFS measures.

    Here's one example:
    In this case the fatigue measured by the FSS correlates better with other measures of CFS, CFS severity, and SF36 scores (both also questionnaires) than the Chalder Fatigue Scale. However, it is still comparing the results of one questionnaire with the results of another.


    Similar to above, and makes the point that asking patients what matters is important. A great example of this sort of problem is the Chalder Fatigue scale, which was largely invented by Simon Wessely
    Note he didn't ask patients what they thought was the best way to describe fatigue, he decided he knew himself - there is no sign he even consulted patients once he had 'invented' the scale. The scale was then psychometrically assessed and found to be 'acceptably valid', but this didn't involved looking for anything outside the original questions.

    As far as I know, this is how most 'patient-reported outcome measures' are invented, without any direct input from paitents.

    3. Failure to show "clinically significant changes" are actually significant to patients.
    Another key point this review highlights is that none of the PROMS reviewed include any evidence on interpretation of changes in scores. So if average Questionnaire scores improve by 3 points during a trial - how do we know that matters to the patients involved? Given it's such an important question, it's amazing that dozens of CFS trials, and no doubt thousands of trials in other areas are done without showing that any gains are meaningful.

    One way used to deterimine 'Clinically Significant Improvement' it to use statistics. Eg the PACE trial decided that a gain of 0.5 Standard Deviation (SD) was a 'clincally useful difference' - though they would have been better to call it a 'Statisically Significant Difference' since patients were not consulted on whether or not they thought this was a worthwhile gain. (For the Chalder scale, a gain of 2 points (=0.5 baseline SD) on a 0-33 scale was declared significant). Given the problems already covered that the Chalder scale might not even be a very good way of measuring CFS fatigue, the use of statistics to decide what is 'clinically useful' - instead of asking patients (which could be a way to check the statistically-dervied threshold) is worrying.

    Summary: if patient measures are unreliable you can't measure the benefit of any treatment properly
     
  8. Valentijn

    Valentijn Activity Level: 3

    Messages:
    6,512
    Likes:
    9,649
    Amersfoort, Netherlands
    Actually studies have used objective measurements, in addition to the usual questionnaires. And they've shown that reporting less disability on questionnaires isn't supported by increased activity levels. Does this mean that those questionnaires have been proven invalid in certain circumstances, such as after an extended course of brain-washing/CBT?
     
  9. peggy-sue

    peggy-sue

    Messages:
    2,494
    Likes:
    2,874
    Scotland
    Going back to Hooper's paragraph (which should read as follows - I've bolded the grammar correction)


    Instead of using actometers, the PIs relied largely on participants subjective responses to questionnaires, which are notoriously unreliable.
    Subjective data is just that. It lacks objectivity and is prone to influences such as participant deference, motivation, gratitude, placebo effect and interpretation. Subjective data is not evidence-based and should be considered unreliable in remitting/relapsing disorders such as ME/CFS. Furthermore, to rely on subjective data in a trial that intentionally set out to modify participants own subjective beliefs cannot be classed as a scientific study.


    It strikes me that subjective evidence has been USED as objective evidence in PACE.
    It is described as an evidence-based study, it touts itself as the best available evidence

    but it's all subjective in the first place - none of the skewing even matters - it's simply not evidence.
     
    Valentijn likes this.
  10. Simon

    Simon

    Messages:
    1,464
    Likes:
    4,666
    Monmouth, UK
    Thought it might be worth putting some numbers on this by comparing gains in fatigue, with gains overall (both self-rated) and with the 6MWT - all for PACE, of course. I've given data for CBT, and also for the control SMC group ( :) brain-washing free). All results at 52 weeks vs baseline

    % with fatigue improving by a 'clinically useful difference': CBT=79%; SMC=65%
    % with physical function (sf36) improving by a 'clinically useful difference': CBT=71%; SMC=58%
    % with positive 'Clinical Global Impression' score: CBT=41%; SMC=25%
    Mean gain in 6 min walking difference: CBT=+21m (6%); SMC= +22m (6%)

    Far more people show clinically significant improvements in fatigue or function than report an overall improvement in their health, which should in itself be cause for concern. If patients don't think they have improved overall but the researchers say they have made a 'clinically useful' gain, something is up. More than twice as may in the control group were classified as clinically improved in fatigue than said they were improved overall. Also, the two measures may not even correlate well, eg some of those saying they have improved overall may not have improved 'clinically' at all.

    And of course the average gain in distance walked was tiny (and not statistically significant) for both CBT and SMC despite large gains in self-rated physical function.

    peggy-sue - not sure I would wholly agree with that: there is no objective measure of fatigue and generally I think patient-reported outcomes are important. However, objective measures are important too, and as a matter of course authors should, in my opinion, check that objective measures back up the subjective measures. If they don't then the authors need to explain why not.

    Looking at the figures above, I think the authors should have at least attempted to explain the large discrepancy.
     
    biophile, Esther12, Dolphin and 2 others like this.
  11. peggy-sue

    peggy-sue

    Messages:
    2,494
    Likes:
    2,874
    Scotland
    I'm very bad in terms of brain function, I'm afraid. Simon.
    Can't hold many ideas in my head at one time, and can't follow details, I'm just grabbing and kind of clinging to little twigs here and there. :redface:

    I want to be simplistic and shout;

    "Why can't they jolly well measure lactic acid production in my muscles if they want physical proof (and an objective measure) of my so-called "fatigue".

    But I was thinking of the gaslighting really - regarding the use of subjective reports within the study, but then CALLING the results of the trial "evidence".
     
  12. Simon

    Simon

    Messages:
    1,464
    Likes:
    4,666
    Monmouth, UK
    Most studies on CFS patients don't find differences in lactic acid production, despite greater levels of fatigue - that may well not be the issue (and believe me, I've slogged through a lot of papers to find this out).

    But also if I was in a Trial and reported my fatigue levels, or overall levels of health I wouldn't want that dismissed as 'not evidence' by anyone. It's a subjective report and should be treated as such, but that doesn't make it unimportant. A lot of people arguing for meaningful patient self-reports are patients themselves. The problem with any subjective report, though, is that there may be self-report bias, especially in clinical trials. That doesn't mean, in my opinion, that patient reports pre se are meaningless.

    Actually, there is a whole debate about patient involvement in research, arguing about moving from studies about patients to studies with or by patients. Not all researchers are very keen on this as it involves a loss of power.
     
    peggy-sue and Valentijn like this.
  13. peggy-sue

    peggy-sue

    Messages:
    2,494
    Likes:
    2,874
    Scotland
    I do believe you have slogged... and slogged... and slogged. :thumbsup:
    But I do not get fatigue.
    I do get horrendous lactic acid pain, very rapidly.

    For example, in my legs if the pavement starts a small incline, or I manage to get half-way up the stairs.
    My problem is energy production.
    I cannot do anything that recruits aerobic metabolism, I am stuck in anaerobic mode. But if I stay there, I can achieve a few things.


    Lactic acid was found to be produced at 20 times the normal rate, in cultured muscle cells from sufferers, by Prof. Julia Newton, published earlier on this year.

    My suspicion about many publications which say they "found nothing" is that they did not look in the right places, or use the correct patient population, and they get published because the "powers that be" want that sort of "nothing to be found" stuff out there.

    Studies which did find things, earlier on, had a nasty habit of going awol, Elaine de Freitas, for example...

    Done a bit of googling on her for accurate info:-

    (copied and pasted the relevant bit from this link. Article by Margaret Williams)

    http://meagenda.wordpress.com/?s=Elaine de


    "
    In 1991, using polymerase chain reaction and in situ hybridisation, Dr Elaine De Freitas, a virologist at the Wistar Institute, Philadelphia (which is America’s oldest independent institution devoted to biological research) and Drs Daniel Peterson, Paul Cheney, David Bell et al found such an association (Retroviral sequences related to human T-lymphotropic virus type II in patients with chronic fatigue immune dysfunction syndrome. Proc Natl Acad Sci USA 1991:88:2922-2926). It is notable that co-author Hilary Koprowski is a distinguished virologist and Professor Laureate who was Director of the Wistar Institute from 1957-1991; he is a member of the US National Academy of Sciences and is Director of the Centre for Neurovirology at Thomas Jefferson University.
    Before publication, the findings were presented on 4th September 1990 by Elaine De Freitas at the 11th International Congress of Neuropathology in Kyoto, Japan.
    Ten days later, on 14th September 1990 Dr Peter White (as he then was) and other members of the Wessely School dismissed the findings: “in the vast majority of CFS cases there is a psychological component. About 75% of CFS sufferers are clinically depressed, according to Peter White, senior lecturer in the department of psychiatric medicine at St Bartholomew’s Hospital in London. White said he believes depression is often a cause, rather than a consequence, of CFS…Les Borysiewicz, a clinical virologist at Addenbrookes Hospital in Cambridge (now Chief Executive of the MRC, having succeeded Professor Colin Blakemore) (said) ‘Whatever causes CFS, it isn’t the virus itself’…Anthony Clare, psychiatrist and medical director of St Patrick’s Hospital in Dublin (now deceased), pointed out that…there have been many ‘fatigue’ diseases with shifting causes: ’Neurasthenia, food allergies, now viruses. Some people would always rather have a disease that might kill them than a syndrome they have to live with’ ” (Science 1990:249:4974:1240).
    In their PNAS article that was published in April 1991, De Freitas et al noted that chronic fatigue immune dysfunction syndrome (CFIDS) “may be related or identical to myalgic encephalomyelitis” and examined adult and paediatric CFIDS patients for evidence of human retroviruses (HTLV types I and II). As the CFIDS Chronicle article noted, the Wistar team looked at the peripheral blood DNA to see if they could find messenger RNA (mRNA) encoding for a viral segment of the HTLV-II virus.
    At that time, known human retroviruses were the human immunodeficiency viruses 1 and 2 (HIV-1 and HIV-2) which are known to cause AIDS, and human T-lymphotropic viruses HTLV-I which causes lymphoma and HTLV-II which causes leukaemia (Hunter-Hopkins ME-Letter, October 2009). The four segments of the HTLV-II virus are referred to as the env, gag, pol and tax.
    After a two year study, De Freitas et al provided evidence for HTLV-II-like infection of blood cells from CFIDS patients (and also to a lesser extent from people closely associated with them). This evidence was further substantiated by patient reactivity to proteins with the molecular weights reported for HTLV-I and HTLV-II antigens.
    In their article, De Freitas et al said: “The frequency of these antibodies in CFIDS patients compared with healthy non-contact controls suggests exposure / infection with an HTLV-like agent rare in healthy non-contact people”.
    Following the Wistar findings, researchers at the US Centres for Disease Control (CDC) allegedly attempted to replicate De Freitas’ work but failed to do so; this was suggested to be because certain scientists appeared eager to discount any possibility of a retroviral association with CFIDS. De Freitas defended her work and insisted that the CDC investigators had modified her assays, with the result that her work could not be replicated by the CDC.
    De Freitas was publicly discredited; her research funding was discontinued and her research abandoned; she was subjected to what appeared to be attempts to destroy her professional reputation. Commenting on the subsequent discovery of XMRV (see below), ME/CFS expert Dr Paul Cheney of The Cheney Clinic was unambiguous: “Her work was unfortunately assaulted by the CDC. Her proposal to fly to the CDC in Atlanta to physically run the assays side by side with the CDC scientists was dismissed by the CDC” (http://cheneyclinic.com/a-retrovirus-called-xmrv-is-linked-to-cfs/538 ).
    Read full article here:
    http://www.meactionuk.org.uk/The-role-of-viruses-in-ME.htm
    http://tinyurl.com/ykjveep
    http://www.meactionuk.org.uk/The-role-of-viruses-in-ME.pdf
    http://tinyurl.com/y8m8s8h
     
  14. Esther12

    Esther12 Senior Member

    Messages:
    5,296
    Likes:
    5,606
    Thanks D and Simon.

    re validity:

    It always pisses me off when they talk of fatigue questionnaires as being 'validated', without there being any good evidence that they are valid measures of what people (and particularly that patients being studied) mean by 'fatigue'. It's invalid 'validity'!
     
  15. Firestormm

    Firestormm Guest

    Messages:
    5,824
    Likes:
    5,976
    Cornwall England
    Simon all sounds like a fine article in the making to me mate :)
     
    Simon likes this.
  16. Snow Leopard

    Snow Leopard Senior Member

    Messages:
    2,373
    Likes:
    1,948
    Australia
    There is no simple measure for fatigue, unfortunately.

    Lots of things have been proposed, from specific cytokines, to a build up of free fatty acids, to oxidative stress, even elevated levels of serotonin. (which by the way, three different groups which investigated this did in fact find evidence of elevated serotonin activity)

    That's why I suggest using objective measures of activity and cognitive function at the same time as self-report questionnaires.
     
    Simon likes this.
  17. alex3619

    alex3619 Senior Member

    Messages:
    7,495
    Likes:
    11,981
    Logan, Queensland, Australia
    More to the point, claiming that subjective data is objective is misleading. Whether this is deliberate or just really bad science is a matter of debate.
     
    peggy-sue likes this.
  18. alex3619

    alex3619 Senior Member

    Messages:
    7,495
    Likes:
    11,981
    Logan, Queensland, Australia
    You can objectively measure activity. You can objectively measure oxygen use, and the AT threshold. You can objectively measure failure to recover from activity by repeat CPET. You can objectively measure cognitive function in various ways using extensive automated cognitive testing batteries. Somehow those who like subjective testing don't want to use any of these.

    In my view the 6 minute walking test is also invalid, based upon published physiology of ME. Only repeat exercise testing has validity now. The same criticism does not appear to apply to actometers. This measures sustained activity over a large period of time. What the science does suggest though is that actometers should be worn over extended periods of time, in part to compensate for the fluctuation in activity - perhaps over weeks.
     
    Valentijn, peggy-sue and biophile like this.
  19. Firestormm

    Firestormm Guest

    Messages:
    5,824
    Likes:
    5,976
    Cornwall England
    How to you measure someone who says they are 'fatigued' to the extent that it effects their life; but who is able to 'push through' if necessary?

    Isn't this something else that needs to be taken into account somewhere? I mean we do need to listen to patients - of course we do - and it is subjective: but measuring biologically cannot account for everything.

    There will ever be the 'human factor'. Be it someone who is healthy, someone with significant cancer, someone being treated for cancer, someone with acute ME or someone with chronic ME.

    We are all different - at least at some level.
     
  20. alex3619

    alex3619 Senior Member

    Messages:
    7,495
    Likes:
    11,981
    Logan, Queensland, Australia
    Firestormm, that is why I suggested long periods with actometers. People pushing through will find they spend time doing less. That should show, most of the time, in prolonged actometer data.

    Where its unclear is with misdiagnosed patients, or patients at the extreme mild end. These might not show up the same, but then we need to see that in the data in order to begin addressing the issue.
     
    Snowdrop and Valentijn like this.

See more popular forum discussions.

Share This Page