Measures of outcome for trials and other studies

[OverTheHills]

Excuse me for being very thick here but can I just clarify by talking about my own case. For me beta- blockers and naltrexone are good symptomatic treatments, each controls one of my ME symptoms (POTS and sleep respectively) quite well.

So using this composite measure in a trial of naltrexone as an overall ME treatment I would (1) rate my overall improvement as say 10% because it acheives good control of one of my top five symptoms (2) my TTT shows no change (3) an my actometer data shows I'm a little bit more active because I am less groggy in the mornings but still have to pace carefully to avoid PEM

SO Naltrexone comes out as an ineffective treatment for my ME although it is good for a single symptom control. Which would be a true reflection in my case.

But if you tested beta blockers as an ME treatment I would (1) rate my improvement as 20-50% because controlling OI allows me to be much more active, up to my PEM limit (2) my TTT would be normalish (3) my actometer would confirm (1)

So beta blockers are a moderately effective treatment for my ME which is also true. So for the subgroup that has OI this test would be good.

Hypothetically a really good treatment (like rituximab) affects a number of my symptoms, including PEM,OI, sleep which would lead to a higher overall improvement rating still supported by 2 TTT & 3 actometer

Is that what we are saying? Sorry again to be so thick

 

[alex3619]

I think that like Fluge and Mella who use flow-mediated dilation to test large blood vessel endothelial functioning in a substudy is important. They will also test microvascular endothelial function using skin laser-doppler measurements.

I was first subjected to tests of this nature about 1993. The research was shut down due to lack of funding and official opposition. You see, "there is no treatment for CFS, so please stop."

Metabolic rate is one probable measure. Its measurable. Unlike a 2 day CPET it does not have to be during exercise.

Actimeters are another. Yet in this illness that would be a long term monitoring situation. Spot measures are unreliable. I am also not convinced they are useful for severe patients.

For research only, and this should be restricted to milder patients, the 2 day CPET is a good choice. Not every patient will be happy with this test though because of the risks. In that respect it is similar to spinal fluid studies ... there is risk there too.

We currently lack a good range of reliable outcome measures. This is just one area we need more study in. One huge issue here is that we don't know causation, so don't know what to track. So every outcome measure is indirect.

What I think is very important is to get away from subjective measures as anything other than a secondary measure looking into quality of life. Quality of life is likely to be highly subjective anyway, there is no way to completely avoid that.

In general I think some things are easier to look at than others. Does the patient walk more? Do they work or study more? Can they engage in more activities to maintain themselves? (This includes household chores.) Is their metabolic rate raised? Are they fitter? Do they still have post exertional relapse, and how severe? Most of these can be answered.

Yet I think, ultimately, this is all a research question. At some point these measures need to be validated. Yet if we do not have effective treatments, how do you do this?

 

[Hip]

In terms of Objective measures:
1) Actigraphy - Has the advantage of measuring activity over an extended period of time, which means it will show whether patients actually have improved energy and patterns of activity or are simply displacing energy from their activities of daily living in order to complete a single test

Very interesting, I had not heard of actigraphy before. I notice there are a few smart phone apps that function as actigraphers.

I wonder how accurate the results are though. In my case, when I am feeling more energetic and sharper mentally, I will often sit motionless behind the computer for many hours, intently researching something or other, because I like to use my good energies for something productive. But when my mind feels vaguer, more tired and less focused, often I may go for a walk.

So in my case, actigraphy would not necessarily be a good measure of my mental energy and cognitive function.

 

[Hip]

Regarding this issue of measuring a change in symptoms, rather than measuring an symptoms in an absolute sense, I will just copy a post I made in another thread:

In my own personal experience of my ME/CFS, I find it quite hard to provide an accurate subjective grading of my own cognitive dysfunction and overall ME/CFS symptoms, in any absolute sense.

I am always able to say whether I am better or worse than yesterday (which is a relative measure of cognitive dysfunction). But on any given day, I could not easily place myself on an absolute scale (from 1 to 10 say) of how I was feeling.

This is party because my long term memory is appalling, so I very quickly forget how I was feeling last week and last month, which makes it hard to gauge in absolute terms how I am feeling on any particular day. But I can remember how I was feeling yesterday, and so it is easy for me to say whether I am feeling better or worse than yesterday.

I see this as a bit of a problem, because it makes it hard for me (and no doubt other ME/CFS patients) to judge whether a given medication regimen is having beneficial effects or not.

However, what I tend to do is use natural objective measures of my health level. For example, by noting the number of bedbound days I have per week, this gives me a natural objective gauge of my health level. Or I might note how many times I tend to leave the house per week (the healthier I am, the more I tend to go out). Or I might note just how much complex reading or writing I was doing in a given week (the less brain fog I have, the more I am able to tackle complex subjects).

I seem to be able to remember these more objective details. If these more objective measures of my heath level indicate an improvement, then I feel more confident that my current regimen is doing me good.

In summary: because of the brain fog and memory problems inherent inME/CFS, patients' own subjective, introspective judgment of their current health and symptom level may be unreliable. But asking patients to gauge their current health using natural objective measures in their daily life will probably be more dependable.

The only difficulty with this is that these natural objective measures will differ from one patient to the next, as the measures will depend on their lifestyle.

 

[beaker]

Quick note on neuro cog tests : Was in study where many tests were done in this area. But if you don't do the tests in the same order every time you can't compare. rough example: If math is first and reading comprehension and memory follow the first time. And the next time reading is first then memory then math. Math will be worse b/c energy now last when before it was first. I pointed this out to neuro psych who was doing this and they changed the study to be this way. Also to consider time of day. Some function at different levels based on time of day. Make sure same tests ( no matter which ones ) are done same time of day.

PS VO2/CPET done in this study too. If can't stand they would hook person up ( in chair ) anyways to get a reading of gas exchange and to record they can do nothing. At least something to compare and then sicker patients can be included.

Tilt table can make people very sick too. Sicker patients can't stand on the machine to get an accurate reading.
Dr. Streeten who published on OT, said "people don't tilt" and preferred measuring in office "poor man's tilt"

 

[WillowJ]

Thank you for the interesting and useful topic.

Should people try to measure things like fatigue quantitatively or is this a bogus idea?

One of my university teachers used to say that people think they are objective because they are assigned a number, but all grades are subjective, because it relates to things like the difficulty of the questions, how well they are understood, and how well they compare to what was taught in class.

Similarly, I think questionnaires and scales for pain, fatigue, other symptoms, wellness, and so on are subjective in that they change from person to person, questionnaire to questionnaire, and from day to day based on how the question is worded, the answerer's mood, their previous experience (I doubt everyone's 5 for pain or fatigue means the same as anyone else's) and all sorts of things.

That doesn't make it bogus. It does mean one should use caution in interpreting such scales, especially when the changes are small or don't correlate to changes in other areas.

Which is why I really like what you said:

The second thing is that although a grade is calculated using a series of point scores or lab measurements these are not added up.

The reason for these composite scores is not really to measure more things or even to measure at finer grain with a wider range of numbers. It is to try to overcome the unreliability of a single measure as a guide to what a reasonable person would think is a modest, moderate or major improvement.

That makes subjective outcome measures very suspect, but it can be argued that more objective surrogate measures are not measuring what is really important.

Treat the patient, not just the lab result? I think your instinct to combine them is right.

I would agree with other people and say that it would be useful to measure and ask about what a patient can actually do (and for this disease, be clear that we are asking can do and keep doing, without suffering adverse effects) instead of or along with asking about symptoms.

Neuropsychiatric testing would be a good objective measure, but as I understand it one has to know what to look for in order to design the test to find it. Slowed processing, difficulty with multi-tasking, and increased difficulty with more duration and intensity of concentration (and with upright posture, for some) would be good places to start.

Rheumatology has a much better foundation in its history than the field of CFS, and it would make more sense to adapt a rheumatology scale than to keep using some of the ones that have been being used in CFS studies (particularly the ones that have floor/ceiling effects and such problems).

Hopefully some tests will show up in a short time that would be useful as a measure of clinical status. Leptin sounds promising for that. Dr. Newton's muscle metabolism studies also. There are probably more possibilities but I've reached the end of my concentration.

 

[Hip]

In terms of objectively measuring symptom levels on an absolute scale, certain classic symptoms of ME/CFS may be amenable to this.

For example, in this thread: How to best gauge levels of brain fog & fatigue, it was suggested that the Stroop test might be used to measure the brain fog level of ME/CFS patients. A free online version of the Stroop test is found here. This is a quick and simple test to do, so might well be useful in measuring brain fog / cognitive dysfunction.

Sound sensitivity symptoms (hyperacusis) might also be measured objectively and on an absolute scale. I notice that when my sound sensitivity is high, certain sounds from the street outside will be very disturbing to my mind, and will really grate on my consciousness. For example, for me personally, any car outside with a thumping bass from a sound system is quite disturbing. I also find the white noise-type sound from a vacuum cleaner quite disturbing.

But on days when my sound sensitivity is low, the same sounds will cease to disturb me. The difference between a good and bad days is startling. And this seems quite objective: when a sound disturbs me, this is very clearcut; and when the same sound ceases to disturb me, this is also very clearcut.

Some ME/CFS patients can have very, very bad sound sensitivity. In one case that I read, even just the sound of a pencil writing on paper was excruciatingly painful and disturbing.

Incidentally, the sound sensitivity common in many ME/CFS patients (and in common in some psychiatric conditions like autism) may be due to sensory gating deficits — ie, a problem with the mental "firewall" that prevents extraneous noises from entering consciousness. The idea is that when the mental firewall is weak or dysfunctional, extraneous noises are not filtered out, and so bombard consciousness. @Marco wrote an excellent article on sensory gating as possibly playing major role in ME/CFS.

 

[oceiv]

I second @Hip's suggestion above regarding rating symptoms compared to the day before. This seems to me a better method to see if the patients' symptoms are trending upwards, downwards or are staying the same. I never know what to answer when asked "what is your pain level?" It can't be a 10, because I had much worse pain at the start of my illness. But it's always above a 5, because this is far from normal and very disabling. Measuring against the previous day would solve this problem.

In addition, since each patient has differing symptoms, a good start would be asking what the patient's most disabling or most bothersome symptoms are and tracking these symptoms' trend over a specified time. One patient may start out with exhaustion, PEM, muscle weakness and sleep problems. Another patient may start out with pain, PEM, neuroimmune symptoms and immune problems.

 

[Marco]

One of the things about the ACR grading is that because you need several measures to meet a threshold (I think it is five) it is pretty demanding. As a result even 50% of patients getting ACR20 is considered not bad (remember that ACR20 means anything up to ACR49, just not quite ACR50). (Note that there are no mean changes here - the result is how many patients reached the target grade. That avoids the false impression that everyone gets a bit of benefit.)

I very much like this idea of individuals having to meet some threshold(s) rather than reporting group means. I suspect you would have a better chance of identifying sub-groups not just in terms of treatment response but also potentially at baseline.

 

[zzz]

The 2-day CPET is clearly the gold standard here - or maybe I should call it the plutonium standard. Like gold, plutonium is extremely valuable, but it's also extraordinarily toxic. That seems to describe the 2-day CPET.

We know that our bodies are already different from healthy peoples' in many different ways on "normal" days. But on Day 2 of the CPET, before the actual test, the abnormalities are even greater, since exertion breaks our bodies so horribly with that second test.

In other words, there's something right on the verge of breaking right before that second test. If we could find it, we could save people from the extremely toxic effects of that second test. This would then give us a test that could be used without danger to patients, as the second exercise part would not be necessary.

Dr. Goldstein found such an abnormality in the brains of CFS patients in the early 1990s. Using SPECT scans, he found that people with CFS had regional hypoperfusion, something that is well known. After exercise, they showed global hypoperfusion, which is not surprising. But this global hypoperfusion persisted a day or two after the exercise - a result that happened only in CFS patients. This appeared to be the sign that their bodies were ready to break if they were stressed any more.

It seems to me that this would be well worth investigating. If a simple brain scan could substitute for the second day of exercise, as apparently it can, then such a test could become a reliable and safe way of diagnosing ME/CFS.

 

[Marco]

Incidentally, the sound sensitivity common in many ME/CFS patients (and in common in some psychiatric conditions like autism) may be due to sensory gating deficits — ie, a problem with the mental "firewall" that prevents extraneous noises from entering consciousness. The idea is that when the mental firewall is weak or dysfunctional, extraneous noises are not filtered out, and so bombard consciousness. @Marco wrote an excellent article on sensory gating as possibly playing major role in ME/CFS.

Thanks @Hip

Sensory gating deficits playing any role, even as an epiphenomenon in ME/CFs is speculative as it stands but I still feel that ERP measures have been underused in ME/CFS and could provide a 'purer' way to measure cognitive dysfunction especially at the pre-conscious level. Whether or not it would be practical as part of a composite measure is another issue.

 

[Jonathan Edwards]

How do you factor the normal relapsing/remitting aspect of our illness and it's impact when measuring any study outcomes of improvement? I would think time would make this part of our illness less of an influence when attempting to get a true measure of improvement. Are there any stastical methods that also do this?

Not sure if that is clear as I'm just kind of brainstorming.

This is quite an interesting thread. Thanks for starting it.

Barb

I may have answered my own question here. Time and replication would cancel this effect as you would get regression towards the mean?

I think fluctuation is a real problem. I guess that taking a total actometer reading over a month or something would help. But ups and downs contribute a statistical noise to assessment of a lot of conditions (MS is probably the key example) and the simplest answer is that you make your sample size big enough for the noise to even out.

 

[Jonathan Edwards]

I think a fair amount of the dilemma centers around perception and politics, as opposed to what is a "true" or accurate measurement of recovery.
Outcomes for studies on Alzheimers for example would be totally based on subjective measures. The problem is that ME/CFS is held to a higher standard based on scepticism about the actual disease and muddiness (some manufactured) around definitions and diagnosis.
So I think a salient question is what outcome measure is the most bullet-proof to criticism?

Ask the biggest ME sceptic you can find what measure they would accept as a positive result in a drug trial?

I am actually thinking it is the other way - that trials in ME, especially those of CBT, do not come up to the basic standards we expect in other diseases. A lower standard has been considered OK.

I think assessment of Alzheimer's is mostly objective because being able to add thirteen and twenty one or remember your address is not something likely to be biased by the way you feel.

 

[Sasha]

For research only, and this should be restricted to milder patients, the 2 day CPET is a good choice. Not every patient will be happy with this test though because of the risks.

I think that any test that might harm patients should not be part of a standard battery of tests in any clinical trial (whether an ME trial or otherwise). In what other disease would this be acceptable? It's one thing to risk harm with a therapeutic intervention (because you're weighing it against benefits) but a risk as part of an outcome measure has no such counterbalance.

As well as the ethical issue, it would introduce bias: patients would self-exclude. Many people want the severely affected to be included in trials wherever possible - but I think they'd self-select out of trials with CPET testing, just as those of us too ill to attend hospital were automatically selected out of PACE.

I accept that the two-day CPET has its place in research specifically on that test but as part of a standard battery, I don't think it's wise or right to include it.

 

[Jonathan Edwards]

@Jonathan Edwards could you provide links to ACR grading system please?

i am not entirely sure if enough studies have been done on the day to day variation of the TTT testing in autonomic dysfunction patient, and in that regards would that mean that patients taking beta blockers for POTS would be excluded of the study?

Sorry I missed that first query. This is a concise site. There have been changes over time and I am not sure I think the criteria are ideal, but this gives an idea of the structure.


http://www.phusewiki.org/wiki/index.php?title=ACR_Response_Criteria

 

[Jonathan Edwards]

@Jonathan Edwards,

I'd like to post a question about Rituximab for you to answer (if you would like to) on a relevant thread. Can you direct me to one?

Thank you.

There is a subsection on the forums page on rituximab issues I think - under treatments. The threads used most recently relate to the Haukeland studies but general discussion of rituximab would work there I think.

 

[A.B.]

Very interesting, I had not heard of actigraphy before. I notice there are a few smart phone apps that function as actigraphers.

I wonder how accurate the results are though. In my case, when I am feeling more energetic and sharper mentally, I will often sit motionless behind the computer for many hours, intently researching something or other, because I like to use my good energies for something productive. But when my mind feels vaguer, more tired and less focused, often I may go for a walk.

So in my case, actigraphy would not necessarily be a good measure of my mental energy and cognitive function.

How big is the improvement you're talking about? It's pretty modest, am I right? I'm still convinced that any substantial improvement will lead to patients doing more.

 

[Jonathan Edwards]

@Jonathan Edwards, is one of the benefits of using a ACR type grade of improvement system is that it can more accurately measure a patient group with widely varying symptoms like in ME/CFS? for example, some patients experience a lot of pain and others experience none. Some patients have orthostatic intolerance while others don't have any, etc.

What were the issues in RA that led to the development of this system?

I would stay far away from the "fatigue" word and measure Functionality/Ability similar to Dr. David Bells "ME/CFS Ability Scale".

The ACR system was not particularly developed to handle variability in presentation. That is more a feature of lupus. There are scoring systems for lupus like BILAG that cover lots of different aspects. Again, there is a threshold component to scoring, rather than adding, although I am not so sure that I find the BILAG structure convincing. At least it demonstrates that you can construct systems for diseases with wide variability.

I know nothing of the history of the origins of the ACR criteria for RA. I have a suspicion they were thought up at a time when common sense was valued more than the recent obsession with measuring everything as if it were a linear quantity. It is now in competition with the EULAR disease activity index (DAS28) which I personally think has a lot of bogus features.

 

[Sasha]

We know that our bodies are already different from healthy peoples' in many different ways on "normal" days. But on Day 2 of the CPET, before the actual test, the abnormalities are even greater, since exertion breaks our bodies so horribly with that second test.

In other words, there's something right on the verge of breaking right before that second test. If we could find it, we could save people from the extremely toxic effects of that second test.

But can we be confident that even the first day of testing is harmless?

 

[alex3619]

From my perspective, most arguments against the 2 day CPET also apply to spinal taps. They are also dangerous. Does that mean we should not have spinal fluid studies? No process is perfect.

We really really need more work on outcome markers.