Measures of outcome for trials and other studies

[Bob]

I haven't read this thread is close detail, so I apologise if I'm repeating stuff or if I've missed important bits of discussion.

I agree with the general sentiment that ME symptoms are near-impossible to quantify.
The symptoms can fluctuate daily, and as soon as you participate in the slightest over-exertion, then all hell breaks loose, and so a fairly mild day can be followed by a drastically bad day. So, single-point measurements are poor indicators of health.
And each patient experiences a different of battery symptoms, or might label different symptoms as their most challenging or most disabling (e.g. exhaustion, pain, light-headedness, or cognitive dysfunction).

It's also the case that some of us can choose to do more activity at the cost of increased symptoms, or we can choose to do less activity with the benefit of controlling our symptoms. So a measure of either activity or symptoms may not be a helpful reflection of health status, as many of us spend our lives balancing the two (activity vs symptoms). However, I think that most of us probably increase our activity levels, at least to some degree, as symptoms permit, even if it's only having an extra shower, of cooking an extra meal, or shuffling around the home a little extra.

Subjective questionnaires struggle to measure subtle or unconscious changes in health, or gradual changes in health.

I consider the CPET test to be dangerous, for many patients, and I would never volunteer to participate in one. A one-day CEPT test would drastically harm me for the long-term.

A tilt table test would only be relevant for a subset of patients - not all ME/CFS/SEID patients have obvious or disabling OI problems. I've rarely experienced any OI issues myself, to my knowledge.

My preferred approach, to assess health of ME patients, would be for actigraphy, because I think this would be a fairly accurate indication of average fluctuations in health for most patients. We pace ourselves, so we control our activity, but I think we would all find it difficult not to do at least a bit extra activity if health permits. And with obvious improvements in health, we adjust our activity accordingly. Average activity is a far more accurate measure of our health than a single-point measure. Also, actigraphy would measure unconscious increases in activity, such as extra walking around the home. It's difficult to mentally record how much activity we are doing around the home. We only really notice large or obvious changes such as when we can cook an extra meal, or when we can make it to a local shop.

Also, as others have suggested, i think that regular cognitive tests would be a good indicator of cognitive function, which is a very challenging aspect of the illness for many. However, i think that the degree of cognitive impairment varies drastically between patients, and it may not be a major challenge for all patients. If using cognitive tests, only some types of tests demonstrate impairment in ME/CFS patients: For some tests we have near-normal results. We demonstrate impairment only with the speed of processing for certain complex tasks including multi-tasking. I can never remember the exact details, but they're easy to find if needed.

 

[Marco]

It may depend on the reliability you want. I read that apple were going to add a heart monitor to their smart watch but didn't in the end because it didn't meet medical standards. This made me wonder about the accuracy of small devices but that may not matter for the type of readings necessary for this type of monitoring. I have read about efforts to use accelerometers to detect heart beats and to be used in non-medical applications as well where I would have thought accuracy was necessary.

What may be interesting is to tie heart beat into other accelerometer data

Heart rate variability is a somewhat different thing from heart rate. It's a measure of responsiveness to demand. A regular heart rate whether low or high isn't a good thing.

 

[Sasha]

A tilt table test would only be relevant for a subset of patients - not all ME/CFS/SEID patients have obvious or disabling OI problems. I've rarely experienced any OI issues myself, to my knowledge.

Actually, I think it's not always clear from symptoms and that tilt-table testing for all could be a good idea (at least at baseline, to establish whether there's a problem).

I have delayed OI and didn't identify myself as having an OI problem until very recent years because I didn't recognise myself from other people's experiences.

 

[Sasha]

BTW, just wanted to say what an interesting and useful discussion this is proving to be.

 

[Jonathan Edwards]

Thanks for the comments, @Bob. I have been thinking along the lines of some sort of threshold based composite score and I still think a 3 component score makes sense. So taking your points into account I am leaning towards something like this at the moment.

1. Does the person actually think they are better, all things considered. In RA all we do here is ask for a global self-rating in relation to a mark on a 10cm line at the beginning. If you start at 2, are you now around 5.5? There are no issues about whether the emphasis is wrong for the individual - they emphasise as they like. Beyond that there are no questionnaires, maybe because they run into all the problems discussed.

2. Is there physiological evidence to support the person's claim that they are better? (To reduce the chance they are just saying they are better to please you, and other stuff.) Tilt table seems to have its pros and cons. Maybe thinking tests should go in here (I hate 'cognitive').

3. Is there a change in real life ongoing activities of daily living that supports the claim of being better? Actometry/ actigraphy seems to have a lot going for it here although it may need to be based on an iWatch7 app not yet invented to be sensitive and flexible enough.

If there is 50% improvement on all three then I think we begin to have an indication that a treatment can be seriously useful. And not all treatments need be judged in this way since things like analgesics do not aim to alter the overall process. But a 'real treatment for CFS' to quote a recent writer on the subject ought I think to hit the mark at least now and again.

 

[Bob]

@Jonathan Edwards, I like your latest thoughts, but I think perhaps they could be further refined...

Perhaps if you were to use both tilt table and cognitive tests, then this world serve quite a range of patients, but maybe it would be quite an intensive set of testing for the trial investigators to manage? Also, both of the tests relate to very specific aspects of the illness and either one may not be pertinent to a large proportion of patients? e.g. I struggle to see how tilt table tests on their own could serve a patient cohort which doesn't require OI issues? But perhaps I've missed some discussion about this on the thread. I hear what sasha says, and I know that there has been a lot of interest in tilt table tests in ME/CFS, but I wonder if tilt table tests would be a bit experimental, and irrelevant to a significant proportion of patients? But perhaps there is substantial research that I'm not aware of.

I really like the idea of your point no.1, above, but, I think there may be problems with that sort of question. I'll try to explain my thinking... If someone asks me how I'm doing on a good day I might say that I'm feeling wonderful - I'm so pleased that my symptoms aren't causing me intense or moderate distress, that I feel relatively wonderful despite not being able to leave my home. Ask me on the next day, after I stupidly decided to vacuum the front room, and the disappointment and frustration of my symptoms flaring up will cause me to tell you that I'm having a really horrible time, in general. It can be hard to answer questions about our health objectively, especially if we only have a moment to think about it. And should we answer such questions relatively and, if so, relative to what? Relative to a good day, a bad day, an exceptionally bad day, or relative to a healthy person? (BTW, we can't remember what it feels like to be healthy - Healthy is not within our reference range.) Also, our cognitive issues can mean that we can forget how we've been feeling generally. So, if asking this sort of question, I think it would be helpful to ask over a number of consecutive days, so that a momentary feeling of positivity or negativity doesn't cause us to answer over optimistically or over persimistically. If asking the question over a number of days, I think it would give people a chance to reflect more and to give a more objective answer.

I think that anything that can measure average function is useful. I remember answering questions about my function for my NHS assessment. On the day that I filled in the questionnaire I couldn't stand in the shower without pain and discomfort, but on the day of my assessment I could walk from a car to the assessment room without too much discomfort. (I'd been persistently resting in a pre-emptive effort, in order to make sure that I could manage the appointment.) The consultant amended my assessment forms, without consulting me, because he could see that I could walk a short distance without obvious discomfort on that particular day.

So, I think that anything that can discern average function would be helpful. Also, i think that anything that makes the measures less subjective and less relative would help.

Sorry not to have any simple answers.

 

[Sasha]

@charles shepherd has just posted dates of the next UK CMRC conference. I had just been thinking that this might be a useful conference topic, though I don't know whether that would be the right conference (especially given the dates).

In practice, how does one take this forwards, @Jonathan Edwards?

 

[Jonathan Edwards]

So, if asking this sort of question, I think it would be helpful to ask over a number of consecutive days, so that a momentary feeling of positivity or negativity doesn't cause us to answer over optimistically or over persimistically. If asking the question over a number of days, I think it would give people a chance to reflect more and to give a more objective answer.

I think that anything that can measure [U]average[/U] function is useful.

Sorry not to have any simple answers.

Yes, I think there is some consensus on that. We need assessments to reflect extended periods. I think Fluge and Mella use a sort of continuing diary with reports over a period of a week or so but I forget.

As someone has said before, these issues tend to be more of a problem if we are looking for small changes. If you no longer have any brain fog, which is the objective, then it may be easier to say 'gee I no longer have any brain fog and I am pretty clear about that'!

Nuances, nuances, but important nuances.

 

[Jonathan Edwards]

@charles shepherd has just posted dates of the next UK CMRC conference. I had just been thinking that this might be a useful conference topic, though I don't know whether that would be the right conference (especially given the dates).

In practice, how does one take this forwards, @Jonathan Edwards?

Just keep discussing - with everyone involved. Fluge and Mella will be in London in May, together with a lot of others. We can exchange ideas then. Whenever people have ideas for trials ideas can be exchanged. I am not a fan of setting anything in stone. Flexibility is essential. But I think there are ways to encourage people at least to consider wider options. Maybe the key thing is to move focus away from complex questionnaires.

 

[Sasha]

Just keep discussing - with everyone involved. Fluge and Mella will be in London in May, together with a lot of others. We can exchange ideas then. Whenever people have ideas for trials ideas can be exchanged. I am not a fan of setting anything in stone. Flexibility is essential. But I think there are ways to encourage people at least to consider wider options. Maybe the key thing is to move focus away from complex questionnaires.

I don't know - I'm a bit of a checklist fan. We have a (rather poor, apparently) research database of 6,000 papers. I wonder how much better they might all have been if they'd had a toolkit of standard measures, devised with the kind of input from patients that's going on here.

Think what the PACE trial could have been with such measures.

Think how much more rapidly all that CBT/GET bullshit would have been blown out of the water.

I'd like it to be that resarchers couldn't do a trial without being expected to use the standard kit (or the relevant elements of it) and would have to make a good case for not using it. At the moment, it's a free-for-all and it's not clear it's a very informed one.

But perhaps that's the fascist dictator in me talking.

 

[Jonathan Edwards]

I don't know - I'm a bit of a checklist fan. We have a (rather poor, apparently) research database of 6,000 papers. I wonder how much better they might all have been if they'd had a toolkit of standard measures, devised with the kind of input from patients that's going on here.

Think what the PACE trial could have been with such measures.

Think how much more rapidly all that CBT/GET bullshit would have been blown out of the water.

I'd like it to be that resarchers couldn't do a trial without being expected to use the standard kit (or the relevant elements of it) and would have to make a good case for not using it. At the moment, it's a free-for-all and it's not clear it's a very informed one.

But perhaps that's the fascist dictator in me talking.

I think toolkits for standard measures are much of the problem - SF36 screwdrivers, Minnesota mole wrenches etc. Every trial is different and all researchers must be free to use new methods for new problems - but they must also be prepared to defend them against reasonable criticism!

 

[user9876]

@Jonathan Edwards,

So, I think that anything that can discern average function would be helpful. Also, i think that anything that makes the measures less subjective and less relative would help.

We need to be careful over average function. If we have a number of readings then the median is good. My concern would be an outlier say due to an immediate but short-lived drug reaction good or bad could skew results,

 

[user9876]

I think toolkits for standard measures are much of the problem - SF36 screwdrivers, Minnesota mole wrenches etc. Every trial is different and all researchers must be free to use new methods for new problems - but they must also be prepared to defend them against reasonable criticism!

I think its important that patient experience is captured in any measures used whether standard of custom designed for a given trial. So perhaps there are some requirements or difficulties worth capturing from these types of discussions.

It seems to me that some of those designing questionnaires didn't really understand the experience of ME and how it could effect measurements.

 

[Sasha]

I think toolkits for standard measures are much of the problem - SF36 screwdrivers, Minnesota mole wrenches etc. Every trial is different and all researchers must be free to use new methods for new problems - but they must also be prepared to defend them against reasonable criticism!

But that's the issue - do we really have new problems in every trial? Especially if we're talking about measuring patients' disability in the kind of real-world terms that we've been discussing, for example?

I understand that the rtx trials might want to measure certain biological things (cytokines or whatever) specific to rtx's action but wouldn't the trials benefit from a standard measure of patient-focused disabilty?

 

[rosamary]

Hi. It is rosemary. But Prof Edwards, b cos I am too thick to start a new topic, I am sticking this right here:

Hi This is a friend of Rosemary. I was interested to read about the Rituximab trial, as my daughter was treated with it last year to combat a very aggressive form of dermatomyositis. Treatment with cyclophosphamide and then IVIG had proved ineffective, but the effect of the Retuximab was rapid and quite dramatic.
In addition to this, my dayghter has suffered since her early teens from simple partial seizures. It took 3 years for that diagnosis to be made, and for a while ME was seriously considered. It is interesting to note, in the light of the current drug trial, that since falling ill with the auto immune condition, she has not suffered any seizures at all. Strangely, we have been given to understand that there is no connection between her two conditions.

Hi rosemary here. Just thought we'd 'share' .

N
Back to the wine for me. The scribe is tea total.

 

[Denise]

I hope I am not misinterpreting what I have read on this thread.

I think that validated (for this illness) outcome measures must be used in studies so that results across studies can be compared.

 

[Jonathan Edwards]

We need to be careful over average function. If we have a number of readings then the median is good. My concern would be an outlier say due to an immediate but short-lived drug reaction good or bad could skew results,

I am a bit sceptical about medians. I agree that a mean sounds dodgy but in reality what Bob means by average may be more like 'total for a month'. You might measure total activity by actimeter for 30 days to even things out - and that would in common sense terms indicate 'what I can actually do over a month'. That seems to me to make more sense than the median count for any one day. And you might say that what matters most is not crashing, or having very bad days. Those very bad days might be the outliers with really low actimeter scores. You might then want a 50% improvement to be at least a 50% increase in your WORST day's actimeter score.

My suspicion is that the maths should follow the common sense interpretation of each particular type of score rather than statistical rules for populations of data points. I think maybe it relates back to the very valid point you made before about these not being linear measures much of the time - at least in terms of their human significance.

 

[Jonathan Edwards]

But that's the issue - do we really have new problems in every trial? Especially if we're talking about measuring patients' disability in the kind of real-world terms that we've been discussing, for example?

I understand that the rtx trials might want to measure certain biological things (cytokines or whatever) specific to rtx's action but wouldn't the trials benefit from a standard measure of patient-focused disabilty?

Very likely yes, but when you have people designing trials who cannot think things through properly the worst thing you can do is give them a rule book to follow. I have seen it too many times before. We get papers talking of 'validated measures' which were validated for something else etc. etc. There may come a time when there is an ACR grade equivalent for ME but for the moment I think things are much more up in the air and each trial does tend to be different in important ways. Even the ACR system is often a millstone around one's neck.

 

[Jonathan Edwards]

I hope I am not misinterpreting what I have read on this thread.

I think that validated (for this illness) outcome measures must be used in studies so that results across studies can be compared.

I know that people often say this but my own experience in doing trials is that it is not that important. If two trials using different methods seem to give different results for the same basic question then you need another study that is designed to sort out why there is a difference. Standardising all too often means that you do not take advantage of the best method for a particular situation. For me validation is mostly about getting information about reproducibility or consistency of a method - not about whether it is the right way to answer a question - that is a matter to be determined by careful logical thought on each occasion.

 

[eafw]

I am interested in what people think about the design of outcome measures for trials and perhaps epidemiological research. Should they be subjective or objective and what exactly do those terms imply? Should people try to measure things like fatigue quantitatively or is this a bogus idea? Should specific neurocognitive or activity (e.g. CPET) test be used or are they too indirect?

I think it is possible to have subjective and quantitative measures which can be very useful - even without being blinded. Particularly if you are looking at improvement per individual.

There are also many cheap and easy to administer tests which could give us these measures.

A simple example, for POTS. Forget the tilt table test (it's unnecessarily unpleasant and demanding for a start). Instead have 24hr ecg followed up by a wrist strap or even a pulse oximeter for the patient to monitor and record day to day. The key is lots of measurements over weeks and months, before and then after starting the treatment regime. You will see trends over time and can ask how the person is feeling too. Add in the actimeter and you will see if any increase in activity is there.

Would this count as good enough for epidemiological studies (as opposed to blinded clinical trials ?)

Other basic tests: HRV, salivary cortisol, grip tests, balance and gait (don't know why we don't see more about gait problems bearing in mind the neuro component of the illness - rhombergs test, 360 deg turn many similar all easy to measure and track over time)

A good sort-of-subjective sort-of-objective measure is how many "discretionary" activities a person is engaging in. That is, what gets cut out of your life when you're bad and what will be there when you're not so bad (I know this will be very different for people with different severities)

But there should be a way to put all this together into something that gives properly useful information, I don't think it's impossible even without obvious disease bio markers, you can still assess improvements in functioning.

http://www.rehabmeasures.org/rehabweb/allmeasures.aspx

Lots to work with already out there as well (above link)