Measures of outcome for trials and other studies

[eafw]

I have been trying to figure out what is wrong with the outcome measures currently being used

What is wrong is that the most of those measures have been drawn up without any real understanding of the illness. I know I'm repeating myself (and others) here but this is fundamental to finding out whether a treatment works or not.

I'd also note that a lot of the ground work is already done, there is a lot of out there that patients and some of the better practitioners already know in terms of basic measuring and monitoring of day to day function. We don't have to completely reinvent the wheel here (though I guess this is why you're asking)

 

[MeSci]

I was thinking it may need to be part of any measurement kit including accelerometers, heart/oxygen monitors etc. You could use a website as a reporting mechanism be I thought that may be harder to use and hence get regular inputs.

Could people just choose which option they find easier? I have never used a smartphone but have considerable difficulty finding my way around my very basic phone. I am much more at ease with websites!

Sorry if someone has already made this point. I'm still catching up on this thread and getting mentally tired so would forget this question if I didn't ask it now.

 

[alex3619]

I think Alex mentioned that a researcher or physician had invented a checklist to assess symptoms (but I didn't see any further details posted about this)...

In a very old thread I think I posted the entire checklist, back in 2009 or 2010. I might have trouble finding the hard copy of it again, but it exists here somewhere.

 

[eafw]

I don't really like the idea of manually keeping track of activities all day. It would a pain in the butt, we'd forget to do it half the time (at least!), and it would be exhausting.

No, but that is a problem with trials on people who are not well enough to engage with the trial. It has to be something that the person is able to do.

 

[MeSci]

Some great points!
Questions that are relative eg are things better or worse than in the past should be avoided altogether. When you have been ill for a long time, you forget what is normal!

Likewise, 'with difficulty' is too vague. Such questions should try to describe what such levels of difficulty actually mean in terms of impact on life activities.

Oh - that brings to mind other issues with questionnaires. So many ask things like "Are you more tired than usual/Is your sleep worse than usual?" etc. What do they mean by usual? Usual before ME or usual-with-ME?

If difficulty had to be quantified I think I would find that very difficult! Trying to do that would bring on questionnaire fatigue very quickly, and I might have to stop completely.

 

[MeSci]

For severe patients at least, they should have caregivers who could answer for them, or help with it. Though it could still present a problem for patients without caregivers who are generally more functional, but have hit a really bad patch.

I have had a couple of days, during what has been diagnosed as migraine, when I could not recognise the name of one of my favourite radio comedy programmes in a listings magazine. It was just ten letters.

But I don't suppose occasional problems like this would cause serious issues in a study. When it is a common problem, adaptations would need to be made.

 

[alex3619]

we'd forget to do it half the time

Forget to do what?

 

[Snow Leopard]

Do you think so? That would appear to be very blind optimism, as there is no reason I can think of why a person would be more likely to be in one group rather than another. I expect my own reply would be "Absolutely no idea" or equivalent! But I am unusually logical, I think (borderline Asperger's).

If you were given a question of yes/no and no 'unsure' option, which would you choose? A second question would be your level of confidence in this question, which you could indicate that you are 'very unsure'.

 

[MeSci]

If you were given a question of yes/no and no 'unsure' option, which would you choose? A second question would be your level of confidence in this question, which you could indicate that you are 'very unsure'.

If it were a written questionnaire, I would cross out both and write "no idea" and perhaps add "why did you not include a 'don't know' or 'unsure' option? Do you suspect delusions of clairvoyance?"

If it were an online questionnaire that did not allow one to skip the question, I would have to abandon it, just as I have to abandon many YouGov surveys that ask questions I cannot answer accurately.

I would also be very cross.

It doesn't make sense to my Spockesque brain.

 

[Jonathan Edwards]

There is a difference between requiring everyone to measure exactly the same things and having a standardised minimum set of data, such as that proposed in the following paper. Researchers are still free to use their own measures beyond this if they believe they would be more useful.

http://www.ncbi.nlm.nih.gov/pubmed/22306456

Brain Behav Immun. 2012 Mar;26(3):401-6. doi: 10.1016/j.bbi.2012.01.014. Epub 2012 Jan 28.
Minimum data elements for research reports on CFS.
Jason LA1, Unger ER, Dimitrakoff JD, Fagin AP, Houghton M, Cook DB, Marshall GD Jr, Klimas N, Snell C.
Author information
Abstract
Chronic fatigue syndrome (CFS) is a debilitating condition that has received increasing attention from researchers in the past decade. However, it has become difficult to compare data collected in different laboratories due to the variability in basic information regarding descriptions of sampling methods, patient characteristics, and clinical assessments. The issue of variability in CFS research was recently highlighted at the NIH's 2011 State of the Knowledge of CFS meeting prompting researchers to consider the critical information that should be included in CFS research reports. To address this problem, we present our consensus on the minimum data elements that should be included in all CFS research reports, along with additional elements that are currently being evaluated in specific research studies that show promise as important patient descriptors for subgrouping of CFS. These recommendations are intended to improve the consistency of reported methods and the interpretability of reported results. Adherence to minimum standards and increased reporting consistency will allow for better comparisons among published CFS articles, provide guidance for future research and foster the generation of knowledge that can directly benefit the patient.

The abstract does not indicate what data elements are minimum. I think there is a lot to be said for minimum data for demographic definition of patients. Whether this applies to outcome measures I am not sure.

 

[Jonathan Edwards]

What is wrong is that the most of those measures have been drawn up without any real understanding of the illness. I know I'm repeating myself (and others) here but this is fundamental to finding out whether a treatment works or not.

I'd also note that a lot of the ground work is already done, there is a lot of out there that patients and some of the better practitioners already know in terms of basic measuring and monitoring of day to day function. We don't have to completely reinvent the wheel here (though I guess this is why you're asking)

I don't think there is any suggestion of reinventing wheels in terms of methods of measurement - my assumption is that we pick from things that are out there. But it seems the we may need to reinvent the wheel in terms of trying to devise a single primary outcome measure that might work for trials of unblinded treatments which mitigated the bias problems of subjective (reported) measures alone but retained the greater relevance that those measures tend to have in compared to more objective measures. I think some form of composite score is almost inevitable but the big question is how you construct that.

 

[Denise]

In terms of clinical outcomes and their assessment in the US (in particular for things that FDA would require), this workshop may be of interest (I have highlighted the webcast info in blue):

"PDUFA V Clinical Outcome Assessments
Public Workshop
The Food and Drug Administration (FDA) is announcing a public workshop entitled: Clinical Outcomes Assessment Development and Implementation: Opportunities and Challenges. The purpose of the public workshop is to provide updates on accomplishments, challenges, and ongoing efforts in the use of clinical outcome assessments (COAs), and plan for the future of COA development and utilization in drug development programs, including how to incorporate the patient voice in drug development using well-defined and reliable patient-centered outcome measures. The public workshop will also discuss standards for COA use and collaborative processes for COA development and dissemination.

The public workshop will be held on:

Date

April 1, 2015

Time

8:30 a.m. to 5:00 p.m.

Location

The public workshop will be held at the FDA White Oak Campus, 10903 New Hampshire Ave., Bldg. 31, the Great Room (Rm. 1503), Silver Spring, MD 20993-0002. Entrance for the public workshop participants (non-FDA employees) is through Bldg. 1 where routine security check procedures will be performed. For parking and security information, please refer to Public Meetings at the FDA White Oak Campus

Registration

Participation can be either in person attendance or by webcast. There is no fee to attend the public workshop, but attendees should register in advance. Space is limited, and registration for in person attendance will be on a first-come, first-served basis.

Before March 27, 2015, we ask all persons who wish to participate either in person or by webcast to Register online .

For those without Internet access, please contact Michelle Campbell at 240-402-6019.

Webcast Information

Access the webcast at: Clinical Outcomes Assessment Public Workshop. This link will be active on the day of the meeting.

Additional Questions

If you have additional questions, please contact COApublicworkshop@fda.hhs.gov"



http://www.fda.gov/Drugs/NewsEvents/ucm431040.htm

 

[Simon]

This thread looks to me like a spectacularly good advertisment for the idea of patients and researchers can collaborate in a productive way.

It's been an incredible discussion (taken me all day to catch up) with bucket loads of high quality input focused on what's going to work in the real world - I've read quite a few papers of questionnaires developed by researchers and often thought 'but is that really relevant'? (Great peer-reviewed critique of such scales in CFS discussed here and here.)

Hats off to @Jonathan Edwards for starting the thread and being so willing to discuss ideas. Hats off to all the patient contributors too (hoping to be one myself, once I have the energy).

 

[eafw]

trying to devise a single primary outcome measure that might work

I'm not sure that would even be possible

I think some form of composite score is almost inevitable but the big question is how you construct that.

Yes (to the composite score). As to how to construct a scoring system, even a composite one, I think we would need to have some tailoring to subgroups, at least in terms of severity and length of illness - rather than a universal. And it is good to start from a point of asking about with patients experience rather than what researchers think they want to measure.

 

[eafw]

as an example, are you thinking you would want to end up with something like this

http://www.aahf.info/pdf/Berg_Balance_Scale.pdf

or less detail, not that sort of measurement ?

 

[A.B.]

This is probably obvious and might have been said already but I expect a real treatment to improve both subjective symptoms and objective / obvious / noticeable improvements in real world functioning and ability to get stuff done. And both of these need to be sustained for a while (six months sounds good).

If the improvements are not sustained: the results mean nothing. The illness fluctuates too much, and lack of stamina is a core symptom.

If subjective symptoms are improved but functioning isn't: the treatment is probably targeting something other than the
core pathology.

If functioning increases but subjective symptoms persist or even increase: the patient is just trying harder while ignoring warning signs.

Merely achieving an improvement of subjective symptoms would still be good but we have to distinguish between lessening suffering and real treatments.

 

[Jonathan Edwards]

I'm not sure that would even be possible

Yes (to the composite score). As to how to construct a scoring system, even a composite one, I think we would need to have some tailoring to subgroups, at least in terms of severity and length of illness - rather than a universal. And it is good to start from a point of asking about with patients experience rather than what researchers think they want to measure.

If no single primary outcome measure is possible I think it is goodbye to trials in ME being of any value. We have to have a definable primary outcome measure if we want to be able to decide if a treatment should be recommended. Without it the trial would not even by publishable (or registerable.)

I think tailoring of subgroups is already on board in the above discussion of options for OI and cognitive problems. I am not sure how severity and length of illness come in to it. You could certainly do trials just on short duration disease or just severe disease etc. And outcome scales could be recalibrated but I am not quite sure how that impacts on the main question to be honest.

 

[Jonathan Edwards]

as an example, are you thinking you would want to end up with something like this

http://www.aahf.info/pdf/Berg_Balance_Scale.pdf

or less detail, not that sort of measurement ?

I am not quite sure why a balance scale would be a priority. It could be one of a range of options for what I have suggested as a test of physiological dysfunction but this particular one looks a bit cumbersome. I am not so much interested in individual measuring techniques as in how one can construct an overall scoring system that will overcome the methodological problems we have seen in trials to date.

 

[Jonathan Edwards]

This is probably obvious and might have been said already but I expect a real treatment to improve both subjective symptoms and objective / obvious / noticeable improvements in real world functioning and ability to get stuff done. And both of these need to be sustained for a while (six months sounds good).

If the improvements are not sustained: the results mean nothing. The illness fluctuates too much, and lack of stamina is a core symptom.

If subjective symptoms are improved but functioning isn't: the treatment is probably targeting something other than the
core pathology.

If functioning increases but subjective symptoms persist or even increase: the patient is just trying harder while ignoring warning signs.

Merely achieving an improvement of subjective symptoms would still be good but we have to distinguish between lessening suffering and real treatments.

That is a pretty good summary of what I am trying to pin down. I am suggesting adding in further some evidence of improved physiology. That may be spurious but I think it has further potential to convince everyone that this is, as you say, a real treatment, and not just someone saying they are better AND trying harder, just to please the therapist or to buy in to an overoptimistic message.

 

[eafw]

I am not quite sure why a balance scale would be a priority.

No, sorry am not being quite clear here. I wasn't suggesting a balance scale. I was asking if this was an example of the sort of thing you were after. I'm trying to get a better hold of what you are asking or what you expect this outcome measure to look like

If no single primary outcome measure is possible I think it is goodbye to trials in ME being of any value. We have to have a definable primary outcome measure if we want to be able to decide if a treatment should be recommended.

I seem to be missing an understanding here. When you say "single measure" you mean a single overall score made up of a number of aspects (like the Berg scale ?) and something that is used in all cases for all trials or not that at all ?