IMMPACT recommendations: Interpreting the clinical importance of group differences in

[Dolphin]

This paper was referenced in the reply by the authors to the letters to the Lancet on the PACE Trial.

None of it is specifically about ME/CFS so it is probably not going to be of interest to many/most.

I underlined bits so thought I'd write them down somewhere for what they are worth.

Interpreting the clinical importance of group differences in chronic pain clinical trials: IMMPACT recommendations.

Pain. 2009 Dec;146(3):238-44.

Dworkin RH, Turk DC, McDermott MP, Peirce-Sandner S, Burke LB, Cowan P, Farrar JT, Hertz S, Raja SN, Rappaport BA, Rauschkolb C, Sampaio C.

SourceDepartment of Anesthesiology, University of Rochester School of Medicine and Dentistry, Rochester, NY 14642, USA. robert_dworkin@urmc.rochester.edu

Abstract
(I made each line a paragraph)

An essential component of the interpretation of results of randomized clinical trials of treatments for chronic pain involves the determination of their clinical importance or meaningfulness.

This involves two distinct processes--interpreting the clinical importance of individual patient improvements and the clinical importance of group differences--which are frequently misunderstood.

In this article, we first describe the essential differences between the interpretation of the clinical importance of patient improvements and of group differences.

We then discuss the factors to consider when evaluating the clinical importance of group differences, which include the results of responder analyses of the primary outcome measure, the treatment effect size compared to available therapies, analyses of secondary efficacy endpoints, the safety and tolerability of treatment, the rapidity of onset and durability of the treatment benefit, convenience, cost, limitations of existing treatments, and other factors.

The clinical importance of individual patient improvements can be determined by assessing what patients themselves consider meaningful improvement using well-described methods.

In contrast, the clinical meaningfulness of group differences must be determined by a multi-factorial evaluation of the benefits and risks of the treatment and of other available treatments for the condition in light of the primary goals of therapy.

Such determinations must be conducted on a case-by-case basis, and are ideally informed by patients and their significant others, clinicians, researchers, statisticians, and representatives of society at large.

PMID:19836888[PubMed - indexed for MEDLINE]

 

[Dolphin]

IMMPACT recommendations for individuals and groups are different

The Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) recently reviewed and
recommended specific methods that can be used for interpreting the clinical importance of treatment outcomes for individual
patients in chronic pain clinical trials [11]. These recommendations included a set of provisional benchmarks for interpreting
changes in measures of pain, physical and emotional functioning, and global improvement that represent outcome domains
recommended previously by IMMPACT [10] and [35]. The methods that were discussed for determining clinical importance and
the recommended benchmarks all involved changes that occur within individuals from the beginning of a clinical trial to its
conclusion. For example, decreases in patients pain intensity of >=30% were considered moderately important
improvements, whereas decreases of >=50% were considered substantial improvements [11].
The determination of criteria for clinically meaningful improvements for individual patients is necessary to categorize patients as
responders or non-responders. Identifying which patients can be considered responders is often a critical aspect of
interpreting clinical trial results. Responder analyses make it possible to compare the percentages of patients who achieve
meaningful outcomes between treatment and control groups or between different treatment conditions, a readily interpretable
approach to presenting clinical trial outcomes [13]. However, recommendations for determining clinically meaningful
improvements in patients do not address an equally important consideration in the interpretation of clinical trial results
specifically, what magnitude of difference between treatment groups should be considered clinically meaningful?
In the IMMPACT recommendations describing the determination of clinically important changes for individual patients, the
authors emphasized that the importance of group differences can only be established in the broader context of the disease
being treated, the currently available treatments, and the overall riskbenefit ratio of the treatment [11,p. 108].

(this is talked about more later)

The second part of this paragraph discusses the some of the factors that can make up a positive increase in a trial:

One possible interpretation of the results of these clinical trials of OA knee pain and painful DPN is that the mean differences in
response that were found between the treatment and placebo groups are not clinically meaningful. However, of existing
pharmacologic treatments, acetaminophen and NSAIDs are internationally considered either first- or second-line for OA pain
[1], [21] and [39] and duloxetine and pregabalin are internationally considered either first- or second-line for painful DPN [2], [9]
and [25]. Of course, widespread clinical use does not provide evidence of clinical meaningfulness. If differences in mean
response of the magnitude found in the trials of these medications were not considered clinically relevant benefits, however,
clinicians would have limited therapeutic options for the pharmacologic treatment of these and other chronic pain conditions.
It is not surprising that the improvements patients consider clinically meaningful are generally larger than the differences found
between efficacious treatments and placebo in chronic pain clinical trials. Meaningful change in individual patients reflects any
effects of the active treatment, placebo and other non-specific effects of the clinical setting, natural history and spontaneous
resolution, and statistical regression to the mean. Differences between treatment and placebo groups, however, reflect the
incremental benefits of active treatments that contribute to improvement after subtracting out placebo and other non-specific
effects, natural history, and regression to the mean, for example, the pharmacologic effects of a medication. In addition, the
differences between treatment and placebo groups in chronic pain clinical trials are limited by the magnitudes of the responses
in the placebo groups, which can be substantial [22] and [26] and reflect multiple factors, especially placebo and other
non-specific effects of clinical trial participation. Although these factors also affect response in patients receiving active
treatment, a substantial response in the placebo group can attenuate the group difference with an active treatment if there is a
floor below which treatment rarely reduces pain.

Table 1. Factors to consider in determining the clinical meaningfulness of group differences.

Statistical significance of the primary efficacy analysis (typically necessary but not sufficient to determine that the group difference <cut-off>

Magnitude of improvement in the primary efficacy outcome with treatment

Results of responder analyses

Treatment effect size compared to available treatments

Rapidity of onset of treatment benefit

Durability of treatment benefit

Results for secondary efficacy endpoints (e.g., improvements in physical and/or emotional functioning)

Safety and tolerability

Convenience

Patient adherence

Cost

Different mechanism of action vs. existing treatments

Limitations of available treatments

Other benefits (e.g., few or no drug interactions, availability of a test that predicts a good therapeutic response)

all other things being equal, treatments that have a rapid onset and last a long time are better than those lacking these features.

4.4. Other factors
The overall benefit of a potential treatment must also be viewed in light of its anticipated ease of use by patients in the
community and the likelihood that they will adhere to treatment. Clinical trials can provide some evidence of patient adherence
to treatment, but their tightly controlled nature can make generalization to the community problematic and judgments about
anticipated ease of use and adherence may require other considerations. For example, oral medications are generally
preferred to those that require injection, and a medication that can be taken in the morning and at bedtime is not only more
convenient for patients than one that must be taken five times daily but will also promote better adherence with therapy. In
addition, a treatment that either does not require titration or that can be rapidly titrated to optimum dosage will be preferred by
patients (and clinicians) to one that requires a lengthy and closely monitored titration to achieve its efficacious dosage. The
cost of a treatment is another important source of patient non-adherence with treatment, of course, and is also important in
considerations of treatment cost-effectiveness.

In approximately half of patients with chronic pain, existing treatments are not effective or are poorly tolerated, and in patients
who do respond, it is relatively rare for pain to be completely relieved or even reduced to mild severity. Because chronic pain
treatments have incomplete efficacy, a treatment whose mechanism(s) of action is different from those of existing therapies
may be effective in patients who are currently non-responders, presumably because the pathophysiologic mechanism of the
patients pain is targeted by the new treatment and not by existing therapies. For similar reasons, treatments with a different
mechanism of action could also be used in combination with existing therapies to augment treatment response in those patients
who are partial responders [9]. Moreover, because chronic pain treatments are often poorly tolerated, a treatment with different
mechanisms of action may be better tolerated by patients who cannot tolerate the adverse effects of existing therapies. For
these reasons, the benefit of a treatment with a different mechanism of action from those of existing therapies could be
considered more clinically meaningful than if the treatment had the same mechanism of action as existing therapies. (contd.)

Comment: this would seem to suggest that the approach in the UK, where only studies of non-pharmacological rehabilitative interventions (and research relating to that model) are generally funded is a flawed model as is the health service only paying for those sorts of treatements.

The limitations of any available treatments should also be taken into account when interpreting group differences. Specifically,
a modest benefit compared with placebo may be more clinically meaningful if existing therapies have important limitations, such
as poor safety and tolerability or limited efficacy, compared with a similar benefit occurring in a context of existing therapies that
have excellent safety, tolerability, and efficacy.

There are other factors specific to certain treatments that must be considered in evaluating the clinical meaningfulness of their
benefits. These include an absence or limited number of drug interactions, especially because many chronic patients are older
and may not only be receiving other analgesics but are likely to be taking a variety of non-analgesic medications. In addition,
modest benefits with a treatment for which there is a simple test or procedure available that predicts whether a patient will have
a positive therapeutic response (e.g., a trial of transcutaneous electrical nerve stimulation) may be more clinically meaningful
than comparable benefits associated with a treatment for which response cannot be predicted. In many circumstances,
patients, clinicians, and third-party payers would consider it beneficial to be able to predict therapeutic response because
increasing the likelihood of a positive response reduces the time, risks, and expenses associated with therapeutic failure. (contd.)

Currently because of the lack of biological research into ME/CFS (which some would like to continue), we don't have a good idea who might benefit from GET programs. Some people may claim everyone with chronic fatigue/chronic fatigue syndrome will benefit but this position I do not believe is justified.

(contd.) Finally, any research design features of the clinical trial that could have attenuated the magnitude of the group difference in
response must also be considered in its evaluation. For example, insufficient pain at baseline could make it difficult or even
impossible for an active treatment to show greater reduction in pain compared with placebo. In addition, patients in placebo
groups are likely to use more rescue medication than patients treated with efficacious analgesics, and it is certainly possible
that a substantially greater use of rescue analgesics in a placebo group could attenuate the difference in pain relief compared
with the active treatment. Of course, clinical meaningfulness cannot be attributed to a group difference simply because the trial
has design limitations. However, such factors can provide possible explanations for more modest benefits than would otherwise
have been expected and thereby provide a basis for conducting additional studies (ideally, without such limitations) to
determine whether the magnitude of the treatment benefit has been underestimated.

 

[Dolphin]

Thought I'd give the Conclusions its own section:

6. Conclusions

The clinical importance of individual patient improvements and of group differences are distinct aspects of the results of a clinical trial, and both play an important role in the evaluation of its outcomes. The clinical importance of individual patient improvements in chronic pain trials can be determined by assessing what patients themselves consider meaningful improvement using well-defined methods. In contrast, the clinical meaningfulness of group differences must be determined by a multi-factorial evaluation of the benefits and risks of the treatment and of other available treatments for the condition in light of the primary goals of therapy. Differences in mean reductions in pain between active treatment and placebo groups do not adequately describe the potential benefits of a treatment in the population of individuals with chronic pain. Although a statistically significant group difference in pain intensity is typically necessary for a demonstration of efficacy, a determination of the clinical meaningfulness of this difference should not be based solely on its magnitude. Group mean differences can obscure meaningful individual patient improvements and other benefits and risks. Information about percentages of responders and evaluation of secondary outcomes, safety and tolerability, and the other factors we have discussed (see Table 1) must all be considered to adequately understand the therapeutic benefit associated with a treatment for chronic pain.

Unfortunately, because this multi-factorial evaluation must consider the benefits and risks of the treatment and of other available treatments for the condition, it is impossible to provide specific guidelines for determining whether or not a specific group difference is clinically meaningful. Such determinations must be conducted on a case-by-case basis, and are ideally informed by patients and their significant others, clinicians, researchers, statisticians, and representatives of society at large. Although such a process may appear daunting, it bears striking resemblance to the advisory committees and similar groups that are routinely assembled by regulatory agencies around the world when evaluating the efficacy and safety of medical interventions. The involvement of multiple stakeholders ensures a comprehensive evaluation that considers the different and complementary perspectives necessary to determine whether the benefits of a treatment outweigh its risks and constitute a clinically meaningful addition to available therapies.

 

[oceanblue]

This looks really interesting, but unfortunately my brain can't read the text with all those line-breaks, I find it too confusing...

 

[Dolphin]

This looks really interesting, but unfortunately my brain can't read the text with all those line-breaks, I find it too confusing...

Are you talking about the abstract or the other posts? I'm not 100% sure what you are saying when you say "line breaks" - would you prefer if it went right the way over to the right hand side (like the last extract)?

Or do you have characters when you read it that say <break> or <b> or whatever?

 

[oceanblue]

When I copy/paste here the line breaks sort themselves out for some reason

The Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) recently reviewed and
recommended specific methods that can be used for interpreting the clinical importance of treatment outcomes for individual
patients in chronic pain clinical trials [11]. These recommendations included a set of provisional benchmarks for interpreting
changes in measures of pain, physical and emotional functioning, and global improvement that represent outcome domains
recommended previously by IMMPACT [10] and [35]. The methods that were discussed for determining clinical importance and
the recommended benchmarks all involved changes that occur within individuals from the beginning of a clinical trial to its
conclusion. For example, decreases in patients pain intensity of >=30% were considered moderately important
improvements, whereas decreases of >=50% were considered substantial improvements [11].
The determination of criteria for clinically meaningful improvements for individual patients is necessary to categorize patients as
responders or non-responders. Identifying which patients can be considered responders is often a critical aspect of
interpreting clinical trial results. Responder analyses make it possible to compare the percentages of patients who achieve
meaningful outcomes between treatment and control groups or between different treatment conditions, a readily interpretable
approach to presenting clinical trial outcomes [13]. However, recommendations for determining clinically meaningful
improvements in patients do not address an equally important consideration in the interpretation of clinical trial results
specifically, what magnitude of difference between treatment groups should be considered clinically meaningful?
In the IMMPACT recommendations describing the determination of clinically important changes for individual patients, the
authors emphasized that the importance of group differences can only be established in the broader context of the disease
being treated, the currently available treatments, and the overall riskbenefit ratio of the treatment [11,p. 108].

The second part of this paragraph discusses the some of the factors that can make up a positive increase in a trial:
One possible interpretation of the results of these clinical trials of OA knee pain and painful DPN is that the mean differences in
response that were found between the treatment and placebo groups are not clinically meaningful. However, of existing
pharmacologic treatments, acetaminophen and NSAIDs are internationally considered either first- or second-line for OA pain
[1], [21] and [39] and duloxetine and pregabalin are internationally considered either first- or second-line for painful DPN [2], [9]
and [25]. Of course, widespread clinical use does not provide evidence of clinical meaningfulness. If differences in mean
response of the magnitude found in the trials of these medications were not considered clinically relevant benefits, however,
clinicians would have limited therapeutic options for the pharmacologic treatment of these and other chronic pain conditions.
It is not surprising that the improvements patients consider clinically meaningful are generally larger than the differences found
between efficacious treatments and placebo in chronic pain clinical trials. Meaningful change in individual patients reflects any
effects of the active treatment, placebo and other non-specific effects of the clinical setting, natural history and spontaneous
resolution, and statistical regression to the mean. Differences between treatment and placebo groups, however, reflect the
incremental benefits of active treatments that contribute to improvement after subtracting out placebo and other non-specific
effects, natural history, and regression to the mean, for example, the pharmacologic effects of a medication. In addition, the
differences between treatment and placebo groups in chronic pain clinical trials are limited by the magnitudes of the responses
in the placebo groups, which can be substantial [22] and [26] and reflect multiple factors, especially placebo and other
non-specific effects of clinical trial participation. Although these factors also affect response in patients receiving active
treatment, a substantial response in the placebo group can attenuate the group difference with an active treatment if there is a
floor below which treatment rarely reduces pain.


In approximately half of patients with chronic pain, existing treatments are not effective or are poorly tolerated, and in patients
who do respond, it is relatively rare for pain to be completely relieved or even reduced to mild severity. Because chronic pain
treatments have incomplete efficacy, a treatment whose mechanism(s) of action is different from those of existing therapies
may be effective in patients who are currently non-responders, presumably because the pathophysiologic mechanism of the
patients pain is targeted by the new treatment and not by existing therapies. For similar reasons, treatments with a different
mechanism of action could also be used in combination with existing therapies to augment treatment response in those patients
who are partial responders [9]. Moreover, because chronic pain treatments are often poorly tolerated, a treatment with different
mechanisms of action may be better tolerated by patients who cannot tolerate the adverse effects of existing therapies. For
these reasons, the benefit of a treatment with a different mechanism of action from those of existing therapies could be
considered more clinically meaningful than if the treatment had the same mechanism of action as existing therapies. (contd.)

The limitations of any available treatments should also be taken into account when interpreting group differences. Specifically,
a modest benefit compared with placebo may be more clinically meaningful if existing therapies have important limitations, such
as poor safety and tolerability or limited efficacy, compared with a similar benefit occurring in a context of existing therapies that
have excellent safety, tolerability, and efficacy.

Comment: this would seem to suggest that the approach in the UK, where only studies of non-pharmacological rehabilitative interventions (and research relating to that model) are generally funded is a flawed model as is the health service only paying for those sorts of treatements.

There are other factors specific to certain treatments that must be considered in evaluating the clinical meaningfulness of their
benefits. These include an absence or limited number of drug interactions, especially because many chronic patients are older
and may not only be receiving other analgesics but are likely to be taking a variety of non-analgesic medications. In addition,
modest benefits with a treatment for which there is a simple test or procedure available that predicts whether a patient will have
a positive therapeutic response (e.g., a trial of transcutaneous electrical nerve stimulation) may be more clinically meaningful
than comparable benefits associated with a treatment for which response cannot be predicted. In many circumstances,
patients, clinicians, and third-party payers would consider it beneficial to be able to predict therapeutic response because
increasing the likelihood of a positive response reduces the time, risks, and expenses associated with therapeutic failure. (contd.)

Currently because of the lack of biological research into ME/CFS (which some would like to continue), we don't have a good idea who might benefit from GET programs. Some people may claim everyone with chronic fatigue/chronic fatigue syndrome will benefit but this position I do not believe is justified.

(contd.) Finally, any research design features of the clinical trial that could have attenuated the magnitude of the group difference in
response must also be considered in its evaluation. For example, insufficient pain at baseline could make it difficult or even
impossible for an active treatment to show greater reduction in pain compared with placebo. In addition, patients in placebo
groups are likely to use more rescue medication than patients treated with efficacious analgesics, and it is certainly possible
that a substantially greater use of rescue analgesics in a placebo group could attenuate the difference in pain relief compared
with the active treatment. Of course, clinical meaningfulness cannot be attributed to a group difference simply because the trial
has design limitations. However, such factors can provide possible explanations for more modest benefits than would otherwise
have been expected and thereby provide a basis for conducting additional studies (ideally, without such limitations) to
determine whether the magnitude of the treatment benefit has been underestimated.

 

[Dolphin]

Hi oceanblue, you've genuinely lost me. Mine looks, to me, like it has exactly the same line breaks as yours has.

 

[Esther12]

it could be that you've got different sized screens, and it's auto-formatting. I've not read this properly yet, but thanks for posting it up.

 

[Dolphin]

it could be that you've got different sized screens, and it's auto-formatting. I've not read this properly yet, but thanks for posting it up.

Are you saying that what I posted looks bad to you? What is happening - are there short lines and long lines?

I generally look at things (and post things) in IE at 100% so I've just opened it in Opera and the line breaks look the same to me as what oceanblue posted.

 

[oceanblue]

Weird. I use firefox, and the copy/paste fixed the line breaks in firefox, but problem remained in IE.

Anway, I was particularly interested in the point that therapies with different modes of working are more valuable due to possibility of complementarity (for responders to other types of treatment) or as an alternative approach for non-responders to other therapy types.

I think Marco posted on the issue of asking patients to establish what counts as 'clinically useful'; what a radical idea.

 

[Dolphin]

Weird. I use firefox, and the copy/paste fixed the line breaks in firefox, but problem remained in IE.

Thanks. Still not 100% sure what the problem is - short and long lines?

Anway, I was particularly interested in the point that therapies with different modes of working are more valuable due to possibility of complementarity (for responders to other types of treatment) or as an alternative approach for non-responders to other therapy types.

Yes, it's interesting.

I think Marco posted on the issue of asking patients to establish what counts as 'clinically useful'; what a radical idea.

Ok.

It also comes up in a paper I posted on recently:
http://forums.phoenixrising.me/show...ient-reported-outcome-measures-used-in-CFS-ME
Quality and acceptability of patient-reported outcome measures used in CFS/ME

Here are the last two paragraphs of discussion section - they give an idea of the paper and hopefully how it chimes in with what you are saying:

Dominance of the assessment of emotional well-being
and the symptomatic impact of CFS/ME illustrated in this
review, and the lack of CFS/ME-specific measure that
captures the broad-ranging experience of the condition
highlights the discrepancies that exist between outcomes
assessed in research and those identified by patients as
significant to their experience of living with CFS/ME,
including social well-being and physical function [6]. The
poor quality of reviewed PROMs combined with the failure
to measure genuinely important patient outcomes suggests
that high quality and relevant information about treatment
effect is lacking [123]. Such dilemmas have been reported
in other conditions, for example, Diabetes [124, 125].
Exemplified in rheumatology [126], resources are required
to drive consensus between patients with CFS/ME and
health professionals towards the standardization of
assessment practice, identification of important and
appropriate health outcomes and selection of good quality,
acceptable measures. The resulting availability of relevant
and credible information about treatment effect will support
informed decision-making and patient choice [127].

A more scientific focus towards the collaborative
development and ongoing evaluation of PROMs in CFS/
ME is essential to the future of evidence-based health care.
Without investment in appropriate and rigorous measurement
of health, the true burden of CFS/ME and the relative
success of health care will not be fully understood and
communicated. This investment should include the development
of a patient-derived assessment that includes
detailed descriptions of CFS/ME-specific health and quality
of life across the range of domains considered important
by patients with CFS/ME. Such a measure should be
developed collaboratively with the full involvement of
people with CFS/ME as partners in the process. The lack of
such a measure is an important omission from the battery
of assessment approaches in CFS/ME and must be
addressed as a matter of urgency.

 

[Esther12]

Thanks. Still not 100% sure what the problem is - short and long lines?

To try to clarify (this isn't important though), this is how it looks to me:

and [25]. Of course, widespread clinical use does not provide evidence of clinical meaningfulness.
If differences in mean
response of the magnitude found in the trials of these medications were not considered clinically relevant
benefits, however,
clinicians would have limited therapeutic options for the pharmacologic treatment of these and other chronic
pain conditions.

 

[Dolphin]

Ok, thanks - that's what I meant by "short and long lines".

I don't want that to happen on lots of threads - what browser do you use?

 

[Esther12]

I use firefox.

Would the screen size not matter more though? eg: If the lines fitted in okay, it wouldn't matter. I may have my text size higher than normal too.

I don't normally find it a big problem.

 

[Dolphin]

I use firefox.

Would the screen size not matter more though? eg: If the lines fitted in okay, it wouldn't matter. I may have my text size higher than normal too.

I don't normally find it a big problem.

Thanks for that. I just downloaded Firefox to check (I know this has been a been a big Firefox plot involving you and oceanblue ). It still looks fine for me - that is to say, what I posted is quite even on the right hand side (and it doesn't go write the way across .

But it could be to do with screen size as you mention. Actually, as I write this in the advanced mailing box, below, I can see short and long lines for my previous posts except where I pasted the Conclusions.

Anyway, Esther12 and oceanblue, feel free to alert me to this in future and I'll see what I can do. Perhaps I did something last night specifically.

If anyone with IT knowledge can lead this blind man, I'd be interested in hearing.