Study findings challenge the content validity of the Canadian Consensus Criteria for adolescent CFS

[A.B.]

Aim
The 2003 Canadian Consensus Criteria for chronic fatigue syndrome (CFS) are often assumed to suggest low-grade systemic inflammation, but have never been formally validated. This study explored the content validity of the Criteria in a sample of adolescents with CFS selected according to a wide case definition

Methods
A total of 120 CFS patients with a mean age of 15.4 years (range 12–18 years) included in the NorCAPITAL project were post hoc subgrouped according to the Canadian Consensus Criteria. Those who satisfied the criteria (Criteria positive) and those who did not (Criteria negative) were compared across a wide range of disease markers and markers of prognosis

Results
A total of 46 patients were classified as Criteria positive, 69 were classified as Criteria negative and five could not be classified. All disease markers were equal across the two groups, except the digit span backward test of cognitive function, which showed poorer performance in the Criteria positive group. Also, the prognosis over a 30 week period was equal between the groups

Study findings challenge the content validity of the Canadian Consensus Criteria for adolescent chronic fatigue syndrome


I have not read the article since it's behind a paywall. Could anyone with access tell us a bit more?

 

[Valentijn]

Study findings challenge the content validity of the Canadian Consensus Criteria for adolescent chronic fatigue syndrome


I have not read the article since it's behind a paywall. Could anyone with access tell us a bit more?

The protocol is at www.oslo-universitetssykehus.no/SiteCollectionDocuments/Om%20oss/Avdelinger/Kvinne-%20og%20barneklinikken/Kronisk%20utmattelsessyndrom/NorCAPITAL%20final%20protocol.pdf

 

[Valentijn]

And http://www.biomedcentral.com/1756-0500/5/418/ seems to be the primary paper produced by the study. The new one is more peripheral.

 

[Simon]

Study findings challenge the content validity of the Canadian Consensus Criteria for adolescent chronic fatigue syndrome
- see above for absract & conclusion that CCC patients no different from CFS ones.

Nice idea, shame about the execution​

The basic idea of this study is pretty interesting. Take a bunch of broadly defined CFS patients, select from that group those that meet the much tighter Canadian Consensus Criteria, and see if these CCC patients differ on a range of objective measures from the non-CCC patients.

The authors found no significant difference between the two groups (pretty minor cognitive one) and concluded the CCC do not select a group of patients who are different from generic any-definition-will-do CFS patients.

Personally, I think the ability to select patients with different biological characteristics IS a key test any worthwhile case definition must pass. Unfortunately, it seems this test wasn't up to much, because they way they defined CCC is highly suspect - using a questionnaire of their own making without any input from the CCC group.

How ALL patients were defined:

To be included we required only three months of unexplained chronic/relapsing fatigue of new onset, causing disability to a degree that prevented normal school attendance. We did not require that patients met any other accompanying symptom criteria. The referring units were required to confirm that the patient did not have any medical or psychiatric disorder that might explain the fatigue, and that they had not experienced any concurrent demanding life event.

Seems fine as broad CFS definitions go

How CCC patients were defined
This is where it goes wrong

We previously developed a CFS symptom inventory for adolescents, assessing the frequency of 24 common symptoms during the preceding month. Each symptom was rated on a five-point Likert scale, ranging from never/rarely present to present all of the time.

All parts of this worksheet correspond to items in the questionnaire of the present study, and questionnaire results were therefore used to split the CFS patients into one subgroup that satisfied the Canadian Consensus Criteria (Criteria positive group) and one subgroup that did not (Criteria negative group). As a general rule, all symptoms required in the Canadian Consensus Criteria had to be present more than once a week for patients in the Criteria positive group, corresponding to a score of three or higher on the CFS symptom inventory.

Note they don't have any severity score (Leonard Jason has done a lot of work showing how important it is to assess severity as well as frequency of symptoms - ignoring severity makes the criteria far too lax). And they only looked at symmptoms for the last month, not six months or more.

This study had some limitations. The questionnaire items used to subgroup CFS patients have not been formally validated for this purpose. However, they correspond closely to the ME/CFS clinical diagnostic worksheet (21 - Sharpe et al).

I’m not sure the authors of the CCC, or case definition experts such as Leonard Jason, would consider a 1991 paper by Michael Sharpe and colleagues (21) a sound basis for diagnosing patients according to the 2003 CCC.

edit: As Dolphin points out, the paper quotes the wrong reference and so their questionnaire is NOT based on Michael Sharpe's 1991 paper, but on their version of the CCC ME/CFS diagnositc worksheet. However, as I argue below, this questionnaire is still unvalidated and doesn't look up to the job of accurately diagnosing Canadian criteria ME/CFS (not enough symptoms and ignoring symptom severity).


To recap, the authors diagnosed CCC using an unvalidated questionnaire based on a 1991 symptom list from Michael Sharpe and colleagues, ignored symptom severity and only considered symptoms over the last month. That doesn't really cut it, so I'm not sure the results of this study tell us very much at all.

However, this is how the authors interpret their study:

Recently, Fink and coauthors provided evidence that bodily distress disorders, which in their study included CFS, might be regard as one entity instead of separate ones (29). The results of the present study seem to support this view.

Per Fink, for the record.

 

[Esther12]

Thanks for looking into that Simon.

My question from just a lazy look at the abstract was: Do they have any evidence that patients who fulfil their loose criteria differ significantly from patients with an even looser criteria (BSS/FSS amd beyond)? Or on to sedentary controls? What value did the measures used have in diagnosing other CFS criteria? Couldn't these sorts of studies be used to justify a steady expansion of criteria until they covered everyone?

 

[Dolphin]

This study had some limitations. The questionnaire items used to subgroup CFS patients have not been formally validated for this purpose. However, they correspond closely to the ME/CFS clinical diagnostic worksheet (21 - Sharpe et al).

I’m not sure the authors of the CCC, or case definition experts such as Leonard Jason, would consider a 1991 paper by Michael Sharpe and colleagues (21) a sound basis for diagnosing patients according to the 2003 CCC.

I'd say the 21 is a typo and they meant 20.

20. Carruthers BM, Jain AK, De Meirleir KL, Peterson DL, Klimas NG, Lerner AM, et al. Myalgic Encephalomyelitis/Chronic Fatigue Syndrome: Clinical Working Case Definition, Diagnostic and Treatment Protocols. J Chron Fatigue Syndr 2003; 11: 7-11.

21. Sharpe MC, Archard LC, Banatvala JE, Borysiewicz LK, Clare AW, David A, et al. A report – chronic fatigue syndrome: guidelines for research. J R Soc Med 1991; 84: 118-21.

They used reference 21 correctly earlier

These include the Oxford definition (21)

Sometimes references get added or subtracted with various revisions before submission and again during peer-review. I notice it not infrequently.

 

[Dolphin]

To test the hypothesis the way they did, I think they needed to show that what was tested showed different results to healthy controls.

If one tests for something that was not different in the patients than in healthy controls, it is not going to be that surprising if one doesn't find differences between different groups of CFS patients. Unfortunately cytokines don't tend to be reliably different in ME/CFS patients than healthies. For example, Hornig et al. (2015) tended to find differences between those ill three years or less and those ill longer, but few differences when these groups were pooled together and compared to healthies.

 

[Dolphin]

One difference between the groups that isn't mentioned in the abstract is step count

Steps per day - number, mean (SD)

Canadian Clinical Criteria group 3958 (2142)

Non-Canadian Clinical Criteria group 5144 (2482)

p value=0.006

 

[Dolphin]

There were two findings that were close to being statistically significant:

BMI - kg/m2, mean (SD)

Canadian Clinical Criteria group: 20.7 (3.7)

Non-Canadian Clinical Criteria group: 22.0 (4.4)

p value=0.081

----

Symptoms suggesting mood disorder - no. (%)

Yes

Canadian Clinical Criteria group: 22 (48)

Non-Canadian Clinical Criteria group: 45 (65)

p-value: 0.064

No

Canadian Clinical Criteria group: 24 (52)

Non-Canadian Clinical Criteria group: 24 (35)

This is potentially a bit interesting. One might expect the more ill group (those who satisfied the Canadian Clinical Criteria, with lower step counts and more symptoms) to have more mood disorders. In fact, the opposite was the case. This suggests the possibility that the Non-Canadian Clinical Criteria group contains some people who don't have ME/CFS/SEID at all and instead might have a primary mood disorder (or at least have a group that are a bit different).

Here are the overall entry criteria again:

To be included we required only three months of unexplained chronic/relapsing fatigue of new onset, causing disability to a degree that prevented normal school attendance. We did not require that patients met any other accompanying symptom criteria. The referring units were required to confirm that the patient did not have any medical or psychiatric disorder that might explain the fatigue, and that they had not experienced any concurrent demanding life event.

 

[SOC]

I'm a bit puzzled. Do I understand correctly that they claimed to confirm no psychiatric disorder that might explain fatigue as a requirement for entry, but then found symptoms suggesting a mood disorder in nearly half of the non-CCC group?

 

[Dolphin]

I'm a bit puzzled. Do I understand correctly that they claimed to confirm no psychiatric disorder that might explain fatigue as a requirement for entry, but then found symptoms suggesting a mood disorder in nearly half of the non-CCC group?

They found symptoms suggesting mood disorder in 48% of CCC patients and 65% of non-CCC patients. However, one can have a mood disorder secondary to being ill.

 

[SOC]

They found symptoms suggesting mood disorder in 48% of CCC patients and 65% of non-CCC patients. However, one can have a mood disorder secondary to being ill.

Ah, of course! Still, I'm not sure how they distinguished whether the patients had a primary psychiatric condition presenting as fatigue or a secondary mood disorder as the result of chronic illness. It would take some very careful evaluation, I think, to be reasonably confident of that differential diagnosis. I doubt they did that careful evaluation and consequently let in quite a few primary psych patients.

It's interesting, if the mood disorders are thought to be secondary to chronic illness, that more non-CCC patients showed symptoms of mood disorder than CCC patients. I wonder how that could be interpreted. No doubt there are multiple plausible explanations.

 

[Simon]

I'd say the 21 is a typo and they meant 20.

Might have been the way I wrote it up
[QUOTE"I said they"]This study had some limitations. The questionnaire items used to subgroup CFS patients have not been formally validated for this purpose. However, they correspond closely to the ME/CFS clinical diagnostic worksheet (21 - Sharpe et al).[/QUOTE]I meant ref 21 was the Sharpe questionnaire they used to diagnose accoriding to Carrduthers/Canadian (20). And my point was that an unvalidated 1991 questionnaire is not fit for diagnosing according to 2003 CCC (which is very much a tool for clinicians). This is a study making judgements about the validity of diffferent case definitions: if you don't make a decent fist of the case definition implementation, it becomes a pointless exercise. Certainly it's not suitable methodology to back up this claim that they went on to make:

Recently, Fink and coauthors provided evidence that bodily distress disorders, which in their study included CFS, might be regard as one entity instead of separate ones (29). The results of the present study seem to support this view.

As you say, they might be right about no cytokine difference, but I don't think this study provides the evidence to back up this claim.

 

[Dolphin]

Might have been the way I wrote it up
I meant ref 21 was the Sharpe questionnaire they used to diagnose accoriding to Carrduthers/Canadian (20). And my point was that an unvalidated 1991 questionnaire is not fit for diagnosing according to 2003 CCC (which is very much a tool for clinicians). This is a study making judgements about the validity of diffferent case definitions: if you don't make a decent fist of the case definition implementation, it becomes a pointless exercise.

My point is that I don't think they used something comparable to a 1991 questionnaire (Sharpe et al., 1991). The 1991 paper doesn't have a big long set of questions comparable to the Canadian criteria - if they had used the 1991 paper as a basis a large percentage of the Canadian criteria symptoms wouldn't have been asked about. (One can see the 1991 paper here: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1293107/)

I think they used their own questionnaire which is asks a lot of questions and they think can be used to try to access who would satisfy the 2003 Canadian criteria

i.e. this sentence should have said reference 20 not 21.

The questionnaire items used to subgroup CFS patients have not been formally validated for this purpose. However, they correspond closely to the ME/CFS clinical diagnostic worksheet (21).

 

[Simon]

My point is that I don't think they used something comparable to a 1991 questionnaire (Sharpe et al., 1991). The 1991 paper doesn't have a big long set of questions comparable to the Canadian criteria - if they had used the 1991 paper as a basis a large percentage of the Canadian criteria symptoms wouldn't have been asked about. (One can see the 1991 paper here: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1293107/)

I think they used their own questionnaire which is asks a lot about a lot of questions and they think can be used to try to access who would satisfy the 2003 Canadian criteria

i.e. this sentence should have said reference 20 not 21.

The questionnaire items used to subgroup CFS patients have not been formally validated for this purpose. However, they correspond closely to the ME/CFS clinical diagnostic worksheet (21).

Thanks, see what you mean about their own questionnaire, not Sharpe's:

In accordance with a CFS symptom inventory for adults (23), we previously developed a CFS
symptom inventory for adolescents, assessing the frequency of 24 common symptoms during the
preceding month

However, this still looks like an inadequate way to assess CCC ME/CFS - which goes to the heart of what the paper is doing:

1. They used 24 common symptoms (we don't know which ones), compared to over 50 listed in the ME/CFS diagnostic worksheet
2. That worksheet is appendix 5 of the CCC paper - appendix 3 is another worksheet, this time focusiing on symptom severity, which CCC stresses is important in making diagnoses: this current paper does not assess severity
3. Their questionnaire has not been validated as they acknowledge.
4. It's all very different from Lenny Jason's work with this 100+ item DePaul Symptom questionnaire, which places a lot of emphasis on symptom severity, and has been psychometrically validated (well, as much as any questionnaire has been)

So the situation is better than I'd orginally posted (will update) but falls a long way short of diagnosis that is good enough to make the claim that all forms of CFS are the same (and part of bodily distress syndrome).

 

[waiting]

One difference between the groups that isn't mentioned in the abstract is step count

Steps per day - number, mean (SD)

Canadian Clinical Criteria group 3958 (2142)

Non-Canadian Clinical Criteria group 5144 (2482)

p value=0.006

@Dolphin, does this mean that the 1st number is the average actual steps/day -- over how long of a period? And so what does the bracketed number mean? I can't recall how to interpret standard deviation....
Thanks!

 

[Dolphin]

Steps per day - number, mean (SD)

Canadian Clinical Criteria group 3958 (2142)

Non-Canadian Clinical Criteria group 5144 (2482)

p value=0.006

@Dolphin, does this mean that the 1st number is the average actual steps/day -- over how long of a period? And so what does the bracketed number mean? I can't recall how to interpret standard deviation....
Thanks!

Yes, average steps per day.
Details:

daily physical activity was monitored during seven consecutive days using the activPAL accelerometer device (PAL Technologies Ltd, Scotland),

Standard deviations give an idea of the variety of scores, whether they are all clustered close to the mean or spread out wide on both sides of a mean.

If data is normally distributed, 68% of scores will be between "mean - 1 standard deviation" and "mean + 1 standard deviation".

 

[waiting]

Yes, average steps per day.
Details:


Standard deviations give an idea of the variety of scores, whether they are all clustered close to the mean or spread out wide on both sides of a mean.

If data is normally distributed, 68% of scores will be between "mean - 1 standard deviation" and "mean + 1 standard deviation".

Thank you for explaining that, and excuse me for being dense -- but does that mean the CCC group walked a average of 3958 steps/day? Then what does the 2142 represent for the CCC group?
Thanks again.

 

[Dolphin]

Yes, average steps per day.
Details:


Standard deviations give an idea of the variety of scores, whether they are all clustered close to the mean or spread out wide on both sides of a mean.

If data is normally distributed, 68% of scores will be between "mean - 1 standard deviation" and "mean + 1 standard deviation".

Thank you for explaining that, and excuse me for being dense -- but does that mean the CCC group walked a average of 3958 steps/day? Then what does the 2142 represent for the CCC group?
Thanks again.

Yes, the CCC group walked an average of 3958 steps/day.
If the distribution is not normally distributed, it is not easy to interpret what the standard deviation means exactly. It just gives an idea of how close or otherwise the scores were on average to the mean i.e. 3958 steps.

 

[waiting]

3958 steps per day seems very high for a group of ME patients.