• Welcome to Phoenix Rising!

    Created in 2008, Phoenix Rising is the largest and oldest forum dedicated to furthering the understanding of and finding treatments for complex chronic illnesses such as chronic fatigue syndrome (ME/CFS), fibromyalgia (FM), long COVID, postural orthostatic tachycardia syndrome (POTS), mast cell activation syndrome (MCAS), and allied diseases.

    To become a member, simply click the Register button at the top right.

Jason: Data mining: comparing the empiric CFS to the Canadian ME/CFS case definition.

CBS

Senior Member
Messages
1,522
Another nice addition to Jason's body of work.He first showed that the "Empiric Definition" (Reeves, 2005) had led to a 10 fold increase in the diagnosis of CFS (creating a much more heterogeneous patient group that was originally diagnosed using the Holmes criteria). However, the question remained, was the Reeves criteria capturing more ME/CFS patients (in addition to more depressed patients)? Did the "Empiric Definition" have any redeeming qualities. This study answers that question with a resounding NO.

Even thought the "Empiric Definition" was based upon the Fukuda criteria, it fails to properly identify as many ME/CFS patients (79%) as the Canadian Case Definition (87%). The "Empiric Definition" is neither the most specific nor the most sensitive diagnostic criteria for ME/CFS.


Data mining: comparing the empiric CFS to the Canadian ME/CFS case definition.

[FONT=&quot]Jason LA, Skendrovic B, Furst J, Brown A, Weng A, Bronikowski C.[/FONT]
[FONT=&quot]
[/FONT]

[FONT=&quot]J Clin Psychol.[/FONT] 2011 Aug 5. doi: 10.1002/jclp.20827. [Epub ahead of print]

Abstract

This article contrasts two case definitions for myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS). We compared the empiric CFS case definition (Reeves et al., 2005) and the Canadian ME/CFS clinical case definition (Carruthers et al., 2003) with a sample of individuals with CFS versus those without. Data mining with decision trees was used to identify the best items to identify patients with CFS. Data mining is a statistical technique that was used to help determine which of the survey questions were most effective for accurately classifying cases. The empiric criteria identified about 79% of patients with CFS and the Canadian criteria identified 87% of patients. Items identified by the Canadian criteria had more construct validity. The implications of these findings are discussed.
 

CBS

Senior Member
Messages
1,522
More from the article:

Results
For the empiric CFS criteria, the risk statistic was .21, indicating that 79% of the cases were identified correctly. The cross validation statistic was .32, which is the average risk of the 10 trees created during the 10-fold cross validation. This indicates the generalizability of the classification: One can expect that 68% of the new cases will be identified correctly. The six items that loaded (were the most significant for the classification) were sore throat, lymph node pain, MFI reduced activity, SF-36 Social Functioning, Joint Pain, and SF-36 Role-Physical.

For the Canadian criteria, the risk statistic was .13, indicating that 87% of the cases were identified correctly, and the cross validation statistic was .27, indicating that 73% of new cases would be identified correctly. The six items that loaded were sore throat, lymph node pain, inability to concentrate, presence of multiple chemical sensitivities, postexertional malaise, and unfreshing sleep.

Discussion

The studys overall findings were that the Reeves et al. (2005) criteria were not as capable of discriminating cases from noncases as the Canadian criteria (Carruthers et al., 2003).

In addition, when examining the items selected in both analyses, it is apparent that the Canadian criteria appear to select cardinal and central features of the illness. CFS is often thought to include postexertional malaise and neurocognitive dysfunction, and both did emerge as predictive factors in the Canadian criteria, but not within the Reeves et al. (2005) empiric case criteria. In addition, sleeping disorders and pain symptoms, other key symptoms of CFS, did emerge from the Canadian criteria, but not within the Reeves et al. criteria. Within the other category of the Canadian criteria, two symptomssore throat and multiple chemical sensitivitiesemerged in the immune areas, and this provides evidence that the Canadian criteria identify more key symptoms of this illness. In contrast, the empiric criteria tended to identify more general areas, including less activity, social-functioning and role-functioning problems, and some pain issues. However, critical symptoms such as postexertional malaise, neurocognitive symptoms, and sleep disorders were not identified as discriminating symptoms.
 

Dolphin

Senior Member
Messages
17,567
Petition against the "empiric" criteria

Thanks CBS. You've summarised it well. This is a fairly short paper with most of the text just explaining the patients, etc.

Even thought the "Empiric Definition" was based upon the Fukuda criteria, it fails to properly identify as many ME/CFS patients (79%) as the Canadian Case Definition (87%). The "Empiric Definition" is neither the most specific nor the most sensitive diagnostic criteria for ME/CFS.
Yes, here's what they say on this:
In addition, the Fukuda et al. (1994) criteria were used to select cases, and this might have biased the study toward having the empiric case definition do better, as it is based upon the Fukuda et al. criteria. However, this was not the case with the Canadian criteria, which were better able to classify cases and noncases.
The figures for each should be read in this context.

There's a petition against the "empiric" criteria at:
http://www.ipetitions.com/petition/empirical_defn_and_cfs_research/index.html which anyone can sign (one can also "like" it by clicking the button on that page).

There is various information on other studies on the site esp. in the blog: http://www.ipetitions.com/petition/empirical_defn_and_cfs_research/blog
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
Thanks for posting more of the study than I could get access to.

I will read more thoroughly of what is there when I am less 'foggy' but one question if I may?

Reeves identified 79% and Canadian 87% of an already establish cohort of patients. Established by Fukuda (CDC) upon which Reeves is based. Right, got that.

Erhm... what about the 21% and 13% respectively, of patients that neither criteria identified? I mean I know this is not looking at actual patients or anything, and it could just be me at this point in my fogginess, but neither method is very accurate is it? I mean it is suggesting (still) that even the same criteria re-applied cannot produce the same results.

I don't see that it can even be said that Canadian reveals 'purer' subjects either. Not based on this study. Can it?

Anyway - as I hypothesised before when posting these results on another thread - this would appear to be what Jason has been working on then. It remains to see if the 'International' ME version will produce an even 'purer' and less subjective set of results I guess...
 

Dolphin

Senior Member
Messages
17,567
Thanks for posting more of the study than I could get access to.

I will read more thoroughly of what is there when I am less 'foggy' but one question if I may?

Reeves identified 79% and Canadian 87% of an already establish cohort of patients. Established by Fukuda (CDC) upon which Reeves is based. Right, got that.

Erhm... what about the 21% and 13% respectively, of patients that neither criteria identified? I mean I know this is not looking at actual patients or anything, and it could just be me at this point in my fogginess, but neither method is very accurate is it? I mean it is suggesting (still) that even the same criteria re-applied cannot produce the same results.

I don't see that it can even be said that Canadian reveals 'purer' subjects either. Not based on this study. Can it?

Anyway - as I hypothesised before when posting these results on another thread - this would appear to be what Jason has been working on then. It remains to see if the 'International' ME version will produce an even 'purer' and less subjective set of results I guess...
I don't really understand the mathematics/similar involved but some observations:

- The Fukuda criteria in this study is taken as the gold standard as that was how the patients were diagnosed in the study - however, that might not be best gold standard.

- When one normally thinks of such issues, one thinks of sensitivity and specificity, however this may involve a different concept:
Statistical Analyses
We used decision trees to help distinguish among individuals with CFS and other conditions (ICF, Exclusions, and Controls) based upon their responses to survey questions. For the empiric CFS case definition (Reeves et al., 2005), there were altogether 14 different items or subscales from the SI symptoms, SF-36 subscales, and MFI questions that were used as features for the decision tree classification, compared with altogether 43 questions that were used as features for the decision tree for the analysis to tap the ME/CFS Clinical Canadian criteria. These items have recently been described in an article by Jason, Evans et al. (2010). SPSS Answer Tree software was used to build our decision tree models. To build the models, we used a Classification and Regression Tree (CART) algorithm with a 10-fold cross validation. The value of the model was measured with risk estimates (risk statistic and cross validation), which give an estimate of how many unknown cases will be misclassified, allowing this technique the ability to be generalized to new data.

- The point about the sort of patients a definition includes/construct validity is based on the important nodes, as far as I can see. The important nodes for the diagnosis were more ME-CFS like for the Canadian criteria than for the empiric criteria.
 

oceanblue

Guest
Messages
1,383
Location
UK
I'm a bit nonplussed by this particular paper.

It compares the ability of Canadian and Empiric criteria to diagnose patients with a confirmed Fukuda CFS diagnosis; it finds that the Canadian criteria diagnose more patients than the Empiric critria, and concludes that the Canadian crtieria are more effective. I'm surprised by the finding.

First, the Empric criteria are essentially Fukuda with lax criteria, so I would have expected the Empiric criteria to diagnose 100% of the Fukuda patients, as opposed to the 79% found in this study. Possibly this is because Fukuda requires 4/8 symptoms, while Empiric also requires a minimum score based on frequency and severity of symptoms.

Second, as the Canadian criteria are usually seen as more specific, I would have expected them to diagnose fewer Fukuda patients, while this study found 89% of Fukuda patients met the Canadian Critria. This contrasts with the recent large Nacul study (n=270, vs n=24 for this one) which found only 53% of Fukuda patients met the Canadian criteria.

What would have been intersting would be to see how the Canadian and Empiric criteria compared when dealing with the 80 fatigued patients in this study who had had a Fukuda CFS diagnoses ruled out. I would have expected the Canadian criteria to be very good at excluding these non-CFS patients and the Empiric criteria to be poor at excluding them. However, this paper gives no data on these non-CFS patients.

I did email Lenny Jason, who says he will be presenting data on this at a later date, and is actively collecting data on a larger group of patients. It looks like what we have here is work in progress, and the final data will hopefully be more revealing.

This looks important, (as Dolphin & CBS highlighted):
when examining the items selected in both analyses, it is apparent that the Canadian criteria appear to select cardinal and central features of the illness.

...and this provides some construct validity to these criteria
This feels like good evidence - and the most interesting part of the paper - but I'm not clear if this is a recognised way of validating case definitions.
 

Dolphin

Senior Member
Messages
17,567
First, the Empric criteria are essentially Fukuda with lax criteria, so I would have expected the Empiric criteria to diagnose 100% of the Fukuda patients, as opposed to the 79% found in this study. Possibly this is because Fukuda requires 4/8 symptoms, while Empiric also requires a minimum score based on frequency and severity of symptoms.

Second, as the Canadian criteria are usually seen as more specific, I would have expected them to diagnose fewer Fukuda patients, while this study found 89% of Fukuda patients met the Canadian Critria. This contrasts with the recent large Nacul study (n=270, vs n=24 for this one) which found only 53% of Fukuda patients met the Canadian criteria.

What would have been intersting would be to see how the Canadian and Empiric criteria compared when dealing with the 80 fatigued patients in this study who had had a Fukuda CFS diagnoses ruled out. I would have expected the Canadian criteria to be very good at excluding these non-CFS patients and the Empiric criteria to be poor at excluding them. However, this paper gives no data on these non-CFS patients.
Firstly, small error, I think the figure you have in mind is 87%.

I think you may not be comparing like with like (with regard to the Nacul study). I think (not 100%) that the percentage may relate to the sum of the cases and non-cases of CFS - put them through the algorithm and see what percentage of the total are classified "correctly" as CFS cases or non-CFS cases as they case may be.
Not 100% sure of this.

It is novel to me and perhaps it does not have too much validity except as a pointer. It may be a bit like using a different statistically method of analysing data.
 

oceanblue

Guest
Messages
1,383
Location
UK
Firstly, small error, I think the figure you have in mind is 87%.

I think you may not be comparing like with like (with regard to the Nacul study). I think (not 100%) that the percentage may relate to the sum of the cases and non-cases of CFS - put them through the algorithm and see what percentage of the total are classified "correctly" as CFS cases or non-CFS cases as they case may be.
Not 100% sure of this.
I asked Lenny about this and it seems the figure of 87% refers to 87% of CFS cases correctly identified and says nothing about diagnostic accuracy for non-cases.
 

Snow Leopard

Hibernating
Messages
5,902
Location
South Australia
First, the Empric criteria are essentially Fukuda with lax criteria, so I would have expected the Empiric criteria to diagnose 100% of the Fukuda patients, as opposed to the 79% found in this study. Possibly this is because Fukuda requires 4/8 symptoms, while Empiric also requires a minimum score based on frequency and severity of symptoms.

Maybe this shows that the Empiric criteria is not merely a lax version of the Fukuda criteria and that is why it lacks sensitivity as well as specificity?
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
I asked Lenny about this and it seems the figure of 87% refers to 87% of CFS cases correctly identified and says nothing about diagnostic accuracy for non-cases.

Thanks Ocean. And presumably the same applies to the other percentile quoted.

I must read the whole paper. Will move it back up the old list of priorities :)
 

oceanblue

Guest
Messages
1,383
Location
UK
Maybe this shows that the Empiric criteria is not merely a lax version of the Fukuda criteria and that is why it lacks sensitivity as well as specificity?
Yes, that's possible. It's worth noting that the Empiric criteria are based on the 2003 Ambiguities paper (basically updated Fukuda), the same definition used to diagnose CFS in this study. The empiric criteria uses odd ways of measuring fatigue and activity which Lenny Jason has elegantly shown leads to poor specificity. Maybe the Empricic's method of requiring a minimum symptom severity/frequency score leads to poor sensitivity too - I'd be interested in hearing more about this possibility. Or maybe the Empiric's fatigue/activity measures lead normal CFS patients being excluded but I don't really understand how that would happen, given how lax they are. It is all a bit puzzling.
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
Hadn't it previously been suggested (by Jason I think) that the CCC effectively reduces the number of patients diagnosed with CFS by the CDC criteria by a third?

I have this figure in my head for some reason and can't for the life of me remember the actual context now...
 

Dolphin

Senior Member
Messages
17,567
I think people are thinking of it as just applying the full definitions in a different format but using the exact same questions that would lead to the exact same percentages of sensitivity and specificity. However, I presume what was done was not that. That may explain the differences.
 

Firestormm

Senior Member
Messages
5,055
Location
Cornwall England
Interesting perhaps to compare Jason's methods and results with this study in England that concluded:

'The estimated minimum prevalence rate of ME/CFS was 0.2% for cases meeting any of the study case definitions, 0.19% for the CDC-1994 definition, 0.11% for the Canadian definition and 0.03% for the ECD. The overall estimated minimal yearly incidence was 0.015%. The highest rates were found in London and the lowest in East Yorkshire. All cases conforming to the Canadian criteria also met the CDC-1994 criteria but presented higher prevalence and severity of symptoms.'

http://www.ncbi.nlm.nih.gov/pubmed/21794183 July 28 2011