Jason - Sensitivity and Specificity of the CDC Empirical CFS Case Definition

[Jerry S]

I don't think this has been posted yet. The full paper can be found at:

http://www.scirp.org/journal/psych/

This is the abstract:

Journal: Psychology, 2010, 1: 9-16, doi:10.4236/psych.2010.11002 Published Online April 2010
Authors:Leonard A. Jason, Meredyth Evans, Abigail Brown, Molly Brown,
Nicole Porter, Jessica Hunnell, Valerie Anderson, Athena Lerch

Affiliation: DePaul University, Chicago, American. Email: <Ljason@depaul.edu>

Received February 10th, 2010; revised March 7th, 2010; accepted March 8th, 2010.

ABSTRACT

In an effort to bring more standardization to the chronic fatigue syndrome (CFS) Fukuda et al. case definition [1], the Centers for Disease Control and Prevention (CDC) has developed an empirical case definition [2] that specifies criteria and instruments to diagnose CFS.

The present study investigated the sensitivity and specificity of this CFS empirical case definition with diagnosed individuals with CFS from a community based study that were compared to non-CFS cases. All participants completed questionnaires measuring disability (Medical Outcome Survey Short-Form-36) [3], fatigue (the Multidimensional Fatigue Inventory) [4], and symptoms (CDC Symptom Inventory) [5].

Findings of the present study indicated sensitivity and specificity problems with the CDC empirical CFS case definition.

The paper concludes:

"The sensitivity and specificity outcomes for the Reeves et al. criteria suggest that these recommended scales and cutoff points would not be considered a good diagnostic tool for selecting CFS cases from the general population."

 

[Jerry S]

Some comments on the discussion section of the study (my bolds):

4. Discussion

The present study investigated the sensitivity and specificity of the empirical CFS case definition [2] with diagnosed individuals with CFS from a community based study that were compared with non-CFS cases. Findings of the present study indicated sensitivity and specificity problems with the CDC empirical CFS case definition. When comparing the overall Reeves et al. criteria, only about 65% of true CFS cases were identified. In other words, these criteria are not able to identify an acceptable high percentage of individuals who have this illness.

JS Note: The Reeves 2007 study [29] found a prevalence rate of 2540 per 100,000. Part of the explanation for this stunningly high rate was greater sensitivity. "The 6- to 10-fold greater prevalence estimates also reflect application of more sensitive and specific measures of the CFS diagnostic parameters specified by the 1994 case definition." Yet this study by Jason finds the empirical case definition lacks sensitivity - 35% of the "true" CFS cases were missed. The only other explanation, if the claim Georgia might be a CFS hotspot is dismissed, is the empirical definition is grossly non-specific.

If samples of CFS are not identified with sensitivity and specificity, it will be difficult to compare samples from different studies, and the search for biological markers will be compromised. Using the Reeves et al. criteria [2], the estimated rates of CFS have increased to 2.54% [29], rates that are about ten times higher than prior CDC estimates [30] [235 per 100,000 - JS] and prevalence estimates of other investigators [31] [ 420 per 100,000 - JS]. It is at least possible that the increases in the United States are due to a broadening of the case definition and possible inclusion of cases with primary psychiatric conditions.

JS Note: This is extremely likely.

Chronic fatigue occurs in about 4-5% of the population [32]. If about 5% of the population has 6 or more months of fatigue, and about half of this is due to clear medical or psychiatric reasons [31], then the critical question is how many of the remaining 2.5% have CFS. The empirical CFS case definition estimates that 2.54% do have this illness, so that research group would suggest that almost all of the remaining 2.5% would fall within the CFS category. However, Jason et al. [7] believe that within this 2.54% are mood disorders, which are one of the most prevalent psychiatric disorders (one-month prevalence rate of major depressive episode is 2.2%) [33].

As an example, one mood disorder is MDD [major depressive disorder - JS], which can be confused with CFS, as it has some overlapping symptoms with CFS. It is possible that some patients with MDD also have chronic fatigue and four CFS Fukuda et al. [1] symptoms that can occur with depression (e.g., unrefreshing sleep, joint pain, muscle pain, impairment in concentration). Fatigue and these four minor symptoms are also defining criteria for CFS, so it is possible that some patients with a primary affective disorder could be misdiagnosed as having CFS. [38 % of patients with only major depressive disorder [7] were misclassified as having CFS - JS]

Yet, these are distinct illnesses, as several CFS symptoms are not commonly found in depression, including prolonged fatigue after physical exertion, night sweats, sore throat, and swollen lymph nodes. Illness onset with CFS often occurs over a few hours or days, whereas primary depression generally shows a more gradual onset. Biological findings also differentiate the two conditions [34]. Including the latter type of patients in the current CFS case definition could confound the interpretation of epidemiologic and treatment studies, and complicate efforts to identify biological markers for this illness.

JS Note: Bottom line - After 25 years, the CDC still hasn't come up with an adequate case definition of CFS.

Partial References:

[2] W. C. Reeves, D. Wagner, R. Nisenbaum, J. F. Jones, B. Gurbaxani, L. Solomon, et al., Chronic Fatigue Syndrome A Clinically Empirical Approach to Its Definition and Study, BMC Medicine, Vol. 3, 2005.

[7] L. A. Jason, N. Najar, N. Porter and C. Reh, Evaluating the Centers for Disease Controls Empirical Chronic Fatigue Syndrome Case Definition, Journal of Disability Policy Studies, Vol. 20, 2009, pp. 93-100.

[29] W. C. Reeves, J. J. Jones, E. Maloney, C. Heim, D. C. Hoaglin, R. Boneva, et al., New Study on the Prevalence of CFS in Metro, Urban and Rural Georgia Populations, Population Health Metrics, Vol. 5, 2007.

[30] M. Reyes, R. Nisenbaum, D. C. Hoaglin, E. R. Unger, C. Emmons, B. Randall, et al., Prevalence and Incidence of Chronic Fatigue Syndrome in Wichita, Kansas, Archives
of Internal Medicine, Vol. 163, 2003, pp. 1530-1536.

[31] L. A. Jason, J. A. Richman, A. W. Rademaker, K. M. Jordan, A. V. Plioplys, R. Taylor, et al., A Community-Based Study of Chronic Fatigue Syndrome, Archives of Internal Medicine, Vol. 159, 1999, pp. 2129- 2137.

[32] L. A. Jason, M. Evans, M. Brown and N. Porter, What is
Fatigue? Pathological and Non-Pathological Fatigue, Manuscript Submitted for Publication, 2010.

[33] D. A. Regier, J. H. Boyd and J. D. Burke Jr., One Month Prevalence of Mental Disorders in the United States: Based on Five Epidemiological Catchment Rea Sites, Archives of General Psychiatry, Vol. 45, 1988, pp. 977- 986.

34] L. A. Jason, K. Corradi, S. Torres-Harding, R. R. Taylor and C. King, Chronic Fatigue
Syndrome: The Need For Subtypes, Neuropsychology Review, Vol. 15, 2005, pp. 29-58.

 

[Dolphin]

I don't think this has been posted yet. The full paper can be found at:

http://www.scirp.org/journal/psych/

Thanks for posting the link for the full paper, Jerry. Haven't seen it posted it elsewhere. Will link to it from the petition.

This paper really vindicates everyone who had doubts about the definition.

 

[mezombie]

Bumping...for others to read. Jason's article is great and Jerry's comments are astute.

 

[Dolphin]

In my opinion, the main finding of this study can be summarised as follows from Table 1:

----------------
For the 3 parts of the CDC's empiric criteria for CFS:

MFI (fatigue criteria)
Sensitivity: 0.95 (i.e. 95% of the true CFS cases will satisfy the criteria)
Specificity: 0.27 (i.e. 73% of the non-CFS cases will satisfy the criteria)


SI (core symptoms criteria)
Sensitivity: 0.59 (i.e. 59% of the true CFS cases will satisfy the criteria)
Specificity: 0.73 (i.e. 27% of the non-CFS cases will satisfy the criteria)


SF-36 (substantial reductions criteria)
Sensitivity: 0.96 (i.e. 96% of the true CFS cases will satisfy the criteria)
Specificity: 0.17 (i.e. 83% of the non-CFS cases will satisfy the criteria)


Meet Criteria (overall)
Sensitivity: 0.65 (i.e. 65% of the true CFS cases will satisfy the criteria for CFS, 35% will not)
Specificity: 0.76 (i.e. 24% of the non-CFS cases will satisfy the criteria for CFS)

[Aside: I checked with the authors and the following is going to be added to the data: "Some of the participants did not complete all three questionnaires, and were thus excluded from the overall sensitivity and specificity figures."]

-----------------

These are not good figures for a set of criteria. They are so "broad" that 24% of the non-CFS cases will satisfy the criteria for CFS, yet even with these "broad" criteria, 35% of the true CFS cases will be missed.
Each of the three parts of the criteria are very poor ("bad") in one way i.e. either the specificity or sensitivity is very poor.

 

[Mithriel]

Thanks for that analysis Tom, I have great trouble getting my head round these papers.

Mithriel

 

[Dolphin]

SF-36 questions

Thanks Mithriel.

As I mentioned on another thread, many people would benefit from filling in a SF-36 questionnaire to know what is being talked about.

This can be done at:
http://www.sf-36.org/demos/SF-36.html
or
http://www.qualitymetric.com/demos/sf-36.aspx

If one goes to (near end of) https://www.fstrf.org/apps/cfmx/apps/actg/html/QOLForms/manualql601-2799.pdf one can see how they are scored. Except for the pain scale (and the composite scores which Lenny Jason may not talk about), once you know the questions, it is just a question of expanding them to fit 100 (in other words, getting a percentage). So the SF-36 physical functioning subscale is 10-30 (which is the same as 0-20) - so one just multiplies each score by five (if you are doing the 10-30 scale, subtract by 10 first). Anyway with the sample tests, they do the scoring for you.

 

[Jerry S]

Thanks, Tom, for breaking down the numbers and and providing links to the SF-36 questionnaire. Thanks, mezombie, for the kind comment.

Another quote from the Jason paper:

It is important for screening tests to have high sensitivity and specificity, particularly for disorders with low prevalence rates such as CFS (about 4.2 in a thousand) [31].

As an example, in a city of 1,000,000, with a true CFS rate of 4.2 per thousand, there would be 4,200 CFS cases. According to Bayes’ theorem [35] if a diagnostic test had a 95% rate of sensitivity, the screening test would correctly identify 3,990 of these cases.

However, if the test had 95% specificity, there would be 50,000 individuals who did not have CFS but were identified as having it using the test. Clearly, being able to identify true negatives with precision is of high importance in the diagnostic process.

[35] E. T. T. Jaynes, “Probability Theory: The Logic of Science,” G. L. Bretthorst Ed.,Cambridge University Press, 2003.

I don't know what numbers you'd get using 65% sensitivity and 76% specificity, but Jason's example gives one an idea of how far off the mark the empirical definition is.

 

[Dolphin]

I don't know what numbers you'd get using 65% sensitivity and 76% specificity, but Jason's example gives one an idea of how far off the mark the empirical definition is.

I pointed out to Lenny they made a slight error with Bayes' theorem and a correction will be placed with the article.
If 4,200 people have CFS, they shouldn't be included in figures to do with the rest.
So one calculates the other figures from 1,000,000-4,200=995,800.
So there would be 49,790 cases who did not have CFS but were identified as having CFS in the scenario he gave.

65% sensitivity and 76% specificity would give:
Number satisfying the criteria from the CFS cases: 1,000,000*0.0042=4200*.65=2730
Number satisfying the criteria from the non-CFS cases: (1,000,000-4200)=995,800*.24=238992
So if it was left to the questionnaires, the number of proper cases would only be: 2730/(2730+238992)=0.0112939 of the number they say have CFS (1.1%)!

Of course, this was done in two stages so many of the healthies were excluded by the phone call stage.

 

[Jerry S]

Amazing!! Thanks, Tom!

 

[rebecca1995]

...I don't know what numbers you'd get using 65% sensitivity and 76% specificity, but Jason's example gives one an idea of how far off the mark the empirical definition is.

Exactly. The numbers from the Jason paper, along with Tom's analysis, scare the daylights out of me.

It's so frustrating that much of the CAA's literature, like the Patient Brochure found here, endorses the empirical definition/Reeves Criteria.

The brochure says, "More than 4 million people in the United States have chronic fatigue syndrome (CFS)..." A number that high can only be obtained using the empirical definition.

Why would the nation's largest ME charity (as fred would say :Retro wink support a definition that's wildly and demonstrably inaccurate?

 

[Dolphin]

Amazing!! Thanks, Tom!

Thanks Jerry.

Regarding the CAA and the empiric criteria, they did come out against it last year.
They were in a sort of "no man's land" before that, neither really supporting it, nor criticising it.
They were in a difficult situation as (i) they were tied to the CDC for the awareness etc contract (ii) Suzanne Vernon's name is on the criteria.

However, with a bit of a nudge, they appear to be more against it now although they may still feel 4 million is a useful figure for lobbying (I don't like that figure myself and as you say, rebecca1995, it comes from the empiric criteria).

 

[Jerry S]

Exactly. The numbers from the Jason paper, along with Tom's analysis, scare the daylights out of me.

It's so frustrating that much of the CAA's literature, like the Patient Brochure found here, endorses the empirical definition/Reeves Criteria.

The brochure says, "More than 4 million people in the United States have chronic fatigue syndrome (CFS)..." A number that high can only be obtained using the empirical definition.

Why would the nation's largest ME charity (as fred would say :Retro wink support a definition that's wildly and demonstrably inaccurate?

Yes, Rebecca. Based on an estimated US population of 310 million the various prevalence rates for CFS give:

Jason, 1999. A community-based study in Chicago - 422 per 100,000

1.3 million

Reyes, 2003. A CDC study in Wichita, KA - 235 per 100,000

730,000

Reeves, 2007. A study using the empirical definition - 2540 per 100,000

7.9 million

I don't know where the 4 million figure comes from, but it certainly isn't based on any study using Fukuda. The empirical definition is the only one which gives figures that high.

ETA: The CAA has come out against the empirical definition, as Tom noted, yet they seem reluctant to abandon the high CFS numbers the empirical definition generated.

 

[Dolphin]

Yes, Rebecca. Based on an estimated US population of 310 million the various prevalence rates for CFS give:

Jason, 1999. A community-based study in Chicago - 422 per 100,000

1.3 million

Reyes, 2003. A CDC study in Wichita, KA - 235 per 100,000

730,000

Reeves, 2007. A study using the empirical definition - 2540 per 100,000

7.9 million

I don't know where the 4 million figure comes from, but it certainly isn't based on any study using Fukuda. The empirical definition is the only one which gives figures that high.

(Not very exciting)

http://www.pophealthmetrics.com/content/5/1/6/comments

Caution required when extrapolating prevalence rates to the full population
Tom Kindlon (11 June 2007) Irish ME/CFS Support Group

This editorial [1] says, with regard to the CDC study[2]: "The CDC has now repeated and extended the Wichita study in Georgia, and found a prevalence of between six and ten times greater, with 2.5% of the population suffering from CFS. If this prevalence was both accurate and representative of the USA as a whole, this would suggest that some 7.5 million Americans were sufferers, compared to the previous estimates of 0.7 to 1.2 million."

Before the 7.5 million figure is quoted, it might be useful to point out that the figure makes a number of assumptions, including that the prevalence rate for those under 18 and over 60 would be similar. However previous studies have suggested this is unlikely to be the case, with prevalence rates for young children in particular being much lower.

The round figure of 7.5 million would be equivalent to a population of 295,275,591.

Using this data the population estimate for 2005 was 296,410,404 (i.e. a similar figure).

Using the same data: The population under 18 years was 73,469,580, the population over 60 was 49,791,976 and population aged 18-60 was 173,148,444.

For a population of those aged over 18 and under 60 of this size, a back of the envelope calculation for CFS prevalence using the prevalence rate of 2.64%[2] would give: (173,148,444*0.0264)= 4,397,971.

Tom Kindlon

[1] How common is chronic fatigue syndrome; how long is a piece of string? Peter D White
Population Health Metrics 2007, 5:6 doi:10.1186/1478-7954-5-6

[2] Prevalence of chronic fatigue syndrome in metropolitan, urban, and rural Georgia
William C Reeves , James F Jones , Elizabeth Maloney , Christine Heim , David C Hoaglin , Roumiana S Boneva , Marjorie Morrissey and Rebecca Devlin

Population Health Metrics 2007, 5:5 doi:10.1186/1478-7954-5-5

Competing interests
No competing interests

 

[Mithriel]

So it might be your fault Tom :Retro smile:

With all these false positives, any intervention could be shown to "significantly" help CFS without a single person with ME/CFS getting any better. Papers using the Oxford definition come to mind.

I am trying to work out the maths, but I keep getting muddled, very annoying!

Mithriel

 

[Dolphin]

So it might be your fault Tom :Retro smile:

No, the CDC had used the 4 million figure before that (by accident in an ad before the study came out). I just thought it was easier to show how they might have arrived at the figure.

With all these false positives, any intervention could be shown to "significantly" help CFS without a single person with ME/CFS getting any better. Papers using the Oxford definition come to mind.

Yes, exactly.

 

[Bellyhill]

hi
This is Gagan.

I am from NJ i just an account on this forum