Jason's Latest Paper on the Reeves / Empiric Criteria: Use of SF-36 Subsets

Anika

Senior Member
Messages
148
Likes
28
Location
U.S.
Leonard Jason and his team keep plugging away at the problems with the CDC Reeves / empirical definition and the cohorts it produces, a la Wichita and Georgia cohorts used in the CDC's XMRV paper.

Some of the major criticisms related to the Reeves definition concern its choice of subscales of the SF-36, and cutoff scores on those subscales, to determine whether a person has the substantial reduction in activity to be characterized as CFS. It uses a score relating to emotions (versus physical or social functioning, for example) as one of the relevant subscales, and uses arbitrary cutoffs, among other things.

In the abstract cited below, Jason says the Role Emotional subscale had the least sensitivity and specificity of all the subscales. Other findings also sound interesting.

Does anyone have access to the whole paper? I'd be interested in how else their findings differ from what the CDC is doing.

The problem is, the CDC should have done something like this before it published any results using the Reeves criteria.

http://www.ncbi.nlm.nih.gov/pubmed/20617920
Disabil Rehabil. 2010 Jul 9. [Epub ahead of print]

Measuring substantial reductions in functioning in patients with chronic fatigue syndrome.
Jason L, Brown M, Evans M, Anderson V, Lerch A, Brown A, Hunnell J, Porter N.

Center for Community Research, DePaul University, Chicago, IL 60614, USA.

Abstract
Purpose. All the major current case definitions for chronic fatigue syndrome (CFS) specify substantial reductions in previous levels of occupational, educational, social, or personal activities to meet criteria. Difficulties have been encountered in operationalizing 'substantial reductions.' For example, the Medical Outcomes Study Short Form-36 Health Survey (SF-36) has been used to determine whether individuals met the CFS disability criterion. However, previous methods of using the SF-36 have been prone to including people without substantial reductions in key areas of physical functioning when diagnosing CFS. This study sought to empirically identify the most appropriate SF-36 subscales for measuring substantial reductions in patients with CFS. Method. The SF-36 was administered to two samples of patients with CFS: one recruited from tertiary care and the other a community-based sample; as well as a non-fatigued control group. Receiver operating characteristics were used to determine the optimal cutoff scores for identifying patients with CFS. Results. The SF-36 Role-Emotional subscale had the worst sensitivity and specificity, whereas the Vitality, Role-Physical, and Social Functioning subscales had the best sensitivity and specificity. Conclusion. Based on the evidence from this study, the potential criteria for defining substantial reductions in functioning and diagnosing CFS is provided.

PMID: 20617920 [PubMed - as supplied by publisher]
 
Messages
1,383
Likes
362
Location
UK
Does anyone have access to the whole paper? ]
Thanks for posting this, the whole idea of using the Role Emotional subscale is a bit daft, since it measures not doing things because you don't feel up to it emotionally, rather than because you are tired.

You could try requesting a copy from Leonard Jason: I asked him for a paper recently and he was good enough to send me a copy.
 

Anika

Senior Member
Messages
148
Likes
28
Location
U.S.
Leonard Jason's flurry of papers

That seems to be different paper, about implementing the Canadian criteria, rather than refuting the use of SF-36 RE scale for implementing the Reeves empirical criteria. Maybe a more interesting paper though!
Thanks Oceanblue and Angela for your suggestions and the reference to the other paper and thread (which I'm also following).

Jason has been VERY busy - in the Revised Canadian Definition paper that Angela refers to, he cites 5 other papers (including the SF-36 subscales paper) of his from 2010 alone- 2 papers look to be still in press.

From their titles, it looks like Jason is trying to further de-legitimize the Reeves / empiric criteria, and to operationalize the Canadian criteria to make them more meaningful for research purposes. Way to go, Lenny!
 
Messages
1,575
Likes
171
Thanks for this Anika,

yay Lenny indeed.

Cohort definition is so critical right now. And experts listen to other experts, not patients, especially on things like definitions. Dr Jason is dealing with one of THE critical issues right now. 5 papers this year - fantastic.
 
Messages
1,383
Likes
362
Location
UK
Yes, Case Definition is probably the most important (and definitely the most tedious) aspect of ME/CFS research. Thank God people like Len Jason are prepared to work at this particular coalface - looks like he's producing some great finds.
 

Anika

Senior Member
Messages
148
Likes
28
Location
U.S.
Jason & group have 15 papers this year

Yes, Case Definition is probably the most important (and definitely the most tedious) aspect of ME/CFS research. Thank God people like Len Jason are prepared to work at this particular coalface - looks like he's producing some great finds.

Manipulation of the Case Definition has been probably the greatest harm done by the CDC (with help from some overseas miscreants). Without someone like Leonard Jason and his group, where would we be to challenge it?

I was looking for other Jason papers and came across this link to articles by Jason and his group at DePaul University:
http://condor.depaul.edu/~ljason/cfs/publications.html
Most of them don't seem to be available online, at least for now.

I think I counted 15 publications so far this year (including those "in press" - I guess not yet formally published, but awaiting publication). Mostly, though not all, research studies.

LJ always comes through. He's one of my heroes in CFS research. Most researchers are the lab kind - which we also need - but without the toil in the this area, all research languishes for lack of comparability.
 
Messages
1,383
Likes
362
Location
UK
Case Definition has become one of the main battlefields of the clash of Biospychosocial vs Biomedical views of CFS. Those who see ME/CFS as simply one end of a spectrum of fatguing illnesses with a psychological basis like their CFS definition drawn broadly. It seems the CDC is now firmly in this camp.

In a recent Journal of Disability Policy Studies paper responding to Jason's criticisms of the CDC empirical case definition, Bill Reeves wrote (well, the whole piece was more of a printed tantrum than a normal article):
"The reality of CFS research is that many medical and psychiatric illnesses (currently termed exclusionary conditions) have symptoms indistinguishable from CFS".

This is presumably why the CDC chose to include the SF36 RE (Emotional Role) sub-scale in it's criteria. Jason pointed out the case of someone with Major Depressive Disorder who scored the maximum score of 100 on the SF36 PHYSICAL function scale (ie no physical problems at all) yet because of a low Emotional Role score was diagnosed with CFS by the CDC criteria.

Leonard Jason is definitely one of those who believes that ME/CFS can - and should - be distinguished from many other disorders. Interestingly, the latest CDC community survey (the best way to measure prevalence) using its empirical criteria gives a CFS rate of 2.5% while Jason's Chicago community survey, using the Fukuda criteria, came up with a rate of 0.4%, a rate over 5 times lower. Likewise in the UK, Simon Wessely estimated CFS at 2.5% - which works out at an implausibly high average of 45 cases per GP (Community Phsyician).
 

Dolphin

Senior Member
Messages
17,555
Likes
28,235
Table 1 lists the 17 studies that used the SF-36 in CFS studies with at least two comparison groups. Of these, the nine studies used to calculate difference scores are noted in table 1. Differences between the CFS and non-ill control groups were: 70.1 for Role-Physical; 48.7 for Vitality; 46.0 for Social Functioning; 41.5 for Physical Functioning; 38.5 for Bodily Pain; 38.4 for General Health; 30.9 for Role-Emotional; and 20.9 for Mental Health. These findings suggest that Role-Physical is clearly the best discriminator between CFS and controls, with the next groups being Vitality, Social Functioning, Physical Functioning, Bodily Pain, and General Health; and dimensions that showed the least discrimination between the CFS and control groups were Role-Emotional and Mental Health.
Three of the 9 are actually empiric criteria studies (Boneva et al, 2007; Reeves et al, 2005 and Nater et al, 2006) but only one is marked as such. They are all from the one cohort. They really should only have used one. I might bring that up with Lenny. It shows that people are not very good at spotting empiric criteria studies (Lenny probably got the postgrads to compile them).
 

Dolphin

Senior Member
Messages
17,555
Likes
28,235
Results

Literature review

Table 1 lists the 17 studies that used the SF-36 in CFS studies with at least two comparison groups. Of these, the nine studies used to calculate difference scores are noted in table 1. Differences between the CFS and non-ill control groups were: 70.1 for Role-Physical; 48.7 for Vitality; 46.0 for Social Functioning; 41.5 for Physical Functioning; 38.5 for Bodily Pain; 38.4 for General Health; 30.9 for Role-Emotional; and 20.9 for Mental Health. These findings suggest that Role-Physical is clearly the best discriminator between CFS and controls, with the next groups being Vitality, Social Functioning, Physical Functioning, Bodily Pain, and General Health; and dimensions that showed the least discrimination between the CFS and control groups were Role-Emotional and Mental Health.
SF-36 Part of empiric criteria (Reeves et al., 2005) http://www.biomedcentral.com/1741-7015/3/19

We defined substantial reduction in occupational, educational, social, or recreational activities as scores lower than the 25th percentile of published US population [11] on the physical function (≤ 70), or role physical (≤ 50), or social function (≤ 75), or role emotional (≤ 66.7) subscales of the SF-36.
 

Dolphin

Senior Member
Messages
17,555
Likes
28,235

Dolphin

Senior Member
Messages
17,555
Likes
28,235
I'm not sure how many people understand ROC. Basically, you want to score as close to 1 as possible. 0.5 would be the same as a random cut off (i.e. useless).

Table 2. AUC Values, Standard Errors (SE), and Confidence Intervals (CI) for Community-Based and Tertiary Care Samples

Column 1: SF-36 Subscale
1st three numbers: Community-Based Sample
2nd three numbers: Tertiary Care Sample
AUC SE 95% CI† AUC SE 95% CI†
Vitality .88 .04 .81 -.96 .91 .03 .85 -.97
Social Functioning .87 .04 .79 -.95 .87 .04 .79 -.94
Role-Physical .86 .04 .77 -.94 .91 .03 .84 -.97
Bodily Pain .85 .04 .77 -.94 .86 .04 .79 -.93
Physical Functioning .84 .05 .75 -.93 .87 .04 .80 -.94
General Health .80 .05 .70 -.90 .91 .03 .86 -.97
Mental Health .75 .06 .63 -.86 .71 .05 .61 -.80
Role-Emotional .67 .07 .54 -.80 .63 .05 .53 -.73

†Confidence intervals not including .5 are significant at the p < .05 level
Only Vitality, Social Functioning, and Role-Physical emerged as within the top 4 for each sample.
 

Dolphin

Senior Member
Messages
17,555
Likes
28,235
This gives you the thresholds they suggest

This gives you the thresholds they suggest:

We next focused on optimal cutoff points for these two subscales. For Vitality, a score of 35 or less in the community-based sample had 90% sensitivity and 81% specificity; whereas in the tertiary care sample, a score of 35 or less had 89% sensitivity and 81% specificity. For Social Functioning, a score of 62.5 or less in the community-based sample had 87% sensitivity and 72% specificity; whereas in the tertiary care sample, a score of 62.5 or less had 88% sensitivity and 72% specificity. For the Role-Physical subscale, a score of 50 or less had 97% sensitivity and 70% specificity for the community-based sample; whereas for the tertiary care sample, a score of 25 or less had 96% sensitivity and 79% specificity (there are only 4 items on this scale, so a person can only get a score of 0, 25, 50 or 100). Based on these findings, we used a cut-off score of ≤ 35 or less for Vitality, ≤ 62.5 for Social Functioning, and ≤ 50 for Role-Physical.

As it is unclear if the optimal diagnosis for CFS would require a person to meet the cutoff for one or more subscales (i.e., ≤ 35 on Vitality, ≤ 62.5 on Social Functioning, and/or ≤ 50 on Role-Physical), we conducted ROCs examining the total number of scales met by participants (range 0 to 3). For the community-based sample, the ROC resulted in an AUC of 0.89 (SE = 0.04, 95% CI 0.82 to .96, p < 0.001). If the person met the cutoff for one or more subscales, the sensitivity was 100% and the specificity was 64%. If the person met the cutoff for two or more subscales, the sensitivity was 93% and the specificity was 75%. If the person met the cutoff for three subscales, the sensitivity was 79% and the specificity was 85%.

For the tertiary care sample, the ROC resulted in an AUC of 0.90 (SE = 0.03, 95% CI 0.83 to .96, p < 0.001). If the person met the cutoff for one or more subscales, the sensitivity was 100% and the specificity was 64%. If the person met the cutoff for two or more subscales, the sensitivity was 96% and the specificity was 75%. If the person met the cutoff for three subscales, the sensitivity was 81% and the specificity was 85%.

If investigators are attempting to identify as many as possible cases of CFS from either community-based or tertiary care samples, then higher sensitivity is better. Even though specificity might be lower, when a medical and psychiatric examination is conducted, those individuals who have other fatigue-causing medical or psychiatric conditions would be excluded from the CFS samples. Therefore, we would recommend that meeting two or more of the following three subscales: Vitality, Social Functioning, or Role-Physical, with our designated cutoff scores, should be used to designate substantial reductions.
 
Messages
1,383
Likes
362
Location
UK
Reeves's rationale for using SF-36 Role emotional scale to diagnose CFS

Lenny Jason has provided good evidence for not using sf36 RE to select for CFS. But I was amzaed when I came across this quote in the 2007 Reeves paper on prevalence of CFS in Georgia which found a rate of 2.5% using the empirical critieria - it's from the Discussion section, explaining why they used the Role Emotional subscare:

In particular, we included the role emotional subscale to capture the relation between functional emotional impairment and reduced social and personal activities.
WTF does that have to do with CFS? They say they are implementinng the International Study Group recommendations, themselves a clarification of Fukuda, and both of these talk about substantial activity reduction as a result of fatigue, not emotional impairment.

I'd always assumed they come up with a better argument than this for using RE. Surely even the CDC themselves don't buy this.
 

Dolphin

Senior Member
Messages
17,555
Likes
28,235
In particular, we included the role emotional subscale to capture the relation between functional emotional impairment and reduced social and personal activities.
WTF does that have to do with CFS? They say they are implementinng the International Study Group recommendations, themselves a clarification of Fukuda, and both of these talk about substantial activity reduction as a result of fatigue, not emotional impairment.

I'd always assumed they come up with a better argument than this for using RE. Surely even the CDC themselves don't buy this.
Well spotted. Might be worth putting as a comment - comments don't have to be anything fancy.

I've now a link in my signature to a petition complaining about these criteria.
 

Sunshine

Senior Member
Messages
208
Likes
5
Location
UK
Do we know of any other Psychologist who is on our side? Is LJ actually the only psychologist in the United States (if not the world) who produces research supporting us?

If so, that's a chilling testimony to the profession.