The scale has good face validity, and reasonable discriminant validity.
Although evidence of validity as an estimator of change has been established in an open-treatment
trial further evidence could be obtained by using the scale before and after treatment in a con-trolled trial."
I assume discriminant validity simply means testing ill or not ill.
The big question is whether it is a good estimator of change as they put it. To me that means if we have a perfect fatigue estimator which is linear then the CFQ would need to approximate this. Important properties being:
If patient A at position Xa on the scale improves on our perfect fatigue estimator by a quantity of I to a new value of Ya and patient B at position Xb improves by the same amount I to Ya then For all I, (Ya-Xa)~=(Yb-Xb).
I'm not sure how you would ever prove such a thing. But thing that can be done is to substitute an alternative fatigue scale for the perfect fatigue estimator. Hence you at least test the consistency of the two scales together (where I is scaled). But this involves more than drawing ROC curves and saying they are equally good at dealing with ill|not ill comparisons.
With the CFQ they have two marking schemes so we could look at the consistency across the two different schemes. I think given the two schemes it is clear that they are not consistent. As we even get the case where one patient can improve and get worse - This can be because there are improvements on some questions that count in the Likert scoring but not in the binomial scoring and a worsening of 1 question where the Binomial scoring counts. Comparing patients has a similar issue.
There will be edge effects as well but I think even in the area of interest at the middle of the scale things are dodgy.
One reason for wanting the property I have stated is that if you want to look at the mean differences over a set of patients (P). Basically the mean = sum(i member P, (Yi-Xi))/|P| so if the improvements are not equal due to the starting position then the mean difference does not have a good meaning.
A further complexity is introduced in that the CFQ acknowledges that mental and physical fatigue can change independently. It also has a different number of questions for mental and physical fatigue.