AVIVA Semi-Finals: National ME/FM Action Network is competing for $100,000
The National ME/FM Action Network in Canada is competing for $100,000 for biomedical research of ME and FM in the Aviva Community Fund contest. With thanks to all who helped, they made it through the first round of voting into the Semi-Finals.
Discuss the article on the Forums.

Identifying Defining Aspects of Chronic Fatigue Syndrome via Unsupervised Machine Learning and Featu

Discussion in 'Latest ME/CFS Research' started by Bob, Feb 1, 2014.

  1. Bob

    Bob

    Messages:
    9,947
    Likes:
    17,109
    South of England
  2. Bob

    Bob

    Messages:
    9,947
    Likes:
    17,109
    South of England
    Quite an interesting paper by Lenny Jason and colleagues.
    The study uses data analysis tools in an attempt to gain information to help operationalise the CCC, ICC and Fukuda.
    The results, from page 136 onwards, are interesting.
    I don't understand all the technical details and I found the methodology impenetrable, so some of my analysis might be incorrect.

    Fig. 1, Table III, and Table IV, on pages 136-137, are most interesting.

    The study analyses the diagnostic 'accuracy' (which I think is determined by a combination of 'sensitivity' and 'specificity') of various specific self-report questions/answers (i.e. how accurately each question/answer diagnoses ME/CFS patients.) Then it combines the questions/answers to assess their combined accuracy. The top 15 most diagnostically accurate questions, out of 54 questions that were tested, are shown in Table IV.

    Fig.1 shows that the diagnostic accuracy doesn't change a great deal after the top 3 most accurate questions are used for diagnosis. But the top 11 questions give the most accurate diagnosis, with 90.2% accuracy (with a corresponding sensitivity of 90.1% and specificity of 90.3%.)

    So, in theory, these quite straightforward questions could be used by clinicians to make a diagnosis, with 90% accuracy.

    I'm not sure if these results (in the tables and graphs) relate to a CCC, ICC or Fukuda diagnosis, or a combination. I need to have another look at the paper.
     
    Last edited: Feb 1, 2014
    SOC likes this.
  3. SOC

    SOC Senior Member

    Messages:
    5,917
    Likes:
    8,960
    USA
    I can't see the paper. I just get a black screen. :( But then, I can't stream video, either. My itunes is permanently busted, as well. Looks like my laptop is dying a slow and painful death. :cry:

    Can someone who can see the paper at least post the top 3 questions to satisfy my curiosity?
     
  4. Sparrow

    Sparrow Senior Member

    Messages:
    688
    Likes:
    836
    Canada
    Thank you for passing this on.
     
  5. Sparrow

    Sparrow Senior Member

    Messages:
    688
    Likes:
    836
    Canada

    I believe these are their top 15. upload_2014-2-1_16-44-32.png

    In case it helps clarify for anyone,

    sensitivity is a measure of how successfully patients who actually have ME would be correctly diagnosed as having it based on these criteria

    specificity is a measure of how successfully it avoids misdiagnosing people who do NOT have ME as having it based on these criteria

    (so a criteria that was very sensitive but not very specific would be more likely to come up with the result that everyone on the planet has ME, while one that was not very sensitive but very specific would be more likely to decide that nobody has ME at all. You need one that's high in both so that you catch the people who do have it, but leave out anybody who doesn't)
     
    Last edited: Feb 1, 2014
    aimossy, alex3619, Valentijn and 2 others like this.
  6. SOC

    SOC Senior Member

    Messages:
    5,917
    Likes:
    8,960
    USA
    Thanks, Sparrow. :hug:

    I'm surprised the first 5 or 6 questions alone wouldn't pull in a lot of people with other illnesses. I suppose that's where specificity comes in.

    These questions wouldn't have caught my ME/CFS in the early years when my ME/CFS was relatively mild. (And when I was probably in some degree of denial. :oops: ) I blamed soreness and tiredness on my lifestyle, and didn't mention them to my doc. Everyone's tired and overworked, right?

    What's remarkable about this research, as far as I can tell, is how the right combination of questions gives very good diagnostic accuracy. Sort of like triangulation. ;)
     
    lnester7 and Bob like this.
  7. Bob

    Bob

    Messages:
    9,947
    Likes:
    17,109
    South of England
    It seems that the study was partly testing the DePaul Symptom Questionnaire (DSQ) which, it says, gauges 54 symptoms most commonly associated with CFS. The list of 15 questions, posted in the table above, are taken from the DSQ, I think. So they're the 15 most helpful diagnostic questions from the DSQ.
     
    Last edited: Feb 1, 2014
    Valentijn likes this.
  8. Bob

    Bob

    Messages:
    9,947
    Likes:
    17,109
    South of England
    The top 11 questions (combined) are the most accurate but not significantly different (in terms of accuracy) to the top 5 questions combined.
    The paper concludes that the top 11 questions have better diagnostic accuracy than Fukuda, the CCC, the ICC, and the combined 54 DSQ (DePaul Symptom Questionnaire) questions.
     
    Last edited: Feb 1, 2014
    SOC likes this.
  9. Sparrow

    Sparrow Senior Member

    Messages:
    688
    Likes:
    836
    Canada
    Unsurprisingly, it seems like the symptoms most specific to ME are the ones that tend to show up more strongly once it gets more severe - muscle weakness, sound sensitivity, mental and physical exhaustion after only mild activity, etc.
     
    Valentijn and SOC like this.
  10. Simon

    Simon

    Messages:
    1,814
    Likes:
    6,369
    Monmouth, UK
    Interesting idea,but as with all such studies I wonder how useful it is to have criteria that divide healthy controls from CFS with only 90% accuracy? I mean that's not something that will be useful to many physicians who can probably do better on their own. Now criteria that would separate CFS from other fatiguing illnesses would be a boon.

    Secondly, although I have only skimmed the paper, they dont' seem to have used a training set and a validation set (certainly they don't mention it in the abstract, where it ought to be). If you analyse a single set of data with machine learning the risk is of 'overfitting' the data where chance differences between controls and patients are given undue importance. The standard way to deal with this issue is rrandomly split the original data into a 'training set' used to develop the model, which is then tested for accuracy on the remaining 'validation' set. The validation set accuracy results are almost always worse, and are the only ones that matter. I'm no expert on this but I have done a short online biostatistics course with Johns Hopkins Medical School that covererd machine learning (inc practical work) so do know a bit about this area.

    Like I said, I have only skimmed the paper, and I am in a bit of a grumpy mood today so maybe being unduly harsh.
     
    Last edited: Feb 2, 2014
    aimossy, SOC, WillowJ and 6 others like this.
  11. MeSci

    MeSci ME/CFS since 1995; activity level 6

    Messages:
    5,332
    Likes:
    6,752
    Cornwall, UK
    Whilst my grasp of stats is rudimentary, I think that if you combine symptoms that exclude different other conditions (e.g. for the sake of argument, one excluding anaemia, one excluding coeliac disease and one excluding Lyme) the overall specificity would be higher, would it not?
     
  12. Firestormm

    Firestormm Guest

    Messages:
    5,823
    Likes:
    5,988
    Cornwall England
    Thanks :)

    What's accuracy then? Sorry I haven't read the paper and if @Bob doesn't get it then I definitely wont understand it! I am crap with technical stuff. I fail to understand how criteria based on self-reported symptoms can be deemed accurate of any disease?

    And you say 'patients who actually have ME' which as a statement is also subjective - no? CCC = ME/CFS ICC = ME and Fukuda = CFS and NICE = ME/CFS, if we are being picky: and all of those 'diseases' are subjective interpretations too.

    The criteria are designed to help group common clusters of reported symptoms and not necessarily define a specific disease. Some maintain that CFS is ME and some believe the latter to be a sub-group of the former etc. etc. This paper is trying to establish something in relation to criteria - but it isn't clear to me what that might be - can you explain? Thanks.

    Asking someone if they suffer from fatgiue/extreme tiredness provides the top results - but is only 87.6% accurate of something... It's not terribly good is it really.

    Maybe 100% accuracy will only come when objective measures are developed that are indicative of a proven and accepted biological anomaly for a subset of patients that may or may not eventually be deemed to have something that is not even known as ME (as most recognise it now).

    Or perhaps I am even more confused than I thought. Perfectly possible of course - it is Sunday :)

    Edit:

    Having re-read my post. I just want to be clear I am not directing my critique at you @Sparrow - far from it. Though it perhaps reads like that unintentionally. I am grateful to anyone helping me to better understand such things :)
     
    Last edited: Feb 2, 2014
    aimossy, Esther12 and Simon like this.
  13. Bob

    Bob

    Messages:
    9,947
    Likes:
    17,109
    South of England
    @Firestormm, I think your points are valid, and I don't know the answers to your questions. I don't know how this study decides what is the exact nature of ME/CFS. The DePaul Symptom Questionnaire (DSQ) is Jason's own pet project and maybe some answers can be found if we were to read some of Jason's other research papers relating to the DSQ. I'm not sure if I've yet read anything substantial about the DSQ.
     
    Last edited: Feb 4, 2014
    aimossy, Esther12 and Firestormm like this.
  14. Bob

    Bob

    Messages:
    9,947
    Likes:
    17,109
    South of England
    I'm not sure if primary care clinicians can diagnose CFS with 90% accuracy. My guess is that they're a lot less accurate than that. We often hear of people taking years to get a diagnosis and there is a 40-50% misdiagnosis rate in primary care. (So is that only a 50-60% specificity rate in primary care?) So I think such a tool would be a useful additional tool for primary care clinicians, who would also be free to use their own clinical judgement. Actually I think my local NHS specialist service could also use such a tool as my 'specialist' diagnosing doctor clearly didn't understand the nature of CFS/ME. He seemed to think that I should only suffer from fatigue after exercise. I had to agree with his question ("do you experience fatigue only after exercise?") or it looked like he wouldn't have given me a diagnosis. (I should have made a complaint about him but I was too ill.)
     
    Last edited: Feb 2, 2014
    aimossy, MeSci and Esther12 like this.
  15. Simon

    Simon

    Messages:
    1,814
    Likes:
    6,369
    Monmouth, UK
    Sorry about your bad experience with the doc.Shame they are so common. My best one was early in my illness where my regular GP - who was steadily working her way to referring me on to a specialist was away so had to see a locum. He explained how he'd felt tired too when working hard and reliably told me the answer was to take a shower - so invigorating! I did breifly worry about my personal hygeine but at least in this case the comments made me amused rather than mad as they were so absurd.

    Don't forget, doctors are not diagnosing CFS vs entirely healthy people - they never get suspected as CFS so that's c50% diagnosis vs CFS-like illnesses (which are many). Which is why these studies comparing accuracy on CFS vs healthy strike me as missing the point - and yup, I think just about any doctor could beat 90% accuracy when they were presented with pre-diagnosed CFS patients vs healthy people (the choice in this study). Wheras a machine learning study on symptoms vs other fatiguing illnesses would be interesting.

    @Firestormm
    As you say, with no 'gold standard' for diagnosis, there is a problem in any case in studies of this type.

    Personally, I think such unsupervised machine learning could very useful in searching both for the key features of a ME diagnosis and any subgroups. But the crucial thing would probably be to include biological data as well as symptoms. Potentially the CDC study (as covered by @Bob) could provide such data, even though it has limitied biologics.
     
    aimossy, WillowJ, Valentijn and 3 others like this.
  16. Esther12

    Esther12 Senior Member

    Messages:
    5,712
    Likes:
    7,351
    A bit OT, but I think that the problems around criteria/diagnosis mean it's really necessary for doctors to be honest about the limited value of the diagnosis to patients, and that it is currently little more than a place holder.
     
    Valentijn, Bob, MeSci and 1 other person like this.
  17. Gijs

    Gijs Senior Member

    Messages:
    242
    Likes:
    278
    It is very very sad that after 30 years we still talking and debate the criteria for ME.... so sad... I think the Fukuda is good for the whole CFS group to start. This group must split up in more objective subgroups like CFS with CPET and/or CFS with NK cel dysfunction etc...
     
  18. Christopher

    Christopher Senior Member

    Messages:
    499
    Likes:
    196
    Pennsylvania
    No one has ownership Gijs.
     
  19. Sparrow

    Sparrow Senior Member

    Messages:
    688
    Likes:
    836
    Canada
    To be honest, I didn't read the paper in full detail either yet. Just skimmed it, reading bits and pieces.

    If you can get the right combination of self-reported symptoms, the accuracy rate can be pretty good. And until we have a consistent and easily testable biomarker, that may be the best we can get. Many symptoms are pretty distinctive, and people can tell whether they have them or not. If you believe the patient, then, you can diagnose them based on what they're reporting. It's the same as getting a diagnosis of anxiety based on saying you're super anxious, etc.

    My impression is that the paper is trying to compare the various diagnostic tools for ME (Fukuda, CCC, etc.) and how successful they are at actually distinguishing ME or CFS patients from healthy controls, as well as comparing that success with the method of using their set of questions instead. In this case, it looks like they were only concerned with distinguishing between ME/CFS patients and healthy controls. But a similar process could be done trying to create and include questions that would distinguish ME patients from people with other fatiguing illnesses as well (because there are probably statements people with MS or Lyme disease would make, for example, that we would not, and vice versa).

    When you make the criteria too vague, you misdiagnose too many healthy people or people with other illnesses. When you make them too specific, you risk excluding people who are legitimately ill with ME but don't happen to have that particular symptom. It's not an easy balance.

    These scientists were suggesting using a group of questions rather than a single one, which would help compensate for what you're talking about with the 87,6% accuracy issue. One question on its own is not particularly helpful, but in combination it's less likely to get false positives or false negatives. Accuracy takes into account how well it successfully identifies both people with the illness and people without it.

    That doesn't mean that a single question is necessarily any good, though. The fatigue question, for example, gets a high rating because it's great at not leaving out people who have ME. Almost everyone with ME said they had either fatigue or extreme tiredness. But tons of healthy people said that too, so it's not a good measure by itself.
    But if you combine that with something like feeling physically trained or sick after mild activity, it's likely to weed out the folks who are healthy but just feeling a bit tired.

    Note that I'm not saying these folks have the answer. Just trying to explain what they're saying with the paper, in case it's helpful. I don't think these particular questions are as optimal as they could be, but it is nice to see people trying to improve the process.
     
    aimossy, Bob, Firestormm and 2 others like this.
  20. alex3619

    alex3619 Senior Member

    Messages:
    8,726
    Likes:
    16,321
    Logan, Queensland, Australia
    I haven't read this paper yet, and wont be doing it today. I was doing my PhD on something similar, at least in terms of technology. Overfitting is indeed a huge worry. In any machine learning exercise, construction of the data sets is the PRIMARY issue. If its well constructed, things go smoothly. If not, then things go fubar. Overfitting occurs because the machine learning algorithm can learn noise or irrelevant detail as well as potentially relevant generalizations. @Simon is right that a test set is really required.

    One issue we have here is who are the patients? Do they all have ME? Misdiagnoses will throw off the numbers.

    I look forward to reading this paper.
     
    aimossy and Simon like this.

See more popular forum discussions.

Share This Page