Can tracker readings predict daily health for PWME?

[TracyD]

For me, the answer is no.

Here's a blog post about my many, failed efforts to create a statistical model to predict my daily health using tracker data.

And here's the gist:
How well can the tracker data that’s available to me upon waking foretell my health status, as a person with ME/CFS (PWME), for the rest of the day? Over the past few years, I have applied a plethora of machine learning techniques to answer this question. The goal was to create an app to automatically predict PWME’s daily health each morning.

The result is disappointing: the best method will accurately predict whether my daily health will be ‘bad’ or ‘not bad’ only 57 percent of the time. Flipping a coin would result in 50 percent accuracy, so this is an improvement of only 7 percent.

I still believe that resting heart rate (RHR) and heart rate variability (HRV) generally change along with my health status, so what went wrong? My observation is that many other factors also influence these measurements, masking the relationship that I’m trying to model. The good-ish news is that a person likely can do a better job of taking these contingencies into account, making this a case when human learning probably outperforms machine learning.

 

[hapl808]

Over the past few years, I have applied a plethora of machine learning techniques to answer this question.

What machine learning techniques did you utilize? I've considered trying to do this, but I think one of the challenges is the time window changes, but I'm not sure how. In other words, sometimes my health on Wednesday is determined by my activity on Monday, but sometimes it's determined by my activity on Tuesday, etc. Possibly some sort of attention mechanism could achieve that, rather than just LSTM or RNN or whatever.

Also for me, there is no bad or not bad since my health is always bad. So I'd probably break it down to 'no crash' vs 'mild crash' vs 'bad crash'. That might be manageable.

Still, the biggest problem for me is how to define the parameters. I've noticed that phone calls are a huge predictor for me - but a 30 minute phone call at 10pm is far, far worse than a 30 minute phone call at 2pm. Very quickly, the complexity of collecting data skyrockets - although vitals like HR make a much easier proxy.

 

[Wishful]

I think there are way too many factors affecting ME symptoms to make worthwhile predictions. Doing exactly the same things on two different days may have wildly different results. You might be able to compose a list of activities or foods or other factors that are very likely to make your symptoms worse, but there will be factors you aren't even aware of to mess up predictions.

Spores in the air, chemicals that you react to at concentrations you don't register, a new microbe acquired from food; the list goes on.

 

[hapl808]

I think there are way too many factors affecting ME symptoms to make worthwhile predictions.

While I generally agree, potentially vitals could be a proxy for some of that. A spore in the air might cause an imperceptible drop in HRV that leads to a decline in function 36 hours later. No human can spot that, but that's exactly where ML algorithms excel. But the architecture chosen and how vitals are tracked, parameters are created, etc - all make a huge difference.

I wouldn't be surprised if you could get somewhat accurate predictions. How useful that would be is another question.

 

[TracyD]

What machine learning techniques did you utilize?

The blog post outlines the ten general approaches I tried, along with variations for each.

 

[hapl808]

The blog post outlines the ten general approaches I tried, along with variations for each.

Thanks - this is great and very cool that you've done that.

I wish I were tracking my response better - I did it for awhile years ago (everything from PEM to GI, etc), but now I only have the Garmin info. Maybe I'll start tracking again. One method I considered was tracking how I felt three times per day - impractical for most people, but since I'm housebound it could be done. I do track every single supplement I take every day, so was mainly doing that hoping to look for patterns if I improved, but haven't really seen improvement.

My total offhand guess for the methods that would work the best is something with a flexible attention mechanism, or at least something with a 'memory' like LSTM. Because of the temporal disconnect, I think that's critical. If my HR (not RHR) spikes badly on Tuesday, there's a good chance I'm going to get reflux on Wednesday or Thursday. A simple neural net won't capture that without well engineered features.

However, how do you track that data? RHR is easy because it's one measurement. For HR, do you count number of minutes above a threshold? Or maximum value? Or average during waking hours or 24 hours?

This is, IMO, the most difficult part. The model architecture and training wouldn't be that hard, but deciding what data to collect and how to do it is endlessly complicated - and a pretty common issue with ML. I think a good ML model could already determine effective treatments for us if they had good data, but people's EMRs and histories are pretty useless for that, sadly.

Anywyas, that's my rambling.

 

[TracyD]

My goal of creating an automated system for all pwme guided my approach. So I limited my predictors to what Garmin makes available for download. They're shown on the correlation matrix in the blog post.

Ideally, a system shouldn't require hundreds or even thousands of observations to train well, because that would require a really long wait. Thus, I think that one of the problems with using a neural network is that they are reputed to require tons of data to train well. Anyway, the time-series modeling, at least with NeuralProphet, takes 'temporal disconnects' into account.