Actometers accurately measuring activity levels/energy expenditure

[Simon]

How do you know if CFS patients have been able to increase their activity levels as the result of therapy? Put more bluntly, how do you know if a therapy really works?

The usual way, used in the PACE trial, is asking patients about their activity or capabilities using a structured questionnaire. However, this leads to concerns over self-report bias: are patients subconsciously over-reporting their activity to please therapists, for instance?

There are other reasons to doubt the accuracy of questionaires. First, different types of self-report (daily diary, different activity questionnaires) don't correlate very strongly with each other, when they really should stronglycorrelate. Secondly, and crucially, there is no good evidence that any questionnaire can accurately measure the relatively modest gains found in most clinical trials. No one ever does the experiment to find out.

The alternative is to use some kind of motion sensing device, called an actometer, that straps to the body and measures acceleration in different directions. The device 'counts' the amount of movement and the higher the count, the more active the wearer has been. These devices are impressively accurate when measuring subjects on a treadmill in a lab, compared with gold-standard measures of exertion eg amount of oxygen consumed.The manufcturers of these devices trumpet these results and researchers tend to blandly cite the manufacturers' assurances. And actometers clearly can't suffer from self-report bias.

However, no one mentions that if the lab treadmill has a gradient (as in running up hill, or downhill) the actometers can't tell the difference.Or that real life doesn't involve a lot of treadmill running. Where the accuracy of actometers has been measured in 'free-living' humans, the studies have usually been small and the results generally unimpressive. The Dutch/Belgian CFS studies used a home-grown actomter without even the usual lab-based validation.

One study, though, does show impressive accuracy for an actometer measuring activity levels (or more strictly, energy expenditure) using the gold-standard methog of Double-labelled water (read the paper if you are interested). There was a robust sample of 42 subjects measured over 10 days. The correlation between actometer and energy-expenditure was 0.81 (zero is no correlation, 1.0 is total correlation). By comparison, most activity questionnaires correlate with each other by around 0.3 to 0.4, sometimes less. Given this, without any good evidence that activity (or disability)questionnairesdo reliably and accurately what patients really do, it would surely make sense to have improvement in actomter-measured activity as a primary outcome in CFS treatment trials.

Here's that study:

Evaluation of a portable device to measure daily energy expenditure in free-living adults. St-onge 2006 (free full text)

ABSTRACT​

Background:Increasing daily energy expenditure (EE) plays an important role in the prevention or treatment of several lifestyle-related diseases; however, its measurement remains problematic.

Objective:The objective was to evaluate a portable armband device for measuring daily and physical activity EE compared with doubly labeled water (DLW) in free-living individuals.

Design:Daily EE and physical activity EE were measured in 45 subjects over a 10-d period simultaneously with 2 techniques: a portable armband and DLW. Resting metabolic rate was measured by indirect calorimetry, and the thermic effect of a meal was estimated (10% of daily EE). Physical activity EE was obtained by subtracting the values for resting metabolic rate and thermic effect of a meal measured with DLW from those measured with the armband. Body composition was measured with dual-energy X-ray absorptiometry. Concordance between measures was evaluated by intraclass correlation, SEE, regression analysis, and Bland-Altman plots.

Results:Mean estimated daily EE measured with the armband was 117 kcal/d lower (2375 ± 366 kcal/d) than that measured with DLW (2492 ± 444 kcal/d; P < 0.01). Despite this group difference, individual comparisons between the armband and DLW were close, as evidenced by an intraclass correlation of 0.81 (P < 0.01).

Conclusions:The portable armband shows reasonable concordance with DLW for measuring daily EE in free-living adults. The armband may therefore be useful to estimate daily EE.

Declaration of interest: I've spent quite a bit of time casting doubt on the accuracy of actometers (as well as questionnaires); not sure why I didn't pay more attention to this study since I did mention it briefly on the PACE thread last year! Though as I mention, plenty of other actometer studies in free-living adults are rather less impressive.

 

[Valentijn]

There are also a series of papers from the Nijmegen CBT group that show pretty well that questionnaires are useless. They got great results on questionnaires, and no improvement on actometers.

Naturally they tried to spin this to suggest that it is feelings that are most relevant in CFS, not actually activity levels.

 

[Sam Carter]

How do you know if CFS patients have been able to increase their activity levels as the result of therapy? Put more bluntly, how do you know if a therapy really works?

The usual way, used in the PACE trial, is asking patients about their activity or capabilities using a structured questionnaire. However, this leads to concerns over self-report bias: are patients subconsciously over-reporting their activity to please therapists, for instance?

There are other reasons to doubt the accuracy of questionaires. First, different types of self-report (daily diary, different activity questionnaires) don't correlate very strongly with each other, when they really should stronglycorrelate. Secondly, and crucially, there is no good evidence that any questionnaire can accurately measure the relatively modest gains found in most clinical trials. No one ever does the experiment to find out.

The alternative is to use some kind of motion sensing device, called an actometer, that straps to the body and measures acceleration in different directions. The device 'counts' the amount of movement and the higher the count, the more active the wearer has been. These devices are impressively accurate when measuring subjects on a treadmill in a lab, compared with gold-standard measures of exertion eg amount of oxygen consumed.The manufcturers of these devices trumpet these results and researchers tend to blandly cite the manufacturers' assurances. And actometers clearly can't suffer from self-report bias.

However, no one mentions that if the lab treadmill has a gradient (as in running up hill, or downhill) the actometers can't tell the difference.Or that real life doesn't involve a lot of treadmill running. Where the accuracy of actometers has been measured in 'free-living' humans, the studies have usually been small and the results generally unimpressive. The Dutch/Belgian CFS studies used a home-grown actomter without even the usual lab-based validation.

One study, though, does show impressive accuracy for an actometer measuring activity levels (or more strictly, energy expenditure) using the gold-standard methog of Double-labelled water (read the paper if you are interested). There was a robust sample of 42 subjects measured over 10 days. The correlation between actometer and energy-expenditure was 0.81 (zero is no correlation, 1.0 is total correlation). By comparison, most activity questionnaires correlate with each other by around 0.3 to 0.4, sometimes less. Given this, without any good evidence that activity (or disability)questionnairesdo reliably and accurately what patients really do, it would surely make sense to have improvement in actomter-measured activity as a primary outcome in CFS treatment trials.

Here's that study:

Evaluation of a portable device to measure daily energy expenditure in free-living adults. St-onge 2006 (free full text)

ABSTRACT​

Background:Increasing daily energy expenditure (EE) plays an important role in the prevention or treatment of several lifestyle-related diseases; however, its measurement remains problematic.

Objective:The objective was to evaluate a portable armband device for measuring daily and physical activity EE compared with doubly labeled water (DLW) in free-living individuals.

Design:Daily EE and physical activity EE were measured in 45 subjects over a 10-d period simultaneously with 2 techniques: a portable armband and DLW. Resting metabolic rate was measured by indirect calorimetry, and the thermic effect of a meal was estimated (10% of daily EE). Physical activity EE was obtained by subtracting the values for resting metabolic rate and thermic effect of a meal measured with DLW from those measured with the armband. Body composition was measured with dual-energy X-ray absorptiometry. Concordance between measures was evaluated by intraclass correlation, SEE, regression analysis, and Bland-Altman plots.

Results:Mean estimated daily EE measured with the armband was 117 kcal/d lower (2375 ± 366 kcal/d) than that measured with DLW (2492 ± 444 kcal/d; P < 0.01). Despite this group difference, individual comparisons between the armband and DLW were close, as evidenced by an intraclass correlation of 0.81 (P < 0.01).

Conclusions:The portable armband shows reasonable concordance with DLW for measuring daily EE in free-living adults. The armband may therefore be useful to estimate daily EE.

Declaration of interest: I've spent quite a bit of time casting doubt on the accuracy of actometers (as well as questionnaires); not sure why I didn't pay more attention to this study since I did mention it briefly on the PACE thread last year! Though as I mention, plenty of other actometer studies in free-living adults are rather less impressive.

I haven't thought this through properly so take the following with a pinch of salt -- but is it not possible to build an objective picture of someone's physical capacity by measuring, for example:
- hours worked (if any)
- benefits claimed
- steps taken
- geographical position relative to home with a GPS enabled device (sick people are less mobile)
- hours spent in bed (which a FitBit can record)

Each of these measures on their own would be accurate 'relative to themselves' in the sense that they would be able to reliably, and objectively, indicate a trend towards improvement or a trend towards relapse.

Taken together they beat the snot out of an SF36 or CFQ score.

 

[Esther12]

Thanks for this Simon. Another interesting piece you've dug up.

I haven't thought this through properly so take the following with a pinch of salt -- but is it not possible to build an objective picture of someone's physical capacity by measuring, for example:
- hours worked (if any)
- benefits claimed
- steps taken
- geographical position relative to home with a GPS enabled device (sick people are less mobile)
- hours spent in bed (which a FitBit can record)

Each of these measures on their own would be accurate 'relative to themselves' in the sense that they would be able to reliably, and objectively, indicate a trend towards improvement or a trend towards relapse.

Taken together they beat the snot out of an SF36 or CFQ score.

I tend to agree with this.

If the therapist knows what is being measured, then I would expect that they could manipulated a patient in to improving any one outcome measure without needing to genuinely improve their health at all. With a range of outcome measures which should all improve if there were genuine improvements in health, it becomes much harder to create an 'effective' treatment simply by substitution and altering patient preferences. I think that, within the mix, there should be questionnaires, but they should not be relied upon in the way that they are for a lot of CFS work (unless it's genuinely double blind).

 

[Firestormm]

Cheers Simon. I need to read through what you've written carefully - and I liked the look of that study. Interesting.

Here's one: how do you judge whether a nurse is providing effective care? Or a GP? or a Neurologist? Is the use of e.g. structured questionnaires (with their assumed inherent bias) a better means of assessing healthcare than simply asking a patient what they think or how they feel before and after?

It's a funny thing, because we are now being given the opportunity to see the relative performance of NHS surgeons: http://www.bbc.co.uk/news/health-20766880 and I'd been contemplating, whether or not it would be 'a good idea' to allow visitors to a website to 'rate' their GP.

To what extent these simple measures are useful or indeed effective when it comes to improving healthcare I really do not know. In a scientific experiment you need to try and measure responses and that's not easy with therapy especially when therapy involves e.g. emotional responses or behavioural ones; but how a patient feels before and after is I think important.

Like other measures - objective measures (which also have restrictions of course in terms for example of how the data is processed and presented or indeed collected) - I suppose you have to do the best you think you can. Choose the most practical and accepted means of obtaining measurable results.

I have problems with studies that go back and re-arrange the means of reporting or the way in which collected data is presented. But apparently that's the norm - based largely on practical issues (such as cost in having to do the whole thing again) but it does lessen the impact of the study in my own mind.

Any study can be critiqued of course because the means of collecting and assessing the effectiveness of the therapy is not as good as it could have been. That's what will drive the next study to do better - assuming improvements can be found. And maybe this 'armband' to which you refer is indeed something that would provide a better means of doing so. It does not however cater to the 'care' aspect of therapy. That personal intervention, understanding, empathy, hand-holding, listening etc. and that's important too.

Good post my man. I shall return and read more thoroughly. NICE job

 

[user9876]

The seem to have used a 2 axis accelarometer rather than a 3 axis one which I believe is more normal in things like game controllers. I wonder if the corrolation would increase if they improved their accelorometer.

There is an issue in that standing in some positions for example leaning over uses muscle strength but does not involve moving. So you would expect some inaccuracy.

Modern accelorometers (or maybe the next generation comming soon) will also pick up heartbeats so I wonder if more accuracy could be gained by monitoring heart rates along with other physical activity in a more conventional sense.

 

[Simon]

I haven't thought this through properly so take the following with a pinch of salt -- but is it not possible to build an objective picture of someone's physical capacity by measuring, for example:
- hours worked (if any)
- benefits claimed
- steps taken
- geographical position relative to home with a GPS enabled device (sick people are less mobile)
- hours spent in bed (which a FitBit can record)

Each of these measures on their own would be accurate 'relative to themselves' in the sense that they would be able to reliably, and objectively, indicate a trend towards improvement or a trend towards relapse.

Taken together they beat the snot out of an SF36 or CFQ score.

Multiple measures would be good and I like the idea of hours in bed too.

If the therapist knows what is being measured, then I would expect that they could manipulated a patient in to improving any one outcome measure without needing to genuinely improve their health at all. With a range of outcome measures which should all improve if there were genuine improvements in health, it becomes much harder to create an 'effective' treatment simply by substitution and altering patient preferences. I think that, within the mix, there should be questionnaires, but they should not be relied upon in the way that they are for a lot of CFS work (unless it's genuinely double blind).

Normally trials focus on one or two measures as primary outcomes, to stop them cherry picking the outcomes that improved from a whole range of possible outcomes. But I haven't come across the idea of improvements being required across a whole range of outcomes. The closest to this is PACE's protocol recovery definition which was something like low fatigue, high function, no symptoms and patients rating themselves as recovred. Unfortunately we;ve never seen the data for 'recovery' defined like this, and probably never will.

Modern accelorometers (or maybe the next generation comming soon) will also pick up heartbeats so I wonder if more accuracy could be gained by monitoring heart rates along with other physical activity in a more conventional sense.

Very interesting idea.

 

[Hope123]

Dolphin had referred me to this paper several months ago although I did not get around to reading it yet. But

1) No, it is considered suscipious or somewhat scientifically unethical to change your outcomes or your standards for your outcomes in medical studies unless you have good reason.

2) It is very common for multiple outcomes to be measured in studies although some are designated primary and others secondary.

3) There is no perfect measure for activity. For example, actometers do not measure mental activity and the brain is one of the most energy-draining organs in the body.

4) The US Food and Drug Administration, which approves medication in the US, is concerned about the types of outcome measures used in CFS and convening a meeting next year to look at this issue. Including but also beyond actometers.

 

[Sean]

As somebody who has pushed hard for actometers to be used as primary outcome measures, my view is that they are very necessary, but not sufficient on their own.

There needs to be a range of measures, and correlations between measures. If, for example, we are getting improved SF36 scores, without comparable improvement in more objective measures, such as actometers and employment levels, then there is a real problem.