Concurrent validation of activity monitors in patients with rheumatoid arthritis

Abstract

Background

Physical activity is frequently reported in rheumatology but it is difficult to measure objectively outside the gait laboratory. A new generation of activity monitors offers this potential but it has not yet been evaluated in patients with rheumatoid arthritis. This study aimed to evaluate three types of activity monitors in patients with rheumatoid arthritis.

Methods

The Step-N-Tune, Activ4Life Pro V3.8, and the Intelligent Device for Energy Expenditure and Activity activity monitors were tested concurrently in 12 patients with rheumatoid arthritis as well as in a healthy control group of 12 volunteers. Participants walked at a self selected speed for two minutes and were filmed for later review. Temporal and spatial gait parameters were also validated against the GAITRite walkway and the total number of steps recorded by each activity monitor was compared to a gold standard derived from half speed video replays.

Findings

Activity monitor performance varied between devices but all showed poorer performance when used in the group with rheumatoid arthritis. Bland–Altman plots demonstrated wider 95% limits of agreement in the group with rheumatoid arthritis and a systematic decrease in agreement between activity monitors and the gold standard with decreasing functional ability.

Interpretation

Despite some variation between devices, all the activity monitors tested performed reasonably well in healthy young volunteers. All except the Activ4Life showed a marked decrease in performance in patients with rheumatoid arthritis, suggesting Activ4Life could be the most suitable for use in this patient group. The marked between group difference in functional ability, and systematic decrease in device performance with deteriorating gait, indicate that activity monitors require specific validation in target clinical populations.

1. Introduction

Functional ability and physical activity have traditionally been used as outcome measures in studies investigating the impact of interventions on physical function. More recently, advances in the management of rheumatoid arthritis (RA) have placed greater importance on associated co-morbidities and their impact on patients (Avina-Zubieta et al., 2012). In particular, cardiovascular risk is elevated in patients with RA, in part due to physical inactivity and the majority of patients fail to achieve recommended activity levels (Cooney et al., 2011; Elkan et al., 2011; van den Berg et al., 2007). Physical activity is therefore not only an important outcome measure in therapeutic studies, but it can also be an important therapeutic intervention in its own right. Physical activity is difficult to measure in community dwelling populations however, as direct observation is often not feasible (Lee et al., 2009).

Subjective measures of activity, such as self reported questionnaires, are widely used in epidemiological studies of healthy and clinical populations, including those with musculoskeletal disease (Bilek et al., 2005; Lee et al., 2009). Concerns have been expressed however that they underestimate low intensity and unstructured activities (Jacobs et al., 1993; Kriska and Caspersen, 1997). This may be particularly problematic in demographic groups such as older adults, many of whom acquire most of their activity through low-intensity physical activities and activities of daily living (Lee et al., 2009). Furthermore, self reported activity questionnaires are also prone to the same limitations as other subjective outcome measures, including phase shift and response bias (e.g. imprecise recall and social desirability) (Coughlin, 1990; Durante and Ainsworth, 1996). Again, recall bias appears to be highest in low-intensity activities that are habitual (e.g. walking and housework) rather than high intensity activities that are structured (e.g. sports), so this would be a particular concern in a population of patients with RA (Durante and Ainsworth, 1996). It is often recommended therefore, that self reported measures are supplemented by objective measures of physical activity (Lee et al., 2009).

Traditional laboratory based measures of physical function and activity such as gait velocity, cadence and single/double support periods are widely used and well validated in RA (Helliwell et al., 2007; Semple et al., 2007; Turner and Woodburn, 2008). They are of limited use however, in that it is not usually possible to record data outside of the artificial environment of the gait laboratory. Devices such as pedometers and a new generation of accelerometer based activity monitors offer the potential to measure global activity in community dwelling populations. Most validation has been undertaken in only healthy populations however (Lee et al., 2009; Zhang et al., 2003, 2004). Despite this, concerns have already been raised about the ability of bi-axial accelerometers to detect acceleration in the transverse plane, leading to under-reporting of activity in healthy volunteers with adducted or abducted feet (Maffiuletti et al., 2008). To date, the validity of such activity monitors in patients with RA, who often have marked gait abnormalities, has not been assessed.

This study aimed to evaluate the concurrent validity of three types of activity monitors in patients with rheumatoid arthritis.

2. Methods

2.1. Activity monitors

Activity monitors were selected to represent three levels of complexity: first, a simple pedometer with a spring mounted lever arm; second, a waist mounted device consisting of a single tri-axial accelerometer with step counts derived from on board software; third, a device consisting of multiple accelerometers able to measure additional data such as temporal and spatial gait parameters.

2.1.1. Step-N-Tune Pedometer, Kinergy Electronics Co., Ltd (Shenzhen, Hong Kong, China)

The Step-N-Tune is a simple, inexpensive battery operated device worn at the waist. Vertical forces at heel strike cause movement of a spring mounted lever arm which opens and closes an electrical circuit to register a step. The cumulative number of steps is then displayed on an LCD screen on the device.

2.1.2. Activ4Life Pro V3.8, Activ4Life Healthcare Technologies Ltd (Boroughbridge, UK)

The Activ4Life (A4L) device comprises a small activity monitor (38 × 32 × 10 mm) that is attached to the patient's skin and an associated docking station. The activity monitor, consisting of a single tri-axial accelerometer, is worn during the day and then placed in the docking station at night where data are downloaded and the activity monitor is charged. Step counts are derived from actuation of the accelerometer and proprietary onboard algorithms to identify patterns that correspond to steps rather than other forms of activity such as driving. Data are then uploaded to the company's servers using mobile phone technology. Here data are processed and then made available for clinicians to view by logging on to a secure website. Total step counts are provided over a 24 h period and can be subdivided into ‘high intensity’ and ‘low intensity’ activities. Although the manufacturers performed 104 tests on the device during its development, no results are available publically.

2.1.3. Intelligent Device for Energy Expenditure and Activity (IDEEA), MiniSun LLC (Fresno, California, USA)

The IDEEA is an accelerometer based activity monitor consisting of five skin-mounted bi-axial accelerometers and a 32-bit, 25 MHz micro processor unit. Two sensors are mounted to the plantar surface of the participant's feet, two on the anterior thigh, and the remaining sensor attached over the sternum. Each sensor is connected via wires to the small (70 × 44 × 18 mm) processor unit that is attached to the participant's belt. Up to 200 MB can be stored on the device allowing it to collect data for up to 7 days, which is then uploaded to a PC for processing and analysis. The location of sensors allows the device to estimate the orientation of different body segments, thus enabling posture and body movements to be identified as well as calculating energy expenditure. In addition to a simple step count the IDEEA also claims to be able to measure a number of temporal and spatial gait parameters such as gait velocity and double support time.

The IDEEA is the only device used in this study that has previously been validated in the literature. Previous studies have evaluated the IDEEA's concurrent validity, confirming that the device is able to discriminate between 32 common physical activities and the energy expenditure calculations have also been validated using direct and indirect calorimetry (Zhang et al., 2003, 2004). These studies were however conducted on young healthy volunteers (mean age 34 and 36 years) and involved periods of vigorous exercise so it was not clear if these results can be generalised to RA patients performing low intensity activities.

2.2. Participants

Consecutive assenting patients were recruited from rheumatology outpatient clinics. All patients with a consultant diagnosis of RA were invited to participate except those who received intra-articular injections on the day, or had a current foot ulcer. Walking aids were permitted as required.

A group of healthy controls was recruited from staff and students of the hospital and university department. Volunteers were eligible to participate if they could walk freely for two minutes without pain and had no known medical conditions affecting their gait. Eligibility was confirmed through history taking. Ethical approval was obtained from the Leeds West Research Ethics Committee (08/H1307/65) and all participants provided written informed consent. All aspects of the study were conducted in accordance with the Declaration of Helsinki.

2.3. Data collection

Activity monitors were mounted concurrently on the participants according to the manufacturer's instructions as shown in Fig. 1. The five sensors of the IDEEA were held in place using Medipore tape with the processor unit attached to the waistband of the participants' shorts. Loose wires were attached to the skin around the medial malleolus and medial proximal quarter of the tibia and the participants wore socks to prevent the plantar foot sensor catching the floor during gait (see Fig. 1). The Step-N-Tune pedometer was fastened to the waistband on the opposite hip and the participants attached the A4L device using the double sided adhesive pads provided by the manufacturer. Prior to attaching the device, the participants were provided with instructions by the researcher. All devices were reset and care was taken to prevent any unnecessary movement or additional steps once they were attached.

Fig. 1.

Experimental setup of activity monitors. A: IDEEA; B: Step-N-Tune; C: Activ4Life (attached directly to skin).

Following attachment of all devices, the participants sat on an adjustable chair that was adjusted to allow positioning of the hips and knees to 90° for calibration of the IDEEA device.

Temporal and spatial gait parameters recorded by the IDEEA were validated against a GAITRite instrumented walkway which has previously been found to be highly reliable in the test population (Menz et al., 2004; Rome, 2005). The participants rose from the chair after the attachment of the activity monitors and walked along the GAITRite walkway before sitting on a second chair positioned at the end of the walkway. Chairs were carefully positioned to ensure that all steps during this stage took place on the instrumented area of the walkway and that no additional steps were required for the patients to get into position prior to sitting on the chair. After pausing for ten seconds, the participants stood and undertook a continuous walk around the gait laboratory for a two minute period before returning to the chair. Throughout the test period, the participants were followed by a member of the research team (DW) who filmed the test procedure.

2.4. Data processing

To establish a gold standard step count, a mean value was derived from three manual counts of half-speed video replays. For the temporal and spatial parameters of gait simultaneously recorded by the GAITRite walkway and IDEEA, the GAITRite outputs were used as the gold standard as this system has been shown previously to be valid and reliable (Menz et al., 2004).

Raw data from the Step-N-Tune and A4L devices are presented directly as step counts and so required no further processing. GAITRite data were processed and analysed using the GAITRite v3.8 software (CIR Systems Inc., Sparta, New Jersey, USA).

For the IDEEA device, postural changes recorded when the participants sat on the two chairs provided discrete markers on the outputs with which to analyse subsections of the IDEEA output within the proprietary ActiView analysis software (MiniSun LLC, Fresno, California, USA). The two sitting events discretely bookended the section of gait along the GAITRite walkway, allowing separate validation of temporal and spatial gait parameters, in addition to the total step count of the full assessment period.

2.5. Statistical analysis

Summary statistics were used to describe demographic and clinical features of the participants.

Agreement between the activity monitors and gold-standard measures was assessed using Bland–Altman plots. The familiar method of calculating the 95% limits of agreement assumes that the mean and standard deviation (SD) of the differences between two methods are constant, irrespective of the magnitude of the measurements. When a non-uniform relationship between difference and magnitude is identified, Bland and Altman have proposed an alternative approach using linear regression. For a given magnitude of measurement, the mean difference between the measurement methods can be obtained from the constant and slope of the regression line, whilst the 95% limits of agreement are obtained from the SD of the residuals (Bland and Altman, 1999; Kirkwood and Sterne, 2006). Formally the mean difference is $\widehat{D}=b_0+b_{1}A$ where A is the estimate of the true value of measurement, and the 95% limits of agreement are calculated as $\widehat{D}\pm 1.96\sqrt{\pi /2}\widehat{R}$ where $\widehat{R}$ represents the residual SD from the regression. Statistical analyses were conducted using PASW 18.0.0.

3. Results

3.1. Participants' profile

The validation sample consisted of 24 participants (12 patients with RA and 12 healthy volunteers). Within the RA group, 10 were female; the group had a mean age of 51.6 years (SD = 18.0), and a mean BMI of 31.1 (SD = 7.1). In the RA group the mean disease duration was 18 years (SD = 13.4), and median DAS28 and HAQ were 5.02 (IQR = 3.76 to 5.75) and 1.5 (IQR = 1.0 to 2.13) respectively. Half of the control group were female (n = 6); the mean age was 41.6 years (SD = 9.8), and mean BMI was 23.4 (SD = 2.8). Although all the participants walked for the same period of time, patients with RA took fewer steps regardless of the method of counting used, as is seen in Table 1.

Table 1.

Group mean step counts for each measure.

Measure	Control group Mean (SD)	RA patients Mean (SD)
Gold standard	221.75 (19.21)	159.50 (38.65)
Step-N-Tune	213.75 (57.42)	86.27 (77.57)
A4L	247.17 (21.68)	177.27 (58.04)
IDEEA	184.67 (61.49)	72.92 (69.65)

Values indicate group average for each measure. Gold standard: mean of 3 counts from half speed video replay; A4L: Activ4Life; SD: standard deviation.

With the exception of the gold standards of video step count and GAITRite, all four of the test devices malfunctioned at least once during the study and in each case this did not become evident until the end of the walk or data processing. The A4L and Step-N-Tune failed once, whereas the IDEEA failed on three occasions. When devices failed, the automated step counts (IDEEA, Step-N-Tune, and A4L) failed to record any steps. Data from defective trials have been excluded from all analyses.

3.2. Step counts

Bland–Altman limits of agreement demonstrated that the automated step counts agreed relatively closely with the gold standard in the control group but differed markedly from the gold standard in patients with RA. With the Activ4Life device, the variation between the measurements in patients with RA was comparable to that seen in controls (see Fig. 2 and Table 2). Furthermore, in patients with low step counts as measured by the video gold standard, the automated measures tended to consistently further underestimate the number of steps taken. Conversely, in RA patients with higher step counts, i.e. nearer the levels seen in the control group, the error in the automated measures decreased.

Fig. 2.

Regression based Bland–Altman plots illustrating agreement between different activity monitors and the gold standard. Solid line: mean agreement; dotted lines: 95% limits of agreement.

Table 2.

Regression based Bland–Altman 95% limits of agreement and estimated mean differences for step counts of each of three automated measures.

	Device
	Step-N-Tune	A4L	IDEEA
RA patients
Variation in bias, per step	− 1.11x	− 0.38x	− 1.53x
Width of 95% LOA	169.74	52.90	225.00
Bias (95% LOA): 100 steps	110.59 (25.72 to 195.46)	6.12 (20.33 to 32.57)	127.84 (15.34 to 240.34)
150 steps	55.09 (− 29.78 to 139.96)	− 12.88 (− 39.33 to 13.57)	51.34 (− 61.16 to 163.84)
200 steps	− 0.41 (− 85.28 to 84.46)	− 31.88 (− 58.33 to − 5.43)	− 25.16 (137.66 to 87.34)
Controls
Variation in bias, per step	0.011x	− 0.13x	− 0.05x
Width of 95% LOA	22.64	44.16	61.52
Bias (95% LOA): 200 stepsa	− 6.51 (− 17.83 to 4.81)	− 20.75 (− 42.83 to 1.33)	21.47 (− 9.29 to 52.23)

Bias = estimated mean difference between gold standard and test device; LoA = limits of agreement; x = mean of gold standard and test device.

^aEstimates are only provided for controls at 200 steps because there were no observed data at lower step counts.

3.3. Temporal and spatial gait parameters

Analysis of temporal and spatial parameters recorded using the IDEEA activity monitor revealed marked differences in agreement between the measures across the two disease groups (see Fig. 3). The wider limits of agreement indicate poorer agreement in the RA patients for all IDEEA derived parameters except gait velocity, which showed slightly increased levels of agreement. In contrast to the step counts, the level of agreement between measures within the RA patients showed a systematic relationship in which there were lower levels of agreement in patients with higher levels of functional ability. Even in the control group however, the IDEEA activity monitor underestimated both gait velocity and double support time relative to the GAITRite gold standard.

Fig. 3.

Regression based Bland–Altman plots for temporal and spatial gait parameters. Solid line: mean agreement; dotted lines: 95% limits of agreement.

4. Discussion

This is the first study to investigate the validity of a range of activity monitors in patients with RA and a control group of healthy volunteers. Although there was some variation between devices, the major finding of this study was the large variation in the ability of the activity monitors to accurately record gait in this pathological sample. Whilst the devices generally showed reasonable agreement with the gold standard in the healthy population, the Step-N-Tune and IDEEA devices performed poorly in patients with RA, whilst the A4L device showed only a slight decrease in performance in patients with RA. The devices also demonstrated a systematic decrease in their performance as RA patients' functional ability fell further below the level of the control group.

This study provides useful information to help inform activity monitor selection in future studies through direct comparison of devices. The magnitude of error of some of the test devices was often very high in comparison to the number of steps taken during the walk, and this was particularly so in the RA group. If the limits of agreement are narrow, bias need not necessarily be problematic as one may apply a conversion factor to maintain consistency with the gold standard, however wide limits of agreement render it difficult to perform such a conversion with any confidence that the results would be accurate. Such wide limits of agreement as observed here call into question the validity of using some devices in patients with RA. In the current study, the A4L device had the highest level of agreement with the gold standard in patients with RA, although larger long-term studies are still needed to investigate the device's usability and durability in clinical populations outside the laboratory. Furthermore, the additional temporal and spatial parameters of gait measured by the IDEEA performed poorly in the RA group when compared to a laboratory gold standard.

Although early studies in healthy volunteers conclude that the IDEEA was 99% accurate in identifying types of activity in healthy volunteers, concerns have been raised more recently about this device's ability to measure temporal and spatial parameters (Maffiuletti et al., 2008; Zhang et al., 2003). Maffiuletti et al. suggest that the device's inability to measure acceleration in the transverse plane leads to large inaccuracies in step length estimates in patients with adducted or abducted feet (Maffiuletti et al., 2008). With such deformity being common in patients with RA, this may go some way to explaining the difference in the device's performance between disease groups in the current study.

The number of activity monitors available on the market is increasing and devices vary considerably in terms of their complexity and usability. They range from simple pedometers and smartphone apps to novel nine degrees of freedom devices that allow geographical tracking of subjects, as well as complex multisensor devices that also record heart and GPS data. This makes evidence based selection of activity monitors for specific clinical populations ever more challenging. To facilitate device comparison, future validation studies should seek to adopt common experimental and analysis frameworks, as well as ensuring that patients' functional ability is appropriately described. Furthermore, the growing complexity of some devices may provide a barrier for use outside the laboratory setting, particularly amongst patients with reduced dexterity and this also warrants consideration.

Although levels of agreement with the gold standard should be a major consideration when selecting a device, other more practical issues such as price and usability also warrant consideration: it would be possible to buy up to 500 Step-N-Tune pedometers for each IDEEA activity monitor purchased, and for simple step counts the Step-N-Tune pedometer performed better in the control group. Some devices, such as the IDEEA, require two episodes of face to face interaction between the researcher and the participant in order calibrate the device and then to download the data, whereas the A4L can transmit data without the need for direct interaction. The visible step count and inexpensive nature of the Step-N-Tune would enable participants to record step counts on a piece of paper and return the device through the post.

All of the activity monitors used in this study rely on acceleration to measure events, such as steps, during the gait cycle. Changes in acceleration are either interpreted using signal processing algorithms in the manufacturer's software, or via movement of a spring mounted lever arm. The threshold for interpreting a change in acceleration as a step is determined by the manufacturer during the device's development. The magnitude of accelerations affecting the limbs and trunk during gait is, in part, derived from a person's gait velocity and it has been widely documented that patients with RA have reduced gait velocity (Turner and Woodburn, 2008; Turner et al., 2003, 2006, 2008). Consequently, if the manufacturers set the threshold using data from healthy populations with normal gait, the device may underestimate activity levels in patients with RA as was found in the current study. Similar findings have also been reported in recent studies of older adults living in nursing homes without RA (Cyarto et al., 2004). Furthermore, spring loaded lever arm pedometers have been reported to miss steps in obese patients and those taken at low frequencies (Feito et al., 2012; Tyo et al., 2011). A potential further refinement to help overcome this limitation might be to enable the adjustment of the acceleration threshold required to record a step so that activity monitors could be calibrated to the particular characteristics of patients exhibiting pathological gait patterns.

A strength of this study was the concurrent testing of all activity monitors during the same walk. This excluded the possibility of fatigue and intra-subject variation between tests affecting the results. Furthermore, the study used the same test procedure to evaluate the activity monitors in a group of patients with RA and a normal control group. This suggests that any change in the magnitude of error between groups was due to differences in the characteristics of the group rather than systematic error or researcher bias.

The current study investigated the ability of a range of activity monitors in participants completing two tasks commonly performed in daily living. As well as a period of walking, the study contained transitions from walking to sitting, and from sitting to walking. A limitation of the study was that it did not investigate the validity of the devices during other common activities such as stair climbing, driving, or getting in and out of bed. Future studies should investigate the ability of devices to accurately measure activity during such activities before results from community dwelling populations can be interpreted with confidence.

It was of concern that all devices apart from the gold standard failed on at least one occasion during data capture. On each occasion, equipment failure and loss of data were not evident until the end of the walk meaning that no remedial action could be taken for that trial. Although this was problematic in our short periods of assessment, the consequences of equipment failure when the devices are used over a period of several days are likely to be magnified. Within the current study, equipment failure occurred more frequently in the RA group and in some devices more than others. Equipment failures of this nature were unexpected however, so the study was not designed to measure failure rates, compare them between different devices, or between disease groups. Appropriately designed and adequately powered studies are needed to properly evaluate failure rates and determine whether they are higher in different disease groups.

In conclusion, the activity monitors tested in the current study performed reasonably well in healthy young volunteers but all except the A4L showed a marked decrease in performance in patients with RA. Furthermore, there was a systematic improvement in performance as patient's functional ability increased towards the level of the control group. Although activity monitors offer potential in patients with RA, they must always be validated in target clinical populations.

Conflict of interest statement

All authors have declared that they have no financial interests that could create a potential conflict of interest or the appearance of a conflict of interest with regard to the work.