Validity of Activity-Based Devices to Estimate Sleep

Abstract

Study Objectives:

The aim of this study was to examine the feasibility of sleep estimation using a device designed and marketed to measure core physical activity.

Methods:

Thirty adolescent participants in an epidemiological research study wore 3 actigraphy devices on the wrist over a single night concurrent with polysomnography (PSG). Devices used include Actical actigraph, designed and marketed for placement around the trunk to measure physical activity, in addition to 2 standard actigraphy devices used to assess sleep-wake states: Sleepwatch actigraph and Actiwatch actigraph. Sleep-wake behaviors, including total sleep time (TST) and sleep efficiency (SE), were estimated from each wrist-device and PSG. Agreements between each device were calculated using Pearson product movement correlation and Bland-Altman plots.

Results:

Statistical analyses of TST revealed strong correlations between each wrist device and PSG (r = 0.822, 0.836, and 0.722 for Sleepwatch, Actiwatch, and Actical, respectively). TST measured using the Actical correlated strongly with Sleepwatch (r = 0.796), and even stronger still with Actiwatch (r = 0.955). In analyses of SE, Actical correlated strongly with Actiwatch (r = 0.820; p < 0.0001), but not with Sleepwatch (0.405; p = 0.0266). SE determined by PSG correlated somewhat strongly with SE estimated from the Sleepwatch and Actiwatch (r = 0.619 and 0.651, respectively), but only weakly with SE estimated from the Actical (r = 0.348; p = 0.0598).

Conclusions:

The results from this study suggest that a device designed for assessment of physical activity and truncal placement can be used to measure sleep duration as reliably as devices designed for wrist use and sleep wake inference.

Citation:

Weiss AR; Johnson NL; Berger NA; Redline S. Validity of activity-based devices to estimate sleep. J Clin Sleep Med 2010;6(4):336-342.

For some time, the importance of assessing physical activity as a critical health behavior has been recognized: reliable measurements of physical activity are useful in evaluating the role of physical activity in health promotion and also for evaluation of the relative effectiveness of public health programs and interventions.¹ Large-scale epidemiological studies have incorporated both subjective and objective tools for assessing physical activity. The measurement of energy expenditure by doubly labeled water is the “gold standard,” though it is both costly and has limited extendibility.² Accelerometry is one of the most commonly used methods for assessing free-living physical activity.³ Compared to other objective measures, it has been shown to be highly sensitive in detecting varying levels of physical activity,^4,5 and is currently being used by the US National Health and Nutrition Examination Survey (NHANES IV).⁶

Sleep has also been recognized to be a key health attribute associated with energy expenditure and other health outcomes. Large-scale epidemiological studies have employed both objective and subjective tools for measurement of sleep-wake behavior. The “gold standard” measure of sleep is made using polysomnography (PSG), though wrist actigraphy has gained popularity over the past several decades as an important lower-cost and minimally invasive measurement alternative.^7,8 Compared to PSG, actigraphy has been shown to be a reliable and valid instrument to assess sleep.^9–11

The need to integrate measurement of both sleep and physical activity in population studies is clear, though no uniform approach exists to measure sleep and physical activity concurrently. Use of the same instrumentation for measuring both sleep and activity could accelerate the collection of more comprehensive physiological data while minimizing cost and participant burden. The same apparatus would likewise facilitate clinical monitoring of sleep and activity in clinical disorders of energy balance. Devices used for the estimation of sleep using wrist actigraphy and for the estimation of physical activity by core accelerometry are, however, very similar. Both are used to record a digitally integrated measurement of gross motor activity. The marketable difference between the 2 devices is as follows: whereas the core accelerometer is typically worn on the trunk and used to predict energy expenditure and activity data, the wrist actigraph is worn on an extremity; and the smaller movements of the wrist as compared to the trunk are used to infer time spent asleep and wake.

BRIEF SUMMARY

Current Knowledge/Study Rationale: Physical activity and sleep both are attributes important for health. Although objective, concurrent measurement of each could improve our understanding of risk of obesity and other health conditions, there are little data that address the validity of measuring these attributes using the same devices.

Study Impact: This study supports the ability to use the same activity monitors to measure both physical activity and sleep, identifying the potential for future epidemiological and public health initiatives to measure both of these important health attributes concurrently.

We aimed to test the hypothesis that a device specifically marketed for core accelerometry could be moved to the wrist and could generate data that would result in accurate estimation of sleep-wake states. We collected concurrent data from several actigraphy/ accelerometry devices in order to compare estimation of sleep. A sample of 30 participants in a Cleveland-based epidemiological research study using actigraphy to estimate sleep participated by wearing 3 devices on a single night concurrent with PSG.

METHODS

Subjects consisted of 30 adolescents participating in the late-adolescent examination of the Cleveland Children's Sleep and Health Study (CCSHS), an ongoing longitudinal cohort study designed to evaluate sleep measures and their health outcomes. Participants were all without sleep apnea or severe comorbidities. As part of the study protocol participants underwent examination at a clinical research center where overnight PSG and concurrent actigraphy (using Sleepwatch actigraph; Ambulatory Montoring, Inc., Ardsley, NY) were performed using a standardized protocol.¹² Subjects for this investigation were consecutively seen participants in the research protocol who agreed to wear the additional monitors on the night of their PSG. The study was approved by the institutional review board, and signed informed consent was obtained for all participants.

The 3 actigraph devices evaluated were: (1) Sleepwatch (Ambulatory Monitoring, Inc., Ardsley, NY); (2) the Actiwatch (Respironics, Pittsburgh, PA); and (3) the Actical (Respironics, Pittsburgh, PA). The Sleepwatch and Actiwatch each were designed to be worn on the wrist to estimate sleep-wake states. The Actical was designed and is marketed for placement around the trunk to measure physical activity. Participants wore each device on the wrist of the non-dominant hand concurrent with PSG.

PSG data were scored based on established criteria for sleep staging¹³ to provide a gold standard measure of sleep state in 30-sec epochs. The beginning and end of the in-bed period in the electronic record were marked as “Lights Out” and “Lights On,” respectively. Total sleep time (TST) was calculated as the total duration of epochs scored as sleep between lights out and lights on. Sleep efficiency (SE) was calculated as the ratio of TST to the total time between lights out and lights on.

Data from the Sleepwatch were collected in 1-min epochs, and data in each epoch summarized using the time above threshold (TAT) method. The TAT mode has previously been shown to correlate best with PSG sleep time in adolescents¹⁴ and was therefore currently being used as part of the study protocol. The UCSD scoring algorithm¹⁵ was used within Action-W software (Ambulatory Monitoring, Inc., Ardsley, NY) to score each epoch as sleep or wake. The algorithm generates an activity score for each epoch as a weighted average of the level of activity for the current epoch and that of the surrounding epochs, which is then used to determine if each epoch is scored as sleep or wake.

Both the Actiwatch and the Actical utilize the same mechanism, a piezoelectric accelerometer, to generate voltage based on movement, and similarly generate activity counts for each epoch. An activity score is generated for each epoch as a weighted average of the activity count for the current epoch and that of the surrounding epochs (± 2 min).^11,16 Data from the Actiwatch and the Actical were collected in 15- or 30-sec epochs, and the appropriate parameters in the scoring algorithm used based on the epoch length, but the window for generating the activity scores remained ± 2 minutes.¹⁶ The scoring algorithm was implemented using a custom program developed in SAS (v. 9.2, SAS Institute Inc., Cary, NC), allowing data from the Actiwatch and the Actical to be scored using the same approach.

Similar to the Sleepwatch, each epoch of data from the Actiwatch and Actical was assessed as sleep or wake, based on whether or not the activity score exceeds a set threshold. A preliminary review of the raw activity counts from the Actiwatch and the Actical revealed that the Actical tended to return lower activity counts; a sample of 4 research volunteers (n = 9648 min of data collection) identified that the Actical indeed returned lower activity scores on average (124.4 from Actiwatch vs. 90.6 from Actical). To account for the lower activity scores, a low sensitivity threshold (activity score of 20) was used to determine sleep in the Actical, while the medium-sensitivity threshold (activity score of 40) was used for the Actiwatch. Both the low and the medium sensitivity thresholds have been shown to agree well with sleep staging from PSG.¹¹

Data from each device were aligned by time and truncated to include only the period from lights out to lights on as indicated in the PSG. An example of the activity counts for one subject from each of the 3 devices is shown in Figure 1. While comparison of the values of the activity counts are not meaningful,¹⁷ similar trends are noticeable and correspond to the scoring of sleep from PSG. Data were then summarized to determine TST and SE from each actigraphy device, utilizing the same definitions used for PSG.

Figure 1.

Sleep stage summary from PSG

Example of the actigraphy output concurrently measured on one person from a single night. The top tracing shows the “hypnogram” output from concurrent PSG, showing sleep onset at approximately 23:15. Plots A, B, and C, indicate the activity counts from the Sleepwatch, Actiwatch, and Actical, respectively. The vertical lines indicate movement, which are used to infer wakefulness. The absence of movement is used to infer sleep.

The difference between TST and SE as assessed by PSG and the 3 actigraphy devices was examined. Agreement between PSG and each device was calculated using Pearson product-movement correlation and calculation of mean differences between the devices. Bland-Altman¹⁸ plots were used to show the relative bias between the data obtained from each wrist device and PSG. All statistical analyses were performed using SAS software version 9.2 (SAS Institute, Inc, Cary, NC).

RESULTS

Subject Characteristics

Subjects were mean age 17.6 ± 0.3 years, and consisted of 19 male and 11 female participants. The mean BMI was 25.8 ± 5.6 kg/m². Participants were otherwise free of serious comorbidities.

TST from Actigraphy Devices and PSG

Average TST measured by PSG was 463 ± 71 min. Actigraphy devices tended to overestimate sleep when compared to PSG: 488 ± 63 min for the Sleepwatch, 489 ± 60 for the Actiwatch, and 497 ± 53 for the Actical. As shown in Table 1, strong, positive correlations were observed for TST estimated by the Sleepwatch with TST estimated by both Actiwatch and Actical (r = 0.881 and 0.796, respectively), though the two latter devices which were produced by a single manufacturer corresponded best with each other (r = 0.955). Shown in Figure 2 are Bland-Altman plots for analysis of Actiwatch vs. Actical and Sleepwatch vs. Actical for TST. The mean difference line for each plot did not significantly vary from zero, showing that the TST measured from the Actical is comparable to both the Sleepwatch and Actiwatch.

Table 1.

Pearson correlation coefficient for total sleep time from polysomnography (PSG) and actigraphy devices^†

N = 30	PSG	Sleepwatch	Actiwatch	Actical
PSG		0.822*	0.836*	0.722*
Sleepwatch	0.822*		0.881*	0.796*
Actiwatch	0.836*	0.881*		0.955*
Actical	0.722*	0.796*	0.955*

^*p < 0.001;

^†Sleepwatch refers to Sleepwatch actigraph (Ambulatory Monitoring, Inc., Ardsley, NY); Actiwatch, Actiwatch actigraph (Respironics, Pittsburgh, PA); Actical, Actical actigraph (Respironics, Pittsburgh, PA)

Figure 2.

Bland-Altman bias plot showing TST made by PSG and compared to the actigraphy estimates from each device

Plots A, B, and C, indicate analyses of Sleepwatch, Actiwatch, and Actical, respectively. The pink horizontal lines correspond to the mean difference between the methods, and the upper and lower blue horizontal lines represent the mean ± 2 SD, respectively.

Statistical analyses also revealed moderate to strong positive associations among the Sleepwatch, Actiwatch, and Actical to PSG (r = 0.822, 0.836, and 0.722 for Sleepwatch, Actiwatch, and Actical, respectively). Each device, regardless of manufacturer or original design (sleep-wake vs. activity), however, provided a systematically higher mean estimate of TST, differing from PSG by 25-31 min (Table 2). Shown in Figure 3 are Bland-Altman plots for the 3 different wrist devices used in this analysis. The mean difference lines for each device were below zero, which graphically shows the systematic bias towards overestimation of TST relative to PSG when using any of the 3 actigraphy devices compared to PSG.

Table 2.

Comparison of total sleep time (TST) from polysomnography (PSG) and actigraphy devices^†

	N	PSG	Sleepwatch	Actiwatch	Actical
TST on PSG night (min)	30	462.9 ± 71.5	488.1 ± 63.3	488.1 ± 59.7	494.3 ± 53.8
		467.0 (417.0 – 512.0)	495.0 (447.0, 527.0)	491.9 (453.5, 523.5)	506.6 (455.8, 529.3)
Difference with TST from PSG (min)	30		−25.1 ± 41.0	−25.1 ± 39.3	−31.4 ± 49.5
			−26.5 (−45.0, −1.0)	−17.1 (−47.5, 5.0)	−22.9 (−58.5, 0.0)
Difference with TST from Actiwatch (min)	30		0.0 ± 30.2		−6.3 ± 18.1
			−2.1 (−5.5, 4.0)		−6.1 (−15.3, 5.3)

(Mean ± SD; Median [IQR])

^†Sleepwatch, Sleepwatch actigraph (Ambulatory Monitoring, Inc., Ardsley, NY); Actiwatch, Actiwatch actigraph (Respironics, Pittsburgh, PA); Actical, Actical actigraph (Respironics, Pittsburgh, PA)

Figure 3A.

Bland-Altman bias plot between Actical and Sleepwatch measurement of TST

The pink horizontal lines correspond to the mean difference between the methods, and the upper and lower blue horizontal lines represent the mean ± 2 SD, respectively.

Figure 3B.

Bland-Altman bias plot between Actical and Actiwatch measurement of total sleep time

The pink horizontal lines correspond to the mean difference between the methods, and the upper and lower blue horizontal lines represent the mean ± 2 SD, respectively.

SE From Actigraphy Devices and PSG

Average SE measured by PSG was 87% ± 10%. Actigraphy devices tended to overestimate SE compared to PSG: 92% ± 7% for the Sleepwatch, 92% ± 7% for the Actiwatch, and 94% ± 6% for the Actical. As shown in Table 3, for sleep efficiency (SE) a strong, positive correlation was observed between the Actiwatch and Actical, the two devices produced by the same manufacturer, but marketed for sleep-wake estimation and activity estimation, respectively (r = 0.820; p < 0.0001). SE estimated using the Sleepwatch was only moderately correlated with SE estimated using Actiwatch (r = 0.653; p < 0.0001), and weaker still with Actical (0.405; p = 0.0266). Compared to PSG, the Sleepwatch and Actiwatch showed moderate positive correlations (r = 0.619 and 0.651, respectively), whereas there was a weaker correlation between PSG and Actical (r = 0.348; p = 0.0598). Table 4 shows the mean estimates of SE (%), which were systematically high, though the mean deviation from PSG was only within 7 percentage points for each device.

Table 3.

Pearson correlation coefficients for sleep efficiency from polysomnography (PSG) and actigraphy devices^†

N = 30	PSG	Sleepwatch	Actiwatch	Actical
PSG		0.619**	0.651**	0.348
Sleepwatch	0.619**		0.653**	0.405*
Actiwatch	0.651**	0.653**		0.820**
Actical	0.348	0.405*	0.820**

^*p < 0.05;

^**p < 0.001;

^†Sleepwatch, Sleepwatch actigraph (Ambulatory Monitoring, Inc., Ardsley, NY); Actiwatch, Actiwatch actigraph (Respironics, Pittsburgh, PA); Actical, Actical actigraph (Respironics, Pittsburgh, PA)

Table 4.

Comparison of sleep efficiency (SE) from polysomnography (PSG) and actigraphy devices^†

	N	PSG	Sleepwatch	Actiwatch	Actical
Sleep efficiency (%)	30	87.3 ± 9.6	92.1 ± 7.0	92.1 ± 6.2	93.3 ± 6.3
		88.4 (80.9, 95.6)	93.4 (89.2, 96.5)	93.8 (88.2, 97.0)	96.6 (86.6, 98.3)
Difference with SE from PSG (%)	30		−4.8 ± 7.6	−4.8 ± 7.3	−6.2 ± 9.5
			−5.2 (−9.0, −0.2)	−3.4 (−10.0, 0.9)	−4.3 (−11.6, −0.1)
Difference with SE from Actiwatch (%)	30		0.0 ± 5.6		−1.4 ± 3.8
			−0.4 (−1.0, 0.9)		−1.2 (−2.8, 1.1)

(Mean ± SD; Median [IQR])

^†Sleepwatch, Sleepwatch actigraph (Ambulatory Monitoring, Inc., Ardsley, NY); Actiwatch, Actiwatch actigraph (Respironics, Pittsburgh, PA); Actical, Actical actigraph (Respironics, Pittsburgh, PA)

Similar analyses were performed for wake after sleep onset (WASO). Generally, the patterns of correlation followed those observed for SE. Specifically, for WASO, the correlations to PSG for measurements derived from the Sleepwatch, Actiwatch, and Actical were 0.432, 0.602, and 0.187, respectively.

DISCUSSION

The present study examined the validity of an accelerometry device designed to measure activity to measure sleep-wake behaviors. We evaluated the agreement among three different actigraphy devices, as well as the comparability of data from each device to the gold standard measure of sleep from PSG. We specifically examined how Actical (specifically marketed to measure physical activity) compared to reference standards. The results showed that an accelerometry device marketed specifically to assess physical activity can provide measures of sleep duration comparable in accuracy to those marketed for sleep-wake estimation and well correlated with measures made from PSG. We also observed variability in SE estimates derived from devices produced by different manufacturers, and poorer levels of agreement for measures of sleep efficiency obtained from each of the actigraphy devices compared to the gold standard PSG.

Wrist actigraphy has emerged over the past several decades as a critical method for estimating sleep duration, sleep efficiency, and sleep timing: it is not only more objective than sleep diaries, but it is also less expensive and invasive than PSG.^5,19 Whereas PSG usually requires participants to be monitored in the sleep laboratory where data can usually only be attained for one or two nights, actigraphy can collect data over multiple nights without interfering with the participant's normal sleep setting.⁵

Core accelerometry has also become widely used to estimate physical activity.²⁰ With technological advances in circuitry and memory capacity,²¹ accelerometers are equipped to measure the intensity, frequency, and duration of body movement in a digitally integrated manner.

In the present study, three actigraphy devices (Sleepwatch, Actiwatch, and Actical) exhibited strong correlations with the gold standard in analyses of TST. Each device, however, systematically overestimated TST relative to PSG. This is consistent with the literature: Overestimation is common in healthy samples,^22–24 and may be due to the limitations of actigraphy in detecting wake periods. Because of the decreased sensitivity of actigraphy devices relative to PSG, overestimation can occur, for example, when a subject lies immobile in bed in a non-sleeping state. This is especially true when subjects are studied in the sleep laboratory for the first time, due to lower sleep efficiency and possibly perceived constraints during monitoring. In a study of healthy older adults, TAT algorithm provided TST estimates that were overestimated compared to PSG by a mean of 33 minutes.²⁵ Underestimation is most likely to occur in younger populations with sleep disorders, as sleep-wake identification by wrist actigraphy may be confounded by external motion. In prior work on a younger sample, actigraphy analyzed in the TAT mode underestimated TST, with the largest biases occurring in boys with sleep apnea.¹⁰ Overestimation or underestimation of TST is also contingent upon the specific scoring algorithm used.¹⁰

Sleep efficiency, a measure of sleep quality that is reduced in individuals with frequent awakenings or arousals, has been associated with multiple health outcomes, including obesity, hypertension, and diabetes.^27,28 SE estimates were attained from each device to further assess validity relative to PSG. Although a high correlation was achieved for sleep efficiency measured from Actical and Actiwatch (the two devices produced by the same manufacturer), all other correlations were weaker, with the weakest correlation for the Actical compared to the PSG. Low SE estimates reported for this group of adolescents is, however, consistent with the literature.²⁹ SE detected via actigraphy is prone to underestimation more so then TST among healthy subjects.²⁴ Though the main confounder is overall lack of sensitivity,²⁹ the differences in algorithms for computing rapid changes in sleep wake state used by the varying devices (including differences in the statistical weighting algorithms used) may introduce more variability in sleep efficiency estimation than in estimation of TST.

Discrepancies between sleep measures assessed via actigraphy and PSG may be largely due to differences in wake-time sensitivity. Differences in manufacturer specifications can result in subtle differences in measures of wake and sleep efficiency. To explore this point further, we looked at measures of wake after sleep onset (WASO) for each of the devices. Like SE, WASO is an important tool for measuring sleep quality. Measures of WASO in this sample correlated strongly with SE, but poorly with PSG indices of WASO. Thus, our data suggest that while TST may be reliably estimated from the three devices we evaluated, there needs to be further refinement and evaluation of actigraphy-based algorithms for improving estimation of wake times that particularly influence assessment of indices such as SE and WASO.

An important finding is that the Actical, the device designed and marketed for physical activity monitoring and the same or similar to those used in large-scale studies of physical activity, showed strong correlations with both the AMI Sleepwatch and Actiwatch in measurements of TST. This finding supports the potential use of this device in studies where the aim is to collect physical activity information during the day and sleep duration information at night (or during the subject's usual sleep period). The ability to use one device to measure physical activity and sleep-wake behaviors could provide researchers an approach for incorporating information from both sleep and physical activity to characterize health behaviors.

A limitation of this study is the small sample size, restricted to healthy young subjects. Future work should include more diverse samples, as well as expand assessment to “field” settings, testing the feasibility in large samples of moving the device from one location (trunk) to another (wrist) and use over multiple days of measurement. Another limitation is the collection of data using different epoch lengths, though appropriate weights were applied to address any potential bias.

The study's strengths include concurrent data collection of PSG and actigraphy data with controlled evaluation that allowed the performance of these devices to be assessed without environmental influences. An adolescent sample, a group at high risk for insufficient sleep, was studied.

CONCLUSIONS

The three wrist devices evaluated produced reliable measures of TST relative to the “gold standard” PSG. Further, we were able to show that Actical, a device to be placed on the trunk and assess physical activity, can be used to measure sleep duration as reliably as devices designed for wrist use and sleep wake inference. The poorer performance of all actigraphy devices, and especially the Actical, in measuring sleep efficiency suggests the need for further adjustments of scoring and sensing algorithms for more reliably capturing sleep-wake transitions. Further development of devices for flexible measurement of physical activity or of sleep may facilitate research of these interrelated health attributes. Additional research, however, is needed to assess the practical consequences of having one device worn during the day on the trunk for physical activity measurement, with movement to the wrist for sleep-wake assessment, as well as assessment of the comparability of information obtained over multiple day assessments.

DISCLOSURE STATEMENT

This was not an industry support study. Dr. Redline has received equipment from Philips Respironics for use in clinical trials and has received support from Dymedix Inc. The other authors have indicated no financial conflicts of interest.