Invalidity of one actigraphy brand for identifying sleep and wake among infants

Abstract

Study Objectives:

Few commercially available brands of actigraphs (ACT) have been subjected to rigorous validation with infant participants. The purpose of this study was to examine the agreement between concurrent polysomnography (PSG) and one brand of ACT (AW-64, Mitter Co. Inc.) using appropriate statistical techniques among a sample of healthy infants.

Methods:

Twenty-two healthy infants (14.1 ± 0.6 months) had one night of ankle ACT recording during research PSG at Kosair Children's Hospital Sleep Research Center in Louisville, Kentucky. Macroanalyses were conducted using the Bland-Altman concordance technique to assess agreement between total sleep time (TST) and wake after sleep onset (WASO) simultaneously measured by PSG and ACT, using two ACT algorithm settings. Microanalyses were also calculated to examine sensitivity, specificity, and accuracy of ACT within each PSG-identified sleep state. Correlations were calculated between PSG-identified arousals and the discrepancies between ACT and PSG.

Results:

The Bland-Altman concordance technique revealed that ACT underestimated TST by 72.25 (SD = 61.48) minutes and by ≥ 60 minutes among 54.55% of infants. Furthermore, ACT overestimated WASO by 13.85 (SD = 30.94) minutes and by ≥ 30 minutes among 40.91% of infants. Sensitivity, specificity, and accuracy analyses revealed that ACT adequately identified sleep, but poorly identified wake. PSG and ACT discrepancies were positively associated with PSG-identified arousals (r = .45).

Conclusions:

Improved device and/or software development is needed before the AW-64 can be considered a valid method for identifying infant sleep and wake.

INTRODUCTION

Polysomnography (PSG), the ‘gold standard’ for identifying sleep state, is time consuming, expensive, intrusive, and its availability to infants and children is relatively limited. Actigraphy (ACT) has been advocated as an alternative method to allow naturalistic recording of a restricted number of infant sleep measures, including time spent in sleep and wake states. An actigraph is an accelerometer, typically worn on the wrist by adults and on the ankle by infants, that can be used to identify periods of sleep and wake from the absence and presence of detectable body movement. ACT is attractive because it is cost effective, unobtrusive, and has ambulatory capabilities.

Various brands of ACT on the market, each with distinct features, design, and software, process and interpret movement signals according to different algorithms. Across studies, ACT has been determined as a viable method for identifying sleep and wake among adults and the validity of such assumption has been reviewed extensively.^1-3

Previous studies support the validity of specific ACT devices among infants (the AMA-32 [Ambulatory Monitoring, Inc.],⁴ an unspecified accelerometer,⁵ and the AW-64 [Mini Mitter Co. Inc.],^6,7) and among adult subjects (Mini Motionlogger Actigraph – Basic 32 C [Ambulatory Monitoring, Inc.],⁸ and the AW-64 [Mini Mitter Co.]⁹).

Regardless of the dynamic developmental characteristics of human sleep,^10,11 ACT assessment methodology has not been modified based on the developmental stage of the target subjects. Therefore, the same measurement and analytical criteria are used to examine sleep, which is distinctly different among infants and adults. The lack of equivalence in measures used for infant and adult sleep monitoring has previously been addressed and has led to emphasis on the need to apply age-sensitive adjustments when similar methodologies are used.¹² While there are distinct PSG criteria for scoring sleep among infants and adults,¹³ few brands of ACT include different scoring criteria for infants.

Experts have established recommended practice parameters for the use of ACT for normal and disordered sleep assessments;¹⁴ upon the strength of this evidence, a Current Procedural Terminology (CPT) Category code was established for actigraphy in 2009. According to Tryon's critical review of theoretical and methodological issues regarding the use of ACT as a valid instrument for sleep assessment, a favorable view of ACT emerged.¹⁵ Many of the studies used to determine the validity of ACT, however, have relied on inappropriate statistical techniques, specifically correlations and comparison of means.

The Bland-Altman concordance technique is a more appropriate analytical approach when compared to correlations or means comparisons and enables critical assessment of agreement between instruments that concurrently attempt to measure the same construct. This is because correlations measure association, not agreement, and the comparison of means relies on measurement error that could bolster ability to “fail to reject the null hypothesis;” hence there is no difference between measures (see review by Altman & Bland, 1983¹⁶). The Bland-Altman concordance technique utilizes graphical comparisons between two methods of measurement by plotting the difference between two measurements against their average; the differences between the measures are fitted to lines that represent the average difference between the measures, ± 1 standard deviations, and ± 2 standard deviations.¹⁶

The Bland-Altman concordance technique has rarely been used to examine ACT validity. The AW-64 (Mini Mitter Co. Inc.) was compared to PSG in a cohort of adult subjects and was considered a useful instrument for sleep assessment.⁸ The Actiwatch Plus AW4 (Cambridge, Neurotechnology) was compared to parental reports in a pediatric sample aiming to validate the use of sleep-wake diaries.¹⁷ But few ACT device brands have undergone rigorous validation in infants. The purpose of this study was to examine agreement between concurrent PSG and one brand of ACT (AW-64) using appropriate statistical techniques in a sample of healthy infants. The current research was conducted in line with the “Recommendations for Future Research” described by the Standards of Practice Committee of the AASM.¹⁴ Specifically, ACT was compared to PSG as a reference standard using improved comparison methodologies, and the ACT device, algorithm, designated start and stop times, and the technical details about ACT scoring are described accordingly.

METHOD

The analyses were conducted on data from 22 healthy infants who participated in a larger, longitudinal study in Louisville, Kentucky. All these subjects wore an actigraph around their ankle during an overnight research PSG.¹⁸ The study was approved by the institutional review boards at the University of Louisville and Kosair Children's Hospital. Informed consent and Health Information Portability and Accountability Act authorization were administered to the parent(s).

Overnight Polysomnography

Overnight PSG was performed at the Kosair Children's Hospital Sleep Research Center in Louisville, Kentucky with one parent present at all times. No study was performed on a night when an infant was sick. Commercially available multichannel data acquisition equipment (MedCare Diagnostics, Amsterdam, The Netherlands) was used to record four-channels of electroencephalography (O1/O2, C3/C4), chin electromyography, bilateral electrooculography, snore sensor, electrocardiogram, chest and abdominal inductance plethysmography, pulse oximetry and waveform, and thermistor-derived oronasal airflow. Simultaneous audio/video monitoring was digitally recorded. All PSG records were scored by a single analyst who was blinded to the findings of the ACT recording.

Sleep stages and events were scored using Rechtschaffen and Kales criteria (1968)¹⁹ to obtain polysomnographically-measured total sleep time (PSG-TST) and wake time after sleep onset (PSG-WASO). At the time of data collection, the new criteria for arousals had not been published for children.¹³ Spontaneous and respiratory-related arousals were, therefore, manually scored, as recommended by the American Sleep Disorders Association Task Force report,²⁰ and are consistent with the recently updated American Academy of Sleep Medicine criteria.¹³ The frequency of all arousals within each sleep stage was also calculated.

Actigraphy

Concurrent with PSG, infants wore an actigraph (Actiwatch AW-64, Mini Mitter, Inc.) around one ankle. ACT data were collected using the highest resolution setting of 15-second epochs.

ACT signals were scored using Actiware software version 5.52.0003 (Mini Mitter, Inc.) to calculate total sleep time (ACTTST) and wake after sleep onset (ACT-WASO). ACT signals were initially scored using a default parameter of a medium wake threshold value = 40 (WTV-40). Based on initial analyses, a higher wake threshold value setting was then evaluated, and the same recorded signals were re-scored, this time using a “high” wake threshold value = 80 (WTV-80). The Actiware software utilized an algorithm that scored individual epochs as sleep or wake by weighting the activity from the 8 epochs that surrounded each epoch, then added together the values for all 9 epochs. The combined value used to represent each individual epoch was then compared to the WTV; epoch values > the WTV were scored as wake.

The ACT-scored interval was manually set, beginning at PSG identified lights-out and ending at PSG identified lights-on. Within the manually set period, each WTV setting was used to calculate the following: Sleep Onset = the first epoch scored as immobile that was sustained for ten minutes or longer; Sleep End = the last epoch scored as immobile that was sustained for ten minutes or longer; TST = the number of minutes scored as sleep within the manually set interval; WASO = the number of minutes scored as wake following sleep onset within the manually set interval.

Statistical Analyses

SPSS 16.0 (SPSS Inc, Chicago, IL) was used for statistical calculations; a p < 0.05 was considered statistically significant and Cohen's d values were reported to demonstrate the size of the effect. Descriptive statistics were calculated for demographic, PSG, and ACT measures.

“Macroanalyses” were initially conducted to examine the global agreement between PSG-TST and ACT-TST and between PSG-WASO and ACT-WASO. These analyses were calculated for each of these two comparisons with ACT set at either WTV-40 (default) or WTV-80 criteria.

“Microanalyses” were conducted to assess the agreement between PSG and ACT within sleep stages (identified by PSG). Finally, PSG-identified arousals were examined for their potential to influence the PSG and ACT agreement. For both macro- and microanalyses, PSG and ACT variables were converted to a minute-based scale.

Macroanalyses

For illustrative purposes, Pearson's bivariate correlations, intraclass correlation coefficients (ICC) with two-way mixed effects models and absolute agreement, and paired sample t-tests were calculated to show the associations and differences among concurrent PSG and ACT measures. Then the more appropriate Bland-Altman concordance technique¹⁶ was used to determine if a meaningful agreement could be found between concurrent PSG and ACT measures. For each of the comparisons, the average of PSG and ACT was plotted on the X-axis, and the differences between PSG and ACT were plotted on the Y-axis. According to the Bland-Altman technique, grouping assumes that both instruments are somewhat valid; however, in the sleep field, PSG is considered the gold standard and is, therefore, presumed as the measure against which validity of other measurement devices must be demonstrated. Thus, in addition to the standard Bland-Altman concordance technique, another series of plots was generated to describe the difference between the concurrent PSG and ACT measures. Instead of showing the average of PSG and ACT on the X-axis, PSG was plotted on the X-axis, and the differences between PSG and ACT were plotted on the Y-axis. The differences between measures were fitted to a line to which a zero crossing was imposed, that is, indicating no difference between the concurrent measures. Lines were also drawn to illustrate ± 30 min. and ± 60 min. from the line of identity.

Microanalyses

In addition to the comparison between global TST and WASO, within-stage comparisons were calculated. Separate computers were used to record PSG and to set up and download the actigraphs. These computers were manually synchronized within one minute from each other. Consequently, the two clocks may have been misaligned by up to 59 seconds, which prohibits epoch-by-epoch analyses of agreement, particularly during transitions between stages and during PSG-identified arousals. Therefore, to examine agreement between PSG and ACT within sleep stages, a conservative two-minute period was identified at the beginning and end of each PSG identified sleep stage bout and was excluded from further analyses. PSG and ACT were time-matched on the remaining portion of each stage bout that was not excluded from analyses (Figure 1); this will be referred to as the adjusted stage bout. The adjusted stage bout allowed us to look at the agreement between PSG and ACT within each PSG identified stage bout, while unequivocally avoiding error due to misalignment in clock time. That is, potentially misaligned times within the 0 - 59 second range were excluded because the transitions between stage bouts were truncated by a time that was twice as great (i.e., two minutes) as the maximum amount of potential misalignment in clock times (i.e., 59 seconds).

Figure 1.

Schema of an adjusted stage bout used for microanalyses of PSG and ACT within PSG-identified S1/2, SWS, REMs, and wake.

Note: 1 = Sleep and 0 = Wake

Sensitivity, specificity, and accuracy of ACT were calculated within each adjusted stage bout. PSG was scored according to 30-second epochs, whereas ACT was scored according to 15-second epochs. Consequently, in order to compare these two methods within stages, each 30-second PSG epoch was separated into two 15-second epochs of the same stage for analyses (See Figure 1). Following the comparisons, these data were reported in minutes.

Sensitivity, specificity, and accuracy of ACT were calculated for minutes of TST; stages 1 and 2 combined (S1/2) because only two infants had stage 1 bouts > 4 minutes to allow calculation of adjusted stage bouts, slow wave sleep (SWS), and rapid eye movement sleep (REMs). Sensitivity was the proportion of minutes that ACT accurately identified sleep by PSG. Specificity was the proportion of minutes that ACT accurately identified wake by PSG. Accuracy was the proportion of total minutes of PSG-identified sleep and wake that were correctly recognized by ACT. Sensitivity, specificity, and accuracy were calculated within participants and then averaged across participants.

Finally, Pearson's correlations were calculated to determine whether the differences between PSG and ACT were associated with arousal frequency.

RESULTS

The demographic and family characteristics of the infant cohort are shown in Table 1. All polysomnographically measured sleep and respiratory measures were within normal limits (Table 2 in online supplemental data). Central apneas accounted for the vast majority of apnea-hypopnea indices and desaturations; only one infant had two obstructive apneas (total).

Table 1.

Demographic and Family Characteristics (N = 22)

Characteristic	Mean ± SD (range)
Age (months)	14.1 ± .56 (13.0-15.0)
Females, %	45.5
Whitea, %	77.3
Birth weight (lb)	7.6 ± .9
Gestational age (week at birth)	38.5 ± 1.8
Primipara, %	47.6
Maternal age (years)	30.0 ± 5.2
Maternal education (years)	16.1 ± 3.3

Note:

^aNon-White: Biracial (13.64%), Hispanic (4.55%), Other (4.55%)

Macroanalyses

Table 3 (online supplemental data) shows descriptive statistics for sleep and wake identified by PSG and ACT for macroanalyses.

WTV-40

There was a statistically significant correlation (r = .83, p < 0.001; ICC = 80, p < 0.001) between PSG-TST and ACT-TST (Figure 2 in online supplemental data); however, there was also a statistically significant difference, with PSG-TST higher than ACT-TST, t(21) = 5.51, p < 0.001, d = .70. According to the standard Bland-Altman concordance technique, ACT underestimated TST by 72.25 (SD = 61.48) minutes, and 18.18% of infants were > 1 SD beyond the difference between measures (Figure 3 in online supplemental data). Furthermore, when compared to PSG-TST, ACT-TST was underestimated by ≥ 60 minutes among 54.55% of infants (Figure 4), and the difference between PSG-TST and ACT-TST ranged between −276.0 to −6.0 minutes.

Figure 4.

Difference between PSG-TST and ACT-TST (WTV-40 [left] and WTV-80 [right]) plotted against PSG-TST with lines indicating no difference, > 60, and > 120 minute differences between measures.

There was a statistically significant correlation (r = .52, p < 0.05; ICC = .65, p < 0.01) between PSG-WASO and ACT-WASO (Figure 5 in online supplemental data); however, there was also a statistically significant difference, with PSG-WASO being significantly lower than ACT-WASO, t(21) = −2.1, p < .05, d = .44. ACT overestimated WASO by 13.85 (SD = 30.94) minutes, and 31.82% of infants were > 1 SD beyond the difference between the two measures (Figure 6 in online supplemental data). Furthermore, ACT incorrectly identified WASO by ≥ 30 minutes among 40.91% of infants (Figure 7), and the range of differences between PSGWASO and ACT-WASO was −46.8 to 76.5 minutes.

Figure 7.

Difference between PSG-WASO and ACT-WASO (WTV-40 [left] and WTV-80 [right]) plotted against PSG-WASO with lines indicating no difference, > 30, and > 60 minute differences between measures.

WTV-80

There was a statistically significant correlation (r = .84, p < 0.001; ICC = .86, p < .001) between PSG-TST and ACTTST (Figure 8 in online supplemental data); however, PSG-TST was significantly higher than ACT-TST, t(21) = 4.04, p < 0.01, d = .50. Accordingly, ACT underestimated TST by 52.05 (SD = 60.38) minutes, and 13.64% of infants were > 1 SD beyond the difference between the two measures (Figure 9 in online supplemental data). Furthermore, ACT-TST was underestimated by ≥ 60 minutes among 31.82% of infants (Figure 4), and the difference between PSG-TST and ACT-TST ranged between −253.5 to 19.5 minutes.

There was a statistically significant correlation (r = .52, p < 0.05; ICC = .67, p < .01) between PSG-WASO and ACT-WASO (Figure 10 in online supplemental data), but there were no differences between PSG-WASO and ACT-WASO, t(21) = 1.11, p = 0.28, d = .23. According to the Bland-Altman concordance technique, ACT underestimated WASO by 6.35 (SD = 26.78) minutes, and 31.82% of infants were > 1 SD beyond the difference between the two measures (Figure 11 in online supplemental data). Indeed, ACT incorrectly identified WASO by ≥ 30 minutes among 18.18% of infants (Figure 7), with the range of differences between PSG-WASO and ACT-WASO being −67.5 to 40.0 minutes.

Microanalyses

Table 4 shows descriptive statistics for sleep and wake identified by PSG and ACT (WTV-40) within stages for microanalyses. Table 5 shows sensitivity, specificity, and accuracy calculations for ACT compared to PSG.

Table 4.

Descriptive statistics for PSG and ACT (WTV-40) presented as mean ± SEM: Microanalyses

Sleep Stage:	PSG	ACT	PSG-ACT difference	PSG arousals
Stage1/2	147.90 min ± 9.37	135.22 min ± 8.88	12.68 min ± 1.79	37.00 ± 6.40*
SWS	101.75 min ± 4.03	98.02 min ± 4.19	3.73 min ± 1.50	6.41 ± 0.80
REMs	86.35 min ± 6.34	77.00 min ± 5.81	9.35 min ± 1.40	19.91 ± 1.56
TST	336.00 min ± 16.41	310.24 min ± 14.23	25.76 min ± 3.59	63.00 ± 7.64**
Wake	36.49 min ± 5.04	22.01 min ± 3.88	14.48 min ± 3.36	-
TS/TW	369.17 min ± 15.02	330.25 min ± 14.26	38.92 min ± 5.90	-

Note: All variables were calculated within participant, and then averaged across participants. - = data were not processed for this variable. Wake = total time scored as wake within the adjusted stage bout intervals. TS/TW = total minutes scored as sleep plus total minutes scored as wake with the adjusted stage bout intervals. Correlations were calculated among PSG identified arousals the discrepancies between ACT and PSG within each adjusted stage bout stage.

^*= p < .05,

^**= p < .01

Table 5.

Sensitivity, specificity, and accuracy of ACT when compared to PSG: Microanalyses

Sleep Stage:	Sensitivity	Specificity	Accuracy
Stage1/2	91.24% (79.55% - 97.99%)	-	-
SWS	96.29% (73.13% - 100.00%)	-	-
REMs	88.95% (75.42% - 97.88%)	-	-
TST	92.36% (79.40% - 97.72%)	-	-
Wake	-	58.85% (0% - 100%)	-
TS/TW	-	-	89.61% (65.37% - 97.72%)

Note: See table 4 for specific data processing information. Values in parentheses indicate range for particular variable. - = data were not processed for this variable.

The difference between ACT-TST and PSG-TST, within individual participants, was significantly associated with their corresponding frequency of PSG-identified arousals (r = .45, p < .05), and this applied to the difference between ACT-TST and PSGTST within S1/2s (r = .59, p < .01). Nevertheless, the differences between ACT-TST and PSG-TST within SWS (r = −.04, p = .85) and within REMs were not significantly associated with PSG-identified arousals (r = .35, p = .12).

DISCUSSION

ACT signals were first processed using the WTV-40 (default) parameter setting. These calculations revealed that when compared to PSG, ACT underestimated TST by 72.25 minutes and overestimated WASO by 13.85 minutes. In an attempt to improve the agreement between PSG and ACT, these analyses were recalculated with a “high” WTV-80 (standard option) setting for ACT. Despite shifting the ACT values in the correct direction, ACT still underestimated TST by 52.05 minutes and WASO by 6.35 minutes.

ACT was examined more precisely using micro (epoch-by-epoch) analyses. ACT had high sensitivity within sleep stages and adequate accuracy. Although the specificity of ACT was poor, it was higher than a previous report using an adult cohort.⁸ Overall, ACT identified sleep relatively well, but was unable to discriminate wake from sleep. The sensitivity, specificity, and accuracy results must be interpreted with caution because our use of the adjusted stage bout only allowed stage bouts of greater than four consecutive minutes to be examined. Therefore, the 4 minutes surrounding transitions, as well as transient stages (importantly, wake) were not included in these analyses. As reviewed by Sadeh and Acebo, ACT and PSG are most discrepant during transitions to and from sleep.³ We speculate that the current profile of results would indicate worse ACT sensitivity, specificity, and accuracy if the entire night was entered into these calculations.

Further examination revealed that ACT became increasingly worse in the identification of TST as arousal frequency increased. Thus, ACT tends to identify epochs that contain arousals as wake, even though the encompassing epoch is scored as sleep by PSG. Overall, ACT systematically underestimated TST and overestimated WASO. Our results are consistent with previous findings among a pediatric sample.²¹

The detection of movement during sleep is a relatively well known general confounder of ACT.^2,3 For example, the ability of ACT to identify periodic limb movements in children was previously examined and revealed that ACT overestimates movement indices compared to electromyography.²² Additionally, Sitnick and colleagues utilized videosomnography to examine the utility of ACT among sleep disordered preschoolers and concluded that ACT underestimated TST and overestimated WASO.²¹ These authors emphasized important considerations when using ACT in children and, most notably, the importance of selecting the appropriate sleep monitoring tool for the question(s) being addressed.

In an adult sample (with a similar design to the current report) ACT overestimated sleep and underestimated intermittent awakenings.⁸ The contradiction between the findings in adults and those reported herein may be due to developmental differences in movements during sleep and the inability of ACT to accurately detect movements.

Contrary to previous reports on accelerometer devices, including the AW-64,^4-7 ACT does not appear to be an appropriate instrument to identify sleep and wake among infants. Unlike the current report, no previous studies used the Bland-Altman concordance technique to examine the agreement between any brand of ACT and PSG in infants.

Despite our use of appropriate methodological and analytical techniques, along with our conservative analytical approach, our research is not without limitations. Even though the sample consisted of healthy infants with normal PSG sleep and respiratory measures, we exclusively studied 14-month-old infants. The generalizability of our findings to other ages is unknown. Additionally, since computer times were synchronized to within one minute of each other, they may have been misaligned by up to 59 seconds. This potential discrepancy prohibited examination of the agreement between PSG and ACT during transitions between stages as well as during arousals from sleep. Future research efforts should more precisely examine the ability of ACT to correctly identify sleep and wake during stage transitions, as well as the ability of ACT to discriminate arousals during sleep. These issues were expected since ACT in adults has several disadvantages, namely the relative inability to identify sleep and wake during periods of high motility during sleep, as well as during periods of wakefulness without motion,³ both of which could also include sleep-wake transitions. Of note, the current study used one specific brand of ACT (AW-64, Mitter Co. Inc.) and scoring algorithm (Actiware, Mitter Co. Inc.). Other ACT brands should be examined with similar rigor to confirm whether our findings are specific to the AW-64 or to accelerometers in general.

In conclusion, the AW-64 ACT device (using two parameter settings) does not appear to be a suitable instrument for accurate and reliable assessment of sleep and nocturnal wake in infants. This problem, which might be applicable to other accelerometry-based devices, should be solved by age-dependent adjustments of the manufacturer settings and by improvements in the algorithm that allow for calculation of sleep and wake values. Thus, absolute values derived from infant sleep and wake measurements using ACT should be interpreted with caution in both clinical and research settings.