Agreement between Actigraphy and Sleep Diaries: A 28-Day Real-Time Monitoring Study Among Suicidal Adolescents Following Acute Psychiatric Care

Abstract

Objective:

To examine the agreement between, and adherence to, wrist actigraphy and digital sleep diaries as methods for sleep assessment among high-risk adolescents in the 28 days following discharge from acute psychiatric care. Sleep parameters included: number of nighttime awakenings (NWAK), sleep efficiency (SE), sleep onset latency (SOL), total sleep time (TST), and wake after sleep onset (WASO).

Methods:

Fifty-three adolescents (12–18 years) were recruited following discharge from acute psychiatric care for suicide risk. Adolescents completed a baseline assessment followed by a 28-day monitoring period with daily sleep diaries and continuous wrist actigraphy. Bland-Altman and multi-level models examined agreement.

Results:

Adherence to actigraphy was high, but lower for sleep diaries; a similar pattern of adherence emerged on weekdays vs. weekends. Bland-Altman analyses revealed no clinically meaningful bias for sleep parameters (except NWAK), but the limits of agreement make interpretation ambiguous. Our base model indicated strong agreement between actigraphy and sleep diaries for TST (r = .850), moderate for SOL (r = .325) and SE (r = .322), and weak for WASO (r = −.049) and NWAK (r = .114). A similar pattern emerged with the insomnia severity models with baseline insomnia influencing agreement on all parameters. There were significant weekday-weekend differences for WASO and NWAK, but not for SOL, SE, and TST.

Conclusion:

Results suggest that it may be beneficial to find a modeling approach to account for the concordant and discordant information and relevant time-level variables.

Adolescent sleep is characterized by shifts in timing and depth, resulting in a change in the amount and type of sleep obtained [1–3]. There is not only a developmental trend of later sleep onset and reduced sleep time with increasing age from childhood to adolescence [4], but also a decrease in biological sleep depth [5]. Insufficient sleep has a negative impact on mood [6] and cognitive performance [7] and has been proposed as a mechanism in the onset, maintenance, and relapse of psychiatric disorders [8]. As research continues, questions remain about sleep measurement among youth with psychiatric disorders.

Polysomnography (PSG) is considered the gold-standard measure of sleep. However, PSG is impractical for non-laboratory studies due to the specialized equipment and staffing and their related costs. Although commercial devices (e.g., FitBits) have become increasingly popular, they have shown varied agreement with PSG among youth so the most common research methods are sleep diaries and actigraphy [9, 10]. Each have their own strengths and limitations. Sleep diaries are daily assessments in which individuals self-report on sleep parameters from the previous night [11]. They are a low-cost method that can be recorded on paper or digitally to provide information on any number of nights of sleep; they also allow for the collection of subjective aspects of sleep (e.g., sleep quality, rumination before bed) that cannot be obtained from objective methods like actigraphy (described next). Limitations include participant burden and declining adherence over long monitoring periods with paper (21 days) [12] and digital sleep diaries (16 weeks) [13]. They have shown reporter bias (over and underestimation depending on the sleep parameter) when compared to PSG and actigraphy on specific sleep parameters, such as total sleep time (TST; i.e., actual time slept) among youth [14, 15] and sleep onset latency (SOL; i.e., how many minutes it takes to fall asleep, starting from when one intends to fall asleep) and number of awakenings (NWAK; i.e., number of awakenings during sleep, excluding final awakening) among psychiatric adults [16].¹ Actigraphy provides a measure of sleep-wake patterns based upon movement captured by an actigraph, a noninvasive device commonly worn on an individual’s wrist. Actigraphy provides parameter estimates that are not influenced by recall bias and places little burden on the individual due to passive data collection with accelerometer technology. Limitations include its high cost (>$1,000), inability to collect information on subjective sleep phenomena (e.g., nightmares), challenges in distinguishing between motionless wakefulness (e.g., adolescent viewing their phone in bed) and sleep [17], and specialized software needed to extract the data and expertise to interpret it. Given the growing use of sleep diaries and actigraphy among youth with psychiatric disorders, it is necessary to understand where these methods overlap and where they do not to inform future assessment and treatment research.

Although sleep diary-actigraphy agreement research can guide method selection, it has been limited among youth with psychiatric disorders which is surprising given the rapid rise in sleep research among this population and the link between sleep and clinical outcomes (e.g., suicidal thoughts) [3, 18]. Existing youth studies have focused on TST agreement among non-psychiatric youth and showed a discrepancy of, on average, an hour with each method overestimating TST, depending on the study [14, 15, 19–21]. To date, no studies have directly examined sleep diary-actigraphy agreement among youth with psychiatric and sleep disorders. Instead, they have examined sleep parameters across diagnoses (e.g., with and without attention-deficit hyperactivity disorder) and not method agreement specifically [22, 23], leaving the field to turn to agreement research among psychiatric adults for methodological agreement guidance. In adults with major depression, wake after sleep onset (WASO; i.e., total amount of time awake during the night, excluding SOL and terminal wakefulness) was overestimated by sleep diaries and TST was underestimated by sleep diaries [24]. In adults with bipolar disorder, sleep diaries overestimated SOL, TST, and WASO compared to actigraphy [16]. Although this agreement research provides useful information, it requires the field to potentially misapply results from non-psychiatric youth or psychiatric adults to the unique population of youth with psychiatric disorders—a group experiencing significant developmental shifts in sleep and psychiatric symptoms [25]. In addition, agreement studies have almost exclusively used an aggregated approach (i.e., averaging across a follow-up period) [26] with varying degrees of agreement [15, 19–21, 27]. This is useful in understanding average trends, but results in temporally sensitive data being examined in a temporally insensitive way. Taken together, it is vital that fine-grained measurement-focused work is completed to clarify method agreement among youth with psychiatric disorders.

The present study extends prior research by examining adherence to, and agreement between, daily sleep diaries and actigraphy among youth with psychiatric disorders in the 28 days following discharge from acute psychiatric care. First, we compared adherence rates for sleep diaries and actigraphy (across the 28-day monitoring period, weekdays vs. weekends). Based on prior research with non-psychiatric adults [13], we hypothesized that adherence would be higher for actigraphy compared to sleep diaries. In exploratory analyses, we examined differences in adherence based on adolescent factors (i.e., age, depressive symptoms, insomnia, sleep quality). Second, we examined agreement between sleep diaries and actigraphy using an aggregated approach and a more temporally sensitive approach. Based on non-psychiatric youth and psychiatric adults [15, 19], we hypothesized low agreement for SOL, SE, TST, and WASO, but that TST would emerge with the strongest agreement. Given actigraphy’s overestimation of wake bouts [28], we hypothesized low agreement on NWAK. In exploratory analyses, we examined the influence of baseline insomnia symptoms on agreement and if there were differences between weekdays (i.e., Sunday-Thursday; consistent with Alfano et al.’s [29] definition of weekdays-weekends in their study of anxious and non-anxious children) and weekends (i.e., Friday-Saturday) on agreement. Based on Arora et al. [19] (non-psychiatric adolescents) and youth’s weekday-to-weekend sleep variability [30], we expected some weekday-weekend influence on agreement, notably with actigraphy overestimating TST on the weekends.

Methods

Participants

Data was collected as part of a larger study examining sleep problems as a short-term risk factor for suicide among adolescents in the 28 days following discharge from acute psychiatric care (e.g., inpatient psychiatric hospitalization) for a suicide crisis (e.g., suicide ideation with intent). See Glenn et al. [31] for a detailed study methodology and safety monitoring and Glenn et al. [18] for how sleep is related to suicidal thinking.

Adolescents, aged 12–18 years, were approached for screening and inclusion if they were transitioning from acute psychiatric care to outpatient treatment at the University’s medical center and if a parent/guardian was willing to participate. Adolescents were excluded for the following reasons: unable to provide informed assent/consent due to cognitive impairment or psychosis, unwilling to participate (e.g., unwilling to wear actigraphy watch), were a safety risk (e.g., necessitated readmission to acute psychiatric care), or if a sibling was enrolled in the study.

The full sample included 53 adolescents (M_age = 14.8 years, SD = 1.6). See Table 1 for sample characteristics. This sample also had several psychiatric comorbidities and can be considered a proxy for severe psychiatric populations.

Table 1.

Major demographic and clinical characteristics of the sample.

	Adolescents (N = 53)
Major Demographics
Age (years): M (SD)	14.8 (1.6)
Gender Identity: % (n)
Female	64% (34)
Male	16% (9)
Nonbinary1	18% (10)
Race and Ethnicity: % (n)
White	77% (41)
Black	7% (4)
American Indian or Alaskan Native	1% (1)
Multi-racial	9% (4)
Hispanic/Latinx	11% (5)
Sexual Orientation: % (n)
Heterosexual	43% (23)
Gay or Lesbian	5% (3)
Bisexual	32% (17)
Pansexual	5% (3)
Asexual	3% (2)
Unsure	9% (5)
Clinical Characteristics
Major Psychiatric Disorders2: % (n)
Anxiety Disorder	88% (47)
Attention-Deficit hyperactivity disorder	26% (14)
Bipolar Disorder	5% (3)
Disruptive Behavior Disorder	24% (13)
Eating Disorder	17% (9)
Major Depressive Disorder	79% (42)
Obsessive Compulsive Disorder	9% (5)
Posttraumatic Stress Disorder	17% (9)
Psychotic symptoms	5% (3)
Substance Use Disorder	7% (4)
Baseline depression severity (BDI-Y T score): M (SD)	70.1 (14.8)
Suicidal Thoughts and Behaviors
Lifetime suicide attempt: % (n)	83% (44)
Past-year suicide attempt: % (n)	75% (40)
Past-month active suicide ideation: % (n)	92% (49)
Sleep Characteristics
Insomnia (ISI total score): M(SD)	13.4 (5.1)
Clinical insomnia (ISI score ≥ 15): % (n)	45% (24)
Sleep Quality (PSQI total score): M (SD)	11.1 (3.8)
Poor sleep quality (PSQI score ≥ 5): % (n)	84% (45)

Note. BDI-Y=Beck Depression Inventory for Youth; DDNSI=Disturbing Dreams and Nightmares Severity Index; ISI=Insomnia Severity Index; PSQI=Pittsburgh Sleep Quality Index.

¹Nonbinary includes adolescents identifying as transgender, nonbinary, or agender.

²Current diagnoses were determined by integration of the adolescent and parent reports (obtained separately).

Anxiety disorder includes any of the following current disorders: panic disorder, agoraphobia, social anxiety disorder, specific phobia, or generalized anxiety disorder; Attention-deficit hyperactivity disorder includes any of the following current subtypes: inattentive only, hyperactive/impulsive only, or combined; Bipolar disorder includes current bipolar I or II disorder; Disruptive behavior disorder includes current conduct disorder or oppositional defiant disorder; Eating disorder includes current anorexia nervosa or bulimia nervosa; Substance use disorder Journal Pre-proof includes current alcohol use disorder or substance (drug) use disorder. Given time constraints, not all disorder modules were administered to all participants, resulting in missing data

Procedure

Study procedures were approved by the University’s Institutional Review Board. Prior to study initiation, adolescent assent and parent/guardian permission (12–17-year-old) or adolescent consent (18-year-old) was obtained. The baseline assessment occurred within two weeks of discharge from acute care (M = 8.8 days, SD = 3.87). The assessment included clinical interviews, self-reports, and orientation to the smartphone-based EMA application and actigraphy watch; each adolescent and parent/guardian were compensated $25/hour. The 28-day monitoring period included daily sleep diaries using their smartphones and wearing the actigraphy watch continuously (unless showering/bathing). Adolescents without a smartphone were loaned an Android phone with a 30-day prepaid data plan. Each week, adolescents were compensated with a $25 Amazon e-gift card if they completed at least 75% of EMA surveys.

Measures

Baseline Assessment

Baseline sleep problems were assessed to characterize the sample and baseline insomnia was used in analyses. The Insomnia Severity Index assessed insomnia [32]. The Pittsburg Sleep Quality Index assessed sleep quality [33]. The Beck Depression Inventory for Youth assessed depressive symptoms [34]. The Columbia-Suicide Severity Rating Scale [35] and Mini International Neuropsychiatric Interview for Children and Adolescents, Child and Parent Versions [36], clinically characterized the sample.

28-Day Monitoring Period

Five sleep parameters were assessed using sleep diaries and actigraphy: NWAK, sleep efficiency (SE; i.e., percent of time in bed spent asleep), SOL, TST, and WASO.

EMA Sleep Diaries.

Adolescents completed a morning sleep diary to assess the previous night’s sleep, consistent with sleep assessment recommendations [37] and items from the Consensus Sleep Diary [11]. Adolescents were instructed to complete the sleep diary upon waking; EMA software provided a time stamp for submission. Surveys were completed on adolescents’ smartphones using HIPAA-compliant software designed for EMA research.² Surveys were completed within two hours of waking and the EMA schedule was set to each participant’s waketime. See Supplemental Table 1 for sleep diary questions.

Actigraphy.

Adolescents’ sleep-wake patterns were assessed continuously with the Actiwatch Spectrum Plus (sampling rate: 32Hz; epochs: 15s). This is a lightweight, waterproof watch-like device worn on the participant’s nondominant wrist and has been used in prior adolescent research [38].

Data Analytic Plan

Prior to analyses, sleep diary and actigraphy data were manually inspected and cleaned (see Supplement for details). Actigraphy data was manually examined to detect potential outliers (e.g., extremely long sleep intervals) or artifacts then were examined using the Philips Actiware software which generated the sleep-wake statistics used in the study. We conducted analyses in R Studio. Regarding outliers, data points +2.5 SD from the mean of each sleep parameter were excluded; 187 data points (3.1% of data) were removed.

Adherence

EMA Sleep Diaries.

Consistent with Thurman et al. [13], sleep diary adherence was examined in two ways: (1) proportion of days in which the adolescent successfully submitted sleep diaries out of the total days they were enrolled in the study, with a maximum of 28 days (adherence rate); and (2) how soon after prompting was the daily sleep diary submitted (time delay: average delay across all days for which the participant was enrolled in the study, up to two hours after prompting). To understand individual differences, we used Pearson’s correlations to examine the relationship between adherence and adolescent factors (i.e., age, depressive symptoms, insomnia, sleep quality). We used the psych package [39].

Actigraphy.

Adherence was defined as the adolescent wearing the actigraphy watch during the sleep period. We used Pearson’s correlations to characterize the relation between adherence and adolescent factors.

Agreement

Agreement was determined in two ways. First, we used the Bland-Altman approach for repeated measures which assesses average agreement and individual agreement [26]. Bland-Altman plots depict the agreement by plotting the difference between the two measures for each person-day against their average, as the true value of the sleep parameter is unknown [40]. We set the following a priori interpretation guidelines, informed by prior work [41], for mean bias and limits of agreement (LoA): ±30 min for TST, ±15 min for SOL, and ±15 min for WASO. For SE and NWAK, we set two ranges to provide conservative and extended guidelines (±5% and ±10% for SE, ±0.5SD and ±1SD of the actigraphy mean for NWAK). Repeated measures concordance correlation coefficients are provided [42]. We used the SimplyAgree package [43]. Second, we used multilevel modeling (MLM) as it is capable of addressing the nested structure of the data [44] and allows us to examine these measures over time. We used bestNormalize [45], DHARMa [46], lme [47], lmerTest [48], and glmmTMB [49] packages. Due to high correlations among adolescent factors (i.e., insomnia, sleep quality, and depressive symptoms), only baseline insomnia was included.

Results

Of the 53 adolescents, 51 had at least one day of matching sleep diary-actigraphy data. See Table 2 for descriptive statistics. See Supplemental Table 2 for circadian variables (i.e., midpoint of sleep and social jetlag) by method.

Table 2.

Descriptive statistics measured by actigraphy and EMA sleep diaries over 28-day monitoring period.

		Descriptive Statistics				Repeated Measures Correlation
	Actigraphy M (SD)	EMA Sleep Diaries M (SD)	Mean Difference	P	95% CI	Concordance Correlation Coefficient	95% CI
Overall Monitoring Period
SE: %	84.53 (9.83)	85.99 (16.22)	−1.46	.017	−2.67, .26	.19	.11, .27
SOL: min	23.52 (22.38)	36.17 (41.90)	−10.97	< .001	−14.33, −7.61	.01	−.05, .09
TST: min	458.26 (101.01)	456.87 (132.46)	4.14	.303	−3.75, 12.03	.51	.47, .56
WASO: min	20.87 (10.21)	13.91 (25.73)	6.88	< .001	4.97, 8.78	−.03	−.11, .05
NWAK: n	41.43 (16.35)	0.90 (1.11)	40.61	< .001	39.43, 41.78	.01	−.02, .02
Weekdays (i.e., Sunday-Thursday)
SE: %	84.41 (9.91)	85.83 (16.25)	−1.44	.042	−2.84, −.04	.22	.12, .32
SOL: min	23.91 (22.86)	37.32 (42.13)	−11.71	< .001	−15.68, −7.73	.02	−.06, .11
TST: min	454.43 (99.40)	447.52 (131.05)	9.28	.045	.17, 18.39	.56	.51, .61
WASO: min	20.48 (9.98)	14.57 (26.44)	6.01	< .001	3.71, 8.31	.11	.02, .19
NWAK: n	40.72 (16.06)	0.88 (1.10)	39.85	< .001	38.50, 41.21	−.01	−.02, .02
Weekends (i.e., Friday-Saturday)
SE: %	84.87 (9.61)	86.42 (16.18)	−1.53	.208	−3.93,.86	.12	.01, .22
SOL: min	22.89 (21.71)	33.09 (41.22)	−9.35	.004	−15.71, −3.01	.05	−.08, .19
TST: min	469.59 (105.85)	481.38 (133.29)	−9.53	.231	−25.18, 6.11	.55	.41, .62
WASO: min	22.07 (10.91)	12.16 (23.68)	9.47	< .001	6.08, 12.86	.16	.01, .31
NWAK: n	43.30 (16.99)	0.96 (1.09)	42.58	< .001	40.25, 44.92	.01	−.04, .05

Note. NWAK = number of nighttime awakenings; SE = sleep efficiency; SOL = sleep onset latency; TST = total sleep time; WASO = wake after sleep onset.

Adherence

EMA Sleep Diaries

Of days actively enrolled in the study,³ adherence was 67.4% (SD = 23.9%). For weekday-weekend adherence, weekday adherence was 66.5% (SD = 27.1%) and weekend adherence was 64.5% (SD = 28.2%). Although sleep diaries took 2 minutes (SD = 5 minutes) to complete (i.e., from survey start time to submission), the time delay was 35 minutes (SD = 33 minutes) from initial prompt.

Actigraphy

Of days actively enrolled,³ the adherence was 88.1% (SD = 21.3%). For weekday-weekend adherence, weekday adherence was 85.1% (SD = 24.2%) and weekend adherence was 88.1% (SD = 22.1%).

Adherence and Individual Differences

For the days actively enrolled, there were weak, nonsignificant correlations between actigraphy adherence and age (r = .001), depressive symptoms (r = .118), and sleep quality (r = .234). There was a medium significant correlation adherence and baseline insomnia severity (r = .346, p = .023), indicating that adolescents with higher baseline insomnia tended to have better adherence. For the days actively enrolled, there were weak, nonsignificant correlations between sleep diary adherence and age (r = .055), depression (r = .030), and insomnia (r = .201). There was a small, but significant correlation between adherence and baseline sleep quality (r = .293, p = .041), indicating that adolescents with greater sleep quality dysfunction tended to have better adherence.

Agreement

Bland-Altman

Bland-Altman plots are depicted in Figure 1; see supplemental figure for weekday-weekend plots.

Fig. 1.

Bland-Altman plots comparing actigraphy and EMA sleep diaries (overall monitoring period).

Note. For each sleep parameter plot, the x-axis is the mean of actigraphy and EMA diaries. The y-axis is the average difference between actigraphy and EMA diaries. The dashed line is the reference ine which is set zero and represents perfect agreement between methods. The solid line is the bias (or mean of the difference between methods). The shaded area around the bias is the 95% CI. The shaded area at the top is the upper limit of agreement (LoA; mean difference ± 1.96 SD) with the 95% CI. The shaded area at the bottom is the lower LoA (mean difference ± 1.96 SD) with the 95% CI. In this figure, Number of Awakenings does not have a reference line.

¹ a) Sleep Onset, b) Sleep Efficiency, c) Total Sleep time, d) Wake After Sleep Onset, e) Number of Awakenings

For SOL, the mean bias was −7.79, 95% CI [−15.87, .29], suggesting the sleep diary overestimated the parameter compared to actigraphy. The 95% CI indicated no fixed bias between measures, meaning that neither sleep diaries or actigraphy were consistently giving higher or lower values across the range of measurement. The Lower LoA [LLoA] was −64.36, 95% CI [−74.79, −55.31] and Upper LoA [ULoA] was 48.78, 95% CI [3.73, 59.21]. The mean bias was not large enough to be clinically meaningful based upon our a priori guidelines, however, LoAs were wide, exceeding maximum allowable differences.

For SE, the mean bias was −3.11, 95% CI [−5.92, −.29], suggesting the sleep diary overestimated the parameter. The 95% CI indicated no fixed bias. The LLoA was −26.04, 95% CI [−29.53, −22.94] and ULoA was 19.82, 95% CI [16.73, 23.31]. The mean bias was not clinically meaningful but exceeded maximum allowable differences.

For TST, the mean bias was −10.32, 95% CI [−29.75, 9.10], suggesting the sleep diary overestimated the parameter. The 95% CI indicated no fixed bias. The LLoA was −204.41, 95% CI [−227.60, −183.22] and ULoA was 183.74, 95% CI [162.56, 206.94]. The mean bias was not clinically meaningful but exceeded maximum allowable differences.

For WASO, the mean bias was 9.01, 95% CI [5.24, 12.75], suggesting that sleep diaries underestimated the parameter. The 95% CI indicated no fixed bias. The LLoA was −24.97, 95% CI [−29.57, −20.82] and ULoA was 42.97, 95% CI [38.82, 47.52]. The mean bias was not clinically meaningful but exceeded maximum allowable differences.

For NWAK, the mean bias was 40.74, 95% CI [38.02, 43.47], suggesting that sleep diaries underestimated the parameter. The 95% CI indicated fixed bias between measures with actigraphy providing consistently higher values across the range of measurement. The LLoA was 19.19, 95% CI [15.75, 22.24] and ULoA was 62.29, 95% CI [59.25, 65.74]. The mean bias was large enough to be clinically meaningful and LoAs exceeded maximum allowable differences.

Multilevel Models

For each sleep parameter, we used spaghetti plots to examine the shape of the raw data and completed residual diagnostics. Results for the base models (weekday/weekend) are in Table 3 and results for the insomnia severity models (weekday/weekend and baseline insomnia) are in Table 4.

Table 3.

Agreement over time between actigraphy and EMA sleep diaries for each sleep parameter: base models.

Sleep Onset Latency
Fixed Effects	Estimate	SE	Statistic	P
Actigraphy Intercept	3.31	.096	34.44	< .001
EMA Sleep Diary Intercept	3.22	.136	23.68	< .001
Study Time Intercept	−.001	.004	−.245	.805
Weekend Intercept	−.018	.077	−.233	.815
EMA Sleep Diary*Study Time	−.005	.005	−.979	.327
EMA Sleep Diary*Weekend	−.079	.107	−.741	.458
Actigraphy Residual Mean	−2.35	.154	−15.29	< .001
EMA Sleep Diary Residual Mean	−19.48	1145.24	−.017	.986
Random Effects
Actigraphy σ2	.508
EMA Sleep Diaries σ2	.854
Actigraphy-EMA Sleep Diaries Correlation	.325
Sleep Efficiency
Fixed Effects	Estimate	SE	Statistic (df)	P
Actigraphy Intercept	−.060	.082	−.727 (94.25)	.468
EMA Sleep Diary Intercept	−.029	.116	−.256 (68.79)	.798
Study Time Intercept	.001	.003	.178 (1374.75)	.858
Weekend Intercept	.016	.066	.244 (1360.06)	.807
EMA Sleep Diary*Study Time	.002	.005	.457 (1378.58)	.647
EMA Sleep Diary*Weekend	.018	.095	.198 (1357.05)	.842
Random Effects
Actigraphy σ2	.429
EMA Sleep Diaries σ2		.721
Observation σ2	.782
Actigraphy-EMA Sleep Diaries Correlation	.322
Total Sleep Time
Fixed Effects	Estimate	SE	Statistic (df)	P
Actigraphy Intercept	463.44	8.57	54.04 (152.88)	< .001
EMA Sleep Diary Intercept	466.91	11.20	41.66 (88.36)	< .001
Study Time Intercept	−.253	.454	−.557 (1341.26)	.577
Weekend Intercept	11.15	8.23	1.35 (1375.68)	.176
EMA Sleep Diary*Study Time	.194	.647	.299 (1327.10)	.764
EMA Sleep Diary*Weekend	14.08	11.71	1.20 (1377.66)	.229
Random Effects
Actigraphy c2	36.44
EMA Sleep Diaries σ2	62.09
Observation σ2	97.51
Actigraphy-EMA Sleep Diaries Correlation	.850
Wake After Sleep Onset
Fixed Effects	Estimate	SE	Statistic	P
Actigraphy Intercept	3.08	.050	61.05	< .001
EMA Sleep Diary Intercept	2.81	.189	14.83	< .001
Study Time Intercept	−.002	.002	−.861	.389
Weekend Intercept	.060	.050	1.19	.233
EMA Sleep Diary*Study Time	−.002	.005	−.452	.650
EMA Sleep Diary*Weekend	−.245	.095	−2.56	.010
Actigraphy Residual Mean	−20.31	1391.07	−.014	.988
EMA Sleep Diary Residual Mean	.008	.083	.101	.919
Random Effects
Actigraphy σ2	.190
EMA Sleep Diaries σ2	1.19
Actigraphy-EMA Sleep Diaries Correlation	−.049
Number of Awakenings
Fixed Effects	Estimate	SE	Statistic	P
Actigraphy Intercept	3.72	.039	95.04	< .001
EMA Sleep Diary Intercept	.160	.164	.978	.327
Study Time Intercept	−.001	.001	−.885	.375
Weekend Intercept	.067	.031	2.12	.033
EMA Sleep Diary*Study Time	−.038	.006	−6.11	< .001
EMA Sleep Diary*Weekend	−.126	.103	−1.22	.219
Actigraphy Residual Mean	−6.58	1.00	−6.57	< .001
EMA Sleep Diary Residual Mean	−3.30	.700	−4.72	< .001
Random Effects
Actigraphy σ2	.201
EMA Sleep Diaries σ2	.983
Actigraphy-EMA Sleep Diaries Correlation	.114

Table 4.

Agreement over time between actigraphy and EMA sleep diaries for each sleep parameter: insomnia severity models.

Sleep Onset Latency
Fixed Effects	Estimate	SE	Statistic	P
Actigraphy Intercept	3.11	.226	13.77	< .001
EMA Sleep Diary Intercept	2.31	.349	6.61	< .001
Study Time Intercept	−.001	.004	−.247	.804
Weekend Intercept	−.021	.077	−.280	.778
Baseline Insomnia Intercept	.016	.015	1.07	.282
EMA Sleep Diary*Study Time	−.007	.006	−1.29	.194
EMA Sleep Diary*Weekend	−.048	.108	−.443	.657
EMA Sleep Diary*Baseline Insomnia	.052	.024	2.15	.030
Actigraphy Residual Mean	−2.48	.161	−15.36	< .001
EMA Sleep Diary Residual Mean	−19.43	1095.58	−.017	.985
Random Effects
Actigraphy σ2	.467
EMA Sleep Diaries σ2	.811
Actigraphy-EMA Sleep Diaries Correlation	.363
Sleep Efficiency
Fixed Effects	Estimate	SE	Statistic (df)	P
Actigraphy Intercept	.317	.197	1.60 (50.57)	.113
EMA Sleep Diary Intercept	.884	.292	3.02 (54.58)	.003
Study Time Intercept	.001	.003	.372 (1310.61)	.709
Weekend Intercept	.025	.068	.375 (1300.47)	.707
Baseline Insomnia Intercept	−.029	.013	−2.23 (45.05)	.030
EMA Sleep Diary*Study Time	.003	.005	.551 (1315.68)	.581
EMA Sleep Diary*Weekend	−.030	.098	−.313 (1296.47)	.754
EMA Sleep Diary*Baseline Insomnia	−.038	.022	−1.75 (50.94)	.085
Random Effects
Actigraphy σ2	.402
EMA Sleep Diaries σ2	.673
Observation σ2	.786
Actigraphy-EMA Sleep Diaries Correlation	.227
Total Sleep Time
Fixed Effects	Estimate	SE	Statistic (df)	P
Actigraphy Intercept	512.61	18.84	27.20 (62.16)	< .001
EMA Sleep Diary Intercept	548.73	27.19	20.17 (58.25)	< .001
Study Time Intercept	−.139	.469	−.296 (1274.54)	.767
Weekend Intercept	9.89	8.55	1.15 (1315.51)	.247
Baseline Insomnia Intercept	−3.60	1.24	−2.90 (50.20)	.005
EMA Sleep Diary*Study Time	.082	.669	.123 (1267.08)	.901
EMA Sleep Diary*Weekend	11.21	12.15	.922 (1317.23)	.356
EMA Sleep Diary*Baseline Insomnia	−2.42	1.52	−1.58 (51.10)	.118
Random Effects
Actigraphy σ2	33.24
EMA Sleep Diaries σ2	58.02
Observation σ2	98.59
Actigraphy-EMA Sleep Diaries Correlation	.837
Wake After Sleep Onset
Fixed Effects	Estimate	SE	Statistic	P
Actigraphy Intercept	3.07	.115	26.61	< .001
EMA Sleep Diary Intercept	1.20	.687	1.75	.078
Study Time Intercept	−.001	.002	−.609	.542
Weekend Intercept	.060	.051	1.15	.246
Baseline Insomnia Intercept	.001	.007	.037	.970
EMA Sleep Diary*Study Time	−.001	.005	−.332	.739
EMA Sleep Diary*Weekend	−.210	.097	−2.15	.031
EMA Sleep Diary*Baseline Insomnia	.113	.042	2.68	.007
Actigraphy Residual Mean	−19.22	855.94	−.022	.982
EMA Sleep Diary Residual Mean	−.032	.101	−.314	.753
Random Effects
Actigraphy σ2	.197
EMA Sleep Diaries σ2	1.05
Actigraphy-EMA Sleep Diaries Correlation	.028
Number of Awakenings
Fixed Effects	Estimate	SE	Statistic	P
Actigraphy Intercept	3.83	.098	38.93	< .001
EMA Sleep Diary Intercept	−.762	.411	−1.85	.063
Study Time Intercept	−.001	.001	−.631	.527
Weekend Intercept	.065	.032	2.02	.042
Baseline Insomnia Intercept	−.007	.006	−1.18	.237
EMA Sleep Diary*Study Time	−.036	.006	−5.78	< .001
EMA Sleep Diary*Weekend	−.043	.104	−.416	.677
EMA Sleep Diary*Baseline Insomnia	.070	.026	2.62	.008
Actigraphy Residual Mean	−6.53	1.00	−6.53	< .001
EMA Sleep Diary Residual Mean	−3.67	.922	−3.98	< .001
Random Effects
Actigraphy σ2	.205
EMA Sleep Diaries σ2	.876
Actigraphy-EMA Sleep Diaries Correlation	.353

Base Models.

For SOL, we ran a zero-inflated model to account for the shape of the sleep diary data. Agreement between methods was moderate (r = .325). For SE, we ran a linear model with restricted maximum likelihood (REML). Agreement between methods was moderate (r = .322). For TST, we ran a linear model with REML. Agreement between methods was very strong (r = .850). For SOL, SE, and TST, there was no significant trend over study time (i.e., from one study data point to the next) and there was no significant difference between weekdays and weekends. For WASO, we ran a zero-inflated model to account for the shape of the sleep diary data. Agreement between methods was weak (r = −.049). There was a significant sleep method * weekend interaction, suggesting that sleep diaries estimated significantly lower WASO on weekends. For NWAK, we ran a zero-inflated model to account for the shape of the actigraphy data. Agreement between methods was weak (r = .114), demonstrating little agreement. The weekend intercept was significant, suggesting that NWAK was significantly higher on weekends across methods. There was a significant sleep method * study time interaction, suggesting that sleep diaries estimated significantly lower NWAK over time.

Insomnia Severity Models.

For SOL, we ran a zero-inflated model. Agreement between methods was moderate (r = .363). The sleep method * baseline insomnia interaction was significant which indicated that greater an adolescent’s baseline insomnia, the greater an adolescent’s SOL was; this association was stronger for sleep diaries. For SE, we ran a linear model with REML. Agreement between methods was low-moderate (r = .227). Baseline insomnia was significant which indicated the greater an adolescent’s baseline insomnia, the lower an adolescent’s SE was. For TST, we ran a linear model with REML. Agreement between methods was very strong (r = .837). Baseline insomnia was significant, suggesting that the greater an adolescent’s baseline insomnia, the lower an adolescent’s TST; this was equally strong across methods. For WASO, we ran a zero-inflated model. Agreement between methods was very weak (r = .028). There was a significant sleep method * weekend interaction, suggesting that sleep diaries estimated significantly lower WASO on weekends. There was a significant sleep method * baseline insomnia interaction which indicated that a greater an adolescent’s baseline insomnia, the greater an adolescent’s WASO; this was equally strong across methods. For NWAK, we ran a log transformation model. Agreement between methods was moderate (r = .353). The weekend intercept was significant, suggesting that NWAK was higher on weekends across methods. There was a significant sleep method * study time interaction, suggesting that there was a difference in the methods in the linear trend over time; this association was stronger with sleep diaries. Additionally, there was a significant baseline insomnia * sleep method interaction, indicating that the greater an adolescent’s baseline insomnia, the higher an adolescent’s NWAK; this association was stronger with sleep diaries.

Discussion

Our study revealed four main findings. First, adherence for actigraphy was high whereas adherence for sleep diaries was moderate. For both methods, adherence was similar on weekdays vs. weekends. Second, Bland-Altman results were ambiguous despite the bias between methods not being clinically meaningful (except NWAK), suggesting a more nuanced analytical method may be needed. Third, our base models indicated very strong agreement for TST, moderate agreement for SOL and SE, and weak agreement for WASO and NWAK. A similar pattern emerged with our insomnia severity models with baseline insomnia explaining some of the association between methods. Fourth, across MLMs, sleep diaries estimated significantly lower WASO on the weekends and NWAK was significantly higher on the weekends across methods. There were no significant weekday-weekend differences for SOL, SE, and TST.

Our adherence results are consistent with prior work [13] and have implications for studies with longer follow-up windows (>2 weeks). Even with notifications and reminders for our sleep diaries, adherence was only moderate, suggesting there may be aspects contributing to adherence (e.g., sleeping through alerts) that are worthwhile to investigate. In addition, we had a restricted time window (2 hours) for sleep diary completion. This may have contributed to our agreement results as filling out a sleep diary close to waking may improve reporting accuracy [11] and may be beneficial for EMA studies to consider.

Although the statistical approaches produced somewhat similar agreement results, the conclusions we can draw from the Bland-Altman analyses are ambiguous. For all sleep parameters, LoAs and visual inspection indicated that the methods are not equivalent through the range of measurement. In one of the only non-psychiatric adolescent sleep diary-actigraphy studies to report LoAs, Arora et al. [19] listed smaller LoAs (−14.15–180.09) for TST compared to ours (−204.41–183.74). Adolescents in our study had a wider range of differences in TST which is consistent with sleep variability among youth with psychiatric disorders [50]. Given that TST is a parameter that relies on SOL and WASO in its calculation, there may be a similar pattern with other parameters and suggests that Bland-Altman may not be appropriate to assess agreement among youth with psychiatric disorders. Despite these caveats, we see a pattern where sleep diaries are suggesting that sleep quantity is better than it may truly be given the underestimation of WASO and NWAK.

Our MLM results align with most non-psychiatric youth agreement work. For TST, our results align with Arora et al. [19] and Lucas-Thompson et al. [20], demonstrating strong and consistent agreement between methods across younger (11–13 years old; Arora et al. [19]) and older (14–21 years old; Lucas-Thompson et al. [20]) adolescents without a sleep or psychiatric disorder. There is some discrepancy on TST with Short et al. [15] who found low agreement with sleep diaries overestimating TST (by 90 minutes) in adolescents aged 13–18. WASO results are consistent with Short et al. [15] where agreement was low and actigraphy WASO estimates were higher than sleep diaries. Regarding weekday-weekend influence, surprisingly, only WASO and NWAK, were affected. Baseline insomnia explained some of the association between methods, increasing or decreasing the agreement depending on the parameter. Given that 45% of our sample met the cut-off for clinical insomnia, this carry through to the monitoring period was unsurprising.

Methodological Implications

Our results indicate simultaneous use of both methods may be warranted, depending on the sleep parameter of interest and method adherence. Overall, actigraphy is recommended for youth with psychiatric disorders to obtain the most sleep data, although adding diaries is recommended to increase data quality. Researchers are encouraged to consider the benefits of simultaneously using both in combination (as is the approach in clinical sleep research) versus one method alone. If only one method can be selected, the following recommendations are made when moderate adherence is expected. For TST, results suggest that the methods are tapping into the same sleep construct so either sleep diaries or actigraphy are appropriate. For SOL, either sleep diaries or actigraphy are appropriate. Regarding aspects of wakefulness, a sleep diary is more appropriate for WASO and NWAK. Relatedly, it may be beneficial to find a modeling approach that combines both methods to account for the concordant and discordant information along with relevant person- (e.g., insomnia) and time-level (e.g., weekdays/weekends) variables. This can leverage one method’s strength to address another’s weakness while including relevant variables. Thurman et al. [13] suggested including weighted averages of the methods along with a built-in bias for measurements then evaluating these combined estimates against PSG.

Limitations and Future Directions

Our study’s three main limitations suggest directions for future research. First, there is no ground truth about which sleep measure is “best” in this study as neither were compared to PSG. Since a 28-day lab-based PSG study for youth with psychiatric disorders is not feasible, a portable PSG, which has demonstrated feasibility in non-psychiatric youth [51], may be promising for future longitudinal studies. Second, the sample was a small, clinically high-risk group during a high-risk transition time so our findings may not generalize to youth who are less clinically severe or during times of more stability. Research could extend this work by examining agreement in youth receiving outpatient psychiatric services and replicate findings in a larger sample. Third, and finally, we did not assess contributing factors such as sleep medication or electronic usage before sleep. These may be important to examine given the upward trend of prescribing sleep medications for youth [52] and use of electronics before bed [53].

Conclusions

Agreement results suggest that sleep methods may provide different information about sleep disturbance among psychiatric youth. Sleep diaries may be the most appropriate when assessing aspects of wakefulness (e.g., NWAK) whereas actigraphy may be most appropriate for SOL. These factors, along with adherence, should be carefully considered when designing studies to measure sleep among youth with psychiatric disorders.

Highlights.

• Adherence to actigraphy was high, but lower for sleep diaries.

• Actigraphy and sleep diary agreement was strongest for total sleep time.

• Actigraphy and sleep diary agreement was weakest for wake after sleep onset.

• Insomnia severity influenced agreement between actigraphy and sleep diaries.