Comparing actigraphy and diary to measure daily and average sleep in firefighters: a Bland–Altman analysis

Abstract

Study Objectives:

This study sought to examine the relationship between actigraphy and the Consensus Sleep Diary to contribute information on their concurrent validity in a sample of career firefighters.

Methods:

Sixty firefighters were recruited from a large, urban fire department in the southwest United States that utilizes a fire-based emergency medical services system and a 5/6 shift schedule. A total of 329 differences were recorded during participants’ 6-day between-shift recovery period. Data was collected utilizing the two most common forms of sleep analysis in an outpatient setting, wrist actigraphy (Actiwatch-2) and the Consensus Sleep Diary. Nine major sleep indices were computed: wake time after sleep onset, total sleep time, sleep onset latency, sleep offset, in-bed time, lights-off time, out-of-bed time, wake time, and sleep efficiency.

Results:

Firefighters overestimated sleep efficiency and underestimated wake after sleep onset by values that were greater than the American Academy of Sleep Medicine a priori clinical significance thresholds. All indices showed very broad limits of agreement. For example, the 95% confidence interval for diary and actigraphic total sleep time estimates fell within a 4.7-hour range.

Conclusions:

Firefighters receiving recovery sleep between tours demonstrated significantly large disagreements between their daily self-reported sleep and measured actigraphic sleep. Sleep findings from actigraphic and Consensus Sleep Diary sleep assessments in this population should be interpreted cautiously until each method is compared against other reliable sleep analysis methods. Currently it is unclear if clinicians are using properly validated tools when diagnosing shift work disorder or other sleep disorders in firefighters.

Citation:

Marmis R, McGoldrick-Ruth L, Kelly MR, Haynes PL. Comparing actigraphy and diary to measure daily and average sleep in firefighters: a Bland–Altman analysis. J Clin Sleep Med. 2024;20(4):497–503.

BRIEF SUMMARY

Current Knowledge/Study Rationale: The study was conducted because outpatient sleep assessments required for the diagnosis of shift work disorder have not been validated in the fire service population. Prior research is limited that compares the Consensus Sleep Diary to actigraphy in firefighters.

Study Impact: Across all sleep indices and both timeframes (daily, summary), there was significant variability between subjective and objective assessments of recovery sleep in firefighters. Clinicians should be informed that the validity of actigraphy and/or sleep diaries is questionnable in firefighters.

INTRODUCTION

More than one-third of firefighters screen positive for a sleep disorder with shift work and insomnia disorder being among the most prevalent.¹ In many fire departments, firefighters work 24-hour shifts.² Although firefighters are shift workers, they are also on-call workers; they are frequently awakened to respond to calls and may have trouble returning to sleep or not receiving sufficient sleep as a result of the working environment,³ arousal from the call,⁴ or the unpredictability of another call.⁵ These consequences of being on-call may partially explain why insomnia is also highly prevalent in firefighters.¹ Sleep disruption is a significant exposure of working in the fire service.

Because firefighters are at high risk for shift work disorder and insomnia disorder as a result of their work, as well as having difficulty achieving recovery sleep,^6,7 it is important to have well-validated and reliable tools for assessing sleep. Since it is common practice for firefighters to receive disrupted sleep on shift, it is crucial to their health and performance that they achieve proper recovery sleep between shifts and that sleep can be accurately monitored by clinicians.^8–10 Current diagnostic criteria for shift work disorder require both actigraphy monitoring and sleep log for at least 14 days to determine if a disturbed sleep and wake pattern exists.¹¹

This criterion requires an objective actigraphic and subjective recorded review of sleep. The portability and ease of actigraphy, with a single button used to indicate intention to sleep, increases its feasibility as an objective tool for measuring firefighter sleep patterns. In the general population, actigraphy, when compared to the gold standard objective sleep measure, polysomnography, shows reliable estimation of total sleep time (TST), sleep percentage, and wake after sleep onset (WASO), although the ability of actigraphy to assess sleep-onset latency (SOL) is less accurate.¹² The gold standard measurement for subjective sleep in patients with insomnia is the Consensus Sleep Diary (CSD).^13,14 The self-reported night-to-night measurement of CSD allows for the accurate assessment of indices such as SOL, which actigraphy struggles to capture in a noninvasive manner. The CSD is widely used across multiple different populations and offers the ability to record wakefulness during times of low movement, unlike actigraphy. The CSD was designed with the purpose of analyzing insomnia in clinical and research applications. Although the CSD provides a comprehensive, self-reported assessment of sleep, it is largely acknowledged as not necessarily aligning with objective sleep measurement.¹⁴

Results from various studies indicate that populations with insomnia or shift work tend to show disagreement on the CSD vs actigraphy compared to those with normal work hours or without insomnia.^15,16 Findings from a recent meta-analysis demonstrated that individuals with insomnia disorder tend to underestimate their sleep by approximately 37 minutes on CSD as compared to actigraphy (n = 40 studies). On the other hand, individuals with insufficient sleep, who were largely recruited from occupations involving extended shifts, were found to overestimate sleep on diaries by about 39 minutes compared to actigraphy (n = 10 studies).¹⁷ Fewer studies have examined the concordance between CSD and actigraphy in individuals with shift work disorder specifically. However, the limited evidence available suggests poor concurrent validity for sleep onset and offset times, which are often used in the calculation of TST; this may be one reason these measures do not correspond.¹⁷

To our knowledge, only one study has compared the validity of various sleep indices in firefighters. Billings¹⁸ examined 24 firefighters over 18 days and 6 shift tours and found that firefighters overestimated their total sleep time on a novel sleep diary developed for emergency service workers and other sleep questionnaires compared to actigraphy. Despite this, the difference between sleep diary and actigraphy was not statistically significant, potentially due to high levels of variability. Billings reported that one participant overestimated their sleep by an average of 37.8 minutes on the sleep diary compared to actigraphy.¹⁸

This overestimation of sleep assessment in firefighters is consistent with prior work in other on-call or shift work occupations exposed to sleep insufficiency. Helicopter Emergency Medical Services pilots have a 24-hour shift schedule where sleep is permitted but frequently interrupted. In a study examining their sleep habits, Helicopter Emergency Medical Services pilots overestimated their TST on the sleep diary compared to that calculated on the actigraph.¹⁹ Astronauts on space shuttle missions²⁰ and medical interns and residents²¹ have also been shown to overestimate TST on sleep diaries compared to the actigraphy. A different study in internal medicine residents demonstrated the same pattern of self-reported overestimation of TST during recovery sleep only; the same pattern was not evident on nights when residents were on-call.²²

To date, most studies examining firefighter sleep utilize sleep indices aggregated over at least 5 days actigraphy and typically 7 days of sleep diary or other global sleep measures.^18,23 While descriptive differences between mean subjective and objective methods of sleep offer insight into sources of assessment bias, research with sleep data scored across different timescales that involve different work/sleep patterns across shift tours is also necessary. An examination of daily variation can contribute to a greater understanding of within-participant factors associated with self-reported vs objective sleep assessments. This is especially relevant for firefighters whose tour schedules are untraditional and often erratic.

The purpose of this study is to evaluate differences in self-reported and objective sleep between nights over a recovery sleep period in career firefighters. The recovery period was assessed because sleep during this period may be more amenable to intervention (since firefighters are not responding to nighttime calls). Using the clinically significant thresholds determined by the American Academy of Sleep Medicine (AASM) expert consensus panel,¹⁷ this project assesses concurrent validity between the two measures across two timescales: day and recovery period.

METHODS

Data were sourced from a parent study conducted at the University of Arizona,⁷ approved by the local Human Subjects Protection Program. All participants gave verbal informed consent prior to telephone screening, and all in-person participants gave written informed consent.

Participants

Participants were recruited from a large, urban fire department in the southwest region of the United States that utilizes a fire-based emergency medical services system. This fire department responded to more than 92,000 emergency calls during fiscal year 2017,²⁴ with the majority being medical calls. At the time of data collection (September 2016–June 2017), the average call volume per day was 17.28 for the assessed fire department. Sixty-one participants were enrolled and completed the study. One participant was excluded due to incomplete data. Participants were recruited with presentations and flyers given at fire stations and through an agency occupational health care provider. To be enrolled, firefighters gave their information and agreed to be contacted by the study staff or contacted the researchers directly.

Inclusion criteria was previously reported.⁷ In brief, participants were 18 to 60 years old and were required to work on a fire crew as a firefighter, captain, engineer, or paramedic. All participants worked a 5/6 shift tour (ie, five 24-hour shifts with a 24-hour off-shift period between shifts followed by six consecutive days off [“recovery period”] before beginning another tour). All participants were screened in an in-person clinical interview for sleep disturbances resulting from a medication or physical etiology. Exclusion criteria consisted of issues that would negatively affect study participants or interfere with a participant’s ability to participate in the study. For instance, participants were excluded if they had untreated mental health disorders; uncontrolled serious medical conditions; frequent use of hypnotic, sedative, or anxiolytic medication; or sleep disorders resulting from a physical etiology (untreated sleep apnea or restless legs syndrome). Individuals who engaged in work outside of the 5/6 shift tour were not excluded due to the high rate of participants picking up additional work shifts from their colleagues or working second jobs. Days in which a participant worked an additional shift during their 6-day recovery period were excluded from the data. A total of 14 participants worked one additional shift during a recovery period. No participant worked more than one additional shift. Twenty-four participants reported that they worked a second job, all of which reported their second job occurred exclusively during the day (ie, no other shift work).

Measures

Sleep diary

Participants’ self-reported assessment of sleep was collected with the CSD completed the subsequent morning following the measured night. The CSD is a valid and reliable method of sleep assessment in healthy individuals and in individuals with sleep disorders.¹³ The CSD data were used to score the indices of: WASO, TST, SOL, sleep offset, in-bed time, lights-off time, out-of-bed time, wake time, and sleep efficiency (SE). In-bed time was calculated as the time that an individual retired to bed and out-of-bed time was when they left the bed. Wake after sleep onset was the amount of time an individual was awake after initially falling asleep. TST was calculated as the time between lights being turned off to sleep and final awakening or time in bed subtracting SOL and WASO. Sleep offset was calculated as the time that an individual had their final awakening. Sleep efficiency was calculated as TST divided by time in bed multiplied by 100. Sleep was compared both on a daily basis and on an average over the 6 recovery days. Both SE and TST were averaged over the 6-day recovery period between shifts. Subjects completed the CSD an average of 5.48 days (standard deviation = .72 days).

Actigraphy

To collect the objective sleep indices, participants wore a wrist-based actigraph (Actiwatch-2; Philips Respironics, Bend, OR) on their nondominant wrist. The Actiwatch-2 utilizes a solid-state Piezo-electric accelerometer with a sensitivity of 0.025G to measure motor activity and a light sensor with a wavelength range of 400 to 900 nm to measure photopic illuminance (Lux). Data were scored based on bedtimes and waketimes determined through the event marker, a button that participants were instructed to press when retiring to and arising from bed. Data from the actigraph were scored using the Actiware Sleep software package, version 6.0, which estimates rest intervals and sleep periods utilizing a proprietary algorithm. The same sleep indices were scored as reported under the CSD. One actigraph malfunctioned resulting in the loss of data from one participant. Data were analyzed across an average of 5.71 days (standard deviation = 0.49 days).

Screening, covariates

Participants were screened via demographic questionnaires and semistructured interviews for medical and mental health problems likely to interfere with study participation. Fire service rank was assessed as part of a structured interview on occupational history. As reported in the parent study,⁷ the Duke Sleep Inventory²⁵ was administered to screen for untreated obstructive sleep apnea, untreated restless legs syndrome, and regular use of hypnotic or anxiolytic medications (Table 1).

Table 1.

Demographics of study participants (n = 60).

	n	%
Sex
Male	58	96.7
Female	2	3.3
Race
White	47	78.3
Asian	1	1.7
Other	6	10
More than one race	6	10
Ethnicity
Hispanic or Latino origin	17	28.3
Not Hispanic or Latino origin	43	71.7
Rank
Firefighter	25	41.7
Engineer	11	18.3
Paramedic	9	15
Fire management	15	25
Education
Graduated high school or high school equivalent such as GED	26	43.3
Graduated 2-year college (associate’s degree)	10	16.7
Graduated 4-year college (bachelor’s degree) or higher degree	24	40

Statistical analyses

Mixed effects limits of agreement were computed using mixed linear modeling with the nlme package²⁶ in R, consistent with methods delineated by Parker and colleagues.²⁷ The model predicting paired differences (daily sleep diary – actigraphy) in sleep indices contained day of recovery period as a fixed effect and participant as a random effect. Because the dataset was imbalanced with regard to the number of measurements, the mean bias was fit using a separate model with only the constant term and random effect for participant. The estimated between-participant and within-participant variance were summed to create a total variance for all differences.

Next, these analyses were repeated using the standard Bland–Altman analysis of variance procedure collapsing sleep data by week (minimum 3 days). This estimate allowed a comparison of mean sleep scores, as is typically reported in the sleep science literature.

To determine whether CSD and actigraphy provided distinct data, two sets of clinical significance thresholds (CSTs) were applied to the interpretation of mean bias and limits of agreement (LoA) for each sleep index. The first set of CSTs were determined using the largest adult CSTs set by the AASM¹⁷ and applied to the sleep in this population to assess the allowable mean difference between the CSD and actigraphy. If the AASM did not specify a CST (in-bed time, out-of-bed time, wake time, sleep offset, lights-off time), then a presumptive CST was adopted based on similar indices (ie, 20-min mean difference for sleep times based on TST).

The second set of CSTs were defined as the maximum allowable 95% confidence interval for the mean difference (or LoA). Since no current mean difference CST exist between actigraphy and the CSD, the CST was adopted from the AASM comparison between actigraphy and polysomnography.

Given the high likelihood of equipment error, scoring error, or recall bias, all difference scores were checked for outliers. Outliers were identified using the 1.5 interquartile range rule (ie, less than the lower quartile score + 1.5*interquartile range or greater than the upper quartile score + 1.5*interquartile range).²⁸ If outliers were found, they were removed, and results were reported both with and without outliers.

RESULTS

Data from a total of 60 participants were examined producing between 329 to 330 (range) valid measurements (observations) across all participants.

Bland–Altman plots demonstrate the paired differences in measurements against the average (see Figure 1) with mixed effects LoA.

Figure 1. Bland–Altman Plots for computed sleep indices showing the paired differences against the average for the two measurements (daily sleep diary and actigraphy).

Mean bias is shown by solid line. CSTs for the maximum allowable mean are presented in gray shading. Means (solid) outside the shaded CST represent bias. LoA are shown by dashed lines. CSTs for the maximum allowable LoA (95% confidence interval) are shown by dotted lines. LoA (dashed) outside of CSTs (dotted) indicate significant variability between the two devices. CST = clinically significant thresholds, LoA = limits of agreement.

Table 2 shows the numerical values of LoA based on the (1) mixed effects model using all data and (2) fixed effects model (averaged over mean = 5.49, mean standard deviation = 0.72 days).

Table 2.

LoA for each of the daily sleep indices by day (mixed effects) and averaged over multiple days (fixed effects).

Sleep Index	Mean Bias (Mixed Effects 95% LoA)	No. Diffs	Mean Bias (Fixed Effects 95% LoA)	n with ≥ 5 Diffs (%)	CST M Bias	CST LoA
Sleep onset latency, min	−0.82 (−53.06 to 51.43)	330	−0.50 (−33.34 to 32.35)	54 (90%)	±20	40
WASO, min	−32.36 (−121.77 to 57.05)	329	−32.29 (−95.30 to 30.71)	54 (90%)	±15	30
Total sleep time, min	−19.69 (−158.97 to 119.59)	329	−19.97 (90.86 to 50.93)	54 (90%)	±20	40
Sleep efficiency, %	9.69 (−12.21 to 31.59)	329	9.61 (−5.20 to 24.42)	54 (90%)	±2.5	5
Sleep offset, min	−4.43 (−60.45 to 51.58)	330	−4.44 (−38.65 to 29.78)	54 (90%)	±20	40
In-bed time, min	−12.03 (−103.20 to 79.14)	330	−12.21 (−63.15 to 38.72)	54 (90%)	±20	40
Lights-off time, min	−8.94 (−100.83 to 82.95)	330	−8.94 (−55.35 to 37.48)	54 (90%)	±20	40
Out-of-bed time, min	23.47 (−48.01 to 94.95)	330	23.82 (−16.12 to 63.77)	54 (90%)	±20	40
Wake time, min	−12.69 (−84.53 to 59.15)	330	−12.64 (−54.29 to 29.02)	54 (90%)	±20	40

n = 60. Bolded items are outside the CST, indicating discrepant information. Mixed effects LoA is calculated using day of recovery period as a fixed effect and participant as a random effect. Fixed effects LoA are calculated at participant-level using 1.96*standard deviation of difference. CST = clinically significant threshold set by the American Academy of Sleep Medicine (presumptive thresholds are italicized), CST M bias = allowable mean difference between sleep log and actigraphy, CST LoA = maximum allowable 95% confidence interval for LoA (scores were taken from the American Academy of Sleep Medicine comparison of actigraphy to polysomnography since the organization has not yet provided a recommended LoA between sleep log and actigraphy), diffs = number of daily differences computed as daily sleep diary score – actigraphy score, LoA = limits of agreement, mean bias = average difference, WASO = wake time after sleep onset.

Similar results for mean bias were observed between the mixed effects and fixed effects models. Mean bias was in the a priori allowable CST range set by the AASM for SOL, TST, sleep offset, in-bed time, lights-off time, and wake time. Mean bias was outside the CST range for WASO, SE, and out-of-bed time. There was clinically significant bias toward underreporting WASO on CSD relative to actigraphy and overreporting SE on CSD relative to actigraphy. Also, individuals reported arising substantially later on the CSD than what was evaluated on actigraphy.

The LoA were outside the CST for all computed indices and all sleep times in both models. The LoA were narrower for sleep indices summarized over multiple days as compared to daily scores. However, significant variability between the CSD and actigraphy remained regardless of sleep assessment period, suggesting that the two forms of assessment provide distinct information and cannot be used interchangeably in firefighters.

Table 3 shows the within-participant, between-participant, and total (combined) standard deviation of the differences for the mixed effects model. As seen in the table, within-participant variability comprises a substantial portion of the total variability between assessment methods, suggesting that day-level factors (within-participant) may play a more significant role in assessment discrepancies than individual person-level (between-participant) factors.

Table 3.

Comparison of the variabilities of differences between the sleep diary and actigraphy across sleep indices.

Index	Mixed Effects Model
	WP SD	BP SD	Combined SD	Max – Diff	Max + Diff
Sleep onset latency, min	23.72	12.16	26.66	−139.00	178.50
Wake time after sleep onset, min	35.69	28.41	45.62	−399.00	109.50
Total sleep time, min	67.73	21.5	71.06	−268.00	337.00
Sleep efficiency, %	9.17	6.39	11.18	−26.30	74.67
Sleep offset, min	24.82	14.17	28.58	−102.00	140.00
In-bed time, min	43.19	17.28	46.52	−385.00	197.50
Lights-off time, min	44.87	13.56	46.88	−389.00	199.50
Out-of-bed time, min	33.5	14.42	36.47	−176.00	182.00
Wake time, min	33.74	14.32	36.65	−191.00	177.50

Combined standard deviation represents the square root of the sum of the within-participant and between-participant variances. Difference is computed as daily sleep diary score – actigraphy score. BP = between participant, Max – Diff = maximum negative difference, Max + Diff = maximum positive difference, SD = standard deviation, WP = within participant.

There were no substantive differences in mean bias between mixed effects models that included and excluded outliers (see Table S1 ^{(204.4KB, pdf)} in the supplemental material).

With the exclusion of outliers in the fixed effects model, the mean bias for TST was also outside the CST range, indicating an underreporting of TST on the CSD relative to actigraphy. The LoA in the fixed effects model were within the CST LoA range for SOL, sleep offset, and wake time. With removal of the outliers, up to 37% of participants in the sample had less than 5 days of daily differences, raising concerns about the stability and generalizability of results with outliers removed (see Table S1 ^{(204.4KB, pdf)} ).

DISCUSSION

The current study examined nine major sleep indices during recovery days in firefighters by analyzing daily differences between actigraphy and sleep diary methods of assessment. Consistent with a previous study examining sleep in on-call workers,²² minimal bias was observed in TST during recovery sleep. However, there was substantial mean bias showing an overestimation of SE and underestimation of WASO. These results are in the opposite direction of what has been reported in patients with insomnia,¹⁷ suggesting that sleep insufficiency may be driving a bias toward positive sleep estimations.

Across all indices, observed firefighters showed a large amount of variability between their daily self-reported sleep and measured actigraphic sleep, raising questions about the validity of each assessment approach. Regardless of time frame, the LoA for all indices were much wider than the clinical LoA recommended by the AASM between actigraphy and polysomnography.¹⁷ There is currently no clinical threshold for the LoA between actigraphy and CSD set by the AASM. Despite the slight superiority for the standard practice of averaging daily values in terms of LoA, these data suggest that there is a large discrepancy between actigraphy and sleep diary in firefighters. This discrepancy raises concerning questions about the validity of the measures when used alone over the course of 1 day or when averaged over the course of multiple nonworking days.

Because diagnostic criteria for shift work disorder require utilization of actigraphy and sleep diary, future research validating these tools should be a high priority in fire service workers. While polysomnography is often considered a gold-standard for comparison, it is rarely used in-home across multiple days and can alter sleep, which raises questions about clinical relevancy or generalizability. Instead, other sleep assessment methods should be assessed and could include contactless bed sensors,²⁹ video-based monitoring,³⁰ ballistocardiogram,³¹ or other methods that offer an alternative comparison to actigraphy and CSD.

Further research is also recommended to understand the cause of the discrepancy between objective and self-reported methods of sleep assessment. Results from this study indicate that day-level variables explain the majority of disagreements for all indices. Day-level variables are those that vary each day within person, such as mood, attention, energy, emotion, sleep perception, diet, exercise, and workload. In contrast, individual-level variables are those that vary between individuals, such as age, sex, social status, or burnout. Most covariates in sleep and health research are at the individual level, indicating a need for future research that incorporates time-varying covariates.

Despite the critical role of fire service response to community health, most studies examining their sleep have done so using global measures of sleep quality¹⁸ that are frequently linked to health outcomes. This project was significant for its investigation of daily variation between actigraphy and the CSD in firefighters. A major strength of the current study is the use of mixed effect LoA, which highlights the importance of within-participant sources of variability. The primary limitation is that neither actigraphy nor sleep diary can currently be verified as an accurate reflection of sleep without further administration of polysomnography in this population. In addition, generalizability may be limited for other fire departments with different call volumes or fire departments that are not combined fire service and emergency medical service. Both factors could cause significant differences in the frequency of on-shift sleep disruptions. Our findings may not generalize to volunteer fire departments or those departments not following a 24-on/24-off shift schedule. Our study is strengthened by our relatively large number of data points (n = 330) and examination of differences both by day and by averaging across multiple days.

The 24-hour-on/24-hour-off shift schedule with frequent interruptions makes firefighters unique even among other shift workers in terms of sleep health. Accurate assessment of recovery sleep is necessary to shed light on whether firefighters are adequately rested prior to another tour. Research focusing on fire service worker sleep habits is key to educating those providing care for this unique population of public safety professionals.

DISCLOSURE STATEMENT

All authors have seen and approved this manuscript. Institution where work was performed: University of Arizona. This work was supported by the University of Arizona Graduate and Professional Student Council Research and Project grant. Ryan Marmis has no conflicts of interest to disclose. Logan McGoldrick-Ruth has no conflicts of interest to disclose. Monica Kelly has no conflicts of interest to disclose. Dr. Kelly is supported by National Institutes of Health/National Heart, Lung and Blood Institute K23 HL157754. Dr. Kelly is no longer affiliated with the University of Arizona and is currently affiliated with the David Geffen School of Medicine at the University of California, Los Angeles and the VA Greater Los Angeles Healthcare System Geriatric Research, Education and Clinical Center, North Hills, California. Patricia Haynes has no conflicts of interests to disclose. Dr. Haynes is supported in part by a professional services agreement with the city of Tucson.

ABBREVIATIONS

AASM

American Academy of Sleep Medicine

CSD

Consensus Sleep Diary

CST

clinical significance threshold

LoA

limits of agreement

SE

sleep efficiency

SOL

sleep onset latency

TST

total sleep time

WASO

wake after sleep onset