Reliability of Actigraphy and Subjective Sleep Measurements in Adults: The Design of Sleep Assessments

Abstract

Study Objectives:

The aim of the study was to investigate how many nights of measurement are needed for a reliable measure of sleep in a working population including adult women and men.

Methods:

In all, 54 individuals participated in the study. Sleep was assessed for 7 consecutive nights using actigraphy as an objective measure, and the Karolinska sleep diary for a subjective measure of quality. Using intra-class correlation and the Spearman-Brown formula, calculations of how many nights of measurements were required for a reliable measure were performed. Differences in reliability according to whether or not weekend measurements were included were investigated. Further, the correlation between objectively (actigraphy) measured sleep and subjectively measured sleep quality was studied over the different days of the week.

Results/Conclusions:

The results concerning actigraphy sleep measures suggest that data from at least 2 nights are to be recommended when assessing sleep percent and at least 5 nights when assessing sleep efficiency. For actigraphy-measured total sleep time, more than 7 nights are needed. At least 6 nights of measurements are required for a reliable measure of self-reported sleep. Fewer nights (days) are required if measurements include only week nights. Overall, there was a low correlation between the investigated actigraphy sleep parameters and subjective sleep quality, suggesting that the two methods of measurement capture different dimensions of sleep.

Citation:

Aili K, Åström-Paulsson S, Stoetzer U, Svartengren M, Hillert L. Reliability of actigraphy and subjective sleep measurements in adults: the design of sleep assessments. J Clin Sleep Med. 2017;13(1):39–47.

INTRODUCTION

The most common cause for sickness cash benefit in Sweden is psychological ill health.¹ The occupational health services (OHS) have a challenging task as special advisers to prevent psychological ill health and stress-related ill health in the work place, providing interventions targeting workplace stress. There is a need for reliable methods to evaluate stress and stress-related health effects in conjunction with these interventions. Measurements of sleep and sleep disturbances may be one option, indicating a (non-beneficial) stress response in a global meaning.

Sleep disturbances are often regarded as secondary to—or symptoms of—stress and psychological ill health. Self-rated stress and self-rated sleep are closely related,² and sleep disturbances have been shown to be associated with the work stress indicator demand/control/support model^3,4 and to be a mediator in the association between relational justice at work and ill health.⁵ Sleeping problems have further been shown to be predictors of sickness absence, multisite pain, acute myocardial infarction, and mortality.^6–10 Good or restorative sleep has, on the other hand, been shown to predict the resolution of multisite pain⁸ and chronic widespread pain.¹¹ Sleep may thus tell us something about an individual's increased risk of ill health or unfavorable prognosis, and could be a feasible marker of stress globally, when, for example, estimating the health effects of interventions.

BRIEF SUMMARY

Current Knowledge/Study Rationale: Previous research indicates that sleep may be a feasible marker to use when evaluating interventions targeting stress-related ill health at workplaces. The knowledge of how many nights of actigraphy measured sleep that is required for a reliable measure of sleep is however scarce.

Study Impact: This study contributes with methodologically new knowledge of the reliability of repeated measurements of sleep. Knowledge of which, is helpful when planning and performing sleep measurements in both research and clinical settings.

There are different approaches to measure sleep. One common way to assess how individuals typically sleep is to use questionnaires referring to sleep disturbances in general over past time periods. If sleep is to be measured when evaluating interventions, it may be of importance to be able to detect alterations in sleep over shorter periods of time, in which case sleep diaries or objective methods, e.g., polysomnography or actigraphy would be better choices. In order to capture (small) changes in sleep quality over time, the method used for assessment must be valid (measure what it claims to measure) and reliable (be consistent when a measurement is repeated). Polysomnography is the gold standard method for objective sleep measurements. However, assessing sleep using actigraphy, has the advantage of being relatively cheap and easy to use, which may make the method more realistic to use when assessing sleep on larger groups.

There are several studies that have investigated the validity of sleep measurements with an actigraph.¹² In a review from 1995, approved by the American Sleep Disorder Association, it was concluded that epoch-by-epoch sleep estimated by actigraphic measures in laboratory studies was strongly associated with polysomnographic measures, with a correlation coefficient of at least 0.85 in healthy individuals.¹³

Research investigating the reliability of actigraphy sleep measures (or, more precisely, how many nights of measurement should be aggregated for a reliable sleep measurement to be obtained) is, however, scarce. Among children and adolescents, it has been found that, in general (across age groups), aggregation of five or more nights of measurement is needed to obtain a reliable measure.¹⁴ Among women, it has been found that three or more nights of measurements are needed for a reliable assessment of sleep efficiency measured with an actigraph.¹⁵ A previous study investigating sleep using an actigraph in adults at two time points approximately one year apart found that intra-individual variation in sleep duration was larger on a day-to-day than on a year-to year basis, and that 5–6 nights of measurement were sufficient for a reliable measure of sleep duration.¹⁶

To the best of our knowledge, the issue of how many nights of measurement with the Karolinska Sleep diary required for a reliable measurement of sleep has not been investigated before. There is thus a need for more knowledge in order to optimize the design of studies including sleep assessments.

The aim of the study is to investigate how many nights of measurement are needed to obtain a reliable measurement of sleep in a working population, and to investigate if including weekends in period of measurements affect the reliability of the measurements. The measurements, objectively with an actigraph and subjectively through self-reports in a sleep diary, have been taken in conjunction with the start-up of an intervention carried out by a Swedish Occupational Health Service, thus in a “real-life” setting.

METHODS

Participants

Participants were recruited as part of an intervention study called SHIP (Study of a Health Intervention Programme) targeting personnel at a correctional institution in Sweden. In total, 259 employees at the institution were invited to participate in the SHIP study in November 2012. The SHIP study was launched in connection with the start-up of an intervention focusing on promoting health among staff, carried out by an OHS provider. The aim of SHIP was to evaluate the potential use of different indicators related to possible health effects. These indicators should preferably function as estimates of the change due to health promoting interventions.

Of the individuals who participated in SHIP, 60 (men and women) were approached for the current sub-study, which involved taking additional sleep measurements with an actigraph and a sleep diary for 7 consecutive nights. The invited individuals were chosen to represent 3 different areas of the workplace, based on the proportions of their daily work that involved contact with prison inmates. The underlying assumption was that these 3 groups would face different degrees of stress in their everyday work situations. In addition, the participants included for additional sleep measurements were not to work night shifts during the period of measurement in conjunction with the days/nights of the measurements. Of the 60 individuals invited, 58 agreed to participate. Eleven of the participants were working during the weekend (5 women and 6 men). For one individual, we had no information on work during the weekend.

All participants in the study gave their informed consent. The study was approved by the Regional Ethical Review Board in Stockholm (Dnr 2013/677-31).

Study Group Characteristics

Lower self-reported sleep efficiency has previously been shown to be associated with over-commitment, low social support, depressive symptoms, lower ratings on happiness, and self-reported health, whereas objective measures of sleep have shown no such associations.¹⁹ In order to describe the study group, information on the following items is presented:

Self-rated health, assessed by the EQ5D thermometer, where the participants rated their current health status from 0 (worst imaginable health) to 100 (best imaginable health).

Three dimensions of fatigue, assessed using the Shirom Melamed Burnout Measurement (SMBM),²⁰ which comprises: (1) physical fatigue (2) cognitive weariness, and (3) emotional exhaustion. The responses to each question were on a 7-point scale (from 1 = never or almost never, to 7 = always or almost always). A mean value for each dimension was calculated.

Sleep disturbances, assessed through an index based on 4 items from the Karolinska Sleep Questionnaire, using a 6-item Likert scale.^21,22 The responses to each question were scaled from 1 to 6, where a higher score indicates worse sleep.

Anxiety and depression, assessed on the Hospital Anxiety and Depression Scale (HAD), which includes 14 items, each with 4 response categories (scores 0–3, where a low score indicates low depression/anxiety). One index for anxiety (scale 0–21) and one for depression (scale 0–21) were formed, as is appropriate with the HAD questionnaire.²³

Data Collection

Measurement by actigraph

A wrist-worn actigraph, Motionwatch 8 (Camntech Ltd), was used for the sleep measurements. The Motionwatch actigraph has a tri-axial accelerometer using MEMs technology, capable of sensing motions in a resultant force range of 0.01 g to 8 g. It registers total gross motor activity for analysis in a software program for sleep-wake analysis. The Motionwatch 8 is a relatively new device and has shown to be valid when tested against polysomnography²⁴ and against other actigraph devices.²⁵ Measurements, from both actigraph and sleep diary, were performed for 7 consecutive nights at some point between the 4th of February and the 6th of March. The first day (night) of measurement was either a Monday (n = 29), a Tuesday (n = 2) or a Wednesday (n = 25). Participants were instructed to wear the watch on their non-dominant wrist, day and night for 7 consecutive days. Recordings were taken in 1-min epochs.

A software program, Actiwatch Activity & Sleep Analysis CamNtech Ltd version 7.38, was used for sleep-wake analysis. A medium sensitivity level was used for estimation of sleep-wake patterns. An epoch is considered as wake if movement of ≥ 40 “activity counts” is registered during the epoch. Time to bed was estimated by using data from the sleep diary and entered into the sleep-analysis software. The software program then calculated when sleep started for the night and when sleep ended in the morning. The software uses an algorithm for determination of sleep start by searching for a period of ≥ 10 min of consecutively recorded immobile epochs (activity count < 40), with no more than one epoch of movement within that time. The sleep end is determined by the program by looking backwards from the last sample in the analysis window, and is set by the last period of ≥ 6 min of “sleep” epochs, with no more than 2 “awake” epochs in between. To estimate sleep onset by identifying the first period of several consecutive minutes scored as sleep has shown to be a highly reliable and valid method to assess sleep onset when tested against polysomnography.²⁶

The total number of minutes between initial sleep start and final sleep end in the morning is defined as sleep period. Time in bed is the total number of minutes between going to bed at night (time to bed) and getting out of bed in the morning (get up time). See Figure 1.

Figure 1. Illustration over an actigraphy recording.

Time in bed is the number of minutes between time to bed and get up time. Sleep period is the number of minutes between sleep start and sleep end. The period between time to bed and sleep start is defined as the sleep onset latency.

Total sleep time (minutes) represents the total number of minutes spent asleep during the sleep period (sleep period minus wake epochs).

Sleep percent (total sleep time × 100 / sleep period) is used as an additional parameter for sleep efficiency.

Sleep efficiency describes total sleep time × 100 / time in bed. Thus, by contrast with the sleep percent parameter, sleep efficiency also includes time spent in bed before initial sleep onset and after final awakening.

Sleep diary

From the validated Karolinska sleep diary,^17,18 a summated index for sleep quality¹⁷ was constructed from 4 of its items: (1) Sleep quality—phrased “How did you sleep?” (very well [5] – very poorly [1]); (2) Restless sleep (not at all [5] – very restless [1]); (3) Difficulties falling asleep (not at all (5) – very difficult (1)], and (4) premature (final) awakening (not at all [5] – woke up far too early [1]). The same index has been shown to be correlated (0.49–0.66) with objectively sleep efficiency (measured by polysomnography).^18,27 Scores on the summated sleep quality index range from 4 to 20, where a score of 4 indicates worst quality, and a score of 20 best quality. In the analysis of reliability, only the index summing 4 items is included, in an attempt to provide a more normally distributed set of data.

The questionnaires and device for sleep measurements were distributed at baseline in February/March 2013.

Statistics

For description of the sleep data during different days of the week, a group mean stratified by gender was calculated for each sleep parameter. The association between the actigraphic parameters and self-rated sleep quality (indexed from the sleep diary) was calculated using Spearman correlation. The correlation coefficient was calculated for each day, ordered by week-day (Monday through Sunday).

An intra-class correlation (ICC) coefficient was calculated using a two-way random model, which provides a ratio of the between-subjects variance to the sum of all the variance components.²⁸ The intra-class correlation estimates the reliability of the measures for each parameter on a single night of measurement. In this study setting, a low single-measure intra-class correlation indicates a high intra-individual day-by-day variation.

The Spearman-Brown formula was then applied to estimate the effects of changes in number of aggregated days of measurement on the reliability of each test.

A Spearman-Brown reliability coefficient of ≥ 0.7 was used to indicate acceptable reliability, in accordance with the previous literature.¹⁴

Further, analyses were made to investigate what effect including individuals working during the weekend had on 1) the correlation between subjective sleep quality and the actigraphy parameters, and 2) the reliability (ICC analyses). To see whether measurements including week nights and weekend nights differed in reliability, analyses of measurements on Monday through Thursday (4 week nights) and Thursday through Sunday (2 week nights and 2 weekend nights) were performed.

The analyses were conducted using SPSS v. 22.

RESULTS

Characteristics of Participants

Of the 54 participants with eligible actigraphy data included in the analysis, 25 were women and 29 were men. Mean age was 45 years (range 26 to 63). Overall, at group level, the characteristics presented in Table 1 indicate that the participants had high self-rated well-being. There were no significant mean differences between men and women. One participant (male) was using sleep medication during the sleep measurements.

Table 1.

Characteristics of participants based on self-rated psychosocial health factors.

Missing Data

For one of the participants, no actigraphy data were recorded, and, for another, data from only one of the 7 days were recorded. These 2 participants were excluded from the study. In their sleep diary, participants could note if anything had happened during the night that might have influenced their sleep. Screening of these notes revealed 2 participants who reported that they had been out of bed several hours during at least one of the nights, and who therefore had a measurement from a non-representative night. These 2 individuals were excluded from the study. In all, actigraphy data from 54 individuals (93% of the 58 individuals) were eligible for analysis and included in the study. Of these, 52 (90% of the 58 individuals) had actigraphy data for all 7 days. Of the remaining 2, one had missing actigraphy data for 4 days, leaving 3 days of valid data; and one had missing data for one day, leaving 6 days of valid data. In the sleep diary, 3 individuals had not filled in time to bed (or showed a bad fit between this item and registration by the actigraph) for one of the 7 nights of measurement. In order ensure comparable measures between subjects, and to avoid introducing a potential bias in the analysis, these 3 (actigraphy) observations were left out of the analysis including actigraphy parameters. In all, when analyzing data, this meant that 49 individuals entered the ICC analysis when including 7 days of measure, 51 when including 4 days of measure, and 38 entered when excluding those who had been working during the weekend.

If data were missing on any of the items in the sleep diary included in the sleep quality index, data were considered missing on the index (as well as in the diary). In total, 54 (93%) of the 58 who had received a sleep diary had data on all the items included in the sleep quality index for all 7 days. From the 54 included in the study with eligible actigraphy data, 52 (96%) had data on all the items included in the sleep quality index for all 7 days. One of the 54 had missing data for 5 of the 7 days, and one for one of the 7 days. In analyses, when testing correlations between subjective sleep quality and actigraphy parameters for each of the day of the week, 50–53 entered the analysis, and 40–42 entered when excluding those working during the weekend.

Mean Values of Sleep Parameters and Correlation between Objective and Subjective Sleep Quality

The group mean values and 95% confidence intervals for the sleep variables by night of the week are presented in Figure 2. This information is presented solely for descriptive purposes. There seemed to be some difference in mean values during weekends and weekdays. The differences between mean values on the sleep parameters representing weekday sleep (aggregating Monday through Thursday) and weekend sleep (aggregating Friday through Saturday) were thus tested (in a paired t-test). On average, total sleep time during weekend nights was 38 min longer than during week nights (64 min longer when excluding those who had been working during the weekend), which was statistically significant (p < 0.05). There was also a significant difference in mean subjectively scored sleep quality during weeknights compared to weekend nights (mean difference 0.79, and 1.1 when excluding those who had been working during the weekend). The sleep percent was on average 1.1% lower during weekend nights than during week nights (however, when excluding individuals working during the weekend, the difference was smaller [0.70] and no longer significant). There were no significant differences in mean sleep efficiency during week nights and weekend nights.

Figure 2. Presentation of group mean values and 95% confidence intervals over different days of the week, stratified by gender.

Sleep parameters from actigraph presented in graphs 1–3, and subjective sleep quality in graph 4.

Correlations between actigraphy measured sleep and subjective sleep quality are presented for the different week nights of the study, when including all participants (Table 2) and when excluding individuals who had been working during the weekend (Table 3). The correlations were very low overall, and mostly nonsignificant. During week nights sleep percent seem to correlate better than total sleep time with subjectively reported sleep quality. Total sleep time seemed to correlate more with subjective sleep quality during weekend.

Table 2.

Spearman correlation coefficients for measurements on Monday through Sunday, including participants working during the weekend (n = 54).

Table 3.

Spearman correlation coefficients for measurements on Monday through Sunday, including only participants not working during weekend (n = 43).

Number of Days Needed to Obtain a Reliable Sleep Measure

An ICC coefficient was calculated on the basis of all the valid measurements of the sleep parameters in the actigraphy data, and for the sleep quality index from the sleep diary. In Figure 3, a reliability coefficient (Spearman-Brown coefficient) is presented for 1 to 7 days of measurement of each sleep parameter in the full group and when excluding individuals working during the weekend. In all, the analysis shows that at least 2 days of measurement were needed for a reliable measure of sleep percent, 5 days or more for sleep efficiency (4 when excluding individuals who had been working during the weekend), and 6 days or more for self-reported sleep quality. For measured total sleep time, 7 days or more were needed for a reliable measure if excluding individuals working during the weekend. When including individuals working during the weekend, sufficient reliability (a Spearman Brown coefficient ≥ 0.7) was not reached over the 7-day period.

Figure 3. Presenting Spearman Brown coefficients (y-axis) by number of days of measurements (x-axis) for All (the full group, n = 49) and when only including participants who had not been working during the weekend (n = 39).

Graph 1–3 present sleep parameters from the actigraphy measures and graph 4 the subjectively reported sleep quality (by index from the sleep diary). Coefficients > 0.7 indicate a satisfactory reliability. The Spearman Brown coefficient for day 1 is equivalent to ICC coefficient for a single measure.

Further, higher reliability was observed for measurements when including only week nights (Monday to Thursday) than when including weekend nights (Table 4).

Table 4.

Estimates from a single measure (ICC for a single measure, equivalent to the Spearman-Brown coefficient for a single measure) and the number of days of measurement (1 to 4) required for the reliable measuring of objective sleep parameters and subjective sleep quality.

DISCUSSION

This study investigated the number of assessments required for a reliable measure of actigraphy measured sleep and subjective sleep quality to be obtained.

The results suggest that measurement of at least five days is to be preferred when estimating actigraphy sleep efficiency, a parameter frequently used in actigraphic sleep studies. Previous studies, however, show that, among teenagers, at least two days should be sufficient.¹⁴ Sleep percent required fewer nights for a reliable measure to be obtained, two or more days in our study. Sleep percent differs from sleep efficiency in that it does not include sleep onset latency, but solely considers the proportion of sleep from sleep start until sleep end. It is rarely used in sleep studies. As shown in Figure 3, excluding individuals who had been working during the weekend had very little impact on the reliability of the sleep measures. The exclusion had highest impact on total sleep time.

Our results indicate that more days of measurement are needed for a reliable measure of self-reported sleep quality than for the actigraphy measures of sleep (except for total sleep time). The effect of repeatedly responding to the same questionnaire over a short period of time (every day for a full week) on the reliability and validity of the questionnaire, should however be investigated further.

Our study suggests that not even measurements lasting a full week would give a satisfactorily reliable measure of total sleep time. Knutson and colleagues found that five or six days of measurement of sleep duration (using the same definition as for the parameter total sleep time that was used in our study) should be sufficient for adults,¹⁶ and in a study including only women it was suggested that at least five days of measurement are needed.¹⁵ Previous research has suggested that the day-byday variability in sleep duration measured by actigraph is associated with worse subjective sleep quality and poor subjective well-being²⁹ and stress.³⁰ Depending on the purpose for which sleep is measured, having information on variability in sleep duration or total sleep time, might be a valuable complement to knowledge of the aggregated mean of sleep duration over several days.

We saw differences in reliability between including and excluding weekend nights of measurement (measuring on weeknights only). Higher reliability was found when weekend nights were not included. We chose to let four nights represent week nights, i.e., the nights of Monday through Thursday, for two reasons. One is that the night between Sunday and Monday may not be representative of a “normal” week night. The other is that, when performing measurements in a natural setting, it is more likely that the first day of measurement does not occur during the weekend (Sunday). Thus, the settings chosen for this study are representative of a “normal” or clinical setting. Previous studies (including five week nights) have found smaller differences between including and excluding weekend nights^14,15 than we did in the current study.

In Figure 2, where mean values of the self-reported sleep quality and the actigraphy sleep parameters are presented, total sleep time seems longer during the weekend, and self-reported sleep quality seems higher. Sleep percent and sleep efficiency did however not follow the same trend. Sleep percent seemed to be somewhat lower during weekend nights than weeknights. Sleep during weekends may from a recovery perspective be interesting to study further. If this is to be done, these differences in the different sleep parameters during weeknights and weekend nights could be worth investigating further.

Overall, the correlation between actigraphy sleep parameters and self-rated sleep quality was low. The correlations between scores on the sleep quality index from the Karolinska sleep diary and sleep efficiency as measured by polysomnography measures have been shown in previous studies to be higher than those shown in the current study. A study performed in a laboratory over nine days on people with irregular sleeping hours showed a correlation of 0.66.¹⁸ However, when investigating measurements of one-night sleep in a natural setting in a working population, a correlation of 0.49 was found between (polysomnography) sleep efficiency and score on the sleep quality index.²⁷ In the current study, the correlation varied substantially across the different days of the week.

As mentioned in the Introduction, a discrepancy between objective and subjective sleep measures has been seen previously,³¹ and it has been suggested that objective and subjective measures capture different dimensions of sleep.^19,32 In this context, our results concerning correlations between subjective and actigraphy data measures on different days of the week are interesting. Overall, the correlations were higher during the week nights (as compared with the weekend), apart from when testing correlations between subjective sleep quality and total sleep time, where the opposite trend was seen. However, the correlations were overall very low and in most cases nonsignificant. To the best of our knowledge, no other study has investigated such variation in correlation over different days of the week during a period of repeated measurements, and further research is needed to understand its potential consequences for sleep assessments.

Eleven of the participants of this study were working during the weekend. Including them in the analyses had very little impact on the results in this study. It is possible that sleep habits during the weekend are different from weeknights even if a person works during the weekend. Being part of a social structure where family and friends are off during weekends may influence how one sleeps during the weekend. When assessing sleep, it may be important to be aware of which days of the week to include and whether work is performed during the weekend or not.

In the current study, all participants worked at the same location, and the studied group is more homogenous than a random sample from a general population would be. Further, knowledge of entering a study together with colleagues may have increased compliance, which might explain the low overall proportion of missing data (90% had full actigraphy data from all seven nights). Previous studies have found up to 28% missing data in weekly actigraphic recordings.¹⁴

CONCLUSIONS

The results in the present study suggest that data from at least two nights are to be recommended when assessing the actigraphy sleep measure sleep percent and at least five nights when assessing (actigraphy) sleep efficiency. For actigraphy measured total sleep time, more than seven nights are recommended. At least six nights of measurements are required for a reliable measure of self-reported sleep quality. However, if measurements are performed on Monday through Thursday, four nights of measurement seem to be sufficient to obtain a reliable objective and subjective measure. Depending on the purpose of the sleep assessment, it may be preferable to collect data during weeknights only. Information on whether weekend nights are included or not are important when presenting the results of sleep studies. The low overall correlations between actigraphy sleep measurements and self-reports in this study suggest that subjectively and actigraphy measured sleep capture different dimensions of sleep.

DISCLOSURE STATEMENT

This was not an industry supported study. This study was in part funded by the Stockholm Stress Center, a FAS/Forte (Swedish Council for Working Life and Social Research) Center of Excellence (dnr 2009-1758). The study was performed under the auspices of Future Occupational Health Services FAS/Forte 2010-1563. The authors have indicated no financial conflicts of interest. All authors were involved during the full process of planning the project, assessing data, analyzing data, and writing the manuscript.

ABBREVIATIONS

ICC

intra-class correlation

OHS

occupational health services