Consumer sleep technology: accuracy and impact on behavior among healthy individuals

Citation:

Khosla S, Wickwire EM. Consumer sleep technology: accuracy and impact on behavior among healthy individuals. J Clin Sleep Med. 2020;16(5):665–666.

We have witnessed a tremendous increase in sleep awareness over the last decade. Although sleep professionals often receive some credit for this increased awareness, the reality is that public awareness is primarily attributable to the explosion of self-tracking devices and increased media attention regarding sleep. As a result of these changes, consumers are more aware of their sleep patterns. Some consumers invest in tracking devices to evaluate their sleep concerns; many others learn about sleep serendipitously from a fitness tracker that was purchased for other reasons.

In addition to increasing public awareness, consumer sleep technology (CST) has already impacted the practice of clinical sleep medicine. Wearable CST devices are ubiquitous and appear in most sleep clinics on a daily basis. Although there are potential benefits in terms of measurement, sleep clinicians are faced with the considerable challenge of interpreting CST data, generally without standardization or guidelines.¹ From a technical perspective, most CST uses proprietary algorithms, resulting in significant variability in data quality from one device to the next. Sleep clinicians also struggle to compare the metrics across devices, asking questions such as “Is deep sleep on one device the same as deep sleep on another? How do we hope to comprehend this vast amount of data?”

To increase the use of CST-derived sleep data, clinically relevant validation is crucial. Such research will empower sleep clinicians to derive meaningful information from the various sleep scores and sleep hypnograms across devices. To provide systematic guidance regarding CST and related issues, the American Academy of Sleep Medicine (AASM) recently commissioned a Clinical and Consumer Sleep Technology Committee. A position statement has been published regarding the use of consumer sleep technology in clinical practice.¹ This committee has also created an interactive database² where sleep-related technologies are analyzed with links to peer-reviewed data.

In this issue of the Journal of Clinical Sleep Medicine, Berryhill and colleagues³ contributed to a growing body of CST research by evaluating the accuracy of CST estimates of sleep quantity and quality, relative to gold-standard polysomnography (PSG). Importantly and in a novel contribution, these authors also investigated a potential Hawthorne effect⁴ of CST by evaluating the impact of wearing a CST device on perceived sleep quality in healthy participants. In this crossover study, healthy volunteers completed 2 weeks of sleep diaries, including 1 week with and 1 week without a WHOOP device. At the midpoint, participants underwent polysomnography (all participants wore the CST device during PSG) before crossing over from 1 arm of the study to the next. The impact of CST on self-reported sleep quality ratings was assessed, and CST results were compared with concurrent PSG.

To assess accuracy of the WHOOP device, the authors used traditional device validation methods that considered both bias error (ie, systematic measurement error) and precision error (ie, inaccuracy caused by the device, investigator, testing environment, or other factors related to these specific testing conditions). Overall, the authors report low bias and precision errors for sleep duration and sleep fragmentation. The authors assessed the intraclass correlation (ICC) between 2 independent, blinded PSG scorers and then between the device and the consensus PSG scoring based on the 2 raters. Not surprisingly, the ICC between the 2 PSG scorers in this AASM-accredited center was excellent (ICC = 0.91), but the overall agreement between the device and PSG was only moderate (ICC = 0.63), including good for dream sleep (ICC = 0.85) and moderate for both slow wave sleep (ICC = 0.74) and light sleep (ICC = 0.63).

Overall, the authors report reasonable performance of this CST in multiple domains. At the same time, results of this well-designed and well-executed study highlight the need for standardized reporting across CST validation studies. Most sleep clinicians are unfamiliar with bias error, precision error, or ICCs, and to maximize the utility of important CST validation studies, straightforward comparisons between devices must be made easy to understand.

Independent of the accuracy of the WHOOP for determining specific sleep stages, the ability to determine wakefulness and sleep (any stage) may be valuable even when specific sleep staging is less robust. For example, CST can empower sleep medicine clinicians by providing low-cost and valid insight into sleep patterns and circadian rhythm disturbances. In our center (SK), we currently use positive airway pressure (PAP) downloads to examine presumed sleep time while using PAP but have no good measure of off-PAP sleep time.⁵ Thus, CST metrics might become easily incorporated into routine sleep visit. Validation studies are critical in this regard. The authors also note strong performance of the device for cardiovascular measures including heart rate, respiratory rate, and heart rate variability, although the heart rate variability findings warrant replication using robust measures of heart rate variability (ie, not a single PSG electrode).

In addition to the accuracy of the WHOOP device, Berryhill and colleagues also examined the impact of CST on perceived sleep quality. This is an important and novel contribution to the literature. For perspective, years ago, when studies began to evaluate the effect steps measured on a fitness tracker had on activity⁶ or weight loss,⁷ many health professionals were disappointed to learn that measuring steps did not routinely translate into improved fitness, sustained weight reduction, or increased exercise time. By contrast, Berryhill and colleagues found that CST was associated with changes in perceived sleep quality, perhaps because of a modified sleep schedule or another impact of self-monitoring. In this study, minimal sleep messaging was provided to participants, although such features are available when in commercial use. Although the study duration was only 2 weeks, this important finding supports prior preliminary results in our field that sleep behavior can be modified via algorithmic messaging.⁸ As mobile health approaches expand in our field, others are currently validating messaging strategies to enhance patient adherence and clinical outcomes. At the same time, one potential risk of overfocusing on CST data is creating an overfocus on sleep among healthy individuals, which others have hypothesized can be insomnogenic.⁹

CST is here to stay. Although much work remains, CST technologies demonstrate potential to provide important insight into long-term sleep patterns and circadian rhythm disturbances. One can easily envision CST being monitored longitudinally, much the same way PAP downloads are currently used in clinical sleep medicine practice. Long-term CST validation studies are critical in this regard. In addition to validating CST over extended time periods, future studies should examine how these devices perform in patients with sleep disorders and medical diseases, in particular obstructive sleep apnea (OSA). Because most of AASM-accredited sleep centers focus heavily on OSA, knowledge of device performance in treated and untreated patients with OSA would be valuable. More ambitiously, given the prevalence of OSA, its high medical and economic burden,^10,11 and poor penetration of diagnosis (80% undiagnosed¹²), CST might allow sleep medicine clinicians to screen, diagnose, and treat more patients. Validation of CST and harnessing the potential for CST to change behavior are important steps in this process.

DISCLOSURE STATEMENT

All authors have seen and approved the manuscript. Conflicts of interest: SK receives salary support from Medbridge Healthcare. EMW’s institution has received research funding from the AASM Foundation, Department of Defense, Merck, and ResMed. EMW has served as a scientific consultant to DayZz, Eisai, Merck, and Purdue, and is an equity shareholder in WellTap.