Development and Evaluation of a Measure to Assess Restorative Sleep

Abstract

Background:

There are validated measures assessing insomnia and disturbed sleep, but few psychometrically sound instruments to assess perceptions of the restorative or inadequate properties of sleep are available.

Study Objectives:

To develop and evaluate a new instrument, the Restorative Sleep Questionnaire (RSQ).

Design and Setting:

Focus groups were conducted using participants with and without nonrestorative sleep complaints. Questions were designed to elicit the feelings and experiences people have about their sleep and their view of daytime consequences of sleep. Expert panels confirmed the importance of nonrestorative sleep (NRS) as a frequently encountered problem either with or without other sleep complaints. The resulting RSQ was administered in three studies: (1) a telephone interview with healthy controls and individuals with sleep problems; (2) a randomized clinical trial of patients with primary insomnia assessed by polysomnography (PSG); (3) a PSG study of subjects with NRS complaints.

Measurement and Results:

Across all studies, the new measures were shown to be significantly correlated with health-related quality of life (HRQL) domains hypothesized to be related to NRS. The RSQ had good psychometric properties (α > 0.90; rtest-retest > 0.80), and factor analysis confirmed the unidimensionality of the measure. The RSQ was able to distinguish between healthy controls, patients with primary insomnia, and insomnia patients with isolated NRS complaints but without PSG defined sleep onset, duration, or maintenance problems. Normal sleepers reported sleep that was about a standard deviation more restorative than that of those with NRS on the RSQ.

Conclusions:

The results of the study provide support for the reliability and validity of the RSQ as a measure of NRS in subjects with and without self-reported or PSG confirmed sleep initiation and maintenance difficulties.

ClinicalTrials.gov Identifiers:

NCT00655369; NCT00705601

Citation:

Drake CL, Hays RD, Morlock R, Wang F, Shikiar R, Frank L, Downey R, Roth T. Development and evaluation of a measure to assess restorative sleep. J Clin Sleep Med 2014;10(7):733-741.

Nonrestorative sleep (NRS) is a distinct component of insomnia and is included in the DSM-IV, but not DSM-V, diagnostic criteria for the disorder.^1,2 The prevalence of NRS (i.e., feeling that sleep was restless, light, of poor quality, or awakening feeling unrestored or unrefreshed) is estimated to be 10% to 25% of the general population and can manifest with or without difficulties initiating and maintaining sleep.^3,4 Therefore, feeling unrefreshed upon awakening is a common complaint associated with a variety of medical, sleep, and psychiatric conditions. However, the ability to further explore the nature, impact, and treatment of NRS is limited by the lack of reliable and valid measures to assess this component of insomnia.⁵

While a large number of sleep scales exist, most deal with aspects of sleep disturbance per se that do not directly address restorative effects (e.g., sleep onset or initiation, sleep interruptions, the overall length of sleep, sleep quality, alertness on awakening, and daytime somnolence). In scales that ask about the restorative value of sleep, the assessment is generally made with a single item often defined as a subjective feeling of being unrefreshed upon awakening.⁶ For example, the Saint Mary's Hospital Sleep Questionnaire⁷ asks a single question concerning the person's sense of feeling “clear headed” upon awakening, and the Daily Sleep Diary⁸ includes a single question about “feeling rested” upon awakening. Systematic reviews report that although NRS is a key component of insomnia, there are only select instruments available to assess it.^6,9

BRIEF SUMMARY

Current Knowledge/Study Rationale: There is a need for tools that adequately assess the restorative quality of sleep. The objective of the present study was to develop a reliable and valid patient-reported measure of NRS through both qualitative and quantitative methods in insomnia subjects with and without difficulty initiating and maintaining sleep.

Study Impact: The Restorative Sleep Questionnaire provides a valid and reliable measure for assessing nonrestorative sleep complaints. Future studies are needed to determine the sensitivity of this measure to therapeutic interventions.

In the studies that have assessed NRS, it has been defined using terms such as “waking with a feeling of fatigue or exhaustion,” or “not feeling really rested.”^10–12 Many studies using single items have emphasized that complaints of unrefreshing/non-restful sleep occur even if the duration of sleep is normal.¹³ A recent study showed that complaints of NRS can exist even in individuals with polysomnographically determined normal sleep onset, duration, and continuity.¹⁴ The restorative quality of sleep is an important component of insomnia as a significant portion of insomnia patients present exclusively with this complaint, reporting normal sleep initiation and maintenance.⁴ Data from the National Comorbidity Study show the prevalence of individuals in an insomnia population with exclusive complaints of NRS is estimated at 7%.⁴ Importantly, in the largest study to date, people with NRS were found to report daytime consequences significantly more often than those with difficulty initiating or maintaining sleep.³

Objective measures of NRS during sleep have been proposed, including most prominently alpha-EEG activity during NREM sleep (i.e., alpha intrusion).^15–17 This EEG correlate has been found in patients with fibromyalgia and chronic fatigue who characteristically report NRS.^16–18 However, no objective correlates have been evaluated systematically as to their sensitivity and specificity for identifying patients with nonrestorative sleep or the ability of treatments to reverse it. In a study of patients with isolated NRS complaints, no signs of EEG sleep disruption or abnormal EEG spectral density were observed.¹⁴ While the objective measurement of NRS warrants further attention including exploration of subcortical or limbic brain correlates (e.g., neuroimaging, magneto-encephalography), identification of the primary symptom(s) and patient descriptors of NRS would complement such research. Reliable and valid measurement of NRS is a prerequisite for the identification of patients with such complaints and further experiential investigation of its underlying etiology. Accurate quantification is important for the development and evaluation of specific treatments for this symptom.

Systematic and comprehensive reviews of the literature regarding the measurement of NRS identified 26 instruments with content related to the assessment of NRS.⁶ Although the Patient-Reported Outcome Measurement Information System (PROMIS) has sleep-related item banks that could be used to develop a specific NRS measure,¹⁹ no specific nonrestorative sleep items were identified. The Sleep Assessment Questionnaire (SAQ)²⁰ has content specific to NRS but includes only one item directly related to this construct. While brief measures of sleep have value and efficiency in clinical and community-based assessment, and correlate with proposed NRS scales, single items are unlikely to fully capture nonrestorative sleep.²¹ Similarly, reports of specific single item nocturnal sleep symptoms correlated with insomnia scales (e.g., insomnia severity index) but fail to capture the broad aspects of the condition. Because the SAQ was developed with fibromyalgia and chronic fatigue patients, its relevance and psychometric performance in insomnia and psychiatric NRS related conditions remains unknown. In concluding their review of NRS, Vernon and colleagues noted that “…little qualitative research with patients has been conducted to develop a measure consistent with patients' experiences and that comprehensively evaluates this concept.”⁶ One recently developed comprehensive sleep measure, the Iowa Sleep Disturbances Inventory, includes an NRS scale focusing on sleep related daytime disturbances.²² While this scale has reasonable psychometric properties and factor structure, it requires further evaluation in larger well characterized insomnia samples.

The objective of this research was to develop a reliable and valid patient-reported measure of NRS through both qualitative and quantitative methods in insomnia subjects with and without difficulty initiating and maintaining sleep. The guiding conceptual model for scale development defined NRS as a feeling of being unrefreshed upon awakening regardless of sleep quality and quantity.

METHODS

Instrument Development

The development of the Restorative Sleep Questionnaire (RSQ) occurred through a series of phases. After a review of the literature confirmed that the concepts of NRS and daytime consequences of poor sleep were not fully covered in existing sleep assessment instruments, focus groups were conducted and expert panels were convened to more fully characterize these concepts and to provide item generation. Next, a preliminary version of the RSQ was included in “cognitive debriefing” interviews to assess the extent to which the draft items were understood by patients and to gain information for revising item wording. In the final phase, 3 studies were carried out to evaluate the factor structure, reliability, and validity of the RSQ (see Table 2).

Table 2.

Instruments used in the 3 validation studies and their characteristics.

The RSQ as evaluated in the studies to follow included 9 items derived from 3 focus groups and 2 expert panels (described below). It included 9 self-report items related to various aspects of the restorative quality of sleep. The questionnaire items were completed upon awakening and included 2 versions. The first version (RSQ-D) was completed on a daily basis while the RSQ-W was a weekly version completed upon awakening that referred to the restorative quality of sleep over the past 7 nights. The RSQ-D and RSQ-W items are shown in the supplemental material. Several analyses were conducted to evaluate the reliability and validity of the RSQ.

Reliability

Internal consistency reliability was assessed using Cronbach coefficient α. Coefficient α was calculated for each administration of the RSQ-W and RSQ-D. Where there were multiple administrations of a measure within a given study (e.g., the insomnia and the NRS studies), average coefficient α values across all administrations were also calculated. An α ≥ 0.70 was deemed acceptable.²³ Coefficient α with item deleted was also examined to determine whether the exclusion of an item increased reliability, which can be indicative of a poor item or the item measuring a distinct domain from the other items.

Test-retest reliability to assess the stability of the measures was calculated, using Pearson product moment correlations between separate administrations, for those studies in which the measures were obtained on multiple occasions (i.e., the insomnia and the NRS studies). For the insomnia study, we calculated test-retest reliability only for the placebo conditions, since the treatment interventions could reduce stability. Intra-class correlation coefficients were also computed to estimate stability across multiple measurement occasions.

We also conducted a factor analysis to confirm the extent to which the items in the RSQ-W and RSQ-D represented unidimensional constructs.

Validity

We estimated the associations between the RSQ-D and RSQ-W with other patient-reported measures (see Table 2 for a brief description of measures) as well as PSG measures. In addition, we examined “known groups” validity by assessing differences in RSQ scores between healthy controls and sleep deprived subjects, subjects with arthritis, and a cohort with adequate sleep initiation and maintenance but a complaint of “nonrestorative” sleep in a community-based sample. Finally, a comparison of RSQ-W and RSQ-D scores was made between healthy controls with adequate sleep, versus (1) insomniacs with PSG verified sleep maintenance and/or sleep onset difficulties and (2) insomniacs without PSG maintenance and/or onset difficulties but complaints of “nonrestorative sleep.” The latter group comparison was specifically designed to determine if the RSQ measures could detect differences from healthy control sleepers even among clinical insomniacs who do not have traditionally defined PSG sleep disturbance.

Focus Groups and Expert Panels

Three focus groups were formed using a convenience sample of volunteers (8 males and 16 females) recruited through local newspaper advertisements and screened by telephone within an age range of 18-65 years of age. All 3 focus groups were held at the University of California Los Angeles campus and were conducted by professional focus group moderators. The first focus group consisted of “normal” sleepers. The second and third groups were “problem sleepers.” The recruitment flyers stated eligibility as “must speak English well and be at least 18 years of age.” For the second and third groups the advertisements also stated that “you should be experiencing insomnia (trouble falling asleep or trouble staying asleep).” Potential focus group participants were characterized by their age, gender, whether they have problems with sleep, how long they have had sleep problems, and what kind of sleep problems they have had. All focus group participants were paid $40 for their time and transportation cost.

A semi-structured interview was used to guide participants through a description of the concepts underlying 2 general descriptors including what they “considered to be a good night's sleep” and separately, what they “considered to be a bad night's sleep.” Participants in each group were asked to identify and discuss: “How do you feel when you wake up after a good night's sleep?” “What is it like when you wake up feeling refreshed?” “Does having a good night of sleep affect your vitality or energy during the next day?” “Do you feel more alert in the afternoon after a good night of sleep?” In contrast, participants discussed the following: “What do you feel like when you wake up after a bad night's sleep?” “What is it like when you wake up feeling unrested (unrefreshed)?” “Does having a bad night of sleep affect your vitality or energy during the next day?” “Do you feel less alert after a bad night of sleep?” Finally, participants were asked to discuss what sleep quality means to them. The focus group leader probed until these concepts were fully explored (no new information or limited new information elicited). Additional details regarding the focus groups and participant characteristics are included in the supplemental material.

Summary of Focus Groups

Several common themes emerged across each group. A good night's sleep was described as falling asleep easily, sleeping deeply without waking, and feeling rested upon awakening. The consequences of a good night's sleep included awakening feeling rested and energetic, physically better and healthier and clearer minded with a sense of refreshment. On the other hand, a poor night's sleep was described as resulting in awakening with a feeling of tiredness that lasts throughout the day. They reported feeling numb, groggy, heavy-headed, forgetful, lethargic, and “spacey,” with other symptoms including headaches, being thirsty, and being “scratchy eyed.” Most agreed that a bad night's sleep leads to behaviors during the day that they would otherwise avoid, such as consuming significant amounts of caffeine, overeating or eating improperly, and not exercising. The concepts identified in these focus groups (energy, mood, feeling refreshed/restored, rested, mentally alert, and sleepy) were used to develop a pool of 9 items related to the concept of nonrestorative sleep. Additional details regarding the focus group methodology and results can be found in the supplemental material.

Expert Panels

Two expert panels of researchers and clinicians in the area of sleep medicine were convened. Experts were identified as actively engaged in treating sleep disorders and/or actively involved in sleep research. The first panel (10 participants) was comprised predominantly of experts from North America, while the second panel (13 participants) was predominantly from Europe. The specific topics of NRS and of daytime consequences of NRS were a central focus of the meetings. During each expert panel, participants were asked about NRS alone, as well as in combination with other sleep disorders, and were asked to describe specific attributes of NRS.

Both expert panels agreed that existing scales with the ability to assess NRS may not be optimal for clinicians interested in making therapeutic decisions or measuring treatment outcomes. The need for measures of NRS based on qualitative as well as quantitative research, including content validity consistent with patient experiences, was emphasized. In addition, panelists indicated that the information reported from the focus groups was meaningful and relevant for the development of instruments assessing NRS. Clinicians described NRS in a fashion similar to patient focus groups. Consensus was obtained around the point that NRS represents a unique symptom that has been difficult to define precisely or measure and separate from nocturnal sleep symptoms and possible comorbid conditions (i.e., arthritis). From the 6 identified concepts related to NRS, a set of 19 potential items were developed through interactions with the expert panel and the focus group outcome report. Of the 19 items identified, a subset of 9 items were thought to reflect the construct under study, while the additional items were considered more related to daytime consequences of nonrestorative sleep (e.g., unable to accomplish things during the day) and were excluded from further NRS scale development.

Cognitive Debriefing Interviews

After using focus groups and expert panels to generate potential NRS items, item selection was reviewed in a series of eight 1-on-1 interviews of recruited participants who did not participate in the focus groups. This was done to ensure that there were no major errors of omission or commission with respect to the concepts covered by the items, and the wording and responses required were comprehensive, meaningful, easily understandable, and unambiguous. Participants were excluded if they reported experiencing sleep apnea, hypersomnia, parasomnia, or using prescribed or over-the-counter sleep medications to fall asleep. Only minor wording changes in the items were made as a result of these interviews. The revised versions of the RSQ-W and RSQ-D are shown in the supplemental material.

Validation Studies

Demographic data for the 3 validation studies are shown in Table 1. Other measures included in the validation studies and used for comparison with the RSQ are included in Table 2.

Table 1.

Demographic data for the 3 nonrestorative sleep (NRS) validation samples

Study 1: Henry Ford Community-Based Study

Data were collected via telephone by DataStat Inc. (Ann Arbor, MI) from adults residing in southeastern Michigan between April 26 and August 25, 2004. Telephone interviews (random digit dial sample from the general population of Southeastern Michigan) were obtained from 774 individuals (74% response rate). The average age of the sample was 45 (range: 18-65); 44% were female; 69% were employed.

A total of 334 of the sample were chronic insomniacs, of whom 23 had a symptom of nonrestorative sleep with no other insomnia symptoms or comorbid conditions (NRS group) and 27 reported difficulty initiating sleep (DIS) and/or difficulty maintaining sleep (DMS) without other comorbid conditions. The remaining people with insomnia in the sample (n = 284) had a combination of sleep initiation, maintenance, and nonrestorative sleep symptoms along with comorbid conditions. The criteria for insomnia included subjective DIS, DMS, or NRS for at least 1 month in the past year, and which occurred at least “sometimes” or “often.” Of the remaining 440 subjects, there were 49 healthy sleep deprived individuals without insomnia complaints or medical or psychiatric conditions. Sleep deprived individuals were limited to those who reported obtaining ≤ 6 hours of sleep per night on average over the past 2 weeks and who reported neither medical or psychiatric conditions nor insomnia (as above). There were also 45 people with arthritis and no other identified comorbid medical or psychiatric conditions. Finally, 121 healthy individuals without comorbid conditions or insomnia symptoms were selected from the larger sample. These healthy controls were required to be free from any medical or psychiatric conditions, did not use alcohol > 3 times per week, did not report insomnia, and self-reported sleeping ≥ 7 h/night (average over the past 2 weeks). The remaining 225 participants were not utilized for the current study.

Questionnaires assessed in the community-based study are shown in Table 2. Additional questions asked included “Approximately what time do you usually go to bed?” “How many times, if any, do you usually wake up during sleep (No times to 3+ times)?” In addition, subjects were asked to report about their overall health (excellent to poor), age, gender, education, employment status, and race/ethnicity.

Community-Based Study Results

Coefficient α reliability estimates from the community-based study for the scales were 0.91 and 0.90 for the RSQ-D and RSQ-W, respectively. Coefficient α estimated from the small (n = 46) insomnia study was 0.93 for the RSQ-D over the 10 daily replications. In none of the cases cited above did deletion of any single item result in a substantive increase in the magnitude of coefficient α thereby lending support to the integrity of the scales. The high levels of coefficient α provide an indication that the items are addressing the same construct. The unidimensionality of the scales is further supported by factor analyses of the scales in the community-based and NRS study (see below). For the community-based study, the separate factor analyses of the RSQ-D and RSQ-W each resulted in a one factor solution, based upon an examination of the scree plot of eigenvalues, as well as retaining factors with principal component eigenvalues ≥ 1.0. The results for the single factor solutions for the RSQ-D and RSQ-W are shown in Table 3.

Table 3.

Factor loadings for the Restorative Sleep Questionnaire-Daily (RSQ-D) and Restorative Sleep Questionnaire-Weekly (RSQ-W): community-based study

As can be seen in Table 4, in the community-based study, the RSQ-D scores were correlated significantly and in the expected direction with the ESS, the Daytime Sleepiness Scale, and the MOS-sleep adequacy and sleep problem scores. In addition, the RSQ-D correlated significantly with the SF-36 physical and mental component summary scores (PCS and MCS); the highest correlation with any single SF-36 scale was with the vitality scale.

Table 4.

Product-moment correlations of the RSQ-D with selected sleep and health-related quality of life measures in the community-based study

There were significant differences among the 5 group means in the community-based study for each measure: F = 9.46, p < 0.0001 and F = 13.23, p < 0.0001 for the RSQ-D and RSQ-W, respectively. Tests for group differences are displayed in Table 5. The normal group scored highest on each measure (p < 0.05 for all). Despite reporting no problems getting to sleep or staying asleep (and excluding subjects with confounding comorbid conditions) the NRS only group reported greater impairment on the RSQ than did those who were sleep deprived, people with self-reported arthritis, and normal cohorts.

Table 5.

Mean (SD) sleep scale scores for five mutually exclusive groups—community-based study

Study 2: Primary Insomnia Study

This study was a randomized, double-blind, active- and placebo-controlled, Latin square 5-way crossover (5 treatment, 5 sequence) study (unpublished) including 40 patients with primary insomnia conducted to test the safety and efficacy of a new pharmacological treatment for insomnia (compound PD200290: doses of 100, 300, and 900 mg) against an active comparator (zolpidem 10 mg), but also included the RSQ (D and W). The study was conducted at 8 different sites in the US.

After informed consent, participants were asked to complete a set of clinical evaluations (including 2 screening polysomnographs [PSGs]) during the screening phase (Days -20 to -1). Once randomized, participants received their study medication one-half hour before bedtime on Night 1 and Night 2, and standard PSG procedures were employed for evaluation during each night.²⁴ Within approximately 30 min of awakening, subjects completed the RSQ-D, the Leeds Sleep Evaluation Questionnaire (LSEQ),²⁵ the Subjective Sleep Questionnaire (SSQ), the Digit Symbol Substitution Test (DSST),²⁶ and the Hopkins Verbal Learning Test (HVLT).²⁷ The SF-36 vitality scale was completed at 22:00 on Nights 2 and 3. These same evaluations were then repeated on Nights 9 and 10, 16 and 17, 23 and 24, and 30 and 31, following receipt of each of the other study medications in the specific sequence on the evenings of Nights 8 and 9, 15 and 16, 22 and 23, and 29 and 30, respectively.

Insomnia Study Results

For the insomnia study, we estimated the test-retest reliability of the RSQ-D by assessing the responses on the consecutive days in the placebo conditions only. The average test-retest reliabilities for consecutive daily measurements was r = 0.77. Correlations of the RSQ with the PSG measures in the insomnia sample are shown in Table S2 (supplemental material). The RSQ-D correlated significantly and positively with the Quality of Sleep (r = 0.40; p = 0.006), the Awakening from Sleep (r = 0.46, p = 0.002), and the Behavior Following Waking (r = 0.74; p < 0.001) scores from the Leeds Sleep Evaluation Questionnaire. The correlation between the RSQ-D and the Getting to Sleep scores was not significant (r = 0.27; p = 0.079). All correlations were in the expected direction.

The correlations of the RSQ-D with the Subjective Sleep Questionnaire were all in the expected direction. The correlations with sleep quality (r = 0.59; p < 0.001) and total sleep time (r = 0.32; p = 0.036) were both significant. Finally, the correlation with the Vitality Questionnaire was in the expected direction and statistically significant (r = 0.61; p < 0.001). The average correlation of the RSQ-D with the DSST was small in magnitude and not statistically significant.

Study 3: NRS Laboratory PSG Study

This study was designed to allow for better characterization of the NRS population. Full details of this study can be found in the published manuscript and included a standard overnight PSG adaptation night (study day 8), 2 baseline PSG night assessments (study days 9-10) and 2 repeat PSGs 1 month later (study days 38 and 39).¹⁴ All NPSG recording periods lasted 8 h. It was a non-treatment, cross-sectional study of 5 different adult cohorts (i.e., ages 18-64), 4 of whom had a complaint of NRS: (a) adults (i.e., ages 18-64) with objective and subjective problems initiating sleep and NRS; (b) adults with objective and subjective problems maintaining sleep and NRS; (c) adults with objective and subjective problems initiating sleep, objective and subjective problems maintaining sleep, and NRS; (d) adults with NRS and no objective or subjective problems initiating sleep and no objective or subjective problems maintaining sleep; and (e) a group of healthy controls without any sleep complaints. Exclusion criteria included neurological, psychiatric, or medical conditions (assessed using the Mini International Neuropsychiatric Interview or defined as any history of Axis I psychiatric diagnosis). Exceptions were mild hypertension, hypercholesterolemia, allergies not requiring treatment, and gender disorders. Individuals with a history of any other sleep disorder (including AHI or PLMI > 10 on NPSG), excessive caffeine intake (> 4 servings per day), smoking > 10 cigarettes per day, regular napping, or excessive alcohol intake (> 15 servings per week or > 5 servings per occasion) were also excluded.

For all subjects, following an extensive assessment of medical, sleep, social, and psychiatric history, initial sleep and HRQL assessments were administered. Home electronic diaries (e-diaries) were completed daily for 1 week; subjects then underwent one adaptation PSG followed by 2 additional PSGs on consecutive nights. Subjects then completed e-diaries every day for 1 month, followed by 2 more nights of PSG. The e-diaries were to be completed 30 min after arising from bed in the morning, and again just prior to going to bed in the evening. Morning assessments included the SSQ and the RSQ-D. The evening assessments included a set of questions concerning performance throughout the day.

Subjects returned for 2 more visits, 1 and 2 months after the second set of PSGs, and completed daily e-diaries in the week preceding each of these final 2 visits. During these visits they again completed sleep and HRQL assessments. Patients were not to receive treatment for sleep complaints, if any, until after the second set of PSGs, but were allowed to receive treatment for their sleep related complaints thereafter. Morning assessments included the SSQ and the RSQ-D. The evening assessments included a set of questions concerning performance throughout the day.

NRS Study Results

For the NRS study, all factor analyses of the RSQ-D and the RSQ-W (factor analyses were performed individually for data collected on different days) indicated single factor solutions.

The α averaged over 5 in-clinic administrations in the NRS study was 0.95 for the RSQ-D, while the average coefficient α over 4 in-clinic applications for the RSQ-W was also 0.95. In no case did deletion of any single item result in a substantive change in the magnitude of coefficient α. For the NRS study, the in-clinic administrations of the RSQ-W were made on Days 1, 8, 38, 68, and 98; while for the RSQ-D, the parallel data were collected on Days 2-8, 32-38, 62-68, and 92-98. The average test-retest reliabilities for consecutive measurements were r = 0.83, and 0.88 for the RSQ-D, and RSQ-W, respectively. For measurement occasions separated by an intervening measurement occasion, the average test-retest reliabilities were r = 0.81, and 0.84 for the RSQ-D, and RSQ-W, respectively. Even with 2 intervening measurement occasions, the average test-retest reliabilities remained high: r = 0.81, and 0.81 for the RSQ-D and RSQ-W, respectively. The intraclass correlation coefficients for the measures, taking into account all 5 measurement occasions, were 0.79 and 0.81 for the RSQ-D and RSQ-W, respectively. Finally, the RSQ-D was administered multiple times on the e-diary; test-retest reliability for consecutive measurement occasions was high (r = 0.87), and was high even for measurement occasions separated by 26 intervening e-diary administrations (r = 0.77).

The RSQ-W was completed on 5 different occasions during in-clinic visits by the subjects. The correlations between the RSQ and the other patient-reported outcomes (Table S1, supplemental material) indicate significant relationships to these measures and the same or even greater magnitude correlations with measures of vitality/fatigue.

The RSQ-D and RSQ-W correlated significantly with several of the PSG measures (Table S2, supplemental material). In particular, all 3 measures correlated significantly with most of the sleep/wake measures from the PSG, including latency to persistent sleep, total sleep time, sleep efficiency, wake time after sleep onset, and total wake time. As expected, the correlations of the daily RSQ-D tended to be of greater magnitude with the PSG measures than was the case for the RSQ-W assessed weekly.

The RSQ correlated significantly with all the evening functioning scales, with correlations generally in the range of 0.50 to 0.70 (Table S2). Correlations with the 3 of the 4 items of the SSQ, while generally statistically significant, were of smaller magnitude, generally in the range of 0.10 to 0.20. Correlations with the “sleep quality” question were in the range of 0.56 to 0.70.

The correlation of the RSQ-D with the EFS asked on the same day (the “appropriate” match, as both dealt with the prior night's sleep) was compared with the correlation of the RSQ-D with EFS scores from the prior day (i.e., an “inappropriate match,” since the EFS reports on the sleep of the night prior to the report provided by the RSQ-D). As opposed to the correlations in the 0.52 to 0.73 range (Table S2), the correlations for the inappropriate match ranged from 0.17 to 0.27 (data not shown). This provides evidence of the ability of the RSQ-D to appropriately discriminate a subject's reports of one night's sleep from another.

The mean scores of the groups recruited into the NRS study were tested to assess whether they differed significantly from one another in the expected direction. For each measure, the differences among groups were significant (F = 66.35, p < 0.0001; and F = 81.12, p < 0.001 for the RSQ-D [average] and RSQ-W, respectively). Tests showed that the healthy controls had significantly higher scores than the 4 other groups on both measures (p < 0.05 for all; Table 7). As with the community-based study, NRS subjects reporting no problems getting to sleep or staying asleep have greater impairment than normal controls as measured by the RSQ.

Table 7.

Mean (SE) sleep scale scores for five mutually exclusive groups—NRS laboratory study

Table 6.

Average product-moment correlations* of RSQ-D score with measures from 10 measurement occasions in the insomnia study

DISCUSSION

Having restorative sleep is an important aspect of the overall sleep experience, yet available measures do not comprehensively assess this experience. The RSQ was developed to address this important gap. The concepts and resulting items used in the instruments were derived from focus groups conducted with both “normal” sleepers and people symptomatic for “non-refreshing” sleep. The concepts identified were similar to those described by two panels of sleep research experts.

The RSQ demonstrated the ability to distinguish subjects with normal sleep and those with subjective and PSG-confirmed difficulty initiating or maintaining sleep. The RSQ daily and weekly scales were shown in three separate studies to have adequate internal consistency reliability (> 0.90). Both measures also exhibited excellent test-retest reliability over both short-term and long-term assessments and in heterogeneous samples. The factorial structure of the RSQ (both the Daily and Weekly versions) suggested a single underlying dimension.

The results of this study also provide support for the validity of the RSQ. The measure development process emphasized maximizing content validity throughout, using patient focus group input to identify the key concepts important to capture in the items, cognitively interviewing other patients on the items to ensure that there were no major errors of commission or omission in the measures, and having the concepts reviewed by two separate expert panels. The three studies described in this manuscript also provide substantial evidence for the validity of the three measures. In general, the measures were shown to correlate with other patient-reported measures of sleep quality and experience in expected directions, but importantly not so highly as to suggest measurement of identical constructs. Although depression is associated with sleep disruption, this study found that the RSQ correlated more highly with vitality/ fatigue than with depression and anxiety. However, a limitation is that the present study was done in individuals with no major active psychiatric disorders. Further validation studies need to be carried out in samples with NRS comorbid with major psychiatric disorders (e.g., Major Depressive Disorder).

Finally, the measures correlated statistically significantly but modestly with several of the physiological measures of various sleep parameters obtained in the laboratory NRS study. The RSQ scale also was able to discriminate among sleep experiences on different nights, as evidenced by the higher correlations with the evening functioning scale that assessed the same night's sleep versus the EFS that had assessed the prior night's sleep. Both the community-based study and the NRS study also provided substantial evidence of “known groups” validity, in that scores known to differ on self-defined sleep quality differed significantly on all three measures. Thus, the RSQ was able to differentiate between insomniacs with difficulty initiating sleep and/or maintaining sleep, sleep deprived subjects, and controls.

CONCLUSION

Nonrestorative sleep is a critical component of many sleep, medical, and psychiatric disorders. The ability to quantify NRS in conjunction with other currently available scales of sleep disturbances will help in identifying patterns of sleep seen in various disorders as well as efficacy of different treatment approaches to these sleep problems. With additional work to support a meaningful interpretation of the scores in different clinical populations, the daytime consequences of the sleep questionnaire will be very helpful in quantifying this important symptom.

The diagnostic criteria for insomnia require the presence of daytime consequences of disturbed sleep. While scales are available to quantify sleep initiation and maintenance, there are few scales available to quantify NRS with or without problems getting to or maintaining sleep alongside associated daytime impairment. In sum, this study provides support for the reliability and validity of the RSQ as a measure of the important restorative aspects of the sleep experience and the consequences of poor sleep.

DISCLOSURE STATEMENT

The work reported here was performed under contract by UCLA and by the Health Care Analytics Group at UBC to Pfizer, Inc. The Henry Ford Hospital Community-based study was supported by NIMH grant 59338 and a follow up study grant from Pfizer. Inc. Drs. Morlock and Wang were employees of Pfizer, Inc at the time of the research. Dr. Frank was an employee of UBC at the time of the research. Dr. Hays was supported by the UCLA/DREW Project EXPORT, NCMHD (P20-MD00182), the UCLA Center for Minority Aging Research Center for Health Improvement in Minority Elders (RCMAR CHIME: P30-AG021684), and the UCLA Older Americans Independence Center, NIH/NIA (P30-AG028748). Dr. Drake has received funding from Merck, Teva, and Zeo; has consulted for Teva; and has been a speaker for Teva, Purdue, and Jazz. Dr. Roth has served as a consultant for Abbot, Accadia, Acogolix, Acorda, Actelion, Addrenex, Alchemers, Alza, Ancel, Arena, AstraZeneca, Aventis, AVER, Bayer, BMS, BTG, Cephalon, Cypress, Dove, Eisai, Elan, Eli Lilly, Evotec, Forest, Glaxo Smith Kline, Hypnion, Impax, Intec, Intra-Cellular, Jazz, Johnson and Johnson, King, Lundbeck, McNeil, MedicNova, Merck, Neurim, Neurocrine, Neurogen, Novartis, Orexo, Organon, Otsuka, Prestwick, Proctor and Gamble, Pfizer, Purdue, Resteva, Roche, Sanofi, Schering Plough, Sepracor, Servier, Shire, Somaxon, Syrex, Takeda, TransOral, Yanda, Vivometrics, Wyeth, Yamanuchi, and Xenoport. He has served on speakers bureau for Purdue and Sepracor. He has received research support from Apnex, Aventis, Cephalon, Glaxo Smith Kline, Merck, Neurocrine, Pfizer, Sanofi, Schering Plough, Sepracor, Somaxon, Syrex, Takeda, Transcept, Wyeth, and Xeno-port. The other authors have indicated no financial conflicts of interest.

SUPPLEMENTAL MATERIAL

Additional Information Regarding the Focus Groups

Subjects in the focus groups also completed a series of questionnaires including the ESS (8 items) and the MOS Sleep Scale (12 items with possible responses: “all of the time,” “most of the time,” “a good bit of the time,” “some of the time,” “a little of the time,” “none of the time”). We also asked: “approximately what time do you usually go to bed?” “how many times, if any, do you usually wake up during sleep?” with possible responses: “0,” “1,” “2,” “3 or more times.” In addition, we asked people to report about their overall health (excellent to poor), age, gender, education, employment status, and race/ethnicity.

Focus Group #1 (“Normal” sleepers)

Five females (4 white, 1 African American; 18-64 years old) participated. Self-rated health was either good (n = 2), very good (n = 1), or excellent (n = 2). As expected from a group of “normal” sleepers, all 5 people reported falling asleep within 15 minutes and none of the time having trouble falling asleep in the past 4 weeks (MOS Sleep Scale). They generally woke early and enjoyed the morning, and self-reported sleep time of 6-8 hours to feel rested and refreshed. Scores from the Epworth Sleepiness Scale³² ranged from 4 to 12 (4, 6, 7, 8, 12) with participants perceiving the chance of dozing most commonly while watching TV or lying down to rest in the afternoon.

In general the normal sleeper group felt their sleep was deep and restful. Most mentioned some bedtime ritual to set the mood for falling asleep such as listening to music or watching television. Others mentioned using a fan or other “white” noise to fall asleep, and reading something light. It was also noted that weekend sleep patterns were different from weekday ones: bedtimes were often later and participants slept longer.

When participants were asked to describe a good night's sleep, the general description was of falling to sleep easily, sleeping deeply without waking, and feeling rested upon awakening. The awakening experience varied across the group. Some arise immediately feeling “ready to go,” while others feel energized after they shower. Others felt that stretching upon wake gets the body going. A few indicated that they linger in bed for a period before getting up, a habit others perceived as a luxury reserved for weekends. Participants felt that they had slept well if they awoke feeling rested and energetic. Most participants agreed that after a good night's sleep they feel physically better and healthier, their mind is clear, and they feel refreshed. There is a sense of vitality and alertness during the day coming from good sleep. Participants felt they were more functional and could get through a heavy schedule when they slept well.

A bad night's sleep, on the other hand, results in awakening with a feeling of tiredness that lasts throughout the day. Some arise with aches and pains. Physically they feel numb, groggy, heavy headed, lethargic, non-reflexive, and “spacey.” They report having headaches, being thirsty, and feeling “scratchy eyed.” Most agreed that a bad night's sleep leads to behaviors during the day that they would otherwise avoid, such as wanting coffee, overeating or eating improperly, and not exercising. For the more avid exerciser, a workout when tired does not have the same renewing and re-energizing quality as when refreshed. This group also expressed mood changes resulting from a bad night's sleep, such as being grouchy and impatient with others, not being cooperative or helpful, and not being pleasant to be around.

All participants agreed that sleeping well consistently was necessary for good health. A person who does not sleep well is under continual physical and mental stress. Each reported a belief that good sleep was essential to the proper functioning of judgment, concentration, and productivity.

Table S1

Average product-moment correlation between patient reported outcome measures and RSQ-W from five in-clinic measurements in the nonrestorative sleep study

Table S2

Correlations^a sf RSQ-W, and RSQ-D scores with summary polysomnographic (PSG) measures and with the subjective sleep and evening functioning scale

Focus Group #2 (“unrefreshed” sleepers)

Five males and four females (8 white, 1 African American; 18-64 years old) participated in this group. Self-rated health was fair (n = 2), good (n = 4), or very good (n = 3).

This group reported needing 9-12 hours of sleep to feel refreshed, more often awoke too early or too late, and most had a latency to sleep of 16-45 minutes. Only one person reported falling asleep within 15 minutes, and the modal response to having trouble falling asleep in the past 4 weeks on the MOS Sleep Scale was a good bit of the time. The modal response for the past 4 weeks was between most of the time and a good bit of the time to feel that sleep was not quiet, a good bit of the time to get enough sleep to feel rested upon waking in the morning, some of the time to awakening short of breath or with a headache, most of the time to feeling drowsy or sleepy during the day, a good bit of the time to awakening during sleep and having trouble falling asleep again, a little of the time to have trouble staying awake during the day, none of the time to snoring, some of the time to taking naps during the day, and some of the time to getting the amount of sleep they needed. Two people reported waking up one time, five two times, and two people three or more times during sleep. Scores from the Epworth Sleepiness Scale³² ranged from 3 to 17 (3, 3, 4, 4, 5, 8, 10, 14, 17).

Focus Group #3 (“unrefreshed” sleepers)

Three males and seven females (7 white, 1 Asian, 1 American Indian, 1 multiracial; 25-74 years old) participated. Self-rated health was fair (n = 1), good (n = 3), very good (n = 5), or between very good and excellent (n = 1). Three people reported falling asleep with 15 minutes and the modal response to having trouble falling asleep in the past 4 weeks was a good bit of the time (MOS Sleep Scale). For the question “feel that sleep was not quiet,” the modal response for the past 4 weeks was some of the time. For “feel rested upon waking in the morning,” the modal response was some of the time. For the question, “feeling drowsy or sleepy during the day,” the modal response was some of the time. For “awakening during sleep and having trouble falling asleep again,” the modal response was a little of the time. Three people reported waking up one time, one 1.5 times, four two times, and two people three or more times during sleep. Scores from the Epworth Sleepiness Scale³² ranged from 5 to 13 (5, 6, 7, 8, 8, 8, 8, 9, 9, 13).

Participants in the self-identified unrefreshing sleep groups generally were in agreement that they did not experience what they considered to be a typical night's sleep. Rather, their sleep patterns either went in cycles or varied from night to night. This is in distinct contrast to the “normal” sleepers' focus group participants who typically experienced a routine night's sleep.

When the problem sleeper participants were asked to describe the nature of their sleep problems, three areas of sleep problems were revealed: trouble falling to sleep, trouble staying asleep, and awaking too early. In contrast, normal sleepers rarely or only occasionally have instances of trouble falling asleep, staying asleep, or waking too early.

The inability to sleep continuously throughout the night appears to be a nearly universal problem for these participants. Occasionally some get no sleep at all and have had periods like this that have lasted for several consecutive days. Most in the group spoke of awaking during their sleep period and not being able to get back to sleep within what they felt was a reasonable amount of time. Some individuals mentioned being hyper-vigilant throughout the night, awakening every few hours for no apparent reason, tossing and turning in their sleep, or being restless in their sleep. Light in the room or noises on the street or from neighbors cause some individuals to awaken; for others it was for a trip to the bathroom. Others mentioned talking in their sleep. The few who did report sleeping through the night felt that they awoke earlier than they believed they should.

While the sleep patterns and experiences of the normal and the problem groups differed, their perceptions of the impact of both a bad night's sleep and a good one's did not. When asked about the next day effects of a bad night of sleep, problem sleepers mentioned feeling and looking beat up, eyes looking tired, having headaches and aches and pains, feeling dehydrated, skin looking bad, and generally feeling unhealthy. A few further indicated that without adequate sleep they became physically ill. Participants also mentioned feeling that they were clumsy and poorly coordinated, foggy thinking, sluggish, dragging through the day, on automatic pilot, forgetful, and unable to think clearly and make decisions.

A few participants discussed feeling “wired,” anxious, or “on edge” all the next day. In spite of their hyperalertness, these individuals had considerable difficulty coping with their daily activities. They found it challenging to hold conversations and to speak properly, to think clearly, and tended to rush through things without being able to perform well.

Lack of sleep affected the mood of problem sleepers much as it does normal sleepers. Problem sleepers reported being irritated by the littlest things, feeling grouchy or cranky, and were more impatient with others. Further, they lack motivation, feel disenfranchised from the world, suffer depressed mood, and feel withdrawn. A common feeling among participants who lacked sleep was that they were not themselves the next day but rather moved around as if in a daze or robotically.

Although these participants had many problems with their sleep, they could easily report on their experience of a good night's sleep. As would be expected, for these people a good night's sleep entails falling asleep readily, staying asleep ideally throughout the night, or if awakening, being able to go back to sleep rapidly, and waking at a reasonable time feeling refreshed.