Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2012 Feb 1.
Published in final edited form as: J Am Med Dir Assoc. 2010 Oct 2;13(2):127–135. doi: 10.1016/j.jamda.2010.05.004

Can Standardized Sleep Questionnaires be Used to Identify Excessive Daytime Sleeping in Older Post-Acute Rehabilitation Patients?

Megan Skibitsky 1, Maria Orlando Edelen 2, Jennifer L Martin 1,3, Judith Harker 3, Cathy Alessi 1,3,4, Debra Saliba 1,2,3,4
PMCID: PMC3128693  NIHMSID: NIHMS202562  PMID: 21450184

Abstract

OBJECTIVES

Excessive daytime sleeping is associated with poorer functional outcomes in rehabilitation populations and may be improved with targeted interventions. The purpose of this study was to test simple methods of screening for excessive daytime sleeping among older adults admitted for post-acute rehabilitation.

DESIGN

Secondary analysis of data from two clinical samples.

SETTING

Two post-acute rehabilitation (PAR) units in southern California.

PARTICIPANTS

Two hundred twenty-six patients aged > 65 years with Mini-Mental State Examination (MMSE) score > 11 undergoing rehabilitation.

INTERVENTIONS

N/A

MEASUREMENTS

The primary outcome was excessive daytime sleeping, defined as greater than 15% (1.8 hours) of daytime hours (8AM to 8PM) sleeping as measured by actigraphy.

RESULTS

Participants spent, on average, 16.2% (SD 12.5%) of daytime hours sleeping as measured by actigraphy. Thirty nine percent of participants had excessive daytime sleeping. The Pittsburgh Sleep Quality Index (PSQI) was significantly associated with actigraphically-measured daytime sleeping (p= 0.0038), but the Epworth Sleepiness Scale (ESS) was not (p = 0.49). Neither the ESS nor the PSQI achieved sufficient sensitivity and specificity to be used as a screening tool for excessive daytime sleeping. Two additional models using items from these questionnaires were not significantly associated with the outcome.

CONCLUSIONS

In an older PAR population, self-report items from existing sleep questionnaires do not identify excessive daytime sleeping. Therefore we recommend objective measures for the evaluation of excessive daytime sleeping as well as further research to identify new self-report items that may be more applicable in PAR populations.

Keywords: sleep, post-acute care, rehabilitation, screen

INTRODUCTION

Sleep disturbances are common among older adults residing in both community and institutional settings.1,2 Daytime sleep disturbance is particularly problematic in the older population as daytime sleepiness has been associated with limitation both in activities of daily living (ADL) 3,4 and in cognitive functioning.5,6 In addition, daytime sleepiness has been associated with increased incidence of cardiovascular disease7 and may be a marker for disease burden leading to overall mortality.8,9,10,11 In the post-acute rehabilitation (PAR) population, increased daytime sleeping has been associated with less functional recovery.12

Behavioral and environmental interventions have been shown to decrease excessive daytime sleeping in institutional settings, and approaches that address excessive time in bed during the daytime (in addition to other measures) may be particularly successful in decreasing daytime sleeping.13 Thus, identification of those PAR patients with problematic daytime sleeping would permit appropriate targeting of these interventions. The Minimum Data Set 2.0 (MDS) includes three questions relating to staff observation of resident sleep habits. However, there is poor correlation between these MDS sleep items and objective measures of sleep, indicating that long-term care facilities fail to detect sleep disturbances in their resident populations.14

Unfortunately, most studies that measure daytime sleep use tests which are not routinely available in PAR facilities. Traditionally, the Multiple Sleep Latency Test (MSLT) has been used clinically as an objective assessment of daytime sleepiness in non-institutional populations. In the MSLT, the subject is given 5 nap opportunities at 2 hour intervals and the tendency to fall asleep is measured. The Maintenance of Wakefulness Test (MWT) measures the ability of the subject to remain awake during 4 periods at 2 hour intervals. Both tests require carefully controlled conditions, specially trained and dedicated personnel to administer and interpret the tests, as well as polysomnography (including electroencephalography, electrooculography and chin electromyography), 15, 16, 17, 18 all of which are difficult to obtain in PAR settings. In addition, the tests require interruption of patient routines and the populations for which the tests were developed are vastly different from the population admitted for post-acute care rehabilitation, which is generally older, with more medical comorbidities, and more medication use.

There are two objective measures of sleep which have been used in institutional settings. The most widely used in nursing home sleep research is actigraphy. Actigraphy measures limb movement over time using a portable device. It does not measure sleep per se, but rather employs standardized algorithms to translate movement data into estimates of sleep/wake. It has been validated as an estimate of nighttime sleep parameters in comparison to polysomnography (PSG) both in community dwelling adults and in older nursing home residents.19, 20,21 The American Academy of Sleep Medicine has indicated that actigraphy can be used for assessment of sleep and circadian rhythm patterns in older nursing home residents where PSG is not feasible.22 Few, if any, post-acute care facilities routinely use PSG equipment to measure sleep.

Another objective modality for assessing daytime sleep in institutional settings that has been used in research studies is timed observation of sleep versus wakefulness by trained observers. In this approach, observations are performed for a specified amount of time at pre-determined intervals, such as 1 minute of observation every 15 minutes, with sleep defined as “eyes closed with no purposeful activity.” 23 The benefit of this method is that it requires no specialized equipment and is less likely to interfere with daytime routines of the patient than the MSLT and MWT protocols. Unfortunately, these research observations are also time consuming and require significant staff resources, and use has been largely limited to research studies.

An alternative to objective measurement of excessive daytime sleep is resident self-report. In outpatient settings, patient unstructured self-report of sleepiness may not always agree with objective measures of daytime sleeping. 24 An alternative to unstructured self-report is the use of standardized questions to elicit information about sleep. Two such questionnaires are the Pittsburgh Sleep Quality Index (PSQI) and the Epworth Sleepiness Scale (ESS). 25, 26 The performance of these questionnaires has been considered acceptable for initial screening in younger, community dwelling populations, but they were not specifically developed for use in older adults and those who are institutionalized. In addition, while the PSQI contains some items relating to daytime sleep, it is largely focused on nighttime sleep disturbance. The ESS does address daytime sleepiness, but the full scale contains several questions that are not applicable to inpatient PAR populations (e.g., likelihood of falling asleep while driving). In summary, although resident self-report using standardized interviews has been used to identify other conditions, 27, 28, 29 the performance of standardized self-report questionnaires to screen for excessive daytime sleeping has not been well characterized in PAR and other inpatient or long-term care populations.

Given the evidence that excessive daytime sleeping during PAR is associated with less functional recovery with rehabilitation as well as the potential for improved sleep patterns with behavioral interventions, the focus of this study was to evaluate the performance of available self-report questionnaires in measuring daytime sleeping in a PAR population. We hypothesized that resident self report would be comparable to objective measures and provide a feasible approach for identifying residents at risk of excessive daytime sleeping.

In this study we used actigraphy as the objective measure of daytime sleep. We used the PSQI, the ESS, and one additional screening question developed for use in our prior work as self-report measures of sleep disturbance. An effective screening tool composed of self-report items would not necessarily identify the etiology of daytime sleeping, but could be used to screen and target patients for further evaluation or for behavioral interventions to decrease excessive daytime sleep in this setting.

METHODS

Selection of subjects

We conducted a secondary analysis of data from two samples of residents undergoing inpatient PAR.30 Residents were recruited from two facilities in southern California. The first (Facility A) was a freestanding, for-profit, community nursing facility with 130 Medicare-certified beds that focused on rehabilitation. The second (Facility B) was a rehabilitation unit within a Veterans Administration (VA) Medical Center. Patients were recruited from September 2002 through April 2004. Patients were screened for inclusion if they were 65 years of age or older, were admitted for rehabilitation, and had not been living in a NH prior to admission. All patients who met inclusion criteria during the recruitment period were invited to participate in the study. Exclusions were made in two phases. For the original study from which these data were taken, patients were excluded if they were deemed too ill to participate based on abnormal vital signs or severe pain; were transferred, discharged, or died within 1 week of admission; posed a threat to research staff due to a severe behavioral disturbance; were not screened within 1 week of admission; or declined to participate. In addition, for the purpose of the analyses reported here, participants who had severe cognitive impairment (defined as Mini Mental State Exam [MMSE] score < 12) were excluded, as were participants who had incomplete MMSE or missing actigraphy data. The University of California, Los Angeles Office for the Protection of Research Subjects and the Veterans Administration Greater Los Angeles Healthcare System Institutional Review Board approved all research methods and informed consent was obtained from all study participants.

Data Collection

Descriptive measures

The Mini-Mental State Examination (MMSE) was used to assess cognitive function.31 We used the 15-item Geriatric Depression Scale (GDS-15) to screen for symptoms of depression.32 The Cumulative Illness Rating Scale for Geriatrics (CIRS-G) was used to assess baseline disease burden (comorbidity).33 The CIRS-G scores severity of illness in each of 14 organ systems based on a structured medical record review and brief physical examination. Disease in each organ system is scored from 0–4, with higher score indicating more severe illness. We also collected medical record data on reasons for admission to rehabilitation.

Self-Report Sleep Measures

Two patient self-report measures, the Pittsburgh Sleep Quality Index (PSQI) and Epworth Sleepiness Scale (ESS) were used to assess sleep disturbance. The PSQI is an 18-item self-report measure of sleep disturbance that asks individuals to describe their sleep, such as hours slept per night or difficulty staying awake during daily activities, as well as to describe the frequency of specific causes of troubled nighttime sleep such as uncomfortable breathing, pain, or nocturia. The PSQI items are grouped into seven domains and each domain is scored from 0–3 based on a scoring algorithm, with a total possible score of 21, with scores above 5 indicative of a sleep disturbance.34 The ESS contains 8 items and focuses on daytime sleepiness. The ESS asks patients to rate their likelihood of dozing during specific activities such as watching TV or reading. Each item is scored from 0 to 3, with 0 corresponding to no chance of dozing, and 1–3 corresponding to slight, moderate, and high chance of dozing, respectively.35 The ESS is scored from 0 to 24, with scores above 10 indicating daytime sleepiness.36 The potential applicability of the ESS in a PAR population may be limited by the fact that two of the eight items ask about sleeping while in car, and by the fact that the ESS has not been well-validated for use in older adults.

For the purpose of identifying potential items for a simple self-report screening instrument for daytime sleeping in this population, candidate items were taken from the ESS and the PSQI. We also considered an additional item (developed by the research team) — “Over the past week, how often did you take a nap during the day?” In total, 25 candidate self-report items were examined (16 PSQI items, 8 ESS items, and 1 additional item). All items were scored from 0–3, with 0 corresponding to no sleep disturbance, and 1, 2, and 3 corresponding to low, medium, and high sleep disturbance, respectively.

Objective Measurement of Sleep

We employed wrist actigraphy as the objective measure of daytime sleeping. This objective monitoring method was selected because it has been widely used in the study of sleep/wake patterns in both clinical care and research among older adults both at home and in nursing home settings. Wrist actigraphs measure movement, which is then used to score sleep and wakefulness using available algorithms. The algorithms used to interpret actigraphy output have been previously evaluated, and correlate highly with EEG-measured sleep (i.e., the gold-standard assessment) among residents of nursing homes (Morgenthaler et al 22 and Ancoli-Israel et al 21). Wrist actigraphy also has been shown to be sensitive to change in nighttime sleep variables with treatment interventions for insomnia in older adults. 22 Since it was not feasible to perform traditional MSLT in this setting, the current study used wrist actigraphy as the objective measure for sleep.

The outcome of interest in the current study was excessive daytime sleeping (by wrist actigraphy), defined as greater than 15% (1.8 hours) time sleeping between 8AM and 8PM. This criterion was based on data that this amount of daytime sleep is likely to interfere with participation in rehabilitation and other activities and may also interfere with nighttime sleep quality.37,38 Percent of nighttime hours spent asleep was also measured, but was not used for the purpose of this analysis.

Participants wore a wrist actigraph (Octagonal-L, Ambulatory Monitoring, Inc., AMI, Ardsley, NY) on their non-dominant arm (unless that arm was paralyzed in which case the dominant arm was used) for 24 hours a day for one week (mean = 6.4 days, SD 1.7 days, range 1–11 days). Actigraphy data was reviewed to eliminate artifact and sleep was scored using a validated algorithm with ACT software.

Statistical Methods

Analyses were carried out using SAS 9.1.39 We used analysis of variance (ANOVA) to test for differences among participants in the two facilities. For actigraphy measures, daytime was defined as the period between 8AM and 8PM. Actigraphy variables were averaged over 7 days. Logistic regression was used to investigate the relationship between the 25 individual questionnaire items and the outcome. Questionnaire items were treated as continuous variables (possible scores were discrete values of 0, 1, 2, or 3). The outcome was a dichotomous variable for excessive daytime sleeping, defined as greater than 15% of daytime hours spent sleeping as measured by actigraphy. Covariates included in the models were age and gender. Results from logistic regression of individual items were used for model selection as described below.

We examined the performance of the PSQI and ESS to identify excessive daytime sleeping. We conducted logistic regression in which the predictors were the total scores of the PSQI and ESS and the outcome was excessive daytime sleeping. We also considered a modified ESS that did not contain the two items which ask about likelihood of falling asleep in a car, since these are not applicable in an institutionalized PAR population.

To identify a potentially briefer set of items that could be more readily collected and scored by institutional staff and that might better target daytime sleep in the PAR population, we used two strategies. In the first selection strategy, we constructed a model including all items that had p-values of regression coefficients ≤ 0.2 in the logistic regression of individual items. This criterion was chosen because 24 of the 25 items were derived from clinically validated instruments and we did not want to exclude any items with possible predictive value. We narrowed the items in this model through sequential modeling based on the area under the receiver operating characteristics curve (AUC) of the models. A perfect diagnostic test with both sensitivity and specificity of 100% would have an AUC of 1.0. Items were removed in order of descending p-value of the regression coefficient in the logistic regression models of the individual items. If removal of an individual item did not decrease the AUC then the item was permanently eliminated. If removal of an item decreased the AUC, then the item was determined to add to the predictive ability of the set and the item was retained. We repeated this process until all items had been tested, and an alternative optimal set was identified. The p-value of the regression coefficient for this set was obtained and the model AUC calculated.

In the second selection strategy, we considered a model that contained only those of the 25 items which pertained to daytime sleeping, were applicable in a PAR population, and could be easily scored by staff using pen and paper. Items meeting these criteria were combined into one model and the item set was narrowed using the same sequential modeling procedure as in the first strategy described above. Again, the final item set was scored as the sum of all individual items, and the p-value and AUC for the set score were obtained. In both selection strategies, covariates included in the models were age and gender.

Sensitivity and specificity for the statistically significant item sets were calculated for all possible scores by constructing two by two tables for excessive daytime sleeping for each possible score. Since the goal was identifying the best self-report screener that would alert clinicians to the need for additional evaluation, optimal sensitivity and specificity are reported for the cut score which yielded the highest sensitivity.

In order to better understand the performance of the models, we investigated performance of the questionnaire sets in sub-groups of our study population. We assessed performance (logistic regression modeling, AUC, and sensitivity and specificity parameters) for cognitively intact (MMSE ≥ 24) versus cognitively impaired (MMSE=12–24), less ill (CIRS-G < 27) versus more ill (CIRS-G ≥ 27), and depressed symptoms (GDS-15 > 5) versus without depressed symptoms (GDS-15 ≤ 5).

RESULTS

Study sample

Figure 1 describes the flow of participants through the study. Of 1128 individuals over age 65 who were admitted to the two study facilities during the larger study’s enrollment period, 1094 accepted screening. Of the 839 who met eligibility criteria, 261 (31%) agreed to participate in interviews and actigraphy placement. Those who declined had an average age of 82.8 (SD 7.8) and were 60% female, compared to an average age of 79.9 (SD 7.2) and 65% male in those enrolled in the study (due to a higher participation rate in the Veterans Administration study site compared to the community study site). For the current study, an additional 35 participants were excluded because of MMSE score < 12 (n=10) and/or missing data (n=26). All 226 participants in the remaining analytic sample answered the study survey that included the 25 candidate sleep items, although 11 participants had some missing items on the Epworth Sleepiness Scale (ESS) and 39 participants had some missing items on the Pittsburgh Sleep Quality Index (PSQI).

Figure 1.

Figure 1

Flow of participants through the study.

*Patients were not eligible if they were being admitted for hospice care (6), respite (6), medical treatment only (15) or nursing home placement (3); if were a prior nursing home resident (13); or died (3), were discharged (136) or were not identified (9) within one week of admission to post-acute rehabilitation; or if they were judged too ill (19) or with severe behavioral disorder (3). In addition, 45 patients were excluded for other communication difficulties (e.g., non-English and non-Spanish speaking).

** Participants of the larger study were excluded from the current study if they had MMSE <12 (10), missing MMSE data (15), missing sleep questionnaire data (6), or missing daytime actigraphy data (5).

Sample characteristics for each facility and the total sample are shown in Table 1. Reasons for admission to rehabilitation were orthopedic (42.9%), cardiac (13.7%), neurologic (9.3%), debility (8.0%), pulmonary (3.1%), and other conditions (23.0%). Based on actigraphy data, participants spent, on average, 16.2% (SD 12.5%) of daytime hours asleep, and 38.9% of participants had excessive daytime sleeping as defined by greater than 15% of daytime hours (8AM to 8PM) spent sleeping. In addition, 16.3% had ESS scores indicative of daytime sleepiness (ESS > 10), and 70.1% had PSQI scores indicative of a sleep disturbance (PSQI > 5).

Table 1.

Sample Characteristics (SD)1

Facility A Facility Combined
n = 130 B n = 96 n = 226
Age* 81.3 (7.0) 77.9 (7.0) 79.9 (7.2)
Male* 43.1% 93.8% 64.6 %
Caucasian* 92.3% 60.4% 78.8 %
Self-rated health good, very good, or excellent 52.8% 53.8% 53.2%
Length of stay (days) in rehabilitation unit* 24.2 (11.5) 15.8 (8.1) 20.6 (11.0)
Length of stay (days) in hospital prior to admission 11.4 (10.2) 10.5 (24.0) 11.0 (16.8)
MMSE 23.9 (5.2) 25.1 (4.6) 24.2 (5.0)
GDS-15 4.3 (3.5) 4.0 (3.1) 4.1 (3.3)
CIRS-G 22.3 (5.8) 22.8 (5.6) 22.5 (5.7)
ESS* 4.7 (4.3) 6.6 (5.1) 5.6 (4.8)
PSQI 8.3 (4.6) 8.9 (4.4) 8.5 (4.5)
Number of medications* 12.4 (5.1) 16.1 (5.4) 13.9 (5.5)
1

All values are means (standard deviation) unless reported as percent.

*

ANOVA testing indicated a significant difference at the p<0.05 level between facilities

Testing of Screening Questionnaires

The associations between the Epworth Sleepiness Scale (ESS), modified ESS, Pittsburgh Sleep Quality Index (PSQI) and actigraphic evidence of excessive daytime sleeping are shown in Table 2. Neither the ESS or modified ESS total scores were significantly associated with daytime sleeping in this sample (p = .49 and .24, respectively). In contrast, the PSQI was significantly associated with excessive daytime sleeping (p-=.0038). The area under the receiver operating characteristics curve (AUC) for the PSQI model was 0.634. Sensitivity and specificity of PSQI for identifying excessive daytime sleeping is shown in Table 3. Maximum sensitivity was 50% with corresponding specificity 61.2%.

Table 2.

Logistic Regression of Candidate Models to Identify Excessive Daytime Sleeping in an Older PAR Population.

Model Subgroup Parameter estimate Standard error Wald Chi- Square Prob > ChiSq Adjusted R2
7 Item Set (Strategy 1) Overall −0.004 0.044 0.009 0.92 0.0002
Cognitively intact (MMSE ≥ 24)
n= 159
0.010 0.054 0.033 0.87 0.001
Cognitively impaired (MMSE < 24)
n= 67
0.001 0.084 0.000 0.99 0.020
Less illness burden (CIRS-G < 27)
n= 167
0.051 0.054 0.991 0.35 0.1012
Greater illness burden (CIRS-G ≥ 27)
n= 59
−0.171 0.101 2.840 0.09 0.189
Less depressive symptoms (GDS-15 ≤ 5)
n= 162
−0.048 0.054 0.786 0.38 0.010
More depressive symptoms (GDS-15 > 5)
n= 64
0.090 0.084 1.145 0.28 0.033

6 Item Set (Strategy 2) Overall 0.1502 0.304 0.121 0.710 0.026
Cognitively intact (MMSE ≥ 24)
n= 159
0171 0.092 3.423 0.06 0.031
Cognitively impaired (MMSE < 24)
n= 67
0.123 0.135 0.839 0.36 0.037
Less illness burden (CIRS-G < 27)
n= 167
0.217 0.093 5.476 0.02 0.055
Greater illness burden (CIRS-G ≥ 27)
n= 59
0.016 0.139 0.013 0.91 0.084
Less depressive symptoms (GDS-15 ≤ 5)
n= 162
0.203 0.093 4.82 0.03 0.047
More depressive symptoms (GDS-15 > 5)
n= 64
0.040 0.133 0.089 0.77 0.027

PSQI Overall −0.104 0.036 8.379 0.00 0.064
Cognitively intact (MMSE ≥ 24)
n= 159
−0.078 0.041 3.504 0.06 0.036
Cognitively impaired (MMSE < 24)
n= 67
−0.158 0.081 3.732 0.05 0/127
Less illness burden (CIRS-G < 27)
n= 167
−0.086 0.042 4.198 0.04 0.045
Greater illness burden (CIRS-G ≥ 27)
n= 59
−0.171 0.084 4.203 0.04 0.198
Less depressive symptoms (GDS-15 ≤ 5)
n= 162
−0.118 0.417 8.032 0.00 0.089
More depressive symptoms (GDS-15 > 5)
n= 64
−0.067 0.073 0.837 0.36 0.023

ESS Overall 0.021 0.030 0.471 0.49 0.004
Cognitively intact (MMSE ≥ 24)
n= 159
0.014 0.036 0.162 0.69 0.005
Cognitively impaired (MMSE < 24)
n= 67
0.052 0.062 0.685 0.41 0.024
Less illness burden (CIRS-G < 27)
n= 167
0.027 0.036 0.562 0.45 0.127
Greater illness burden (CIRS-G ≥ 27)
n= 59
0.009 0.060 0.021 0.89 0.077
Less depressive symptoms (GDS-15 ≤ 5)
n= 162
0.009 0.037 0.054 0.82 0.004
More depressive symptoms (GDS-15 > 5)
n= 64
0.027 0.057 0.220 0.64 0.044

Modified ESS Overall 0.042 0.036 1.361 0.24 0.009
Cognitively intact (MMSE ≥ 24)
n= 159
0.037 0.043 0.761 0.38 0.009
Cognitively impaired (MMSE < 24)
n= 67
0.072 0.075 0.901 0.34 0.029
Less illness burden (CIRS-G < 27)
n= 167
0.055 0.043 1.621 0.23 0.022
Greater illness burden (CIRS-G ≥ 27)
n= 59
0.010 0.074 0.020 0.89 0.087
Less depressive symptoms (GDS-15 ≤ 5)
n= 162
0.019 0.044 0.189 0.66 0.005
More depressive symptoms (GDS-15 > 5)
n= 64
0.071 0.071 0.997 0.32 0.051

Table 3.

Sensitivity and Specificity of PSQI to Identify Excessive Daytime Sleeping in an Older PAR Population.

Cut Score Sensitivity Specificity
≥ 1 (N=185) 38.4 58.5
≥ 2 (N=176) 37.5 56.0
≥ 3 (N=170) 36.5 53.6
≥ 4 (N=162) 35.2 51.6
≥ 5 (N=144) 32.6 50.0
≥ 6 (N=131) 32.8 52.6
≥ 7 (N=119) 31.1 52.3
≥ 8 (N=105) 30.5 53.7
≥ 9 (N=88) 33.0 57.2
≥ 10 (N=80) 30.0 56.2
≥ 11 (N= 68) 27.9 56.3
≥ 12 (N=59) 23.7 55.7
≥ 13 (N=39) 20.5 57.2
≥ 14 (N=25) 24.0 59.2
≥ 15 (N=17) 23.5 59.8
≥ 16 (N=12) 25.0 60.3
≥ 17 (N=8) 25.0 60.6
≥ 18 (N=4) 25.0 60.8
= 19 (n=2) 50.0 61.2
*

A total of 187 participants had complete PSQI scores; 2 participants had a PSQI total score of 0.

Results from logistic regression of the 25 individual screening items are shown in Table 4. Four of the 25 items were significantly associated with excessive daytime sleeping.

Table 4.

Logistic Regression of Individual Items to Identify Excessive Daytime Sleeping.

Item Scale Parameter Estimate Standard Error Wald Chi- Square Pr > ChiSq
1. Over the past week, when did you usually go to bed at night?*+ PSQI −0.257 0.121 4.492 0.03
Over the past week, when did you usually get up in the morning? *+ PSQI
Over the past week, how many hours of actual sleep did you get at night? *+ PSQI
2. Over the past week, how long (in minutes) did it usually take you to fall asleep each night? * PSQI −0.199 0.144 1.920 0.17
3. Over the past week, how often have you had trouble sleeping because you…
Cannot get to sleep within 30 minutes?**
PSQI −0.111 0.103 1.157 0.28
4. Wake up in the middle of the night or early morning? ** PSQI −0.039 0.103 0.144 0.70
5. Have to get up to use the bathroom? ** PSQI −0.082 0.112 0.537 0.46
6. Cannot breathe comfortably? ** PSQI −0.070 0.147 0.229 0.63
7. Cough or snore loudly? ** PSQI 0.181 0.177 1.043 0.31
8. Feel too cold? ** PSQI 0.089 0.142 0.392 0.53
9. Feel too hot? ** PSQI 0.077 0.184 0.177 0.67
10. Had bad dreams? ** PSQI 0.175 0.194 0.818 0.37
11. Have pain? ** PSQI −0.047 0.114 0.169 0.68
12. Other reason? ** PSQI 0.030 0.111 0.070 0.79
13. Over the past week, how often have you taken medicine to help you sleep? **+ PSQI −0.230 0.102 5.014 0.03
14. Over the past week, how often had you had trouble staying awake while eating meals, talking with others, or engaging in social activity? ** §+ PSQI 0.310 0.187 2.731 0.10
15. Over the past week, how would you rate your sleep quality overall? **+ PSQI −0.269 0.155 3.018 0.08
16. Over the past week, how much of a problem has it been for you to keep up enough enthusiasm to get things done? ** § PSQI −0.048 0.137 0.124 0.73
17. Over the past week, how often did you take a nap during the day? **+§ Investigators 0.253 0.117 4.697 0.03
18. How likely are you to doze off and fall asleep if you are…
Sitting and reading?*** §
ESS −0.005 0.132 0.002 0.97
19. Watching TV? ***§ ESS −0.022 0.127 0.031 0.86
20. Sitting inactive in a public place? ***+§ ESS 0.329 0.157 4.389 0.04
21. As a passenger in a car for an hour without a break? *** ESS −0.136 0.147 0.857 0.35
22. Lying down to rest in the afternoon when circumstances permit? ***+§ ESS 0.218 0.128 2.895 0.09
23. Sitting and talking to someone? ***§ ESS 0.007 0.234 0.001 0.98
24. Sitting quietly after a lunch without alcohol? ***§ ESS 0.214 0.148 2.093 0.15
25. In a car, while stopped for a few minutes in *** Traffic? ESS −0.132 0.309 0.183 0.67
*

These items require an algorithm to convert to categorical 0 to 3 score. The first item is a composite item with 3 parts.

**

Possible responses to items were 0= Not at all, 1= Once a week, 2= Twice a week, or 3= 3 or more times a week.

***

Possible responses to items were 0= Would never doze, 1= Slight chance of dozing, 2= Moderate chance of dozing, or 3= High chance of dozing.

+

Items in the final item set from strategy 1 in which all items with p-value ≤ 0.2 (Items 1, 2, 13, 14, 15, 17, 20, 22, and 24) were included in a model and the model was narrowed through a sequential narrowing process.

§

Items in the final item set from strategy 2 in which all items related to daytime sleep, applicable in PAR populations, and easily scored by staff (Items 14, 16, 17, 18, 19, 20, 22, 23, and 24) were included in a model and the model narrowed through a sequential modeling process.

Results from the two model selection strategies to identify a briefer set of items are shown in Tables 2 and 4. The first model selection strategy included 9 items that met the criteria of p-value of regression coefficient for individual item = 0.2; 7 items remained after the model was narrowed through a process of sequential modeling. Scores on this 7-item set ranged from 0 to 18, but the model was not statistically significant (p=.92). The second model selection strategy included 9 items that met the criteria of being applicable in a PAR population, pertaining to daytime sleep, and easily scored; 6 items remained after the model was narrowed. Scores on this 6-item set ranged from 0 to 15 (Model p = .06).

To test whether the relatively poor performance of the self-report analyses could be related to cognitive impairment, illness, or depression we stratified the sample based on MMSE score, CIRS-G score, and GDS-15 score. Results from the subgroup analysis are shown in Table 2. The PSQI was not significantly associated with excessive daytime sleeping in the groups with higher MMSE score (cognitively intact) or higher GDS-15 score (more depressive symptoms). In participants with less depressive symptoms (scores ≤ 5 on the GDS-15), the maximum sensitivity of the PSQI was 50% with corresponding specificity 63.8%. Neither the ESS nor the modified ESS was significantly associated with the outcome in any of the subgroups. The 6-item set developed with the second selection strategy was significantly associated with excessive daytime sleeping in the subgroup of participants with less illness burden (p= 0.03); area under the receiver operating characteristics curve (AUC) in this subgroup was 0.627. Sensitivity and specificity of the 6-item set for identifying excessive daytime sleeping in the less ill subgroup is shown in Table 5. Maximum sensitivity in this subgroup was 53.3%. Pearson correlation coefficients between the models investigated are shown in Table 6. Interestingly, the PSQI was the only model which showed a significant correlation with excessive daytime sleeping in the overall sample, and this model was significantly associated with the 7-item set developed from selection strategy 1 but was not significantly associated with the 6-tem set developed from selection strategy 2 nor with the ESS models. There was a significantly positive (but weak) correlation between time spent sleeping at night and time spent sleeping during the day (Pearson correlation coefficient 0.38, p <0.05).

Table 5.

Sensitivity and Specificity of 6-Item Set to Identify Excessive Daytime Sleeping in an Older PAR Population with Less Illness Burden (CIRS-G < 27, n= 167).

Cut score Sensitivity Specificity
≥ 1 (n= 146) 34.2 71.4
≥ 2 (n=141) 35.5 76.9
≥ 3 (n= 131) 34.4 69.4
≥ 4 (n=120) 36.7 74.5
≥ 5 (n=108) 36.1 71.2
≥ 6 (n=87) 40.2 73.8
≥ 7 (n=69) 40.6 71.4
≥ 8 (n=52) 42.3 70.4
≥ 9 (n=36) 47.2 70.2
≥ 10 (n=25) 48.0 69.0
11 (n=15) 53.3 68.4
≥ 12 (n=11) 36.4 66.7
≥ 13 (n=6) 50.0 67.1
=14 (n=5) 40.0 66.7
*

A total of 159 of the 167 participants had complete 6-Item set scores; 13 participants had a total score of 0 on the 6-item set.

Table 6.

Correlations Between Models

7 Item Set (Strategy 1) 6 Item Set (Strategy 2) PSQI ESS Modified ESS
7 Item Set (Strategy 1) Correlation coefficient 0.691 0.626 0.527 0.546
p-value <0.0001 <0.0001 <0.0001 <0.0001
n 183 184 180 181
6 Item Set (Strategy 2) Correlation coefficient 0.077 0.825 0.852
p-value 0.30 <0.0001 <0.0001
n 181 201 210
PSQI Correlation coefficient 0.008 0.011
p-value 0.92 0.88
n 179 180
ESS Correlation coefficient 0.976
p-value <0.0001
n 215

DISCUSSION

To our knowledge, this study represents the first attempt to target screening for daytime sleep disturbance in a PAR population. Of two standardized questionnaires used in community dwelling populations, only the Pittsburg Sleep Quality Index (PSQI) was associated with an objective measure of daytime sleep, and this questionnaire only achieved a maximum sensitivity of 50%, making it an unlikely candidate for routine screening. We also tested whether a subset of items from these questionnaires might perform better in identifying PAR residents with excessive daytime sleeping. Using candidate items from existing questionnaires as the basis of our survey, we used two selection strategies to test whether a reduced set of self-report items could be used to screen for excessive daytime sleeping. Neither of these strategies resulted in a model which achieved statistical significance. Although the 6-item set was significantly associated with the daytime sleeping in a less ill subset of the study population, the operating characteristics (sensitivity and specificity) of the set were not appropriate for screening purposes even in this subgroup.

It was interesting that the ESS, which focuses on daytime sleepiness, was not significantly associated with excessive daytime sleeping among older people undergoing PAR, while the PSQI, which assesses overall sleep quality, was significantly associated with objectively-estimated daytime sleeping. This may be due to the fact that the ESS items are not readily applicable to rehabilitation populations in which patients’ schedules are centered on rehabilitation activities and nursing care. Poor nighttime sleep can contribute to daytime sleep disturbances and this is perhaps the reason that the PSQI total score was significantly associated with daytime sleeping. However, despite this significant association, the PSQI did not achieve the sensitivity and specificity needed for a screening tool in this population. One might expect that more sleep during the night would be correlated with less daytime sleep however our data did not support this hypothesis; the correlation between the two was a positive one. A commonly used level for abnormal nighttime sleep in older adults is less than 80% of time in bed spent sleeping. Based on this definition, 92% of our sample population had abnormal nighttime sleep. The association between day and nighttime sleep in the rehabilitation population is an interesting one and merits further investigation.

One possible reason for the poor performance of the item sets is the heterogeneity of the rehabilitation population. Specifically, the decision to include moderately cognitively impaired residents as respondents might have negatively influenced the association between self-report and objective measures of daytime sleep. When we stratified the analysis between no/mild cognitive impairment and moderate impairment, however, the models did not perform better in the no/mild impairment group. Sleep disturbances may be due to primary sleep disorders or may be secondary to other disease processes. Sleep disturbance is common in depression and this condition might influence the performance of self-report items. This may be the reason that the PSQI was not significantly associated with excessive daytime sleeping in the subgroup with more symptoms of depression. Although the 6-item set was not significantly associated with the outcome in the overall group, it was significant in the subgroup with less illness burden. This suggests that different strategies may be needed to assess daytime sleep disturbance that is due to a primary sleep disorder as opposed to sleep disturbance associated with comorbidities.

There was a low participation rate in our study, with only 31% of eligible patients electing to participate, thereby limiting the generalizability of our findings. Patients are generally admitted to PAR following hospitalization and it is possible that patients did not desire any additional activities during this already demanding time. It is also possible that patient decisions regarding participation were affected by the time of day and manner in which the study was presented. Studies which examine factors influencing research participation of older adults PAR populations would be highly valuable.

In this study, actigraphy was averaged over 6.4 days (SD 1.7 days). It is possible that this data did not capture all of the potential variability in daytime sleep patterns for individual participants as evidenced by a SD of 12.5% in daytime hours spent sleeping. However, this length of time was selected as one that would provide the maximum amount of information on sleep patterns while still being economically feasible.

Despite these limitations, the current study suggests that neither the ESS nor the PSQI are useful as screening instruments to identify excessive daytime sleeping in older PAR populations. In addition, we were unable to identify a subset of items from these questionnaires that performed adequately. Further research is needed to develop a tool that can be used to screen for excessive daytime sleeping among older adults in inpatient settings. Until such a tool is developed, objective measurements of daytime sleeping, such as actigraphy, may be the best available option. While actigraphy is significantly less expensive than full polysomnographic recording of sleep, the cost of actigraphy devices may be a significant barrier to their routine use, as validated devices typically cost $500-$1000 with additional costs for computer interface devices and software. Also, the review and interpretation of actigraphy data requires training and expertise not generally available in clinical settings outside of sleep laboratories. It is conceivable that the personnel time necessary to measure daytime sleep using actigraphy would be less than time to administer and score clinical questionnaires. While timed patient observation is another available measure of daytime sleeping; current protocols require multiple observations over an extended period during the day, which may not be feasible in busy clinical settings with limited staff resources. Future studies might investigate whether a modified protocol requiring fewer observations could be substituted in a PAR population.

Based on our findings, we recommend the investigation and development of new self-report items not currently included on commonly used sleep questionnaires for the identification of excessive daytime sleeping in an older PAR population. We recommend that such items ask about the effect of daytime sleeping on activities which are relevant to rehabilitation. Until a screening tool is developed to identify those patients with excessive daytime sleeping, institutions may consider implementing general, facility-wide procedures to decrease in-bed time during daytime hours, since in-bed time is one of the strongest predictors of excessive daytime sleeping in institutional settings.

CONCLUSIONS

Self-report questionnaires developed to identify sleep disorders in community dwelling populations perform poorly when tested in an older post acute rehabilitation (PAR) population to identify excessive daytime sleeping. Given the fact that excessive daytime sleeping is associated with less functional recovery with rehabilitation and evidence suggests that behavioral interventions can decrease daytime sleeping in institutional settings, it is important to identify rehabilitation patients who suffer from this condition and who could benefit from intervention. Available methods for assessing daytime sleep disturbance include actigraphy and timed observations; however these traditional modalities require the use of monitoring and trained personnel which are not routinely available in PAR settings. Other objective measures of sleep exist but also have significant costs in terms of equipment and/or staff time. Given the current lack of questionnaires that accurately assess daytime sleep, rehabilitation facilities wishing to measure daytime sleeping might wish to weigh the costs of actigraphy versus timed observations performed by staff. Future studies are needed to develop feasible and valid approaches to screen for excessive daytime sleeping in rehabilitation populations.

Acknowledgments

Research supported by the Veterans Administration Health Services Research and Development (SDR 03-217, Saliba; IIR 01-053-1, Alessi; and AIA 03-047, Martin), NIH/NIA K23 AG028452 (Martin), and the VA Greater Los Angeles Healthcare System Geriatric Research, Education, and Clinical Center (GRECC), and the AFAR Medical Student Summer Research Training in Aging Program.

References

  • 1.Foley DJ, Monjan AA, Brown SL, et al. Sleep Complaints Among Elderly Persons: An Epidemiologic Study of Three Communities. Sleep. 1995;18(6):425–432. doi: 10.1093/sleep/18.6.425. [DOI] [PubMed] [Google Scholar]
  • 2.Ganguli M, Reynolds CF, Gilby JE. Prevalence and persistence of sleep complaints in a rural older community sample: the MoVIES project. J Am Geriatr Soc. 1996;44:778–784. doi: 10.1111/j.1532-5415.1996.tb03733.x. [DOI] [PubMed] [Google Scholar]
  • 3.Whitney CW, Enright PL, Newman AB, et al. Correlates of Daytime Sleepiness in 4578 Elderly Persons: The Cardiovascular Health Study. Sleep. 1998;21(1):27–36. doi: 10.1093/sleep/21.1.27. [DOI] [PubMed] [Google Scholar]
  • 4.Hays JC, Blazer DG, Foley DJ. Risk of napping: excessive daytime sleepiness and mortality in an older community population. J Am Geriatr Soc. 1996;44:693–698. doi: 10.1111/j.1532-5415.1996.tb01834.x. [DOI] [PubMed] [Google Scholar]
  • 5.Ohayon MM, Vecchierini MF. Daytime Sleepiness and Cognitive Impairment in the Elderly Population. Arch Int Med. 2002;162:201–208. doi: 10.1001/archinte.162.2.201. [DOI] [PubMed] [Google Scholar]
  • 6.Cohen-Zion M, Stepnowsky C, Marler, et al. Changes in cognitive function associated with sleep disordered breathing in older people. J Am Geriatr Soc. 2001;49:1622–1627. doi: 10.1046/j.1532-5415.2001.t01-1-49270.x. [DOI] [PubMed] [Google Scholar]
  • 7.Newman AB, Spiekerman DF, Enright P, et al. Daytime Sleepiness Predicts Mortality and Cardiovascular Disease in Older Adults. J Am Geriatr Soc. 2000l;48:115–123. doi: 10.1111/j.1532-5415.2000.tb03901.x. [DOI] [PubMed] [Google Scholar]
  • 8.Rockwood K, Davis HS, Merry HR, et al. Sleep disturbances and mortality: results from the Canadian study of health and aging. J Am Geriatr Soc. 2001;49:639–641. doi: 10.1046/j.1532-5415.2001.49125.x. [DOI] [PubMed] [Google Scholar]
  • 9.Fetveit A, Bjorvatn B. Sleep disturbances among nursing home residents. Int J Geriatr Psychiatry. 2002;17:604–609. doi: 10.1002/gps.639. [DOI] [PubMed] [Google Scholar]
  • 10.Jacobs D, Ancoli-Israel S, Parker L, et al. Twenty-four-hour sleep-wake patterns in a nursing home population. Psychol Aging. 1989;4:352–356. doi: 10.1037//0882-7974.4.3.352. [DOI] [PubMed] [Google Scholar]
  • 11.Martin JL, Webber AP, Alam T, et al. Daytime sleeping, sleep disturbance, and circadian rhythms in the nursing home. Am J Geriatr Psychiatry. 2006;14:121–129. doi: 10.1097/01.JGP.0000192483.35555.a3. [DOI] [PubMed] [Google Scholar]
  • 12.Alessi CA, Martin JL, Webber AP, et al. More daytime sleeping predicts less functional recovery among older people undergoing inpatient post-acute rehabilitation. Sleep. 2008 Sep 1;31(9):1291–300. [PMC free article] [PubMed] [Google Scholar]
  • 13.Alessi CA, Martin JL, Webber AP, et al. Randomized, controlled trial of a nonpharmacological intervention to improve abnormal sleep/wake patterns in nursing home residents. J Am Geriatr Soc. 2005;53:803–810. doi: 10.1111/j.1532-5415.2005.53251.x. [DOI] [PubMed] [Google Scholar]
  • 14.Martin JL, Alessi CA. Limited validity of minimum data set items on sleep and hypnotic use in predicting falls and hip fracture in nursing home residents. J Am Geriatr Soc. 2006 Jul;54(7):1150–1. doi: 10.1111/j.1532-5415.2006.00777.x. [DOI] [PubMed] [Google Scholar]
  • 15.Littner MR, Kushida C, Wise M, et al. Standards of Practice Committee of the American Academy of Sleep Medicine. Practice parameters for clinical use of the multiple sleep latency test and the maintenance of wakefulness test. Sleep. 2005 Jan 1;28(1):113–21. doi: 10.1093/sleep/28.1.113. [DOI] [PubMed] [Google Scholar]
  • 16.Arand D, Bonnet M, Hurwitz T, et al. The clinical use of the MSLT and MWT. Sleep. 2005 Jan 1;28(1):123–44. doi: 10.1093/sleep/28.1.123. [DOI] [PubMed] [Google Scholar]
  • 17.Carskadon MA, Dement WC, Mitler MM, et al. Guidelines for the multiple sleep latency test (MSLT): a standard measure of sleepiness. Sleep. 1986 Dec;9(4):519–24. doi: 10.1093/sleep/9.4.519. [DOI] [PubMed] [Google Scholar]
  • 18.Mitler MM, Gujavarty KS, Browman CP. Maintenance of wakefulness test: a polysomnographic technique for evaluation treatment efficacy in patients with excessive somnolence. Electroencephalogr Clin Neurophysiol. 1982 Jun;53(6):658–61. doi: 10.1016/0013-4694(82)90142-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Ancoli-Israel S, Cole R, Alessi C, et al. The role of actigraphy in the study of sleep and circadian rhythms. Sleep. 2003;26:342–392. doi: 10.1093/sleep/26.3.342. [DOI] [PubMed] [Google Scholar]
  • 20.Jean-Louis G, von Gizycki H, Zizi F, et al. Determination of sleep and wakefulness with actigraphic data analysis software (ADAS) Sleep. 1996;19:739–743. [PubMed] [Google Scholar]
  • 21.Ancoli-Israel S, Clopton P, Klauber M, et al. Use of wrist actigraphy for monitoring sleep/wake patterns in demented nursing home patients. Sleep. 1997;20:24–27. doi: 10.1093/sleep/20.1.24. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Morgenthaler T, Alessi C, Friedman L, et al. Standards of Practice Committee; American Academy of Sleep Medicine. Practice parameters for the use of actigraphy in the assessment of sleep and sleep disorders: an update for 2007. Sleep. 2007 Apr 1;30(4):519–29. doi: 10.1093/sleep/30.4.519. [DOI] [PubMed] [Google Scholar]
  • 23.Bliwise DL, Bevier WC, Bliwise NG, et al. Systematic 24-hr behavioral observations of sleep and wakefulness in a skilled-care nursing facility. Psychol Aging. 1990 Mar;5(1):16–24. doi: 10.1037//0882-7974.5.1.16. [DOI] [PubMed] [Google Scholar]
  • 24.McCrae CS, Rowe MA, Tierney CG, et al. Sleep complaints, subjective and objective sleep patterns, health, psychological adjustment, and daytime functioning in community-dwelling older adults. J Gerontol B Psychol Sci Soc Sci. 2005 Jul;60(4):P182–9. doi: 10.1093/geronb/60.4.p182. [DOI] [PubMed] [Google Scholar]
  • 25.Johns MW. A new method for measuring daytime sleepiness: The Epworth Sleepiness Scale. Sleep. 1991;14:540–545. doi: 10.1093/sleep/14.6.540. [DOI] [PubMed] [Google Scholar]
  • 26.Buysse DJ, Reynolds CF, Monk TH, et al. The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research. Psychiatry Research. 1989;28:193–213. doi: 10.1016/0165-1781(89)90047-4. [DOI] [PubMed] [Google Scholar]
  • 27.Cadogan MP, Orlando Edelen M, Lorenz KA, et al. The Relationship of Reported Pain Severity to Perceived Effect on Function of Nursing Home Residents. Journal of Gerontology: Medical Sciences. 2008;63:969–973. doi: 10.1093/gerona/63.9.969. [DOI] [PubMed] [Google Scholar]
  • 28.Teresi J, Abrams R, Holmes D, et al. Prevalence of depression and depression recognition in nursing homes. Soc Psychiatry Psychiatr Epidemiol. 2001 Dec;36(12):613. doi: 10.1007/s127-001-8202-7. [DOI] [PubMed] [Google Scholar]
  • 29.Parmelee PA, Lawton MP, Katz IR. Psychometric Properties of the Geriatric Depression Scale Among the Institutionalized Aged. Psychological Assessment: A Journal of Consulting and Clinical Psychology. 1989;I:331–338. [Google Scholar]
  • 30.Alessi CA, Martin JL, Webber AP, et al. More daytime sleeping predicts less functional recovery among older people undergoing inpatient post-acute rehabilitation. Sleep. 2008 Sep 1;31(9):1291–300. [PMC free article] [PubMed] [Google Scholar]
  • 31.Folstein MF, Folstein SF, McHugh PR. ‘Mini-mental state.’ A practical method for grading the cognitive status of patients for the clinician. J Psychiatr Res. 1975;12:189–198. doi: 10.1016/0022-3956(75)90026-6. [DOI] [PubMed] [Google Scholar]
  • 32.Sheikh JI, Yesavage JA. Geriatric Depression Scale (GDS): Recent evidence and development of the shorter version. Clin Gerontol. 1986;5:165–172. [Google Scholar]
  • 33.Miller MD, Paradis CF, Houck PR, et al. Rating chronic medical illness burden in geropsychiatric practice and research: application of the Cumulative Illness Rating Scale. Psychiatry Res. 1992;41:237–248. doi: 10.1016/0165-1781(92)90005-n. [DOI] [PubMed] [Google Scholar]
  • 34.Buysse DJ, Reynolds CF, Monk TH, et al. The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research. Psychiatry Research. 1989;28:193–213. doi: 10.1016/0165-1781(89)90047-4. [DOI] [PubMed] [Google Scholar]
  • 35.Johns MW. A new method for measuring daytime sleepiness: The Epworth Sleepiness Scale. Sleep. 1991;14:540–545. doi: 10.1093/sleep/14.6.540. [DOI] [PubMed] [Google Scholar]
  • 36.Johns M, Hocking B. Daytime sleepiness and sleep habits of Australian workers. Sleep. 1997;20:844–849. doi: 10.1093/sleep/20.10.844. [DOI] [PubMed] [Google Scholar]
  • 37.Martin JL, Webber AP, Alam T, et al. Daytime sleeping, sleep disturbance, and circadian rhythms in the nursing home. Am J Geriatr Psychiatry. 2006;14:121–129. doi: 10.1097/01.JGP.0000192483.35555.a3. [DOI] [PubMed] [Google Scholar]
  • 38.Alessi CA, Martin JL, Webber AP, et al. More daytime sleeping predicts less functional recovery among older people undergoing inpatient post-acute rehabilitation. Sleep. 2008 Sep 1;31(9):1291–300. [PMC free article] [PubMed] [Google Scholar]
  • 39.SAS [computer software]. Version 9.1. Cary, NC: SAS Institute, Inc; 2004. [Google Scholar]

RESOURCES