Assessing the psychometric properties of the PROMIS sleep measures in persons with psychosis

Abstract

An accumulation of research has indicated that persons with psychotic disorders experience a variety of sleep disturbances. However, few studies have examined the psychometric properties of sleep assessments that are utilized in this population. We conducted two studies to examine the reliability and validity of the PROMIS^TM Sleep Disturbance and Sleep-Related Impairment scales in outpatient samples of persons with psychosis. In Study 1, we examined the internal consistency and convergent validity of the PROMIS sleep scales in individuals with various psychotic disorders (N = 98) and healthy controls (N = 22). The PROMIS sleep scales showed acceptable internal consistency and convergent validity in both healthy controls and individuals with psychotic disorders. In addition, replicating prior research, the PROMIS scales identified greater sleep disturbance and sleep-related impairment in participants with psychotic disorders compared to healthy controls. In Study 2, we examined the test-retest reliability (M = 358 days) of the PROMIS sleep scales in a subset (N = 37) of persons with psychotic disorders who previously participated in Study 1. We also assessed the relation between these self-report measures and actigraph sleep parameters. The results showed that PROMIS sleep measures demonstrated modest temporal stability in the current sample. Contrary to our hypothesis, there was a lack of correspondence between these scales and actigraph sleep parameters. Overall, these findings indicate that the PROMIS sleep scales are psychometrically sound measures for populations with psychosis and highlight the importance of utilizing a multi-method approach to assess sleep.

Statement of Significance.

Sleep disturbances are commonly experienced by persons with psychotic disorders. Few studies have examined the reliability and validity of sleep measures that are used in this population. We conducted two studies to evaluate the performance of the PROMIS Sleep Disturbance and Sleep-Related Impairment scales in samples of persons with psychosis. We also examined the association between the PROMIS sleep scales and actigraphy. Results showed that the PROMIS sleep scales are sound measures to assess sleep in persons with psychosis. Findings also suggest that it is important to measure sleep using multiple assessment methods.

Introduction

Sleep disturbances, such as difficulty falling asleep, excessive sleep, and early wakening, are prevalent in many psychiatric disorders, including post-traumatic stress disorder [1], anxiety disorders [1], major depressive disorder [1–4], and bipolar disorders [5]. An accumulation of research indicates that sleep disturbances commonly occur across the course of the illness in clinical samples with psychosis as well [5–9]. Researchers have proposed that sleep disturbances are a transdiagnostic problem that often arise before the onset of psychiatric symptoms, persist after other symptoms remit, and predict the development of future symptoms [3–5, 8–11].

Sleep disturbances are experienced by thirty to eighty percent of people with psychosis [12–14]. Common sleep disturbances in this population include insomnia [5, 7, 15–18], hypersomnia [5, 8], and poor sleep quality [19–21]. Sleep disturbances in psychosis are associated with decreased medication compliance [19] worse quality of life [13, 16, 19, 20, 22], poorer behavioral skills [23], and worse psychosocial and community functioning [5, 23, 24]. Although some studies have also found that sleep disturbances are associated with increased severity of symptoms of psychosis [5, 6, 22, 25–28], other studies have failed to find this association [6, 16, 20, 29, 30]. It is difficult to interpret these variable results because researchers often used different methods of sleep assessment across studies [6, 31].

Given the growing clinical and research interest in the association between sleep problems and psychosis, it is important to examine the validity of sleep assessments in this population. Previous research has relied on various self-report measures to examine sleep disturbances in individuals with psychosis. Some studies utilized non-validated sleep measures [13, 15, 32], often with a small number of items (e.g. one to four [6, 13]). Empirically validated questionnaires that assess sleep disturbances, such as the Pittsburgh Sleep Quality Index (PSQI) [33], remain the most commonly utilized method of assessment [34]. Although the PSQI has frequently been used to assess sleep in persons with psychosis [16, 20, 21, 24, 29, 30, 35, 36], few researchers have thoroughly investigated its psychometric properties in this population [36]. Some studies have reported that the PSQI demonstrates acceptable internal consistency in samples with psychosis [20, 37], but there is limited research regarding additional psychometric properties of the PSQI (e.g. test–retest reliability, convergent validity; see exceptions [16, 21, 38]).

The National Institutes of Health Patient-Reported Outcomes Measurement Information System (PROMIS) Sleep Disturbance and Sleep-Related Impairment scales were developed to provide psychometrically sound measures of sleep that may be administered to diverse groups in various contexts including clinical trials, epidemiological studies, and general clinical settings [34, 39]. To the best of our knowledge, only one study has evaluated the performance of the PROMIS sleep measures in individuals with psychosis. Although Strainge and colleagues [40] reported that the PROMIS Sleep Disturbance scale has excellent internal consistency in a mixed psychiatric group that included a subset (24%) of persons with psychosis, it remains unknown how this performs specifically in samples with psychosis.

Despite the advantages of sleep questionnaires, including the PROMIS sleep measures, there are lingering questions related to the performance of these scales in people with psychotic disorders. One issue is that individuals with insomnia and psychiatric disorders demonstrate increased difficulty accurately reporting their sleep patterns [41]. As found by Hartmann et al. [41], persons with psychiatric disorders (e.g. anxiety and depression) and insomnia showed worse retrospective recall of their sleep patterns compared to persons with only insomnia. Although only one person with psychosis was included in this study [41], these findings are noteworthy because persons with psychosis often experience anxiety and depression [42]. Another concern is that persons with psychosis may frequently experience changes in their sleep, which may make it difficult for them to accurately rate their overall sleep patterns [36]. A final limitation of sleep questionnaires is that individuals with cognitive impairments may incorrectly report their sleep quality [43]. This is a notable issue for persons with psychosis because they often demonstrate cognitive deficits [44, 45], and sleep problems may exacerbate memory difficulties in this population [46]. Given the above questions, it is imperative that researchers empirically confirm that these sleep questionnaires demonstrate acceptable psychometric properties for use in individuals with psychotic disorders.

In addition to exploring the psychometric characteristics of sleep questionnaires in psychosis, it is important to understand how sleep questionnaires relate to other indicators of sleep behavior, such as actigraphy. Actigraphy commonly consists of wrist-worn devices with light detectors and accelerometers that estimate sleep-wake patterns, such as total sleep time and number of awakenings [43, 47]. Although several studies [25, 27, 29, 35, 47, 48] have successfully utilized actigraphy in research involving persons with psychotic disorders, most [25, 35, 48, 49] have not evaluated the correspondence between sleep questionnaires and actigraphy. When Bromundt and colleagues [29] compared these methods of sleep assessment in a small sample (n = 14) of persons with schizophrenia, they found that self-reported sleep quality and two actigraph sleep variables (sleep efficiency and time awake after sleep onset) showed large, but non-significant, effect sizes. More research is needed to examine the association between these methods of sleep assessment in persons with psychotic disorders.

Following the National Institute of Mental Health research domain criteria (RDoC) framework [50–53], we conducted two studies that adopted a symptom-oriented dimensional approach to examine the psychometric properties of the PROMIS sleep assessments in individuals with psychosis. The RDoC approach recommends that researchers implement an agnostic view of categorical diagnostic groups, which are associated with substantial heterogeneity [51], and focus on the shared features of psychopathology across groups [52, 53]. Adhering to this framework gives us the advantage of examining sleep problems across diagnostic groups with psychosis rather than focusing on one group at a time. In the first study, we sought to replicate previous studies by assessing sleep disturbance and sleep-related impairment among participants with clinical psychosis and healthy controls. In addition, we sought to extend previous research by assessing the internal consistency and convergent validity of the PROMIS sleep measures in these groups. Given the concerns about the accuracy of responses on sleep questionnaires and cognitive impairment, we examined cognitive functioning in the current sample. We hypothesized that participants with a psychotic disorder would endorse greater sleep disturbance and sleep-related impairment compared to healthy controls. We also expected that the PROMIS Sleep Disturbance and Sleep-Related Impairment scales would show acceptable psychometric properties in a sample of persons with psychosis. In the second study, we evaluated the temporal stability of the PROMIS sleep questionnaires in individuals with psychosis. Further, we examined the relation between responses on sleep questionnaires and actigraphy. We hypothesized that self-reported sleep disturbance and sleep-related impairment would be associated with actigraph sleep parameters. Specifically, we expected that greater sleep disturbance and sleep-related impairment would be related to increased sleep latency, decreased total sleep time, decreased sleep efficiency, and increased number of awakenings after sleep onset.

Study 1

The first study assessed self-reported sleep disturbance and sleep-related impairment in persons with a psychotic disorder and healthy controls. In addition, the study evaluated the internal consistency and convergent validity of the PROMIS Sleep Disturbance and Sleep-Related Impairment scales in these groups. We have previously reported on symptom and social functioning correlates of sleep in a subset of this sample [23, 28].

Methods

Participants

Participants were enrolled in an ongoing grant-funded neuroimaging study that assessed social affiliation deficits in psychosis from an RDoC perspective between April 2017 and February 2020 (National Institutes of Health grant R01MH110462). The sample consisted of 98 clinical participants with a psychotic disorder (e.g. schizophrenia, schizoaffective disorder, bipolar disorder with psychosis, major depressive disorder with psychosis) and 22 healthy controls. All clinical participants were prescribed medications and dosages as determined by their outpatient treatment providers. Participants were recruited from outpatient mental health programs in the Baltimore and Washington D.C. metro areas or from online community posting websites (e.g. Craigslist).

Inclusion criteria for clinical participants included (1) aged 18–60, (2) lifetime history of a psychotic disorder, (3) clinical stability (i.e. no inpatient hospitalizations for 3 months before enrollment, no changes in psychoactive medication four weeks before enrollment) as indicated by approval of clinician and medical record review, and (4) fluent in English. Inclusion criteria for community participants included (1) aged 18–60, (2) no current clinical disorder or psychiatric medications, (3) no lifetime history of a psychotic or mood disorder, (4) no avoidant, paranoid, schizotypal or schizoid personality disorder, and (5) fluent in English. Exclusion criteria for all participants included (1) current substance use disorder, (2) neurological conditions (e.g. epilepsy, multiple sclerosis), (3) evidence of intellectual disability as determined by medical evaluation or prior cognitive testing, (4) any history of serious head injury, (5) any MRI contraindications (e.g. MRI unsafe metal in body, weight that exceeds the limitations of MRI machine), and (6) unwillingness to have assessments videotaped during study participation. Of note, the inclusion/exclusion criteria were not based on medication type or dosages.

Measures

Diagnostic and clinical assessments

The Structured Clinical Interview for the Diagnostic and Statistical Manual of Mental Disorders, 5th edition (SCID-5) [54] screener and SCID-5 clinical interview were administered to determine psychiatric diagnoses. All clinical participants completed the SCID-5 mood and psychotic disorder modules. Those who endorsed items related to the alcohol or substance use on the SCID-5 screener were administered the corresponding module(s) to rule out possible alcohol or substance use disorder(s). Community participants were administered the SCID-5 screener and any corresponding SCID-5 module related to items endorsed on the screener. Community participants who met diagnostic criteria for a current psychiatric disorder were excluded from the study.

The Brief Psychiatric Rating Scale-expanded version (BPRS) [55, 56] is a 24-item semi-structured clinical interview that assesses the severity of current clinical symptomatology over the previous one week. The BPRS has shown acceptable test-retest reliability, internal correlation coefficients, and discriminant validity [57, 58]. For the current study, the BPRS total score was used to assess overall symptom severity.

Sleep assessments

The National Institutes of Health Patient-Reported Outcomes Measurement Information System (PROMIS) Sleep Disturbance and Sleep-Related Impairment short form scales [39] were utilized to assess sleep disturbance and sleep-related impairment over the past one week. The Sleep Disturbance scale is an 8-item questionnaire that inquiries about various sleep disturbances, such as restlessness, difficulty falling asleep, and trouble staying asleep. It includes items like “I had difficulty falling asleep” and “I had trouble staying asleep” that are rated on a 5-point Likert scale. The Sleep-Related Impairment scale inquiries about daily challenges related to sleep disturbance, such as trouble getting things done, poor concentration, and feeling irritable. It includes items like “I had a hard time concentrating because of poor sleep” and “I had a hard time getting things done because I was sleepy” that are also rated on a 5-point Likert scale. Higher scores indicate greater sleep disturbance or sleep-related impairment. Notable strengths of these scales are their brevity and they have shown greater precision assessing the severity of sleep problems compared to traditional questionnaires (i.e. the Pittsburgh Sleep Quality Index [33] and Epworth Sleepiness Scale [59]) [39]. Both PROMIS sleep measures have shown acceptable convergent, construct, and discriminant validity in healthy populations and those with clinical sleep disorders [39].

Cognitive assessment

The MATRICS Consensus Cognitive Battery (MCCB) [60] is the gold standard assessment for cognitive functioning in persons with psychosis [44]. It consists of ten measures that assess seven cognitive domains, including working memory, attention, and verbal learning [60]. All measures included in the MCCB have demonstrated acceptable reliability and validity [60, 61].

Procedures

Study procedures were approved by the University of Maryland, Baltimore Institutional Review Board. All participants completed a standardized informed consent process with trained research staff. Research staff administered a brief questionnaire to verify that participants understood the informed consent document and were competent to sign it. After the consent process, participants completed clinical interviews and self-report questionnaires related to diagnoses, clinical symptomatology, social affiliation, and community and social functioning. Participants were compensated for their participation.

Data analysis

All data analyses were completed using SPSS 24. For one participant, one item for the PROMIS Sleep Disturbance scale was replaced with a non-pathological score. No other data was missing for self-report measures. For both PROMIS sleep measures, the raw total scores were converted to T-scores using conversion tables based on the normative adult sample from the general population of the United States [39, 62–64]. Following the Health Measures’ scoring guide, the PROMIS T-scores for sleep disturbance and sleep-related impairment may be categorized into mild, moderate, or severe symptoms, which corresponds to 0.5, 1.0, and 2.0 standard deviations above the mean T-score of the normative sample, respectively [65]. We utilized the MCCB computerized scoring program, controlling for age and gender, to calculate domain and composite T-scores that were based on the performance of the normative sample of community participants in the original study [61].

Independent Sample Welch’s t-tests and Pearson’s Chi-Square tests were used to assess demographic and clinical characteristics across groups. Independent Sample Welch’s t-tests were conducted to examine differences between these groups regarding self-reported sleep disturbance and sleep-related impairment. Due to the differences in sample size between groups, effect sizes were calculated utilizing the Cohen’s d statistic by calculating the differences between the group means and dividing by the pooled sample’s standard deviation [66, 67]. Cronbach’s alphas (α) were calculated to evaluate the internal consistency of the Sleep Disturbance and Sleep-Related Impairment scales. Pearson r correlations were used to evaluate the relation between the Sleep Disturbance and Sleep-Related Impairment scales.

Results

Demographic and diagnostic characteristics of the sample are provided in Table 1. There were no significant differences between groups related to age (t (31.63) = −.27, p = 0.79), race (X² = 4.67, p = 0.32), and sex (X² = 2.78, p = 0.10). However, participants with psychosis had fewer years of education (t (30.49) = −3.69, p = 0.001), and they performed significantly worse in 8 out of 10 cognitive domains compared to healthy controls (see Table 2). Most participants with psychotic disorders were prescribed antipsychotic medications, including atypical antipsychotics (n = 64, 53.3%), typical antipsychotics (n = 11, 9.2%), or a combination of atypical and typical antipsychotics (n = 10, 8.3%).

Table 1.

Sample characteristics

	Clinical (n = 98) Mean (SD) or n (%)	Healthy Control (n = 22) Mean (SD) or n (%)
Age (years)	43.67 (12.23)	42.91 (12.02)
Education (years)	12.59 (2.29)	14.64 (2.37)
Sex
Male	57 (58.2)	17 (77.3)
Female	41 (41.8)	5 (22.7)
Race
African American	65 (66.3)	16 (72.7)
White	24 (24.5)	4 (18.2)
Asian	2 (2.0)	2 (9.1)
More than one race	6 (6.1)	-
Not reported	1 (1.0)	-
Ethnicity
Non-Hispanic or Latino	88 (89.8)	21 (95.5)
Hispanic or Latino	9 (9.2)	1 (4.5)
Unknown	1 (1.0)	-
Current Employment
Yes	24 (27.6)	11 (50.0)
No	71 (72.4)	11 (50.0)
Diagnosis
Schizophrenia	39 (32.5%)	–
Schizoaffective bipolar type	17 (14.2%)	–
Schizoaffective depressive type	19 (15.8%)	–
Delusional sisorder	1 (.8%)	–
BP I with psychotic features	13 (10.8%)	–
MDD w/ psychotic features	10 (8.3%)	–
Antipsychotic medication
Atypical	64 (53.3%)	–
Typical	11 (9.2%)	–
Combined (typical and atypical)	10 (8.3%)	–
Neither typical nor atypical	12 (10.0%)	–
Unknown	1 (.8%)	–

Note: BP = bipolar; MDD = major depressive disorder

Table 2.

Descriptive statistics of assessments

	Clinical Mean (SD)	Healthy Control Mean (SD)	t
BPRS total score	40.60 (9.80)	27.50 (4.14)	9.88***
PROMIS sleep disturbance	18.41 (8.01)	14.95 (5.27)	2.50*
T-score	46.68 (10.92)	42.59 (8.22)
PROMIS Sleep-Related Impairment	18.12 (7.26)	13.55 (5.71)	3.22*
T-score	50.98 (10.44)	43.70 (9.93)
MATRICS Consensus Cognitive Battery Domains (T-score)
Speed of processing	30.11 (13.11)	43.82 (17.01)	–3.54**
Attention/vigilance	27.81 (10.67)	39.68 (14.62)	–3.58**
Working memory	33.85 (12.24)	43.55 (10.54)	–3.76**
Verbal learning	34.58 (9.02)	46.82 (13.07)	–4.17***
Visual learning	34.85 (13.76)	45.14 (16.08)	–2.77*
Reasoning and problem solving	38.05 (8.14)	40.62 (10.41)	–1.06
Social cognition	41.46 (14.44)	42.64 (13.43)	–0.363
Overall composite	24.49 (12.83)	38.86 (15.78)	–3.87**

Note: BPRS = Brief Psychiatric Rating Scales; PROMIS = Patient-Reported Outcomes Measurement Information System; ns for the cognitive domains for the clinical group ranged from 87 to 96 while ns for healthy control group ranged from 21 to 22 because not all participants completed each subscale in the cognitive battery

* p < 0.05; ** p = 0.001; *** p < 0.001

Sleep measure scores are presented in Table 2. Persons with psychosis reported significantly greater sleep disturbance (t (45.76) = 2.50, p = 0.02) and sleep-related impairment (t (37.90) = 3.22, p = 0.003) than healthy controls. The group difference in sleep disturbance approached a medium effect size (Cohen’s d = 0.45) while the effect size for sleep-related impairment was medium (Cohen’s d = 0.65). To understand the occurrence of clinically relevant sleep problem more clearly, the frequency of mild, moderate and severe PROMIS scores are presented in Table 3 [65]. Among persons with psychosis, 19.4% reported at least mild sleep disturbance and 38.8% reported at least mild sleep-related impairment. Conversely, 9.1% of healthy controls endorsed at least mild sleep disturbance and 13.6% endorsed at least mild sleep-related impairment.

Table 3.

Frequency of PROMIS Sleep Assessments T-Scores

	Clinical (n = 98)	Healthy control (n = 22)
PROMIS Sleep Disturbance T-Scores
Within the normal limits	79 (80.6%)	20 (90.9%)
Mild sleep disturbance	10 (10.2%)	2 (9.1%)
Moderate sleep disturbance	7 (7.1%)	–
Severe sleep disturbance	2 (2.0%)	–
PROMIS Sleep-Related Impairment T-Scores
Within the normal limits	60 (61.2%)	19 (86.4%)
Mild sleep-related impairment	19 (19.4%)	2 (9.1%)
Moderate sleep-related impairment	16 (16.3%)	1 (4.5%)
Severe sleep-related impairment	3 (3.1%)	-

Note: PROMIS = Patient-Reported Outcomes Measurement Information System. The T-score distribution has a mean of 50 and a standard deviation of 10 [39, 62]. The cut-points of mild, moderate, and severe correspond to 0.5, 1.0, and 2.0 standard deviations above the mean of the normative adult sample, respectively [65].

Internal consistency

Overall, the Cronbach alphas for both groups ranged from good to excellent. In the group with psychotic disorders, the Sleep Disturbance scale had excellent internal consistency (α = .91) while the Sleep-Related Impairment scale had good internal consistency (α = .85). Similarly, in the healthy control group, the Sleep Disturbance scale showed good internal consistency (α = .82) and the Sleep-Related Impairment scale showed excellent internal consistency (α = .90). These findings suggest that the PROMIS sleep assessments demonstrate acceptable reliability for both persons with psychotic disorders and healthy controls.

Convergent validity of PROMIS sleep measures

There were positive correlations between the Sleep Disturbance and Sleep-Related Impairment scales for participants with psychosis (r = 0.59, p < 0.001) and healthy controls (r = 0.71, p < 0.001), which indicates that these sleep measures were moderately related.

Study 2

In the second study, we aimed to extend our findings on the psychometric properties of the PROMIS sleep measures in a transdiagnostic subset of participants with psychosis who were previously enrolled in Study 1. We first assessed the long-term temporal stability of self-reported sleep disturbance and sleep-related impairment. We also investigated the relation between the PROMIS sleep questionnaires and actigraphy.

Methods

Participants

Thirty-seven individuals with a psychotic disorder (e.g. schizophrenia, schizoaffective disorder, bipolar disorder with psychosis, major depressive disorder with psychosis) who participated in Study 1 were recruited to complete clinical interviews, self-report questionnaires, and actigraphy assessments between March 2019 and November 2019. The mean duration between Study 1 and Study 2 was 358.05 days (SD = 257.31).

Measures

Participants provided written consent for research staff to incorporate their responses on the PROMIS sleep measures [39] and SCID-5 clinical interview [54] that were obtained during Study 1 into the current study. All participants were re-administered the Brief Psychiatric Rating Scale-expanded version [55, 56] as well as the PROMIS Sleep Disturbance and Sleep-Related Impairment scales [39].

Actigraph wristwatches (Phillips Respironics, USA) that contain an accelerometer and light sensors were utilized to estimate sleep parameters, such as total sleep time and total number of awakenings [43, 47]. Participants wore actigraph watches on their non-dominant wrists for seven consecutive nights. Data collection was based on the default settings of the actigraph watches, including 30-s epochs, medium sensitivity, and 10-minute intervals for sleep onset and sleep end. We used the automated algorithms of the Actiware software (version 6.0.9; Phillips Respironics) to calculate the averages for the following sleep variables: total sleep time, wake after sleep onset, number of awakenings after sleep onset, and sleep efficiency (i.e. percentage of time asleep during a designated sleep period). Our actigraph data collection [68] and analyses [69–71] are consistent with previous research. Visual inspection of the actograms was consistent with expected sleep-wake patterns (e.g. marked differences in levels of physical activity to indicate wake and sleep intervals) for most participants. For one participant, the actogram did not appear to demarcate sleep intervals for three nights; therefore, data from these three nights were not included in analyses.

Procedure

Study procedures were approved by the University of Maryland, Baltimore Institutional Review Board. Similar to Study 1, participants completed a standardized informed consent process. After providing consent, participants completed clinician-rated interviews and self-report questionnaires to assess clinical symptomatology and sleep. Following the completion of these assessments, participants were given an actigraph watch to wear for seven nights. They were instructed to wear the actigraph watch at all times, except when bathing or swimming for more than twenty minutes. At the end of the actigraphy phase of the study, participants returned study equipment and were compensated for their participation.

Data analysis

All data analyses were completed using SPSS 24. Data from Study 1 related to clinical diagnoses (i.e. SCID-5 [54]) and the PROMIS sleep measures [39] were included in the current study’s data analyses. Missing data for the Sleep Disturbance and Sleep-Related Impairment scales (one item on each scale) were replaced with non-pathological scores. There was no missing data for other self-report assessments. As described in Study 1, the raw total scores on the PROMIS Sleep Disturbance and Sleep-Impairment scales were converted to T-scores using a conversion table that was based on the normative sample of adults from the general population of the United States [39, 62–64], and these T-scores were categorized as mild, moderate, or severe following the Health Measures’ guideline [65]. All participants who completed at least one night of actigraphy were included in analyses. The averages for the actigraph variables (i.e. total sleep time, sleep efficiency, wakening after sleep onset, and number of awakenings) were used in all analyses. Exploratory analyses removing participants (n = 2) who only completed one night of actigraphy were conducted, but overall significance was unchanged.

Descriptive statistics pertaining to demographic information and total scores for all measures were calculated. Interclass correlation coefficients (ICC) were calculated to analyze the test-retest reliability of the PROMIS Sleep Disturbance and Sleep-Related Impairment scales for Time 1 (Study 1) and Time 2 (Study 2). Paired Sample t-tests were completed to compare mean differences between sleep assessments at these time points. Bivariate correlations were conducted to assess the convergent validity between the PROMIS sleep scales and actigraph sleep parameters. Pearson r correlations were calculated to assess the relation between Sleep Disturbance total score, actigraph total sleep time, and actigraph sleep efficiency because these variables were normally distributed. Spearman rho (rs) correlations were calculated when sleep variables (i.e. Sleep-Related Impairment total score, actigraph awakening after sleep onset and number of awakening) were not normally distributed.

Results

Demographic and diagnostic characteristics of the sample are provided in Table 4. Most participants were prescribed atypical antipsychotics (n = 24, 64.9%), while others were prescribed typical antipsychotics (n = 6, 16.2%), or a combination of atypical and typical antipsychotics (n = 3, 8.1%). All participants completed at least one night of actigraphy. Further, 86.5% of participants wore the actigraph watch for at least five nights.

Table 4.

Sample characteristics

	Mean (SD) or n (%)
Age (years)	42.76 (12.86)
Sex
Male	22 (59.5%)
Female	15 (40.5%)
Race
African American	22 (59.5%)
White	9 (24.3%)
Asian	1 (2.7%)
More than one race	5 (13.5%)
Ethnicity
Non-Hispanic or Latino	32 (86.5%)
Hispanic or Latino	5 (13.5%)
Education (years)	12.76 (2.07)
Current Employment
Yes	11 (29.7%)
No	26 (70.3%)
Diagnosis
Schizophrenia	15 (40.5(%)
Schizoaffective bipolar type	8 (21.6%)
Schizoaffective depressive type	6 (16.2%)
BP I w/ psychotic features	5 (13.5%)
MDD w/ psychotic features	3 (8.1%)
Antipsychotic medication
Atypical	24 (64.9%)
Typical	6 (16.2%)
Combined (typical and atypical)	3 (8.1%)
Neither typical nor atypical	3 (8.1%)
Missing	1 (2.7%)

Note: BP = bipolar; MDD = major depressive disorder.

Descriptive statistics related to clinical symptoms and sleep results are shown in Table 5. Replicating findings from Study 1, participants showed a wide range of T-scores for self-reported sleep disturbance (M = 48.18, range 28.90–76.50) and sleep-related impairment (M = 49.65, range 30.00–76.90). Specifically, 29.7% of participants reported at least mild sleep disturbance and 54.1% of participants reported at least mild sleep-related impairment (see Table 6). As shown in Table 7, mean actigraph total sleep time was 470.25 min (approximately 7.84 h), which indicates that at the group level the average sleep duration was within the recommended sleep range for adults (e.g. 7−9 h) [72]. However, the sample demonstrated a large range (270.63−753.00 min) of actigraph total sleep time. The actigraph results also showed that participants demonstrated a large number of sleep awakenings (M = 36.32, SD = 17.71) during rest-sleep intervals, which suggests that this sample experiences various sleep problems. Replicating findings from Study 1, the Sleep Disturbance and Sleep-Related Impairment scales were moderately correlated (r = 0.57, p < 0.001).

Table 5.

Descriptive statistics of assessments for Study 1 and Study 2

	Study 1		Study 2
	Mean (SD)	Range	Mean (SD)	Range
BPRS Total Score	39.11 (8.68)	25.00–59.00	35.86 (9.96)	22.00–61.00
PROMIS Sleep Disturbance	17.70 (8.40)	8.00–36.00	19.59 (8.45)	8.00–40.00
T-score	45.48 (11.28)	28.90–67.50	48.18 (11.55)	28.90–76.50
PROMIS Sleep-Related Impairment	17.76 (7.79)	8.00–38.00	19.86 (7.77)	8.00–31.00
T-score	50.53 (10.80)	30.00–75.00	52.90 (10.97)	30.00–66.30

Note: BPRS = Brief Psychiatric Rating Scale; PROMIS = Patient-Reported Outcomes Measurement Information System

Table 6.

Frequency of PROMIS Sleep Assessments T-Scores for Study 1 and Study 2

	Study 1	Study 2
PROMIS Sleep Disturbance T-Scores
Within the normal limits	27 (73.0%)	26 (70.3%)
Mild sleep disturbance	6 (16.2%)	8 (23.5%)
Moderate sleep disturbance	4 (10.8%)	2 (5.4%)
Severe sleep disturbance	-	1 (2.7%)
PROMIS Sleep-Related Impairment T-Scores
Within the normal limits	24 (64.9%)	17 (45.9%)
Mild sleep-related impairment	7 (18.9%)	9 (24.3%)
Moderate sleep-related impairment	4 (10.8%)	11 (29.7%)
Severe sleep-related impairment	2 (5.4%)	-

Note: PROMIS = Patient-Reported Outcomes Measurement Information System. The T-score distribution has a mean of 50 and a standard deviation of 10 [39, 62]. The cut-points of mild, moderate, and severe correspond to 0.5, 1.0, and 2.0 standard deviations above the mean of the normative adult sample, respectively [65].

Table 7.

Descriptive statistics for actigraph sleep variables

	Mean (SD)	Range
Average TST (min.)	470.25 (113.07)	270.63–753.00
Average SE (%)	81.31 (8.32)	60.11–92.10
Average WASO (min.)	55.99 (33.54)	15.14–182.00
Average number of awakenings	36.32 (17.71)	10.00–89.43

Note: TST = total sleep time, SE = sleep efficiency, WASO = wakening after sleep onset.

Test-rest reliability

The test-retest reliability was fair for both the Sleep Disturbance (ICC = 0.53, p < 0.001) and Sleep-Related Impairment scales (ICC = 0.50, p < 0.001). Further, there were no significant mean differences between the two assessments for the Sleep Disturbance (t (36) = 1.51, p = 0.14) or Sleep-Related Impairment (t (36) = 1.28, p = 0.21) scales. Thus, it appears that self-reported sleep disturbance and sleep-related impairment in this sample were modestly stable over time.

Convergent validity

The bivariate correlations conducted to assess the convergent validity between the Sleep-Disturbance and Sleep-Related Impairment scales and actigraph sleep parameters are presented in Table 8. Contrary to our hypothesis, results showed that there were no significant correlations between the PROMIS sleep measures and actigraph sleep variables (ps > 0.05). Further, the effect sizes for the bivariate correlations were small [66]. It is important to note that the relation between actigraphy computed average sleep efficiency and self-reported sleep disturbance approached a moderate effect size (r = −0.25, p = 0.143).

Table 8.

Correlations between self-report sleep measures and actigraphy

Actigraphy	PROMIS Sleep Disturbance	PROMIS Sleep-Related Impairment
Average TST (min)	–.19	.05
Average SE (%)	–.25	–.12
Average WASO (min)	.21	.18
Average number of awakenings	.17	.19

Note: PROMIS = Patient-Reported Outcomes Measurement Information System, TST = Total Sleep Time, SE = Sleep Efficiency, WASO = Wakening After Sleep Onset; Conducted Spearman correlations for the PROMIS Sleep-Related Impairment scale, actigraph wakening after sleep onset, and actigraph number of awakening because these variables were not normally distributed.

Discussion

We conducted two studies to investigate the performance the PROMIS Sleep Disturbance and Sleep-Related Impairment scales in outpatient individuals with psychotic disorders. The first study examined the level of sleep disturbance and sleep-related impairment in a sample with psychosis and a non-clinical community sample. In addition, we compared the psychometric properties of the PROMIS scales between these groups. In the second study, we assessed the temporal stability of the PROMIS sleep scales, and we examined the relation between these self-report questionnaires and actigraph sleep parameters.

Main findings of Study 1

Confirming our first hypothesis, we found that persons with psychosis, who were considered clinically stable, still endorsed increased levels of sleep disturbance and sleep-related impairment compared to healthy controls. Further, the effect size for sleep-related impairment was of a medium magnitude. Using available clinical cut-offs [65], 19.4% of individuals with a psychotic disorder experienced at least mild sleep disturbance, which was double the rate reported by healthy controls (9.1%). In addition, 38.8% of persons with psychosis endorsed at least mild sleep-related impairment, which was almost three times the rate reported by healthy controls (13.6%). Our results replicate past research [2–4, 9, 71] that has reported persons with schizophrenia spectrum, major depressive, and bipolar disorders demonstrate sleep disturbances even when their clinical symptoms are remitted. These findings also suggest that the PROMIS Sleep Disturbance and Sleep-Related Impairment scales can successfully distinguish between groups with and without clinical psychotic disorders. Of note, the rate of reported sleep-related impairment within persons with psychotic disorders was fifty percent higher than reports of sleep disturbance in this group (38.8% versus 19.4%, respectively). This finding suggests that it may be important to examine individuals’ perceptions of how sleep impacts their daily functioning even when they endorse low levels of sleep disturbance. Our second hypothesis was supported given that the Sleep Disturbance and Sleep-Related Impairment scales showed acceptable internal consistency, despite persons with psychotic disorders demonstrating poor performance in most cognitive domains. Thus, the PROMIS sleep questionnaires appear to effectively assess sleep difficulties in samples with both psychosis and deficits in cognitive functioning. Findings also showed that the Sleep Disturbance and Sleep-Related Impairment scales were moderately correlated. This suggests that while these scales are similar, each assesses a unique aspect of sleep problems within samples with psychosis and healthy controls. These results replicate prior reports indicating that the PROMIS sleep scales demonstrate strong psychometric qualities [1, 40, 73–77] and extends these findings to individuals with psychosis.

Main findings of Study 2

Results showed that over an extended period of time, on average 358 days, the PROMIS sleep scales demonstrated acceptable test-retest reliability in a sample of persons with psychosis. Replicating past research [39, 62–64], approximately 35%−54% of individuals with psychosis in the current study reported sleep disturbance and/or sleep-related impairment. The longitudinal assessment of these self-reports indicates that sleep problems for some with psychosis are persistent, which reflects the clinical importance of providing more attention to sleep complaints in this population.

Previous studies have only investigated the temporal stability of the PROMIS sleep measures during a short period of time (e.g. up to three weeks [73, 78]); our study appears to be the first to examine the long-term temporal stability of the PROMIS Sleep Disturbance and Sleep-Related Impairment scales. Our findings suggest that the PROMIS sleep measures are moderately stable over time within individuals with psychosis and support their use as a valid self-report measure of sleep in this population.

The sample’s actigraph total sleep duration was approximately 7.84 h, which is within the recommended amount for adults [72]. Paralleling the results of the self-report measures, the actigraph total sleep duration varied widely. Some individuals demonstrated a shorter sleep duration that was almost half (4.50 h) of the recommended amount for adults, while others showed a sleep duration that was over one third longer (12.55 h) than the recommended length of time [72]. While the actigraph results showed that our participants demonstrated higher number of awakenings compared to the general population [79], these findings replicate a previous study involving persons with schizophrenia [80]. Similar to previous research in psychosis [27, 47], actigraphy was well tolerated in our sample. All participants in our study wore the actigraph watch for at least one night, and 86% of participants wore it for at least five nights.

Relation between sleep questionnaires and actigraphy

Contrary to our third hypothesis, there were no significant relations between the PROMIS Sleep Disturbance and Sleep-Related Impairment scales and any actigraph sleep parameters. The non-significant association between the PROMIS sleep questionnaires and actigraphy in our sample with psychosis is comparable to a previous study that did not find a significant relation between the PSQI and actigraphy in adults with schizophrenia [29]. The current findings replicate research from the broader literature that has reported a lack of correspondence between sleep questionnaires and actigraphy in various populations [81–83]. On occasions when researchers have found significant associations between sleep questionnaire and actigraph sleep variables, this overlap has only pertained to some sleep domains, such as total sleep time [84–86]. Thus, our results reflect a common lack of convergence between sleep questionnaires and actigraphy.

Although sleep questionnaires and actigraphy are common methods of sleep assessment, they clearly do not consistently correspond with each other. This discrepancy could be due to the nature of these assessments and their unique limitations. Sleep questionnaires are designed to capture individuals aggregated and subjective experiences, but they may be fallible to error because of changes in mood, cognitive functioning, and sleep patterns [36, 41, 44–46]. Alternatively, actigraphy estimates sleep parameters by passively measuring light and physical activity [43, 47], and it may mistake reduced motion for the onset of a sleep period [87, 88]. This notable error in actigraphy often occurs with populations that experience insomnia [87], such as persons with psychosis [12, 89]. The differences between sleep questionnaires and actigraphy suggest that these types of sleep assessments capture distinct facets of sleep and sleep problems [69, 81, 83], and that researchers should follow previous recommendations to utilize a multi-method assessment of sleep in future studies [27, 83, 86, 90]. The use of multiple methods of sleep assessment may be especially important in samples with psychosis given that varying forms of sleep assessment have been differentially associated with non-sleep variables in this population [25]. For example, Mulligan and colleagues [25] found that self-report sleep diaries and actigraphy provided convergent and divergent information about sleep patterns in persons with schizophrenia spectrum disorders. More research is needed to fully investigate how different facets of sleep are related to other variables of interest in populations with psychosis.

Limitations

The current studies have several limitations that constrain interpretation of our results. Regarding Study 1, our recruitment criteria required clinical participants to be from an outpatient mental health clinic with stable symptoms and medications. Therefore, results from our sample may not be generalizable to those who are experiencing acute symptoms of psychosis, such as those in an inpatient setting. Related to Study 2, the convenience sampling method and small sample size are notable limitations. Our results regarding the temporal stability of the PROMIS scales may have been adversely impacted by the variable time duration between Time 1 and Time 2, which ranged from approximately two weeks to two-and-a-half-years. Another limitation of Study 2 is that we did not include a sleep diary to confirm actigraph wake and sleep periods. Although some researchers recommend utilizing sleep diaries to confirm actigraph intervals [68, 86], others have argued against this practice because self-report measures may also be inaccurate [91]. A potential limitation across both studies is that we did not evaluate for all psychiatric disorders and cannot assess the impact that these disorders (e.g. anxiety, depression) may have on sleep problems in this sample. In addition, we did not evaluate the influence that psychiatric medications (e.g. antipsychotic and benzodiazepine medications) have on sleep. Previous research has shown that these medications may affect sleep-wake patterns [10, 89, 92], such as contributing to increased daytime sleepiness, napping, sleep duration, and sedation [19, 29, 49, 93]. As we have mentioned in past publications [28, 94], all clinical participants were prescribed medications based on their outpatient treatment providers’ discretion and we are unable examine the potential impact that medications may have had on our findings. Finally, we did not assess for clinical sleep disorders, which prevents us from being able to characterize the frequency or type of sleep disorders in the current samples.

Future directions

Future studies may want to screen participants for sleep patterns prior to enrollment to ensure that the study sample represents the full continuum of sleep disturbances. Longitudinal studies, with larger samples sizes, are needed to fully assess the temporal relation between sleep disturbances and psychosis [6, 7, 31, 93]. This research could also evaluate the relation between daily variations in sleep patterns as measured by actigraphy and sleep questionnaires, which would allow us to thoroughly investigate their differential contributions in assessing sleep. More research is also needed to examine whether persons with psychosis consistently demonstrate a discrepancy between self-reported sleep disturbance and sleep-related impairment. Future studies would benefit from examining the influence that various psychiatric medications and clinical symptomology have on sleep disturbances [10, 32]. Although some researchers have published recommendations for the utilization of actigraphy in clinical and research settings [68, 86, 87], a gold standard for the implementation and collection of data using actigraphy has not been created. Therefore, it is imperative that future investigators develop established guidelines for actigraphy in various populations. Finally, our studies did not compare the performance of the PROMIS sleep measures to other empirically validated sleep questionnaires. Although the original validation study [39] found that the PROMIS Sleep Disturbance and Sleep-Related Impairment forms more precisely estimated the severity of sleep problems compared to the Pittsburgh Sleep Quality Index [33] and Epworth Sleepiness Scale [59], more research is needed to thoroughly assess the advantages and disadvantages of implementing these measures in samples with psychosis.

Conclusion

In summary, our research demonstrates that the PROMIS sleep questionnaires are a suitable assessment of sleep disturbance and sleep-related impairment in persons with psychosis. These findings give us greater confidence in the utilization of these measures in future studies involving this population. It also supports the importance of using multi-method assessments of sleep, including well-validated sleep questionnaires and actigraphy, to assess sleep patterns in this population.

Funding

This work was supported by the the National Institutes of Health grant [R01MH110462] and the University of Maryland Brain Behavior Initiative.

Deposit of Material in Data Repository

The data generated for Study 1 can be found in the NIMH Data Archive (NDA study account in process). The data for Study 2 will be shared on reasonable request to the corresponding author.

Disclosure Statement

The authors have no conflicts of interest to report.