Validation of the Insomnia Severity Index (ISI) for Identifying Insomnia in Young Adult Cancer Survivors: Comparison with a Structured Clinical Diagnostic Interview of the DSM-5 (SCID-5)

Abstract

Objectives:

Insomnia is a common problem affecting young adult cancer survivors (YACS) even years after treatment, yet it often goes unidentified and untreated. The Insomnia Severity Index is a widely-used insomnia measure, but has not been studied as an insomnia screener for YACS. The goal of this study was to validate the ISI in YACS by determining its utility in identifying YACS with insomnia disorder diagnosed with the Structured Clinical Interview for the DSM-5 (SCID-5).

Methods:

250 YACS completed the ISI and SCID-5 Insomnia Module. Area under the curve (AUC) was calculated to reflect the ISI’s discrimination between YACS with and without SCID-5 insomnia disorder. An ISI cut-off score with sensitivity ≥.85 and specificity ≥.75 was deemed acceptable.

Results:

Of 250 participants, 52 met criteria for SCID-5 insomnia disorder diagnosis. The ISI had excellent discrimination, with an AUC=.91. A cut-off score ≥8 met study clinical screening criteria with sensitivity of .85 and specificity of .77. A cut-off score ≥7 with a higher sensitivity (.96) but lower specificity (.70) was noted as a potential alternative. Cut-off scores ≥12 and ≥14 were recommended for applications prioritizing overall accuracy.

Conclusions:

Results support validity of the ISI for identifying YACS with insomnia disorder. For clinical screening, data support the use of an ISI cut-off score ≥8 in YACS, and additional cut-off scores were found for research purposes or higher sensitivity. Results of this study and prior studies of the ISI offer important reminders that cut-off scores derived from different populations are not generalizable.

1. Introduction

With improvements in treatment for many cancers, the majority of young adult cancer survivors (YACS; ages 18–40) can now expect to be cured of their cancer.¹ Unfortunately, YACS are particularly susceptible to late-effects of cancer treatment, which interfere with their health and adjustment. ^2–4 Insomnia is a well-known late-effect of cancer therapy associated with poorer quality of life and psychosocial adjustment.^5–8 Nearly 20% of YACS report chronic insomnia even many years after their treatment.^9,10 If left untreated, chronic insomnia is associated with many negative physical and psychosocial outcomes, interfering with cognitive functioning and psychological well-being just as YACS are trying to reengage with normal life and catch up on missed educational and social opportunities.^8,9,11–14

If identified, insomnia is highly treatable, with considerable evidence demonstrating the efficacy of CBT-I in cancer survivors. To insure that YACS with insomnia are identified and treated, routine assessment as part of clinical care will be required.^15,16 Though follow-up care guidelines for cancer survivors support the need for clinicians to inquire about insomnia,¹⁷ they provide little empirical evidence to guide clinicians in selecting or implementing insomnia assessments. Moreover, research confirms that clinicians treating adult and adolescent cancer patients do not systematically assess insomnia as part of their survivorship care.^16,18 To address this need, patient-reported measures of insomnia that are both brief and valid are critically important to facilitate identification of insomnia in YACS.

Because of the time and training needed, structured diagnostic interviews are not widely available outside of specialty clinics and research settings. The Insomnia Severity Index (ISI) is a well-regarded self-report measure of insomnia that has been widely used clinically and in intervention studies of cancer survivors and other populations.^19–26 As it is brief and easily administered, the ISI is potentially appropriate for clinical screening. While the ISI has been supported by several psychometric studies, not all have recommended the same clinical cut-off score.^22,23,25,26 In addition, we are not aware of studies validating the ISI specifically in YACS, which may limit its adoption in YACS programs, as validity and optimal cut-off scores of self-report measures can vary from one population to another. To address this, we set out to validate the ISI in YACS and identify optimal cut-off scores for identifying insomnia disorder as defined by current DSM-5 criteria and assessed with a structured diagnostic interview. By selecting DSM-5 insomnia disorder as our criterion, we aimed to focus on YACS with significant insomnia and provide medical providers caring for them with the type of diagnostic criterion they commonly use for treating and referring these patients.

2. Methods

2.1. Participants and Procedures

Participants were 250 YACS enrolled on an NCI-funded study called E-Quest Stress and Coping, (EQuest-SC) a study of posttraumatic stress symptoms and health outcomes in YACS (1R21CA223832, 1R21CA223832). To be eligible for EQuest-SC, YACS had to be 18–40 years of age, able to complete study measures independently in English, and at least 6 months off treatment. YACS were recruited at a scheduled oncology clinic visit at a single cancer center. After consenting, participants completed self-report measures, including the ISI, during a single study visit. After completing self-report measures, participants completed a structured diagnostic interview, the SCID-5. Of note, for this study we intentionally enrolled equal numbers of males and females, as well as an equal number of participants diagnosed < age 21 and those diagnosed ≥ age 21. To protect participant confidentiality, this was an anonymous study with no identifying information linked to any study measures. All procedures were approved by the Cancer Center’s Institutional Review Board.

Two hundred and ninety-nine YACS were screened for eligibility, and 8 participants were ineligible (6 were < 6 months off-treatment, and 2 had severe cognitive impairment). Of the 291 eligible participants, 4 actively opted-out of participation, and 28 passively opted-out (e.g. expressed interest but failed to enroll after multiple attempts to contact them). 259 survivors enrolled on the study but 9 failed to complete study measures and were unevaluable, leaving 250 enrolled and evaluable participants reported on here.

2.2. Measures

Insomnia Severity Index (ISI):¹¹

The ISI is a self-report 7-item checklist asking about insomnia symptoms over the prior two weeks. The first three items inquire about problems falling asleep, maintaining sleep, and early-morning awakenings. The last four items inquire about satisfaction with current sleep, noticeability of sleep problem(s) to others, worry about sleep, and interference of sleep problems with daily functioning.

Items are rated on a Likert scale and summed to obtain a total score, ranging from 0 to 28, with a higher score indicating greater insomnia severity. As noted (see section 1), the ISI is commonly used in insomnia research and has been widely used in clinical applications and research,^20–25 including with cancer populations.^20,26

Structured Clinical Interview for the DSM-5 (SCID-5):¹²

The SCID-5 is a semi-structured clinical interview for identifying psychiatric diagnoses based on DSM-5 criteria. The SCID-5 is the most recent version of the SCID; previous versions of the SCID have been amongst the most widely used diagnostic measures for studying mental disorders in psychiatric and medical populations.^27,28 The SCID-5 incorporates a skip logic so that once a respondent denies a critical symptom or fails to meet a diagnostic criterion, no other items for that diagnosis are administered. The SCID-5 was administered by a single trained clinical research coordinator who was blind to all responses on study measures. Only “current” symptoms were queried in the interview, defined by the SCID-5 as present in the last 30 days.

The SCID-5 has a modular structure allowing it to be tailored so only relevant sections can be administered. The screener is a preliminary section of the SCID asking respondents initial questions from several diagnostic modules, to help determine what diagnostic modules should be administered. For E-Quest-SC, we included the initial question from the insomnia disorder module asking about dissatisfaction with sleep in the screener. Based on pilot testing of the SCID-5 with YACS, we noted many respondents did not initially understand the intent of the insomnia screener question item and made minor modifications to the item to make it clearer and easier for respondents to answer. Specifically, to highlight the DSM-5 criterion the item is intended to assess, “Dissatisfaction with sleep over the past three months,” we added the words “dissatisfied with your sleep” to this item (Table 1).

Table 1:

Modifications Made to the SCID-5 Insomnia Module

Insomnia Module
	SCID Criterion	Description	Change Made
Removal of Questions
	Criterion G	G [Primary Insomnia]: The insomnia is not attributable to the physiological effects of a substance (e.g., a drug of abuse, a medication).	This criterion was not applied in scoring the insomnia module.
	Criterion H	H: Coexisting mental disorders and medical conditions do not adequately explain the predominant complaint of insomnia.	This criterion was not applied in scoring the insomnia module.
	Criterion F	F: The insomnia is not better explained by and does not occur exclusively during the course of another Sleep-Wake Disorder (e.g., Narcolepsy, a Breathing-Related Sleep Disorder, a Circadian Rhythm Sleep-Wake Disorder, a Parasomnia).	The criterion was not applied in scoring the insomnia module.
Modification of Questions	SCID Item #	Description	Change Made
	OH2	Item is designed to inquire about DSM-5 criterion: “Dissatisfaction with sleep over the past three months.”	This item was modified to more closely match DSM-5 criterion. Item 1 from the Insomnia Module was administered as part of the screener, and the text was modified for clarity (text in bold added).
		Over the past 3 months, since (3 MONTHS AGO), have you been dissatisfied with your sleep? For example, have you not been getting enough rest or not feeling rested?

YACS reporting insomnia on the screener item were administered the SCID-5 insomnia module to determine if they met Criterion A (dissatisfaction with sleep quantity), Criterion B (sleep disturbance causing distress or impairment), Criteria C and D (sleep difficulty occurring ≥ 3 nights per week for at least 3 months), and Criterion E (sleep difficulty occurs despite adequate sleep opportunity). These items were scored according to standard SCID scoring algorithms, and if all criteria were met, then participants were classified as meeting SCID-5 insomnia disorder diagnosis. As the goal of the study was to evaluate the ISI’s ability to accurately identify individuals with clinically significant insomnia, we did not administer SCID items inquiring about Criterion F (insomnia not better explained by other sleep-wake disorder), Criterion G (insomnia attributed to physiological effects of a substance), and Criterion H (insomnia not due to coexisting mental or medical disorders). These “rule-out” criteria for insomnia disorder are not covered by the ISI or other self-report measures of insomnia, and we expect clinicians would be interested in using the ISI to identify individuals with significant insomnia symptoms, even if they were secondary to substance use or another disorder.

2.3. Statistical Analyses

Descriptive statistics were used to describe the sample on demographic, medical, and insomnia characteristics. Receiver operating characteristic (ROC) analyses were used to describe the classification agreement between the ISI and the SCID-5 insomnia module. Area under the curve (AUC) was calculated to reflect the ISI’s discrimination between YACS with and without an insomnia disorder across the full range of ISI scores; AUC values were defined to have good ( ≥.80) or excellent (≥.90) discrimination.^29–33

To evaluate utility of the ISI for screening YACS for insomnia disorder, potential cut-off scores between 5 and 15 were evaluated. Sensitivity was calculated to capture the ISI’s accuracy in identifying YACS with a SCID-5 insomnia diagnosis (true positives), and specificity was calculated to capture the ISI’s ability to correctly identify YACS without a SCID-5 insomnia diagnosis (true negatives). Total predictive value was also analyzed, reflecting the percentage of correctly identified participants overall. For evaluating the ISI as a clinical screening measure, we prioritized identifying cut-off scores with high sensitivity, to ensure all affected survivors were identified, while maintaining at least moderate specificity. As we have done in prior studies,^34,35 we specified a priori that an ISI cut-off score with a sensitivity ≥.85 and a specificity ≥.75 was adequate to recommend its use for clinical screening. Analyses were conducted using the Statistical Package for Social Sciences Version 234.0 (SPSS 24.0).

3. Results

3.1. Descriptive Statistics

Participants were 125 males (50.0%) and 125 females (50.0%) aged 18–40 (M = 29.3, SD = 6.2; Table 2). Participants first cancer diagnoses were lymphoma (106, 42.4%), leukemia (49, 19.6%), breast (27, 10.8%), sarcoma (27, 10.8%), neuroblastoma (9, 3.6%), brain tumor (7, 2.8%), and other solid tumors (25, 10.0%). One hundred and twenty-five (50.0%) participants were diagnosed before the age of 21. Participants were an average of 9.6 years from their first cancer diagnosis (SD = 7.6). Of the 250 participants, 106 (42.4%) endorsed the SCID-5 insomnia screener question and were administered the Insomnia Module; 52 participants (20.1%) met criteria for an insomnia diagnosis on the SCID-5. On the ISI, scores ranged from 0–28 with mean score of 7 (SD = 5.8). Internal consistency of the ISI was high (α = .91).

Table 2:

Demographic & Medical Characteristics of Participants (N=250)

Participant Characteristics	M (SD)	N (%)
Demographic Characteristics
Age	29.2 (6.2)
Gender
Female		125 (50.0)
Male		125 (50.0)
Race/Ethnicity
Caucasian		203 (81.2)
Other Ethnic Background		47 (18.8)
Highest Education Level
Postgraduate Level (e.g., MA, MBA, MD, PhD)		75 (30.0)
College Graduate		100 (40.0)
Some College or a Two-Year Degree		42 (16.8)
Training after High School		5 (2.0)
Completed High School or GED		26 (10.4)
Some High School (or less)		2 (0.8)
Medical Characteristics
Years Since Cancer Diagnosis	9.6 (7.6)
First Cancer Diagnosis
Lymphoma		106 (42.4)
Leukemia		49 (19.6)
Breast		27 (10.8)
Sarcoma		27 (10.8)
Neuroblastoma		9 (3.6)
Brain Tumor		7 (2.8)
Other Solid Tumorsa		25 (10.0)

^aOther Solid Tumors Include: Colon or Colorectal, Kidney or Wilms, Testicular, Retinoblastoma, Melanoma, Thyroid, and Head & Neck.

^bOther Ethnic Background includes: African American, Hispanic, Asian or Pacific Islander, Native American or Alaskan Native, or Mixed Race

3.2. Comparison of ISI with SCID-5 Insomnia Diagnosis

Using the SCID-5 insomnia diagnosis as the criterion variable, ROC analysis showed the ISI had overall excellent discrimination, with an AUC of .91. Evaluating sensitivity and specificity of potential ISI cut-off scores, we found a cut-off score of ≥ 8 met study criteria for clinical screening with sensitivity of .85 and specificity of .77 (Table 3). As expected, lower cutoff scores increased the sensitivity of the ISI for the detection of survivors with an insomnia diagnosis, but decreased specificity, indicating that more false-positive findings would result. We did note that a slightly lower cut-off score of ≥ 7 had a very high sensitivity of .96, indicating it would be expected to miss only 4% of YACS with an insomnia diagnosis on the SCID-5, though its specificity of .70 indicates 30% of YACS identified by this cut-off score as having insomnia would be expected to be false positives cases without a SCID insomnia diagnosis. Nonetheless, in settings where this lowered specificity is acceptable, with its very high sensitivity, a cut-off score of ≥ 7 could be considered for insomnia screening in YACS.

Table 3:

Sensitivity and Specificity for Insomnia Diagnosis Cut-Off Scores (N=250)

ISI Scores	Insomnia Diagnosis (N=52/250)
Alternative Cut-Offs	Sensitivity	Specificity	% Correct
≥ 5	1.00 (0.91–1.00)	0.54 (0.47–0.61)	63.6
≥ 6	1.00 (0.91–1.00)	0.64 (0.57–0.71)	71.6
≥ 7	0.96 (0.86–0.99)	0.70 (0.63–0.76)	75.2
≥ 8	0.85 (0.71–0.93)	0.77 (0.71–0.83)	78.8
≥ 9	0.79 (0.65–0.88)	0.83 (0.77–0.88)	82.0
≥ 10	0.75 (0.61–0.86)	0.89 (0.83–0.93)	86.0
≥ 11	0.67 (0.53–0.79)	0.90 (0.85–0.94)	85.6
≥ 12	0.60 (0.45–0.73)	0.93 (0.89–0.96)	86.4
≥ 13	0.52 (0.38–0.66)	0.93 (0.89–0.96)	84.8
≥ 14	0.52 (0.38–0.66)	0.96 (0.92–0.98)	86.8
≥ 15	0.37 (0.24–0.51)	0.96 (0.93–0.98)	84.0

Figure 1 demonstrates the practical implications of applying these two ISI cut-off scores to a hypothetical example of screening 100 survivors (20% with an insomnia diagnosis). Applying an ISI cut-off score of ≥ 8 (Figure 1a), we would expect 17 (85%) of the 20 YACS with an insomnia disorder to be correctly identified and referred for appropriate follow-up, and the other 3 YACS with insomnia disorder to be missed (false negative cases) and not receive further evaluation. For the 80 YACS without insomnia disorder, 62 (77%) would be expected to have true negative results, but 18 of these YACS would be erroneously sent for further evaluation (false positive results). In all, 35 of the 100 survivors would be referred for further evaluation of insomnia, although half (18 YACS; 51.4%) would not actually have an insomnia diagnosis. If the alternative cut-off score of ≥ 7 were used (Figure 1b) we would expect 19 (96%) of the 20 YACS with an insomnia disorder to have to be correctly identified and referred for appropriate follow-up, and the other 1 survivor with insomnia disorder to be missed (false negative case). For the 80 YACS without an insomnia disorder, 56 (70%) would be expected to have true negative results, but 24 would be erroneously sent for further evaluation (false positive cases). In all, 43 of the 100 survivors would be referred for further evaluation of insomnia, although 55.8% (24 YACS) would not actually have an insomnia disorder diagnosis.

Figure 1a.

Expected clinical decisions using the Insomnia Severity Index (ISI) to screen for a diagnosis of Insomnia (cut-off ≥ 8).

Figure 1b.

Expected clinical decisions using the Insomnia Severity Index (ISI) to screen for a diagnosis of Insomnia (cut-off ≥ 7).

For comparison, we also prepared a similar figure (Figure 1c) for a widely used ISI cut-off score of ≥ 10,²³ which had a sensitivity of .75 and a specificity of .89 in our YACS sample. Applying this to the same hypothetical example (i.e.,100 survivors, 20 with an insomnia diagnosis), we would expect 15 (75%) of the 20 YACS with an insomnia disorder to be correctly identified and referred for appropriate follow-up, and the other 5 survivors with insomnia disorder to be missed (false negative case). For the 80 YACS without an insomnia disorder, 71 (89%) would be expected to have true negative results, but 9 would be erroneously sent for further evaluation (false positive cases).

Figure 1c.

Expected clinical decisions using the Insomnia Severity Index (ISI) to screen for a diagnosis of Insomnia (cut-off ≥ 10).

As some users may be interested in applying the ISI to YACS for purposes other than for clinical screening, we did note the potential utility of other possible cut-off scores. For some purposes, such as identifying YACS with diagnosable insomnia disorder for a research study, users may be interested in ISI cut-off scores that prioritize high specificity and minimize false positives. For that purpose, we noted that scores of ≥ 12 (specificity = .93), or ≥ 14 (specificity = .96), would be best suited based on our data. As these cut-off scores demonstrate the highest proportions of total correct classification (86.4% and 86.8% respectively), they may also be useful for studies where overall accuracy of classification is most important without regard to differences in false positive or false negative screening results.

4. Discussion

In this study we sought to evaluate the validity of the ISI as a clinical screening measure in YACS by comparing it to the SCID-5 Insomnia Module. Structured diagnostic interviews are the gold standard for assessing mental disorders in both clinic and research settings, and both the overall SCID-5 interview and the SCID-5 Insomnia Module are well regarded.^27,28,36,37

Our results indicate the ISI has a very strong association to the SCID-5 Insomnia Module, with an AUC of .91. This result provides important new information verifying validity of the ISI for identifying YACS who have significant symptoms of insomnia that require intervention.

In addition to supporting validity of the ISI in YACS overall, results supported the utility of specific cut-off scores for clinical and research application. Though a cut-off score of of ≥ 8 (sensitivity = .85; specificity = .77) met our study criteria for clinical screening, we also noted that an ISI cut-off score ≥ 7 (sensitivity = .96; specificity = .70) could be considered for clinical screening because of its very high sensitivity, with a specificity just below our prior criteria. In addition, we recommended a slightly higher cut-off score of ≥ 12 for use in determining eligibility for studies of insomnia disorder in YACS, and noted that a cut-off score of ≥ 14 had greatest overall accuracy (86.8%).

In previous studies, recommended cut-off scores for the ISI has been highly variable, and typically higher than the scores of 7 and 8 that our data support for screening YACS. For example, one of the most widely cited ISI studies (Morin et al.),²³ found a cut-off score of ≥ 10 was optimal in their community sample when comparing the ISI to a single yes/no item on insomnia (sensitivity = .86, specificity = .87). As noted (see section 3.2), this cut-off score has an unacceptably low sensitivity (.75) in our sample.

Similarly, our recommended clinical cut-off scores were also lower than the cut-off score of ≥ 14 found in a primary care population when the ISI was compared to the Insomnia Diagnostic Interview (IDI; sensitivity = .82, specificity = .82),²² and ≥ 11 in a psychiatric population when the ISI was compared to DSM-5 criteria collected by interviewers using the Brief Insomnia Questionnaire (sensitivity = .88, specificity = .65).²⁴ Differences in recommended cut-off scores across these studies are likely due to the different criterion measures used for validating the ISI as well as differences in insomnia burden, age, and comorbidities in the samples. A previous study by Savard et al. in breast cancer survivors²⁶ also supported a cut-off score of ≥ 8 for clinical screening when the ISI was compared to the Insomnia Interview Schedule (IIS); however, their reported sensitivity with this cut-off scores was somewhat higher than we found (0.94 vs. 0.85) and their specificity reported was much lower (0.47 vs. 0.77). Taken together, results of our report and prior studies of the ISI offer important reminders that cut-off scores may not be generalizable from one population to another.

There are several study limitations to be noted. As intended, our results are specific to YACS and users should be cautious about applying them to other groups. Additionally, because our sample was predominantly white and well educated (Table 2), future studies with more socioeconomic and demographically diverse participants will be needed to investigate how well these results generalize. Similarly, though we enrolled YACS across a variety of cancers, results may have been affected by the particular distribution of cancer, treatments, and late-effects in the sample. Replication of our finding in new YACS will be useful to address this. As we assessed each participant at only one time-point, we were unable to assess temporal stability of ISI ratings or classification using recommended cut-off scores. However, because reliability is necessary but not sufficient for validity,³⁹ findings of their close concordance with the SCID-5 provide strong indirect support for their reliability. This is the case because temporal instability of the ISI or cut-off scores would be a source of measurement error which would limit their association with a valid criterion measure. For that reason, our finding that the ISI scores had excellent concordance with the SCID-5 (AUC = .91) supports both the validity and reliability of the ISI. Similarly, it is also important to note that the ISI and SCID-5 were administered on the same day, using their standard reference periods of 2 weeks and 3 months, respectively. It is possible that agreement between the ISI and the SCID-5 may have been attenuated by this difference. Future studies investigating the temporal fluctuations in insomnia severity would be needed to assess this,⁴⁰ however given the strong relationship found between the ISI and the SCID-5, any attenuation is likely to be small. Finally, we evaluated the concordance of the ISI to the SCID-5 insomnia module using specific a priori criteria for sensitivity and specificity, and results would have been different had we applied different criteria. However, the criteria used are consistent with those applied in similar validation studies,^34,35 and we provide sensitivity and specificity information for a large range of ISI cut-off scores so that our data are informative to users who wish to apply different criteria for selecting an ISI cut-off score.

4.1. Conclusions

Despite these limitations, the current study provides important new information validating the ISI as a screening measure for insomnia in YACS. Clinical programs providing follow-up care to YACS aim to assess their health across a wide range of medical and psychological outcomes. This limits provider time for any specific outcome, and requires that assessments be brief and efficient.^17,41 Moreover, availability of sleep specialists is often limited and likely contributes to the observed low rates of insomnia assessment in survivorship care.^16,18,42 Because of its ease of administration and low burden, the ISI is particularly well-suited to address these challenges.^20–26 The 7-item ISI can be completed in less than 5 minutes, and providers can be confident that YACS with low scores are unlikely to have an insomnia disorder. YACS with higher scores can be offered additional assessment and treatment for insomnia in the survivorship setting if available, or referred out for appropriate care. Incorporating the ISI into routine assessment in this way should allow survivorship programs to identify those YACS with significant insomnia requiring treatment, with minimal burden to other patients or providers.

Highlights.

• Validity of the ISI in YACS was evaluated by comparing it to the SCID-5

• The ISI effectively discriminated YACS with or without insomnia disorder (AUC=.91)

• An ISI cut-off score ≥8 is recommended for screening YACS for insomnia

• Results suggest utility of ISI cut-off scores may vary across patient groups

Abbreviations:

YACS

Young adult cancer survivors

SCID-5

Structured clinical interview for the DSM-5

AOC

Area under the curve

ROC

Receiver operating characteristics