Validation of the Delirium Observation Screening Scale (DOS) in a Hospitalized Older Population

Abstract

Delirium is challenging to diagnose in older populations. It is often reversible, and when detected, treatment can improve patient outcomes. Delirium detection currently relies on trained staff to conduct neurocognitive interviews. The Delirium Observation Screening (DOS) Scale¹ is a screen designed to allow faster, easier identification of delirium. In this validation study, conducted at an academic tertiary care center, we attempt to determine the accuracy of the DOS as a delirium screening tool in hospitalized patients over 64 years old. We compared DOS results to a validated delirium diagnostic tool, the Delirium Rating Scale-Revised-98 (DRS-R-98). We also assess the user-friendliness of the DOS by nurses via electronic survey. In 101 assessments of 54 patients, the DOS had sensitivity of 90% and specificity of 91% for delirium. The DOS is an accurate and easy way to screen for delirium in older inpatients.

INTRODUCTION

Delirium is a rapidly developing, fluctuating disturbance in consciousness, caused by a medical condition. The diagnosis of delirium is often missed, potentiating negative outcomes.^{2, 3} Regular delirium screening by nurses results in increased recognition and treatment.⁴ While multiple screening tools exist, many are cumbersome to execute. Efforts have been made to shorten them, but while the screening tools may predict adverse outcomes, there are concerns about their specificity.^{2, 3, 5–7} The Delirium Observation Screening (DOS) Scale is a brief screening tool based on observation. It has been validated in several patient populations, but no published studies took place in the United States or focused on an older, general medicine, inpatient population. Given the low numbers of patients in earlier validation studies, the effectiveness of the DOS for screening hospitalized, older patients is not yet fully established.

This study aimed to determine the ability of the DOS to screen hospitalized, older patients for delirium compared to a validated delirium diagnostic tool, the Delirium Rating Scale-Revised-98 (DRS-R-98).⁸ In addition, DOS acceptability, ease of use, and benefit were explored by surveying nurses.

METHODS

Participants

After Institutional Review Board approval, participants were selected by convenience sample from general medicine inpatients at a large, tertiary care, academic hospital. Eligible patients were age 65 years or older, admitted to a medicine inpatient unit, and spoke English. If participants were unable to consent, consent was obtained from the participant’s legally authorized representative.

Delirium Observation Screening (DOS) Scale

The DOS is a 13-point screen for delirium, based on DSM-IV delirium criteria, designed to be completed by a nurse (Appendix 1). Responses are dichotomous. Scores ≥ 3 were considered positive delirium screens.¹

Nurses on medicine units attended educational in-services on delirium recognition and use of the DOS. The DOS was embedded in the electronic medical record (EMR) and nurses are electronically prompted to chart DOS results every 12 hours for patients, age 65 years or older. Nursing staff utilized the DOS for one year prior to study start.

Delirium Rating Scale-Revised-98 (DRS-R-98)

The DRS-R-98 was used as the study reference standard.⁸ Scores ≥15 are indicative of delirium.⁹ All assessments were performed by a medical student (KG) trained to administer the DRS-R-98.

Data Collection

After consent, hospitalized participants were evaluated daily (Monday-Friday) using the DRS-R-98. Enrollment took place over a ten-week period. Nurses and researchers were blinded to other delirium assessment results until after participant discharge. Following discharge, additional data was collected from the EMR: age, gender, cognitive comorbidities, and nurse-charted DOS score. Cognitive comorbidities were classified as “no impairment,” “dementia,” or “cognitive impairment” based on the problem list and admission note. A psychiatrist (MW) confirmed questions of cognitive impairment.

The DOS score closest in time, within 24 hours of DRS-R-98 assessment, was used for comparison. If a DOS score was not charted within 24 hours of the DRS-R-98 evaluation, that assessment was excluded. Partial DRS-R-98 assessments were included only if there was enough information to classify a subject as delirious or not.

Nursing Survey

A 13 question nursing survey was developed and consisted of demographic, Likert-style, and multiple-choice questions, with opportunities for open-ended responses (Appendix 2). Survey design followed similar surveys investigating staff experiences and clinical functionality of other brief delirium screening tools, like the CAM-ICU.^{10, 11} The survey was distributed by e-mail to 435 nurses on 16 units. Coffee gift cards were raffled as participation incentive.

Statistical Analysis

Statistical analysis was completed using SPSS and SAS. DOS results were compared to the DRS-R-98 and validity statistics were calculated for delirium. Confidence intervals were calculated using the Clopper-Pearson method for binomial data. The Spearman rank correlation coefficient between DOS and DRS-98 score was calculated. PROC LOGISTIC modeled the relationship between positive DOS screens and delirium and created a receiver operating characteristic (ROC) curve using continuous DOS score to predict delirium. Because these models did not control for multiple observations per individual, PROC GENMOD was used to confirm the relationship between a positive DOS screen and delirium using a marginal logistic regression model accounting for repeated measures. In addition, we selected ten random samples of one observation per person, and validity statistics were calculated for each sample.

The nursing survey results were analyzed using descriptive statistics. Open-ended comments were reviewed in aggregate.

RESULTS

Participant Characteristics

Fifty-four participants enrolled in the study. Fifty-three were able to complete one DRS-R-98 and comprise the study sample (Table 1). Participants completed 1–5 daily DRS-R-98 assessments (mean 1.94, SD 0.90, mean length of admission 6.06 days). Of the 105 DRS-R-98 assessments, 101 were classifiable for delirium. Of the 101 DRS-R-98 assessments classifiable for delirium, 100 had a corresponding DOS score within 24 hours. Participant characteristics are listed in Table 1. Eight of the 53 participants (15%) had at least one positive DRS-R-98. Overall, 10 of the 101 delirium assessments diagnosed delirium (DRS-R-98 score ≥ 15).

TABLE 1.

Participant Characteristics

Characteristic	No Delirium (n=45)	Delirium* (n=8)
Age (years)
65–74 (n=26)	22	4
75–84 (n=15)	13	2
85+ (n=12)	10	2
Age (y), mean (SD, range)	77 (SD, 10, range 65–92)	76 (SD, 8.6, range 65–92)
Gender
Female (n= 33)	28	5
Male (n=20)	17	3
Cognitive status per chart
No impairment (n=45)	43	2
Cognitive impairment without dementia (n=5)	1	4
Dementia (n=3)	1	2

^*Any DRS-R-98 score > 15

DOS Validity

The mean and standard deviation of delirium screening scores are as follows: DRS-R-98 (mean 6.13, SD 4.74, range 0–20) and DOS (mean 1.22, SD 2.37, range 0–9). The Spearman correlation coefficient between DOS and DRS-R-98 scores was 0.58. DOS had a sensitivity of 90% (95% CI 56%–100%) and specificity of 91% (95% CI 83%–96%) compared to the DRS-98-R standard. There was only one false negative DOS screen out of 83 negative assessments (negative predictive value=99%, 95% CI 93%–100%). Out of the 17 positive assessments, nine were true positives (positive predictive value=53%, 95% CI 28%–77%) and seven scored in the subsyndromal range for delirium (DRS-R-98 score 8–14).

In analyses using ten samples, with one randomly selected observation per person, the mean sensitivity was 84.6%, ranging from 80% (95% CI 28%–99%) to 87.5% (95% CI 47%–100%). The mean specificity in these samples was 92%, ranging from 87% (95% CI 74%–95%) to 96% (95% CI 85%–99%).

Logistic Regression Models

All models confirmed positive DOS screens significantly predicted delirium. The traditional logistic regression model produced an odds ratio estimate of 92 (95% CI 10–824, p<0.0001) for a positive DOS screen predicting delirium. The marginal logistic regression model accounting for repeated measures produced a consistent estimate (OR 93, 95% CI 11–800, p< 0.0001). Continuous DOS scores predicted delirium (OR 2.1, 95% CI 1.5–2.9, p<0.0001) and the ROC curve supported the cutoff of DOS ≥ 3, corresponding to a predicted probability of 0.12 (Figure 1).

FIGURE 1.

DOS ROC Curve

Nursing Survey

The nursing survey had a response rate of 23% (N=98). The most robust results related to DOS administration: 87% (N=83) of nurses were confident in DOS administration; 92% (N=86) could complete the DOS in under three minutes; and 79% (N=74) agreed that performing the DOS is easy. There was less agreement on the value of the DOS: 37% agreed that the DOS is worth the time to perform; 25% agreed that the DOS enhances patient care; 36% agreed that the DOS provides valuable information for patient care. Over half the nurses (55%) reported that they perform the DOS 75–100% of the prompted times and 62% stated if the DOS was no longer required, they would not use it. Open-ended questions generated a wide range of responses, from supportive to critical of delirium screening and the DOS (Appendix 3).

DISCUSSION

This study demonstrated the effectiveness, efficiency and ease of use of the DOS as a delirium screening tool. The DOS exhibited a high sensitivity (90%) and specificity (91%). Similar to previous findings, the positive predictive value was only 53%, but the negative predictive value was 99%.¹² These results support that the DOS is consistently able to rule out delirium, with only one false negative in this study.

Nursing responses regarding user-friendliness are consistent with other studies; however, there was a knowledge gap related to how positive delirium screens can inform and change care for patients.¹ Education is a known barrier to integrating delirium screening tools secondary to the need for regular and extensive education, frequent reminders to screen, and regular evaluations of assessment quality.^{11, 13–15} Developing guidelines for responding to positive DOS screens and documenting its impact on care may incentivize use.

Study strengths include: strong evaluator consistency, blinding of evaluator and nurses, and responses from a broad range of nurses (14 of 16 units represented). Additionally, this study demonstrated the efficacy and ease of use of an EMR-prompted delirium screen. However, this study had several limitations, including a small sample size and a low incidence of delirium. The lower incidence is likely secondary to selection bias that resulted from difficulty consenting delirious subjects. The discordant time between DOS and DRS-R-98 assessments may have also influenced results; however, inclusion of data from the previous 8–24 hrs in both tools makes the temporal separation of assessments less impactful.

The ability of the DOS to accurately identify patients at high risk of delirium is useful for healthcare staff. Future work will include nurse and physician education to emphasize delirium understanding, the importance of regular screening, and the use of non-pharmacological interventions. Additional studies will include examination of the interventions and outcomes of patients who screen positive for delirium, in order to determine the long-term impact of delirium screening.

APPENDIX 1. Delirium Observation Screening Scale (DOS)¹

1. Doses off during conversation or activities. (Yes=1/No=0)

2. Is easily distracted by stimuli from the environment. (Yes=1/No=0)

3. Maintains attention to conversation or action. (Yes=0/No=1)

4. Does not finish question or answer. (Yes=1/No=0)

5. Gives answers that do not fit the question. (Yes=1/No=0)

6. Reacts slowly to instructions. (Yes=1/No=0)

7. Thinks they are somewhere else. (Yes=1/No=0)

8. Knows which part of the day it is. (Yes=0/No=1)

9. Remembers recent events. (Yes=0/No=1)

10. Is picking, disorderly, restless. (Yes=1/No=0)

11. Pulls IV tubing, feeding tubes, catheters, etc. (Yes=1/No=0)

12. Is easily or suddenly emotional. (Yes=1/No=0)

13. Sees/hears things which are not there. (Yes=1/No=0)

APPENDIX 2. DOS Nursing Survey

• This survey is designed to elicit your attitudes toward and comfort using the Delirium Observation Screening Scale (DOS). Thank you for taking the time to fill it out.

• Please report your number of years of experience/practice.

• What is your typical unit assignment? (choice of units plus free response)

• Performing the DOS is easy. (Strongly disagree, disagree, neither agree nor disagree, agree, strongly agree)

• How confident do you feel administering the DOS? (Very unsure, somewhat unsure, somewhat confident, very confident)

• The DOS provides me with valuable information for the treatment of my patients. (Strongly disagree, disagree, neither agree nor disagree, agree, strongly agree)

• Patient care is enhanced by using the DOS.(Strongly disagree, disagree, neither agree nor disagree, agree, strongly agree)

• How much time does it take you to complete the DOS? (Less than 1 minute, 1–3 minutes, 3–5 minutes, more than 5 minutes)

• The DOS is worth the time required to perform the assessment. (Strongly disagree, disagree, neither agree nor disagree, agree, strongly agree)

• What percentage of the time do you perform the DOS? (0–25% of the time, 25–50% of the time, 50–75% of the time, 75–100% of the time)

• In your opinion, how often should the DOS be performed? (It is currently done twice per day at 2AM and 2PM). (Once per day, twice per day, once per shift, more than once per shift, other [free response])

• What are some of the barriers to performing the DOS assessment? (Time it takes to perform assessment, difficulty using assessment, low applicability of results to patient care, other [free response])

• If the DOS was not required, would you still use it? Why or why not? (yes [free response], no [free response])

• Any additional comments about the DOS [free response]

APPENDIX 3. Nursing Survey Free Responses

If the DOS was not required, would you still use it? Why or Why not? (N=49)
Responses and Themes	Examples
Yes (n=15)	“It is a great tool for stopping and thinking about changes in my patient’s mental status” “It helps me prioritize where pt. needs/my interventions will be” “It makes one realize to what extent your patient is at risk for or has delirium” “I think it helps encourage nurses to consider factors they may be too busy or distracted to consider otherwise”
Provisionally (n=8)	“Only if delirium suspected” “I would use it if conversation with pt is not tracking” “If I was starting to notice my patient was getting confused and I was unclear on the reason”
No (n=26)	“Even with a score indicating a patient has delirium nothing happens to help treat the condition so it is a waste of my time” “It doesn’t seem to change the way that I care for my patient” “Just another thing to chart that takes up time” “I tend to look at the neuro assessment on whether they are confused” “Based on my personal nursing skills/judgment, you can tell if someone has delirium” “It is mainly for the physicians. I can interpret the mentation of my patients without it.”
Any additional comments about the DOS?
Supportive (n=5)	“I think having a scale that assesses the delirium risk/status of a patient is very important, as sometimes delirium goes unnoticed. I love that our hospital is making efforts to better patient care, especially for the geriatric populations. I would like more information on HOW the scale can be better implemented and used to provide better care for my patients.”
Non-supportive (n=13)	“It can be helpful to compare small changes in mental status, but it does not always represent an accurate assessment of the patient due to the lack of further details” “Be careful with have so many assessments- sometimes it can be unrealistic the amount of charting nursing has to do. My concern is that they really don’t get done or just seen as ‘another thing that nurses have to do’ instead of seeing it as a useful tool.”