The Minimal Clinically Important Difference of the Richmond Agitation-Sedation Scale in Cancer Patients with Agitated Delirium

Abstract

Background

The Richmond Agitation-Sedation Scale (RASS) is commonly used to assess psychomotor activity; however, its minimal clinically important difference (MCID) has not been determined. The objective of this study is to identify the MCID for RASS using two anchor-based approaches.

Methods

This is a secondary analysis of a randomized controlled trial to compare the effect of lorazepam versus placebo as an adjuvant to haloperidol for persistent agitation in patients with delirium. The primary outcome was change in RASS (10-point numeric rating scale ranging from −5 [unarousable] to +4 [combative]) from baseline to 8 hours after treatment administration. The sensitivity-specificity and within-patient change methods were used to identify the MCID, with patient comfort after study intervention as perceived by caregivers and nurses being the anchor.

Results

90 patients were randomized and 58 (64%) received the study medication for restless/agitation (mean baseline RASS 1.6). 23 (61%) caregivers and 23 (55%) of nurses perceived that the patient was more comfortable after treatment. Using the sensitivity-specificity method, the optimal RASS reduction was ≥4-points according to both caregivers (sensitivity 61%, specificity 80%; area under the curve [AUC] 0.71) and nurses (sensitivity 73%, specificity 84%; AUC 0.78). The RASS cutoff based on within-patient change method was similar (−4.2 for caregivers; −4.0 for nurses).

Conclusion

For patients with persistent restlessness/agitation, a reduction of ≥4 points in RASS was considered to be the MCID for both nurses and caregivers. These preliminary findings may have implications for sample size calculation and interpretation of treatment effect in future delirium trials.

Introduction

Delirium is a highly distressing and difficult to manage neuropsychiatric syndrome that commonly occurs in the last weeks to days of life.^{1, 2} Over half of patients with delirium have hyperactive features characterized by restlessness and agitation, which can result in significant psychological distress among patients, their caregivers and healthcare professionals, and may pose a safety concern for patients.³

Management of agitated delirium is complicated by the paucity of randomized controlled trials to inform clinical practice, leading to few evidence-based treatment options for this devastating syndrome. There are many contributors to the lack of research on delirium, such as difficulties in enrolling extremely ill patients, scarce research funding, shortage of academic investigators, and limited number of validated assessment tools to examine delirium-related outcomes.¹

The Richmond Agitation-Sedation Scale (RASS) is one of the most commonly used instruments to assess the level of psychomotor agitation and sedation in clinical practice and research. This clinician-based observation scale is simple to use, easy to interpret, and has been validated in multiple care settings.⁴ It has been incorporated as the primary or secondary outcome measure in several clinical trials on delirium.^5–8 However, the minimal clinical important difference (MCID, i.e. the smallest change in this scale that is clinically meaningful) for patients who were being treated for agitation has yet to be defined. Availability of the MCID for RASS would facilitate proper sample size justification for studies on agitated delirium, and also enable clinicians to properly interpret the study findings. In this study, we conducted a post-hoc secondary analysis of a recently completed randomized trial on agitated delirium to determine the MCID for RASS using the sensitivity-specific approach.

Methods

Study Design

This is a secondary analysis of a double-blind, randomized controlled trial to compare the use of single dose lorazepam versus placebo as adjunctive therapy to haloperidol in patients with persistent agitated delirium. Details of study methodology has been published recently.⁶ Briefly, inclusion criteria included patients who were adults (aged ≥18), had a diagnosis of advanced cancer, admitted to the acute palliative care unit at the University of Texas MD Anderson Cancer Center had a diagnosis of delirium based on DSM-IV-TR criteria, and had a RASS score of ≥+2 within 24 hours of enrolment despite scheduled haloperidol of 1–8 mg/day. Exclusion criteria included dementia, use of benzodiazepines or chlorpromazine within the past 48 hours, contraindications to neuroleptics (i.e. Parkinson’s disease, myasthenia gravis, acute narrow angle glaucoma, seizure disorders, documented QTc prolongation, hypersensitivity) or contraindications to benzodiazepines (i.e. hypersensitivity).

Upon enrolment, all patients were given scheduled haloperidol 2 mg intravenously every 6 hours and 2 mg intravenously every 1 hour as needed for agitation. Once they became restless/agitated (i.e. RASS ≥+1) requiring the rescue medication in the opinion of the bedside nurse, they were given either lorazepam 3 mg in 25 cc of 0.9% normal saline plus haloperidol 2 mg intravenously, or identically appearing placebo plus haloperidol 2 mg intravenously.⁶ At the beginning of the study, the RASS threshold for study medication administration was set at ≥+2. This threshold was revised to RASS ≥+1 after 7 months to ensure patients who were restless could also receive study medications.

Data Collection

The primary outcome was change in RASS from baseline to 8 hours after study medication administration. RASS is a single item assessed by the bedside nurse. It is a 10-point ordinal scale that ranges from +4 to −5, where 4=combative, 3=very agitated, 2=agitated, 1=restless, 0=alert and calm, −1=drowsy, −2=light sedation, −3=moderate sedation, −4=deep sedation and −5=unarousable. RASS has been tested extensively for construct validity and has high inter-rater reliability (0.91).^{4, 9–12}

We also asked the caregivers (legally authorized surrogate decision makers) and nurses to independently report their perceived level of patient comfort.⁶ This was considered as the anchor for MCID determination in this study because one of the key goals of pharmacologic therapy in the agitated delirium setting was to improve patient comfort. Both caregivers and nurses were blinded to study assignment and answered the following question on the day after treatment: “In my opinion, the patient was more comfortable after the study medication.” The response was recorded on a 5-point Likert scale ranging from “strongly agree”, “agree”, “neutral”, “disagree”, and “strongly disagree”. For analysis purposes, “strongly agree” and “agree” were considered as agreement. In this study, caregiver assessment was used as the main anchor, and nurses as the secondary anchor.

Statistical analysis

The study sample size was based on our calculation that 26 patients per arm would allow us to detect a mean between-arm difference in RASS score of 0.32 (0.5 effect size, based on a standard deviation of 0.63) with alpha=5% using a two-sided t-test.⁶ We summarized our data with descriptive statistics.

To determine the MCID for RASS, we adopted the statistical analysis plan previously used to identify the MCID for the Edmonton Symptom Assessment System.^{13, 14} In the sensitivity-specificity approach, caregiver perceived patient comfort was used as the benchmark against which RASS changes was anchored and calibrated. Specifically, improvement was indicated by the caregiver answering “agree” or “strongly agree”, while other answers (“neutral”, “disagree” and “strongly disagree”) were considered as no improvement. We plotted the receiver-operating characteristic (ROC) curves with true positive rate (sensitivity) on the y-axis and false positive rate (1 – specificity) on the x-axis. We then calculated the area under the curve (AUC), and determined the optimal cutoff for improvement based on the Youden J’s index and top left approaches. We repeated the same analysis using perceived patient comfort by the nurse as the anchor.

We also estimated MCID using within-patient change by computing the average RASS change for the perceived comfort question categories “agree” because this category represented the smallest perceived change.¹⁵

MCID is often estimated using the distribution-based approach, in which 0.5 standard deviation of baseline score is considered as a cutoff.¹⁶ However, the nature of this study design resulted in a highly homogenous RASS score distribution at baseline because patients only started the study intervention when they reached the pre-specified RASS threshold. Thus, distribution approach was not used in this study to estimate the MCID.

The Statistical Analysis Software 9.3 (SAS Institute Inc., Cary, NC, USA) software was used for statistical analysis. Statistically significance was declared when the P-value is <0.05.

Results

Patient characteristics

The demographics has been reported previously.⁶ Among the 58 patients who received study medication, 52 (89%) completed the first 8 hours of observation. The mean age was 65 (range 30, 90), 27 (47%) were female, 44 (76%) were White Race, 46 (79%) had metastatic disease, and 53 (91%) had Karnofsky Performance Status of 30% or lower (Table 1). The most common malignancies were respiratory (n=14, 24%) and gastrointestinal (n=13, 22%). The median Memorial Delirium Assessment Scale was 29.5 (IQR 20, 30). 28 (48%), 26 (45%) and 4 (7%) of patients had RASS of +1, +2 and +3 at the time of study medication administration, respectively, with a mean RASS of 1.6 (SD 0.6). Table 2 shows the characteristics of the 58 caregivers included in this study.

Table 1.

Characteristics of Patients involved in this Study (N=58)

	Total N (%)a
Age, mean (range)	65 (30, 90)
Female sex	27 (46.6)
Race
Caucasian	44 (75.9)
Black	8 (13.8)
Hispanic	2 (3.4)
Other	4 (6.9)
Education
High school or less	15 (25.9)
Some college	17 (29.3)
Completed college	24 (41.4)
Not available	2 (3.5)
Cancer type
Breast	5 (8.6)
Gastrointestinal	13 (22.4)
Genitourinary	3 (5.2)
Gynecological	4 (6.9)
Head and neck	1 (1.7)
Hematological	10 (17.2)
Respiratory	14 (24.1)
Other	8 (13.8)
Cancer stage
Metastatic	46 (79.3)
Locally advanced	1 (1.7)
Recurrent or persistent	11 (18.9)
Karnofsky Performance Status
10%	12 (20.7)
20%	27 (46.6)
30%	14 (24.1)
40%	5 (8.6)
Reason(s) for APCU admissionb
Delirium	31 (53.4)
Pain	48 (82.8)
Dyspnea	21 (36.2)
Others	23 (39.7)
Memorial Delirium Assessment Scale, median (IQR)	29.5 (20.0, 30.0)

^aunless otherwise specified

^bSome patients had more than one reason for admission

Table 2.

Caregiver Characteristics

Variable		Lorazepam + Haloperidol N=29 (%)*	Placebo + Haloperidol N=29 (%)*	Both N=58 (%)*
Age	Mean (range)	57 (18, 81)	61 (40, 84)	59 (18, 84)
Sex	Male	8 (28)	9 (31)	17 (29)
	Female	21 (72)	20 (69)	41 (71)
Race	Asian	3 (10)	1 (3)	4 (7)
	Black or African American	4 (14)	4 (14)	8 (14)
	Hispanic	1 (3)	0	1 (2)
	White	21 (72)	23 (79)	44 (76)
	Other	0	1 (3)	1 (2)
Relationship	Children	11 (38)	7 (24)	18 (31)
	Parent	3 (10)	2 (7)	5 (9)
	Sibling	1 (3)	0	1 (2)
	Spouse/significant other	14 (48)	19 (66)	33 (57)
	Other	0	1 (3)	1 (2)

^*Unless otherwise specified

Change in RASS

As reported previously,⁶ patients in the lorazepam/haloperidol arm had a significant within-arm decrease in RASS within the first 30 minutes of study medication administration (mean change −3.6, 95% CI −4.3, −2.9) and this effect was sustained at 8 hours (mean change −4.1, 95% CI −4.8, −3.4). Patients in the placebo/haloperidol arm also had a reduction in RASS at 30 minutes (mean change −1.6, 95% CI −2.2, −1.0) and at 8 h (mean change −2.3, 95% CI −2.9, −1.6). By 8 hours, 17 of 26 (65%) of patients had a 4 point or greater reduction in RASS in the lorazepam/haloperidol group compared to 7 of 26 (27%) of patients in the placebo/haloperidol arm (P=0.01, Fisher’s exact test).

Level of Perceived Comfort

Data on perceived comfort was available from 38 of 52 (73%; lorazepam/haloperidol n=19; placebo/haloperidol n=19) caregivers and 42 (81%; lorazepam/haloperidol n=22; placebo/haloperidol n=20) nurses. Missing data were due to death, patients being discharged before we were able to administer questions, or research staff unavailable (e.g. weekend). As reported in the primary study,⁶ 16 (84%) of patients in the lorazepam/haloperidol group and 7 (37%) in the placebo/haloperidol group were perceived by caregivers to be more comfortable after the study intervention; moreover, 17 (77%) of patients in the lorazepam/haloperidol group and 6 (30%) in the placebo/haloperidol group were perceived by nurses to be more comfortable.

In 12 patients, both the caregivers and nurses agreed that patient was more comfortable. In 14 patients, both the caregivers and nurses perceived that the patient was not more comfortable after study medication administration. The caregivers felt that the patient was more comfortable in 7 patients while the nurse disagreed, and the nurse reported that the patient was more comfortable in 3 patients while the caregiver disagreed. The inter-rater agreement was moderate (kappa=0.45, 95% CI=0.17, 0.73; P=0.008).

Minimal Clinically Important Difference

Using the sensitivity-specificity approach, the optimal cutoff for RASS improvement was a 4-point reduction for both caregivers (sensitivity 61%, specificity 80%) and nurses (sensitivity 73%, specificity 84%) (Table 3). Based on this cutoff, the sensitivity was 61%, the specificity was 80%, positive predictive value was 82% and negative predictive value was 57% for caregiver assessment; the sensitivity was 73%, the specificity was 84%, positive predictive value was 84% and negative predictive value was 73% for nursing assessment. The area under the receiver-operating characteristics curve was 0.71 (95% CI 0.54–0.87; P=0.03) for caregivers and 0.78 (95% CI 0.64–0.92; P=0.002) for nurses, suggesting moderate to high discrimination (Figure 1).

Table 3.

Optimal RASS Cutoff based on Sensitivity-Specificity Approach

Decrease in RASS	Sensitivity	Specificity	Youden’s Ja	Top Leftb	AUC (95% CI)c
Caregivers
≥5 points	0.35	0.87	0.21	0.67	0.71 (0.54, 0.87)
≥4 points	0.61	0.80	0.41	0.44
≥3 points	0.74	0.47	0.21	0.59
≥2 points	0.83	0.33	0.16	0.69
≥1 points	0.91	0.13	0.05	0.87
Nurses
≥5 points	0.36	0.90	0.12	0.78	0.78 (0.64, 0.92)
≥4 points	0.73	0.84	0.57	0.32
≥3 points	0.86	0.53	0.39	0.49
≥2 points	0.91	0.32	0.23	0.69
≥1 points	0.96	0.16	0.12	0.84

Abbreviations: AUC, area under the receiver-operating characteristic curve; CI, confidence interval; RASS, Richmond Agitation-Sedation Scale

^aThe cutoff value is selected based on the largest Youden’s J value, which represents the point on the ROC curve that represents the largest vertical distance from the ROC curve to the diagonal line of equality. The optimal cutoff is highlighted in bold.

^bThe cutoff value is chosen based on the smallest top left value, which is on the point on the ROC curve that represents the shortest distance to the top left corner of the graph (where sensitivity = 100% and specificity = 100%). The optimal cutoff is highlighted in bold.

^cThe area under the curve for each receiver-operating characteristic curve is shown. AUC is an indicator of the ability of the scale to discriminate change. The AUCs for the ROC curves were >0.70, which suggest moderate to high discrimination. The P-values for the AUCs were 0.03 for caregivers and 0.002 for nurses.

Figure 1. Receiver-operating characteristic curves for RASS using different anchors.

(A) The area under the curve for caregivers perceived patient comfort was 0.71 (95% confidence interval 0.54, 0.87; P-value 0.03; n=38); (B) The area under the curve for nurse perceived patient comfort was 0.78 (95% confidence interval 0.64, 0.92; P-value 0.002; n=41).

The RASS cutoff based on within-patient change method was also highly consistent with the above analysis, being −4.2 (SD 0.6) for caregivers and −4.0 (SD 1.8) for nurses.

Discussion

In this study, we found that the MCID for RASS improvement was a 4-point reduction using both the sensitivity-specificity approach and the within-patient change approach. Remarkably, this cutoff was the same for both caregivers and nurses. Our preliminary findings have potential implications for clinical trial designs and interpretation.

MCID is often determined using an anchor-based approach, and the choice of anchor is critical to define clinical significance. There is no universally accepted anchor to define clinical significance in the agitated delirium setting. We elected to use perceived comfort as the anchor in this study because one of the most important goals of palliative care is to maximize comfort through alleviation of suffering. Patient comfort is also a key determinant of personalized symptom goals.¹⁷ While patient comfort may be affected by many factors other than agitation, such as pain and dyspnea control, the significant association between patient perceived comfort and RASS change in this study supports that the level of agitation/restlessness remains a main driver of comfort. The moderate to high level of discrimination in RASS score further confirms this assessment’s responsiveness to change.

To our knowledge, this is the first study to examine the MCID for RASS in the setting of agitated delirium. We were surprised to find that the MCID for RASS was 4 points over the 10-point Likert Scale, suggesting that a large magnitude of reduction in RASS was necessary for it to be considered as clinically meaningful. We believe our findings were robust because all 4 methods (2 assessors and 2 anchor-based approaches) arrived at the same cutoff. One reason for this large MCID value may be related to the fact that we used perceived comfort as an anchor. A significant shift in RASS may be necessary for patients to be considered as comfortable by surrogate observers, who likely had been troubled by the aggressive behaviour and/or sleepiness nights. In this specific setting of persistent agitation, a consistent level of moderate sedation may be preferred over light sedation.

This large MCID value has implications for future delirium clinical trial design. First, the sample size required may be smaller than in other symptoms. Given that delirium trials are complex by nature and difficult to conduct, this new insight supports that even small studies (N<100) may be adequately powered for hypothesis-testing. Second, the intervention needs to have a relatively big impact on agitation. For pharmacologic therapies, this means that the dose would need to be high enough or that medication combinations may be required to achieve the desired effect.^{18, 19} Third, it is important to point out that a smaller magnitude does not necessarily mean that the intervention is not clinically significant. Smaller changes in RASS may be associated with other clinical outcomes (e.g. fewer use of rescue neuroleptics, absence of any RASS ≥+1 during the 8 hour period) and need to be further investigated.

In addition to study design, findings from this study may help us to better interpret clinical trial outcomes. For a patient with agitation at RASS of +1 or +2 at baseline, a reduction to RASS −3 (moderately sedated) and −2 (mildly sedated), respectively, would be necessary for the patient to be considered as comfortable. Our finding suggests that RASS −1 or 0 may not be the optimal level of sedation, perhaps because patients on this trial had persistent delirium with agitation which could be extremely distressing for caregivers and nurses alike. Applying this 4-point cutoff, 65% patients on the lorazepam/haloperidol arm and 27% patients on the haloperidol arm were able to achieve a clinical response by 8 hours.

A better understanding of the MCID may also have clinical implications for defining a clinical response. Our current study suggests that caregivers desire a significant shift in consciousness for patients with persistent agitation. The use of pharmacologic therapy for delirium in the palliative care setting is an area of intense debate. Although current clinical practice guidelines recommend neuroleptics such as haloperidol and olanzapine as first line pharmacologic treatments for agitation in delirium,^{20, 21} some have cautioned against their use because a recent 3-arm randomized trial found that placebo was superior to risperidone and haloperidol.⁵ However, several methodologic issues complicate its interpretation, including the composite primary outcome which had not been validated and enrolment of patients with relatively low delirium symptom severity.²² In contrast, our randomized clinical trial in the terminal delirium setting found that haloperidol alone resulted in a significant reduction of RASS (a validated primary outcome) and the addition of lorazepam was significantly more effective.⁶ More research is needed to identify the right medication(s) for the right indication in the right population. Patients with hyperactive delirium often receive higher doses of neuroleptics compared to those with hypoactive delirium; however, the optimal dosing has not been defined, resulting in wide variations in how neuroleptics are prescribed in practice.¹⁹

One unique aspect of this study was that we assessed patient comfort from the perspective of caregivers and nurses. Given that patients were delirious and unable to communicate, we needed to rely on these observers, who often spend a large amount of time by the bedside, to provide a surrogate response. Interestingly, there was only moderate concordance between caregiver’s assessment and bedside nurses’ assessment. This is consistent with previous studies on patient-reported outcomes suggesting that caregivers’ and nurses’ impression correlated with patient self-report, although the two groups had different perspectives.²³ It was reassuring that the same cutoff value for RASS reduction was considered clinically meaningful by both groups.

This study has several limitations. First, the MCID was derived from a single randomized trial conducted in an acute palliative care unit. Patients in this study were being treated for terminal restlessness/agitation. The findings may not be generalizable to other settings. Second, it is important to point out that a majority of study participants had a RASS score of +1 or +2 at the time of study medication administration. Further research is needed to determine if the 4-point MCID cutoff is applicable to patients with higher RASS scores. Third, the sample size was small and not specifically powered for this post-hoc analysis. This limitation was further compounded by the fact that the anchor was missing in some patients. Thus, the MCID cutoff should be considered as preliminary. Fourth, the primary outcome was change in RASS reduction from baseline to 8 hours while the anchor was collected between 8 and 24 hours after study intervention. The level of sedation/agitation and other symptoms that occurred during this time interval may have contributed to perceived comfort. Fifth, we only assessed perceived comfort using a single question. Qualitative studies may help to better understand how caregivers and nurses define comfort in the context of refractory terminal delirium and to identify a personalized sedation goal.

In summary, using 4 different anchor-based methods, we found that a reduction of 4 points in RASS was considered to be the MCID in the setting of persistent agitation and delirium. Our findings may support future study designs and help with interpretation of findings from delirium trials.