Gold Standard Assessment of Delirium Severity in Patients with Dementia: Evaluation of an Expert Panel Adjudication Process

Abstract

Background:

Patients with Alzheimer’s Disease and Related Dementias (ADRD) have an increased risk for delirium and subsequent complications. Rating delirium severity in the presence of co-occurring dementia is challenging due to overlapping features of delirium and ADRD. The multi-site prospective Better ASsessment of ILlness (BASIL)-II study will develop and validate a new delirium severity instrument for use in patients with and without ADRD.

Objective:

Describe an expert panel process used to rate delirium severity.

Methods:

Clinical assessors conducted standardized cognitive tests. A separate panel of experts independently reviewed assessors’ reports, rated delirium severity using a 0–10 scale, and assigned dementia diagnoses using DSM-5 criteria. Panel agreement was defined using a priori criteria. Cases without agreement after initial review were discussed as a group and re-rated using a modified Delphi approach until achieving consensus.

Results:

Patients (N=488) were on average 79 years old, 58% female, and 75% White. After initial review, 80% of cases were in agreement for delirium severity. Kappa was 0.86 (95% CI, 0.78, 0.82) before expert panel discussion and 0.90 (95%CI, 0.89, 0.92) after consensus. Final delirium severity ratings were no delirium (48%); subsyndromal (22%), mild-moderate (25%), or severe (6.4%). Disagreement in delirium severity was associated with ADRD (OR 3.02), nursing home setting (2.63), and vision impairment (2.42).

Conclusions:

This rigorous process provides confidence that delirium severity can be rated accurately in patients with and without ADRD. We will use this expert panel adjudication to provide the reference standard for validation of a future delirium severity instrument.

Introduction

Delirium is an acute change in mental status that occurs frequently across many settings and is associated with poor outcomes.^1–4 Delirium severity, defined as the degree or intensity of delirium features and symptoms, has emerged as a sensitive and responsive measure of delirium for guiding clinical care and research. Most existing delirium severity instruments, such as the Confusion Assessment Method – Severity Scale (CAM-S),⁵ rate and sum the presence and severity of delirium signs and symptoms⁶ to quantify severity on a continuous scale. This allows for identifying early and/or subsyndromal cases of delirium, following patient response to treatment interventions, and calculating prognosis for longer-term recovery. Continuous scales can also be useful for risk stratification for research study enrollment, as an outcome measure in clinical trials or biomarker studies, and in estimating staffing burden and costs related to delirium.

Delirium is especially common among patients with Alzheimer’s Disease and Related Dementias (ADRD). Persons living with ADRD have a 4–5 fold increased risk of delirium and may experience a greater severity of delirium compared to those without ADRD.² Studies have found that when delirium is superimposed on ADRD, there is a more rapid rate of cognitive decline, increased rates of hospitalization and institutionalization, and increased mortality when compared to delirium in persons without ADRD.^3–5,7 A number of challenges exist in measuring delirium severity in persons with ADRD, for example, none of the existing delirium severity instruments have been developed or validated specifically for use in this population.⁸ Moreover, the substantial overlap in symptoms and features between delirium and dementia pose meaningful challenges to rating delirium severity in the presence of underlying dementia. An additional diagnostic challenge is the lack of an accepted laboratory test or biomarker for diagnosing delirium or monitoring its severity.⁹ For this reason, clinical research studies on delirium often rely on reference standard delirium diagnoses, adjudicated by expert panel consensus.^10,11 Likewise, rating delirium severity requires expert clinical evaluation and interpretation of both case reports and cognitive testing data. While these approaches are essential for establishing accurate ratings, few studies have formally described or evaluated their reference standard procedures,¹² or examined the frequency and potential reasons for lack of concordance among expert panel members.

In this report, the overarching goal was to develop, implement, and evaluate the agreement of a rigorous expert panel process to adjudicate delirium severity in support of the Better ASsessment of ILlness (BASIL)-II study, a multisite prospective cohort study designed to develop and validate new delirium severity instruments. We describe the consensus process used by our clinical expert panel to generate reference “gold standard” ratings of delirium severity in patients with and without dementia, highlight challenges in this process, and provide insights for improving the process for future studies. To this end, we examine the delirium severity ratings across interdisciplinary experts (geriatrics, neurology, psychiatry, neuropsychology) and clinical practice settings (medical inpatients, surgical inpatients, nursing home) and highlight common challenges in making these ratings. We assess demographic and clinical patient factors associated with expert agreement and disagreement and perform stratified analyses on subgroups of patients with and without mild cognitive impairment (MCI) and dementia to examine the specific impact of co-occurring dementia on expert panel agreement.

Methods

BASIL-II Study Design

The Better ASsessment of ILlness (BASIL)-II study is a three-site prospective cohort study designed to develop and validate new delirium severity instruments, the Delirium Severity (DEL-S) rating for all older adults and the Delirium Severity Rating in Alzheimer’s and other Dementias (DEL-S-AD) specifically for persons with mild to moderate ADRD.¹³ Patients aged 70 years and older and English-speaking, with and without ADRD, were enrolled across 3 sites: patients with acute medical conditions admitted to the hospital with an expected length of stay greater than 48 hours (Site 1, Beth Israel Deaconess Medical Center (BIDMC), Boston, MA), patients scheduled to undergo elective surgery with an expected length of stay greater than 48 hours (Site 2, University of Florida (UF) Hospital, Gainesville, FL), and long-term and post-acute care residents with expected length of stay of at least 4 or more weeks who were followed for development of an acute illness episode (Site 3, The New Jewish Home (TNJH), New York, NY). Potentially eligible participants were identified through review of electronic or paper-based medical records and they or their legal surrogate decision-makers were approached by research staff to obtain verbal informed consent for participation. All study procedures, including verbal informed consent, were approved by the BIDMC Institutional Review Board with ceded review from UF and TNJH.

Study Procedures

All cognitive and health data used by the expert panel to adjudicate delirium severity were collected by clinical assessors (CAs) at each site. CAs were clinicians and advanced clinical fellows, including neuropsychologists (n=5), research coordinators/program managers (n=4), geriatricians (n=2), a clinical research nurse (n=1), and a neurologist (n=1), who had experience in the diagnosis of delirium and dementia. The CA performed bedside interviews of every participant and informant using standardized instruments and reviewed the medical record. At Sites 2 and 3, an initial interview (baseline) was conducted by the CA prior to the surgery or acute illness episode, and a second interview (acute) was conducted on Day 1 or 2 following the surgery or acute illness onset. At Site 1, the CA interview occurred only during the hospital admission (acute). At Sites 1 and 2, the CAs interviewed a nurse and a designated proxy informant, usually a family member or friend, and at Site 3, the CA interviewed a facility nurse or nurse’s aide as an informant.

Study Variables

Cognitive function was assessed using the Montreal Cognitive Assessment (MoCA)¹⁴ at all sites, and additionally, at Site 3, the Severe Impairment Battery (SIB-8).¹⁵ The 8-item Informant Interview to Differentiate Aging and Dementia (AD8)¹⁶ and the family or proxy-rated Dementia Severity Rating Scale (DSRS)¹⁷ were completed with the informant. The CA reviewed the medical record for evidence of a diagnosis of dementia as well as history of delirium or other cognitive diagnoses. Major neurocognitive disorder (dementia) and minor neurocognitive disorder (Mild Cognitive Impairment, MCI) were rated by the CA according to DSM-5 criteria, and dementia severity was rated using the Clinical Dementia Rating Scale (CDR),¹⁸ with severity rated as mild, moderate, or severe. These ratings were shared with the GS along will all other data collected for use in their consensus process.

Delirium was assessed at baseline and at the acute time point using the Confusion Assessment Method (CAM),⁵ a standardized 10-question delirium assessment with 94% sensitivity, 89% specificity, and inter-rater reliability of 0.70–1.00.¹⁹ The CAM was rated by CAs using items in the MoCA, SiB-8, and the Abbreviated Delirium Symptom Interview (DSI),²⁰ as well as from their direct interactions with participants.¹⁵ The Family Confusion Assessment Method (FAM-CAM)²¹ was used to identify the presence of symptoms of delirium and acute change in mental status in the period leading up to the acute episode. Nurse interview also identified evidence for acute change in mental status, including new medications for agitation or sleep (e.g., antipsychotic, benzodiazepine, ramelteon), and use of a sitter that might signal new agitated behavior. After completing the CAM, CAs also rated the presence of delirium using DSM-5 criteria.²² The CAM-Severity (CAM-S)²³ score (scored 0–19, 19=most severe delirium) was calculated as an initial estimate of delirium severity.

Upon completion of interviews, CAs rated participants for delirium severity using a quantitative rating scale from 0–10, where 0=no delirium, 1–3=subsyndromal delirium, 4–6=mild-moderate delirium, and 7–10=severe delirium. Descriptors for each degree of delirium severity were used to select a score, and were provided in the study manual, used for training and standardization of the clinical assessors. For example, the descriptor for mild delirium severity provides detailed examples to assist with scoring as follows: “Patient awake throughout the interview but demonstrated a few attention lapses requiring repeating the question. Makes several mistakes in cognitive testing, especially on the attention and memory items. If baseline available, some new errors and/or positive signs. Caregiver notes some subtle changes from baseline. Acute changes may or may not be documented in medical record.” (Supplemental Table 1)

Additionally, clinical data were collected from the medical record including medical diagnoses, Charlson Comorbidity Index score,²⁴ and intercurrent medical illnesses. CAs administered the Patient Health Questionnaire (PHQ-9)²⁵ to assess symptoms of depression. Major depressive disorder was present when five or more out of nine symptoms (including at least one of depressed mood and loss of interest or pleasure) was present nearly every day in the same 2-week period.²⁵ Functional variables obtained from patients and informant included Activities of Daily Living (ADL),²⁶ Instrumental Activities of Daily Living (IADL),²⁷ and Medical Outcomes Study Short Form 12 (MOS SF-12),²⁸ Self-rated Health and Quality of Life scales. Patient demographics including age, sex, education, English as first language, vision and hearing impairment, and prior history of depression, dementia, or depression were obtained from patient and informant interview, and medical record review.

Expert Panel Adjudication Process

A panel of individuals with expertise in delirium and dementia was formed that included experts in geriatrics (n=2), cognitive neurology (n=1), geriatric psychiatry (n=1) and neuropsychology (n=1). On average, panel members had more than 20 years of experience in clinical practice caring for patients with delirium and dementia, and participating in research on delirium, dementia, and delirium superimposed on dementia. Outside of this research project, experts had face-to-face contact with patients at least monthly. (Supplemental Table 2)

A standardized adjudication process was used to assign a reference “gold” standard delirium severity rating (including delirium presence or absence), along with diagnosis and rating of MCI, dementia, dementia severity, and depression. To develop and implement this process, the expert panel met via Zoom approximately once per month from September 2021 until September 2024. Initial meetings consisted of training by a panel member with prior experience in expert adjudication of delirium (Fong). Subsequent meetings consisted of iterative adaptation of adjudication processes. In preparation for each adjudication meeting, the following steps occurred: 1) a study coordinator or research associate identified 10–20 cases ready for review by verifying all assessment and case report data (Figure 1) for completion and accuracy; 2) single-page summary reports for each case were distributed to expert panelists; data contained in these reports (see Supplemental Table 3) were collected by a single CA; 3) experts independently reviewed the reports and rated delirium severity using the 0–10 quantitative rating scale (analogous to the scale used by the CAs, described above), as well as assigned delirium and other diagnoses according to DSM-5 criteria; most panel members spent approximately 10 minutes per patient record; 4) experts submitted their independent ratings via an online form in advance of the group meeting.

Figure 1.

Flow Diagram of Expert Panel Process. AD8 = 8-item Informant Interview to Differentiate Aging and Dementia; ADLs = Activities of Daily Living; CAM = Confusion Assessment Method; CAM-S = Confusion Assessment Method – Severity Scale; DSI = Abbreviated Delirium Symptom Interview; DSRS = Dementia Severity Rating Scale; FAM-CAM = Family Confusion Assessment Method; IADLs = Instrumental Activities of Daily Living; MoCA = Montreal Cognitive Assessment; PHQ-9 = 9-item Patient Health Questionnaire; SIB-8 = 8-item Severe Impairment Battery.

Ά priori consensus was defined as 4 of 5 experts agreeing on the presence or absence of delirium, with no more than a 2-point total disagreement range among members’ 0–10 delirium severity rating scores. Ratings of MCI, dementia, and dementia severity required at least 4 of 5 experts to agree on the diagnosis or level. During each one-hour consensus meeting, only cases which did not meet initial agreement were presented to the full panel. This typically included approximately 6 cases. In contrast to the initial ratings, the consensus meeting was not blinded. For each case, discussion began with review of the voting, i.e. 3 rated the case as delirium and 2 rated the case as no delirium. Panelists were then encouraged to discuss their interpretation of the data and how it led to their diagnosis. At any point during the discussion, any panelist could opt to change their rating. Discussion continued until consensus (4 of 5) was achieved, and the panel was asked to verify agreement with the final diagnosis. Time spent on each individual case varied due to numerous factors, such as complexity (i.e. DSD), dispersion of initial ratings (i.e. severity ratings with greater differences between raters), and for the reasons outlined in Tables 4 and 5.

Table 4.

Risk factors for disagreement in initial delirium severity rating

Characteristic	OR for disagreement	CI lower	CI upper
Age	1.02	0.99	1.05
Female	1.48	0.94	2.33
Non-white	1.53	0.94	2.50
Black or African American	1.50	0.85	2.65
Race Uncertain	1.44	0.61	3.40
Low Education (<=8)	0.94	0.29	3.09
High Education (>=18)	0.80	0.47	1.37
English as second language	0.47	0.16	1.36
Lives in NH	2.63	1.66	4.15*
Hearing impairment (ever)	1.50	0.95	2.36
Vision impairment (ever)	2.42	1.49	3.95*
Delirium severity continuous	1.13	1.05	1.22*
Delirium severity 4–6	0.81	0.47	1.40
Delirium severity 7–10	1.14	0.42	3.12
Dementia severity (GS) continuous	1.90	1.52	2.36*
Dementia	3.02	1.95	4.66*

^*p <.05. Separate bivariate models were fit for each covariate category. In the model for race, white is the reference group, and Asian, other, and Hispanic are not included because of small sample sizes. Non-white is any race/ethnicity other than White, non-Hispanic. In the model for education, medium (9–17 years) is the reference group. Lives in Nursing Home (NH) is independent of dementia status. In the model for delirium severity, low or no severity (0–3) is the reference group.

Table 5.

Challenges in rating delirium severity

Challenge	Detail(s)
Degree of attribution of symptom severity to delirium vs. dementia	Given the overlap between delirium and dementia symptoms, how do symptoms individually and jointly contribute to delirium severity? Patients with dementia may have delirium features at baseline, so the presence of these features may not contribute as much to delirium severity Interpreting cognitive test performance in a patient with potential delirium and underlying cognitive impairment The magnitude and relative change in test score may have different implications, for example if the score falls from 27 to 25 vs. 20 to 18 The domain(s) affected may have different implications
Multidimensionality of delirium severity	Delirium severity can be influenced by the intensity and duration of cognitive and functional symptoms
Delirium does not meet criteria (subsyndromal)	Subsyndromal delirium occurred frequently among patients with dementia; often there was no acute change specifically documented but other delirium criteria were present and new
Patient features	Patients with more severe dementia were more difficult to rate vs. those with only mild impairments
Lack of baseline data	Knowledge of acute change (vs. chronic) is essential for establishing delirium and rating its associated severity
Timing of data collection	Fluctuations in symptoms may be hard to capture Cognitive/functional performance might appear “normal” at one time point but later that same day be “impaired”
Limitations of available data	Inconsistent reliability of caregiver report Reporter may have lack of familiarity of patient’s baseline

Figure 1. Flow Diagram of Expert Panel Process. AD8 = 8-item Informant Interview to Differentiate Aging and Dementia; ADLs = Activities of Daily Living; CAM = Confusion Assessment Method; CAM-S = Confusion Assessment Method – Severity Scale; DSI = Abbreviated Delirium Symptom Interview; DSRS = Dementia Severity Rating Scale; FAM-CAM = Family Confusion Assessment Method; IADLs = Instrumental Activities of Daily Living; MoCA = Montreal Cognitive Assessment; PHQ-9 = 9-item Patient Health Questionnaire; SIB-8 = 8-item Severe Impairment Battery.

Analysis approach

Inter-rater agreement was quantified using Cohen’s kappa. All unique pairs of raters (n=10 pairs from 5 raters) were considered in agreement calculations. Each patient’s data was duplicated for each unique rater pair, resulting in a dataset with repeated observations per patient to facilitate inter-rater agreement analyses.²⁹ Initial disagreement was defined as the absolute difference in rater delirium severity scores. Ordinal logistic regression analyses were completed to examine risk factors associated with disagreement in initial delirium severity ratings, including patient demographics, clinical characteristics (e.g., dementia, hearing and vision impairment), and functional status. In this analysis, a higher odds ratio indicates that the risk factor was associated with higher odds of an additional level of disagreement among pairs of raters.

To capture factors that contributed to disagreement in ratings, panelists were asked to explain their ratings in the online rating form. Panelists were also asked as a group to identify challenges that contributed to disagreement.

Results

Study Cohort

The BASIL-II study cohort (N=488) was on average 79 (SD 6) years old, 58% female, 75% White, 17% Black or African-American, and 2% Hispanic. Average education was 14.5 (SD 3.2) years. (Table 1). Missing data was minimal overall (less than 7%; mainly due to patient refusal).

Table 1.

Patient Characteristics

Characteristic	BASIL-II Sample (N = 488)*
Female, n/N (%)	283 / 488 (58%)
Age, years, mean (SD)	79 (6)
Racial group, n/N (%)
White	364 / 488 (75%)
Black or African American	84 / 488 (17%)
Asian	5 / 488 (1.0%)
All other racial groups	4 / 488 (0.8%)
Uncertain	31 / 488 (6.4%)
Hispanic, n/N (%)	8 / 488 (1.6%)
Education, mean yrs (SD)	14.5 (3.2)
English as second language, n/N (%)	26 / 486 (5.3%)
Lives alone, n/N (%)	145 / 484 (30%)
Lives in nursing home, n/N (%)	107 / 488 (22%)
Married, n/N (%)	206 / 483 (43%)
Charlson Comorbidity Index Score, n/N (%)
0	98 / 486 (20%)
1	93 / 486 (19%)
2	91 / 486 (19%)
3 or more	202 / 486 (42%)
Hearing impairment, n/N (%)	172 / 485 (35%)
Vision impairment, n/N (%)	112 / 454 (25%)

^*Denominators less than 488 are due to missing data

Consensus Process

Following independent rater review, the expert panel discussed and adjudicated 31 (6%) of the cases for delirium presence or absence and 97 (20%) of the cases for delirium severity (0–10 score). Consensus discussion was also required for 27 (6%) of dementia and 78 (16%) of MCI ratings. The final delirium severity rating was no delirium in 232 (48%), subsyndromal delirium in 105 (22%), mild-moderate delirium in 120 (25%), and severe delirium in 31 (6%). The final cognitive diagnosis was MCI in 158 (32%), dementia in 152 (31% (21% of mild severity; 10% of moderate severity)), and no cognitive impairment in 176 (36%) (Table 2).

Table 2.

Cognitive Diagnoses and Proportion Requiring Consensus Discussion

Cognitive diagnosis	BASIL-II Sample (N = 488)	Proportion requiring discussion*
Delirium, No. (%)	146 (30%)	6%
Delirium severity, No. (%)		20%
0	232 (48%)
1 – 2	105 (22%)
3 – 5	120 (25%)
6 – 10	31 (6.4%)
Mild Cognitive Impairment, No. (%)	158 (32%)	16%
Dementia / Major Neurocognitive Disorder, No. (%)	152 (31%)	6%
Dementia severity rating, No. (%)		19%
1 No dementia	176 (36%)
2 Very mild	160 (33%)
3 Mild	101 (21%)
4 Severe	51 (10%)

^*Number of cases that required consensus discussion divided by the number of cases counted towards the prevalence. Discussion was required if less than 4 of 5 expert panelists agreed in their initial rating for delirium, MCI, Dementia, or Dementia severity, or the range of initial delirium severity ratings was >2.

The pre-adjudication kappa for delirium severity was 0.86 (95%CI 0.85, 0.87). Delirium severity agreement was higher in those without cognitive impairment (kappa 0.89; 0.87, 0.90) than in those with co-occurring MCI (0.86; 0.85, 0.88) or dementia (0.82; 0.80, 0.84) (Table 3). Pre-adjudication agreement for diagnosis of MCI was 0.58 (0.56, 0.61), compared with 0.81 (0.79, 0.83) for dementia.

Table 3.

Agreement on diagnosis (pre-adjudication)

Diagnosis	Subgroup or co-occurring condition	Agreement (K)*	95% CI
Delirium	All (N = 488)	0.80	(0.78, 0.82)
	No MCI or dementia (N = 178)	0.81	(0.78, 0.84)
	MCI (N = 158)	0.80	(0.77, 0.83)
	Dementia (N = 152)	0.77	(0.73, 0.80)
Delirium Severity	All (N = 488)	0.86	(0.85, 0.87)
	No MCI or dementia (N = 178)	0.89	(0.87, 0.90)
	MCI (N = 158)	0.86	(0.85, 0.88)
	Dementia (N = 152)	0.82	(0.80, 0.84)
MCI	All (N= 488)	0.58	(0.56, 0.61)
	No delirium (N = 342)	0.60	(0.57, 0.63)
	Delirium (N = 146)	0.54	(0.50, 0.59)
Dementia	All (N= 488)	0.81	(0.79, 0.83)
	No delirium (N = 342)	0.83	(0.80, 0.85)
	Delirium (N = 146)	0.78	(0.74, 0.81)
Dementia Severity	All (N= 488)	0.86	(0.85, 0.87)
	No delirium (N = 342)	0.88	(0.87, 0.89)
	Delirium (N = 146)	0.82	(0.80, 0.84)

^*Kappa (K) was calculated after creating all possible unique pairs of five raters (n=10 pairs), with each participant’s record duplicated x10 for each unique rater pair.

Pre-adjudication agreement was lower for both MCI and dementia when co-occurring delirium was present, kappa 0.54 (0.50, 0.59) and 0.78 (0.74, 0.81), respectively, than when delirium was absent. In addition, pre-adjudication agreement for dementia severity was 0.88 (0.87, 0.89) when delirium was absent, and 0.82 (0.80, 0.84) when delirium co-occurred. By study design, all agreement increased with adjudication (Supplemental Table 4).

Associations between patient characteristics and expert agreement

We examined associations between patient characteristics and pre-adjudication disagreement in delirium severity (Table 4). The presence of dementia had the strongest association, with an OR of 3.02 (CI 1.95, 4.66) for a higher level of disagreement. Similarly, greater dementia severity was associated with a higher level of disagreement (OR 1.90 (1.52, 2.36)). Living in a nursing home, with OR 2.63 (CI 1.66, 4.15), and vision impairment OR 2.42 (CI 1.49, 3.95) were also associated with a higher level of disagreement. Education and English as a second language were not associated with delirium severity disagreement, even though these factors were cited by the panelists as influencing their interpretation of the data. Age, sex, and race were also not associated with delirium severity disagreement.

Challenges to expert ratings

Challenges identified by the expert panel in determining the presence or absence of delirium included: 1) limitations due to data availability, such as lack of baseline data at Site 1 which was necessitated by study design, 2) collection of data at different time points in different cases and settings, and 3) informant data that was not always available or reliable.

There were also challenges in cases in which CAM and DSM-5 classification disagreed, as in some cases of subsyndromal delirium by CAM that met the DSM-5 criteria for delirium. Challenges in assigning delirium severity ratings raised by the expert panelists included: 1) deciding to what degree co-occurring delirium and dementia each contributed to abnormal cognitive features and severity (i.e., attribution decision), 2) the impact of underlying cognitive impairment on delirium severity, and 3) the multidimensional nature of the concept of delirium severity (Table 5).

Discussion

This report describes an expert adjudication process for rating delirium severity in BASIL-II, a study designed to develop and test new delirium severity instruments in 3 care settings in which delirium is common: inpatient general medical care, peri-operative care, and nursing home acute illness care; and in patients both with and without ADRD. Results show high pre-discussion expert panel agreement on rating delirium severity (kappa 0.8), and excellent post-discussion agreement (kappa 0.9). The improvement in agreement after discussion supports the added value of consensus discussion, as long as discussion was limited to those cases that did not satisfy initial consensus criteria (20% of the total), to make the process feasible and efficient. The final kappa of 0.9 indicates that this process resulted in “gold” standard level reliability. The training time for this study’s multi-disciplinary panel members was especially important since the study encompassed patients in different settings, across which cognitive and function expectations differed widely. Importantly, this adjudication process will be used for the reference standard ratings to validate the new delirium severity instrument for those with and without ADRD to be developed in BASIL-II.

Expert adjudication panels play an important role in clinical research by providing a standardized outcome review process, enhancing reproducibility and mitigating bias. Systematic reviews and meta-analyses suggest that central adjudication may not significantly alter overall treatment effects in trials where outcomes are objective and site investigators are blinded.³⁰ However, expert adjudication panels are important for complex endpoints or those that require clinical expertise,³¹ including multifactorial geriatrics syndromes such as such as delirium. Clinical trial stakeholders indicate that expert panel reliability is maximized by: 1) rigorous committee selection, 2) establishment of clear adjudication criteria and definitions, 3) robust training for adjudicators, and 4) discussions that allow for careful consideration of event complexity, as implemented in this study.³² In studies of delirium severity, this would include training support to help panelists weigh attribution of cognitive features to either delirium or dementia.

This study builds on BASIL-I, in which a new instrument for rating delirium severity (the DEL-S) was developed and validated using state-of-the art psychometric approaches in combination with cognitive test items, patient self-report, and observer-rated items.^8,33 Results of this (BASIL-II) study will similarly enable testing the performance characteristics of the DEL-S-AD, a new instrument for rating delirium severity specifically in patients with ADRD, against the adjudicated expert panel reference “gold” standard ratings. The lower pre-adjudication agreement of delirium severity diagnoses for patients with co-occurring dementia (adjusted OR for disagreement = 3.02) validates the importance of expert panel discussion, and the excellent post-adjudication agreement indicates the adjudication process was effective.

This success could be explained by case report forms that facilitated reconciliation of cognitive testing results across instruments and time points, the high level of expertise of the panelists, and focused discussion on weighing attribution of cognitive impairments to either dementia or delirium. Of note, panelists had relatively lower agreement assigning individuals to an MCI diagnosis (kappa 0.72 after consensus discussion), likely because of the relative limits of the measures of cognitive function and self-care ability. Some patients had a physical cause for their functional impairment, which is a key determinant in distinguishing MCI from dementia. This was compounded by variability in functional assessment between Site 3 (a nursing facility) and Sites 1 and 2 (community) in that the caregiver at Site 3 was often a nursing assistant whose ratings were based on their interactions with the patient and may not fully reflect the patient’s level of functioning outside of the nursing home environment. However, panelists’ delirium severity rating agreement remained high in individuals with co-occurring MCI (kappa 0.91 after discussion), suggesting the impact of MCI on the primary goal of the expert panel adjudication process was minor.

A strength of this study was our systematic evaluation of our expert panel process along with exploration of challenges. The adjudication process was efficient in that after training, only 20% of cases required consensus discussion, even in patients with a wide variety of profiles across 3 different care settings, and initial concordance was high. We selected an interdisciplinary expert panel representing multiple disciplines and settings of care (acute medical, surgical, and long-term care settings) relevant to rating delirium severity in individuals with and without dementia, including internal medicine, geriatrics, psychiatry, neurology, and neuropsychology. This expert panel adjudication process to generate reference “gold” standard ratings provides confidence that delirium severity can be used as a key measurable outcome in delirium intervention studies in patients with and without dementia.

Our panelists identified limitations in the process First, differences in data collected across the sites presented some challenges in consistent ratings of delirium severity. For example, at Site 1 (as opposed to Sites 2 and 3), there was no preadmission baseline rating of cognition, since all participants were assessed at the time of hospital admission. Another example is at Site 2, where the preoperative assessment was often done by telephone, and thus, baseline data included a T-MoCA instead of the full MoCA. Despite this, agreement remained high. Second, the process was relatively time intensive, which may not be generalizable to all research studies or feasible for clinical application, though with practice the panelists were increasingly efficient. Third, the multidisciplinary panel composition and experience may not be exactly replicable. Our expert panel consisted of physicians and neuropsychologists, but expansion to include a nurse and/or research staff would create a wider breadth of expertise. However, not all represented disciplines (geriatrics, neurology, psychiatry, and neuropsychology) are necessarily needed to provide experienced delirium severity adjudication. Fourth, panelists did not have the benefit of face-to-face interaction with patients. However, three GS raters were also Site PIs and responsible for overseeing the CA ratings at their site. Thus, they sometimes relayed information that arose from their discussions with the CA. In the future, including video recordings of the CA assessments may aid in the consensus process. Lastly, data had to be interpreted within the context of known confounders to dementia diagnosis, such as education, vision/hearing impairment, or functional impairments.

To date, no existing delirium severity instruments have been designed specifically for use in patients with dementia. Since outcomes of delirium are worse in patients with ADRD, there is an imperative to fill this gap in knowledge. Measurement of delirium severity is important because increased severity is associated with worsened adverse outcomes.³⁴ As a continuous measure, it complements binary (presence or absence) delirium instruments, facilitating detection of change and responsiveness to intervention. Our report points to areas where challenges remain in ascertaining delirium severity in patients with co-occurring dementia. However, our successful development and implementation of an expert panel adjudication process with excellent agreement holds promise as a means of using delirium severity in delirium prevention or treatment protocols.

Data availability statement

The data supporting the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy restrictions.