Delirium Diagnosis Methodology used in Research: A Survey-Based Study

Abstract

Objectives

To describe methodology used to diagnose delirium in research studies evaluating delirium detection tools.

Design

A survey addressing reference rater methodology for delirium diagnosis, including rater characteristics, sources of patient information and diagnostic process.

Setting

Survey completed via web or telephone interview according to respondent preference.

Participants

Authors of 39 studies included in 3 recent systematic reviews of delirium detection instruments in hospitalized patients.

Results

Authors from 85% (n=33) of the 39 eligible studies responded to the survey. The median (Interquartile Range [IQR]) number of raters per study was 2.5 (2–3); 79% were physicians. The raters’ median (IQR) duration of clinical experience with delirium diagnosis was 7 (4–10) years, with 5% having no prior clinical experience. Inter-rater reliability was evaluated in 70% of studies. Cognitive tests and delirium detection tools were used in the delirium reference rating process in 61% (n=21) and 45% (n=15) of studies, respectively, with 33% (n=11) using both and 27% (n=9) using neither. When patients were too drowsy or declined to participate in delirium evaluation, 70% (n=23) of studies used all available information for delirium diagnosis, while 15% excluded such patients.

Conclusions

Significant variability exists in reference standard methods for delirium diagnosis in published research. Increasing standardization by documenting inter-rater reliability, using standardized cognitive and delirium detection tools, incorporating diagnostic expert consensus panels and using all available information in patients declining or unable to participate with formal testing may help advance delirium research by increasing consistency of case detection and improving generalizability of research results.

Introduction

Delirium is a clinical syndrome characterized by an acute and fluctuating cognitive impairment, occurring over hours to days, primarily associated with inattention and other cognitive and behavioral changes.¹ Delirium occurs very frequently throughout the healthcare system and is associated with substantial suffering and loss of dignity,^2,3 longer hospital stays, ^4–6 institutionalization at hospital discharge,^7,8 increased healthcare expenditures, ⁹ increased 1-year mortality ^10–13 and long-term cognitive impairment.^14–17

Common to most diagnoses in the psychiatric literature, no pathognomonic test (e.g., laboratory, imaging, or biomarker) can identify delirium; therefore the diagnosis is predicated upon the careful clinical examination. In addition to a thorough patient history and physical examination, a delirium diagnosis requires: 1) history from collateral sources documenting an acute and fluctuating change in cognitive function and behavior from baseline, 2) examination of the patient’s mental state documenting decreased attention and other associated cognitive and behavioral impairments (e.g., disorientation, impaired short- and long-term memory, perceptual disturbances, hallucinations, delusions, motoric abnormalities and sleep disturbances), and 3) review of laboratory and other investigations (e.g. radiological testing) which may help detect underlying cause(s) of delirium.¹⁸

Reliable and valid delirium diagnoses, using a reference standard, are crucial for the advancement of clinical research in this field. To better understand its epidemiology, risk factors, phenomenology, etiology and prevention, accurate case detection is important. Given the fluctuating nature of its presentation, accurate and reproducible diagnoses are an especially challenging issue for delirium research of all types.¹⁹ One particular type of study that must, by design, regularly employ an independent reference rater evaluation to serve as the reference standard, is the development and evaluation of delirium detection tools. ^20–22 Details of these reference rater methods are scant in most research publications. Although reference standards are important in all types of delirium research, understanding the reference rater methodology that underpins the development of detection instruments is arguably the most important standard to characterize, as it underlies the assumptions made in studies, where only the delirium detection instrument is used as evidence of the diagnosis. Hence, the objective of this inquiry is to characterize the methodology used as a reference standard in studies evaluating delirium detection tools in hospitalized patients, using a survey-based study design.

Methods

Study Sample

A sampling frame of 37 unique investigators representing 39 studies, published between 1990 and 2012, were obtained from 3 systematic reviews of the literature evaluating delirium detection tools against a reference standard for delirium diagnosis.^20–22 Of these 39 studies, 16 (41%) were performed in ICU and 23 (59%) in non-ICU in-patient hospital settings (See Appendix).

Survey Design and Testing

The survey was designed by co-authors with expertise in the clinical diagnosis of delirium, the development and use of delirium detection tools and/or survey design methodology. The survey included questions about the following areas of reference rater methodology: 1) characteristics of delirium reference raters (number per study, professional background, training and experience), (2) sources of information used in determining the delirium diagnosis (including use of standardized cognitive testing and delirium detection tools), (3) the use of consensus panels in making a final delirium diagnosis, and (4) methodology for evaluating patients who did not answer questions due to decreased arousal or patient refusal. The survey was pilot-tested with two delirium researchers and revised based on their feedback. A web-based version of the survey was developed using “SurveyMonkey ” (www.surveymonkey.com) and pilot tested by two additional delirium researchers. As a token of appreciation, an electronically issued $US 50 gift card was offered to participants. This study was reviewed and approved by the Johns Hopkins University Institutional Review Board; all survey respondents provided informed consent.

Survey Distribution and Response

In September 2012, an email was sent to the corresponding author of each of the 39 studies eligible for this research, requesting completion of the web-based survey with respect to the reference rater methodology used in the specific eligible study. At least 2 additional emails inviting participation were resent to non-responders in October and November, 2012. The authors who did not respond to email invitations were contacted by phone and offered the option of completing the survey via a telephone interview (conducted by KJN). The last survey was completed in May 2013. Respondents included 31 of 37 (84%) authors, representing 33 of 39 (85%) studies; 29 authors responded to the web-based survey, and 4 completed the survey by phone. No response was received from 5 authors and 1 declined to participate.

Survey Analysis

Standard descriptive statistics were performed using Excel (version 2010). Median values and 25–75% interquartile ranges (IQR) were reported for data with non-normal distributions.

Results

Respondent and Reference Rater Characteristics

Respondents identified themselves as the principal investigator in 76% of the 33 studies, with 55% of respondents serving as a delirium reference rater in the study being evaluated. Of 82 reference raters participating in the 33 studies, 79% were physicians from the following specialties: 51% psychiatry, 29% geriatrics, 9% critical care, 9% neurology and 2% not described. The background of the physician reference raters (n=65) were described as: 51% attendings/consultants, 20% fellows/senior registrars and 14% house staff (i.e., interns, residents, and junior registrars), with 15% not reported. The remainder of the reference raters (n=17) were nurses (7%), psychologists (4%) and research assistants (1%), with 7% not reported. Reference raters’ median (IQR) duration of clinical experience with diagnosing delirium was 7 (4–10) years, with 5% having no clinical experience prior to conducting the eligible study.

Reference Rater Training Practices

The median number (IQR) of reference raters per study was 2.5 (2–3) with a range of 1 to 15; 18% (n=6) of studies used only one reference rater. Of the remaining 27 studies with more than one reference rater, inter-rater reliability was evaluated in 70% (n=19) of studies, not evaluated in 15% (n=4), and not reported in 15%. Based on the 23 studies acknowledging reference rater training approaches, a median (IQR) of 20 (11–100) patients per study were evaluated by the study’s reference raters prior to starting the study as part of standardization and training. However narrative descriptions by some of the respondents in the survey reporting the highest numbers of patients, suggest that this often included patients who had been seen in routine clinical practice by the reference rater previous to the start of the study, and not patients evaluated using the standardized procedures unique to the given study.

Reference Rater Examination Practices

The sources of information used in the reference rater examination are outlined in Figure 1. The median (IQR) time spent with the patient, and time taken for the entire examination including medical record review were reported as 18 (15–30) and 30 (20–45) minutes, respectively. Respondents were asked about the use of formal cognitive tests and delirium detection tools in the reference standard diagnosis of delirium. At least one cognitive test was used in 20 studies (61%), at least one delirium detection tool was used in 15 studies (45%); 11 studies (33%) used both standardized cognitive testing and delirium detection tools, 13(39%) used one or the other, and 9 (27%) used neither. The types of cognitive and delirium detection tools used are outlined in the Table. The most frequently used cognitive tests were the Mini Mental State Examination (MMSE), Digit Span, and Testing for Orientation. The most frequently used delirium detection tools were the Confusion Assessment Method (CAM), the Memorial Delirium Assessment Scale (MDAS), and the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU). (Place Table Here)

Figure 1.

Sources of Information for Reference Raters’ Delirium Evaluation (n=33 studies)

Table.

Frequency and Type of Standardized Cognitive and Delirium Detection Tool Used by Reference Rater as Part of Delirium Evaluation

Cognitive Tests (Used in 20 (61%) of 33 studies)a	N	% of All Studies N=33
Mini Mental Status Exam (MMSE)	17	52%
Digit Span	12	36%
Testing for Orientation (e.g. person, place, time)	12	36%
Months of the Year Backwards	7	21%
Days of the Week Backwards	5	15%
Clinician’s Global Rating	5	15%
Clock Drawing Test	2	6%
Short Portable Mental Status Questionnaire (SPMSQ)	2	6%
Comprehensive Geriatric Assessment	2	6%
Clinical Dementia Rating (CDR)	2	6%
Trail Making Test A	2	6%
Trail Making Test B	2	6%
Blessed Dementia Scale	2	6%
Digit Cancellation Test	1	3%
Vigilance “A” Test (VAT)	1	3%
Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE)	1	3%
Delirium Detection Tools (Used in 13 (39%) of 33 studies)a
Confusion Assessment Method (CAM)	7	21%
Memorial Delirium Assessment Scale (MDAS)	4	12%
Confusion Assessment Method for the ICU (CAM-ICU)	4	12%
Delirium Rating Scale- Revised 1998 (DRS-98R)	3	9%
Intensive Care Delirium Screening Checklist (ICDSC)	2	6%
Clinical Assessment of Confusion (CAC)	1	3%
Nursing Delirium Screening Scale (Nu-DESC)	1	3%
Cognitive Test for Delirium (CTD)	1	3%
Delirium Index	1	3%
Other (list)	6	18%

^aMore than one cognitive test or delirium detection tool could be used in a given study.

Delirium Diagnostic Assignment

The diagnostic classification systems used in the 33 studies were as follows: 58% DSM IV, 24% DSM III-R, 9% ICD 10, and 9% CAM algorithm. The incorporation of cognitive and delirium screening tool information and clinical impression in the diagnostic assignment is displayed in Figure 2. Respondents described the use of a consensus panel to review delirium diagnoses in 70% (n=23) of studies; 27% (n=9) did not use such a procedure and 3% (n=1) reported another procedure where only select patients were reviewed.

Figure 2.

Information Used in Reference Raters’ Delirium Diagnosis (n=33 studies)

Special Examination Considerations

Respondents reported procedures used when patients declined to answer the cognitive testing questions during the evaluation. Studies used all available information to decide on delirium diagnosis in this case in 70% (n=23), excluded the patient in 15% (n=5), and were not certain about the procedure in 15% (n=5). The procedure for evaluating patients with a decreased level of arousal that precluded formal cognitive testing revealed that 70% (n=23) of the studies used all available information to decide on delirium diagnosis, 15% (n=5) excluded the patient from the study, 6% (n=2) were uncertain about the procedure, and in 9% (n=3) another procedure was employed but the description was not included in the response to the survey.

Discussion

This is the first systematic evaluation of reference rater practices for the diagnosis of delirium in clinical research studies of hospitalized patients. The reference rater methods reported in the survey are quite variable. The majority of reference raters were physicians, from a number of different specialties, with wide-ranging clinical experience in diagnosing delirium. Almost without exception, the reference standard diagnosis was reported to be based on “clinical judgment”, and while diagnosis was reported to incorporate diverse sources of information, including the patient’s performance on a wide range of cognitive tests and delirium rating tools, a substantial proportion of reference raters based their diagnosis on clinical judgment in the absence of any standardized testing.

The literature suggests that delirium under-diagnosis by physicians in clinical practice is as high as 46–66% among non-critically ill, hospitalized patients. ^{23, 24} Findings in an ICU population reveal that physicians were able to identify delirium with 29% sensitivity when using clinical judgment alone without the aid of objective testing.²⁵ While clinical judgment will be required in reference standard diagnosis, it needs to be informed by data gained from objective cognitive testing and collateral information gathered in a systematic examination.

There are no definitive objective diagnostic tests for delirium at this time; however this is true of many other psychiatric diseases as well. As is typically the case in most psychiatric research, clinical and research practice has typically relied on determining a categorical presence or absence of a given diagnosis, using clinical examination, with the clinician both eliciting and interpreting the significance of patients’ signs and symptom.²⁶ This approach relies on the experience of the clinician for making accurate (or valid) diagnoses in this process and has often regarded the more experienced physician as the “gold standard”. ²⁷ However there are potential shortcomings with this approach; for example, literature has demonstrated the fallibility of radiologic diagnosis, with as many as 30% of pulmonary lesions missed when using a single radiologist’s review.²⁸ Since the potential for diagnostic error might be high in delirium given its fluctuating presentation, even a one-time assessment by an experienced physician cannot necessarily be regarded as a “gold standard”, with serial assessments and collateral information from sources such as family, care providers and validated chart review to detect fluctuations over time potentially being required for the most accurate diagnostic assessment.

The results from our survey suggest that only a minority of reference raters were non-physicians. Our survey does not determine the appropriateness of non-physicians in completing such assessments. There is some evidence that well-trained, non-clinical research personnel can use standardized tools to gather information relevant to delirium assessment with high reliability and validity when compared to physicians.²⁹ Rigorous training, before and during studies, led by experienced clinicians expert in delirium diagnosis, accompanied by inter-rater reliability evaluation, is one method to improve reliability of delirium diagnoses.²⁷ Quality control efforts, such as co-rating in-person, videotaped interviews or sequential evaluations of the same patient on the same day, should be used to assess inter-rater reliability throughout a study. Due to the heavy reliance on clinical impression by reference raters, a consensus panel of multiple experts reviewing all available data after a patient evaluation may also increase the validity of the delirium diagnostic process. ³⁰ A majority of studies in this survey indicated convening a consensus panel.

Two thirds of survey respondents endorsed the use of either standardized cognitive testing and/or the use of delirium detection tools as part of their diagnostic assessment; one third did not use any such instruments, basing their diagnosis solely on clinical impression. The use of standardized tools might be an additional method for improving the reliability, and ultimately validity and generalizability of delirium diagnoses. The use of itemized dimensional symptom scales, such as the MDAS ³¹ or the Delirium Rating Scale-Revised 98³² allows for the calculation of delirium severity and further study of specific delirium phenomenology. Approaches to operationalize criteria in delirium diagnosis detailed in the literature, include the careful description by Gottlieb and colleagues.³³ Each DSM III criteria is demonstrated clinically with specific cognitive testing and rating of behavioral symptoms. A similar approach is employed by other authors in the comparing delirium identification by different diagnostic systems including the DSM editions III, III-R, IV and the ICD10.^34–36 Routinely reporting operationalization approaches used by reference raters in the describing study methodology would greatly enhance the ability to compare findings between studies of delirium.

Findings from this survey highlight variability in how studies incorporate patients who may be unable or unwilling to participate in cognitive testing. Patients who decline to answer questions, or are too drowsy to participate in cognitive testing may be more likely to have delirium than those who are willing and able. A previous study reported that 50% of patients refusing to answer attention testing questions were assessed as delirious by a second examiner using all available clinical information.³⁷ Studies excluding patients who refuse to participate on a given day, or are too drowsy to comply with active testing may be subject to ascertainment bias, especially if not including information from collateral informants and medical record review. The exclusion of non-responders who represent patients with the hypoactive motoric subtype of delirium is a serious problem; previous research suggests that the hypoactive subtype is associated with worse clinical outcomes when compared to the hyperactive subtype.³⁶

This is the first study to systematically describe the reference rater methodology used for delirium diagnosis in clinical research evaluating hospitalized patients, and to our knowledge, is a unique contribution to the delirium literature. Advancing reference rater methodology is critical because, in the absence of laboratory tests or biomarkers, clinical expert opinion will remain the standard to establish diagnostic criteria for the field moving forward. This study also has potential limitations. First, the survey does not include information from all possible studies in the field; however, it does include 33 (85%) of 39 studies from 3 independent systematic reviews of delirium screening tools, and as such it is likely representative of the main body of available literature. Moreover, our survey response rate is better than the 68% rate previously described for physicians ³⁸ and 56% for academics in general.³⁹ Second, our survey did not delineate exactly how each reference raters used data from the exam to operationalize the delirium criteria employed in the various diagnostic classification systems (e.g., DSM-IV) used in these studies. More detail in this regard would be helpful in understanding the generalizability of the diagnostic process and help develop research standards for diagnosis. This detail would be better elucidated via future qualitative interviews of reference raters. Finally, this study cannot offer guidelines for best diagnostic practices; however, until present the reference standard delirium diagnosis has remained a “black box” in much of the literature. This survey clearly describes the variability of practice and reveals areas where increased standardization of reference examinations may help advance the field.

Conclusion

This survey demonstrates significant variability in reference rater methods for delirium diagnosis in published research of hospitalized patients. Based on these findings, reference rater methods could be further improved in the following areas: 1) performing and reporting inter-rater reliability among multiple reference raters, 2) using standardized cognitive and validated delirium detection tools, 3) incorporating consensus panels for adjudication of diagnoses, and 4) using all available information in patients declining or unable to comply with formal testing. Such measures have the potential to advance delirium research by increasing consistency of case detection and improving generalizability of studies conducted in different settings.