Developing the CAM-BERT: Enhancing delirium screening in hospitalized older adults using natural language processing

Abstract

Background:

Delirium is a common condition affecting hospitalized older adults, often leading to adverse outcomes. Nevertheless, delirium frequently goes unrecognized due to various clinical and systemic challenges. We aimed to develop and evaluate a deep-learning natural language processing (NLP) model trained on Brazilian Portuguese clinical notes, aiming to improve the identification of delirium symptoms in electronic health records (EHR) and to facilitate the detection of delirium.

Methods:

We extracted free-text clinical notes from 500 hospitalizations of older adults at a tertiary care hospital in São Paulo, Brazil, for annotation and analysis. Delirium symptoms were identified and labeled by expert clinicians using a structured protocol. The deep learning model BERTimbau was employed alongside a classical Random Forest approach for comparison, with performance metrics derived from F1 scores. We also developed the CAM-BERT framework, an algorithmic approach to categorize potential delirium cases by aligning symptoms classified by the model with CAM criteria.

Results:

The BERTimbau model showcased superior performance, with an F1-macro score of 77 % compared to the baseline model’s 39 %. It achieved F1 scores of around 90 % for identifying confusion and disorganized thinking. For the CAM-BERT framework, which mapped detected symptoms to CAM criteria, the overall F1-macro score was 83 %. The agreement with expert chart review yielded a Cohen’s kappa coefficient of 0.72.

Conclusions:

The study highlights the potential of artificial intelligence, particularly NLP, in supporting the recognition of delirium in non-English-speaking clinical settings. Further research is needed to validate the model’s applicability across diverse healthcare environments.

1. Introduction

Delirium, a serious and often underdiagnosed geriatric syndrome, affects an estimated 23 % of older adults in medical settings [1]. Characterized by its sudden onset and fluctuating cognitive disturbances, delirium contributes to adverse health outcomes, including prolonged hospital stays, delayed recovery, a decline in both cognitive and functional abilities, and increased mortality [2,3]. The impact of delirium extends to care partners and the healthcare system, where it is associated with substantial costs [4]. However, up to two-thirds of delirium episodes go unrecognized in hospital settings [5,6]. Factors contributing to this oversight include the complex and multifaceted nature of delirium, its overlapping symptoms with dementia, and the variability in its clinical presentation, compounded by a general lack of provider awareness and training in delirium diagnosis [7]. Furthermore, inconsistencies in the documentation of delirium within medical records present additional barriers to improving patient care continuity and hospital workflow efficiency [8].

Leveraging technology, particularly the field of linguistic modeling within machine learning, offers transformative potential to enhance the recognition and documentation of delirium. Linguistic modeling encompasses a range of interdisciplinary applications designed to interpret, process, and transform human language into actionable data [9], Among these, natural language processing (NLP) stands out for its ability to analyze vast amounts of unstructured text, such as that found in electronic health records (EHR), to identify clinical conditions [10]. With the advent of artificial neural networks and deep learning, our capacity to model complex patterns within human language has significantly advanced, often surpassing traditional computational methods [11].

Prior research exploring NLP in English-language EHRs has yielded promising outcomes in delirium detection [12–18]. Fu et al. developed an algorithm capable of extracting explicit mentions of delirium and identifying symptoms aligning with CAM criteria, demonstrating over 90 % accuracy against manual chart reviews [13]. The same model applied to a cohort of older adults hospitalized with COVID-19 outperformed traditional clinician-based diagnoses, which relied on ICD-10 codes and Brief-CAM assessments [17]. Additionally, an NLP tool designed to detect behavioral disturbances indicative of delirium in an intensive care setting showed potential for anticipating the need for antipsychotic medications compared to assessments using the CAM-ICU [14]. A separate investigation involving a multi-hospital dataset illustrated that various NLP methodologies, including Support Vector Machines, recurrent neural networks, and transformers, were effective in categorizing delirium-related narrative elements, with transformers achieving the highest precision [16].

Nonetheless, the reach of these technologies has been largely confined to English-language medical records. A recent study explored large language models to classify German nursing reports, with finetunning outperforming prompting for delirium detection [18]. The scarcity of health-related datasets, particularly from less-resourced languages, limits the global applicability of these technologies to train NLP models in the nuances of different languages [19]. Portuguese, for instance, is a low-resource language whose linguistic complexities make the development of tailored models significantly more challenging than for English [20]. It features a complex verb conjugation system, where verbs change according to tense, mood (indicative, subjunctive, imperative), person, and number, with numerous irregular verbs. In contrast, English verbs have fewer forms, primarily changing between past, present, and participle forms. Additionally, in Portuguese, adjectives and articles must agree with the noun in gender, while English lacks grammatical gender, making noun-adjective agreement simpler. Portuguese also presents challenges, such as the use of diacritics and accentuation.

This study introduces a deep-learning NLP model trained on a corpus of Brazilian Portuguese clinical notes to accurately classify delirium symptoms in the electronic health records of hospitalized older adults in Brazil. Furthermore, we investigate the utility of this model in inferring Confusion Assessment Method (CAM) items and thus enabling the detection of presumed delirium cases, potentially filling a critical gap in delirium screening within non-English speaking healthcare environments.

2. Methods

2.1. Study design and population

This retrospective study analyzed data from single hospitalizations of 500 older adults at Hospital Israelita Albert Einstein in São Paulo, Brazil – a 600-bed tertiary facility – recorded between January 2018 and June 2022. Participants in the study were aged 65–99 years to comply with data anonymization protocols, as the inclusion of the rare admissions of patients over 100 years could increase the risk of re-identification. The minimum length of stay was set at 48 consecutive hours to ensure adequate time for delirium symptoms to manifest and be observed by the clinical staff, with a cap at 26 days to avoid the confounding effects of prolonged hospitalization on delirium assessment. The local institutional review board approved the study, and it was granted a waiver of informed consent due to the study’s retrospective nature and the negligible risk of confidentiality breaches through the rigorous data pseudonymization process in place.

To favor model training, we included stratified random admissions based on the presence or absence of delirium indicators: 45 % of the admissions had a positive CAM screening or a corresponding delirium-related ICD-10 code (see Supplementary Table S1 for codes utilized for identifying potential delirium cases) [21]; 35 % had negative CAM screening results; and 20 % had no recorded CAM screening or delirium-related ICD-10 codes.

2.2. Clinical notes and annotation of delirium symptoms

We extracted all free-text clinical notes regarding admission and routine daily assessments from the EHRs, which were authored by physicians, nurses, nurse assistants, and physical therapists. To maintain the focus on the active phase of hospital care, we omitted discharge notes and those recorded during stays in intensive care units, acknowledging the distinct nature of delirium manifestations in critically ill patients.

An expert panel, comprised of seven physicians specialized in geriatrics, was trained to conduct the annotation process. These experts reviewed clinical notes to tag passages suggestive of the following delirium symptoms: acute onset or fluctuation; inattention; disorganized thinking; altered level of consciousness; psychomotor agitation; hypoactivity or lethargy; disorientation to time, place, or person; hallucinations; and sleep-wake cycle disturbances [22]. We collected a robust dataset to train and validate our machine learning models through a meticulous process involving blinded pairs of reviewers labeling these symptoms, while a third expert oversaw the process to ensure adherence to rigorous standards and to resolve discrepancies among reviewers to achieve consensus. A definitive curation stage was then conducted, streamlining the annotations to align with established guidelines. This step was critical for eliminating extraneous data that could detract from the machine learning model’s capacity to accurately learn and identify symptoms of delirium. In other words, we first employed a high-quality double-review triage process for identifying potential delirium symptoms, relying subsequently on expert curation and consensus for the final classifications. The detailed description of data processing for annotation and the accounts of the annotation criteria and exemplars, provided in both the original Brazilian Portuguese and its English translation, are available in the Supplementary Materials (Supplementary Box S1 and S2). Finally, through the systematic examination of clinical notes, experts also identified delirium during hospitalizations, employing a validated chart review methodology [23].

2.3. Complementary data extraction and delirium screening comparative analysis

Structured variables were extracted from the administrative databases and EHRs to characterize the study cohort, encompassing demographics (sex and age), admission specifics (total duration of hospital stay, length of ward stay, surgical vs. clinical admission, admission urgency, admission to intensive care), alongside comorbidities quantified using the Charlson Comorbidity Index [24]. Data on CAM screenings were also extracted, as recorded in the EHR.

2.4. Data preprocessing for model training

The textual data in the annotated dataset was segmented into 53,128 discrete sentences using newline characters to preserve the context for accurate model training. A structured dataset was constructed, assigning one or more labels to each sentence based on the annotated delirium symptoms or their noted absence. Terms related to hallucinations were encountered infrequently (with just nine instances recorded), posing potential challenges to the model’s learning efficacy and generaliz-ability, and were omitted from the model development process.

The dataset was then divided by admissions, ensuring that clinical notes from the same participant did not appear in multiple subsets: training subset 48 % (240 admissions), validation subset 32 % (160 admissions), and test subset 20 % (100 admissions). To address the imbalance, where only 3.4 % of sentences were annotated for delirium symptoms, the training set was enriched by keeping all 1532 sentences flagged for symptoms and supplemented with a random selection of 5000 symptom-free sentences. Such refinement, however, was not extended to the validation and test sets to more accurately reflect real-world clinical scenarios. The distribution of sentences featuring each annotated symptom across the training, validation, and test sets is detailed in Table 1. Data manipulation and analysis were conducted using Python 3.9 (libraries for handling the data: Pandas 1.5, Datasets 2.8, NLTK 3.8.1, and dkpro-cassis 0.7.5).

Table 1.

Sentence annotations for delirium symptoms in training, validation, and test sets.

Delirium symptom	Training seta n = 6532	Validation setb n = 27,798	Test setb n = 18,798
Acute onset or fluctuation, n (%)	574 (8.8)	389 (1.4)	282 (1.5)
Inattention, n (%)	134 (2.1)	59 (0.2)	69 (0.4)
Confusion/disorganized thinking, n (%)	543 (8.3)	319 (1.1)	265 (1.4)
Spatial-temporal disorientation, n (%)	123 (1.9)	66 (0.2)	47 (0.3)
Altered level of consciousness, n (%)	474 (7.3)	256 (0.9)	257 (1.4)
Psychomotor agitation, n (%)	413 (6.3)	319 (1.1)	265 (1.4)
Hypoactivity or lethargy, n (%)	63 (1.0)	51 (0.2)	57 (0.3)
Sleep wake cycle disturbance, n (%)	65 (1.0)	34 (0.1)	20 (0.1)

^aSentences were obtained by splitting text on newlines. The training set includes all sentences with one or more annotated symptom and 5000 randomly selected sentences without symptoms.

^bThe validation and test sets contain all sentences from the clinical notes.

2.5. Statistical analysis

The characterization of the study population was articulated through means and standard deviations for normally distributed continuous variables, medians and interquartile ranges for skewed variables, and frequencies for categorical variables (Python libraries: NumPy1.24, SciPy 1.9.1).

2.6. Development and evaluation of machine learning models

We employed the BERTimbau model, a deep learning neural network predicated on the principles of Bidirectional Encoder Representations from Transformers (BERT) [25]. This model extracts linguistic features by analyzing the sequence of words in both forward and reverse order within a sentence. Initially leveraging an English pre-trained BERT foundation, BERTimbau was calibrated to the nuances of Brazilian Portuguese, ensuring a better understanding of the language’s structure and idiomatic expressions [26].

We fine-tuned the BERTimbau model using an efficient sentence embedding-based approach with contrastive learning on our annotated clinical dataset to recognize and classify various delirium symptoms described in clinical notes [27]. During training, the model learned to distinguish symptom patterns, for instance, recognizing that “Continues agitated and confused. Slept little” and “Alert, very confused and disoriented” both contain confusion symptoms, while only the first includes psychomotor agitation. Fine-tuning was performed on an AWS g4dn.2xlarge instance (12 h, 9 epochs) with detailed hyperparameters provided in Supplementary Table S2.

Additionally, we employed a Random Forest classifier, a more conventional machine learning methodology that served as a baseline for performance evaluation. This model used a text preprocessing step, converting the unstructured clinical notes into a quantifiable format using Term Frequency-Inverse Document Frequency (assignation of weight to each semantic unit based on its frequency, rendering the raw text suitable for machine learning applications) [28].

Performance metrics for both the BERTimbau and Random Forest models were derived from the test set. We measured sensitivity (recall), specificity, positive predictive value (precision), and negative predictive value for each symptom classified. Precision and recall informed the calculation of the F1 score. The macro-average F1 score, calculated as the mean F1 score across all symptoms, was used to evaluate the overall performance. The hyperparameters guiding the learning processes were optimized using Bayesian optimization to maximize the macro-average F1 score (Supplementary Table S2). Python libraries: SetFit 1.5, Transformers 4.36, Sentence-Transformers 2.2.2, Torch 2.1.2, and Optuna 3.5.

2.7. CAM-BERT

We developed the CAM-BERT framework, an algorithmic approach leveraging the BERTimbau-based model to identify potential delirium cases by aligning delirium symptoms detected by the model with corresponding CAM items [13]. Symptoms of acute onset or fluctuation, psychomotor agitation, hypoactivity or lethargy, and sleep-wake cycle disturbances were grouped under the CAM criterion “acute onset or fluctuation.” Inattention was aligned with the “inattention” CAM item, while confusion or disorganized thinking and spatial-temporal disorientation were related to the “confusion or disorganized thinking” CAM item. Similarly, altered level of consciousness was associated with the respective CAM item. The CAM-BERT framework was assessed using the same aforementioned performance metrics.

The determination of delirium was predicated upon the identification of at least three CAM items within a rolling 24-h period. Fig. 1 illustrates the application of CAM-BERT in classifying delirium symptoms for a single patient case from the test set. The framework’s diagnostic output was then compared to that of the expert’s identification of delirium through conventional chart review, the gold standard, through the same performance metrics and Cohen’s kappa coefficient.

Fig. 1.

This timeline illustrates the sequence of delirium symptom detection for a patient from our test dataset using the BERTimbau-based model. Initially trained on a broad range of Brazilian Portuguese texts, this deep-learning model was further refined with our specific study data. True positives are instances where both the model and experts agree on symptom classification. False positives are symptoms the model detected but experts did not validate, and false negatives are symptoms experts identified, but the model missed. The dashed line marks when the CAM-BERT framework, which links model-detected symptoms to CAM criteria, suggests a potential delirium case based on the presence of three CAM items within 24 h.

3. Results

The cohort had a mean age of 80.2 ± 9.0 years, and 51 % were female. Medical admissions comprised 70 % of the cases, while surgical admissions accounted for 30 %. Most hospitalizations (55.8 %) originated from the emergency department, and 34.6 % of patients required intensive care. The average length of stay was 6.9 ± 4.7 days, with a mean of 4.7 ± 2.7 days spent in wards. The median Charlson Comorbidity Index was 6 (interquartile range 5–8). Delirium, as identified through chart reviews by experts, was noted in 192 individuals (38.4 % of the sample). Positive delirium screening based on the CAM was reported for 148 patients (29.6 %), and delirium indicated by ICD-10 codes was present in 96 patients (19.2 %).

The BERTimbau-based model’s ability to classify delirium symptoms outperformed the baseline model, with F1-macro scores of 77 % and 39 %, respectively (Table 2, Fig. 2). It demonstrated proficiency in recognizing symptoms of confusion, disorganized thinking, and spatial-temporal disorientation, with F1 scores approximating 90 %. Classification performance for other delirium symptoms was also robust, ranging from 70 to 84 %, aside from identifying sleep-wake cycle disturbances, which had an F1 score of 49 %. Both models presented extremely high specificity and negative predictive values. Examples of false-positive and false-negative results of the BERTimbau-based model can be found in the Supplementary Box S3.

Table 2.

Model performance in delirium symptom classification: BERTimbau vs. Baseline Model.

Delirium symptom	BERTimbau-based modela					Baseline modelb
	Sensitivityc	Specificity	PPVd	NPV	F1 scoree	Sensitivityc	Specificity	PPVd	NPV	F1 scoree
Acute onset or fluctuation	0.88	1	0.71	1	0.78	0.61	1	0.76	1	0.68
Inattention	0.77	1	0.65	1	0.70	0.40	1	0.49	1	0.44
Confusion/disorganized thinking	0.96	1	0.89	1	0.92	0.66	1	0.89	1	0.76
Spatial-temporal disorientation	0.85	1	0.95	1	0.90	0.25	1	0.46	1	0.32
Altered level of consciousness	0.91	1	0.77	1	0.84	0.30	1	0.20	1	0.24
Psychomotor agitation	0.91	1	0.73	1	0.81	0.34	1	0.24	1	0.28
Hypoactivity or lethargy	0.65	1	0.79	1	0.71	0.16	1	0.30	1	0.21
Sleep wake cycle disturbance	0.55	1	0.44	1	0.49	0.29	1	0.13	1	0.18

NPV: negative predixtive value; PPV: positive predictive value.

^aThe BERTimbau-based model is a deep learning model that was initially trained on a broad range of Brazilian Portuguese texts and was further refined with our study data to classify delirium symptoms.

^bThe baseline model was developed using a classical NLP approach by applying a Term Frequency - Inverse Document Frequency to generate weights for words, that were later used as input to train a random forest algorithm to classify delirium symptoms.

^cAlso named recall.

^dAlso named precision.

^eThe harmonic mean between sensitivity and positive predictive value.

Fig. 2.

The radar chart illustrates the F1 scores achieved by the BERTimbau-based model and the baseline model in classifying each delirium symptom. The BERTimbau model, employing advanced deep learning techniques, is contrasted with the baseline model, which uses a more traditional approach to natural language processing that requires lower computational resources.

The overall F1-macro score of the CAM-BERT framework was 83 %. The “disorganized thinking” item exhibited the most robust performance, whereas the “inattention” item showed room for improvement, reflective of the individual symptom classification metrics (Table 3). When benchmarked against the gold standard of expert chart review, CAM-BERT demonstrated high sensitivity at 97.4 % and specificity at 78.7 %, with a positive predictive value of 74.5 % and a negative predictive value of 98 %. The concordance between CAM-BERT and expert chart review methods was significant at 0.72.

Table 3.

CAM-BERT^a performance metrics for CAM item identification.

CAM item	Sensitivityb	Specificity	PPVc	NPV	F1 scored
Acute onset or fluctuation	0.89	1.00	0.72	1.00	0.79
Inattention	0.77	1.00	0.65	1.00	0.70
Disorganized thinking	0.95	1.00	0.90	1.00	0.92
Altered level of consciousness	0.91	1.00	0.77	1.00	0.84

Note: The table details the performance of the CAM-BERT algorithm, a novel approach that utilizes symptom classifications from the BERTimbau-based model to infer CAM items. The table lists sensitivity, PPV, and F1 score for each CAM item, which represents the algorithm’s precision in classifying potential delirium cases.

^aThe CAM-BERT framework applies rules to assign delirium symptoms classified by the BERTimbau-based model to their corresponding CAM item.

^bAlso named recall.

^cAlso named precision.

^dThe harmonic mean between sensitivity and positive predictive value.

4. Discussion

We developed and tested a deep-learning NLP model engineered to discern delirium symptoms from EHRs of hospitalized older adults in Brazilian Portuguese. The model demonstrated proficiency in identifying a wide selection of delirium symptoms by leveraging a neural network pre-trained on general texts in Brazilian Portuguese and refining it with expertly annotated clinical notes. Furthermore, our CAM-BERT framework, which assigns CAM criteria to model-identified symptoms, has shown promise in accurately identifying delirium, as indicated by a good correlation with delirium identified by experts.

Our findings highlight the potential of NLP as a tool to aid in the more accurate screening and diagnosis of delirium, a condition that is often overlooked yet carries significant morbidity and healthcare costs. Current clinical practices fall short in delirium detection compared to the heightened recognition rates reported in controlled research environments [29]. This discrepancy points to the role artificial intelligence could play in enhancing the detection of delirium symptoms documented in EHRs, which, due to the current lack of adequate training and awareness among healthcare providers, are not always integrated into clinical decision-making [7]. The efficacy of the BERTimbau-based model, which demonstrated superior performance in symptom classification over a traditional model, validates the investment in more sophisticated computational resources that underpin deep learning techniques. The introduction of the CAM-BERT technology to the clinical workflow could alert healthcare professionals about potential delirium in hospitalized older adults, prompting timely diagnosis confirmation and management to prevent adverse outcomes.

The variability in model performance across different delirium symptoms can largely be attributed to the frequency and quality of symptom descriptions in the EHR. The model showed exceptional efficacy in recognizing confusion, disorganized thinking, and spatial-temporal orientation, as reflected by the high F1 scores. Conversely, identifying sleep-wake cycle disturbances was less accurate, possibly due to their underrepresentation in the training data. The classification of symptoms less often documented, like inattention, and hypoactivity, was competent but did not reach the accuracy levels of more commonly reported symptoms. This may suggest that the model’s learning ability was impacted by the diversity of language used to describe these less frequent symptoms. Possible strategies to mitigate these performance disparities include incorporating rule-based approaches using regular expressions to capture less frequent and more diverse symptom descriptions, augmenting the training dataset through automatic labeling of additional clinical notes, and increasing the sample size for underrepresented symptoms to improve the model’s ability to generalize from diverse linguistic patterns.

When applying the CAM-BERT framework, the identification of “acute onset or fluctuation” remained unimpaired, likely given the possible allocation of multiple delirium symptoms to different CAM criteria. A notable issue emerged with the CAM item inattention in the EHR data, where a mere 2.1 % of sentences included the symptom. This observation indicates the urgent need for improved staff training on the varied presentations of delirium, as well as the adoption of more effective bedside inattention screening methods, which remain underutilized in clinical practice [30].

Our study had limitations. Initially, the scope of delirium symptom identification adhered strictly to the nine characteristics outlined in the original CAM framework [22]. While comprehensive, this approach may have overlooked or inaccurately categorized certain manifestations of delirium. Moreover, the model was not trained to identify hallucinations due to their scarce mention in clinical documentation, possibly reflecting a gap in the training of healthcare personnel to recognize and record such symptoms. Additionally, the EHR data often lacked detailed descriptions of inattention, prompting us to formulate an annotation guideline that prioritized the sensitivity of detection over specificity. While potentially increasing the incidence of false positives, this decision aligns with the overarching goal of employing artificial intelligence to enhance the sensitivity of delirium screening in clinical practice. Furthermore, we did not explore uncertainty quantification of predictions for clinical usage due to computational constraints. We prioritized model accuracy and generalization over uncertainty estimation, achieving robust performance metrics across our three independent datasets. Future work should explore methods such as Monte Carlo dropout or ensemble approaches [31]. Finally, as a single-center study, the model’s training and testing were confined to the same patient population, raising questions about its external validity and scalability, which should be addressed in the future.

In summary, our study showcases the prospective role of artificial intelligence, through NLP in Brazilian Portuguese, in classifying delirium symptoms from clinical notes in EHRs, thereby aiding in the accurate evaluation of CAM items and potentially refining delirium diagnosis in hospitalized older adults. Integrating such technology in healthcare settings may improve diagnostic accuracy and serve as a catalyst for enhancing delirium awareness among healthcare professionals. Ongoing research is required to further investigate the deployment and impact of this NLP model on the diagnosis of delirium in hospitalized older adults.

Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.compbiomed.2025.110781.