A Hybrid Model to Identify Fall Occurrence from Electronic Health Records

Abstract

Introduction:

Falls are a leading cause of unintentional injury in the elderly. Electronic health records (EHRs) offer the unique opportunity to develop models that can identify fall events. However, identifying fall events in clinical notes requires advanced natural language processing (NLP) to simultaneously address multiple issues because the word “fall” is a typical homonym.

Methods:

We implemented a context-aware language model, Bidirectional Encoder Representations from Transformers (BERT) to identify falls from the EHR text and further fused the BERT model into a hybrid architecture coupled with post-hoc heuristic rules to enhance the performance. The models were evaluated on real world EHR data and were compared to conventional rule-based and deep learning models (CNN and Bi-LSTM). To better understand the ability of each approach to identify falls, we further categorize fall-related concepts (i.e., risk of fall, prevention of fall, homonym) and performed detailed error analysis.

Results:

The hybrid model achieved the highest f1-score on sentence (0.971), document (0.985), and patient (0.954) level. At the sentence level (basic data unit in the model), the hybrid model had 0.954, 1.000, 0.988, and 0.999 in sensitivity, specificity, positive predictive value, and negative predictive value, respectively. The error analysis showed that that machine learning-based approaches demonstrated higher performance than a rule-based approach in challenging cases that required contextual understanding. The context-aware language model (BERT) slightly outperformed the word embedding approach trained on Bi-LSTM. No single model yielded the best performance for all fall-related semantic categories.

Conclusion:

A context-aware language model (BERT) was able to identify challenging fall events that requires context understanding in EHR free text. The hybrid model combined with post-hoc rules allowed a custom fix on the BERT outcomes and further improved the performance of fall detection.

INTRODUCTION

Falls are a leading cause of unintentional injury, especially among older adults (1–3). Studies have estimated that between 30–40% of patients over the age of 65 will fall at least once, (4, 5) and that over 40–60% of falls lead to injuries, which include body injuries, fractures, and brain trauma (6). The estimated direct medical costs were $616.5 million and $30.3 billion for fatal and non-fatal fall injuries in 2012 (7). Identifying individuals at high risk for future falls can provide the opportunity to implement multifactorial interventions to reduce the risk of future falls (8). The rich data available in electronic health records (EHRs) offer the unique opportunity to develop models that can identify fall events and persons at risk of falling. However, the use of billing codes may underestimate true fall events. A study of patients treated by the Veteran’s Administration found that fall-related codes were assigned to about 74% of patients with falls documented in the medical record (9). Tremblay and colleagues have noted that the failure to identify falls with only billing codes may occur because billing codes are designed for reimbursement purposes, and may be applied inconsistently across clinical practices (10). In particular, billing codes may underestimate the frequency of less severe falls that do not require immediate medical attention, but that may be documented by health care providers during the course of other health care visits. To fully leverage the EHR data to accurately, clinical free text, such as clinical notes, are required identify fall events.

Identifying fall events in clinical notes requires natural language processing (NLP) to simultaneously address multiple issues because the word “fall” is a typical homonym. Word sense disambiguation is needed to identify the correct semantic meaning of a word given a particular context (11, 12). Specifically, there are various fall related synonyms and semantic variations in clinical notes. Based on the definition proposed by the American Nurses Association (ANA), a true fall in the context of EHRs is “an unplanned descent to the floor with or without injury”(13). Traditional symbolic approaches that use keywords (e.g., “falls”) and description patterns (e.g., regular expressions) can be overwhelmed by the variation of the context. For example, common expressions found in patient clinical notes include the following: 1) True fall: “patient fell yesterday at about 1 p.m.”; 2) False fall (did not happen): “almost falling at the end”; 3) Not a fall event (fall risk): “patient remains at fall risk due to right hemiplegia, right hip pain”; or 4) False fall (homonym): “TSH (thyroid-stimulating hormone) normal this fall”; “Skin falls apart”; “lives in Cannon Falls”; “symptoms worse in fall and winter”.

Recently, researchers have developed text mining methods to extract falls information from clinical free text in EHRs (10, 14, 15). Tremblay et al. used term frequency-based representation on a k-mean algorithm for cluster analysis and a logistic regression model for fall classification (10). Similarly, McCart leveraged a term frequency–inverse document frequency (tf-idf) based statistical machine learning approach to identify falls in ambulatory care settings (16). Toyabe combines statistical testing methods (chi-square or Fisher’s exact) with syntactic category rules to classify fall related incidental reports (17). However, these models lack the ability to understand context to discern the true meaning of falls. More recently, Santos et al attempted to compare classic machine learning methods with deep learning (i.e., word embedding and recurrent neural networks [RNN]) to detect fall in EHR documents (18). They reported deep learning models (best model: f1-score 0.90 in a document level) outperformed classical machine learning models. This work, however, has potentials to be enhanced by using context-aware language models that are necessary to fully leverage context to identify complex fall events in the EHR free text.

To address these gaps, we explored the ability of a contextual pre-trained language model, Bidirectional Encoder Representations from Transformers (BERT) to identify falls from the EHR text. BERT leveraged the bidirectional training of transformers from the large unlabeled corpus, resulting in a strong ability to capture complex dependencies and context (19). To further improve the performance, we fused the BERT model into a hybrid architecture coupled with post-hoc heuristic rules built under an open-source UIMA–based(20) information extraction framework (21). The models were evaluated on real world EHR data and were compared to conventional rule-based and deep learning models. To better understand the ability of each approach to identify falls, we further categorize fall-related concepts (i.e., risk of fall, prevention of fall, homonym) and performed detailed error analysis.

MATERIALS AND METHODS

Study Cohort

Our study population consisted of persons who were participants in the Mayo Clinic Biobank (22, 23), and who were ≥65 years at the time of consent to enroll (n=22,772). Among these participants, we randomly selected 302 persons for manual chart review to annotate fall events: 152 persons who had an ICD-9 or ICD-10 code for “accidental fall” after the enrollment in the Mayo Clinic Biobank and 150 persons without any of these codes. ICD-9 codes used for fall identification were as follows: E804, E833-E835, E843, E880-E888, E917.5-E917.8, E929.3, and E987. ICD-10 codes were: W00.0XXA-W18.49XS.

Annotation and Data Set

There were two phases involved in the annotation process: 1) training and annotation guideline development phase to become familiar with the annotation process and to refine annotation guidelines, and 2) production phase to create the gold standard for NLP algorithm development and evaluation (24).

During the training and guideline development phase, the annotation guidelines were developed by a geriatrician (ZX) and a palliative care physician (BT). We used the all-inclusive definition of falls proposed by the ANA National database for nursing quality indicators (13):“An unplanned descent to the floor (or extension of the floor, e.g., trash can or other equipment) with or without injury. All types of falls are included, whether they result from physiological reasons or environmental reasons.” Two subject matter experts (ZX and BT) manually reviewed an age-stratified sample of 30 persons with an ICD-9 or ICD-10 code for “fall” and determined the point of interests for annotation (Table 1). A true fall event should satisfy the following attributes: confirmed certainty, present status or follow-up visit, experiencer (patient), and exclusion (no).

Table 1.

Summary of fall concepts for annotation

Concept	Examples	Attribute
Mention	fall/fell, tripped, slipped, slid	Certainty (negated, possible, hypothetical, confirmed) Status (present, follow-up visit, history) Experiencer (patient, other) Exclusion (yes, no) (e.g., fall asleep - exclusion: yes)
Indication	seizure, syncope/fainting, narcolepsy

The full annotation task was performed by two trained abstractors (DMI, CMP). The matched cases were determined by comparing the bipartite set alignment for two annotated sentences using the Kuhn-Munkres algorithm (25). The final corpus yield a concept-level IAA of 0.72 in f1-score. A consensus meeting was organized to resolve disagreements and annotation issues. The gold standard dataset includes 302 patients (5,447 documents, 308,967 sentences). In our study, we randomly selected 242 patients (148 positives, 94 negatives) as a training set (4,357 documents [862 positives, 3495 negatives], 243,582 sentences [4,256 positives, 239,326 negatives]) and 60 patients (32 positives, 28 negatives) as a test set (1,090 documents [263 positives, 827 negatives], 65,385 sentences [1,073 positives, 64,312 negatives]).

Modeling

We examined multiple approaches (26) for fall detection: a rule-based, word embedding (i.e., fit embedding features to machine learning models), pre-trained language (BERT), and hybrid (heuristic rules and BERT) approach. These approaches use different architecture designs and training processes (Figure 1). A comprehensive evaluation and error analysis were conducted for each approach. We evaluated the results in a different level of granularity: a concept (sentence), document, and patient level of falls. Each model first identifies sentences that contain fall events and aggregates them to determine a document level and patient level fall event status based on presence or absence of sentence-level fall events. The details of each model are described in Figure 1.

Figure 1.

Approaches to detect falls using EHR text

Rule-based Model

A rule-based model uses a comprehensive set of rules and keyword-based features to identify pre-defined patterns in text. This method was adopted because of its advantage of simplicity, transparency and tractability. We used the open-source clinical NLP pipeline MedTaggerIE to handle clinical context and apply lexicon search. MedTaggerIE is a resource-driven open-source information extraction (IE) framework based on the Apache Unstructured Information Management Architecture (21), where the knowledge sources (i.e., keywords, rules) are separated from the generic NLP processes in order to increase algorithm portability and customizability. Three steps were involved in developing the rule-based model. First, we compiled the initial keywords and regex search patterns based on existing studies (14) and domain experts (ZX, BT). Second, the initial model was iteratively evaluated using the training data. False positive and negative cases were manually reviewed for rule refinement. This process was repeated on the training set after acceptable performance was reached (f1-score > 0.95). Lastly, the final algorithm was evaluated on the independent test set. The fall-related keywords were provided in Supplementary Appendix.

Word Embedding Model

A deep learning model paired with embedding representation is a common approach for extracting clinical concepts (27). We defined the task as a sequential sentence classification, and explored two well-established deep learning models: Convolutional neural network (CNN) and Bi-directional long short-term memory networks (Bi-LSTM). Both models used word vectors as input representation, where each word from the input sentence is represented as the k-dimensional word vector. The embedding model is based on the Global Vectors for Word Representation, an unsupervised learning algorithm proposed by Pennington et al (28). The pretrained data were based on Wikipedia and Gigaword. LSTM is a variant of recurrent neural network (RNN) with the ability to learn long-term dependencies. An LSTM cell contains a forget gate, an update date and an output gate. We used sigmoid functions to calculate the gate value and tanh function to calculate candidate value. To enhance the ability of capturing additional contextual information, we applied bi-directional forward and backward passing components to LSTM cells. Lastly, we used a fully connected linear layer for outputting a prediction.

Context-aware Language Model

A pretrained language representation model (BERT) was used because of its capability of context understanding and recent impact in various areas of research and state-of-the-art results (29). Each sentence is represented as a consecutive array of tokens. The first word of the sequence is labeled as the [CLS] token. The last layer of the [CLS] token is connected with a fully connected layer and a SoftMax layer for computing the sentence level classification. Minimal processing methods were applied including sentence segmentation and converting all text to lowercase. For computational efficiency, we used BERTbase, a pre-trained model with pre-trained sentences on 11,038 unpublished books and Wikipedia. The model contains 768 hidden layers and 12 self-attention heads. For the model fine-tuning, we set 218 as the maximum sequence length with a batch size of 32. We applied the early stopping technique to prevent overfitting. We also explored ClinicalBERT and other transformer variants such as XLNet, XLMs and Albert. These models, however, did not outperform BERT in this study.

Hybrid Model

A hybrid model leverages both a rule- and machine learning-based approach into one system and potentially offers some learning advantages and yields better performance. We adopted a post-hoc hybrid design that leverages rules to correct false BERT fall detection sentences. The hybrid model consists of a rule-based (MedTaggerIE), BERT and post rule adjustment module. As illustrated in Figure 1, the input sentence is processed independently by the BERT model and the rule-based model. Results of the models are reconciled by a summarization engine, where BERT results are overwritten by heuristic rules (Figure 1). These post-hoc rules are different from the rules in the pure rule-based model, which uses complex and comprehensive rules, but the rules in the hybrid model are rather complimentary focused on fixing the errors of the BERT model. For example, any generic fall expressions follow by the predefined positive trigger patterns such as “chief complaint”, “cause of wound”, or “landed on” reconcile the case as a true fall. Similarly, negative trigger patterns such as “risk” or “educated” will determine the case as negative event regardless of BERT’s output. These trigger patterns were determined based on the error cases in the training data. The implementation details for all models can be found in supplementary appendix.

Evaluation

All fall detection models were evaluated on a same independent test set at three levels (sentence, document, and patient level). Performance was assessed through sensitivity (recall), specificity, positive predictive value (PPV, precision), negative predictive value (NPV), and f1-score [2(precision × recall)/(precision + recall)] (30). The error analysis was performed by the same annotation team through manually reviewing incorrect cases from EHRs.

We also conducted comprehensive analyses of incorrect fall detection to understand each model’s behavior. The Oxford dictionary (31) was used to categorize different types of fall detection errors (Table 2). The cases with the highest number of misclassifications from all models were qualitatively analyzed.

Table 2.

Semantic categories of incorrect fall events

Category of fall detection errors	Example
Negation (Neg)	Denies a fall event.
Historical fall (Hx)	Patient was injured after falling in the last year.
Fall prevention (Prv)	Patient required handhold assist to prevent falling.
Fall risk (Rsk)	Risk of falls with injuries.
Lack of context (L)	Fall.
Homonyms (Hmn)
Pass into physical or mental condition (Cond)	Falls back to sleep easily.
Pass at a certain time (T)	Until the fall of 2010.
Become of a lower level (Low)	With falling blood pressure and mental status changes.
To come by chance (Chc)	Fall into the following categories.
To come by chance into a particular position (Pos)	To fall among.
To have its proper place (Plc)	It falls on the last syllable.

RESULTS

Concordance between Fall Detection Models and Gold Standard

As shown in Table 3, the models produced f1-score of 0.877 to 0.971 at the sentence level (basic data unit in the models) in detecting fall events from clinical notes. Machine learning-based methods had the higher performance than a rule-based method except CNN. The hybrid model achieved the highest f1-score at all levels; sentence (0.971), document (0.985), and patient (0.981). Specifically, the highest performance among all models at the sentence level was a sensitivity=0.954, specificity=1.000, PPV=0.988, and NPV=0.999, which matches with the performance of the hybrid model. All models performed high in f1-score at a patient level (≥0.9).

Table 3.

Model performance on three levels of evaluation

Level	Model	Sensitivity	Specificity	PPV	NPV	f1-score
Sentence P: 1073, n: 64312)	Hybrid	0.9543	0.9998	0.9884	0.9992	0.9711 (0.96, 0.98)
	BERT	0.9534	0.9995	0.9724	0.9992	0.9628 (0.95, 0.97)
	ClinicalBERT	0.9422	0.9992	0.9520	0.9990	0.9471 (0.94, 0.96)
	Rule	0.8509	0.9985	0.9040	0.9975	0.8766 (0.86, 0.89)
	CNN	0.9422	0.9965	0.8220	0.9990	0.8780 (0.86, 0.89)
	Bi-LSTM	0.8779	0.9993	0.9554	0.9979	0.9150 (0.9, 0.93)
		0.9696	0.9927	0.977	0.9903	0.9733
Document (p: 263, n: 827)	Hybrid	0.9924	0.9927	0.9775	0.9976	0.9849 0.97, 0.99)
	BERT	0.9696	0.9927	0.977	0.9903	0.9733 (0.96, 0.99)
	ClinicalBERT	0.9544	0.9867	0.958	0.9855	0.9562 (0.94, 0.97)
	Rule	0.9468	0.9637	0.8925	0.9827	0.9188 (0.9, 0.94)
	CNN	0.9734	0.9166	0.7877	0.9908	0.8707 (0.83, 0.89)
	Bi-LSTM	0.9202	0.994	0.9798	0.9751	0.949 (0.93, 0.97)
Patient (p: 32, n: 28)	Hybrid	0.9688	0.9286	0.9394	0.9630	0.9538 (0.91, 1.0)
	BERT	0.9688	0.8929	0.9118	0.9615	0.9394 (0.87, 0.99)
	ClinicalBERT	0.9688	0.8929	0.9118	0.9615	0.9394 (0.87, 0.99)
	Rule	0.9375	0.8571	0.8824	0.9231	0.9091 (0.82, 0.98)
	CNN	0.9688	0.7500	0.8158	0.9545	0.8857 (0.8, 0.96)
	Bi-LSTM	0.9375	0.9643	0.9677	0.9310	0.9524 (0.9, 1.0)
Level	Model	Sensitivity	Specificity	PPV	NPV	f1-score(95% CI‡)
Sentence (p=1073, n=64312†)	Hybrid	0.9543	0.9998	0.9884	0.9992	0.9711 (0.96, 0.98)
	BERT	0.9534	0.9995	0.9724	0.9992	0.9628 (0.95, 0.97)
	Rule	0.8509	0.9985	0.9040	0.9975	0.8766 (0.86, 0.89)
	CNN	0.9422	0.9965	0.8220	0.9990	0.8780 (0.86, 0.89)
	Bi-LSTM	0.8779	0.9993	0.9554	0.9979	0.9150 (0.9, 0.93)
Document (p=263, n=827†)	Hybrid	0.9924	0.9927	0.9775	0.9976	0.9849 0.97, 0.99)
	BERT	0.9696	0.9927	0.9770	0.9903	0.9733 (0.96, 0.99)
	Rule	0.9468	0.9637	0.8925	0.9827	0.9188 (0.9, 0.94)
	CNN	0.9734	0.9166	0.7877	0.9908	0.8707 (0.83, 0.89)
	Bi-LSTM	0.9202	0.994	0.9798	0.9751	0.949 (0.93, 0.97)
Patient (p=32, n=28†)	Hybrid	0.9688	0.9286	0.9394	0.9630	0.9538 (0.91, 1.0)
	BERT	0.9688	0.8929	0.9118	0.9615	0.9394 (0.87, 0.99)
	Rule	0.9375	0.8571	0.8824	0.9231	0.9091 (0.82, 0.98)
	CNN	0.9688	0.7500	0.8158	0.9545	0.8857 (0.8, 0.96)
	Bi-LSTM	0.9375	0.9643	0.9677	0.9310	0.9524 (0.9, 1.0),

^†p: positive fall event, n: negative fall event;

^‡bootstrap confidence interval

Analysis of Semantic Categories of Incorrect Fall Detection

We analyzed fall detection errors at the sentence level based on the contextual and semantic categories shown in Table 2. Figure 2 shows errors in different categories for each model. In general, machine learning-based approaches could correctly identify challenging fall events that required context understanding better than a rule-based approach. A rule-based model relatively had more error in negation but less in fall risk than the BERT. A hybrid model (BERT + post-hoc rules) was able to take advantage from both approaches – e.g., low error both in fall risk and negation. However, no single model yielded the best performance for all semantic categories. For example, a hybrid model performed better than Bi-LSTM in Rsk (fall risk) and Low (become a lower) but opposite in T (Time) and Pos (Position).

Figure 2.

Errors of fall detection in different semantic categories. Count: the number of false positive cases. Neg: Negation, Hx: Historical fall, Prv: Fall prevention, Rsk: Fall risk, L: Lack of context, Hmn: Homonyms, Cond: Pass into physical or mental condition, T: Pass at a certain time, Low: Become a lower level, Chc: To come by chance, Pos: To come by chance into a particular position, Plc: To have its proper place

Sentence Level Error Analysis

We analyzed challenging fall detection cases that were difficult to identify correctly. Table 4 lists common false negative (FN) and false positive (FP) examples. These cases either presented a complex syntactic sentence structure (e.g., #3, 10), expressions that may be caused by spelling or typographical errors (e.g., #4), or rare semantics patterns (e.g., #2, 8). A hybrid model was able to correctly detect true falls missed by BERT (#1) and rules (#2–5) and exclude incorrect fall cases miss-identified by BERT (#6, 8–10) and rules (#6–8). The false positive events (#6, 8, 9, 10) suggest some misclassification of the fall risk related expressions by the BERT model. The true positive rate for BERT model among the fall risk-related expressions was 0.74, indicating that the BERT model was able to distinguish more than half of the cases related to fall risk. We believe that a lack of relevant training data results in incorrect identification. However, 26% of the cases failed to be classified by the model, which provided the rationale for using a hybrid approach.

Table 4.

Challenging fall event cases (H: Hybrid, B: BERT, R: Rules, C: CNN, L: Bi-LSTM, TP: true positive, FN, false negative, FP: false positive).

True fall events prone to be missed	H	B	R	C	L
1. #2 fall from ladder on 12/3/2016 transfer from la crosse 12/5/16	TP	FN	TP	FN	FN
2. When he went down, his hands were behind him	FN	FN	FN	FN	FN
3. Continue cervical collar for now, however, given that it was ground level fall with negative c-spine CT will discuss with dr about removal of cervical collar.	TP	TP	FN	FN	FN
4. Fallin in parking lot of bowling lanes	TP	TP	FN	FN	FN
5. Denies any dizziness or light-headedness at any point, including preceding the two recent falls.	TP	TP	FN	FN	FN
False fall events prone to be considered as falls	H	B	R	C	L
6. Patient presents for evaluation of fall, while walking or running, landing on hard surface, landing on left side.	TN	FP	FP	FP	FP
7. These will likely fall off a few weeks after application	TN	TN	FP	FP	FP
8. Patient remains at fall risk due to right hemiplegia, right hip pain, and weight bearing as tolerated restrictions	TN	FP	TN	TN	FP
9. He was also somewhat impulsive, quickly getting up out of his wheelchair, and he tried to step over the edge with his right foot, which put him at high risk of falling.	TN	FP	TN	TN	TN
True fall events prone to miss	H	B	R	C	L
10. #2 fall from ladder on 12/3/2016 transfer from la crosse 12/5/16	TP	FN	TP	FN	FN
11. When he went down, his hands were behind him	FN	FN	FN	FN	FN
12. Continue cervical collar for now, however, given that it was ground level fall with negative c-spine CT will discuss with dr about removal of cervical collar.	TP	TP	FN	FN	FN
13. Fallin in parking lot of bowling lanes	TP	TP	FN	FN	FN
14. Denies any dizziness or light-headedness at any point, including preceding the two recent falls.	TP	TP	FN	FN	FN
False fall events prone to be considered as falls	H	B	R	C	L
15. Patient presents for evaluation of fall, while walking or running, landing on hard surface, landing on left side.	TN	FP	FP	FP	FP
16. These will likely fall off a few weeks after application	TN	TN	FP	FP	FP
17. Patient presents for evaluation of fall, while walking or running, landing on hard surface, landing on left side.	TN	FP	FP	FP	FP
18. patient remains at fall risk due to right hemiplegia, right hip pain, and weight bearing as tolerated restrictions	TN	FP	TN	TN	FP
19. he was also somewhat impulsive, quickly getting up out of his wheelchair, and he tried to step over the edge with his right foot, which put him at high risk of falling.	TN	FP	TN	TN	TN

To illustrate the ability to handle complex syntactic and semantic expressions from real-world EHRs data, we summarized the challenging cases correctly identified by the hybrid model) in Table 5. Based on our results, machine learning models are able identify fall events without a word ‘fall’ if there were enough relevant training data. However, events without keyword ‘fall’ are rare and thus more likely to be missed but the ML models.

Table 5.

Examples of challenging cases that are correctly identified by the hybrid model

Category	False Fall Events
Homonym / Fall risk and prevention	She falls asleep within 15 minutes Lives in Cannon falls You have requested information on fall prevention. Patient declines further evaluation of their fall risk.
Ambiguous Context	The patient does clearly describe difficulty with balance and a propensity to fall to the right side This will help decrease the number and severity of falls A cardiac cause of syncope cannot be completely excluded Patient has no apparent pain or injury aside from some vague right knee pain, but it is not known if she fell to floor, or had a syncopal event? Avoid falls related to diphenhydramine component
	True Fall Events
Rare Expression	He remembers waking up on the ground Landing on a small concrete structure and then flipped backwards Backwards hitting the back of his head I was wearing my satin gown and I slipped off the edge of the bed and hit the table She says this happened after tripping on a rug at home
Complex Sentence Structure	Mucous membranes moist, Patient complains of fall/head and knee injury, 68 yo male presents accompanied by wife after fallin in parking lot of bowling lanes He stated that patient had been evaluated for his 10 foot fall from his trauma team and found to have no additional injuries aside from his right intertrochanteric hip fracture
Negation	Denies any dizziness or light-headedness at any point, including preceding the two recent falls. She was not using this at the time of her fall when she got up quickly to help her husband He continues to be able to ambulate without an aid and denies any falls since his initial fall

DISCUSSION

In this study, we developed a hybrid model to identify fall events from the EHR free text. The evaluation of the hybrid approach against other approaches demonstrated excellent performance in identifying fall occurrences that are typically described in patient clinical notes. The ability to accurately detect true fall events from EHRs is largely dependent on the context understanding of clinical narratives. Leveraging the advantage of context-aware language models and domain knowledge represented by heuristic rules demonstrated the ability to handle complex syntactic and semantic fall expressions from real-world EHR data compared with other methods. Our hybrid model was able to identify indirect fall expressions and negated concepts reasonably well as in Table 5. For example, “waking up on the ground” has no direct fall indicative keywords; “denies any falls” indicate a false fall. However, the hybrid model correctly identified both expressions. The BERT model incorrectly identified some explicit fall events, and this might be due to lack of appropriate training data. The hybrid model using post-hoc heuristic rules combined with the BERT model was able to fix those misclassified cases, showing the potential to further improve the performance. We applied the ClinicalBERT(32) (0.9471 for sentence, 0.9562 for document, and 0.9394 for patient level) to evaluate the effect of pretraining on clinical notes; however, we did not observe any performance improvement over the BERT. This may be due to the different nature of clinical notes, especially fall event description, in our institution.

There are noticeable variations in fall events as shown in Table 2 and 5. We found that no single model outperformed all other models in all fall event categories. The hybrid model demonstrated similar learning patterns with the two other statistical machine learning models with a better ability to handle complex sentence structure and negation patterns. Although the rule-based approach yielded lower performance (sentence level f1-score) than the others, its consistency and customizability to handle explicit fall events cannot be ignored. Despite the overall promising performance of BERT, we found a handful of misclassified explicit fall events that could be fixed by rules. For example, a very common documentation pattern by Mayo Clinic clinicians is the use of “#” to document important diagnostic findings or problem lists (e.g., “# falls in 30 seconds”). Another example is the section information (e.g., section name + documentation structure patterns + fall expressions). The hybrid model was able to fix these issues and further improve the performance over the BERT model. In addition, we believe that the hybrid model can help customize fall detection in EHR data when the model is used in different institutions.

Despite the high performance in fall detection, our study has several limitations. The model may behave differently on the EHR data in other institutions due to heterogeneity of clinical documentation. Our study cohort consists of older, less racially or ethnically diverse, and more affluent than the general population of the United States. Other populations may have different care-seeking behaviors and health care providers may document falls differently. In the future, we plan to refine and deploy our model on EHR data from other institutions to evaluate the model portability.. A word embedding pretrained on a given institution may has a potential to improve the performance of fall detection. Transfer learning techniques may be considered to augment the learning process in different EHR data, but nonetheless needs to be paired with rigorous evaluation to ensure site-specific heterogeneity, and population variation.

CONCLUSION

In this study, we demonstrated that a context-aware language model (BERT) can be used to accurately detect challenging fall events from EHR free text that are difficult to be captured by simple methods. In addition, the hybrid model coupling with post-hoc rules allowed customizability to overcome the weakness of the BERT model and further enhance the performance, rectifying incorrect fall detection determined by the BERT model.

Summary Table.

Known on the topic	Newly added to our knowledge
Falls are a leading cause of unintentional injury among older adults EHR documents contain many fall events, not captured by ICD-9/10 codes The word “fall” is a homonym requiring context understanding to correctly detect complex true falls A context-aware language model has yielded high performance in other complex clinical NLP tasks	A context-aware language model is able to accurately capture complex fall events that require context understanding A hybrid model incorporated with post-hoc rules offer customizability to fix errors from a context-aware model and further improved the fall detection performance