Using AI to identify in-hospital falls: a comparative analysis with incident reports and manual chart review

Abstract

Introduction

Falls remain a problem in hospitalised patients, especially in older adults. In-hospital falls occasionally lead to injuries. Currently, the rate of falls in hospitals is monitored using incident reports, which often leads to underreporting. Recently, natural language processing (NLP) models gained interest. In this study, an NLP model (AI algorithm) to detect falls in nursing records was evaluated.

Methods

In this retrospective study conducted in a large teaching hospital in the Netherlands, hospitalised patients aged 18 years and older were included. Data were collected from free text in nursing records; all data were manually checked for fall incidents. For comparative analysis, the AI algorithm and incident reports were used. A descriptive comparison of the three different methods is provided by using absolute counts. For analysis, descriptive statistics and an area under the curve (AUC) were calculated. For time efficiency a comparative analysis was performed.

Results

76 fallers and 1547 non-fallers with a total of 30 615 nursing records were included. Using manual chart review, a total of 124 falls were identified. The AI algorithm showed good sensitivity (79.8%), good specificity (98.3%), excellent negative predictive value (NPV, 99.9%) and moderate positive predictive value (PPV, 16.4%). Incident reports had a sensitivity of 37.0% and a specificity of 99.9%, with a good PPV and NPV (97.9% and 99.7%, respectively). The AI algorithm had good discriminatory ability with an AUC of 0.891; incident reports had a moderate discriminatory ability with an AUC of 0.685. Utilising the algorithm, only 604 nursing records needed to be manually reviewed to identify 79.2% of all fall incidents, a higher detection rate than that achieved through incident reports (36.8% of all fall incidents). The semi-automated workflow using the AI algorithm requires significantly less time compared with incident reports, with a 92.6% reduction in time.

Conclusion

The AI algorithm has a good discriminatory ability to timely detect in-hospital falls. Using this technique, more falls are identified than by using incident reports, while the administrative burden of nursing personnel is reduced. Moreover, it will enable more effective, near real-time preventive measurements.

WHAT IS ALREADY KNOWN ON THIS TOPIC

• Currently, falls are monitored using incident reports, which leads to underreporting and is a time-consuming method. NLP algorithms have the ability to detect in-hospital falls.

WHAT THIS STUDY ADDS

• The AI algorithm has a good discriminatory ability, and using this algorithm, time spent on detecting in-hospital falls is reduced.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

• Using the AI algorithm more, in-hospital falls can be detected near real time. Enabling the possibility to improve fall-preventative measurements during a hospital stay and to improve quality of care.

Introduction

Falls remain a common problem in hospitalised patients, especially in older adults. In a Dutch study, fall risk was 5% for hospitalised patients aged 70 years and older.¹ In the Catharina Hospital in Eindhoven, the incidence of falls is around 4%. In-hospital falls occasionally lead to injuries. While most injuries are minor, such as lacerations, sprains and strains, 4.3%–11% of falls lead to major injuries, including fractures and subdural hematomas.2,6

Currently, the number of falls in hospitals is monitored using incident reports. However, numerous barriers have been identified for using incident reports, including lack of feedback, lack of time, fear of negative consequences and the assumption that an incident was too trivial to report.^{7 8} For falls specifically, studies demonstrated that non-injurious falls may often not be recorded because of time pressure on staff, discrepancies in the definition of falls or fear of criticism and blame.^{9 10} One study investigated the characteristics of in-hospital falls not reported in incident reports, showing that falls of male patients, falls that occurred during holiday periods, first falls, non-witnessed falls and falls observed by physicians were more likely to not be reported.¹¹ These disadvantages contribute to underreporting. One additional disadvantage of using incident reports lies in the post-reporting process. Each incident report is evaluated by an incident review committee, which assesses trends and patterns. When necessary, follow-up questions are asked to the individual who submitted the report for further clarification. As a result, in addition to the time spent completing the initial report, additional time is required for the processing and review of the incident, further increasing the overall time spent on incident reports.

Because of underreporting in incident reports, other ways of reporting in-hospital falls are needed. Electronic health records contain a considerable amount of unstructured data, such as progress notes. These notes may contain key patient information that cannot be found in structured data.¹² Manual chart review has been considered the gold standard for detecting adverse events. However, this method is time-consuming and imperfect.¹³ Recently, NLP models gained interest for the analysis of free text, and several algorithms have been developed to detect falls without the need for time-consuming efforts such as incident reports.14,18 These algorithms all showed good discriminatory ability in detecting falls.

An NLP-based AI algorithm was previously developed in our hospital to detect patient fall incidents from free-text nursing records. This model is based on RobBERT (Robustly Optimized Bidirectional Encoder Representations from Transformers), a fine-tuned version of a pre-trained Dutch language model, specifically adapted for this classification task. During its initial development, the model achieved an F1-score of 0.81 and an accuracy of 0.91.¹⁹ Since its original implementation, the algorithm has undergone further refinements and improvements. The model was developed using the Simple Transformers library, which builds on the Transformers framework from Hugging Face. All development and training were performed within the hospital’s environment using Python 3 (Python Software Foundation, Wilmington, DE, USA) on the HPE Ezmeral Runtime Enterprise platform containing one NVIDIA Tesla T4 GPU. The objective of this study is to compare the discriminatory ability for in-hospital falls of the AI algorithm, incident reports and the gold standard; manual chart review of nursing records. Furthermore, this study aims to assess whether the AI algorithm can offer a more time-efficient approach for the identification of in-hospital falls than using incident reports.

Methods

Study design

This is a single-centre, retrospective study that used historical data from daily nursing reports in electronic health records and incident reports. It was conducted in the Catharina Hospital, a large teaching hospital in Eindhoven, The Netherlands. The hospital has 690 beds and facilitates regional as well as specialist care for patients from within and outside the region.

Patient selection and study population

All patients aged 18 years and older, admitted to the departments of geriatrics, neurology, orthopaedics, surgery and internal medicine from 1 June 2022 and 5 October 2022 were included. Patients who indicated in their online hospital portal that they did not consent to the use of their electronic health record (EHR) data for scientific research were automatically excluded, since their records could not be accessed for research purposes.

Patient and public involvement

Patients or the public were not involved in the design or conduct or reporting or dissemination plans of this study.

Data collection

For this study, two databases were created. The first database comprised nursing records extracted from the EHR system (HiX, V6.3, ChipSoft B.V.). The first database consisted of free-text nursing notes extracted from the EHR database using standardised SQL scripts. Each record represented a single patient during one nursing shift. The database contained all nursing records from patients admitted to the departments of geriatrics, neurology, orthopaedics, surgery and internal medicine from 1 June 2022 and 5 October 2022. The second database contained all incident reports related to fall events during the same period. Incident reports were retrieved from the institutional incident reporting system in Excel format. Data extraction was performed by a single researcher (LL) affiliated with the Department of AI Expertise. Subsequently, data collection, data preprocessing and validation were conducted by three researchers (WL, MG and BH). The entire process of data extraction, compilation and validation was completed within approximately 3–4 weeks. Due to the nature of the dataset, there were no missing data. Furthermore, the average time required for generating incident reports and conducting manual chart reviews was measured throughout the study.

Data analysis

All data were manually checked for fall incidents and labelled according to the definition of the WHO: ‘inadvertently coming to rest on the ground, floor or other lower level, excluding intentional change in position to rest in furniture, wall or other objects’.²⁰ All nursing records were labelled as a 0 (no fall incident), a 1 (fall incident) or a 2 (doubtful). A doubtful fall is defined as an event where the circumstances do not allow for a definitive classification as a fall, for example, when a patient is found lying on the ground without any witnessed fall or clear evidence of the mechanism leading to the position. All doubtful cases were checked by another researcher (MG or CvdL). The database was later validated by an independent second researcher through manually checking a sample of 10% of the nursing records (BH). Data were preprocessed by removing diaeresis, special characters (characters that are not alphabetic or numeric), extra spaces and single letters. In addition, we excluded the single letters from the ABCDE and SCEGS framework used by nurses as a framework in writing their reports. Manual identification of fall incidents was considered the gold standard. For comparative analysis, a previously developed AI algorithm that was designed to identify fall incidents from free text in nursing records was used.¹⁹ Additionally, incident reports related to fall incidents were used to identify patients that experienced a fall. Incident reports were manually linked to the corresponding nursing record. The efficacy of all three methods for identifying fall incidents was assessed to determine the optimal performing method. Data analysis was conducted in consecutive studies between November 2022 and January 2025.

Statistical analysis

Baseline characteristics

Continuous variables such as age and length of stay were presented as medians with IQRs. Differences in continuous variables were calculated using the Mann-Whitney U test. For categorical variables such as sex, differences were assessed using the chi-square test.

Outcome measure

For validation of the prior manual assessment of nursing records, a 10% sample will be re-scored. Subsequently, a Cohen’s kappa will be calculated, using a threshold of 0.8. A 10% sample size was chosen since this is a common size for calculating the Cohen’s kappa for machine learning.^{21 22}

Fall incidents were described both as an absolute number and as a percentage. Additionally, a descriptive comparison of the three different methods will be provided by using absolute counts. This research uses descriptive statistics, including sensitivity, specificity, positive and negative predictive value (PPV and NPV) and an area under the curve (AUC). For visualisation of the results, a Venn diagram was created.

A comparative analysis was performed to evaluate the time efficiency of incident reports and the AI algorithm. For the comparative analysis, an average time of 15 min for generating an incident report was selected, based on clinical expertise and established practice.

All text mining procedures and statistical analyses were performed using R, V4.3.1 (2023–06-16 ucrt) ‘Beagle Scouts’.²³

A significance threshold of p<0.05 was used to determine statistical significance.

Ethical statement

The study has been reviewed by the Medical Ethics Review Committee (MECU) and has been classified as not being subject to the Medical Research Involving Human Subjects Act (WMO), thereby receiving a non-WMO declaration (reg. nr. W24.142). The study was conducted in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration and its later amendments.

Informed consent was not requested because the study posed no safety risks, did not affect patient treatment and involved anonymised data that ensured participants’ privacy, serving the public interest without causing any burden.

Results

Baseline characteristics

Baseline characteristics of the study participants are presented in table 1. 76 fallers and 1547 non-fallers were included. Median age was higher in patients who experienced a fall (81 years (IQR: 14 years) versus 71 years (IQR: 28 years), p<0.0001). Length of stay in fallers was 15 days (IQR: 18.5 days), whereas length of stay in non-fallers was 3 days (IQR: 6 days, p<0.0001).

Table 1. Baseline characteristics of included patients.

Baseline characteristics	Fallers (n=76)	Non-fallers (n=1547)	P value
Sex, n (%)*
Male	46 (60.5)	808 (52.2)	0.16
Age in years, median (IQR) †	81 (14)	71 (28)	0.0001
Length of stay in days, median (IQR)†	15 (18.5)	3 (6)	0.0001

Data are presented as numbers (percentage) or medians (IQR).

^*Chi-square test.

^†Mann-Whitney U test.

Number of falls

A total of 30 615 nursing records were included in the study. Using manual chart review, a total of 124 falls in 76 patients were identified. Fall incidence rates varied across specialties: 13 patients (1.9%) in surgery, 14 patients (4.6%) in internal medicine, 21 patients (17.4%) in geriatrics, 19 patients (5.3%) in neurology and 9 patients (5.4%) in orthopaedics. Some patients experienced more than one fall during their admission.

The Cohen’s kappa on the 10% sample was 1.0, indicating perfect agreement. A CI is not defined due to lack of disagreement. The AI algorithm marked 604 fall incidents; incident reports marked 47 falls.

Sensitivity of the AI algorithm was 0.80; specificity was 0.98. The AI algorithm showed a good NPV (1.00) and moderate PPV (0.16). Incident reports had a sensitivity of 0.37 and a specificity of 1.00, with a good PPV of 0.98 and an NPV of 1.00. Results are shown in table 2. The AI algorithm had good discriminatory ability with an AUC of 0.891 (95% CI 0.855 to 0.926); incident reports had a moderate discriminatory ability with an AUC of 0.685 (95% CI 0.643 to 0.728). The Receiver Operating Characteristic (ROC) curve is shown in figure 1.

Table 2. Comparison of modalities for identifying falls in nursing records.

Method	True positive, n	False positive, n	True negative, n	False negative, n	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
AI algorithm	99	505	29 987	25	0.80 (0.72 to 0.87)	0.98 (0.98 to 0.98)	0.16 (0.14 to 0.20)	1.00 (1.00 to 1.00)
Incident reports	46	1	30 491	78	0.37 (0.29 to 0.46)	1.00 (1.00 to 1.00)	0.98 (0.89 to 1.00)	1.00 (1.00 to 1.00)

NPV, negative predictive value; PPV, positive predictive value.

Figure 1. ROC curve for AI algorithm and incident reports. AUC, area under the curve.

Using manual chart review, 18 falls were identified that were not identified by the AI algorithm or incident reports. Using the AI algorithm, 504 falls were identified that were not identified using manual chart review or incident reports. All falls identified using incident reports were also identified using manual chart review or by the AI algorithm. Only one fall was identified by both incident reports and the AI algorithm that had not been identified by manual chart review. Overlap of all modalities is shown in figure 2. When using the AI algorithm, only a small proportion of all nursing records (604 of 30 615, thus 2.0%) should be checked manually to find 79.2% of all fall incidents. Using incident reports, only 36.8% of fall incidents were found.

Figure 2. Overlap between modalities for identifying falls in nursing records.

Time comparison between modalities

Time required to complete all 124 incident reports is 1860 min or 31 hours (15 min per incident report). Hereby, we did not include time needed for the post-reporting process. On average 264 nursing reports can be read per hour. A semi-automated workflow, where the AI algorithm screens all nursing records and human reviewers validate the 604 records flagged as positive, requires only about 137 min (approximately 2 hours) for manual review. The comparison between incident reports and the semi-automated workflow demonstrates a time savings of roughly 29 hours, corresponding to a 92.6% reduction of time spent on identifying in-hospital falls.

Discussion

This study was conducted to compare the detecting abilities of three methods to detect in-hospital falls incident reports, an AI algorithm and manual chart review. In addition, compare time efficiency between the AI algorithm and incident reports.

The AI algorithm showed a good sensitivity and specificity with an AUC of 0.891. However, the PPV of the AI algorithm was low because of high false positive rates. The incident reports showed moderate sensitivity and good specificity, with an AUC of 0.685. Incident reports had a good PPV and NPV; however, 78 of 124 fall incidents were missed.

When using the AI algorithm, only a small proportion of all nursing records should be checked manually to find 79.2% of all fall incidents, while using incident reports only 36.8% of fall incidents were found. Moreover, a time reduction of 92.6% is achieved using the AI algorithm in a semi-automated workflow.

To the best of our knowledge, three prior studies have developed algorithms aimed at identifying falls in clinical settings. Dolci et al developed an algorithm for fall detection using physicians’ and nurses’ progress notes, physicians’ discharge summaries and nurses’ anamneses. Their algorithm demonstrated a PPV of 47%, due to a relatively high number of false positives. However, it successfully identified all 15 fall incidents (100% sensitivity). They suggested that employing their algorithm could reduce the time spent detecting fall incidents by 89%.¹⁴ Toyabe et al explored the use of an NLP algorithm to detect falls from progress notes. While the PPV was 6%, the algorithm exhibited 100% sensitivity. This indicated that only 6% of progress notes required manual review, compared with 100%, thus significantly reducing the time required from medical personnel.¹⁷ Shiner et al conducted a pilot study in which they developed an NLP model to detect falls among inpatients using progress notes. Their model demonstrated excellent specificity (97%) in comparison to manual chart review, but its sensitivity was low (44%) due to a high number of false negatives.¹⁸

In contrast with the results of Dolci et al, our algorithm had a lower sensitivity and lower PPV. One possible explanation for this is that our algorithm only used nursing records, and their study used more sources of narrative information in the electronic health records. Specificity and NPV and time reduction for identifying fall incidents were comparable for both algorithms. Compared with the algorithm of Toyabe et al our algorithm had a higher PPV but a lower sensitivity. This indicates that our study produced fewer false positives, although it did not identify all fall incidents, in contrast to the algorithm developed by Toyabe et al. The studies of Dolci and Toyabe compared incident reports with manual chart review, in which 67% and 65% of falls were found using incident reports, which is higher than the 36.8% that we found in our study. A possible explanation for this can be lack of time or lack of effort for nurses in our hospital to write incident reports. Compared with the NLP model of Shiner et al, our model performed better, with a higher sensitivity and comparable specificity.

Our study has several limitations. First, the gold standard was manual chart review, which is considered the gold standard for identifying adverse events.¹³ Despite all nursing records being reviewed by a single researcher, with a 10% random sample validated by a second researcher yielding a Cohen’s kappa of 1, falls may still have been missed due to factors such as time constraints, researcher fatigue or other limitations.¹⁴ Second, we did not incorporate all available information from the electronic health records; instead, we focused solely on free-text nursing records to identify falls. Nursing records were chosen because they are the most comprehensive progress notes in the electronic health records. Nurses spend most time at the patients’ bedside and write narrative notes with a lot of relevant information on the patients’ status.²⁴ Furthermore, both AI algorithm and manual chart review outcomes are dependent on the quality of the nursing records.²⁵ There are many challenges for automated detection of falls in free text, such as the use of abbreviations, spelling errors and ambiguity in vocabulary.^{26 27} If a fall is not clearly documented in the nursing records, both the AI algorithm and manual chart review may fail to detect the fall incident.

We believe that our study has several strengths. We conducted a methodologically robust study with large sample sizes (30 615 nursing records) to include as many fall incidents as possible. Second, although manual chart review has its limitations, it is the gold standard for identifying adverse events. In our study, there was perfect agreement (Cohen’s kappa of 1) for identifying fall incidents, and all doubtful cases were checked by a second researcher. Moreover, we included data from several departments, creating a heterogenous study group with departments with both high and low fall rates.

An advantage of the algorithm over incident reports is its ability to provide timely detection of falls on a larger scale. This does not replace existing prevention measures but adds an extra layer of quality assurance by enabling continuous monitoring and timely retrospective follow-up at the level of nurse managers and hospital leadership, thereby supporting patient safety and care quality. Moreover, utilising the algorithm, only 604 nursing records needed to be manually reviewed to identify 79.2% of all fall incidents, a higher detection rate than that achieved through incident reports (36.8% of all fall incidents). The manual review of 604 nursing records in the semi-automated workflow requires significantly less time compared with processing 124 incident reports, with a 92.6% reduction in time. Consequently, this could result in a reduction of the administrative burden on nursing personnel, should incident reports no longer be required to identify in-hospital falls. The 92.6% reduction is likely an underestimation, as it was calculated based on an assumption of 15 min required to file an incident report. This calculation does not account for the additional time spent on the evaluation of the report by an incident review committee. Most likely, the use of the algorithm would result in even greater time savings, since incident review committees can focus on aggregated data. Additionally, this algorithm has the potential to identify larger numbers of patients who have experienced falls, which could be valuable for future research and further improvements in healthcare quality.

For future research, we will continue to refine the algorithm to enhance its sensitivity while maintaining its PPV. Moreover, we are currently developing a dashboard in our hospital using the AI algorithm and the semi-automated workflow to monitor all fall incidents and collect all relevant patient data. This will allow for the systematic analysis and evaluation of fall events, improving fall prevention strategies.

In conclusion, the AI algorithm has a good discriminatory ability to detect in-hospital falls near real-time. Using this technique, more falls are identified than by using incident reports, while the administrative burden of nursing personnel is reduced. Furthermore, this algorithm has the potential to identify larger numbers of patients who have experienced falls, offering valuable insights for future research and advancements in healthcare quality. A better understanding of the true incidence of falls will enable more effective, near-real-time preventative measurements, ultimately reducing fall occurrences and improving overall care quality.

Data availability statement

No data are available.