Abstract
Background
Falls in hospitals are the most common risk factor that affects the safety of inpatients and can result in severe harm. Therefore, preventing falls is one of the most important areas of risk management for health care organizations. However, existing methods for predicting falls are laborious and costly.
Objective
The objective of this study is to verify whether hospital inpatient falls can be predicted through the analysis of a single input—unstructured nursing records obtained from Japanese electronic medical records (EMRs)—using a natural language processing (NLP) algorithm and machine learning.
Methods
The nursing records of 335 fallers and 408 nonfallers for a 12-month period were extracted from the EMRs of an acute care hospital and randomly divided into a learning data set and test data set. The former data set was subjected to NLP and machine learning to extract morphemes that contributed to separating fallers from nonfallers to construct a model for predicting falls. Then, the latter data set was used to determine the predictive value of the model using receiver operating characteristic (ROC) analysis.
Results
The prediction of falls using the test data set showed high accuracy, with an area under the ROC curve, sensitivity, specificity, and odds ratio of mean 0.834 (SD 0.005), mean 0.769 (SD 0.013), mean 0.785 (SD 0.020), and mean 12.27 (SD 1.11) for five independent experiments, respectively. The morphemes incorporated into the final model included many words closely related to known risk factors for falls, such as the use of psychotropic drugs, state of consciousness, and mobility, thereby demonstrating that an NLP algorithm combined with machine learning can effectively extract risk factors for falls from nursing records.
Conclusions
We successfully established that falls among hospital inpatients can be predicted by analyzing nursing records using an NLP algorithm and machine learning. Therefore, it may be possible to develop a fall risk monitoring system that analyzes nursing records daily and alerts health care professionals when the fall risk of an inpatient is increased.
Keywords: fall, risk factor, prediction, nursing record, natural language processing, machine learning
Introduction
Background
Falls are the most common risk factor affecting the safety of hospital inpatients. They often result in a severe injury, such as a femoral fracture or head trauma, which can be life-threatening or affect the patient’s quality of life. After analyzing data from 1263 hospitals, Bouldin et al [1] reported that the rate of falls in the United States was 3.56 per 1000 patient-days during a 27-month study period and that 26.1% of these falls (0.93 per 1000 patient-days) resulted in injury. In Japan, a 2016 report from the Japan Federation of Democratic Medical Institutions indicated that the rates of falls and falls causing injury were 4.40 and 0.29 per 1000 patient-days, respectively [2]. Therefore, the prevention of falls is one of the most important areas of risk management for health care organizations. The Joint Commission, which is involved in the accreditation and certification of US health care organizations and programs, has strongly recommended taking strategic action for fall prevention, including the use of a standardized assessment tool to identify risks [3].
Prior Work
A variety of methods have been developed to predict the risk of falls for hospital inpatients, such as the Morse Fall Scale [4], St Thomas’s Risk Assessment Tool in Falling Elderly Inpatients (STRATIFY) [5], Hendrich Fall Risk Model (HFRM) [6], and the revised Hendrich II Fall Risk Model [7]. All these methods have been used and evaluated [8-11]. However, such risk assessment methods invariably involve time-consuming processes, such as interviews, observation, and intervention [4-7], which interrupt the work of health care professionals, and the additional workload contributes to an increase in medical costs.
Moreover, several studies, including systematic reviews, have demonstrated that no single intervention, including patient tags and movement sensors, efficiently reduces fall incidents in any setting, whereas multifactorial assessment linked to appropriate interventions is successful [12-16]. However, no common combination of risk factors was discovered in these studies [17], indicating that health care professionals still need to conduct multiple assessments for each risk factor in daily practice, including motor function, continence, mental state, and medication. Thus, a less laborious assessment tool that can predict the risk of falls with high precision without initial intervention is desirable.
With recent advances in information technology, several groups have attempted to apply natural language processing (NLP) to text analysis of electronic medical records (EMRs) to achieve the early diagnosis of conditions such as peripheral arterial disease [18], asthma [19], and multiple sclerosis [20]. In these studies, NLP was used to find specific words or phrases in a predefined dictionary that described the symptoms or signs of each disease. Following these studies, we apply artificial intelligence to EMRs to analyze the risk of falls.
Goal of This Study
Our primary objective is to determine whether hospital inpatient falls can be predicted through the analysis of the unstructured text of hospital nursing records in Japanese EMRs using an NLP algorithm and machine learning. In nursing records, nurses write daily information about a patient’s nursing care, the patient’s response, and other events or factors that may affect the patient’s well-being based on observation and experience [21]. Thus, nursing records contain valuable information for clinical practice but have not been widely used for any type of risk assessment because they require a technique, such as NLP, to analyze and extract meanings of interest from free text or unstructured documents.
We constructed a predictive model to assess the linguistic differences between the nursing records of fallers and nonfallers using our proprietary algorithm applying NLP in combination with machine learning and evaluated its performance using receiver operating characteristic (ROC) analysis. The advantages of our approach are that it allows us to assess various risk factors from a single input (nursing records), and it is less laborious and costly than previous approaches because it does not require additional observation or interviews.
Methods
Study Design
We used a case-control study because of the easy availability of nursing records in EMRs, limited computational capacity, and low rate of falls among inpatients. Because our main objective is to verify the feasibility of using nursing records to predict falls, we used only one hospital and one year of data to limit the cost and time of data extraction. For this study, we considered NTT Medical Center Tokyo (Tokyo, Japan), which is an acute hospital with 606 beds and an average hospital stay of 11.4 days. The Institutional Review Board of the hospital approved the study (Approval #15-267, June 25, 2015). The study period was from July 2014 to July 2015.
Data
Among 18,045 inpatients during the study period, 335 patients with one or more fall incidents (fallers) were identified from the incident reports of the hospital. As a control group, 408 patients without falls (nonfallers) were randomly selected. More nonfallers than fallers were chosen as a contingency if extracted data had to be discarded for unexpected reasons. Data were not discarded; therefore, all usable data were considered in the analysis. We are aware that the substantial difference between the total number of fallers and nonfallers can affect machine learning; however, we believe this is mitigated by the use of a case-control study, which is often used in rare medical cases such as rare diseases.
Data on the two groups of patients were extracted from the EMR system by the EMR vendor and provided to the researchers after anonymization. The researchers constructed a case data set (fallers) and control data set (nonfallers). The nursing records were written in the EMR once a day or more frequently as necessary by several nurses using the subjective, objective, assessment, and plan style or free description. These contained (1) patients’ statements, (2) observations of the nurses, (3) results of vital check and various assessments, (4) descriptions of medical treatment and administration of drugs (or plan for them), (5) messages to and from patients, and (6) any other comments by nurses. Some parts of (3) and (4) were entered as preset form data, and others were unstructured data. Several records for one patient made on the same day were integrated into one nursing record. Thus, 25,145 nursing records were obtained, which consisted of 18,912 nursing records for fallers and 6233 for nonfallers. The prevalence of falls was 2.61 falls per 1000 patient-days during the study period. The characteristics of the patients and nursing records are shown in Table 1.
Table 1.
Characteristics of the patients and nursing records.
| Characteristics | All patients | Fallers | Nonfallers | P valuea | |
| Patients, n (% of total) | 743 (100) | 335 (45.1) | 408 (54.9) | —b | |
| Gender, n (% of total) |
|
|
|
— | |
|
|
Female | 342 (100) | 156 (45.6) | 186 (54.4) |
|
|
|
Male | 401 (100) | 179 (44.6) | 222 (55.4) |
|
| Age (years), mean (SD) | 67.0 (17.1) | 73.3 (13.3) | 65.5 (18.1) | <.001 | |
| Nursing records, n | 25,145 | 18,912 | 6233 | — | |
| Nursing records per patient, mean (SD) | 45.3 (43.5) | 68.1 (49.1) | 26.6 (26.4) | <.001 | |
| Nursing record length,c mean (SD) | 5392.1 (4138.2) | 5628.4 (4202.6) | 4675.1 (3848.8) | <.001 | |
aWelch t test between fallers and nonfallers used.
bNot applicable.
cNumber of Japanese or Chinese characters.
The entire nursing record data set was divided into a learning data set and test data set by generating random numbers for patient identification numbers assigned after anonymization.
Data Exclusion
The nursing records that did not satisfy the criterion of more than 50 Japanese or Chinese characters were excluded during tokenization and vectorization. This was a requirement of the Concept Encoder, which is described subsequently.
Data Processing by Concept Encoder
A model was constructed to sort the nursing records into two groups (“risk” and “no risk”) from the learning data set. The probability of being categorized in the risk group, hereafter referred to as the risk probability, was calculated for each nursing record in the test data set using an in-house algorithm for NLP and machine learning called Concept Encoder (FRONTEO, Inc, Tokyo, Japan; will be published elsewhere), which was constructed on a Python platform.
Document and Word Embedding
Concept Encoder performs text analysis by defining the line vector obtained from the document-word matrix as a document vector. First, each document is decomposed into morphemes (the smallest meaningful units of a language) by morphological analysis using MeCab version 0.996 [22], and rules are applied to label each element at the morpheme level with a word. Morphemes that were not words were discarded before each element was labeled. Then the word labels are embedded in k-dimensional vector space [23-25]. Documents can also be embedded in the k-dimensional vector space by expanding the word-embedding method. Assuming that there are m documents and n words in all the nursing records used in the study, and they are embedded, these documents and words can be expressed as matrices D and W:
where each row vector of matrices D and W corresponds to m documents and n words, respectively, from the nursing records in the study.
It is well known that embedded vectors have interesting features, such as word analogy, and outperformed bag of words approaches in several linguistic tasks. These interesting features are retained after two matrices are multiplied because of the linearity of multiplication. For example, if 
for two row vectors in W, then the inner product with d, which is a row vector in matrix D, holds 
. Expanding this to the word analogy, if 
, where 
holds for four row vectors in W, then 
holds for any row vector d in D. Hence, the product of these two matrices generates the DW matrix, which is a document-word matrix that also has these interesting features:
As seen in previous studies [23-25], neural networks have been used to calculate D and W, and if the number of documents becomes large, then the calculation of these matrices is computationally intensive. Hence, the words included in the neural embedding are restricted to the top 1000 most popular words that occur in the documents in the learning data set, hereafter referred to as the “top 1000 words.”
In this study, for W, the skip-gram with the negative sampling algorithm was used. The hyperparameter number of negative sampling was set to 5, and the number of dimensions for W was set to 300. For D, the distributed bag of words version of the paragraph vector (PV-DBOW) was used with the same negative sampling and embedding dimensions as W. After obtaining W and D, the DW matrix was calculated using matrix multiplication.
Construction of the Fall Prediction Model
For the construction of the fall prediction model, the DW matrix was derived from all documents and words in the learning data set. By attaching tags of 1 (for fallers) and 0 (for nonfallers) to each document, each line vector of the DW matrix (which corresponds to m documents) was associated with a tag of 0 or 1. Each word was subjected to adaptive weighting for optimum separation between fallers and nonfallers using a logistic regression model, and the weighted parameters were estimated by the Markov chain Monte Carlo (MCMC) method with a normal distribution as the prior distribution of weights. For the MCMC approach, the weighted parameters were estimated using posterior distributions, and uncertainty of the estimate was also considered by observing the distribution. The weighted parameters thus obtained were used as the fall prediction model to evaluate the test data set. Random bisection of the learning data set was conducted three times, and six models were constructed using the six bisected data sets. Because the sample size was not balanced between fallers and nonfallers, we used the synthetic minority oversampling technique in this step [26] by using the function of “imblearn.over_sampling.SMOTE” from the library [27] with the default setting and checked that samples for not majority class (“faller” or “imminent”) were resampled to be equal to those the major one in number. Morphemes that significantly contributed to the separation of the fallers and nonfallers in at least four of the six primary models (ie, “significant vocabulary”) were extracted and were used to construct the final model by the generation of the trimmed DW matrix followed by MCMC optimization.
Evaluation of Documents in the Test Data Set
For evaluation, documents in the test data set were tokenized to generate another matrix (hereafter called “DW for test”) using the top 1000 words followed by trimming it down using the significant vocabulary. The risk probability was calculated as the element-wise product of the corresponding line vector of the DW for test matrix and the final model. To assess the significance of differences, the Student t test was performed using R studio software (version 1.0.143).
Results
Analysis of the Data Set
Differences were observed between the groups of fallers and nonfallers for age, number of nursing records per patient (strongly correlated with the duration of hospitalization), and the length of nursing records (Table 1; P<.001 by Welch t test). The ratios of fallers and nonfallers also varied among some clinical divisions of the hospital, as shown in Table 2. However, matching for such factors was not performed because our primary aim was to determine whether it was possible to predict falls through comprehensive risk assessment using text analysis of nursing records regardless of risk factors already known or presumed from other information.
Table 2.
Number of inpatients per clinical division.
| Clinical division | Total (N=743), n | Fallers (n=335), n | Nonfallers (n=408), n |
| Gastroenterology | 107 | 51 | 56 |
| Surgery | 104 | 42 | 62 |
| Cardiology | 53 | 22 | 31 |
| Gynecology and obstetrics | 49 | 4 | 45 |
| Stroke unit | 44 | 27 | 17 |
| Orthopedic surgery | 41 | 23 | 18 |
| Respirology | 37 | 20 | 17 |
| Urology | 36 | 12 | 24 |
| Hematology | 32 | 27 | 5 |
| Neurosurgery | 31 | 19 | 12 |
| Psychiatry | 30 | 23 | 7 |
| Pain clinic | 27 | 10 | 17 |
| Otorhinolaryngology | 21 | 1 | 20 |
| Medical cooperation | 17 | 7 | 10 |
| Nephrology | 16 | 9 | 7 |
| Dermatology | 16 | 3 | 13 |
| Ophthalmology | 15 | 4 | 11 |
| Palliative care | 14 | 9 | 5 |
| Gamma knife center | 13 | 1 | 12 |
| Dentistry and oral surgery | 9 | 3 | 6 |
| General thoracic surgery | 8 | 4 | 4 |
| Neurology | 8 | 6 | 2 |
| Emergency medicine | 5 | 5 | 0 |
| Cardiovascular surgery | 4 | 2 | 2 |
| Endocrinology and metabolism | 3 | 0 | 3 |
| General medicine | 2 | 0 | 2 |
| Psychosomatic medicine | 1 | 1 | 0 |
Model to Predict Falls
The entire data set was divided into a learning data set and test data set as shown in Table 3. To construct a model to predict falls, tokenization and vectorization were performed on the learning data set. During this step, 12 nursing records (five for fallers and seven for nonfallers) that did not contain more than 50 Japanese or Chinese characters were excluded, leaving 9094 nursing records for fallers and 3513 nursing records for nonfallers. Using NLP and machine learning for the unstructured text of the learning data set, 378 morphemes that corresponded to significant vocabulary (ie, they contributed to separating fallers from nonfallers in at least four of the six primary models) were selected (a partial list is shown in Textbox 1). To construct the final model, 378 columns that corresponded to the selected morphemes were extracted from the 1000 columns of the DW matrix generated using the learning data set and were again subjected to optimization to separate fallers from nonfallers using the MCMC method. Using the final model, the probability of each nursing record in the test data set being in the risk category was evaluated next.
Table 3.
Characteristics of patients and nursing records in the learning data set and test data set for prediction of falls.
| Entire data set | Total | Fallers | Nonfallers | P valuea | ||
| Learning data set |
|
|
|
|
||
|
|
Patients, n (% of total) | 371 (100) | 167 (45.0) | 204 (55.0) | —b | |
|
|
Gender, n (% of total) |
|
|
|
— | |
|
|
|
Female | 159 (100) | 78 (49.1) | 81 (50.1) |
|
|
|
|
Male | 212 (100) | 89 (42.0) | 123 (58.0) |
|
|
|
Age (years), mean (SD) | 67.0 (17.0) | 73.4 (12.9) | 61.7 (18.1) | <.001 | |
|
|
Nursing records, n | 12,619 | 9099 | 3520 | — | |
|
|
Nursing records per patient, mean (SD) | 45.4 (41.9) | 66.4 (45.3) | 28.2 (29.3) | <.001 | |
|
|
Nursing record lengthc, mean (SD) | 4879.1 (2212.3) | 5559.4 (1961.9) | 4323.8 (2090.9) | <.001 | |
| Test data set |
|
|
|
|
||
|
|
Patients, n (% of total) | 372 (100) | 168 (45.2) | 204 (54.8) | — | |
|
|
Gender, n (% of total) |
|
|
|
— | |
|
|
|
Female | 183 (100) | 78 (42.6) | 105 (57.4) |
|
|
|
|
Male | 189 (100) | 90 (47.6) | 99 (52.4) |
|
|
|
Age (years), mean (SD) | 67.1 (17.1) | 73.2 (13.8) | 62.1 (18.1) | <.001 | |
|
|
Nursing records, n | 12,526 | 9813 | 2713 | — | |
|
|
Nursing records per patient, mean (SD) | 45.2 (45.1) | 69.8 (52.6) | 25.0 (23.0) | <.001 | |
|
|
Nursing record length,c mean (SD) | 4739.6 (2127.5) | 5522.9 (2005.8) | 4094.5 (2009.1) | <.001 | |
aWelch t test between fallers and nonfallers used.
bNot applicable.
cNumber of Japanese or Chinese characters.
Morphemes used in the model for predicting falls. Morphemes related to known or potential risk factors (indicated in brackets) were extracted from 378 morphemes used in the final model of the first experiment.
[Psychotropics]
Seroquel, Lendormin, Serenace
[Mental status]
recognition, dementia, arousal, mental status, somnolence willingness, cognitive function, orientation, esthesia, sleeplessness, anxiousness, Myslee
[Motor function]
postural change, aid, assistance, support, lower limb, rehabilitation, slippers, wheelchair, sitting square, torpor, self-standing, parallel bars, limb, daily life behavior, lumbar region, ride, body posture, dorsal region, gait, extension (of limbs), walking stick
[Excretion]
excretion, defecation, constipation, incontinence, Lasix, Pursennid, Biofermine
[Oropharyngeal]
mouth, sputum, hospital food, oral, water drinking, nausea, swallowing, vomiting, dentures, fluid, mouth rinse, eat
[Circulation]
WBC (white blood cells), blood pressure, transfusion, anemia, mmHg, oxygen, neutrophil, blood, pulse, vein, bleeding, blood vessel, heartbeat, platelet
Similar to the process used for the learning data set, nursing records with fewer than 50 characters (13 and 4 nursing records for fallers and nonfallers, respectively) were deleted from the test data set, leaving 9800 nursing records for fallers and 2709 nursing records for nonfallers. For each patient in the test data set, the mean value of the risk probabilities for all their nursing records was calculated as a patient risk score that was used to evaluate the performance for predicting falls by ROC analysis. To draw the ROC curve, we calculated the true positive rate and false positive rate using the patient risk score (continuous variables that range from 0 to 1) and category (faller or nonfaller) for each patient. Scanning the cutoff values from 0 to 1, the true and false positive rates were calculated from the confusion matrix for each cutoff value.
As shown in Figure 1A, the area under the ROC curve (AUC) was 0.835, which indicates excellent separation between fallers and nonfallers. Applying a threshold score of 0.5602, corresponding to the point on the ROC curve closest to the coordinate (0, 1), each patient was sorted into risk and no risk categories, as shown in the confusion matrix (Table 4). Then the sensitivity, specificity, and odds ratios were calculated (Table 5). Sensitivity and specificity are the most commonly used measures for diagnostic performance from the viewpoint of actual medical practice, in which the former is the rate of correct diagnosis among all disease patients and the latter is the rate of correct diagnosis among all normal patients. The odds ratio is the most commonly used measure in case-control studies.
Figure 1.
Precision and reproducibility of the model for predicting falls using the test data set. Five independent experiments were conducted for the learning and testing steps. A: receiver operator characteristic (ROC) curve for experiment 1; B: scatterplot of patient risk scores for two of the five experiments (1 and 4). AUC: area under the curve.
Table 4.
Confusion matrix of fall prediction for experiment 1.
| Prediction | Patients | ||
|
|
Fallers, n | Nonfallers, n | Total, N |
| Risk | 128 | 39 | 167 |
| No risk | 40 | 165 | 205 |
| Total | 168 | 204 | 372 |
Table 5.
Reproducibility of the model for predicting falls. A summary of evaluation indexes for the five experiments are shown.
| Statistic | Experiment | Mean (SD) | ||||
|
|
1 | 2 | 3 | 4 | 5 | |
| Area under the curve | 0.835 | 0.831 | 0.832 | 0.842 | 0.831 | 0.834 (0.005) |
| Sensitivity (95% CI) | 0.762 (0.714-0.813) |
0.75 (0.702-0.801) |
0.774 (0.726-0.824) |
0.78 (0.730-0.823) |
0.78 (0.732-0.830) |
0.769 (0.013) |
| Specificity (95% CI) | 0.809 (0.766-0.854) |
0.794 (0.751-0.839) |
0.779 (0.736-0.825) |
0.789 (0.724-0.801) |
0.755 (0.712-0.801) |
0.785 (0.02) |
| Odds ratio (95% CI) | 13.54 (8.23-22.27) |
11.57 (7.11-18.83) |
12.09 (7.40-19.73) |
13.26 (8.07-21.78) |
10.9 (6.72-17.71) |
12.27 (1.11) |
Next, the reproducibility of the analysis was examined by conducting similar experiments four more times (experiments 2 to 5). The model was constructed with a new learning data set, and the test data set was evaluated by generating random numbers for patient identification numbers, after which scatterplots were drawn to check correlations of patient risk scores between all combinations of two experiments (an example for experiments 1 and 4 is shown in Figure 1B). The analytical indexes for the five independent experiments demonstrated the high precision (Table 5) and reproducibility (Figure 1B and Table 6) of the model for the prediction of falls. These results demonstrated that text analysis of nursing records was an efficient method for predicting falls with high reproducibility.
Table 6.
Correlations (R2 for linear regression) of all combinations of two out of five experiments are shown.
| Experiment | 1 | 2 | 3 | 4 | 5 |
| 1 | — | 0.939 | 0.952 | 0.946 | 0.945 |
| 2 | — | — | 0.932 | 0.937 | 0.957 |
| 3 | — | — | — | 0.948 | 0.957 |
| 4 | — | — | — | — | 0.945 |
| 5 | — | — | — | — | — |
Imminent Precursors of Falls
In the next step, the detection of the imminent precursors of falls was attempted by extracting specific features from the nursing records written several days before each incident. For the purpose, nursing records of all fallers were collected as “Faller data set” and then tagged with imminent (1-7 days before the fall) or not imminent (Table 7). After bisecting the faller data set into a learning data set and a test data set, the former was used to construct a model for discrimination of the tags by the same method described previously for risk/no risk categorization; that is, the final model was built from morphemes identified in at least four of the six primary models constructed using the learning data set. Then the final model was used to evaluate the probability of each faller nursing record in the test data set being placed in the imminent category, after which the performance of the detection of imminent precursors was evaluated using ROC analysis (Figure 2A) and the confusion matrix (Table 8). After four more independent examinations were performed in the same manner to check reproducibility, the average AUC of the ROC curve was 0.567 for the five experiments (Table 9), which demonstrates limited prediction of nursing records for imminent falls.
Table 7.
Characteristics of patients and nursing records in the faller data set for detection of imminent precursors.
| Faller data set | All fallers | >60 Nursing records | ≤45 Nursing records | ||
| Learning data set |
|
|
|
||
|
|
Patients, n | 167 | 56 | 91 | |
|
|
Gender, n |
|
|
|
|
|
|
|
Female | 78 | 32 | 38 |
|
|
|
Male | 89 | 24 | 53 |
|
|
Age (years), mean (SD) | 73.4 (12.9) | 74.7 (11.2) | 73.0 (12.7) | |
|
|
Nursing records, n | 9094 | 5809 | 2231 | |
|
|
|
Imminenta | 1114 | 487 | 464 |
|
|
|
Not imminent | 7980 | 5322 | 1767 |
|
|
Nursing records per patient, mean (SD) | 54.5 (45.7) | 103.8 (45.7) | 24.5 (12.3) | |
|
|
Nursing record length, mean (SD) | 5559.4 (1961.9) | 5363.34 (1879.5) | 5628.6 (2081.0) | |
| Test data set |
|
|
|
||
|
|
Patients, n | 168 | 56 | 95 | |
|
|
Gender, n |
|
|
|
|
|
|
|
Female | 78 | 21 | 48 |
|
|
|
Male | 90 | 35 | 47 |
|
|
Age (years), mean (SD) | 73.2 (12.8) | 72.4 (12.9) | 74.0 (14.2) | |
|
|
Nursing records, n | 9813 | 6693 | 2239 | |
|
|
|
Imminenta | 984 | 424 | 463 |
|
|
|
Not imminent | 8829 | 6269 | 1776 |
|
|
Nursing records per patient, mean (SD) | 58.4 (54.1) | 119.5 (51.9) | 23.6 (12.6) | |
|
|
Nursing record length, mean (SD) | 5522.9 (2005.8) | 5022.2 (2187.5) | 5662.8 (1890.6) | |
aNursing records registered within seven days before a fall.
Figure 2.
Precision of the model for detecting imminent precursors using the faller data set. Five independent experiments were conducted for the learning and testing steps to identify imminent precursors of falls among all fallers (A) and among fallers who were short-term patients (B). Receiver operating characteristic (ROC) curves for experiment 1 out of the five experiments are shown. AUC: area under the curve.
Table 8.
Results of discrimination of imminent precursors of falls among all fallers. Confusion matrix for experiment 1 out of five experiments is shown.
| Prediction | Nursing records | ||
|
|
Imminent | Not imminent | Total |
| Imminent | 553 | 4281 | 4834 |
| Not imminent | 429 | 4536 | 4965 |
| Total | 982 | 8817 | 9799 |
Table 9.
Reproducibility of the model for detecting imminent precursors using the faller data set. Five independent experiments were conducted for the learning and testing steps to identify imminent precursors of falls among all fallers and among fallers who were shot-term patients.
| Group and statistic | Experiment | Mean (SD) | |||||
|
|
1 | 2 | 3 | 4 | 5 |
|
|
| Fallers |
|
||||||
|
|
Area under the curve | 0.562 | 0.576 | 0.568 | 0.566 | 0.564 | 0.567 (0.005) |
|
|
Sensitivity (95% CI) | 0.563 (0.546-0.581) |
0.543 (0.526-0.560) |
0.611 (0.593-0.630) |
0.576 (0.559-0.594) |
0.536 (0.519-0.553) |
0.566 (0.030) |
|
|
Specificity (95% CI) | 0.514 (0.509-0.520) |
0.576 (0.571-0.582) |
0.477 (0.472-0.482) |
0.517 (0.512-0.522) |
0.558 (0.552-0.563) |
0.529 (0.039) |
|
|
Odds ratio (95% CI) | 1.37 (1.20-1.56) |
1.62 (1.42-1.84) |
1.43 (1.25-1.64) |
1.46 (1.27-1.66) |
1.45 (1.27-1.66) |
1.47 (0.09) |
| Fallers who were short-term patients |
|
||||||
|
|
Area under the curve | 0.613 | 0.607 | 0.595 | 0.602 | 0.618 | 0.607 (0.009) |
|
|
Sensitivity (95% CI) | 0.547 (0.522-0.572) |
0.649 (0.621-0.677) |
0.492 (0.470-0.515) |
0.607 (0.581-0.635) |
0.623 (0.596-0.651) |
0.584 (0.063) |
|
|
Specificity (95% CI) | 0.626 (0.613-0.641) |
0.524 (0.512-0.536) |
0.653 (0.639-0.668) |
0.548 (0.535-0.560) |
0.560 (0.547-0.573) |
0.582 (0.055) |
|
|
Odds ratio (95% CI) | 2.02 (1.64-2.49) |
2.03 (1.64-2.51) |
1.83 (1.48-2.25) |
1.87 (1.52-2.31) |
2.10 (1.70-2.59) |
1.97 (0.12) |
Based on the hypothesis that the medical conditions of long-term inpatients would be stable, and changes in risk factors for falls would be difficult to detect, we also performed separate analyses of long-term and short-term inpatients. Fallers with more than 60 nursing records or 45 or less nursing records were selected as long-term and short-term inpatients, respectively, and the prediction of imminent falls was conducted for each group (Table 7).
We found that improved prediction of imminent falls was achieved for short-term inpatients, with an AUC of mean 0.607 (SD 0.009) (for five independent experiments, Figure 2B and Tables 9 and 10), whereas prediction was poor for long-term inpatients (AUC mean 0.496, SD 0.011; summary table for the five experiments not shown). Confusion matrices were constructed for the short-term group, and the sensitivity, specificity, and odds ratios were calculated (Table 9). The results suggested that the calculated risk probability could be used to assess the imminent risk of falls for short-term inpatients at the time when each nursing record was written.
Table 10.
Results of discrimination of imminent precursors of falls among fallers who were short-term patients. Confusion matrix for experiment 1 out of five experiments is shown.
| Prediction | Nursing records | ||
|
|
Imminent | Not imminent | Total |
| Imminent | 252 | 663 | 915 |
| Not imminent | 209 | 1112 | 1321 |
| Total | 461 | 1775 | 2236 |
Discussion
Principal Results
Our results confirmed it is possible to predict inpatient falls using text analysis of nursing records in a hospital EMR system, with an AUC of 0.834 across an average of five independent experiments. In many previous studies, the prediction of falls was based on specified risk factors, such as the use of psychotropic drugs [28-32], mental state (eg, disorientation, confusion, and delirium) [4,5,30,33-35], impaired motor function (eg, unstable gait and muscle weakness) [4,5,29,32,35], and excretory condition (eg, incontinence and frequent toileting) [5,33,35]. Additionally, the usefulness of nursing records for inpatient fall prediction was discussed recently [36], and it was shown that nursing records contained words known as risk factors for inpatient falls and interventions used in daily practice using NLP analysis. However, all the words identified in the analysis were preselected using prior reports, risk assessment tools, and subject matter expert’s knowledge. By contrast, we did not focus on any specific factor or emphasize any specific keywords, topics, concepts, or fields throughout our NLP analysis of unstructured text in nursing records and subsequent machine learning. Despite this, we found many words closely related to the previously mentioned risk factors in the list of morphemes that contributed to the prediction of fall risk (Textbox 1). Thus, the Concept Encoder successfully extracted known risk factors for falls as words with a statistically significant correlation to actual incidents. It is possible that several other words (or related concepts) that contribute to the model might be unknown risk factors. These candidate novel risk factors may not only be useful for predicting falls but also for determining the causes of falls or selecting interventions for prevention. In future work, we will conduct further numerical analyses of these candidates to examine their similarities or relationships, such as cluster analysis or context analysis based on the document-word embedding matrix (DW). If it is proven that words related to known and novel risk factors are effective for predicting falls, this might encourage hospital nurses to write nursing records that emphasize these factors, thus improving the quality of nursing records and allowing falls to be predicted with higher precision.
There was a statistically significant difference between nursing records recorded one to seven days before a fall and others. This suggests that a fall risk monitoring system designed to analyze nursing records daily and alert health care professionals when an increase of fall risk is detected could be an effective tool for the prevention of falls. Recently, the authors developed a new version of Concept Encoder with improved computational capacity and deployed for a currently ongoing study using a larger data set (all nursing records for three years; approximately 520,000 nursing records from 900 fallers and 28,000 nonfallers). Encouraged by the early results of the study, which has shown considerable improvement in the prediction for imminent falls (AUC of approximately 0.73), the authors have developed the first version of the fall risk monitoring system.
Because nursing records contain continuous information covering a broad context regardless of the underlying disease or complications and results of various medical tests and vital signs, this algorithm can be applied to construct models for predicting other specific medical interests, such as a sudden change of the patient’s condition or recurrence of acute illness. It also has the potential to be used as the basis of a multipurpose diagnosis and caregiving support system.
Recent developments in machine learning technology have enhanced the range of application, but it is still rarely used in the health care field. One reason is that neural network analysis, such as deep learning, cannot provide human-interpretable models or rules because of the numerous layers in the learning process. This “black box problem,” that is, poor traceability of the learning and analysis processes, is one reason that machine learning has not been widely applied in the health care field. The algorithm that we used (Concept Encoder) achieves very efficient transformation from documents to a document-word matrix, after which even simple logistic regression analysis can successfully predict falls. Moreover, the characteristics and probability distribution of the data are provided in an interpretable manner. Thus, even after a machine learning process is used, it can perform statistical analyses with high levels of stability, reproducibility, and verifiability that are required in the health care field. In this field, evidence-based decision making is valued, and vast amounts of medical data have been accumulated over many years for this purpose. It seems possible that Concept Encoder can be applied to mine these precious assets with verifiable analysis.
Limitations
The low quantity of data may be a limitation in this study. However, due to the oversampling technique that we used, in which minority data were resampled to balance the two-group data set, we believe that the results of the study were not substantially affected by the low rate of falls. However, meta-analysis and a multicenter study will be considered in future work, which will generate more data. Additionally, we defined imminent as one to seven days before the fall. When we considered shorter time periods, such as one to three or one to five days before the fall, this reduced the number of imminent nursing records, which resulted in poorer prediction. In future work, larger data sets will enable the analysis of shorter time periods. Finally, as this is the first study to analyze nursing records using NLP and machine learning, there is no prior work available for comparison.
Conclusions
We verified that text analysis of a single input—nursing records—using an NLP algorithm and machine learning was effective for the prediction of falls among hospital inpatients and the detection of imminent precursors of fall incidents. The approach was also able to extract useful information related to various types of fall risk factors, whether they are known or unknown, from the unstructured description of the nursing records. This can serve as a basis for a fall risk monitoring system (eg, screen-based) that can output risk factors for each high-risk patient together with the risk probability. We have already developed a prototype monitoring system and plan to start testing in collaboration with several hospitals. We are also developing an English version of our system for testing in English-speaking countries. Studies have reported that intervention is more successful when various health care professionals are involved as a team rather than taking a nursing-centric approach [17,37,38]. Thus, the output of data and risk factors provided by the system could be helpful for information sharing among teams of health care professionals at safety huddles or during handover.
Acknowledgments
HU was affiliated with FRONTEO Inc. at the time of the study and is currently affiliated with Neopharma Japan Co Ltd, which has no involvement in this study. We thank Hideki Takeda, Kohei Matsumoto, and Hiroki Ego for general support during the study. We thank Maxine Garcia, PhD, from the Edanz Group for rewriting a draft of this manuscript.
Abbreviations
- AUC
area under the curve
- EMR
electronic medical record
- HFRM
Hendrich Fall Risk Model
- MCMC
Markov chain Monte Carlo
- NLP
natural language processing
- PV-DBOW
paragraph vector-distributed bag of words
- ROC
receiver operating characteristic
- STRATIFY
St Thomas’s Risk Assessment Tool in Falling Elderly Inpatients
Footnotes
Authors' Contributions: CO, HN, and MN contributed to the conception and design of the study. HN and MN collected the data. HT designed and developed the system. HU performed the data analysis. HU and HT wrote the manuscript, and all other authors reviewed and provided feedback with each draft. All authors read and approved the final manuscript.
Conflicts of Interest: HT has patent JP 2017-214388. HT and HU have patents JP2018-088828 and JP2018-088829 pending. HT is and HU was an employee of FRONTEO Inc, which developed and marketed a fall prediction system based on the results of this research. All other authors have no conflicts to declare.
References
- 1.Bouldin ELD, Andresen EM, Dunton NE, Simon M, Waters TM, Liu M, Daniels MJ, Mion LC, Shorr RI. Falls among adult patients hospitalized in the United States: prevalence and trends. J Patient Saf. 2013 Mar;9(1):13–7. doi: 10.1097/PTS.0b013e3182699b64. http://europepmc.org/abstract/MED/23143749. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Japan Federation of Democratic Medical Institutions. [2020-04-08]. Rate of fall incident: report of quality improvement of medical care project 2016 https://www.min-iren.gr.jp/hokoku/hokoku_h28.html.
- 3.Joint Commission Sentinel Event Alert. 2015. [2020-04-08]. Preventing falls and fall-related injuries in health care facilities https://www.jointcommission.org/assets/1/6/SEA_55_Falls_4_26_16.pdf. [PubMed]
- 4.Morse JM, Tylko SJ, Dixon HA. Characteristics of the fall-prone patient. Gerontologist. 1987 Aug;27(4):516–22. doi: 10.1093/geront/27.4.516. [DOI] [PubMed] [Google Scholar]
- 5.Oliver D, Britton M, Seed P, Martin FC, Hopper AH. Development and evaluation of evidence based risk assessment tool (STRATIFY) to predict which elderly inpatients will fall: case-control and cohort studies. BMJ. 1997 Oct 25;315(7115):1049–53. doi: 10.1136/bmj.315.7115.1049. http://europepmc.org/abstract/MED/9366729. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Hendrich A, Nyhuis A, Kippenbrock T, Soja ME. Hospital falls: development of a predictive model for clinical practice. Appl Nurs Res. 1995 Aug;8(3):129–39. doi: 10.1016/s0897-1897(95)80592-3. [DOI] [PubMed] [Google Scholar]
- 7.Hendrich AL, Bender PS, Nyhuis A. Validation of the Hendrich II Fall Risk Model: a large concurrent case/control study of hospitalized patients. Appl Nurs Res. 2003 Feb;16(1):9–21. doi: 10.1053/apnr.2003.YAPNR2. [DOI] [PubMed] [Google Scholar]
- 8.Oliver D, Daly F, Martin FC, McMurdo MET. Risk factors and risk assessment tools for falls in hospital in-patients: a systematic review. Age Ageing. 2004 Mar;33(2):122–30. doi: 10.1093/ageing/afh017. [DOI] [PubMed] [Google Scholar]
- 9.Oliver D, Papaioannou A, Giangregorio L, Thabane L, Reizgys K, Foster G. A systematic review and meta-analysis of studies using the STRATIFY tool for prediction of falls in hospital patients: how well does it work? Age Ageing. 2008 Nov;37(6):621–7. doi: 10.1093/ageing/afn203. http://europepmc.org/abstract/MED/18829693. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Aranda-Gallardo M, Morales-Asencio JM, Canca-Sanchez JC, Barrero-Sojo S, Perez-Jimenez C, Morales-Fernandez A, de LME, Moya-Suarez AB, Mora-Banderas AM. Instruments for assessing the risk of falls in acute hospitalized patients: a systematic review and meta-analysis. BMC Health Serv Res. 2013 Apr 02;13:122. doi: 10.1186/1472-6963-13-122. https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-13-122. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Matarese M, Ivziku D, Bartolozzi F, Piredda M, De Marinis MG. Systematic review of fall risk screening tools for older patients in acute hospitals. J Adv Nurs. 2015 Jun;71(6):1198–209. doi: 10.1111/jan.12542. [DOI] [PubMed] [Google Scholar]
- 12.Oliver D, Connelly JB, Victor CR, Shaw FE, Whitehead A, Genc Y, Vanoli A, Martin FC, Gosney MA. Strategies to prevent falls and fractures in hospitals and care homes and effect of cognitive impairment: systematic review and meta-analyses. BMJ. 2007 Jan 13;334(7584):82. doi: 10.1136/bmj.39049.706493.55. http://europepmc.org/abstract/MED/17158580. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Oliver D, Healey F, Haines TP. Preventing falls and fall-related injuries in hospitals. Clin Geriatr Med. 2010 Nov;26(4):645–92. doi: 10.1016/j.cger.2010.06.005. [DOI] [PubMed] [Google Scholar]
- 14.Cameron ID, Gillespie LD, Robertson MC, Murray GR, Hill KD, Cumming RG, Kerse N. Interventions for preventing falls in older people in care facilities and hospitals. Cochrane Database Syst Rev. 2012 Dec 12;12:CD005465. doi: 10.1002/14651858.CD005465.pub3. [DOI] [PubMed] [Google Scholar]
- 15.Miake-Lye IM, Hempel S, Ganz DA, Shekelle PG. Inpatient fall prevention programs as a patient safety strategy: a systematic review. Ann Intern Med. 2013 Mar 05;158(5 Pt 2):390–6. doi: 10.7326/0003-4819-158-5-201303051-00005. [DOI] [PubMed] [Google Scholar]
- 16.Sahota O, Drummond A, Kendrick D, Grainge MJ, Vass C, Sach T, Gladman J, Avis M. REFINE (REducing Falls in In-patieNt Elderly) using bed and bedside chair pressure sensors linked to radio-pagers in acute hospital care: a randomised controlled trial. Age Ageing. 2014 Mar;43(2):247–53. doi: 10.1093/ageing/aft155. http://ageing.oxfordjournals.org/cgi/pmidlookup?view=long&pmid=24141253. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Morris R, O’Riordan S. Prevention of falls in hospital. Clin Med. 2017 Aug 01;17(4):360–362. doi: 10.7861/clinmedicine.17-4-360. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Afzal N, Sohn S, Abram S. Identifying peripheral arterial disease cases using natural language processing of clinical notes. IEEE EMBS Int Conf Biomed Health Inform 2016 Feb. 2016:126–131. doi: 10.1109/bhi.2016.7455851. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Wi C, Sohn S, Rolfes MC, Seabright A, Ryu E, Voge G, Bachman KA, Park MA, Kita H, Croghan IT, Liu H, Juhn YJ. Application of a Natural Language Processing Algorithm to Asthma Ascertainment. An Automated Chart Review. Am J Respir Crit Care Med. 2017 Aug 15;196(4):430–437. doi: 10.1164/rccm.201610-2006OC. http://europepmc.org/abstract/MED/28375665. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Chase HS, Mitrani LR, Lu GG, Fulgieri DJ. Early recognition of multiple sclerosis using natural language processing of the electronic health record. BMC Med Inform Decis Mak. 2017 Feb 28;17(1):24. doi: 10.1186/s12911-017-0418-4. https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911-017-0418-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Stevens S, Pickering D. Keeping good nursing records: a guide. Community Eye Health. 2010 Dec;23(74):44–5. http://europepmc.org/abstract/MED/21311663. [PMC free article] [PubMed] [Google Scholar]
- 22.Kudo T. Project Web Site. [2020-04-08]. MeCab: Yet another part-of-speech and morphological analyzer https://sourceforge.net/projects/mecab/
- 23.Mikolov T, Chen K, Corrado G. arXiv.org. 2013. [2020-04-08]. Efficient estimation of word representations in vector space https://arxiv.org/abs/1301.3781.
- 24.Pennington J, Socher R, Manning C. Glove: Global Vectors for Word Representation. 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP); October 26–28, 2014; Doha, Qatar. Glove: global vectors for word representation. Empir Meth Nat Lang Process (EMNLP) 2014; 2014. Oct, https://www.aclweb.org/anthology/D14-1162/ [DOI] [Google Scholar]
- 25.Dai A, Olah C, Le Q. arXiv.org. 2015. [2020-04-08]. Document embedding with paragraph vectors https://arxiv.org/abs/1507.07998.
- 26.Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. jair. 2002 Jun 01;16:321–357. doi: 10.1613/jair.953. [DOI] [Google Scholar]
- 27.imblearn SMOTE. [2020-04-08]. over_sampling https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html.
- 28.Ballinger BR, Ramsay AC. Accidents and drug treatment in a psychiatric hospital. Br J Psychiatry. 1975 May;126:462–3. doi: 10.1192/bjp.126.5.462. [DOI] [PubMed] [Google Scholar]
- 29.Lichtenstein MJ, Griffin MR, Cornell JE, Malcolm E, Ray WA. Risk factors for hip fractures occurring in the hospital. Am J Epidemiol. 1994 Nov 01;140(9):830–8. doi: 10.1093/oxfordjournals.aje.a117331. [DOI] [PubMed] [Google Scholar]
- 30.Passaro A, Volpato S, Romagnoni F, Manzoli N, Zuliani G, Fellin R. Benzodiazepines with different half-life and falling in a hospitalized population: The GIFA study. Gruppo Italiano di Farmacovigilanza nell'Anziano. J Clin Epidemiol. 2000 Dec;53(12):1222–9. doi: 10.1016/s0895-4356(00)00254-7. [DOI] [PubMed] [Google Scholar]
- 31.Gales BJ, Menard SM. Relationship between the administration of selected medications and falls in hospitalized elderly patients. Ann Pharmacother. 1995 Apr;29(4):354–8. doi: 10.1177/106002809502900402. [DOI] [PubMed] [Google Scholar]
- 32.Chu LW, Pei CK, Chiu A, Liu K, Chu MM, Wong S, Wong A. Risk factors for falls in hospitalized older medical patients. J Gerontol A Biol Sci Med Sci. 1999 Jan;54(1):M38–43. doi: 10.1093/gerona/54.1.m38. [DOI] [PubMed] [Google Scholar]
- 33.Schmid NA. 1989 Federal Nursing Service Award Winner. Reducing patient falls: a research-based comprehensive fall prevention program. Mil Med. 1990 May;155(5):202–7. [PubMed] [Google Scholar]
- 34.Salgado R, Lord SR, Packer J, Ehrlich F. Factors associated with falling in elderly hospital patients. Gerontology. 1994;40(6):325–31. doi: 10.1159/000213607. [DOI] [PubMed] [Google Scholar]
- 35.Gluck T, Wientjes HJ, Rai GS. An evaluation of risk factors for in-patient falls in acute and rehabilitation elderly care wards. Gerontology. 1996;42(2):104–7. doi: 10.1159/000213779. [DOI] [PubMed] [Google Scholar]
- 36.Bjarnadottir RI, Lucero RJ. What Can We Learn about Fall Risk Factors from EHR Nursing Notes? A Text Mining Study. eGEMs. 2018 Sep 20;6(1):21. doi: 10.5334/egems.237. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Jones KJ, Venema DM, Nailon R, Skinner AM, High R, Kennel V. Shifting the paradigm: an assessment of the quality of fall risk reduction in Nebraska hospitals. J Rural Health. 2015;31(2):135–45. doi: 10.1111/jrh.12088. doi: 10.1111/jrh.12088. [DOI] [PubMed] [Google Scholar]
- 38.Cracknell A, Lovatt A, Winfield A, Arkhipkina S, McDonagh E, Green A, Rooney M. Huddle up for safer healthcare: how frontline teams can work together to improve patient safety. Future Hosp J. 2016 Jun 01;3(Suppl 2):s31. doi: 10.7861/futurehosp.3-2s-s31. http://europepmc.org/abstract/MED/31098260. [DOI] [PMC free article] [PubMed] [Google Scholar]