Improved Accuracy and Efficiency of Primary Care Fall Risk Screening of Older Adults Using a Machine Learning Approach

Abstract

Background

While many falls are preventable, they remain a leading cause of injury and death in older adults. Primary care clinics largely rely on screening questionnaires to identify people at risk of falls. Limitations of standard fall risk screening questionnaires include suboptimal accuracy, missing data, and non-standard formats, which hinder early identification of risk and prevention of fall injury. We used machine learning methods to develop and evaluate electronic health record (EHR)-based tools to identify older adults at risk of fall-related injuries in a primary care population and compared this approach to standard fall screening questionnaires.

Methods

Using patient-level clinical data from an integrated health care system consisting of 16-member institutions, we conducted a case-control study to develop and evaluate prediction models for fall-related injuries in older adults. Questionnaire-derived prediction with three questions from a commonly used fall risk screening tool were evaluated. We then developed four temporal machine learning models using routinely available longitudinal EHR data to predict future risk of fall injury. We also developed a fall injury prevention clinical decision support (CDS) implementation prototype to link preventative interventions to patient-specific fall injury risk factors.

Results

Questionnaire-based risk screening achieved AUC up to 0.59 with 23% to 33% similarity for each pair of three fall injury screening questions. EHR-based machine learning risk screening showed significantly improved performance (best AUROC=0.76), with similar prediction performance between 6-month and one-year prediction models.

Conclusions

The current method of questionnaire-based fall risk screening of older adults is suboptimal with redundant items, inadequate precision and no linkage to prevention. A machine learning fall injury prediction method can accurately predict risk with superior sensitivity while freeing up clinical time for initiating personalized fall prevention interventions. The developed algorithm and data science pipeline can impact routine primary care fall prevention practice.

Introduction

Preventable falls are the leading cause of injury and accidental death among adults aged 65 and older.¹ Despite growing evidence that falls can be prevented,^2,3 deaths from falls continue to rise.⁴ A single fall frequently results in a fear of falling that can begin a downward spiral of reduced mobility, leading to loss of function, activity restriction and further risk for falls.^5,6 The challenge is to ensure that older adults at risk of falling receive appropriate referrals, education and support from their primary care provider. Many primary care providers (PCPs) are unaware of their patients’ falls, as half of older adults do not discuss falls with their PCP.⁷

Primary care clinics largely rely on screening questionnaires to target their fall prevention practices.⁸ However, the utility of existing screening questionnaires is limited by suboptimal accuracy of patient-reported falls, missing data, and non-standard formats (Figure 1).^9,10 Fall risk screening and assessment is often included in the Medicare Annual Wellness Visit (AWV), but more than half of older patients do not participate in AWVs, and fall risk is not routinely addressed in other types of health care encounters. Fall risk screeners also have limited clinical impact because they are broad and do not inform individualized fall prevention plans.

Figure 1.

Comparison between current questionnaire-based approach and a data-driven approach for screening of fall and related injury

Given the large number of risk screeners recommended during the Medicare AWV and limited clinicians’ time, machine-learning based screening could potentially help streamline and enable provision of more efficient care (Figure 1). EHR data, with its high-intensity measures of patient status, could provide a useful data source to automate or at least inform primary care screening practices and tailored interventions.^11–13 Using EHR data and machine learning algorithms could achieve more accurate prediction power for complex clinical conditions^14–17 Recent work has shown that EHR-based models have robust performance with predicting fall risk in inpatient and outpatient settings^18–20, suggesting that EHR derived algorithms could be used to augment the accuracy and streamline the number of items required to adequately screen for fall and fall-related injury risk. However, previous studies have several limitations making it challenging to translate these algorithms to clinical practice. First, most EHR-based models used patient reported fall events, as the prediction outcome, making it difficult to validate the algorithms across studies and health systems where this outcome in not consistently captured. Second, patients report fall events mainly when they sustain an injury and seek medical care. This may cause a high number of missing cases and result in model bias. Third, fall-related injuries, which is a more objective measurement, is rarely used as the model outcome. Furthermore, questions remain about how to effectively use longitudinal and highly-granular EHR data (including temporal features) to develop more accurate models for fall and related injury events while avoiding overfitting.²¹ Despite the promising studies to date, investigation is clearly needed to standardize fall and fall-related injury model outcome definitions using routinely captured EHR features and to develop risk models that can be linked to preventative interventions and inform routine care.

The goal of this project was to conduct a case-control study to develop and evaluate different screening tools for fall-related injuries in a primary care population. We also aimed to develop a fall injury prevention clinical decision support (CDS) implementation prototype to link patient-specific determinants of risk to evidence-based interventions to increase the utility of machine learning models in primary care practice.

Methods

Data source

We used clinical databases within the Mass General Brigham (MGB) Healthcare system, which includes a centralized clinical data warehouse consisting of data from 16-member institutions including academic medical centers, specialty and community hospitals, a rehabilitation network, and a primary and specialty care provider network. Patient records from primary care practices were extracted for the study.

Study cohort development and temporal design

We applied temporal criteria during cohort development (Figure 2). An “index time” was identified for the first fall risk screening time from 2018 (e.g., the time that all patients in the cohort were asked fall risk screening questions). We extracted model input features from a multi-year duration (2015–2017) before index time. By consulting with clinical experts, we used various time durations to extract these EHR variables, based on their time-sensitive nature, e.g. time sensitive features versus static features (see details in below sections and Figure 2). We used patient records in 2019 to capture the model outcome: fall-related injury based on validated diagnosis codes.²²

Figure 2.

Summary of temporal EHR-based machine learning prediction model

We then applied multiple data mining steps (Step 1 to 4 were applied for both case and control groups): First, we identified patients age 65 and above and with at least one primary care visit within MGB network from 2015 to 2019. We only used clinical data from before 2020 to avoid potential impact of the COVID epidemic on clinical operations and data collection. Second, we selected patients with both a primary care visit and related fall risk screening records in 2018. Third, we captured records for a total of 17 types of diagnoses and 5 types of medication use information from 2015 to 2017 (see Supplementary Table 2). Fourth, we captured records of hospitalization (defined as >24 hours stay) and physical therapy in 2017 to identify patients who were more likely to experience severe health problems and to have mobility issues including gait and balance problems. These two variables were selected based on clinical expert recommendations as they could be associated with higher risk for fall injury (model outcome). Fifth, we identified patients with at least one fall injury diagnosis record in 2019 as the case group. Controls did not have any fall injury records during the same period. Through these steps, patients with any missing record or fall injury diagnoses were excluded from the study cohort.

Model development

1. Questionnaire-based screening tool evaluation

Questionnaire feature extraction

The Mass General Brigham fall risk screening questions recommended by the Centers for Disease Control and Prevention (CDC),²³ are widely used in practice,^24,25 and include: 1) have you fallen and hurt yourself in the past year?, 2) are you afraid that you might fall because of balance and walking problems?, and 3) have you fallen 2 or more times in the past year? The information collected from the three questions focuses on different aspects of fall and fall related injury, such as fall history, timing of previous falls with injury and fear of falling. Using the MGB study cohort from 2018, we obtained patient answers (YES or NO) to all three questions. We then developed three binary variables based on patient responses (YES as 1 and NO as 0).

Pair-wise similarity evaluation among three fall risk screening questions

Due to the binary format for all three questions, we chose the Jaccard similarity coefficient,²⁶ rather than the correlation coefficient, to estimate their relationships. We calculated it for each pair among three questions to estimate their similarities. The Jaccard coefficient is defined as the size of the intersection divided by the size of the union.

Evaluation of fall risk screening questions

Multiple logistic regression models were developed using all three screening questions (full model) or part of the three question with different combinations (sub-models with two or one question) as independent variables. We used question records from 2018 as input and fall injury within one year after index time as the model outcome. To evaluate the model performance, we used the area under the receiver operating characteristic curve (AUC), sensitivity and specificity. All metrics were calculated using the test set from train-test split.

2. EHR-based machine learning prediction models (summarized in Figure 2)

EHR feature selection

Based on the literature and clinician expert opinion, we selected 31 EHR-derived variables as the model input, including patient demographics, diagnosis of comorbidities and fall-risk related medications. Data cleaning processes were conducted and patients with missing model input/outcome information were removed to develop the final study cohort.

Prediction outcome time windows

We defined two sequential time windows to capture model prediction outcomes: 1) fall injury occurred within 6 months after index visit time and 2) fall injury occurred within 1 year after index visit time.

Model development and evaluation

Using input features and outcomes defined above, four fall injury prediction models were developed, including regularized logistic regression (LR), support vector machines (SVM), random forest (RF), and neural network (NN). To overcome the overfitting issue, we tuned models through cross-validation to select the best set of parameters and evaluated their performance on an independent test set (see detail descriptions in Supplementary Material).

Development of fall injury prevention clinical decision support (CDS) implementation prototype (Summarized in Figure 3)

Figure 3.

Fall injury prevention CDS implementation prototype

We integrated machine learning model design and real-world clinical scenarios to develop a CDS implementation prototype. First, we categorized model input variables into clinically actionable features (e.g. comorbidity diagnosis) or non-actionable features (e.g. gender), based on whether or not the features are treatable (e.g., can be mapped to fall prevention guideline recommendations); Second, our clinical team generated a set of guideline-based fall prevention treatment strategies^27,28 for each actionable model feature; Third, clinical experts linked actionable features to specific interventions, including treatment of comorbidities, medication adjustment, referral to specialist, and recommendations for exercise and/or physical therapy.

Results

Evaluation of current fall risk screener based on the primary care practice questionnaire

1). Similarity evaluation among fall risk screening questions

The pair-wise Jaccard similarity coefficient was calculated to evaluate the relationships among three questions (Supplementary Table 1). Question 1 and question 3 were most similar -- 33% of patients had the same answer (either both YES or both NO), while question 2 and question 3 were least similar -- 23% of patients had the same answer to these two questions. Jaccard similarity suggests that there was moderate redundancy in these questions.

2). Evaluation model for fall risk screening questions

Using 1,000 cases and 2,000 control patients from MGB system, seven logistic regression predictive models were implemented to predict fall injury with a different number of questions as input variables, in which the full model used all three questions and sub-models used either two or single question with different combinations. The full model achieved AUC of 0.59, with sensitivity and specificity of 0.39 and 0.77, respectively. Among three sub-models with two questions, similar performance was observed with AUC ranging from 0.57 to 0.59. For sub-models with a single question, only the model with question 2 showed a decreased performance with AUC of 0.53, while models with question 1 or question 3 achieved a comparable performance (AUC of 0.56 and 0.57) with the full model and two-question model (Table 1).

Table 1.

Performance of questionnaire-based models (full model vs sub-models)

	Input Variables	AUC (SD)	Sensitivity (SD)	Specificity (SD)
Full Model	All Three Questions	0.59 (0.01)	0.39 (0.01)	0.77 (0.01)
Sub-Model with two questions	Question 1+Question 2	0.57 (0.01)	0.29 (0.02)	0.84 (0.01)
	Question 1+Question 3	0.59 (0.01)	0.39 (0.01)	0.77 (0.01)
	Question 2+Question 3	0.57 (0.02)	0.28 (0.02)	0.84 (0.01)
Sub-Model with single question	Question 1	0.57 (0.01)	0.28 (0.01)	0.84 (0.01)
	Question 2	0.53 (0.01)	0.12 (0.01)	0.94 (0.02)
	Question 3	0.56 (0.01)	0.25 (0.02)	0.86 (0.01)

Machine learning models with EHR variables to predict six-month and 12-month fall injury

Based on previous literature^18,20,29 and recommendations from clinical experts, we developed models to predict fall injury within 6 months or 1 year after the index visit time. For each outcome prediction, we obtained 1,000 cases and 2,000 controls for model development.

Four predictive models were developed using a list of 31 potential predictive variables, including demographics, comorbidities, and medications (Supplementary Table 2). All four models showed decent performance across two prediction time windows. The 6-month prediction showed slightly better performance with the best AUROC =0.76 compared to AUROC =0.74 for the best one-year prediction model. Similar performance was observed across all models on all metrics (Table 2 and Supplementary Figure 1). Among the statistically significant predictors from the logistic regression model, patient age, was found to be the most predictive feature of fall injury followed by previous ICD code-based fall injury history (within 3 years before index visit) in both prediction outcome time windows. Other important predictors included hospitalization (within one year before index visit), abnormalities of gait and mobility (within 3 years before index visit) and total number of total comorbidities (within 3 years before index visit).

Table 2.

Performance of EHR-based models and important predictors

	Fall injury within 6 months			Fall injury within 1 year
	AUC (SD)	Sensitivity (SD)	Specificity (SD)	AUC (SD)	Sensitivity (SD)	Specificity (SD)
Logistic Regression	0.75(0.03)	0.65(0.06)	0.76(0.06)	0.74(0.02)	0.62(0.06)	0.76(0.06)
Random Forest	0.76(0.02)	0.69(0.03)	0.75(0.05)	0.74(0.02)	0.64(0.05)	0.74(0.09)
Support Vector Machine	0.76(0.02)	0.70(0.08)	0.71(0.07)	0.74(0.02)	0.65(0.05)	0.72(0.04)
Neural Network	0.76(0.02)	0.70(0.04)	0.73(0.07)	0.73(0.02)	0.66(0.07)	0.71(0.05)
	Top Ten Significant Predictors (based on Logistic Regression)
	Age (Standardized coefficient =0.39), Fall injury history (0.21), Hospitalization (0.20), Abnormalities of gait and mobility (0.18), Total comorbidities (0.15), Has comorbidity history (0.11), Stroke (0.08), Gender (0.08), Orthostatic.hypotension (0.06), monoplegia_of_lower_Extr (-0.06)			Age (Standardized coefficient =0.59), Fall injury history (0.25), Abnormalities of gait and mobility (0.23), Hospitalization (0.21), Total comorbidities (0.11), Has comorbidity history (0.11), Parkinsons (0.11), Gender (0.10), Vision_impairment (0.09), Osteoporotic_fractures (0.08)

Machine learning models with both EHR and questionnaire variables to predict fall injury

We developed a sub-cohort (750 cases and 1500 controls) with both EHR information and all three screening questions. The same four predictive models were developed using EHR variables and questionnaire variables. Similar performance was observed compared with EHR variable only models. Three screening questions showed negligible contribution to the model performance (Supplementary Table 3 and Supplementary Figure 2).

Fall injury prevention CDS implementation prototype

We developed a fall injury prevention CDS implementation prototype to demonstrate how our EHR-based machine learning model could link preventative interventions to patient-specific determinants of risk and facilitate real-world implementation of our algorithm (illustrated in Figure 3)

Discussion

Injury caused by a fall is a major contributor to morbidity and death among community dwelling older adults.³⁰ Falls are also associated with significant costs to health organizations.³¹ Studies have shown that appropriate interventions, such as deprescribing fall risk increasing drugs, fall prevention exercises that address strength, balance and gait impairments, and treatment for osteoporosis can reduce fall risk.³² A personalized and accurate tool to predict high risk patients and their specific risk factors will be important for informing personalized clinical decisions and for applying timely interventions. The current standard approach for fall risk screening is based on multiple CDC-endorsed questions that relate to different aspects of fall risk.³³ These questions are widely used in primary care practices across the country to address regulatory requirements³⁴ but the accuracy of these screening questions is poor and they are too broad to inform preventive interventions. We found that a machine learning fall injury prediction method using longitudinal EHR-derived data was superior to the usual screening approach as the machine learning approach can accurately predict risk with superior sensitivity while reducing time needed in a clinical encounter and facilitating personalized fall prevention interventions.

While we identified that machine leaning-based screening can be more accurate than manual questionnaires, we also conducted a comparative analysis to assess the performance of manual screening questionnaires by comparing the predictive accuracy of the full version (includes all three fall risk screening questions) and shortened versions (includes two questions or one question). We found that a shortened fall risk screening questionnaire identifies the risk of fall injury with similar accuracy as the commonly used²³ three-question screener. Shortening screening questionnaires without compromising accuracy presents an opportunity to alleviate documentation burden and enhance the quality of patient care. For example, during a Medicare Annual Wellness Visit, multiple screenings are required (summarized in Supplementary Figure 3). Reducing time spent completing screening questionnaires allows providers more time during the visit to address patient needs and concerns, thus improving provider-patient communication and care.³⁵ Specifically, due to the high similarity among the three questions, a two-question screener could be used without compromising accuracy. Furthermore, question 1 (i.e. falling twice or more in the past year) could be used as a one-question screener with comparable performance. However, we found that the sensitivity of all of the fall risk screening questions is poor, ranging from .12–.39 (Table 1) versus .62 –.70 for EHR-based models (Table 2), suggesting that replacing the manual fall risk screening questionnaire with the machine learning model that can be linked to preventative interventions may be a more accurate, efficient, and effective solution.

We explored a data-driven approach using machine learning to improve clinical accuracy and efficiency of the fall risk prediction process. We used EHR variables as predictors (listed in Supplementary Table 2) extracted from a multi-site study cohort to develop prediction models for fall injury in a primary care population. A novel feature is that temporal criteria are defined using specific time windows for inputs and outcomes and included in the model design to reflect their sequential order, e.g., predictors must precede outcomes. By consulting clinical experts, we defined three levels of time constraints to select input features: 1) non-time sensitive variables, e.g. race and gender; 2) chronic features within a 3-year duration before screening time (index time), e.g., diagnosis of comorbidities; 3) time sensitive features within a 1-year duration before screening time, e.g., records of hospitalization (Figure 2).

Our machine learning model achieved decent performance (AUC=0.76 and 0.74, sensitivity .62–.70) for both short-term and long-term prediction models, suggesting the promising potential of developing and applying predictive models for fall injury risk screening using EHR data. We also identified multiple important EHR variables which could also be used to facilitate fall injury prevention decisions, thus making the resulting clinical decision support actionable.

Our results suggest that the fall injury screening questions have limitations in real-world practice. Considering that these questions are widely used in primary care practices and consuming providers’ and patients’ valuable time, significant adjustments of these questions, including question design, number of questions needed and best time for these questions to be asked, as well as the most beneficial patient populations, should be further studied. ln light of the improved performance we obtained in EHR-based models, the use of existing EHR variables and machine learning algorithms, could be an efficient and effective strategy for future fall risk and other screening in large patient populations.

Clinical actionability and generalizability, (e.g., “making machine learning matter to clinicians”³⁶), are increasingly important topics for applying machine learning in health care.^37,38 Our team included several elements to facilitate translation into clinical practice: 1) Our model uses standard EHR features as input variables, making it easy to implement in diverse EHR systems across organizations; 2) All variables are routinely captured during typical encounters, requiring no new data acquisition or additional documentation to initialize model use; 3) We also collaborated with clinicians and informaticians to develop a fall injury prevention CDS implementation prototype to link CDS output with guideline-based intervention recommendations to help providers efficiently initiate personalized fall prevention recommendations during a visit; As illustrated in Figure 3, during the primary care visit, the machine learning model can automatically identify high-risk patients using existing EHR records and provide corresponding alerts to providers during the clinical encounter. The CDS output will also include patient-level intervention recommendations based on patient-specific actionable features (e.g., each patient’s fall risk factors). Providers can then make timely decisions based on this information. Informed by the CDS, providers can recommend evidence-based preventative interventions, including treatment of comorbidities, medication adjustment and/or exercise recommendations, to reduce a patient’s actionable risk factors identified from our algorithm. These interventions are widely recommended by the CDC’s STEADI toolkit^2,8,39 and other national fall prevention guidelines but currently are not well integrated into clinical practice;⁴⁰ 4) Based on our design, the fall injury prevention CDS will run in the background of the EHR system and will not require providers to alter their EHR workflow. The CDS can be applied for both individual and sub-population levels for risk screening. By outputting fall risk score during primary care encounters, this algorithm can be used on patient-level risk screening for individual providers. In addition, using the EHR information stored in existing large-scale EHR databases, our algorithm can also be used to provide a summative risk index for selected patient cohorts on the hospital or system levels, which can be used for population health purposes. Lastly, we expect the data science design pipeline and experience gained in this study can help other researchers to better translate their informatics tools to real-world clinical practice.

Limitations

Our study has several limitations. First, we only evaluated three fall risk screening questions. These questions were recognized by the CDC³³ and different versions and combinations are commonly used in primary care practices. Thus, we believe these three questions are a good representation of the fall injury screening process commonly used in primary care practices across the country. Second, we evaluated one type of screener from primary care practices. There are many other types of screening tools which need to be evaluated (e.g. depression risk, substance abuse risk, deep vein thrombosis risk). As discussed above, our analytic pipeline could be used to evaluate different screeners due to the structural similarity of these tools. Third, we applied temporal design for EHR model development for both input and outcomes. We used these time components to reflect a real-world clinical time flow. To optimize these time window definitions, results from prospective evaluation is needed. We have obtained approval from MGB Predictive Analytics Committee to test our model in the real-time clinical environment as part of an existing NIH-funded fall prevention research study (5R61AG068926–02).

Conclusion

This work provides evidence that the current method of questionnaire-based fall risk screening in primary care is suboptimal and that a machine learning risk screening tool using EHR data showed improved performance in predicting fall injury events. Specifically, machine learning screening using EHR data had superior sensitivity to detect patients at risk of future fall injury events compared to the standard questionnaire-based models. We also utilized standard EHR variables to improve model generalizability and demonstrated how CDS could be used to link patient specific risk factors to evidence-based fall injury prevention interventions to facilitate real-world implementation. Next steps include building on these findings to help primary care practices eliminate the unnecessary elements of manual screeners (fall risk and other commonly used screening tools) and to evaluate our model in other time-constrained settings including inpatient and emergency departments. Furthermore, we will optimize fall prevention management in clinical practice by testing the effect of personalized interventions guided by machine learning algorithm outputs.

Supplementary Material

Supindo

Supplementary Table 1. Pair-wise similarities (Jaccard similarity coefficient) among 3 screening questions

Supplementary Table 2. Input variables for EHR models

Supplementary Table 3. Performance of EHR-questionnaire integrated models

Supplementary Figure 1. ROC curve of EHR models

Supplementary Figure 2. ROC curve of EHR-questionnaire integrated models

Supplementary Figure 3. Summary of multi-component tasks for a health professional during Medicare annual wellness visit

Key Points.

• Multiple limitations of current fall risk screening questionnaires hinder early identification of risk and prevention of fall injury.

• In this case-control study, we developed and evaluated prediction models for fall-related injuries in a primary care population.

• Our results suggest that the current method of questionnaire-based fall risk screening in primary care settings is suboptimal and a data-driven fall injury prediction method using longitudinal EHR-derived features should be used.

Why does this paper matter?

Primary care clinics largely rely on screening questionnaires to identify people at risk of falls. This work provides evidence that the current method of questionnaire-based fall risk screening in primary care is suboptimal and that a machine learning risk screening tool using EHR data showed improved performance in predicting fall injury events.

Data sharing statement

The datasets generated during the current study are not publicly available due to hospital IRB regulation and patient privacy.