Machine Learning Applied to Electronic Health Record Data in Home Healthcare: A Scoping Review

Abstract

Objective:

Despite recent calls for home healthcare (HHC) to integrate informatics, the application of machine learning in HHC is relatively unknown. Thus, this study aimed to synthesize and appraise the literature describing the application of machine learning to predict adverse outcomes (e.g., hospitalizations or mortality) using electronic health record (EHR) data in the HHC setting. Our secondary aim was to evaluate the comprehensiveness of the predictors used in the machine learning algorithms guided by the Biopsychosocial Model.

Methods:

We conducted a literature search in four databases: PubMed, Embase, CINAHL, and Scopus on March 2022. Inclusion criteria were 1) describing services provided in the HHC setting, 2) applying machine learning algorithms to predict adverse outcomes, defined as outcomes related to patient deterioration, 3) using EHR data and 4) focusing on adult population. Predictors were mapped to the Biopsychosocial Model. A risk of bias analysis was conducted using the Prediction Model Risk Of Bias Assessment Tool.

Results:

The final sample included 20 studies. Eighteen studies used predictors from standardized assessments integrated into the EHR. The most common outcome of interest was hospitalization (55%), followed by mortality (25%). Psychological predictors were frequently excluded (35%). Tree based algorithms were most frequently applied (75%). Most studies demonstrated high or unclear risk of bias (75%).

Conclusion:

Future studies in HHC should consider incorporating machine learning algorithms into clinical decision support systems to identify patients at risk. Based on the Biopsychosocial model, psychological and interpersonal characteristics should be used along with biological characteristics to enhance risk prediction. To facilitate the widespread adoption of machine learning, stakeholders should encourage standardization in the HHC setting.

Introduction

In the United States, home healthcare (HHC) is provided to approximately 5.2 million Medicare beneficiaries through 7.4 million home visits per year.¹ HHC patients include individuals recently discharged from an acute hospital stay or individuals who are managing chronic conditions in the community. The services provided during HHC include skilled nursing care, physical therapy, occupational therapy, speech therapy, and social work.² As part of routine HHC visits, clinicians assess for early signs of deterioration to reduce the risk of adverse events.³ Despite comprehensive assessments and interventions, nearly one in five patients are hospitalized or visit the emergency department (ED) during the time they receive HHC services.^4–6 Other initiatives aimed at hospitalization reduction include financial incentives;⁴ however, readmission rates have not improved.⁷ Estimates show that approximately 40% of hospitalizations and ED visits are due to preventable ambulatory care-sensitive conditions⁸ that could be avoided with timely and tailored interventions.⁹

In recent years, there has been growing evidence that machine learning algorithms can predict the risk of deterioration in patients by analyzing electronic health record (EHR) documentation.¹⁰ HHC agencies have their own EHR which provides a central place to store information describing patients’ diagnosis, problems, interventions, and outcomes. In HHC, nurses are the largest group of clinicians, hence nursing documentation accounts for a majority of EHR documentation. Other users of the EHR in HHC include physical, occupational, and speech therapists, and social workers.¹¹ Machine learning algorithms can identify patients who may experience an adverse event by discovering patterns in previously documented data.¹² The value of machine learning has been demonstrated in the acute care setting to detect various adverse events^13,14 such as mortality¹⁵ and the activation of rapid response teams.¹⁶ This has led to the development of clinical decision support systems that proactively notify clinicians to patients at risk for experiencing an adverse outcome.^17,18 However, the extent to which machine learning has been applied in the HHC setting for risk prediction has not been previously reported.

Another consideration when assessing the value of machine learning is understanding the predictors driving the algorithm. Previous studies have demonstrated the versatility of using machine learning to examine a wide range of predictors from symptoms to social determinants of health.^19,20 However, while it is known that social determinants²¹ and psychological factors influence health outcomes,²² the extent to which these factors are included in machine learning algorithms in the HHC setting is unknown.^19,20

Recent literature has called for an increase in informatics, such as machine learning, in the HHC setting to bring documentation-driven evidence to clinicians at the point of care.^23–25 Despite HHC being one of the fastest growing sectors of healthcare²⁶ and the benefits of machine learning to predict patients at risk,²⁷ no previous studies have summarized the state of the science on machine learning applied to EHR data in the HHC setting. To address these knowledge gaps, this study aims to critically appraise and synthesize information on the application of machine learning to predict a priori specified adverse outcomes using EHR data in the HHC setting. Our secondary aim was to summarize the different dimensions of predictors (i.e., biological, psychological, and interpersonal) used to build the machine learning algorithms guided by the Biopsychosocial Model.

Materials and methods

This scoping review was conducted and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines.²⁸ The protocol describing this study was registered in Open Science Framework (osf.io/g5zrj). Studies were eligible to be included if they addressed skilled health services provided in HHC, applied machine learning algorithms to predict adverse outcomes a priori, used EHR data, focused on the adult population, and were published in English. Full text papers and conference proceedings covering quantitative machine learning algorithms were included. For this study, adverse events were defined as outcomes related to patient deterioration.²⁹ Studies were excluded if they took place in a nursing home or skilled nursing facility or focused on smart homes, sensors, or robots because they were outside the scope of this manuscript. Qualitative studies, dissertation papers, and research proposals were excluded.

For a comprehensive synthesis of machine learning methodological approaches, we refined a previous literature query that examined the application of artificial intelligence in healthcare.³⁰ Four databases were searched (PubMed, Embase, CINAHL, and Scopus) in March 2022. There was no restriction on the publication date. The full search strategy can be found in Appendix A. Our screening team included two registered nurses and two advanced practice nurses with informatics expertise (MH, JS, DS, MT). After duplicates were removed, all relevant studies were screened by two reviewers. Any conflicts were resolved through discussion between the three reviewers until consensus was reached. This process was repeated with full-text studies.

The data extraction template was drafted by two co-authors (MH, MT). Other co-authors were invited to modify the template prior to charting the items. The extraction template included information about the study and sample characteristics, algorithm overview, machine learning approaches, and algorithm performance. Upon consensus with the appropriate variables to extract (e.g., purpose, data source, best-performing algorithm), two reviewers (JS, DS) independently charted the data using the software, Covidence. A third reviewer went through and resolved any disagreements (MH).

Data were synthesized by mapping predictors to the Biopsychosocial Model.^31–33 This model assumes that to understand health holistically, a range of characteristics should be considered, including an individual’s social, psychological, and behavioral dimensions.^31,32 Lehman et al adapted this model to include the following dimensions: biological, psychological, and interpersonal.³³ The biological dimension encompasses medical histories influencing the physical body; the psychological dimension encompasses coping and health behaviors; and the interpersonal dimension encompasses social support.³³ Two reviewers were tasked with categorizing the predictors into one of the three dimensions of the Biopsychosocial Model. Discrepancies were resolved through consensus meetings with the screening team (MH, JS, DS).

Four reviewers (MH, JS, DS, MT) performed a risk of bias analysis using the Prediction Model Risk of Bias Assessment Tool (PROBAST).³⁴ One reviewer (MH) resolved disagreements after discussing the conflicts with another member of the screening team who has expertise in machine learning (MT). PROBAST facilitates risk of bias assessment in four domains: participants, predictors, outcome, and analysis, and is specific to prediction model studies. Per the PROBAST guidelines, if a study has at least one criterion indicating an unclear or high risk of bias, the overall risk of bias for that study would be unclear or high.

Results

Our initial search yielded 1,644 studies from four databases (PubMed, Embase, CINAHL, and Scopus). After duplicates were removed (n = 190), there were 1,454 studies eligible for title and abstract screening. A total of 1,378 studies were excluded, and 76 studies were moved to full-text review. Of those, 20 studies were included for synthesis. Among the excluded studies (n = 56), studies were most commonly excluded because they did not predict an a priori specified adverse event (n = 32) or did not use EHR data (n = 17). The PRISMA flow diagram³⁵ presented in Figure 1 summarizes the study selection process.

Figure 1.

PRISMA Diagram

Study characteristics

Results revealed the global advances in integrating machine learning into the HHC setting. Six countries were represented in this review (30%). Ten studies (50%) did not include a clear definition of HHC in their region. Most studies took place in the United States (n = 9; 45%)^36–44 and Canada (n = 4; 20%)^45–48 and used data retrospectively collected from 2015 and earlier (n = 13; 65%)36–39,42–44,46,47,49–52 although some studies did not specify dates (n = 3; 15%).^40,48,53 The majority of studies were published between 2019 and 2022 (n = 13; 65%).^{37,40–44,46,47,49,51,52,54,55}

Sample characteristics

Most studies did not focus on patients with a specific disease (n = 12; 60%). Among studies that did focus on a disease-specific population, patients with heart failure (n = 3; 15%)^36,39,53 and advanced cancer (n = 2; 10%)^50,55 were the most common. The number of participants included in the studies ranged from 284 to 317,621. A summary of study and sample characteristics is provided in Table 1.

Table 1.

Study & Sample Characteristics

Author	Purpose	Disease Focus	Participants	Country	Retrospective Data Collection Period	HHC Definition
Calvo et al., 201951	To evaluate sampling approaches using blood tests to detect successful versus unsuccessful (i.e., hospitalization) HHC stays.	General HHC population	1,951	Spain	January 2012 – December 2015	Home-based medical and nursing services are provided when patients no longer need hospital facilities, but still require active and complex surveillance.
Calvo et al., 202052	To assess the performance of models for predicting mortality and in-hospital admission	General HHC population	1,936	Spain	January 2009 – December 2015	A service delivered by trained hospital personnel providing acute, home-based, complex interventions for a period that is not longer than the expected length of hospital stay.
Jones et al., 201845	To compare the performance of machine learning models predicting ED and hospital utilization among HHC patients.	General HHC population	88,345	Canada	2014 – 2016	Not specified
Jones et al., 201942	To develop a model to predict hospital readmissions from HHC in Medicare beneficiaries.	General HHC population	43,407	United States	January 2012 – January 2013	Home-based skilled services, including nursing and therapies (e.g., physical, occupational, and speech) for patients after hospitalization.
Kang et al., 201636	To examine risk factors for hospitalization among patients with heart failure who received telehomecare during their HHC episode.	Heart failure	552	United States	2010	Monitoring patients remotely using telehomecare.
Kuspinar et al., 201946	To develop an algorithm to predict falls among HHC patients.	General HHC population	126,703	Canada	2002 – 2014	Not specified
Lo et al., 201940	To identify factors that predict fall risks.	General HHC population	59,006	United States	Not specified	Not specified
Madigan and Curet, 200639	To determine the drivers of HHC service outcomes (i.e., discharge destination, length of stay).	COPD, Heart failure, Hip fractures	580	United States	2000	Not specified
McArthur et al., 202047	To develop a risk model to predict one-year incidence of hip fractures.	General HHC population	317,621	Canada	April 2011 – March 2015	Not specified
Nabal et al., 201450	To develop a model to predict survival based on symptoms identified by palliative HHC teams.	Advanced cancer	698	Spain	April 2009 – July 2009	Not specified
Nuutinen et al., 202049	To develop a model that predicts nursing home admission risk.	General HHC population	2,523	Finland	January 2012 – September 2015	Home-based services that are provided to patients for longer than 12 months.
Olson et al., 201638	To characterize patients at risk for medication-related hospital readmissions.	General HHC population	911	United States	2004	Not specified
Song et al., 202144	To identify risk factors for wound infection-related hospitalization or ED visits among patients who received HHC.	Wounds	54,316	United States	January 2014 – December 2014	HHC episodes typically last for 30 to 60 days or until the patient is discharged from HHC or admitted to the hospital or ED.
Song et al., 202241	To compare the performance of risk models that predict hospitalization and ED visits in HHC.	General HHC population	66,317	United States	January 2015 – December 2017	HHC episodes include services provided to a patient received between admission and discharge (30–60 days).
Sullivan et al., 202037	The determine how well the OASIS-SQ predicts ED mortality risk over time compared to alternative OASIS items.	General HHC population	69,097	United States	January 2012 – 2013	Nurses and therapists assess older adults residing in the community using the Outcomes and Assessment Information Set (OASIS).
Topaz et al., 202043	To identify patients at high risk for hospitalization or ED visits.	General HHC population	89,459	United States	January 2014 – December 2014	HHC encounters (e.g., nursing visits) typically happen a few days apart, and most of the data is generated by nurses and physical or occupational therapists.
Witt et al., 202254	To evaluate the performance of the model used to predict hospitalization in HHC patients.	General HHC population	1,282	Denmark	January 2016 – December 2017	Not specified
Yang et al., 202155	To predict survival and medical expenses among patients with advanced cancer receiving HHC.	Advanced cancer	310	China	January 2016 – December 2018	The Homecare Service Program for advanced cancer patients provides free medical services in patients’ homes.
Zhang et al., 201353	To identify patients recently discharged from the hospital at risk for death or hospitalization related to heart failure.	Heart failure	284	Germany, UK, Netherlan ds	Not specified	Not specified
Zhu et al., 200748	To explore if machine learning can predict rehabilitation potential.	General HHC population	24,724	Canada	1 year (no dates cited)	Not specified

^*HHC: Home healthcare; ED: Emergency department; OASIS: Outcome and Assessment Information Set

Algorithm overview

While all studies used data from the EHR, some studies included standardized assessments stored in the EHR (n = 18; 90%).^{36–50,52,54,55} Standardized assessments, housed within the EHR, include a set of standardized data elements allowing for easier comparison across different settings.⁵⁶ For example, the United States’ federally mandated Outcome and Assessment Information Set (OASIS) assesses the patient’s functional status by collecting data about activities of daily living using a standardized set of questions such as describing the patient’s ability to “transfer in and out of bed.”⁵⁷ In the United States, EHR data (e.g., demographics, clinical assessments) are typically collected by nurses within the HHC agency,¹¹ siloed from healthcare data collected in other care settings.²³ The most cited standardized assessment was the OASIS (n = 8; 40%)^{36–38,40–44} and Residential Assessment Instrument - Home Care (RAI-HC) (n = 5; 25%).^45–49 OASIS is a patient-specific assessment tool required by Medicare in the United States to guide a patient’s plan of care, determine reimbursement, and collect quality measures in HHC.⁵⁷ Similarly, the RAI-HC is used in Canada to assess “long stay home care clients’ health status, need for care, and basic background on housing and informal caregivers,” helping to guide care and collect quality indicators.⁵⁸ In the included studies, only three studies included clinical notes as a data source (15%).^41,43,44

Most studies considered variables from all three domains of the Biopsychosocial Model (n = 12; 60%): biological (e.g., age, hematocrit, fall history), psychological (e.g., depression, memory deficit, coping), and interpersonal (e.g., lives alone, social support, informal caregiver status).^{38,40–42,44–50,52} Predictors that did not map to one of the three domains were categorized as “Other” which included variables such as the number of days in the hospital, hospital utilization history, and the number of HHC visits. A list of variables from each study mapped to the Biopsychosocial Model is provided in Appendix B. Biological characteristics were included in all the studies. Psychological characteristics were excluded in seven studies.^{36,37,39,43,51,54,55} Interpersonal characteristics were excluded in three studies.^37,51,53

Most studies were aimed at predicting hospitalization (n = 11; 55%).^{36,38,41–45,51–54} The next most common outcome of interest was mortality (n = 5; 25%).^{37,50,52,53,55} Though less frequent, other studies focused on predicting additional adverse events including falls,^40,46 nursing home admission,^39,49 hip fractures,^39,47 and lack of rehabilitation potential.⁴⁸ Table 2 includes details about each study’s data source, predictors and outcomes included in the algorithms.

Table 2.

Algorithm Overview

Study	Data Source	Predictors’ Categories (Biopsychosocial Model)	Outcome of Interest
Calvo et al., 201951	EHR	Biological	Hospitalization
Calvo et al., 202052	EHR, Standardized assessments (GMA, SF-36, Barthel Index)	Biological, Psychological, Interpersonal, Other	Mortality and Hospitalization
Jones et al., 201845	EHR, Standardized assessments (RAI-HC)	Biological, Psychological, Interpersonal	Unplanned ED visit, Hospitalization
Jones et al., 201942	EHR, Standardized assessment (OASIS, Patient health questionnaire, Elixhauser comorbidity index), Medicare claims data, MEDPAR	Biological, Psychological, Interpersonal, Other	Hospitalization
Kang et al., 2016†36	Standardized assessments (OASIS)	Biological, Psychological, Interpersonal	Hospitalization
Kuspinar et al., 201946	Standardized assessments (RAI-HC)	Biological, Psychological, Interpersonal	Fall
Lo et al., 201940	EHR, Standardized assessments (OASIS)	Biological, Psychological, Interpersonal	Fall
Madigan and Curet, 200639	Standardized assessments (2000 National home and hospice care survey)	Biological, Interpersonal	Discharge destination, Length of stay
McArthur et al., 202047	Standardized assessments (RAI-HC)	Biological, Psychological, Interpersonal	Hip fracture
Nabal et al., 201450	EHR, Standardized assessments (Charlson scale, Cullum scale, Karnofsky index, Barthel index)	Biological, Psychological, Interpersonal	Mortality
Nuutinen et al., 202049	EHR, Standardized assessments (RAI-HC)	Biological, Psychological, Interpersonal, Other	Nursing home admission
Olson et al., 201638	EHR, Standardized assessments (OASIS)	Biological, Psychological, Interpersonal, Other	Hospitalization
Song et al., 202144	EHR, Standardized assessments (OASIS), Clinical notes	Biological, Psychological, Interpersonal, Other	Wound related hospitalization
Song et al., 202241	EHR, Standardized assessments (OASIS), Clinical notes	Biological, Psychological, Interpersonal, Other	Hospitalization or ED visit
Sullivan et al., 2020†37	Standardized assessment (OASIS)	Biological	Mortality
Topaz et al., 2020†43	Standardized assessment (OASIS), Clinical notes	Biological, Interpersonal	Hospitalization or ED visit
Witt et al., 202254	EHR, Danish national patient register, Standardized assessment (Triaged changing table)	Biological, Interpersonal	Hospitalization
Yang et al., 202155	EHR, Standardized assessment (Numeric pain rating scale, Karnofsky performance scale, Quality of life score)	Biological, Interpersonal	Mortality, Medical expenses
Zhang et al., 201353	EHR	Biological	Mortality, Hospitalization
Zhu et al., 200748	Standardized assessments (RAI-HC)	Biological, Psychological, Interpersonal	Lack of rehabilitation potential

^*EHR: Electronic Health Record; NA: Not Applicable; GMA: Adjusted Morbidity Groups; SF-36: Short Form Health Survey; RAI-HC: Residential Assessment Instrument-Home Care; ED: Emergency department; OASIS: Outcome and Assessment Information Set; MEDPAR: Medicare Provider Analysis Review; COPD: Chronic obstructive pulmonary disease

^†Although standardized assessments (e.g., OASIS) have biological, psychological, and interpersonal dimensions, some studies only used a subset from one or more domains as inputs for the predictive models.

^‡In this table, the term “EHR” covers variables not specific to standardized assessments. Standardized assessments (e.g., OASIS, RAI-HC) are listed separately.

Machine learning approaches

Of the machine learning approaches applied, most were categorized as supervised machine learning (n = 17; 85%).^{36,39–48,50–55} Supervised machine learning uses a deductive approach with a human-labeled dataset as a gold standard to train classification algorithms; unsupervised machine learning uses an inductive approach that does not include a labeled dataset and instead focuses on topic discovery.⁵⁹ Among the machine learning algorithms, tree based algorithms (e.g., Decision tree, ADABoost, Classification and regression trees, Random forest) were most commonly applied (n = 15; 75%).^{36,39–44,46,47,50–53,55} Decision tree,^{36,39,43,46,47,52,53} Logistic regression,^{41,42,44,45,52–54} and Random forest^{40,41,43–45,52,55} were each applied in seven studies (35%). Neural networks were applied in three studies (15%).^44,45,55 Alternatively, unsupervised approaches such as Clustering (n = 3; 15%) were less frequently applied in the included studies.^37,38,49 Three studies discussed using natural language processing (NLP) which supports the analysis of narrative text (e.g., clinical notes) (n = 3; 15%).^41,43,44

Algorithm performance

Area under the curve (AUC) was the most reported performance metric (n = 9; 45%).^{36,38,40,44–47,49,50,52–54} For the performance metrics, F-score, AUC, and c-statistics values ranged between 0–1, with higher values denoting better-performing algorithms.^60,61 The best performing algorithm varied across the different studies, as shown in Table 3 with the corresponding performance metrics.

Table 3.

Machine Learning Approaches and Algorithm Performance

Study	Machine Learning Algorithms	Best Algorithm Performance
Calvo et al., 201951	ADABoost	Precision (0.10) Specificity (0.12) Sensitivity (1) F-score (0.11)
Calvo et al., 202052	Decision trees, Logistic regression, Random forest	Random forest AUC (0.80) Sensitivity (0.75) Specificity (0.71)
Jones et al., 201845	Logistic regression, Neural networks, Random forest, Gradient boosted trees	Gradient boosted trees Logarithmic score (−0.30) Brier score (0.17) AUC (0.68)
Jones et al., 201942	Logistic regression, Gradient boosted trees	Gradient boosted trees c-statistic (0.67) Brier score (0.12)
Kang et al., 201636	Decision trees	c-statistic (0.59) Specificity (0.65) Sensitivity (0.49)
Kuspinar et al., 201946	Decision trees	c-statistic (0.60)
Lo et al., 201940	Random forest	AUC (0.67) Precision (0.10) Balanced Accuracy (0.62)
Madigan and Curet, 200639	Decision trees, Classification and regression trees	Classification and regression trees Classification rate (0.71)
McArthur et al., 202047	Decision trees	Decision trees c-statistic (0.66)
Nabal et al., 201450	Classification and regression trees	Decision trees AUC (0.88)
Nuutinen et al., 202049	Principle component analysis, K means clustering	ACC (0.80) AUC (0.80) TPR (0.44) FPR (0.12) FNR (0.56) TNR (0.88)
Olson et al., 201638	K means clustering, Hierarchical clustering	AUC (0.77)
Song et al., 202144	Logistic regression, Neural networks, NLP, Random forest	Logistic regression Sensitivity (0.88) Specificity (0.64) PPV (0.03) NPV (0.99) AUC (0.82)
Song et al., 202241	Logistic regression, SVM, Random forest, Naïve bayes, NLP, Bayesian network	Random forest Sensitivity (0.93) PPV (0.72) F-score (0.81) PRC (0.86)
Sullivan et al., 202037	K means clustering, Survival analysis	PPV (0.49) NPV (0.72) K (0.20; 95%CI: [0.19, 0.21])
Topaz et al., 202043	Decision trees, Random forest, Naïve bayes	Random forest Recall (0.81) Precision (0.83) F-score (0.82) PRC (0.76)
Witt et al., 202254	Logistic regression, RUSBoost	RUSBoost AUC (0.99) PRC (0.71)
Yang et al., 202155	Neural networks, SVM, Random forest	Random forest Accuracy (0.82) Normalized mean square error (0.42)
Zhang et al., 201353	Chi-square automatic interaction detector decision tree, Logistic regression	Chi-square automatic interaction detector decision tree AUC (0.80)
Zhu et al., 200748	K nearest neighbor	FPR (0.34) FNR (0.36) DLR+ (1.88) DLR− (0.55)

^*AUC: Aera Under the Curve; TPN: True-positive-rate; FPN: False positive-rate; FNR: False-negative-rate; TPN: True-negative-rate; PPV: Positive Predictive Value; NPV: Negative Predictive Value; NLP: Natural language processing; SVM: Support Vector Machines; RUSBoost: Random UnderSampling and Boosting; PRC: Precision-Recall Curve; FPR: False positive rate; FNR: False negative rate; DLR+: Positive diagnostic likelihood ratio: DLR−: Negative diagnostic likelihood ratio

Risk of bias analysis

Similar to recent literature, most studies demonstrated high or unclear risk of bias.^62,63 Of the studies screened with PROBAST, five (25%) demonstrated an overall low risk of bias,^{36,42,43,45,52} seven demonstrated an overall unclear risk of bias (35%),^{38,40,46,47,49,50,53} and eight demonstrated an overall high risk of bias (40%).^{37,39,41,44,48,51,54,55} Studies most frequently (n = 8; 40%) had an overall unclear risk of bias^{38,40,46,47,49,50,53} because there was a lack of detail about overfitting,^37,46,53 handling of missing data,^{40,46,47,49,50,53} use of univariable analysis,^47,49,50 and handling of complexities in the data.^{38,40,46,47,49,50} Most commonly, studies were designated high risk of bias because they selected predictors using univariable analysis,^41,44,51 did not account for model overfitting,³⁹ assigned weights in the final model based on the multivariable analysis,³⁹ did not handle missing data appropriately,⁵⁴ and did not evaluate model performance measures appropriately.^48,55 Full results of the PROBAST risk of bias analysis are provided in Table 4.

Table 4.

Risk of bias analysis

Study	Participants	Predictors	Outcome	Analysis	Overall
Calvo et al., 201951	Unclear	Low	Low	High	High
Calvo et al., 202052	Low	Low	Low	Low	Low
Jones et al., 201845	Low	Low	Low	Low	Low
Jones et al., 201942	Low	Low	Low	Low	Low
Kang et al., 201636	Low	Low	Low	Low	Low
Kuspinar et al., 201946	Low	Low	Low	Unclear	Unclear
Lo et al., 201940	Low	Low	Low	Unclear	Unclear
Madigan and Curet, 200639	Low	Low	Unclear	High	High
McArthur et al., 202047	Low	Unclear	Unclear	Unclear	Unclear
Nabal et al., 201450	Low	Low	Low	Unclear	Unclear
Nuutinen et al., 202049	Unclear	Unclear	Unclear	Unclear	Unclear
Olson et al., 201638	Low	Unclear	Unclear	Unclear	Unclear
Song et al., 202144	Low	Low	Unclear	High	High
Song et al., 202241	Low	Low	Unclear	High	High
Sullivan et al., 202037	Low	Low	High	Unclear	High
Topaz et al., 202043	Low	Low	Low	Low	Low
Witt et al., 202254	Low	Low	Low	High	High
Yang et al., 202155	Low	Low	Low	High	High
Zhang et al., 201353	Low	Low	Low	Unclear	Unclear
Zhu et al., 200748	Unclear	Low	Low	High	High

^*In this table, low indicates low bias indicating better quality

Discussion

This scoping review included 20 studies that described the application of machine learning to predict a priori specified adverse outcomes using EHR data in the HHC setting. Although similar reviews were conducted in the acute and ambulatory settings,^64,65 this study was the first to focus on the HHC setting, which represents one of the fastest-growing healthcare sectors.²⁶

Most studies (65%) included in this review were published between 2019 and 2022, suggesting a recent uptake of machine learning in the HHC setting.^64,66 With that, 65% of the studies used data collected from 2015 or earlier, indicating a delay between retrospective data availability and analysis. This may be related to the lack of regulation and standardization of EHR systems and data collected in the HHC setting globally, which may increase the difficulty and time needed to extract, clean, and analyze the data.^67,68 Generating global policies similar to the Improving Medicare Post-Acute Care Transformation (IMPACT) Act, which advances standardized data in post-acute care settings⁶⁹ could lead to improved interoperability and analytical efficiency in the near future.

This review paper highlights a gap in current research around the implementation of machine learning algorithms into the HHC clinical setting.³⁰ A previous systematic review describing prediction algorithms in post-acute care found that among the 37 models reviewed, very few were ever implemented in practice.⁷⁰ While clinical validation is underway in the acute care setting through randomized controlled trials,^71–73 there is still a significant gap in the HHC setting. Shifting focus from an end goal of performance optimization to practical clinical implementation is imperative to achieve the intention of almost all health-related machine learning algorithms – to advance health and improve outcomes. Seneviratne et al recommends that when building machine learning algorithms, developers should proactively consider clinical actionability, patient safety, and cost-utility to optimize implementation in the clinical setting.⁷⁴

Although this study examined the global scope of machine learning in HHC across six countries, it is important to highlight some regional differences. For example, in the United States, HHC is defined as a period of 30 to 60 days where skilled services such as nursing, physical, occupational, and speech therapy are provided to help patients transition back to the community.^42,44 Alternatively, in Spain, the purpose of HHC is to provide interventions aimed at substituting inpatient hospitalization in a period of time that does not exceed the expected length of an inpatient hospitalization.⁵² Concerningly, half of the studies included in this review did not provide detailed information on the structure of HHC in their country. The differences in HHC structure and purpose also influence EHR documentation⁷⁵ which was not thoroughly discussed in all articles. Given the variation of HHC and EHRs globally,^76–78 future studies should provide a detailed description of their regional HHC structure and EHR systems to help distinguish differences in patient populations, goals, timelines, and care structures. Providing further clarity around regional differences in the structure of HHC and EHR systems could lead to better tailoring and predictor selection when developing machine learning algorithms globally.

Standardized assessments, such as OASIS and the RAI-HC, are commonly stored in the EHR. Using standardized assessments can support interoperability, or the ability to exchange health information, across systems or within a health system.⁷⁹ In addition, the utilization of standardized assessments has led to more evidence-based, consistent, and holistic assessments among clinicians.^80,81 In fact, all three domains of the Biopsychosocial Model were aligned with studies that used a standardized assessment tool as their data source, suggesting that standardized assessments lead to more comprehensive documentation. Thus, incorporating standardized assessments into the EHR could improve comparability, access to data, interoperability, and generalizability in the HHC setting.

Our results reveal that most studies include predictors from all three domains of the Biopsychosocial Model. Unsurprisingly, and similar to a previous study,⁸² biological characteristics were included in all studies. Fewer studies included psychological (e.g., mental health) and/or interpersonal (e.g., social determinants of health) predictors in their machine learning algorithms. Of the studies that did include these dimensions, they were often from the structured standardized assessments. For psychological predictors, this may be related to using an assessment tool that does not comprehensively consider psychological characteristics. For interpersonal predictors, this might be related to limited data capture and documentation of social determinants of health in EHR systems.⁸³ Previous studies have suggested that including social determinants of health can improve machine learning prediction among individuals from racial or ethnic minorities.^21,84,85 The inclusion of psychological and interpersonal characteristics in machine learning algorithms could lead to improved equity in risk identification⁸⁵ and proactive interventions for patients with mental health conditions or social risk factors who are often stigmatized in healthcare.^86,87

The most frequently cited outcome of interest was hospitalization. In HHC, one of the primary goals is to prevent avoidable hospitalizations.⁸⁸ Hospitalization rates reflect care quality,⁴ with reducing readmission being financially incentivized;⁴ thus, we anticipate the number of machine learning algorithms focused on hospitalization to continue to be common. Other outcomes specific to HHC, such as self-management, should be explored in future studies to investigate other ways to reduce the risk of specific adverse outcomes prior to hospitalization.⁸⁹ In terms of disease conditions, a majority of the studies in this review did not focus on patients with a specific disease but rather on the general HHC population. The few studies that specified a condition focused on patients with heart failure or cancer. Future studies might examine specific disease-related predictors to identify characteristics that put specific populations at risk and inform the development of tailored interventions to improve patient outcomes.

The wide range of sample sizes in the included studies demonstrates the versatility of machine learning algorithms in studies with various sample sizes.⁹⁰ Although small samples can be problematic for specific algorithms, such as Neural networks, other algorithms, such as Naïve bayes, can better accommodate smaller samples since it assumes class independence.⁹¹ However, data-demanding algorithms like Neural networks may be more prone to overfitting with smaller samples, leading to overestimated algorithm performance.⁹² Thus, we encourage researchers to consider sample size when determining the appropriate algorithm to apply to the data.

Tree based algorithms (e.g., Decision tree, ADABoost, Classification and regression trees, Random forest) were frequently cited in this review and are known to be a reliable method in biomedical literature.⁹³ An advantage of a Decision tree is its interpretability,⁹⁴ meaning that humans can understand the informative features driving the algorithm’s performance.⁹⁵ Alternatively, “black box” algorithms, such as Neural networks, are less interpretable, which may indicate why they were used less frequently⁹⁴ in studies included in this review. However, there has been a rise in the use of Neural networks^96,97 applied to EHR data, given its increased performance and reduction in preprocessing and feature engineering compared to other explainable algorithms.⁹⁷ Recent literature has published model-agnostic methods to increase the interpretability of “black box” models, such as Shapley Values, which help to explain features that are more informative (i.e., feature importance).⁹⁵ Therefore, future studies may be more apt to apply Neural networks in the HHC setting. However, the studies in this review that applied Neural networks did not outperform alternative algorithms, suggesting that other models should not be abandoned altogether.

Additionally, NLP, an algorithm used to process unstructured text in clinical notes, was infrequently applied in studies included in this review. NLP is a method that takes text contained in clinical notes and makes it accessible to machine learning algorithms by transforming the text into a structured form.⁹⁸ Eighty percent of healthcare data is unstructured data (e.g., clinical notes).⁹⁹ Previous studies have demonstrated the value of analyzing clinical notes to reflect clinician concerns to improve risk prediction in HHC.^44,100–102 More specifically, studies have found that additional information about patients’ social determinants¹⁰³ and mental health¹⁰⁴ can be found in clinical notes. Thus, future studies in HHC might explore using NLP to extract additional information about a patient’s biological, psychological, and interpersonal characteristics to improve risk predictive performance.

Overall, there were variations in algorithm performance among the included studies; however, comparing algorithms with different performance metrics is challenging. Each performance metric has its advantages and disadvantages. F-score, which handles imbalanced data, gives more weight to the majority class regardless of if it is the outcome of interest.¹⁰⁵ AUC is better for ranking overall prediction because it equally weights both classes.¹⁰⁵ The variety of measures used to report algorithm performance suggests no general consensus on a particular measure of best use.¹⁰⁶ Therefore, it is important that the selection of performance metrics is thoughtfully considered and explained in the manuscript.¹⁰⁷

Limitations

A few limitations of this scoping review need to be addressed. First, we recognize that sub-optimal machine learning performance may be driven by the dataset’s quality and influence the results. Additionally, the development of machine learning in recent years may have contributed to improved machine learning performance. The scope of this study did not include an in-depth analysis of how the data was collected, but future studies should consider the bias contained in these data. Similarly, no studies rigorously evaluated bias (e.g., racial, age) in algorithm performance recognizing an important gap to consider in future research. This review did not find HHC studies that compared prediction of machine learning models with clinical expert judgment and we recommend exploring this comparison in the future studies. This scoping review contains multiple studies published by the same research team, which suggests that minimal teams have published in this area within the last few years, potentially skewing the results. We mapped predictors to the Biopsychosocial Model to get a comprehensive overview of predictors considered in the included machine learning models; however, an alternative framework may have led to a more granular categorization of predictors.

Conclusion

With the increased demand for healthcare provided in the home, we anticipate the number of machine learning algorithms in HHC to continue to increase. Results from this review demonstrate the feasibility and potential of machine learning to support clinical care in the HHC setting. Future studies should consider how these machine learning algorithms can be packaged into decision support systems that can guide clinical decision-making at the point of care to reduce adverse events, such as hospitalizations. In addition, incorporating psychological and interpersonal characteristics into machine learning models will support comprehensive, holistic risk prediction. To enable widespread adoption of machine learning, stakeholders should consider how they can further promote the standardization of documentation elements in the HHC setting.

Summary table.

What is already known on the topic

• Machine learning is a powerful technique used to help predict adverse events.

• Machine learning applied to EHR data in HHC is bringing documentation-driven evidence to clinicians at the point of care.

What this study adds to our knowledge

• Summarizes the literature of machine learning used in the HHC setting to predict adverse events.

• Explores the basic components of the machine learning algorithms applied to EHR data in the HHC setting to predict adverse events.