Evaluation of machine learning methodology for the prediction of healthcare resource utilization and healthcare costs in patients with critical limb ischemia—is preventive and personalized approach on the horizon?

Abstract

Background

Critical limb ischemia (CLI) is a severe stage of peripheral arterial disease and has a substantial disease and economic burden not only to patients and families, but also to the society and healthcare systems. We aim to develop a personalized prediction model that utilizes baseline patient characteristics prior to CLI diagnosis to predict subsequent 1-year all-cause hospitalizations and total annual healthcare cost, using a novel Bayesian machine learning platform, Reverse Engineering Forward Simulation™ (REFS™), to support a paradigm shift from reactive healthcare to Predictive Preventive and Personalized Medicine (PPPM)-driven healthcare.

Methods

Patients ≥ 50 years with CLI plus clinical activity for a 6-month pre-index and a 12-month post-index period or death during the post-index period were included in this retrospective cohort of the linked Optum-Humedica databases. REFS™ built an ensemble of 256 predictive models to identify predictors of all-cause hospitalizations and total annual all-cause healthcare costs during the 12-month post-index interval.

Results

The mean age of 3189 eligible patients was 71.9 years. The most common CLI-related comorbidities were hypertension (79.5%), dyslipidemia (61.4%), coronary atherosclerosis and other heart disease (42.3%), and type 2 diabetes (39.2%). Post-index CLI-related healthcare utilization included inpatient services (14.6%) and ≥ 1 outpatient visits (32.1%). Median annual all-cause and CLI-related costs per patient were $30,514 and $2196, respectively. REFS™ identified diagnosis of skin and subcutaneous tissue infections, cellulitis and abscess, use of nonselective beta-blockers, other aftercare, and osteoarthritis as high confidence predictors of all-cause hospitalizations. The leading predictors for total all-cause costs included region of residence and comorbid health conditions including other diseases of kidney and ureters, blindness of vision defects, chronic ulcer of skin, and chronic ulcer of leg or foot.

Conclusions

REFS™ identified baseline predictors of subsequent healthcare resource utilization and costs in CLI patients. Machine learning and model-based, data-driven medicine may complement physicians’ evidence-based medical services. These findings also support the PPPM framework that a paradigm shift from post-diagnosis disease care to early management of comorbidities and targeted prevention is warranted to deliver a cost-effective medical services and desirable healthcare economy.

Electronic supplementary material

The online version of this article (10.1007/s13167-019-00196-9) contains supplementary material, which is available to authorized users.

Introduction

Critical limb ischemia (CLI) is a serious form of peripheral artery disease (PAD). It affects approximately 1% of patients diagnosed with PAD in the United States (US) [1] and is associated with significant morbidity and mortality [1, 2]. Patients with CLI are particularly predisposed to a variety of PAD-related complications, including major amputations, cardiovascular disease, cerebrovascular events, disability, and death due to cardiac and non-cardiac causes [3–6], leading to a substantial disease and economic burden from patients, families, to healthcare systems.

Healthcare resource utilization among CLI patients is high, has increased in recent years, and is associated with substantial direct and indirect medical costs [1, 4, 5, 7–12]. In 2014, Medicare patients with CLI accounted for 173,000 hospital admissions, 56,000 inpatient surgical or endovascular revascularizations, and 37,000 major lower extremity amputations. There were 257,000 hospital-based outpatient cases for CLI among Medicare patients alone in 2013 [12]. Analysis of the 2013 to 2014 Nationwide Readmission Databases (NRD), which includes all discharge records of patients admitted to US community hospitals excluding rehabilitation and long-term care facilities, indicated a total of 60,998 index CLI hospitalizations. The 30-day readmission rate was 20.4% for hospitalized patients for CLI who underwent endovascular or surgical interventions [13]. Similar results were evident in an analysis of the State Inpatient Databases for Florida (2009 to 2013), New York (2010 to 2013), and California (2009 to 2011), with unplanned 30- and 60-day readmission rates of 23.6% and 47.7%, respectively [14]. The aggregate cost of all index hospitalizations for CLI exceeded $4.2 billion based on the 2013 to 2014 NRD [13]. Inpatient and outpatient care of CLI in Medicare patients was estimated at $3.6 billion in 2014, which is considered an underestimate of the total cost of care for CLI patients because physician visits and services provided in other clinical settings are not included in the amount borne by Medicare [12]. Resource utilization and costs are greater for elderly CLI patients and those with more severe CLI or comorbid health conditions such as diabetes [15].

Risk prediction and early prevention has the potential to evaluate and improve the use of healthcare resources and reduce costs [16–18]. Among CLI patients, the innovative PPPM concepts have the potential to shift disease care from tertiary to secondary prevention in relation to disease and economic burden. The White Paper of the “European Association for Predictive, Preventive and Personalised Medicine (PPPM)” (EPMA) [19] suggested that model-based medicine, i.e., the patient-specific model that allows for probabilistic prediction of an individual patient’s potential health outcome based on a holistic approach such as machine learning and big data, could be a valuable supplement to evidence-based medicine.

Data are critical to the development of accurate risk prediction models to guide clinicians’ decision-making, identify patients who will obtain the greatest benefit from specific interventions, and predict outcomes [20] such as hospital readmissions or amputations. Machine learning uses statistical techniques and algorithms to model large clinical datasets [21] and identify patterns in large datasets [20, 22, 23]. Machine learning is similar to regression modeling [20, 24], with the added advantage of using computer programs to handle large sets of predictor variables and patients without introducing bias. Machine learning also eliminates the need for a predefined set of predictors based on published research and the experience and judgment of investigators, with the unique advantage of potentially identifying novel predictors. Machine learning has been used to predict treatment responses, develop prognostic models, increase the accuracy of diagnosis, and generate predictions for cost analyses [20, 23].

This study aimed to analyze demographic and disease profiles of CLI patients and associated healthcare resource utilization and cost burden and to develop a personalized prediction model that utilizes baseline patient characteristics prior to CLI diagnosis to predict subsequent 1-year all-cause hospitalizations and total annual healthcare cost, using a novel Bayesian machine learning platform, Reverse Engineering Forward Simulation™ (REFS™), to support a paradigm shift from reactive healthcare focusing more in late-stage disease management to PPPM-driven healthcare and to empower early prediction, targeted prevention, and personalized medicine.

Methods

Data source

This was a retrospective cohort analysis of the linked Optum administrative claims and electronic health record (EHR) databases over an observation period of 1 January 2007 through 30 September 2015. Optum coordinates Clinformatics, a real-world database of 145 million unique patients residing in the US. Clinformatics contains information on administrative claims, medical and pharmacy claims, and laboratory results for enrollees in Medicare Part D only or medical plus Part D coverage and all types of commercial insurance plans. Humedica obtains EHR data supplemented with claims, prescription, and practice management information for more than 28 million patients in the US, with inpatient and outpatient data provided by hospitals, outpatient clinics, and physicians.

A person-level de-identification algorithm integrates data from the Optum Clinformatics and Humedica databases to yield comprehensive health information for 5.1 million unique patients in 20 states in the US. Administrative information, physician and facility claims, pharmacy claims, and laboratory results are available at the level of individual patients. More than 4 years of data are available for 80% of patients, providing a longitudinal perspective on the continuum of patient care. All data are de-identified at the patient and provider levels, ensuring full compliance with the Health Insurance Portability and Accountability Act of 1996.

Subjects

Patients who were 50 years or older at the index date with a confirmed diagnosis of CLI were included in this analysis. The International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for atherosclerosis of native arteries of extremities with rest pain (440.22), ulceration (440.23), or gangrene (440.24) or embolism and thrombosis of abdominal aorta (444.0), or atheroembolism of the lower extremities (445.02) identified patients diagnosed with CLI, with the date of the first available diagnosis of CLI considered the index date. Additional eligibility criteria included clinical activity for the 6-month pre-index and the 12-month post-index periods or confirmation of death during the 12-month post-index interval. To assess economic outcomes, 12 months continuous enrollment in the post-index period was also required. Exclusion criteria included a diagnosis of intracranial hemorrhage (432.9), ischemic stroke (433.X1, 434.X1, 436, and 437.1), or transient ischemic attack (TIA) (435.9) during the pre-index period.

Variables and assessments

Outcomes

The primary outcomes were healthcare utilization and costs during the post-index period, starting 1 day after the index date. Healthcare resource utilization included any all-cause and CLI-related inpatient visits defined as any healthcare services provided by an inpatient hospital, skilled nursing facility, custodial care facility, hospice, inpatient psychiatric facility, intermediate care facility for intellectually disabled, residential substance abuse treatment facility, psychiatric residential treatment facility, and comprehensive inpatient rehabilitation facility. All-cause and CLI-related acute inpatient visits (hospitalizations), outpatient visits, and emergency department (ED) visits were also tracked. Total annual all-cause and CLI-associated healthcare costs were determined for all inpatient, acute inpatient, outpatient, ED, and pharmacy services.

Potential predictors

The REFS model evaluated associations between the full set of predictor variables and the outcomes of all-cause hospitalizations and total annual all-cause healthcare costs. The models were run without pre-specified hypotheses, which included but were not limited to patient demographic (age, sex, race, ethnicity, type of health plan, and location of residence at the index date) and clinical characteristics (comorbid health conditions and medications commonly associated with CLI, tobacco use, alcohol use, measures of health status, and CLI severity score). All covariates were collected during the 6-month pre-index period. Comorbidities were identified by ICD-9-CM codes with these codes mapped to Clinical Classification Software level 2 and 3 codes. Tobacco and alcohol status were assessed closest to the index date.

We also calculated the Charlson Comorbidity Index (CCI) [25, 26] score based on ICD-9-CM codes. The CCI is a validated measure of 1-year mortality risk and burden of disease [26]. A CLI severity score was calculated for each patient. This score was computed by summing the total number of PAD symptomology or sign measures using the ICD-9-CM codes including rest pain, ulceration, gangrene, and amputation as an indication of disease severity. Higher scores indicated more severe CLI. Medications including antiplatelets, anticoagulants, lipid-lowering agents, and medications for the treatment of heart failure, angina, or hypertension that were prescribed during the 6-month pre-index period were also included in the model.

Statistical analysis

Descriptive statistics were calculated for all variables, including frequencies and percent responses for categorical variables and means and standard deviation (SD) for continuous variables; if continuous variables are skewed, median and interquartile range (IQR) are reported instead. Healthcare resource utilization was analyzed as a continuous variable as well as a categorical variable based on the number of times a patient was hospitalized during the pre-index period using the categories of 0, 1, or ≥ 2 hospitalizations.

We did the predictive ensemble of models using the REFS platform (GNS Healthcare, Cambridge, MA, USA) to evaluate associations between the full set of predictor variables in the aggregate Optum and Humedica databases and the each study outcome. REFS is a proprietary machine learning platform based on a Bayesian ensemble-model predictive analytics approach [27–29]. Non-informative priors, which have little-to-no influence on the predictor effects learned from the data, were used to develop the REFS ensemble. This approach decreases the likelihood of over-fitting models and generates a framework to estimate the distributions of individual variable effects.

Each element of an ensemble is called a network, which is a generalized linear model. To create each network, REFS scores the posterior probability of many putative models using a maximum entropy structural prior [27]. REFS generates 256 models for the ensemble to ensure the best fit. The Bayesian score for a model is approximated by marginalizing out model parameters and applying the Bayesian Information Criterion, which penalizes complexity. We used a Markov Chain Monte Carlo approach to generate samples from the equilibrium distribution of models weighted by their score. Each successive evaluation makes a slight transformation to the model by adding or removing a single predictor variable. A simulated annealing approach to obtain samples from the desired posterior distribution was used to accelerate convergence. This approach is particularly well suited to small sample, high-dimensionality datasets, such as used here, when gradient-based learning runs into problems due to the Tanner-Donoho phase boundary [28]. The model performance was assessed using five-fold cross-validated area under the receiver operating curve (AUC) for healthcare utilization and R² for cost.

From the ensemble for each study outcome, predictors can be ranked and evaluated by their edge frequency (EF) (proportion of models in the ensemble in which the variable is selected) and distributions of effect estimates across models in which the variable is selected. EF is a measure of the degree to which a predictor is related to the outcome, and a higher EF for a given predictor represents an increased probability of a true predictive association with the outcome [29]. For the effect estimates, the average odds ratios (AORs) for the all-cause hospitalizations outcome (binary: any vs. none), and the average percentage change in costs (APCCs) was calculated to asses impact on the total annual all-cause healthcare costs (continuous: log-transformed). Small SDs for the AOR or APCC indicate similarity between the AORs or APCCs for a specific predictor resulting from different models. The interpretation of the effect estimates is more meaningful when selected variables are considered “high confidence predictors”—defined as those appearing in more than 5% of the models of an ensemble.

Results

Demographic and clinical characteristics

The study included 3,189 patients after inclusion criteria were applied (Fig. 1). Mean patient age was 71.9 years (SD, 9.0) and 47.5% were ≥ 75 years. More than one-half of patients were male, 74.0% were white, 2.7% were Hispanic, and the majority resided in census region of the Midwest (35.7%) or the South (33.9%). Commercial insurance was the most common insurance plan type (Table 1).

Fig. 1.

Disposition of subjects from the integrated Optum and Humedica administrative claims and electronic health record databases: January 1, 2007 through September 30, 2015

Table 1.

Demographic and clinical characteristics of the patient cohort

Demographic Characteristics at the Index Date	N = 3,189
Age group, n (%)
50–54	160 (5.02)
55–64	550 (17.25)
65–74	963 (30.20)
≥ 75	1,516 (47.54)
Age, mean (SD)	71.9 (9.02)
Sex, n (%)
Female	1,515 (47.51)
Male	1,672 (52.43)
Unknown	2 (0.06)
Insurance plan type, n (%)
Commercial	802 (25.15)
Medicaid	24 (0.75)
Medicare	684 (21.45)
Other	34 (1.07)
Uninsured	34 (1.07)
Unknown	1,611 (50.52)
Geographic location, n (%)
Northeast	461 (14.46)
Midwest	1,137 (35.65)
South	1,081 (33.9)
West	419 (13.14)
Unknown	91 (2.85)
Race, n (%)
African American	297 (9.31)
Asian	25 (0.78)
White	2,361 (74.04)
Other or unknown	506 (15.87)
Clinical characteristics during the 6-month pre-index period	N = 3,189
CLI-related comorbid health conditions, n (%)
Hypertension	2,536 (79.52)
Dyslipidemia	1,959 (61.43)
Coronary artery disease	1,357 (42.55)
Atherosclerosis and other heart disease	1,348 (42.27)
Acute myocardial infarction	188 (5.90)
Acute and unspecified renal failure	452 (14.17)
Congestive heart failure: non-hypertensive	745 (23.36)
Cardiac dysrhythmias	991 (31.08)
Atrial fibrillation and atrial flutter	630 (19.76)
Amputation: all	83 (2.60)
Amputation: minor	58 (1.82)
Amputation: major	25 (0.78)
Type 1 diabetes	333 (10.44)
Type 2 diabetesa	1,251 (39.23)
Diabetes with neurological manifestation	613 (19.22)
CCI score, mean (SD)a,b	6.91 (2.78)
CLI Severity score, n (%)
0	910 (28.54)
1	1,929 (60.49)
2	301 (9.44)
3	49 (1.54)
Medicationsc, n (%)
Anti-angina
Beta-blocker	1,317 (41.30)
Diuretic	1,184 (37.13)
ACE inhibitor	971 (30.45)
Calcium channel blocker	829 (26.00)
Angiotensin receptor blocker	336 (10.54)
Anticoagulant
Heparin	453 (14.20)
Coumarin	355 (11.13)
Thrombin inhibitor	49 (1.54)
Xa inhibitor	19 (0.6)
Antidiabetic
Insulin	628 (19.69)
Sulfonylurea	396 (12.42)
Biguanide	396 (12.42)
Thiazolidinedione	94 (2.95)
DPP-4 inhibitor	87 (2.73)
GLP-1 receptor agonist	21 (0.66)
Meglitinide analogue	17 (0.53)
Amylin analogue	2 (0.06)
Antiplatelet
Aspirin	381 (11.95)
Thienopyridine	561 (17.59)
NSAID	419 (13.14)
Lipid-lowering
Statin	1,416 (44.40)
Fibrate	170 (5.33)

ACE angiotensin-converting enzyme, AIDS acquired immunodeficiency syndrome, CCI Charlson Comorbidity Index, CHF congestive heart failure, CKD chronic kidney disease, CLI critical limb ischemia, COPD chronic obstructive pulmonary disease, DPP-4 dipeptidyl peptidase-4, ESRD end-stage renal disease, GLP-1 glucagon-like peptide-1, HIV human immunodeficiency virus, NSAID non-steroidal anti-inflammatory drug, SD standard deviation

^aDiabetes computed in Charlson Comorbidity Index included only ICD-9 codes: 250.0X-250.7X

^bOverall, 92.7% of patients had a CCI score ≥ 4

^cPatients could be prescribed multiple, concurrent medications

The most frequently reported CLI-related comorbid health conditions were hypertension (79.5%), dyslipidemia (61.4%), coronary atherosclerosis and other heart disease (42.3%), and type 2 diabetes (39.2%; Table 1). The mean CCI score for the pre-index period was 6.91 (SD, 2.78), with at least one of the CLI severity score factors reported for 71.5% of patients. The most frequently reported medications used during the pre-index interval were statins (44.4%), beta-blockers (41.3%), diuretics (37.1%), and angiotensin-converting enzyme (ACE) inhibitors (30.5%).

All-cause and CLI-related healthcare resource utilization

During the 12-month post-index period, the median number of all-cause inpatient visits was 1 (IQR, 0–3), with a median length of stay (LOS) of 7 days (IQR, 3–16) among admitted patients. The median number of all-cause acute inpatient visits was 1 (IQR, 0–2). At the median, patients had 23 (IQR, 11–46) all-cause outpatient visits and 0 (IQR, 0–2) ED visits. The median number of CLI-associated inpatient visits was 0 (IQR, 0–0) with a median LOS of 5 days (IQR, 2–9) among admitted patients. The median number of CLI-associated acute inpatient visits was 0 (IQR, 0–0). The median number of CLI-related outpatient and ED visits were 0 (IQR, 0–1) and 0 (IQR, 0–0), respectively.

Approximately 60% of patients had at least one all-cause inpatient visit and 53.6% had at least one all-cause acute inpatient visit during the post-index period. Almost all patients had two or more all-cause outpatient visits and 44.1% had at least one all-cause ED visit during the post-index period (Fig. 2). CLI-related inpatient visits and acute inpatient visits during the post-index interval were reported for 14.6% and 13.7% of patients, respectively, with at least one CLI-related outpatient visit documented for approximately 32% of patients. One or more CLI-related ED visits was reported for less than 1% of patients during the post-index period (Supplemental Fig. 1).

Fig. 2.

Annual all-cause healthcare resource utilization during the 12-month post-index period

All-cause and CLI-related healthcare costs

The median annual all-cause healthcare costs per patient for the post-index period was $30,514 (IQR, 11,542–74,573), with all inpatient services accounting for the highest costs at $27,296 (IQR, 9,139–65,929) followed by acute inpatient services ($26,426; IQR, 9,321–62,210) and outpatient services ($9,946; IQR, 4,385-22,487) among patients with at least one associated visit. Pharmacy costs and ED visits contributed lesser amounts to the median annual all-cause expenditures per patient (Fig. 3).

Fig. 3.

Total annual all-cause healthcare costs during the 12-month post-index period (median $PPPY among patients with at least 1 associated visit). Abbreviation: PPPY per patient per year; ED emergency department

Evaluation of the median annual CLI-related costs per patient during the post-index interval revealed total healthcare costs of $2,196 (IQR, 596–7,691), with all inpatient services accounting for $27,296 (IQR, 9,139–65,929) and $26,426 (IQR, 9,321–62,210) for acute inpatient care among patients with at least one associated visit. Costs for outpatient services and pharmacy were substantially lower (Supplemental Fig. 2).

REFS model for all-cause hospitalizations

Predictors of all-cause hospitalizations identified by the REFS model are shown in Table 2. The top 5 predictors selected with the highest frequency in the model ensemble included skin and subcutaneous tissue infections (57.8%), cellulitis and abscess (27.0%), nonselective beta-blockers (12.5%), other aftercare (12.5%), and osteoarthritis (8.2%). The strongest predictors with the highest AOR were medical therapy with potassium-sparing diuretics with an AOR (SD) of 2.21 (0.09), cellulitis and abscess (AOR, 2.12; SD, 0.04), skin and subcutaneous tissue infections (AOR, 2.08; SD, 0.04), chronic ulcer of the leg or foot (AOR, 2.01; SD, 0.06), and chronic ulcer of the skin (AOR, 2.0; SD, 0.08); however, these effect estimates need to be interpreted with caution if variables had low EFs. The average AUC based on the five-fold cross-validation for the all-cause hospitalization REFS model was 0.63.

Table 2.

REFS™ prediction model for all-cause hospitalizations

Variable type	Predictor	Edge frequency, %	AORa	SD of AORa
Comorbidity	Skin and subcutaneous tissue infections	57.81	2.08	0.04
Comorbidity	Cellulitis and abscess	26.95	2.12	0.04
Medication	Beta-blockers, nonselective	12.50	1.92	1.26
Comorbidity	Other aftercare	12.50	1.61	0.06
Comorbidity	Osteoarthritis	8.20	1.24	0.03
Medication	Potassium-sparing diuretics	4.30	2.21	0.09
Comorbidity	Chronic ulcer of leg or foot	2.73	2.01	0.06
Demographic	Current smoker	2.34	1.43	0.02
Demographic	Smoking status unknown	2.34	1.15	0.03
Comorbidity	Substance-related disorder	2.34	1.85	0.06
Medication	Indomethacin	1.56	0.80	0.05
Comorbidity	Chronic ulcer of the skin	1.17	2.00	0.08
Comorbidity	Upper GI disorders	1.17	1.18	0.001

REFS Reverse Engineering Forward Simulation, GI gastrointestinal, AOR average odds ratio, SD standard deviation

^aAOR and SD of AOR were calculated across the models in which the variable was selected

REFS model for total annual all-cause healthcare costs

The top 5 predictors selected with highest consistency across the 256 networks included region of residence (75.4%), other diseases of kidney and ureters (64.5%), blindness of vision defects (35.6%), chronic ulcer of skin (32.8%), and chronic ulcer of leg or foot (28.9%). Leading predictors associated with higher total annual all-cause healthcare costs (Table 3) were other diseases of kidney and ureters (APCC, 2.20; SD, 0.17), chronic ulcer of the skin (APCC, 1.95; SD, 0.13), chronic ulcer affecting the leg or foot (APCC, 1.93; SD, 0.11), skin and subcutaneous tissue infections (APCC, 1.85; SD, 0.21), and cellulitis and abscess (APCC, 1.84; SD, 0.16). The mean five-fold cross-validated R² was 0.12.

Table 3.

REFS™ prediction model for total annual all-cause healthcare costs

Variable type	Predictor	Edge frequency, %	APCCc	SD of APCCc
Demographic	Regiona: midwest	75.39	0.84	0.03
Demographic	Regiona: south	75.39	1.16	0.03
Demographic	Regiona: west	75.39	0.42	0.02
Demographic	Regiona: unknown	75.39	1.38	0.05
Comorbidity	Other diseases of kidney and ureters	64.45	2.20	0.17
Comorbidity	Blindness and vision defects	35.55	0.91	0.05
Comorbidity	Chronic ulcer of skin	32.81	1.95	0.13
Comorbidity	Chronic ulcer of leg or foot	28.91	1.93	0.11
Comorbidity	Varicose veins of lower extremity	26.17	0.92	0.04
Comorbidity	Peri-, endo-, and myocarditis; cardiomyopathyb	25.00	1.22	0.07
Interaction	Blindness and vision defects; peri-, endo-, and myocarditis; cardiomyopathyb	28.83	0.26	0.03
Comorbidity	Hypertension	22.27	1.55	0.06
Comorbidity	Cellulitis and abscess	21.88	1.84	0.16
Interaction	Hypertension; blindness and vision defects	19.53	0.92	0.07
Comorbidity	Other non-epithelial cancer of skin	19.53	1.03	0.05
Interaction	Other non-epithelial cancer of skin; varicose veins of lower extremity	18.36	0.77	0.06
Comorbidity	Mood disorders	16.41	1.32	0.05
Comorbidity	Depressive disorders	16.02	1.29	0.06
Comorbidity	Skin and subcutaneous tissue infections	13.28	1.85	0.21
Medication	Amlodipine	11.72	1.42	0.04
Interaction	Hypertension; amlodipine	11.72	0.75	0.03
Comorbidity	Osteoarthritis	10.16	1.32	0.04
Medication	Insulin glargine	10.16	1.77	0.08

REFS Reverse Engineering Forward Simulation, APCC average percentage change in cost, SD standard deviation

^aNortheast as the reference group

^bCardiomyopathy caused by tuberculosis or sexually transmitted disease were excluded

^cAPCC and SD of APCC were calculated across the models in which the variable was selected

Discussion

We applied a proprietary Bayesian machine learning analysis platform, REFS, to identify predictors of all-cause hospitalizations and total annual all-cause healthcare costs during the 12 months following a diagnosis of CLI. The most frequently identified predictors of all-cause hospitalizations were a pre-index diagnosis of skin and subcutaneous tissue infections, cellulitis and abscess, use of nonselective beta-blockers, other aftercare, and osteoarthritis. Region of residence and comorbid health conditions including other diseases of kidney and ureters, blindness of vision defects, chronic ulcer of skin, and chronic ulcer of leg or foot were the most frequently selected predictors of total annual healthcare costs during the post-index period. These findings support the PPPM framework that a paradigm shift from post-diagnosis disease care to early management of comorbidities and targeted prevention is warranted to deliver a cost-effective medical services and desirable healthcare economy.

Clinical and economic disease burden of CLI

Our finding that comorbid health conditions were among the most important predictors of higher healthcare utilization and costs in CLI is consistent with results reported by other efforts that attempted to identify factors associated with patterns and costs of care in CLI [13, 30, 31]. The risk of hospital readmission following endovascular or surgical interventions for CLI was associated with the severity of CLI at diagnosis and comorbid health conditions, such as chronic kidney disease, hypertension, a history of coronary artery disease, and CCI score. Ulcer or gangrene and infections and persistent or recurrent manifestations of PAD were also significant predictors of increased healthcare utilization and higher costs [13]. General PAD patients with comorbid diabetes had longer hospital LOS and higher hospital costs, with costs and LOS greatest in patients with severe PAD who required amputation or bypass surgery [31].

Our study confirms descriptively that CLI imposes a substantial burden on the US healthcare system, with high rates for all-cause inpatient services, hospitalizations, outpatient care, and ED visits during the post-index period. This is not surprising given the increased risk of atherosclerotic events in multiple vascular beds in patients with CLI, which are likely contributors to high rates of all-cause resource utilization [30]. While CLI-related healthcare utilization rates were lower for specific services, a significant number of patients required outpatient and inpatient care. These findings are consistent with those in previously published literature that report higher healthcare resource utilization in CLI patients compared to the general PAD patient population [13, 15, 31].

In addition to healthcare services, CLI patients also have a disparately high cost of care compared to healthcare expenditures associated with the management of general PAD. The mean total annual all-cause healthcare costs in our cohort of CLI patients was $63,367. This is substantially greater than the $13,907 reported for Medicare patients with general PAD [32] and $11,553 for patients with general PAD and various types of health insurance, including Medicare, private payers, and the uninsured [33]. Similarly, we reported a CLI-associated mean annual per-patient cost of $10,884, which is significantly higher than the total mean expenditures for patients with general PAD at $1,868 [32]. Prescription medication expenditures also appear to be important drivers of the high cost of care for patients with PAD and CLI. Scully and colleagues [33] reported that the mean out-of-pocket cost for prescription medications was $386, ranging from $179 for Medicare patients to $1,196 for uninsured patients with general PAD. An analysis of total annual costs in a large managed care database of general PAD patients revealed that medications including, but not limited to, antihypertensive agents and lipid-lowering therapies totaled $610, representing 10.2% of total expenditures [34]. These amounts are significantly lower than the mean annual per-patient cost of $1,360 for pharmacy services in our cohort of CLI patients, which accounted for 18.6% of the total annual mean CLI-related expenditures.

PPPM model as a promising strategy for CLI care

Our prediction models identified several comorbid health conditions before CLI diagnosis as significant drivers of the increased need for healthcare services and greater costs in the subsequent year post-CLI diagnosis. These findings are consistent with results reported by other efforts that attempted to identify factors associated with patterns and costs of care in CLI and reinforces the importance of aggressively targeting evidence-based interventions for patients with CLI, and highlight the emerging opportunity of model-based, data-driven predictive medicine to complement traditional evidence-based interventions and medical decision-making for patients with CLI, including revascularization, wound healing interventions, pharmacotherapy, and lifestyle interventions [35]. The challenge for vascular clinicians is to identify ways to maintain or improve patient outcomes while reducing or holding constant expenditures by optimizing the cost-effectiveness and “value” of the health care provided to patients [8]. Machine learning models such as REFS offer one methodology to support clinicians’ efforts to deliver data-driven care tailored to early prediction, targeted prevention, and personalized medicine [36]. For example, our results suggest that skin and subcutaneous tissue infections, cellulitis and abscess, and use of nonselective beta-blockers before CLI diagnosis is individually associated with doubled the risk of hospitalization. Although these identified comorbidities and medications may not be readily modifiable after their clinical manifestation, early intervention that focuses on preventing or slowing the clinical onset of these comorbid conditions could be useful to alleviate both the clinical and economic burden of CLI, and individualized disease management and care plan may be warranted based on patients’ comorbidity profiles to achieve optimal outcomes.

Study strengths and limitations

Our study has several unique features that may provide additional insights about the current management and costs associated with CLI. First, REFS accommodates the complexity of enormous datasets with novel and traditional risk factors as well as large samples of patients, thereby reducing the chance that relevant predictors are overlooked and increasing the likelihood that the resulting models are representative of the general population of CLI patients treated by vascular surgeons [18]. In addition, our analysis was based on a large, well-validated contemporary dataset that included insured as well as uninsured patients rather limiting patient inclusion to Medicare-only [37] or private payer patient populations [38, 39]. Importantly, the majority of previous studies on this topic are based on patients with general PAD and CLI [37–39]. In contrast, our study focused specifically on patients with CLI, which is likely to more accurately characterize resource utilization and costs in CLI [30].

Due to the need for medical interventions beyond first-line therapy for CLI [4, 8, 30] efforts to accurately identify predictors of costs and healthcare resource utilization will be enhanced by the assessment of outcomes beyond the 12-month post-index period in our study [11]. Further, the development of a consensus on the optimal end points for CER in CLI will improve the relevance of predictive models for healthcare utilization and costs [18]. For example, outcomes such as wound healing, time to wound healing, quality of life, and pain control directly impact readmissions, costs, and major adverse limb events. Inclusion of these endpoints in machine learning models may further clarify our understanding of predictors of costs and healthcare utilization in patients with CLI [4].

We did not examine predictors of all-cause hospitalizations or total annual healthcare costs stratified by the type of intervention performed for patients, which introduces the possibility that REFS would generate alternative models for patients undergoing different guideline-based interventions [35] such as revascularization procedures, wound healing therapies, medical therapies, lifestyle modifications, or some combination of these interventions. The development of such models using machine learning methods might form the basis for future studies.

Potential study limitations also need to be noted. First, the use of an observational study design based on de-identified administrative claims database and EHR introduces the potential for selection bias (e.g., US-based, strict inclusion/exclusion criteria, enrollment requirements, etc.) and errors or missing information in the databases. The non-randomized, observational design also introduces the possibility of confounding due to unmeasured predictors of healthcare utilization and costs, notably, absence of robust clinical data such as severity of comorbidities, controlled status of disease, complete laboratory data, quality of life measures, etc. However, the Optum and Humedica databases contain extensive claims, clinical, and practice management data for more than 5 million unique patients. This is a large patient sample that has been followed for an extended period of time, thereby reducing the likelihood of omitting relevant predictors of healthcare costs and utilization. Second, machine learning models offer an opportunity to uncover proxies of unmeasured variables that can both improve model performance, as well as offer investigators alternative measures for evaluating patient health when such measures are unavailable in the data. Third, the study lacks the external validation data to evaluate generalization accuracy. However, the predictive performance of the models was assessed via a commonly used internal validation technique in machine learning (k-fold cross-validation) which can test the success of classification on cases but also avoids the problem of over-fitting and reduces bias and variance. We also acknowledge that the analysis of data from patients in 20 states might not yield results that can be generalized to the entire US population, or more generally, to populations outside the US. However, recent reports suggest similar utilization rates in the US compared to other developed countries despite higher costs in the US [40]. This work assesses both utilization and cost to provide a more complete summary of burden incurred by CLI populations.

Future directions

Our study supports that machine learning can be a useful tool to identify patients at risk of incurring high healthcare burden and target them for interventions. Disease management and care plan may be personalized based on individual patients’ comorbid health conditions and medication use. The model-based medicine can be an effective complement to traditional evidence-based medicine. Future steps may be warranted, for example: (1) increase model accuracy by adding data—including additional patients, additional variables, data from more recent time frames, longer time windows, etc.; (2) validate the models against external datasets; (3) deploy the models in a payer or provider system to help caregivers make more informed decisions based on model predictions; (4) use discovered variables to stratify populations on a more general level to prioritize interventions.

Conclusions and expert recommendations

As suggested in a Special Session “Patient-Specific Modeling” of 2012 EPMA White Paper [19], model-based medicine based on a holistic approach such as machine learning and big data could help shifting paradigm of reactive healthcare to PPPM-driven healthcare and also serves as a valuable complement to evidence-based medical services. Using a machine learning analysis approach, the current study results of comorbidities and medication use being predictive of subsequent hospitalization risk and increased healthcare cost after CLI diagnosis supported that an earlier diagnostics or targeted intervention based on these patient risk profiles to prevent or slow clinical manifestations of these comorbidities before CLI diagnosis may help reducing subsequent hospitalization risk and healthcare cost. In particular, we found that even before patients were formally diagnosed with CLI, the concurrent comorbidities such as hypertension and diabetes and early symptoms of CLI, such as skin infections or chronic ulcer, were predictive of 12-month risk of all-cause hospitalization and healthcare cost. Due to its substantial clinical, societal, and economic burden, it is absolutely critical to apply the PPPM concept to the CLI care. From a view of public health, one prevention strategy may involve daily practice of health providers to pay a particular attention to hypertensive or diabetic patients, especially when they also present delayed wound healing or skin infections, in order to identify predisposed CLI individuals to be diagnosed and treated well in time. Furthermore, due to the cascaded effect of comorbidities on subsequent CLI burden, early prevention of these CLI comorbidities through modifications of known unhealthy lifestyles and behaviors such as imbalanced nutrition, overweight and obesity, and stress factors through both population-level and individual-level interventions are warranted, despite the unavailability of such data in this current study. Applying this promising and innovative PPPM screening to successful identification of at-risk population at the early stage is crucial to shift the healthcare from disease management and care after disease manifests to earlier intervention and prevention and to alleviate the burden of CLI. In summary, innovative machine learning models such as REFS have the potential wide-ranging applications in CLI care and other disease areas to support health care research and clinicians’ efforts to provide data-driven, personalized therapy for CLI patients to empower prediction, prevention, and early diagnostics [41].

Abbreviations

CLI

Critical limb ischemia

PAD

Peripheral artery disease

US

United States

PPPM

Predictive, Preventive and Personalized Medicine

REFS

Reverse Engineering Forward Simulation

EHR

Electronic health record

ICD-9-CM

International Classification of Diseases, Ninth Revision, Clinical Modification

ED

Emergency department

CCI

Charlson Comorbidity Index

SD

Standard deviation

IQR

Interquartile range

AUC

Area under the receiver operating curve

EF

Edge frequency

AOR

Average odds ratios

APCC

Average percentage change in costs

Funding information

This study was funded by Janssen Pharmaceuticals (Titusville, NJ). The publication of study results was not contingent on the sponsor’s approval or censorship of the manuscript.

Compliance with ethical standards

Conflict of interest

Authors Haskell, Crivera, and Schein have direct financial relationships with Janssen Pharmaceuticals. Authors Berger, Ting, and Lurie have indirect financial relationships with Janssen Pharmaceuticals. Authors Chang, Meuller, Elder, Rich, and Alas declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.