Early Detection of Heart Failure with Reduced Ejection Fraction using Perioperative Data among Non-Cardiac Surgical Patients: A Machine Learning Approach

Abstract

Background:

Heart failure with reduced ejection fraction (HFrEF) is a condition imposing significant healthcare burden. Given its syndromic nature and often insidious onset, the diagnosis may not be made until clinical manifestations prompt further evaluation. Detecting HFrEF in precursor stages could allow for early initiation of treatments to modify disease progression. Granular data collected during the perioperative period may represent an underutilized method for improving the diagnosis of HFrEF. We hypothesized that patients ultimately diagnosed with HFrEF following surgery can be identified via machine learning approaches using pre- and intra-operative data.

Methods:

Perioperative data were reviewed from adult patients undergoing general anesthesia for major surgical procedures at an academic quaternary care center between 2010 and 2016. Patients with known HFrEF, heart failure with preserved ejection fraction, preoperative critical illness, or undergoing cardiac, cardiology, or electrophysiologic procedures were excluded. Patients were classified as healthy controls or undiagnosed HFrEF. Undiagnosed HFrEF was defined as lacking a HFrEF diagnosis preoperatively, but establishing a diagnosis within 730 days postoperatively. Undiagnosed HFrEF patients were adjudicated by expert clinician review, excluding cases for which HFrEF was secondary to a perioperative triggering event, or any event not associated with HFrEF natural disease progression. Machine learning models, including L1 regularized logistic regression, random forest, and extreme gradient boosting were developed to detect undiagnosed HFrEF, using perioperative data including 628 preoperative and 1,195 intraoperative features. Training/validation and test datasets were used with parameter tuning. Test set model performance was evaluated using area under the receiver operating characteristic curve (AUROC), positive predictive value, and other standard metrics.

Results:

Among 67,697 cases analyzed, 279 (0.41%) patients had undiagnosed HFrEF. The AUROC for the logistic regression model was 0.869 (95% confidence interval 0.829-0.911), 0.872 (0.836-0.909) for the random forest model, and 0.873 (0.833-0.913) for the extreme gradient boosting model. The corresponding positive predictive values were 1.69% (1.06-2.32%), 1.42% (0.85-1.98%), and 1.78% (1.15-2.40%) respectively.

Conclusions:

Machine learning models leveraging perioperative data can detect undiagnosed HFrEF with good performance. However, the low prevalence of the disease results in a low positive predictive value, and for clinically meaningful sensitivity thresholds to be actionable, confirmatory testing with high specificity (e.g. echocardiography or cardiac biomarkers) would be required following model detection. Future studies are necessary to externally validate algorithm performance at additional centers, and explore the feasibility of embedding algorithms into the perioperative electronic health record for clinician use in real-time.

Introduction

Heart failure (HF) is a chronic condition affecting over 5.8 million patients in the US, at an annual cost over $39 billion and with a 5-year mortality rate of 45-60%.^1-3 Despite advances in management, limited progress has been made to reduce health costs of HF, due to its insidious onset, varied presentation, and syndromic nature, which impede simple, reliable, and inexpensive early diagnostic measures.⁴ As a result, patients with HF in early stages may go unrecognized and do not receive treatments proven to reduce mortality.⁵

During the perioperative period, HF is of critical concern to anesthesiologists. It is a well-described risk factor for cardiac events and is associated with an 8-fold increase in perioperative cardiovascular mortality.^6,7 Given such risks, anesthesiologists carefully screen patients for HF. A wealth of information is collected preoperatively, including a history and physical, laboratory testing, and when indicated, cardiovascular testing. In patients without previously diagnosed HF at the time of surgery, this information may be useful for identifying patients with early or precursor stage HF.

Beyond this wealth of preoperative data, the intraoperative period shares similarities with a cardiac stress test, during which hemodynamic responses to surgical and anesthetic stimuli are recorded with high resolution. Yet, this vast array of data has not been formally evaluated as a diagnostic adjunct, to improve diagnosis of cardiovascular disease such as HF and guide cardiologist referral early in the disease course. Physician-dependent synthesis of EHR and intraoperative data to identify and characterize HF is challenged by limited personnel resources to perform comprehensive, timely reviews. Machine learning techniques offer a means to streamline EHR review processes, synthesizing large amounts of data from disparate sources to improve a physician’s ability to recognize clinical syndromes such as HF.^8,9

In this observational study of perioperative data from an academic quaternary care center, we leveraged machine learning techniques to understand patterns within preoperative data as well as hemodynamic responses to intraoperative interventions which may serve to detect HF with reduced ejection fraction (HFrEF) in early or preclinical stages. We hypothesized that patterns within perioperative data exist among patients with undiagnosed HFrEF which may be detected via machine learning techniques.

Methods

Study Design

This manuscript follows guidelines for reporting machine learning models in biomedical research.¹⁰ For this retrospective observational study at our academic quaternary care center, we obtained Institutional Review Board approval (Ann Arbor, MI; HUM00143523) and written informed patient consent was waived by the institutional review board. Our study design was presented at a multidisciplinary peer-review forum on September 19, 2018 prior to data access.¹¹ We extracted a limited dataset from the Multicenter Perioperative Outcomes Group (MPOG) database for this study, as well as cardiac imaging data from our institution (Epic Systems, Verona, WI). Methods for ensuring data completeness and veracity within MPOG are described elsewhere¹¹ and utilized in previous studies^12-14.

Study Population

We extracted all non-cardiac surgical procedures performed at a single institution (Michigan Medicine, Ann Arbor, MI) between January 1, 2010 and October 31, 2016. As described in Figure 1, we excluded cardiac surgical procedures, as undiagnosed HFrEF would be unexpected in this population due to extensive preoperative cardiac evaluation. Additional cases were excluded on this basis, including patients undergoing cardiology or cardiac electrophysiologic procedures, as well as patients with indicators of preoperative critical illness. To target identification of patients with more typical HFrEF disease progression, we restricted cases to adults ≥40 years without a previous heart or lung transplantation. To focus the analysis on patients with sufficient hemodynamic response profiles to intraoperative interventions, we restricted cases to longer procedures performed under general anesthesia with mechanical ventilation. Minor procedures, identified by 2018 Anesthesia Common Procedural Terminology code base unit values ≤3, were excluded.¹⁵ To target a patient population not acutely developing HFrEF as a consequence of the surgery (e.g. septic/hemorrhagic shock) and with the capacity to benefit from outpatient treatment interventions (if HFrEF detected), patients with postoperative in-hospital mortality were excluded. Finally, patients with multiple eligible procedures were indexed to one procedure per phenotype (described below).

Figure 1:

Study inclusions/exclusions, heart failure phenotypes, and machine learning methods (algorithm training, tuning, and testing).

ASA = American Society of Anesthesiologists, F/S = feature selection, preop = preoperative, intraop = intraoperative, HFrEF = heart failure with reduced ejection fraction, HFpEF = heart failure with preserved ejection fraction

* Minor procedures defined as cases with Anesthesia Current Procedural Terminology code base unit value ≤3

** Additional details provided in Supplemental Digital Content 3

Heart Failure with Reduced Ejection Fraction - EHR Phenotypes

Cases meeting inclusion criteria were classified into one of four phenotypes based upon a combination of EHR data sources, including International Classification of Diseases (ICD-9/10) codes, left ventricular ejection fraction, HF diagnoses within the anesthesiology history & physical (structured data or positive mention of HF keywords within free-text), and medications unique to HF or associated with cardiomyopathy. We defined EHR phenotypes as (i) healthy controls without HF at any point; (ii) known HFrEF patients with pre-existing HFrEF prior to surgery; (iii) undiagnosed HFrEF patients without a HF diagnosis (of any type) preoperatively but with HFrEF diagnosed within 730 days postoperatively; and (iv) ambiguous or known HF with preserved ejection fraction (HFpEF) among remaining unclassified patients. A cardiac anesthesiologist domain expert (MRM) verified undiagnosed HFrEF phenotype patients via manual review of all available EHR data, reclassifying patients as necessary. Importantly, cases for which HFrEF was secondary to a perioperative triggering event were excluded, such as a perioperative myocardial infarction, cardiac arrest, circulatory overload from massive transfusion, sepsis, initiation of cardiotoxic medications (e.g. adjuvant chemotherapeutic agents with surgical tumor resection), or any other event not associated with HFrEF natural disease progression. Table 1 provides criteria and definitions for each phenotype.

Table 1:

Heart Failure Phenotype Definitions

	Preoperative Data *			Postoperative Data up to 730 Days		Domain Expert Manual Review
	ICD-9/10 Codes	Cardiac Imaging	Anesthesia History & Physical / Medications	ICD-9/10 Codes	Cardiac Imaging
Healthy Control	No HF ICD-9/10 Code **	Unavailable, or LVEF ≥50% on all available cardiac imaging data	No positive mention of HF keywords****   No uniquely HF medications *****	No HF ICD-9/10 Code **	Unavailable, or LVEF ≥50% on all available cardiac imaging data	--
Known HFrEF	HF ICD-9/10 Code(s) ***	LVEF ≤40% within 1,825 days preoperative	--	--	--	--
Undiagnosed HFrEF	No HF ICD-9/10 Code **	Unavailable, OR LVEF ≥50% on all available cardiac imaging data	No positive mention of HF keywords****   No uniquely-HF medications *****	--	LVEF ≤40%	Performed
Ambiguous or Known HFpEF	All patients not meeting above criteria					--

Each phenotype (rows) defined by the intersection (as opposed to union) of all EHR types (columns) above. HF = heart failure; HFpEF = heart failure with preserved ejection fraction; HFrEF = heart failure with reduced ejection fraction; ICD = International Classification of Diseases; LVEF = left ventricular ejection fraction.

^*Includes all dates prior to surgery

^**Defined by Elixhauser Enhanced ICD-9-CM and ICD-10 Indices: Congestive Heart Failure, as well as ICD-9 and ICD-10 codes for cardiomyopathy (425.X, I42.X) and cardiomegaly (429.3, I51.7)

^***Defined by 428.X (ICD-9) or I50.X (ICD-10), X = wildcard

^****“HF”, “heart fail*”, “cardiac fail*”, “ventricular fail*”, “cardiomyop*”, or “*ICM”; * = wildcard

^*****Digoxin (Lanoxin), sacubitril/valsartan (Entresto)

Final Analytic Dataset and Target Output

Following HF phenotype classification, known HFrEF and ambiguous or known HFpEF patients were excluded from the final analytic dataset. The target output (primary outcome) was undiagnosed HFrEF, extracted as a binary event and independent of the number of postoperative days (up to 730) the diagnosis was established.

Input Features - Preoperative Data

Preoperative features were selected based upon availability within the database and domain expert (MRM) clinical judgment. These included demographic, medical, surgical, and anesthetic characteristics, totaling 258 features (628 features when expanded via one-hot encoding) detailed in Supplemental Digital Content 1-3. Patient demographics, laboratory values, and comorbidities were validated utilizing pre-computed, publicly available, MPOG-specific perioperative EHR phenotype algorithms.¹⁶ For a selected subset of continuous features with informative missingness, features were transformed from continuous to categorical data binned into clinically meaningful ranges, with missing data as a distinct category. Home medications were classified using the Veterans Affairs National Formulary.¹⁷ Medications prescribed to >1% of patients in the final analytic dataset, as well as all cardiovascular medications (regardless of prevalence) were considered for feature selection.

Input Features - Intraoperative Data

Intraoperative features were similarly identified based upon availability and domain expert judgment. These included physiologic monitoring and ventilator data, fluids, medications, anesthetic requirements, and case times, described in Supplemental Digital Content 4-5. For intraoperative data collected as repeated measures, data were available to the nearest minute, and summary characteristics (e.g. minimum, median, mean, maximum, standard deviation) were calculated and handled as distinct features. For specific repeated measures, additional complex feature engineering potentially associated with HFrEF was performed (e.g. root-mean-square standard deviation of heart rate, average real variability of blood pressure).^18,19 We processed blood pressure data utilizing algorithms to reduce artifact, impute missing values, and resolve simultaneous values (e.g., non-invasive cuff and invasive arterial line blood pressures) as described in Supplemental Digital Content 6. Similarly, we removed artifact from physiologic and ventilator data, medications, fluids, utilizing pre-computed perioperative EHR phenotype algorithms publicly available within MPOG.¹⁶ For remaining missing data, previous values were carried forward, or handled via imputation as described in Supplemental Digital Content 4.

To analyze intraoperative response profiles contextualized to intraoperative surgical and anesthetic interventions, intraoperative data were segmented into discrete, overlapping phases of intraoperative care (e.g. induction, intubation, incision, maintenance, extubation). Intraoperative phases of care are summarized in Supplemental Digital Content 5; considering the combinations of intraoperative data types, feature engineering, and overlapping phases of care, a total of 1,195 intraoperative features were rendered. We provide example data for a fictitious patient in Supplemental Digital Content 7.

Feature Selection

All processed features were inspected for validity, data leakage, or perfect separation¹⁰ through visualizations of summary data and manual review of 1% of cases without the target output (healthy) and all cases with the target output (undiagnosed HFrEF). Given differences in preoperative versus intraoperative data collected, we adopted a two-stage feature selection process in which intraoperative features were first considered, followed by consideration of all variables. During the first stage, intraoperative features with a correlation coefficient >0.70 were reduced to a single feature by eliminating the second feature. In the second stage, remaining intraoperative features were combined with all preoperative features, and a least absolute shrinkage and selection operator (LASSO) feature selection method with L1 regularization was used. We applied linear support vector machines with LASSO as the second feature selection procedure in pipeline, and used the LASSO penalty factor to control the number of features selected.

Model Development

We partitioned the analytic dataset into training/validation (80%) and test (20%) sets, with equal ratios of the target output (Figure 1). We performed statistical analyses using the Scikit-learn machine learning packages in Python. We provide sample code for the primary analyses, available via the Open Science Framework at https://osf.io/kqj4f/. Figure 2 provides an overview of the preoperative features, intraoperative segmentation, and intraoperative model input features used for machine learning model development.

Figure 2:

Summary overview - preoperative features, intraoperative segmentation, and intraoperative features used for feature selection and machine learning model development.

Following preprocessing and feature selection, we developed machine learning classifiers using logistic regression, random forest, and extreme gradient boosting. We used grid search cross validation and pipeline functions in the scikit-learn package, to respectively perform 5-fold cross validation of our training dataset and assemble feature selection procedures and machine learning classifiers together. This enabled hyperparameter tuning for both feature selection methods (e.g. penalty factor for LASSO) and machine learning classifiers (e.g. maximum tree depth for random forest). For the logistic regression model, continuous features were mean-centered and normalized. We iteratively reduced the number of features by tuning a correlation threshold and L1 penalty to train the models.

We expected our target output distribution to be skewed, given the low prevalence of HFrEF diagnosed in a post-surgical population within 730 days. To mitigate the influence of an imbalanced dataset, we used a computationally efficient cost-sensitive approach with class weighting inversely proportional to sample size in the training dataset.

Within the training/validation set, we trained machine learning algorithms using 5-fold cross validation using the area under the receiver operating characteristic curve (AUROC) as a primary performance metric. We used machine learning algorithms shown in Figure 1; each algorithm predicted a logistic output (probability of undiagnosed HFrEF [0-1]).

Model Tuning and Performance

Following hyperparameter tuning, we calculated an AUROC using the validation fold of the training set averaged over all folds and repetitions. We describe machine learning model functions, packages, and tuning parameters in Supplemental Digital Content 8.

With each best model determined, we evaluated performance on the hold-out test set (20%) using bootstrapping. The size of stratified sampling was the same as the size of the test set, and we repeated the test 500 times. For each test, in addition to evaluating model performance via AUROC, we evaluated area under the precision recall curve (AUPRC), accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. We selected an optimal threshold based on AUROC thresholds by following equation:

$$min(\mid Sensitivity-Specificity\mid )$$

The optimal threshold minimizes the difference between sensitivity and specificity, as has been previously described,²⁰ and was used for calculating all other performance metrics. To evaluate test set model performance at varying mean predicted value thresholds, we explored histograms of mean predicted values for healthy control and undiagnosed HFrEF cases for each model. Similarly, to evaluate test set model performance at varying sensitivity thresholds, we explored AUROC plots and plots of positive predictive value and specificity versus sensitivity.

Feature Importance

Within the tuned models, to assess which model features most strongly influenced the undiagnosed HFrEF detection algorithms, we explored measures of feature importance. These included regression coefficients for logistic regression, Gini importance for random forest, and gain for extreme gradient boosting. To inspect the relative influence of feature subgroups (e.g. preoperative comorbidities, intraoperative physiologic features, etc.) on machine learning models, we explored permutation feature importance of pre-specified feature subgroups within the test set. Permutation feature importance is a model inspection technique, defined as the decrease in model performance (AUROC for this study) when a subgroup of features is randomly shuffled (repeated 100 times and averaged for this study). Applications and limitations to this technique are previously described.^21,22

Pre-planned Sensitivity Analysis

To assess the robustness of the machine learning models developed, we performed a sensitivity analysis in which patients with known HFrEF were additionally included in both training and test set, the target output was defined as undiagnosed HFrEF or known HFrEF, and models were retrained and features were restricted to only the intraoperative features, given the potential for preoperative data leakage among known HFrEF patients. We additionally performed a sensitivity analysis in which healthy control cases were combined with ambiguous or known HFpEF cases into a single non-HFrEF phenotype.

Results

Among 90,357 adult non-cardiac surgical procedures reviewed, 74,758 met inclusion criteria (Figure 1). Within this cohort, 2,354 cases involved patients with known HFrEF. Among cases not meeting diagnostic criteria for HF preoperatively, 415 cases were eligible for undiagnosed HFrEF expert review. Following review, 279 cases were determined to be undiagnosed HFrEF developing unrelated to a perioperative triggering event (Supplemental Digital Content 9), resulting in a phenotyped dataset of 67,418 healthy controls, 279 patients with expert-adjudicated undiagnosed HFrEF, 2,354 patients with known HFrEF, and 4,707 patients with an ambiguous or known HFpEF diagnosis. From the phenotyped dataset, healthy controls and undiagnosed HFrEF patients were included in the final analytic dataset for primary analysis.

Patient Population - Baseline Characteristics

We provide summary statistics of perioperative characteristics among 67,697 cases in the entire analytic cohort, comprised of 279 (0.41%) undiagnosed HFrEF patients and 67,418 (99.6%) healthy controls, in Table 2. Our study population had a median age of 59 years and 47% were men. Compared to healthy controls, patients with undiagnosed HFrEF more commonly were older males with comorbid conditions, were prescribed cardiovascular medications and analgesics, underwent inpatient vascular surgical procedures, and presented with elevated preoperative heart rates and lower hemoglobin and sodium levels. Intraoperatively, patients with undiagnosed HFrEF more commonly underwent longer procedures with arterial line monitoring, were administered larger doses of vasoactive medications and opioids, experienced more estimated blood loss, and demonstrated elevated heart rates and less blood pressure variability. Surgical procedures most commonly included upper and lower abdominal procedures, thoracic procedures, and spine or spinal cord procedures for both healthy and undiagnosed HFrEF cases (Supplemental Digital Content 10).

Table 2-.

Selected Characteristics of the Entire Cohort and Univariate Analyses

Category	Feature		Entire Cohort N = 67,697 n(%) or median (IQR)	Healthy Controls N= 67,418 n(%) or median (IQR)	Undiagnosed HFrEF, N=279 n(%) or median (IQR)	p- Value1
Demographic / Anthropometric Data	Age, years		59 (51 - 68)	59 (51 - 68)	65 (58 - 74)	<0.001
	Body Mass Index		28.8 (24.9 - 33.5)	28.8 (25.0 - 33.5)	27.9 (23.8 - 32.3)	0.026
	Male Gender		32,200 (47.6)	32,028 (47.5)	172 (61.7)	<0.001
Patient Medical History	Cardiac Arrhythmias		8,044 (11.9)	7,910 (11.7)	134 (48.0)	<0.001
	Coronary Artery Disease		4,989 (7.4)	4,867 (7.2)	122 (43.7)	<0.001
	Coagulopathy		1,893 (2.8)	1,849 (2.7)	44 (15.8)	<0.001
	Valvular Disease		1,418 (2.1)	1,382 (2.1)	36 (13.0)	<0.001
	Hypertension, Complicated		3,115 (4.6)	3,072 (4.6)	43 (15.4)	<0.001
	Peripheral Vascular Disorders		3,751 (5.5)	3,64 (5.5)	67 (24.0)	<0.001
	Liver Disease		3,328 (4.9)	3,289 (4.9)	39 (14.0)	0.004
	Chronic Pulmonary Disease		8,180 (12.1)	8,117 (12.0)	63 (22.6)	0.005
	Diabetes, Complicated		1,376 (2.0)	1,351 (2.0)	25 (9.0)	0.007
	Paralysis/Other Neurological Disorders		3,492 (5.2)	3,461 (5.1)	31 (11.1)	0.067
	Pulmonary Circulation Disorders		1,283 (2.0)	1,268 (1.9)	15 (5.4)	0.162
Preoperative Status	Heart Rate		73 (65 - 84)	73 (65 - 84)	77 (67 - 87)	0.008
	Mean Arterial Pressure		95 (87 - 104)	95 (87 - 104)	95 (85 - 106)	0.940
	SpO2 (%)		97 (96 - 98)	97 (96 - 98)	97 (95 - 98)	0.038
	Respiratory Rate		16 (16 - 18)	16 (16 - 18)	16 (16 - 18)	0.081
	Temperature		36.7 (36.5 - 36.8)	36.7 (36.5 - 36.8)	36.7 (36.5 - 36.8)	0.994
Preoperative Testing	Median Left ventricular ejection fraction within 1825 days , %		62.5 (60 - 65)	62.5 (60 - 65)	60 (60 - 65)	0.013
	Median Hemoglobin, g/dL		13.3 (11.8 - 14.4)	13.4 (12.0 - 14.5)	11.9 (10.2 - 13.6)	<0.001
	Median White blood cell count, k/ul		7.2 (5.7 - 8.9)	7.2 (5.7 - 8.9)	7.3 (5.6 - 9.7)	0.461
	Median Serum Sodium, mEg/L		140 (139 - 142)	140 (139 - 142)	140 (138 - 141)	<0.001
Preoperative Medications – Veterans Affairs Formulary	Beta Blockers (CV100)		15,882 (23.5)	15,784 (23.4)	98 (35.1)	<0.001
	Calcium Channel Blockers (CV200)		9,979 (14.7)	9,903 (14.7)	76 (27.2)	<0.001
	Antiarrhythmics (CV300)		425 (0.6)	419 (0.6)	6 (2.2)	<0.001
	Antilipemics (CV350)		22,582 (33.4)	22,455 (33.3)	127 (45.5)	<0.001
	Thiazide Diuretics (CV701)		10,778 (15.9)	10,732 (15.9)	46 (16.5)	0.796
	Loop Diuretics (CV702)		3,328 (4.9)	3,292 (4.9)	36 (12.9)	<0.001
	Potassium Sparing Diuretics (CV704)		2,389 (3.5)	2,374 (3.5)	15 (5.4)	0.094
	ACE Inhibitors (CV801)		13,859 (20.5)	13,764 (20.4)	95 (34.1)	<0.001
	Opioid Analgesics (CN101)		8,087 (12.0)	8,033 (11.9)	54 (19.4)	<0.001
	Non-opioid Analgesics (CN103)		33,782 (49.9)	33,609 (49.9)	173 (62.0)	<0.001
Surgical Characteristics	Admission Status	Admit	39,704 (58.7)	39,556 (58.7)	148 (53.1)	0.083
		Inpatient	7,780 (11.5)	7,688 (11.4)	92 (33.0)	<0.001
		Outpatient	20,213 (29.9)	20,174 (29.9)	39 (14.0)	0.015
	Surgical Service	Dentistry	345 (0.5)	344 (0.5)	1 (0.4)	0.492
		General	13,632 (20.1)	13,578 (20.1)	54 (19.4)	0.443
		Medical	24 (0.0)	23 (0.0)	1 (0.3)	0.432
		Neurosurgery	6,903 (10.2)	6,867 (10.2)	36 (12.9)	0.296
		Obstetrics / Gynecology	4,548 (6.7)	4,542 (6.7)	6 (2.2)	0.327
		Ophthalmology	1,304 (1.9)	1,302 (1.9)	2 (0.7)	0.450
		Oral / Maxillofacial	2,381 (3.5)	2,372 (3.5)	9 (3.2)	0.481
		Orthopedics	8,583 (12.7)	8,551 (12.7)	32 (11.5)	0.419
		Otolaryngology	9,279 (13.7)	9,254 (13.7)	25 (9.0)	0.244
		Plastics	3,872 (5.7)	3,861 (5.7)	11 (3.9)	0.399
		Radiology	37 (0.1)	37 (0.1)	0 (0.0)	NA
		Thoracic	3,927 (5.8)	3,915 (5.8)	12 (4.3)	0.412
		Transplant	1,950 (2.9)	1,938 (2.9)	12 (4.3)	0.384
		Trauma	1,582 (2.3)	1,565 (2.3)	17 (6.1)	0.153
		Urology	6,591 (9.7)	6,564 (9.7)	27 (9.7)	0.496
		Vascular	2,402 (3.6)	2,369 (3.5)	33 (11.8)	0.005
		Unknown	337 (0.5)	336 (0.5)	1 (0.4)	0.492
	Anesthesia CPT Base Unit Value		6 (5 - 8)	6 (5 - 8)	7 (6 - 8)	0.001
	ASA Status	1	1,974 (2.9)	1,971 (2.9)	3 (1.1)	0.425
		2	31,068 (45.9)	31,025 (46.0)	43 (15.4)	<0.001
		3	32,856 (48.5)	32,658 (48.4)	198 (71.0)	<0.001
		4	1,799 (2.7)	1,764 (2.6)	35 (12.5)	<0.001
Intraoperative Characteristics for Entire Case	Median Mean Arterial Pressure (mmHg)		79 (73 - 86)	79 (73 - 86)	79 (73 - 86)	0.961
	Standard Deviation of Mean Arterial Pressure (mmHg)		11.44 (9.22 - 13.98)	11.44 (9.23 - 13.99)	11.60 (9.44 - 14.27)	0.435
	Average Real Variability of Mean Arterial Pressure (mmHg)		5.89 (4.41 - 7.79)	5.90 (4.41 - 7.79)	5.42 (3.96 - 7.57)	0.007
	Median Heart Rate - EKG		69 (62 - 78)	69 (62 - 78)	73 (63 - 83)	0.001
	Standard Deviation of Heart Rate - EKG		7.9 (6.1 - 10.2)	7.9 (6.1 - 10.2)	8.2 (6.1 - 10.5)	0.368
	RMS standard Deviation of Heart Rate - EKG		4.4 (3.2 - 6.4)	4.4 (3.2 - 6.4)	4.4 (2.8 - 7.3)	0.952
	Crystalloid, mL		1,750 (1,200 - 2,600)	1,750 (1,200 - 2,600)	2,000 (1,300 - 3,250)	0.002
	Estimated Blood Loss, mL		50 (5 - 200)	50 (5 - 180)	100 (5 - 300)	0.001
	Cumulative Phenylephrine, mcg		100 (0 - 500)	100 (0 - 500)	300 (0 - 1605)	<0.001
	Cumulative Ephedrine, mg		0 (0 - 10)	0 (0 - 10)	0 (0 - 5)	0.711
	Cumulative Opioid, IV morphine equivalents, mg		60 (34 - 90)	60 (34 - 90)	75 (45 - 110)	0.006
	Median End-tidal isoflurane concentration, %		0.7 (0.4 - 0.9)	0.7 (0.4 -0.9)	0.7 (0.5 -0.9)	0.023
	Arterial line used		16,508 (24.4)	16,369 (24.3)	139 (49.8)	<0.001
	Case Duration, min		163 (113 - 242)	163 (113 - 242)	183 (121 - 300)	0.043

ACE = angiotensin converting enzyme, HFrEF = heart failure with reduced ejection fraction, IQR = interquartile range, RMS = root-mean-square

Full list of model input features described in Supplemental Digital Content 1-5 and 7.

Test of differences of medians were done using Wilcoxon rank-sum test

Test for differences of proportions was done using the normal approximation, z=((p1-p2))/(√(p (1-p)(1/n1 +1/n2 ))

Machine Learning Model - Features and Performance

Following feature selection, within the L1 regularized logistic regression model, a total of 207 features were selected, including 60 preoperative and 147 intraoperative features. The random forest model selected 212 preoperative and 161 intraoperative features (373 total) and the extreme gradient boosting model selected 499 preoperative and 263 intraoperative features (762 total) as detailed in Supplemental Digital Content 11. We describe tuning parameters and performance metrics for the test set (n = 13,539) in Table 3. Performance metrics were comparable across each model: the AUROC using L1 regularized logistic regression was 0.869 (95% confidence interval 0.829-0.911), random forest 0.872 (0.836-0.909), and extreme gradient boosting 0.873 (0.833-0.913); additional details provided in Supplemental Digital Content 11-12. Although models demonstrated good performance by AUROC, low positive predictive values were observed due to the low prevalence of HFrEF diagnosed within 2 years of surgery. The positive predictive value using L1 regularized logistic regression was 1.69% (1.06-2.32%), random forest 1.42% (0.85-1.98%), and extreme gradient boosting 1.78% (1.15-2.40%). Positive predictive values remained low, even at high specificity thresholds with low sensitivity (Supplemental Digital Content 13).

Table 3 -.

Performance of Machine Learning Models on Test Set

	AUROC (95% CI)	AUPRC (95% CI)	Accuracy, % (95% CI)	Sensitivity, % (95% CI)	Specificity, % (95% CI)	PPV, % (95% CI)	NPV, % (95% CI)
Logistic Regression	0.869 (0.829-0.911)	0.037 (0.016-0.059)	79.93 (75.02-84.84)	79.97 (74.75-85.18)	79.93 (75.02-84.84)	1.69 (1.06-2.32)	99.89 (99.85-99.94)
Random Forest	0.872 (0.836-0.909)	0.043 (0.015-0.070)	76.80 (71.69-81.91)	76.85 (71.30-82.39)	76.80 (71.69-81.91)	1.42 (0.85-1.98)	99.87 (99.82-99.92)
Extreme Gradient Boosting	0.873 (0.833-0.913)	0.040 (0.012-0.067)	80.82 (76.86-84.79)	80.84 (76.44-85.24)	80.82 (76.86-84.79)	1.78 (1.15-2.40)	99.90 (99.86-99.94)

AUPRC = area under the precision recall curve, AUROC = area under the receiver operating characteristic curve, CI = confidence interval, NPV = negative predictive value, PPV = positive predictive value

Optimal tuning parameters for models:

Logistic Regression: ‘C': 0.0001, ‘penalty': 'l2', ‘solver': ‘liblinear'; Feature Selection: LinearSVC C:0.001

Random Forest: 'max_depth': 30, 'min_samples_leaf': 100, 'n_estimators': 500; Feature Selection: LinearSVC C:0.01

Extreme Gradient Boosting: 'max_depth': 50, 'min_child_weight': 5, 'reg_lambda': 1000, 'scale_pos_weight': 242, 'learning_rate': 0.1, 'n_estimators': 300, 'subsample': 0.8, 'colsample_bytree': 0.6; Feature Selection: LinearSVC C:1

Feature Importance

We describe the reduced feature sets and feature importance for the tuned models in Supplemental Digital Content 14. Given comparable model performance and for ease of computation and interpretation, we present the feature importance for the L1 regularized logistic regression model (regression coefficients) in Figure 3. Preoperative comorbidities (arrhythmias, fluid electrolyte disorders, valvular disease, coagulopathy, and hypertension) were among the most important features for detection of undiagnosed HFrEF. The most important intraoperative features included presence of sinus tachycardia over the entire case, elevated root-mean-square standard deviation of heart rate over the entire case, elevated maximum systolic blood pressure prior to surgical incision, elevated standard deviation of expired concentration of nitrous oxide prior to surgical incision, and a lower standard deviation of heart rate surrounding extubation. Among feature subgroups assessed via permutation feature importance, importance varied widely by the type of machine learning model used; however generally, preoperative comorbidities were the most important feature subgroup overall, and intraoperative physiologic features were the most important intraoperative feature subgroup (Supplemental Digital Content 15).

Figure 3 -.

Feature importance of top 50 (of 207) selected features in the tuned L1 regularized logistic regression model for detection of undiagnosed HFrEF on the test set. In the case of logistic regression, feature importance is calculated as the regression coefficients. Blue bars indicate positive coefficients (positively associated with undiagnosed HFrEF), red bars indicate negative coefficients (negatively associated with undiagnosed HFrEF). Black font indicates preoperative features; red font indicates intraoperative features.

ACE = angiotensin converting enzyme; EGFR = estimated glomerular filtration rate; HbA1c = hemoglobin A1C; HFrEF = heart failure with reduced ejection fraction; LVEF = left ventricular ejection fraction MAP = mean arterial pressure

Pre-planned Sensitivity Analysis

When including patients with known HFrEF in the analytic dataset and re-training the models using intraoperative data only, model AUROCs demonstrated modest decreases, but positive predictive values demonstrated significant improvements, in the setting of increased prevalence of target output (Supplemental Digital Content 16). When combining healthy control cases with ambiguous or known HFpEF cases into a single non-HFrEF phenotype, machine learning model performance metrics were similar to the primary analysis (Supplemental Digital Content 17).

Discussion

In this study of adult patients without a prior diagnosis of heart failure undergoing major non-cardiac surgeries and surviving past hospital discharge, we report HFrEF diagnosed within 730 days postoperative to occur in 0.41% of cases. Through machine learning, we developed models using data potentially available immediately following surgery to detect patients demonstrating features of HFrEF in precursor stages. Machine learning models demonstrated good performance by AUROC, but were limited by low positive predictive values. Nonetheless, our study demonstrates the feasibility of machine learning to analyze granular preoperative data and intraoperative response profiles, contextualized to surgical and anesthetic cardiac stressor events, as an opportunity to potentially improve detection of HFrEF. Such a tool could prove useful for future precision health systems, leveraging complex EHR data to trigger additional clinical evaluation and initiate medical therapies among patients with previously undiagnosed cardiovascular disease.

The machine learning techniques in this study capture complex relationships within the EHR (“clinical signatures”)^23,24 which may be further developed to guide clinical diagnostic inferences. Such techniques have been explored for diabetes,⁸ coronary artery disease,⁹ and pulmonary hypertension²⁵. Similarly, single-center studies using machine learning applied to HF include work by Blecker et al.²⁶ using EHR data to derive an epidemiologic cohort of HF patients, and Choi et al.²⁷ using deep learning to predict HF defined by ICD-9 codes. Our work builds upon these studies, through use of (i) a feature set potentially available in real-time prior to HFrEF diagnosis, (ii) an expert-adjudicated HFrEF target output, and (iii) intraoperative data contextualized to surgical and anesthetic interventions as a means for improved detection of HFrEF.

Our work explores hemodynamic responses recorded within the intraoperative record as a novel diagnostic data source. As analogous to a cardiac stress test, this study demonstrates the intraoperative record to be a potentially valuable source of diagnostic information. Our technique of intraoperative record time segmentation was critical to intraoperative feature development; techniques for segmenting the perioperative EHR into periods have been previously described.^28,29 Our work builds upon these definitions by exploring hemodynamic perturbations occurring during transition phases overlapping across periods, including induction of anesthesia, intubation, surgical incision, and extubation. Although such perturbations are not induced in a prescribed manner and segments vary by surgery type (e.g. non-cardiac versus cardiac), response profiles to perturbations represent a source of underutilized diagnostic information. Of note, pre-incision, peri-extubation, and peri-induction segments contained important features, and could be targets for future studies exploring intraoperative data to unmask latent cardiovascular disease.

Additionally, as surgical characteristics including longer duration and blood product transfusion were associated with undiagnosed HFrEF, it is possible that measures related to the degree of surgical insult and accelerating the course of chronic HFrEF (but not triggering acute overt postoperative deterioration, as were excluded) were identified in our study. However, given the observational nature of the study, such associations are not to be interpreted as causal mechanisms, and warrant further evaluation.

Study Limitations

Our models have important limitations which must be considered and addressed via external validation and prospective studies of real-time clinical decision support systems implementing such models:

1. We used observational data among patients within a single center. Although the large sample size of this study permitted analysis of a low-prevalence target output, the lack of a test dataset from a separate institution limits the generalizability of our findings. As the study involved a quaternary care center, availability of preoperative data likely differed from non-referral centers, and the undiagnosed HFrEF target output may have been underestimated due to follow-up evaluation at other institutions.

2. Despite good model performance as measured by AUROC, the low prevalence of the target output may have led to the low positive predictive value of the models developed. Therefore, for the model to be clinically actionable at acceptable sensitivity thresholds, highly specific confirmatory testing (e.g. echocardiography and/or cardiac biomarkers) would be required following detection and prior to clinical decision making (e.g. referral for potential diagnosis and management of HFrEF).

3. Although all machine learning model features used were potentially available immediately following surgery, features dependent on manual entry - including procedural codes and diagnoses - require techniques for health data capture beyond the scope of this study (e.g. natural language processing of free text, automated procedure classification).

4. Our undiagnosed HFrEF target output operated under the assumptions that the disease process is chronic and progressive, and that EHR clinical signatures of the evolving disease - contained within preoperative data or unmasked by intraoperative stressors - can be detected up to 730 days prior to current practice standards. This approach is congruent with national guidelines for HF classification, which follow a stepwise progression from Stage A (at risk for HF) to Stage D (medically refractory HF)³⁰ and has been validated in recent machine learning-based heart failure predictive modelling literature.³¹ Furthermore, threats to these assumptions were mitigated by clinical expert adjudication of all undiagnosed HFrEF cases, excluding those iatrogenically induced or explainable by processes other than natural progression. However, a ‘gold standard’ HFrEF target output, involving prospective cardiologist evaluation on the day of surgery, was infeasible.

5. It remains unclear what provider actions would most benefit to patient care, should the models be deployed in a real-time clinical decision support system.

6. To address concerns of data leakage potentially undermining the machine learning models developed, we excluded patients with known HFrEF, for which an early detection system would offer limited clinical utility. Thus, although the prevalence of known HFrEF among cases eligible for study inclusion (2,354 [3.1%] of 74,758 cases) was comparable to previous studies of non-cardiac surgical populations,^32,33 our model was optimized to detect patients with subtle, precursor stage HFrEF, rather than patients with a previous diagnosis of HFrEF potentially demonstrating obvious signs of the disease. This remains important when applying our models to prospective surgical populations potentially including known HFrEF patients. To estimate how our models may handle such patients, we performed a sensitivity analysis including known HFrEF patients, but limiting features to intraoperative data less susceptible to data leakage. However, this analysis should be regarded as hypothesis-generating; our model should not be used to infer judgments of disease progression among patients with known HFrEF.

Conclusion

In summary, we report newly diagnosed HFrEF within 730 days postoperative to occur in 0.41% of adult patients surviving major non-cardiac surgery. Through machine learning, we model patterns within preoperative data and intraoperative response profiles to surgical and anesthetic interventions to detect HFrEF potentially in early stages. As developed by this study, the technique of intraoperative record segmentation by overlapping anesthetic and surgical interventions may prove to be a useful paradigm for future studies seeking analytic approaches to granular intraoperative data. Our findings may guide development of perioperative systems for early diagnosis and management of HFrEF; however, future studies must first be performed to (i) externally validate the detection algorithms, (ii) assess the feasibility of embedding algorithms into the EHR at the point of care, and (iii) understand the clinical effectiveness of such clinical decision support algorithms.

KEY POINTS SUMMARY.

Question:

Among adult patients undergoing general anesthesia for non-cardiac surgical procedures, can machine learning techniques leveraging granular perioperative data be used to detect patients who are later diagnosed with heart failure with reduced ejection fraction (HFrEF) within 2 years following surgery?

Findings:

Preoperative data combined with intraoperative data integrated via machine learning models demonstrated good performance by area under the receiver operating characteristic curve, but had low positive predictive value, largely driven by the low prevalence of HFrEF in this cohort.

Meaning:

Machine learning-based approaches using perioperative data to detect undiagnosed HFrEF are feasible, yet require further evaluation to determine usability for improving early diagnosis and medical management of such patients.

Glossary of Terms

ASA

American Society of Anesthesiologists

ACE

angiotensin converting enzyme

AUPRC

area under the precision-recall curve

AUROC

area under the receiver operating characteristic curve

BMI

body mass index

BNP

B-type natriuretic peptide

CI

confidence interval

CKD-EPI

Chronic Kidney Disease Epidemiology Collaboration

CPT

Current Procedural Terminology

DBP

diastolic blood pressure

EGFR

estimated glomerular filtration rate

EHR

electronic health record

EKG

electrocardiogram

ETCO₂

end-tidal carbon dioxide concentration

FFP

fresh frozen plasma

F/S

feature selection

FiO2

fraction of inspired oxygen

HbA1c

hemoglobin A1c

Hgb

hemoglobin

HF

heart failure

HFpEF

heart failure with preserved ejection fraction

HFrEF

heart failure with reduced ejection fraction

HIV

human immunodeficiency virus

iCal

ionized calcium

ICD-9/10

International Classification of Diseases, Ninth/Tenth Editions

ICD

implanted cardioverter-defibrillator

INR

International Normalized Ratio

IQR

interquartile range

LASSO

least absolute shrinkage and selection operator

LVEF

left ventricular ejection fraction

MAP

mean arterial pressure

MPOG

Multicenter Perioperative Outcomes Group

NPV

negative predictive value

NSR

normal sinus rhythm

PACs

premature atrial complexes

PEEP

positive end-expiratory pressure

PIP

peak inspiratory pressure

PP

pulse pressure

PPV

positive predictive value

PVCs

premature ventricular complexes

PRBC

packed red blood cells

RMS

root-mean-square

SBP

systolic blood pressure

SD

standard deviation

SVT

supraventricular tachycardia

Vfib

ventricular fibrillation

WBC

white blood cell count