Graph convolutional network-based fusion model to predict risk of hospital acquired infections

Abstract

Objective

Hospital acquired infections (HAIs) are one of the top 10 leading causes of death within the United States. While current standard of HAI risk prediction utilizes only a narrow set of predefined clinical variables, we propose a graph convolutional neural network (GNN)-based model which incorporates a wide variety of clinical features.

Materials and Methods

Our GNN-based model defines patients’ similarity based on comprehensive clinical history and demographics and predicts all types of HAI rather than focusing on a single subtype. An HAI model was trained on 38 327 unique hospitalizations while a distinct model for surgical site infection (SSI) prediction was trained on 18 609 hospitalization. Both models were tested internally and externally on a geographically disparate site with varying infection rates.

Results

The proposed approach outperformed all baselines (single-modality models and length-of-stay [LoS]) with achieved area under the receiver operating characteristics of 0.86 [0.84–0.88] and 0.79 [0.75–0.83] (HAI), and 0.79 [0.75–0.83] and 0.76 [0.71–0.76] (SSI) for internal and external testing. Cost-effective analysis shows that the GNN modeling dominated the standard LoS model strategy on the basis of lower mean costs ($1651 vs $1915).

Discussion

The proposed HAI risk prediction model can estimate individualized risk of infection for patient by taking into account not only the patient’s clinical features, but also clinical features of similar patients as indicated by edges of the patients’ graph.

Conclusions

The proposed model could allow prevention or earlier detection of HAI, which in turn could decrease hospital LoS and associated mortality, and ultimately reduce the healthcare cost.

INTRODUCTION

Hospital acquired infections (HAIs), also referred to as healthcare-associated infections, are nosocomial infections that include central line-associated bloodstream infections (CLABSIs), catheter-associated urinary tract infections (CAUTIs), methicillin-resistant Staphylococcus aureus (MRSA), Clostridioides difficile infection (CDI), and surgical site infections (SSIs).¹ In 2018, the annual incidence of HAI among hospitalized individuals was approximately 3%.² HAIs, which up to one-third or more are preventable, contribute to increased hospital length of stay (LoS), morbidity, mortality, and costs.³ HAIs are among the top 10 leading causes of death within the United States, with an annual attributed cost estimated to be up to $10 billion.^4–6 Due to the significant nature of HAI and risk for patient safety, many hospitals have adopted HAI reduction programs^7,8 and report HAIs to National Healthcare Safety Network (NHSN) within the CDC for tracking (https://www.cdc.gov/hai/data/index.html [accessed October 10, 2022]). We propose a graph neural network (GNN)-based model for individualized risk prediction for all types of HAI.

BACKGROUND AND SIGNIFICANCE

Current practices at identifying patients at risk for HAI tend to utilize a “one-size fits all” approach by using certain static variables (eg, intensive care unit admission and LoS) as predictors for relative risk. Individualized risk and probability determination for HAI could allow more informed decisions at a patient level for potential targeted preventive intervention. Moreover, utilization of static variables (eg, age) may not allow dynamic risk assessment during a hospitalization or notably, may not be possible if data are not available. In contrast, deep learning (DL) models are capable of learning from large amounts of minimally processed data and can ingest data from multiple modalities, such as patient clinical notes and imaging data, to potentially improve the prediction accuracy.^9,10 However, most prior DL models treat individual patients or hospitalizations as independent data points, which ignores the similarity between patients or their hospitalizations which is often considered in real-life clinical decision-making. In this study, we proposed leveraging graph convolutional neural networks (GNNs) as the backbone for HAI risk prediction to model the topological structure between patients. Graph convolutional modeling has been proven to be an effective DL architecture in recent years for several tasks where instances can form a meaningful and informative “neighborhood”.^11–15 Graph-based modeling allows the model to not only learn from features of the data instance (a node in a graph) but also from features of similar nodes where similarity definition is left up to the model designer. This scenario is different from conventional convolutional processing where data elements are processed in reference to their spatial neighborhood. GNNs have been widely applied in predictive modeling for autism spectrum disorder (ASD) and Alzheimer and dementia with patients as nodes and “neighborhood” defined as edges between patients with similar demographic characteristics.^11,12 GNNs have the advantage of handling missing data and have higher data modality agnosticism, both useful when applying risk prediction models to real life problems.^14,16

Thus, the purpose of this study was to develop a GNN-based fusion model to predict risk of HAI using electronic health record (EHR) data and compare model performance to traditional feed-forward neural network-based models and current clinical factor LoS in hospitals. We evaluated the model on both internal and external datasets. In an effort to better inform decisions on utility of such a model from an economic perspective, we performed cost-effectiveness analysis to determine cost-saving strategy based on the current model performance.

MATERIALS AND METHODS

This Health Insurance Portability and Accountability Act-compliant, retrospective study was approved by the institutional review board (IRB) of our healthcare institute and a waiver of informed written consent was granted. Our goal was to predict the HAI risk 24 h prior to the event under the notion that 24-h advanced notification may provide a practical timeframe for earlier intervention and possibly prevention.

Cohort selection

Patients were eligible if they were hospitalized for at least 2 days at 2 of the major sites within our healthcare institute (blinded to the reviewers) during 2018 and 2019. These years were chosen to avoid any potential confounding effects of the recent Covid-19 pandemic on the incidence of HAI (https://www.cdc.gov/hai/data/portal/covid-impact-hai.html). We included common HAIs tracked by NHSN for surveillance and as part of the National Action Plan to Prevent Health Care-Associated Infections released by the U.S. Department of Human and Health Services (https://www.hhs.gov/oidp/topics/health-care-associated-infections/targets-metrics/index.htm). The 5 types of HAI included were: (1) central-line associated bloodstream infection (CLABSI), (2) catheter associated urinary tract infections (CAUTI), (3) Methicillin-resistant Staphylococcus aureus (MRSA), (4) CDI, and (5) surgical site infection (SSI). We divided the training and test cohort based on HAI subtypes. Our first cohort included 4 types of HAIs (Figure 1A) but did not include SSI by nature of the inclusion criteria. This cohort is referred to as the “HAI cohort”. For SSI model training, referred to separately as “SSI cohort” (Figure 1B), we included every hospitalization with recorded Current Procedural Terminology (CPT) for anesthesia or recorded “operative note” in our dataset to include all patients who underwent surgery and therefore may be at risk for an SSI. The criteria used for building these cohorts also provided relevant control negative groups.

Figure 1.

Cohort selection flowchart for 2 HAI subgroups: (A) CLABSI, CAUTI, CDI, MRSA; (B) SSI. CAUTI: catheter-associated urinary tract infection; CDI: Clostridioides difficile infection; CLABSI: central-line associated bloodstream infection; HAI: hospital acquired infection; MRSA: methicillin-resistant Staphylococcus aureus; SSI: surgical site infection.

External data

We performed external validation by using data from another, geographically disparate site within our health system (Jacksonville, FL) using the same time period. As there are significant differences in practice patterns and patient demographics at the 3 sites, we speculated that such data division potentially emulates realistic external (to our training data) validation. Table 1 shows characteristics of both internal and external test sets for HAI and SSI cohorts, respectively.

Table 1.

Cohort characteristics for HAI cohort

		HAI cohort				SSI cohort
	Split (hospitalization, patients)	Train (34 171, 15 967)	Validation (4156, 1824)	Test (9703, 4459)	External test (7374, 4451)	Train (18 609, 14 615)	Validation (2149, 1662)	Test (5195, 4079)	External test (4818, 4127)
Age, avg±SD		59.9±20.5	59.5±20.5	59.9±19.9	62.4±15.5	58.9±19.7	58.9±19.6	59.2±19.3	62.1±15.1
	Female	59.5±21.3	58.8±21.4	60.0±19.9	61.6±16.2	58.4±19.9	58.4±19.9	59.3±18.9	60.9±15.9
	Male	60.3±19.8	60.2±19.7	59.9±20.0	63.1±14.9	59.4±19.4	59.3±19.3	59.1±19.7	63.1±14.3
Gender	Female	7522 (47.1%)	870 (47.7%)	2137 (47.9%)	2163 (48.5%)	6737 (46.1%)	731 (44.0%)	1931 (47.3%)	1962 (47.5%)
	Male	8444 (52.9%)	954 (52.3%)	2322 (52.1%)	2288 (51.4%)	7878 (53.9%)	931 (56.0%)	2148 (52.7%)	2164 (52.4%)
	Unknown	1 (∼0%)	0 (0%)	0 (0%)	0 (0%)				1 (0.0%)
Race	White	14 270 (89.4%)	1621 (88.8%)	3970 (89.0)	3572 (80.3%)	13 076 (89.5%)	1468 (88.3%)	3647 (89.4%)	3429 (83.1%)
	Black	433 (2.7%)	60 (3.9%)	127 (2.8%)	583 (13.1%)	362 (2.5%)	42 (2.5%)	94 (2.3%)	465 (11.3%)
	Asian	283 (1.8%)	34 (1.9%)	85 (1.9%)	103 (2.3%)	250 (1.7%)	43 (2.6%)	74 (1.8%)	78 (1.9%)
	American Indian/Alaskan Native	169 (1.1%)	16 (0.88%)	46 (1.0%)	16 (0.35%)	139 (1.0%)	19 (1.1%)	48 (1.2%)	9 (0.2%)
	Native Hawaiian/Pacific Islander	23 (0.1%)	2 (0.1%)	10 (0.22%)	8 (0.17%)	23 (0.2%)	2 (0.1%)	8 (0.2%)	6 (0.1%)
	Unknown	789 (4.9%)	91 (4.9%)	221 (4.9%)	169 (3.8%)	765 (5.2%)	88 (5.3%)	208 (5.1%)	140 (3.4%)
Ethnicity	Not Hispanic or Latino	14 792 (92.6%)	1680 (92.1%)	4129 (92.6%)	4092 (91.9%)	13 420 (91.8%)	1529 (92.0%)	3749 (91.9%)	3779 (91.6%)
	Hispanic or Latino	632 (3.9%)	82 (4.5%)	187 (4.2%)	232 (5.2%)	609 (4.2%)	65 (3.9%)	164 (4.0%)	224 (5.4%)
	Unknown	541 (3.4%)	62 (3.4%)	143 (3.2%)	127 (2.9%)	586 (4.0%)	68 (4.1%)	166 (4.1%)	124 (3.0%)
LoS	All infection	9.4±11.2	9.9±12.2	9.3±10.7	9.9±11.9	8.4±10.2	8.7±10.2	8.5±9.8	8.4±10.7
	Negative	9.1±10.5	9.6±11.6	8.9±9.6	9.7±11.5	8.4±10.1	8.7±10.1	8.4±9.6	8.4±10.6
	Infection positive	28.4±27.9	29.5±25.2	30.7±32.6	26.1±23.1	15.2±19.5	14.0±16.9	18.4±21.6	12.0±13.8
Surgery duration (minutes)	All					196.5±150.1	201.3±152.7	193.4±148.3	188.2±142.1
	SSI-negative					195.5±149.4	200.9±153.0	192.3±147.2	188.1±142.2
	SSI-positive					311.2±173.0	222.4±132.5	283.0±208.7	202.3±144.3
Comorbidities	Diabetes	3914 (24.5%)	410 (22.5%)	1075 (24.1%)	1324 (29.7%)	3084 (21.1%)	340 (20.5%)	844 (20.7%)	1055 (25.6%)
	Hypertension	8723 (54.6%)	963 (52.8%)	2441 (54.7%)	2862 (64.3%)	7904 (54.1%)	873 (52.5%)	2228 (54.6%)	2546 (61.7%)
	Cardiovascular disease	9434 (59.1%)	1093 (59.9%)	2642 (59.3%)	2744 (61.6%)	7434 (50.9%)	855 (51.4%)	2089 (51.2%)	2030 (49.2%)
Infection rate	Total	475 (1.4%)	57 (1.4%)	134 (1.4%)	114 (1.5%)
	CLABSI	182 (0.5%)	24 (0.58%)	52 (0.54%)	52 (0.71%)
	CAUTI	50 (0.1%)	8 (0.19%)	13 (0.13%)	8 (0.11%)
	MRSA	29 (0.08%)	4 (0.09%)	5 (0.05%)	10 (0.14%)
	CDI	230 (0.7%)	28 (0.67%)	70 (0.7%)	50 (0.67%)
	SSI					131 (0.9%)	18 (1.1%)	36 (0.9%)	45 (1.1%)

Abbreviations: CAUTI: catheter-associated urinary tract infection; CDI: Clostridioides difficile infection; CLABSI: central-line associated bloodstream infection; HAI: hospital acquired infection; MRSA: methicillin-resistant Staphylococcus aureus; SSI: surgical site infection.

Feature engineering for clinical data

For HAI positive cases, we included all the clinical data recorded starting from the time of hospital admission to 24 h prior to infection. For HAI negative cases, we used a time period from admission to 1-day prior to discharge so that minimal features related to discharge would be used for prediction. Data collected during the last 24 h of the hospital stay may include hints toward discharge from the hospital (eg, stopping IV fluids, shifting to oral medication, etc.) and inclusion of these data could result in a leak of an unfair signal to the model where each HAI-negative case can be identified by such discharge indicators. Similarly, very long hospitalization may result in a very dense feature vector. To avoid any bias for long but HAI-negative hospitalization, we clip the time period to 31 days from admission for HAI-negative hospitalization. We chose 31 days as the mean+SD for days between admission and HAI was 31 days in our dataset. Thus, this methodology could ensure that models’ decisions were not based on just the density of EHR features. For the SSI cohort, time period formation included time between admission and the surgery. We did not extend the time period to the date of SSI as we found that patients may be discharged between the date of surgery and the date of SSI event reported.

In the given timeframe, we incorporated multiple data elements from the EHRs without manual filtering and designed minimal computational processing blocks for each type.

Demographics

We included common demographic features, such as age, race, gender, and ethnicity information as categorical features after dividing age into bins of 10 years. Missing values were filled with “unknown” category labels.

Comorbidities

We extracted all comorbidities information recorded since the time of admission to the hospital as International Classification of Diseases (ICD-10) codes. Taking advantage of the ICD-10 hierarchy, we mapped individual codes to their corresponding subgroups to reduce the dimensionality. This mapping reduced dimensionality of ICD-10 information (from ∼70k individual codes to 284 subgroups). Each subgroup was used as a discrete feature. While we used count-based representation of comorbidities for node feature vector formation, edge-feature vector contained binary representations of comorbidities (presence/absence).

Procedure

Information regarding medical procedures performed on the patients was extracted from CPT codes. Like ICD-10 codes, we mapped individual CPT codes to their corresponding subgroups, thus capturing the similarity between codes and reducing the dimensionality to 215 subgroups. Each subgroup was used as a discrete feature which was treated for node and edge feature vector formation the same way as comorbidities.

Laboratory tests

Based on clinical experience, we selected 61 of the most commonly encountered laboratory tests. We mapped values of these test results to normal/abnormal based on the recorded normal range of test results. Missing tests for any hospitalization are recorded as “UNKNOWN.”

Vitals

Vitals for hospitalized patients are more frequently captured than other EHR elements. We included temperature, mean arterial pressure (MAP), heart/pulse rate, SpO₂, and pain score in our experiments. We converted the sequence of each vital record for a patient to its mean rate-of-change where we assume that the normal values form the first element of the sequence. This preprocessing allowed us to capture temporal change in vitals.

In addition, we calculated the length of the stay in hospital (LoS) as a continuous feature for HAI prediction model while days spent in the hospital before surgical procedure (days-prior-to-procedure) was used as a feature for SSI prediction.

Graph neural network modeling

We designed the GNNs for predictive modeling with patients as nodes and “neighborhood” defined as edges between patients with similar demographic and clinical characteristics. As shown in Figure 2A, each node contains all EHR data elements collected for the corresponding hospitalization. Two types of feature vectors are created from these data elements for ith node; $n_{i }$and $e_{i }$. Vector $n_{i }$denote node features and is a rather straightforward concatenation of all available EHR data elements listed in the previous section. We experimented with different combinations of data elements for vector $e_{i }$ and found a combination of demographics, CPT and ICD-10 diagnosis codes to be the most effective. Therefore, we define $e_{i }$ by concatenating categorical demographic features and binary representation of CPT and ICD subgroups. Edge between nodes i and j is defined when similarity between vectors $e_{i }$ and $e_{j }$ exceeds a certain threshold. We use cosine similarity to define similarity between the node vectors ($e_{i }$, $e_{j }$) and treat threshold value as model hyperparameter. This setting allows the model to incorporate not only a diverse set of EHR data elements in the prediction process but also perform 2-fold learning for 3 important data elements, including demographics, comorbidities, and medical procedures which are employed as both node features and basis for edge formations.

Figure 2.

(A) Graph construction—node features and edge formation, (B) GCN training operation assesses importance of different edges. GCN: graph convolution.

As shown in Figure 2B, the model is provided with a graph structure with defined node features and weighted edges ($w_i$). Model, through the process of training, can learn aggregate functions to merge node features with “messages” received from neighboring nodes in the form of weighted combination of node features of the neighboring nodes. Hence, the model learns to assign certain “importance” (${w\prime }_i$) to neighbors of a node while making a prediction for the node. Since we designed 2 layers of graph convolution (GCN), the prediction of a node not only depends on its own neighbors but the neighbors of its neighbors as well. While the first layer of GCN aggregates node features with their neighbors’ “messages”, the next layer creates an abstraction of these already aggregated messages to derive high-level knowledge. Thus, even the nodes not connected in the original graph through a direct edge are able to influence the prediction for a node through a multilayer graph neural network and derive new relational information.

Many variations of graph convolution have been proposed which often differ from each other in terms of incorporation of “messages” from neighboring nodes while learning node representation. Many of these variations are designed for transductive learning with limited ability to process unseen nodes or graph structures. We used the SAGE (SAmple and aggreGatE) graph convolution network (GraphSAGE) in our modeling scheme that optimizes a sample aggregate function to collect “messages” from neighboring nodes while generating vector embedding of a node.¹⁷ At the time of inference, this optimized aggregate function is used to collect and process messages from neighboring nodes in potentially unseen graph structures. Thus, it can inductively reason to assign labels to unseen nodes in unseen graph structures.

Cost-effective analysis

We performed a simple, decision-analytic model with Monte Carlo simulation (TreeAge Pro 2022, Williamstown, MA, USA) to evaluate the preferred strategy based on effectiveness and cost of using an AI model to screen for HAI risk. We followed the consolidated Health Economic Evaluation Reporting Standards (CHEERS [http://www.equator-network.org/reporting-guidelines/cheers/]) reporting guideline.¹⁸ Our model assumptions and parameters are outlined in Table 2. Our base case scenario assumes a hospitalized patient with demographics reflecting the mean we encountered in our cohort. For model simplicity, we assumed all patients had the same susceptibility and risk of developing HAI. Treatment costs were not incorporated. For each patient, 2 strategies could be employed (Figure 3): (1) using our best performing fusion model to predict 24-h risk of developing HAI starting from 3 days after admission; or (2) using the traditional LoS² as a marker for identifying patients at risk using performance metrics from our LoS model on our cohort. For the window of predicting and developing HAI, we considered a time horizon as a period of hospitalization for 24 h. Because of a short time horizon, we did not incorporate Markov modeling with different health states with transitions between noninfected to infected and vice versa. For both strategies, we assumed that true positive prediction of the models decreases the cost of HAI, under the assumption that earlier detection and screening could lead to a more protracted course. For each model parameter, low and high ranges were provided from the literature. One-way sensitivity analysis of all model parameters was performed. Finally, to further address uncertainty around input parameters, Monte Carlo probabilistic sensitivity analysis was performed using a sample size of 100 000. Gamma distributions were assumed for costs and beta for probabilities. We used a willingness-to-pay (WTP) threshold of $50 000. Outcomes were expressed as quality-adjusted life years (QALYs). A strategy with higher cost and lower QALY was considered dominated. Incremental cost-effectiveness ratio (ICER) was calculated by dividing the differences in costs and QALYs. Net monetary benefit was also calculated by multiplying WTP and effectiveness and subtracting the cost. The strategy with higher ICER below WTP threshold and Net Monetary Benefit (NMB) represented the most cost-effective strategy.^19,20

Table 2.

Model input parameters

Parameter	Baseline value	Sensitivity analysis range	Source(s)
Costs ($USD)
Cost of AI inference per patient	10	5–15	Assumption
Cost of having HAI	28 395.48	603–65 245	Zimlichman et al21
Reduced cost of HAI from early screening/detection (a realistic assumption)	14 197.74	300–32 622.50	Assumption
Cost of intervention if screen positive for HAI	3375	1000–5000	Waters et al22
Costs of screening tests	100.55	56.41–124.22	Barker et al23
Probabilities
GNN fusion model sensitivity	0.754	0.702–0.806	This study
GNN fusion model specificity	0.694	0.687–0.700	This study
GNN fusion model false negative	1—GNN_sensitivity	0–0.246	This study
GNN fusion model false positive	1—GNN_specificity	0–0.306	This study
LoS model sensitivity	0.667	0.609–0.726	This study
LoS model specificity	0.624	0.616–0631	This study
LoS model false negative	1—LoS_sensitivity	0–0.333	This study
LoS model false positive	1—LoS_specificity	0–0.376	This study
Probability of having HAI	0.03	0.01–0.04	This study https://www.cdc.gov/hai/data/index.html Magill et al2 Zimlichman et al21
Probability of not having HAI	1—Prob_HAI	0.0–0.96	This study, https://www.cdc.gov/hai/data/index.html Magill et al2 Zimlichman et al21
Utilities
Utility of not having HAI	0.88	0.84–0.92	Konijeti et al24
Utility of having HAI	0.81	0.70–0.86	Zhang et al25
Utility of early HAI (protracted course)	0.845	0.81–0.88	Assumption

Abbreviations: AI: acquired infection; GNN: graph convolutional neural network; HAI: hospital acquired infection.

Figure 3.

Decision tree with 2 model strategies.

RESULTS

Our proposed model is a novel graph-based model that combines information from several EHR data elements; intuitive comparative baselines are formed by single-modality models using one EHR data element or traditional fusion modeling techniques. For comparative analysis, we trained multilayer perceptron (MLP)-based classifiers as single-modality models—5 separate models for each involved clinical data element. We chose late fusion technique as a representative of traditional fusion modeling and trained a meta-classifier over label probability estimates collected from MLP-based single modality classifiers. In Table 3, we present the performance of the baseline and GNN models on the internal and external dataset using the standard statistical measures (area under the receiver operating characteristics [AUROC], sensitivity, specificity) along with their standard error (SE). Figure 4 shows the visual representation of the ROC plots.

Table 3.

Performance comparison for HAI and SSI prediction tasks with standard error (SE); bold font shows the best performance. “ –” shows blank due to not being applicable

Modality	HAI prediction						SSI Prediction
	Internal test set			External test set			Internal test set			External test set
	Sensitivity	Specificity	AUROC	Sensitivity	Specificity	AUROC	Sensitivity	Specificity	AUROC	Sensitivity	Specificity	AUROC
Length of hospital stay (HAI)/Days prior to the procedure (SSI)	73.9±3.7	62.3±0.5	73.7±2.1	66.7±4.5	62.4±0.6	65.1±2.7	71.1±7.0	48.4±0.7	60.2±3.6	76.8±5.6	46.9±0.7	60.8±3.2
Surgery duration	–	–	–	–	–	–	70.4±9.0	53.4±1.0	63.5±5.2	66.7±12.5	47.3±1.4	52.1±6.9
Demo	45.5±4.5	66.2±0.5	54.7±2.8	43.0±4.7	58.8±0.6	50.3±2.9	51.1±7.5	54.6±0.7	54.8±4.1	41.1±6.5	54.6±0.7	46.9±3.6
CPT	39.6±4.5	74.3±0.4	56.0±2.7	44.7±4.8	56.6±0.6	51.8±2.6	95.6±3.0	8.5±0.4	67.1±3.9	60.7±6.4	64.6±0.7	66.2±3.3
ICD	53.0±4.4	72.9±0.5	64.9±3.1	54.4±4.5	77.2±0.5	66.2±3.1	93.3±3.8	5.3±0.3	62.3±4.5	51.8±6.6	62.0±0.7	60.8±3.7
Lab tests	56.0±4.4	60.3±0.5	62.0±2.6	57.9±4.6	54.6±0.6	58.0±2.8	66.7±7.1	51.2±0.7	57.5±3.6	66.1±6.3	49.2±0.7	56.6±3.7
Vitals	67.2±4.2	60.1±0.5	67.9±2.0	57.9±4.6	51.4±0.6	53.6±2.7	48.9±7.6	72.8±0.6	61.4±4.6	53.6±6.6	60.8±0.7	54.5±4.0
Late fusion	80.6±3.4	70.3±0.5	82.4±1.6	67.5±4.4	67.1±0.6	69.0±2.6	68.9±7.0	57.7±0.7	68.5±3.6	58.9±6.6	63.7±0.7	61.5±3.8
GNN	72.4±3.9	84.6±0.4	86.2±1.5	75.4±4.0	69.4±0.5	79.3±2.1	80.0±6.0	66.9±0.6	79.3±2.9	64.3±6.5	72.7±0.7	76.0±3.2

Abbreviations: AUROC: area under the receiver operating characteristics; CPT: Current Procedural Terminology; GNN: graph convolutional neural network; HAI: hospital acquired infection; ICD: International Classification of Diseases; SSI: surgical site infection.

Figure 4.

ROC plots for HAI prediction task on (A) internal and (B) external (B) test sets, and SSI prediction task on (C) internal and (D) external (B) test sets. HAI: hospital acquired infection: ROC: receiver operating characteristics; SSI: surgical site infection.

The proposed GNN outperformed all baselines in terms of AUROC for HAI prediction. For internal testing, late fusion achieved better sensitivity than GNN at the cost of significantly lower specificity which would generate more false negative cases. External validation highlights generalization capabilities of the various models. While late fusion suffered a huge performance loss on external data (13 percentage points in AUROC), relative performance loss for GNN was smaller (6 percentage points in AUROC). In addition, GNN achieved better sensitivity and specificity on the external set than all baselines. It can be also observed that length of hospital stay, which is one of the current clinical predictors of HAI, only produced a moderate performance on the internal (0.74 AUROC) and external data (0.65 AUROC).

The model for estimation of SSI demonstrated a similar performance trend. The GNN-based model performed better than all baselines in terms of AUROC. It also showed a smaller drop in performance when tested over external data, 3% points drop compared with 7% points drop for traditional fusion. Surgery duration, which is one of the prime factors for SSI risk estimation, only produced 0.63 AUROC on the internal dataset and 0.52 on the external data.

Cost-effective analysis

Cost-effective analysis demonstrated that the GNN fusion model strategy dominated the standard LoS model strategy predominantly on the basis of lower mean costs of the fusion model ($1651 vs $1915) in all iterations of the Monte Carlo simulations. Under a WTP threshold of $50 000, the fusion model had the higher net monetary benefit ($42 284 vs $42 015). Sensitivity analysis on all input parameters demonstrated that the ICER can shift based on certain parameters, namely diagnostic performance of the 2 models, cost of HAI, and the probability of getting an HAI, not unexpectedly (Figure 5A). However, overall, the fusion model maintains dominance with lower costs and greater effects than the baseline model at a WTP of $50 000/QALY. One-way sensitivity analysis on all parameters of the model showed that only the diagnostic performance parameters of the fusion and LoS models could impact NMB, expectedly. All other input parameters would have no effect on the NMB (Figure 5B).

Figure 5.

Cost-effective analysis results: (A) Net monetary benefit plot of the 2 models using a WTP of $50 000; (B) Tornado diagram demonstrating 1-way sensitivity analysis for all model parameters. WTP: willingness-to-pay.

Visualization dashboard

We designed a dashboard for visualization for our model prediction in an intuitive way which derived the model explanation based on the learned similarity between the neighbors and features weights in the nodes (see Supplementary Material). Figure 6 shows the panels used for explanation. Top-left panel shows a subgraph that the model deemed important for making a prediction for the node depicted in the center of the circular network layout. Nodes are colored based on their groundtruth infection labels (red for infection positive and green for infection negative). Top right panel shows the importance assigned to the top 25 features of the node corresponding to the patient (depicted by the center node in subgraph). Color of bars indicates whether the feature value was available (orange) or unknown (blue) in the EHR of the patient. Table at the bottom shows significant characteristics of the patients and its closest neighbor in terms of model assigned edge importance. This helps the user understand how patients in the model-selected subgraph are similar. In the example shown below, the proposed model successfully assigned high infection probability to this patient while connecting it very closely with another patient of similar age, comorbidities (both patients belong to racial minority communities in the dataset). Even though the closest patient does not get an infection, the model is able to learn from the second hop neighbors which contains one positive case which shows that multilayer graph learning helps to create information abstraction. Node features’ importance plot shows several abnormal lab values for the patient.

Figure 6.

Visualization dashboard for the HAI model where the graph shows target node neighbors and panel shows the important node features. HAI: hospital acquired infection.

DISCUSSION

In this study, we developed a GNN-based fusion model to predict HAI and SSI 24 h prior to the event. The proposed model creates a novel representation of topological relationship between hospital admissions using a graph, where each node is one hospital admission and the edges represent the similarity between admissions based on patient characteristics. Our GNN fusion model outperformed all other traditional fusion and single-modality models with a higher AUROC. Further, we validated our model on patients admitted to a geographically disparate site within our health system, which we considered as external to the training data, and demonstrated similar higher performance than baseline models with good generalizability. We also designed a dashboard for the direct interaction with the model and allow visualization of the learnt subgraph.

There have been previous attempts at building an HAI risk estimation model, though with some limitations. Friedant et al²⁶ proposed a simple scoring method for HAI prediction, but their methodology was focused on a narrow set of patients (ie, patients admitted to the hospital after acute ischemic stroke). Thus, their cohort has a high positivity rate for HAI (∼14%), different from our cohort that consists of all hospitalized patients (positivity rate of ∼1.5%). Our approach does not restrict the prediction to certain disease groups and can be applied to a wider population without preselection. Practically, this would increase the applicability of the model as development of a downstream in-hospital event (eg, acute stroke) may not be known. Similarly, Habibi et al²⁷ developed a predictive model for only shunt infection among pediatric patients with hydrocephalus with a cohort with high positivity rate for infection. They used expert-selected pediatrics-focused clinical features for prediction such as birth weight and prematurity. Development of such a model needs manual curation of clinical data which is a heavily time-consuming process and may limit the model’s ability to explore new knowledge.

CLABSI has been a popular target for HAI prediction models, though these predictive models are limited in terms of features used for prediction.^28,29 A small number of manually selected features have been used for prediction such as demographic features and comorbidities like HIV, leukemia, and lymphoma.²⁸ Parreco et al relied on structured scores for predictive modeling including Oxford Acute Severity of Illness Score (OASIS), Sequential Organ Failure Assessment (SOFA), Simplified Acute Physiology Score (SAPS), SAPS II (SAPS II), Acute Physiology Score III (APS III), Logistic Organ Dysfunction Score (LODS), and Elixhauser comorbidities. It is unclear if these models can generalize to unseen populations with different comorbidities distribution or with limited availability of structured scores. Our GNN-based model shows generalizability and allows increased eligibility for model applicability by reducing reliance on the need of specific selected input features.

Another strength of our model is that our model predicts all types of HAI rather than focusing on a single subtype (eg, C. difficile).^30,31 Li et al³¹ focused on predicting complications after acquiring CDI. Oh et al³⁰ used a relatively large set of static and dynamic variables for CDI prediction and experimented on datasets collected from 2 different centers, though they developed 2 models for 2 centers rather than performing external validation. Chang et al³² devised a scoring system for prediction of HAI and also performed external validation, though they used a limited set of clinical features for prediction as fine-grained EHRs system was not available for the time period of their cohort (2003–2004) at the institute data were collected from Chang et al. Their randomly selected control group showed significantly different distribution for demographic features and major comorbidities.

 

Limitations

Our study has important limitations. The models were trained externally on a population of patients who were cared for in a highly integrated academic healthcare system with 89.5% Caucasian patients. Even though we externally validated the model on a geographically distanced center with different care practice, the patient population remained homogeneous. Further studies are needed to assess performance in a different patient population or health system with more diverse race and ethnic background.

CONCLUSION

The clinical significance of our study is that it provides a risk prediction model for all hospitalized patients. Recent studies have shown the cost and quality benefit of implementation of preventive measures for HAI.^21,23,33 Models such as ours could be used to better inform or triage constrained resources utilized in such preventive programs. Given the cost of HAI prevention programs, we also conducted a cost-effective analysis to highlight effectiveness from a hospital perspective. Our cost-effectiveness analysis revealed that the fusion model dominated the traditional risk factors since it costs less and yields better outcomes for the HAI risk prediction.

FUNDING

The work is partially funded by National Institute of Health grant 1U01CA269264-01.

AUTHOR CONTRIBUTIONS

AT developed and trained the proposed models, ran evaluation experiments, and contributed to manuscript writing. LL contributed to data collection. PE contributed to data curation and problem formation. ES contributed to data collection. KN contributed to data collection and problem formulation. BB contributed to cost-effectiveness analysis. JM contributed to cost-effectiveness analysis. IB led the technical design of the model and contributed to manuscript writing. BNP formulated the problem, performed cost-effectiveness analysis, and contributed to manuscript writing.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

Authors have no competing interests to declare.

DATA AVAILABILITY

The data underlying this article was provided by a third party by permission. Data will be shared on request to the corresponding author with permission of the third party.