Performance Degradation between Development and Deployment of a Predictive Model for Central Line-Associated Bloodstream Infections in Hospitalized Children

Abstract

Background

Central line-associated bloodstream infections (CLABSIs) are associated with substantial pediatric morbidity and mortality. The capacity to predict which children with central lines are at greatest risk of CLABSI could inform surveillance and prevention efforts. Our team previously published in silico predictive models for CLABSI.

Objective

To prospectively implement a pediatric CLABSI predictive model and achieve adequate performance in offline validation for implementation in clinical practice.

Methods

Most performant predictive models were deep learning models requiring substantial pre-processing of many features into 8-hour windows including the current day and up to 56 days prior for the current admission. To replicate this pre-processing, we created a novel infrastructure to (1) organize current-day data for all the relevant features and (2) create a staged historical data store for those same features with application programming interfaces to connect the two. We compared predictive performance of these scores for CLABSI in the next 48 hours with two labels, one based on manual review of positive blood cultures in children with central lines and another based on positive blood culture and receipt of at least 4 days of new IV antibiotics.

Results

The area under the receiver-operating characteristic (AUROC) fell from 0.97 from retrospective data to <0.60 despite multiple iterations of troubleshooting. Primary root causes included train/serve skew, feature leakage, and overfitting. Hypothesized secondary drivers were complex model specification, poor data governance, inadequate testing, challenging feature translation between real-time and historical data models, limited monitoring and logging infrastructure for troubleshooting, and suboptimal handoff between the model development and deployment teams.

Conclusion

Bridging the gap from predictive model development to clinical deployment requires early and close coordination between data governance, data science, clinical informatics, and implementation engineers. Balancing predictive performance with implementation feasibility can accelerate the adoption of predictive clinical decision support systems.

Background and Significance

Central line-associated bloodstream infections (CLABSIs) cause substantial morbidity and mortality, occurring in US children at least 1,463 times in 2023 in 114 pediatric hospitals reporting to the National Healthcare Safety Network (NHSN) (incidence rate of 0.94 CLABSIs per 1,000 device days). 1 CLABSIs in adults are associated with an attributable mortality of 12 to 25%, 2 with limited estimates in children in intensive care units showing similar values. 3 Each pediatric CLABSI represents a mean attributable cost of $55,646 in the United States. 4 Known risk factors for CLABSIs span patient factors (e.g., immunocompromised status, history of central line infections, etc.), line factors (e.g., central line type, location, number of lumens, etc.), treatment factors (e.g., increased line accesses, intralipids, etc.), and inadequate prevention bundles (e.g., mouth care, antimicrobial baths, etc.). 5 6 7 8 In addition to these risk factors, real-time clinical features such as fever and other vital sign trends, laboratories, and other documented care are likely signals that differ between children in the process of developing a CLABSI compared with those who will not develop a CLABSI.

Many CLABSI preventative interventions exist such as dressing changes, hygiene bundles, early line removal, and changes to treatment regimens to reduce line accesses or other risk factors. 5 6 However, each of these interventions requires extra resources and may lead to other risks—for example, fewer laboratory draws to reduce line accesses may lead to inadequate surveillance for other complications in hospitalized children. At our institution, a multidisciplinary team of infectious diseases physicians, infection preventionists, and nurses aims to reduce CLABSI through line “rounds” where they review CLABSI risk factors on a subset of children in the hospital with central lines to identify feasible preventative measures. However, the line rounds team does not have sufficient resources to evaluate every hospitalized patient with a central line as there are usually >150 across the health system at any given time.

We aimed to develop a predictive model to identify which hospitalized children with central lines were at highest risk of CLABSI in the next 48 hours. The intended workflow of this model was to be used as a non-interruptive monitoring system that would (1) help line rounds teams prioritize which patients to focus on and (2) help clinical teams in their decision-making about the risks versus benefits of adjusting care to reduce CLABSI incidence without compromising other aspects of care.

Solution

CLABSI Predictive Model Development

A full description of our predictive model development process and predictive performance based on retrospective datasets are described elsewhere. 9 10 11 Briefly, we collected retrospective data for children hospitalized for >24 hours at one of three freestanding children's hospitals in a single pediatric health system between January 1, 2013 and December 31, 2018 with a documented central line at some point during the hospitalization, including those inserted at outside facilities. This work was determined to be non-humans subject research by the institutional review board of Children's Healthcare of Atlanta (STUDY STUDY00001526).

CLABSI Labels

Two CLABSI labels were extracted, one based on the determination of a hospital committee that met regularly to review all positive blood cultures from children with central lines to discuss whether they met NHSN definitions for CLABSI. 12 These data were manually collected and entered in a REDCap 13 database without automated data validation for medical record numbers and specific dates. This list was felt to be highly specific for CLABSI but imperfectly sensitive for clinically relevant CLABSIs and at risk of missing examples due to missing or inaccurate entry of identifiers leading to inability to join to predictive features. A second label was created based on electronic health record (EHR) queries for a positive blood culture and at least 4 days of new IV antibiotics not administered between 24 and 72 hours prior to the blood culture time.

Collection of Potential Features

Potential predictive features were extracted for all hospitalized children with central lines in the cohort. Data were extracted in multiple tables with an encounter key and patient key to join across. Tables extracted included patients, encounters, diagnoses, indwelling line entry data (e.g., line characteristics at the granularity of each line), line timestamped data (e.g., ongoing line care that could be one line to many documentations such as flushes, dressing changes, etc.), medication administrations, vital signs, laboratory results, other flowsheet data, advanced therapies (e.g., endotracheal tubes, extra-corporeal membrane oxygenation), and immunizations. In addition to raw data, a set of features were derived from these data based on consultations with a series of subject matter experts (e.g., on total parenteral nutrition [yes/no], on intralipids [yes/no], etc.). Of note, at the time of model development, there was not yet a data governance program in place at our institution to validate commonly used features or common code repositories.

Model Development Processes

Retrospective EHR data were extracted for model training and shared by a clinical informaticist (EWO) at Children's Healthcare of Atlanta with data scientists at the Emory Department of Biomedical Informatics. A PhD candidate in Biomedical Informatics created scripts in MATLAB and Python with supervision by a PhD in Computer Science and a clinician and informaticist to pre-process the data. The initial training dataset included 338 features that were further narrowed down using LASSO feature selection. Subsequently, a series of predictive models were created beginning with a logistic regression model with L1L2 (elastic net) regularization and eXtreme Gradient Boosting (XGBoost), 9 followed by sequential deep neural network models (bidirectional long short-term memory [LSTM] with attention and focal loss) requiring longitudinal data 10 and EHR clinical notes. 11 The deep learning models were noted to have substantially improved predictive performance and particularly higher positive predictive value given the low prevalence of CLABSI, especially when viewed in 48-hour windows (i.e., the probability a hospitalized child with a central line at a particular time point would develop a CLABSI within the subsequent 48 hours).

The team determined that only the deep learning models met performance criteria necessary to proceed to prospective implementation. The final model included 135 unique features, 96 of which needed to be tracked across the hospitalization in 8-hour intervals. For example, for a prediction to be made at 8 a.m. on the current day, the model required the most recent hemoglobin in the 12 a.m. to 8 a.m. time window as well as a vector of all prior hemoglobin levels split into 8-hour windows (i.e., the most recent hemoglobin level between 12 and 8 a.m., 8 a.m. and 4 p.m., and 4 p.m. and 12 a.m. on the day before the prediction is being made and all prior hospital days up to 56 days). This processing needed to be repeated for each time-varying feature.

Model Translation

To prospectively implement the model in the EHR where it could eventually impact workflow, a multi-disciplinary team was convened including the original model developers (clinical informatics and data scientists), as well as a health system software engineer, data engineer, senior EHR analyst, and a data scientist.

Infrastructure Development

We developed novel infrastructure to merge historical data for a hospitalized patient (i.e., from prior to 12 a.m. on the date of prediction) with current-day data (i.e., after 12 a.m. on the date of prediction), as summarized in Fig. 1 .

Fig. 1.

Novel infrastructure for central line-associated bloodstream infection (CLABSI) model implementation. API, application programming interface; CVAD, central venous access device; DMZ, de-militarized zone; EHR, electronic health record; ETL, extract, transfer, load; IIS, Internet Information Services; RWB, Reporting Workbench; SQL, Structured Query Language.

At the time of model translation, the health system EHR, Epic Systems©, had a nightly extract, transfer, and load process from a hierarchical real-time transactional database to a relational database (Clarity®) to facilitate analytic work. To merge historical data for each feature with current-day data, we developed a staged database accessible via an application programming interface (API) that would find all children hospitalized with a central line as of 12 a.m. on the day of prediction and extract all features.

We then developed a Reporting Workbench (RWB) report in Epic to find all patients currently hospitalized with a central line and extract current-day data for all time-varying features split into 8-hour windows for the current day (e.g., for a 4 pm prediction, the RWB report would have columns for the most recent heart rate, hemoglobin, and other levels from 12–8 a.m. and 8 a.m.–4 a.m.).

Using Epic's Nebula cloud computing platform for custom machine learning models, we developed an application that was triggered by batched run of the RWB extract. When the scheduled batch was run, encounter IDs and the current-day features were delivered to Nebula's containerized cloud-hosted environment. Our application would then make a HTTP call through an API to the staged database for those same encounter IDs to gather historical features. Next, the application would aggregate and organize current-day and historical features into the necessary model data structure. The model was then run on the structured data to make a prediction. Finally, the prediction was filed as a score back to the Epic Chronicles® database.

The deployment environment required running all transformations and model calls using a set of pre-installed Python libraries. Therefore, the original pre-processing and model prediction code, partially written in MATLAB, had to be translated to work in the new environment. Of note, the PhD student who originally developed the code had taken a full-time position at another institution so new developers were responsible for both pre-processing translation, mathematical transformations (e.g., to standardize continuous variables), and developing new code to integrate with the EHR.

Additionally, the deployment environment was a stateless container, so in the initial implementation no intermediate steps or features were logged—only the score and a timestamp were available as part of API responses. To enable troubleshooting and monitoring after we observed a reduction in predictive performance, we extended the environment with custom logging infrastructure so that with each prediction, a separate API would log features used at the time of prediction into a relational database.

Results

The predictive model was implemented prospectively beginning April 2022. The time from deployment project kick-off to initial implementation took over 18 months. During validation, the model was available in production as a Patient List Column displaying the model score for users who manually added the column, but was not advertised to any clinical stakeholders.

The first evaluation of predictive performance was performed in November 2022. The area under the receiver operating characteristic (AUROC) was found to have fallen from 0.97 in the initial publications to 0.502 for the REDCap Label (manually curated CLABSIs) and 0.536 for the PSI label (automated surveillance). Our primary hypothesis for the root cause of this drastic drop was train/serve skew, defined as a difference in the distribution of data or the way data are processed between the training phase and the serving phase, likely due to a mistranslation of logic.

A team aimed to review the original code and transformations including reaching out to the original student who developed the model. However, we had no logs of what the features were at the time of each prediction to check against the EHR. To enable further investigation into our train/serve skew hypothesis, we developed novel logging infrastructure and created reporting to plot multivariate feature distribution profiles using tableplot visualizations ( Fig. 2 ). 14 These tableplots allowed the team to quickly visualize the relative density of each feature, or how often it was present or absent, and made it easy to compare with expected rates (e.g., we might expect more hemoglobin features to be available but fewer obscure laboratory tests). As a result of these initial exploratory analyses on data filed between January and March 2023, we identified several laboratory results that were not crossing appropriately into the model (feature leakage). Code updates were made and subsequent feature profiling showed laboratory results crossing, but the AUROC only improved to 0.575 (REDCap label) and 0.559 (PSI label) between April 2023 and January 2024 ( Fig. 3 ).

Fig. 2.

Example tableplot of feature distribution profiles.

Fig. 3.

Area under the receiver operating characteristic for two central line-associated bloodstream infection (CLABSI) labels before and after updates to ensure laboratory results crossed to the model container.

To further investigate the cause of the skew in performance between model training and model serving, we manually reviewed individual predictions and their features (from the new logging infrastructure) and attempted to check all mathematical transformations relative to what was expected. We also replicated the retrospective data extraction done for model development but for the time period after prospective implementation and then compared results to the prospective implementation. Despite these investigations and changes at each stage, we were unable to achieve an AUROC close to the original publications based on retrospective data. Additional proposed root causes included feature leakage (when the training data include information that would not be available at the time of prediction due to a mishandling of time dependencies) and model overfitting (when the model perfectly learns the training data due to highly complex model relative to the amount and noisiness of the training data). Investigating both of these root causes would require the ability to access and fully reproduce the training dataset, which the local data science team did not have the ability to do. Due to resource constraints, the CLABSI predictive model was decommissioned in January 2024 for data science resources to be prioritized elsewhere.

Discussion and Lessons Learned

A pediatric CLABSI predictive model based on a sequential deep neural network that appeared to predict extremely well in initial development on a test dataset from a retrospective data extraction exhibited significant train/serve skew with near-complete loss of predictive performance when converted to a prospective implementation in the EHR. Although the investigator team was unable to identify the exact root causes and mitigate the errors, numerous challenges likely played a role in this predictive clinical decision support (CDS) failure including people, process, and technology challenges.

People

Challenge: Many predictive model publications are completed by a team composed of data scientists and clinical subject matter experts. These teams may lack technical awareness of EHR constraints for model implementation as well as clinical informatics and human factors expertise to understand the workflow and leverage points where a predictive model could improve care processes.

In our case, the original model conception, code development, and publication were done by an academic team of data scientists and a junior (at the time) clinical informaticist (EWO) with limited expertise in predictive model implementation in the EHR. The lack of awareness of EHR constraints led to model development in an unsupported language (MATLAB versus Python) and complex feature engineering that was difficult to replicate for current-day data.

Recommendation: Model goals, definitions, and development plans should include not only data scientists and clinical subject matter experts but also human factors engineers, clinical informaticists, and implementation engineers with experience in EHR integration of predictive models. This approach can ensure that (1) workflow is prioritized and prediction targets are more appropriately defined, (2) the data science team can be aware early of EHR constraints to reduce implementation work later on, and (3) the team can intentionally make tradeoffs in predictive performance versus implementation feasibility.

Challenge: Many predictive models are developed by independent academic teams whose primary focus is publication and future grants as opposed to implementation and the many operational challenges associated with enterprise-grade software, project management, and maintenance.

In our case, much of the coding work was done by a talented PhD student with supervision. Much of the data wrangling occurred in working sessions with a clinician, such that limited documentation was generated. Additionally, by the time of transition to deployment that student had left the organization and, while generous with their time, could only provide limited support given new responsibilities.

Recommendation: Predictive model implementation teams should establish roles and responsibilities (e.g., RACI: Responsible, Accountable, Consulted, Informed) early in the project including redundancies and handoff plans, understanding that specific team members may become unavailable over the course of the implementation. Documentation standards should be defined and reinforced throughout the model life cycle from development through implementation for clinical use to facilitate on-boarding of new team members, allow team members with different perspectives to provide feedback, and reduce risks from staff turnover.

Processes

Challenge: Predictive model development is often done by independent research teams with variable connection to health system data governance structures. This approach can lead to inadequately validated labels and features that may not translate prospectively.

In our case, a very large number of CLABSI labels and potential predictive features were extracted by a single clinical informaticist (EWO) for model development with only informal validation of accuracy and without code review. Thus, there may have been inaccuracies in both labels and predictive features in the training data used.

Recommendation: Data scientists and clinical investigators should engage operational analytics stakeholders early to ensure data feeds align with organizationally defined metrics. Health systems aiming to leverage predictive analytics may first need to ensure their data governance structures are well developed and that trusted data assets are available.

Technology

Challenge: Epic Systems architecture at the time of implementation required integration of very different data structures to include both current-day and historical features. Although some predictive models may only require current state variables or simple transformations, a sequential deep neural network required longitudinal data to capture trends and patterns in a patient's health trajectory, enabling it to identify signals that predict the onset of specific events, such as CLABSI. Mixing these data models increases the chance of translation errors.

In our case with a complex model of 135 features, many requiring longitudinal data, there was a strong chance of mistranslation accentuated by the different data models for current-day (transactional database) and historical data (relational database).

Recommendation: First, model developers should recognize EHR constraints and intentionally decide on tradeoffs between predictive performance and ease of implementation. Second, EHR vendors could develop tools (e.g., the DEPLOYR technical framework) 15 that reduce translation needs and better approximate the data structures in which model development and model deployment occur to reduce train/serve skew.

Challenge: At the time of implementation, there was no ability to log the data that informed a particular prediction as seen by the model at the time the prediction was made. Thus, while we could log the prediction score itself, we could not easily figure out what the inputs were for that particular prediction. This limitation made troubleshooting much more challenging.

In our case, an entire class of data (laboratory results) was not crossing appropriately into the model. However, we were unable to identify this issue until developing custom logging infrastructure.

Recommendation: EHR vendors could simplify the ability to log data throughout the lifecycle in an application including initial ingestion, transformation, and eventual predictions. This would allow predictive model implementation teams to more easily pinpoint translation problems. Such an infrastructure would also be very helpful for monitoring feature drift even with a successful implementation.

Limitations

This study has several limitations. First, it was performed at a single health system with a specific EHR vendor and unique documentation patterns and organizational structures, which have changed over time. Challenges and lessons may not be applicable to many other health systems, although we believe that many are despite differences in data structures. Second, technology and EHR vendor capabilities are changing rapidly. Many of the challenges we experienced are now being addressed by new applications and vendors and may not be the most important predictive CDS challenges going forward.

Conclusion

Predictive CDS has the potential to improve outcomes for patients, families, and clinicians. We developed a predictive model for CLABSI in hospitalized children that failed to maintain its predictive performance when implemented in the EHR prospectively due to model complexity and multiple implementation challenges. Future work will focus on developing models that are simpler to implement to see if they can provide clinical utility even if predictive performance is not equal to what appears to be achievable with complex models on retrospective datasets. Nonetheless, many of the most powerful predictive algorithms require large volumes of data to train and complex data structures to ingest with each prediction. To achieve the promise of predictive CDS, health systems need to address challenges with people, processes, and technology in concert with EHR vendors and other stakeholders.

Clinical Relevance Statement

Predictive analytics has the potential to substantially improve patient outcomes through better targeting of resources and therapies. Most predictive models are developed with retrospective datasets. Deploying complex predictive models in production requires re-engineering how the model runs, leading to the potential for degraded performance. In this example, a pediatric central line-associated bloodstream infection model failed to replicate predictive performance in production compared with a retrospective dataset. Lessons learned can help improve translation of other complex models into the EHR correctly to achieve the potential of predictive models.

Multiple-Choice Questions

1. A clinical stakeholder and data science research team are planning to develop a new predictive model for a clinical outcome of interest with the goal of eventually implementing the model in clinical workflows via the electronic health record (EHR). Which of the following is NOT a best practice to facilitate implementation?

   1. Use organizationally governed data definitions for labels and features whenever possible

   2. Educate the data science team on EHR implementation constraints and tradeoffs

   3. Perform a workflow analysis to define prediction targets prior to creating the dataset

   4. Ensure the data science team uses the most cutting-edge machine learning techniques

   Correct Answer : The correct answer is option d. Use of governed data definitions for labels and features whenever possible increases the likely validity of the predictive model and reduces the risk of prediction errors from custom data processing that does not align with organizational workflows. Educating the data science team on EHR implementation constraints can help the team make decisions on tradeoffs between predictive performance and implementability—for example, a simpler model that aligns with EHR technical constraints may be much less resource intensive to implement and may provide adequate predictive performance to achieve the improvement goals. Performing a workflow analysis ideally with a human factors engineer before creating the dataset for prediction can help ensure the right problem is being solved and target where a prediction model is actually needed compared with other less resource-intensive solutions. Ensuring the data science team uses the most cutting-edge machine learning techniques is not critical to implementation success and may in fact increase the cost of implementation and the risk of train/serve skew if the modeling techniques are not easily supported in a real-time production EHR.

2. What is a critical implementation benefit for data science research teams from engaging with operational EHR stakeholders early in predictive model conception?

   1. To protect intellectual property and commercialize the predictive model once implemented

   2. To understand and reduce implementation challenges from EHR technical constraints

   3. To demonstrate the value of the research team to the organization

   4. To gain access to EHR built environments

   Correct Answer : The correct answer is option b. Combining the expertise of clinical subject matter experts, data scientists, clinical informatics, human factors engineers, and EHR implementation engineers as early as possible in the predictive model life cycle is recommended to ensure the right problem is being solved, the right dataset is produced to make the appropriate prediction in the workflow, and the predictive modeling approach is as simple as possible to implement in the EHR. In the absence of these stakeholders, data science teams can easily use software and techniques that are challenging to translate into the EHR, leading to errors and waste. Operational stakeholders are not likely to help with commercialization of a predictive model. Although connecting stakeholders can build relationships that eventually demonstrate value of the research team, the primary goal is to increase the likelihood of success of the predictive model to improve outcomes, which later on would demonstrate value of the research team. Although data scientist may want direct access to EHR built environments, often EHR analysts have much greater knowledge of EHR build and may be better positioned to support direct EHR configuration.

Funding Statement

Funding None.