In Response to, “Development, Implementation and Impact of an Automated Early Warning and Response System for Sepsis”

We greatly appreciate the letter (1) regarding our publication describing the development, implementation and impact of an automated Early Warning and Response System (EWRS) for Sepsis at our institution (2). We used established criteria for severe sepsis as predictors in our algorithm to expedite the derivation, validation and implementation of the EWRS. This resulted in implementation across a multi-hospital healthcare system within one year from the date that vital signs first became available in our electronic health record (EHR). Although the ability of our model as a predictive classifier was fair, its clinical utility was robust. With a positive likelihood ratio >5 and screen positive rate <5%, the model enabled the timely identification and care of patients at risk for sepsis, improved sepsis documentation, and potentially reduced mortality, thereby supporting the notion that simple classifiers can compete with more complex algorithms (3).

Although the original model implemented could have been improved using other variables and approaches, we were concerned at the time about the diminishing returns of additional complexity, particularly when deploying the model in a production environment with high demands. We thus favored simplicity at all stages, with the knowledge that complexity could be added once the clinical feasibility of the system was established.

More recently, we have applied machine learning algorithms to leverage “big data” available from our EHR, using random forest models to predict hospital readmissions (4) and improve sepsis predictions (in an initiative labelled EWRS 2.0). As Drs. Bhattacharjee and Edelson suggest, these methods can have advantages over using more manual regression approaches to select predictors, a particularly cumbersome process when an acutely ill inpatient can generate hundreds of variables and >1,000 data points daily (5). Yet, although machine learning algorithms can theoretically be constructed using all variables in real-time, from an implementation perspective, it can be difficult to continuously collect and maintain such a complete information set for each patient. Therefore, we strive to develop models using subsets of predictors without sacrificing performance.

What we’ve learned is that the development of highly accurate predictive algorithms using these approaches is often less complex than the technical and administrative aspects of implementing these algorithms into practice. To address these challenges, we are currently developing an open platform to allow researchers and data scientists to tap into the wealth of data available in the EHR and other connected devices in order to build, test, and deploy novel intelligent predictive solutions. Our goal in developing this platform is to accelerate the development and deployment of innovative solutions. Once in production, algorithm performance can then be improved iteratively as more data (in terms of volume and velocity) become available through technological advances such as streaming health data from wearables and other connected devices.

With the implementation of EWRS 1.0, we thus set the stage for the development of high performance predictive models, as well as the implementation of these models into practice, with the ultimate goal of improving the quality and value of care we provide.