Principles for Real-World Implementation of Bedside Predictive Analytics Monitoring

Abstract

A new development in the practice of medicine is Artificial Intelligence-based predictive analytics that forewarn clinicians of future deterioration of their patients. This proactive opportunity, though, is different from the reactive stance that clinicians traditionally take. Implementing these tools requires new ideas about how to educate clinician users to facilitate trust and adoption and to promote sustained use. Our real-world hospital experience implementing a predictive analytics monitoring system that uses electronic health record and continuous monitoring data has taught us principles that we believe to be applicable to the implementation of other such analytics systems within the health care environment. These principles are mentioned below:

• To promote trust, the science must be understandable.

• To enhance uptake, the workflow should not be impacted greatly.

• To maximize buy-in, engagement at all levels is important.

• To ensure relevance, the education must be tailored to the clinical role and hospital culture.

• To lead to clinical action, the information must integrate into clinical care.

• To promote sustainability, there should be periodic support interactions after formal implementation.

Background and Significance

While illness scoring systems have been in use since the introduction of Acute Physiology and Chronic Health Evaluation (APACHE) by Knaus and coworkers in the early 1980s, 1 new hospital information technology allows the real-time presentation of risk estimates for individual patients. 2 Many predictive analytics tools can provide a snapshot of the patient's clinical state by integrating multiple pieces of information. 3 4 Moreover, some potentially catastrophic illnesses have prodromal signatures that can be detected by algorithms analyzing continuous cardiorespiratory monitoring. 5 6 7 This approach, called predictive analytics monitoring, does not rely on clinician-initiated data and has a true predictive quality 8 in that a risk estimate based on these measures might rise in a patient with no overt signs or symptoms of illness.

For the bedside clinician, predictive analytics monitoring presents a new kind of information and a new paradigm of care. Its continuous nature can lead to very early diagnosis and brings the opportunity to be proactive—to take steps toward diagnosis and treatment prior to severe symptoms—rather than the conventional reactive stance. Thus, we distinguish proactive measures from preventive ones like vaccination or cancer prevention. Implementing predictive analytics monitoring, especially in a busy, complex, clinical environment, 9 can prove challenging, especially with bedside clinicians whose hard-earned wisdom may not be easily modified. 10 New strategies are required.

In 2020, we implemented predictive analytics monitoring in five intensive care units (ICUs) and nine non-ICU settings at an academic quaternary care hospital and a regional hospital, both on the east coast. We have engaged in more than 100 sessions and meetings with clinicians dedicated to the education and implementation of this system.

This case study describes our experiences and lessons learned during implementation in each of those units. We propose six principles that we believe to be generally useful.

The Predictive Analytics Monitor

We implemented a continuous predictive analytics monitoring system ( Fig. 1 ; Prediction Assistant, CoMET-inside, Premier, Inc., Charlotte, North Carolina, United States) that is representative in that it uses vital signs and laboratory test results from the electronic health record (EHR), and its basis is regression models trained on clinical event data. 3 11 12 Additionally, it incorporates continuous cardiorespiratory monitoring data that is analyzed mathematically for illness signatures. For each patient in a clinical care unit, the CoMET monitor assigns a comet-like icon and displays the fold-increase in risk from 1 to 5 of a future critical event. The risk is displayed visually, such that those patients at higher risk have larger and darker comets. The plot is displayed on a centralized wall monitor, so that the interprofessional team (physicians, nurses, respiratory therapists, and other support staff) can easily see each patient's risk.

Fig. 1.

The predictive monitor. ( A ) CoMET icons (left) become more prominent and darker orange as the patient's risk on either axis increases. A leaderboard (right side of display) rank orders patients within the unit according to relative risk. The head of the CoMET represents the current scores (x and y), and the tail contains three hours of prior data. The patients in beds 93, 94, and 97 have the highest risks within the unit, as evidenced by the size, color and position on the axes and the leaderboard. In the sample display, the cardiovascular event (x-axis) on which the model was trained was hemorrhage leading to transfusion of three units of packed red blood cells; the respiratory event (y-axis) was emergency intubation documented by a note from an attending physician. The predictive models are optimized to specific venues and target outcomes, differing from a standard “one-size-fits-all” approach). 21 ( B ) Bed 93 is selected from the Leaderboard. The bottom right graph displays the patient's CoMET scores as a function of time (24, 48, or 72 hours). ( C ) An increase in the respiratory instability score (green line) over a 4-hour period led to further evaluation and treatment (data were de-identified). (©2021 AMP3D, Charlottesville, Virginia, United States; With permission, all rights reserved).

Principles for Implementing Predictive Analytics Monitoring

These principles derive from our experience, including challenges we faced or anticipated and feedback from stakeholders in each unit at both hospitals. The Table 1 lists the principles and challenges that led to their development and examples of success.

Table 1. Principles for implementation of AI-based predictive analytics monitoring, associated challenges, and examples of success.

Principle	Challenge	Success example
1. Trust: understandable science	Clinicians questioned how the algorithm works to compute the score. “What are you using to get that score?” Clinicians questioned what additional information the predictive analytic score provides. “I am not sure why we need it. I can see the vitals and look up the labs myself.”	6 months after implementation, a clinical nurse specialist presented a case to her peers to demonstrate opportunity for improvement in time to escalation of care. She showed that the predictive analytic score had been elevated for >6 hours before the patient deteriorated and was diagnosed with sepsis. During the presentation, she showed an understanding of the algorithm, pointing out the subtle changes in the patient's clinical data that contributed to the increase in the calculated risk score prior to obvious signs of deterioration. 3 months after implementation, a surgeon reported on a patient that he was treating in hospital with a cocktail of antibiotics for new bacteremia several months after a complicated surgery. Even on treatment, the risk scores had begun to increase further over a 12-hour period, indicating a high risk for sepsis. The surgeon reviewed the laboratories and vital signs, noting a slight increase in the white blood cell count. He consulted the ID team and susceptibility testing showed bacteria resistance to 2 of the 3 antibiotics. The regimen was changed, and the patient recovered. The surgeon wrote “Your data predicted a problem,” “Very powerful tool!”
2. Uptake: minimal impact	Clinicians are busy and have competing unit and system-wide priorities. “How much extra work will this be? It's like, with COVID, they are already throwing new initiatives at us left and right.”	One nurse said that it is hard to miss the high risk patients on the big monitor, but that she also always looks at the scores in the flow sheet when validating her patients' vital signs. “We look at the big monitor to see which way the score is headed, and we look at the scores when we use the flow sheet.” One physician noted “I just pull up the synopsis view in the (EHR) and I can see the (predictive analytic) scores right below the vitals.” In several units, nursing leadership established the policy that with any MD notification for change in status (regardless of whether the scores drove the notification), the nurses would write the scores in their note. The score was incorporated into the existing workflow. In a cardiac ICU, the CNS noted, “We just added a checkbox to our rounding checklist to say, “Were the CoMET scores reviewed for this patient?'”
3. Buy-in: engagement and empowerment of clinicians at all levels	Nurses were concerned that the physicians wouldn't be receptive to notification of an increased score. Clinicians were skeptical as to how discussing the scores would affect care.	In an ICU at the academic hospital, the CNS noted, “When we do rounds the senior nurses expect the bedside nurses to report on the score trends for their patient along with other things like drip titration and readiness for extubation assessment.” She reports that the physicians now expect to hear this information and about unexpected elevations in scores. In an intermediate care unit, a nurse reported to us that she had noted “worrisome” changes in the character of the patient's breathing though there were no gross vital sign changes. Because the respiratory risk score had increased over the prior 4 hours to a five-fold risk of a critical respiratory event ( Fig. 1C ), the nurse reported having more confidence in calling the provider with this objective data. The physician examined the patient. A chest X-ray showed unsuspected pneumonia and an antibiotic was started.
4. Relevance: tailored education	Clinicians questioned the relevance of event prediction if the event never occurred. “It said the patient was at high risk for intubation, and he did have trouble, but we didn't end up intubating.”	In our earliest implementation in 2015, the scores were displayed on the graph as risks for specific events, such as intubation and hemorrhage. But it was confusing when the event itself didn't occur. With clinician input, we revised the education to include the use of the risk score as an index of clinical instability as well as event prediction. Ultimately, the axes titles were changed to reflect that new view. Clinician feedback for this change was very positive across disciplines. Nurses found the in-the-moment information useful. A charge nurse said, “It makes more sense now. I can tell the nurse that the patient has more respiratory instability and we figure out if there is an issue. Like in this one patient with a rising CoMET, we went in and realized there was a big mucous plug. We had a heads up and could take care of it ourselves before the patient got into trouble.” One surgeon recently said that when a patient's respiratory risk score increases unexpectedly, indicating risk for respiratory failure, “The team knows we need to examine that patient. We may draw some (arterial blood gases) or get a chest X-ray if it makes sense.” He remarked that, “Picking things up early gives us a chance to try and manage them with high-flow O 2 or CPAP and not have to intubate.”
5. Actionability: integration into care	Clinicians were unsure of the appropriate next step when the score increased. “My patient's cardiovascular risk is at 4, what am I do with that?”	Each of the units at both hospitals adopted or modified the suggested protocol in which bedside nurses were to notify the next level clinician for a prespecified increase in the risk score. At the academic hospital, for example, the ward nurses notify the resident physicians of an unexpected rise by 2 units in a patient's risk score. The residents are then to evaluate the patient to determine if further evaluation or treatment is appropriate. An ICU nurse realized her post-op patient “was more unstable” because the cardiovascular risk score had continued to rise despite administration of blood products. Seeing the score, she reviewed the vital signs and notified the team, according to protocol, that the risk score was rising, and the blood pressure had been subtly trending downward. Labs were drawn and additional blood products transfused. Ultimately, the patient was taken back to OR for ligation of an artery.
6. Sustainability: follow-up interactions	Clinicians need to continue to use the innovation after the initial implementation	Several months after implementation, a physician who cares for complicated cardiac patients had questions about the risk score in 2 of his patients. We worked through those patients with him, demonstrating what risk predictors had accounted for the change in scores. A nurse told us that when he came onto the unit recently (11 months after implementation), she told him that one of his patient's' scores had risen. He told her that he knew already because he'd been “checking on his phone” and that was why he was there. 12 months after Go-Live, the regional hospital expanded use of the predictive analytic to two additional units: an intermediate care unit, and a medical ward. Additionally, physician interest moved beyond the initial hospitalist and intensivist targets. The surgeons requested education sessions to understand the analytics potential for their patient management. Now that the regional hospital has begun to integrate scores into their EHR, they are looking at ways to flag rising scores within the medical record to increase clinicians' awareness. Used for 6 years in the surgical ICU at the academic hospital, the predictive analytic was also woven into the unit's broader escalation process map two years ago by the interprofessional leadership team. Scores that increase by different specified amounts, or that fail to decrease after treatments, lead to specific care escalations to be taken either by nursing, respiratory therapists, or medical providers.

Abbreviations: AI, artificial intelligence; CNS, central nervous system; COVID, novel coronavirus disease 2019; CPAP, continuous positive airway pressure; EHR, electronic health record; ICU, intensive care unit; OR, operating room.

Framework

This work extends that of Keim-Malpass et al 11 and Kitzmiller and coworkers 13 who used qualitative research methods and diffusion of innovation theory to examine the earliest implementations of predictive analytics monitoring at the University of Virginia Hospital. The principles are consistent with Proctor and coworkers who suggested that acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability are implementation outcomes and necessary preconditions for desired clinical change from the tool itself. 14

1. To promote trust, the science must be understandable

   Clinicians can be skeptical of new clinical tools, especially those that seem to be black boxes. 10 11 13 15 To accept new tools like predictive analytics, particularly those that derive information from the physiologic waveforms, clinicians need to understand what is happening inside. 11 13 16 17

   We presented the published evidence base 6 18 19 20 21 22 to provide transparency into the algorithms' underpinnings and to emphasize the strengths of the scientific foundation. 16 We focused other sessions on current patients to give clinicians a more detailed examination of how data elements interacted within the algorithm to produce a risk score. Correlating the change in the risk score in real time with the patient's clinical course helped develop confidence in the tool. We used case-based learning to demonstrate the hours of early warning the predictive analytic monitoring provided before an actual patient event.

2. To enhance uptake, the workflow should not be impacted greatly

   An essential feature of a predictive analytics monitoring tool, like other clinical decision support tools, is minimal disruption in workflow. 11 13 23 24 We note that this particular predictive analytics tool uses only electronically available data; thus, clinicians did not need to enter any additional documentation.

   Prior to implementation, we assessed existing unit workflows, including interprofessional methods of communication, clinical escalation or event management processes, and documentation practices. We customized an integration strategy that was complementary to and meshed with existing practices. 25 We identified opportunities in which discussion of predictive scores would fit seamlessly into established inter- and intraprofessional interactions. Large wall monitors made it easy for clinicians to view at a glance, without opening the EHR, their patient's changing clinical stability and risk. Clinicians could interact with the display on any device to select individual patients for a deeper review of the scores' trajectory. Because the tool integrates with the vital sign flow sheet in the EHR at the academic hospital, nurses and respiratory therapists could review the scores as a part of clinical documentation used in existing workflows.

3. To maximize buy-in, engagement at all levels is important

   The care of a patient proceeds at multiple clinical levels, and there is complex but essential communication among team members. 26 27 28 29 30 Each clinician plays a knowing role in patient surveillance, and describing relevance to their clinical practice facilitated adoption of the tool with multiple stakeholders. 10 Thus, the target audience for use of the tool included physicians, nurses, and other support disciplines, such as respiratory therapy. Educating and supporting the interprofessional teams' efforts toward incorporating the predictive tool into standard care improved institutionalization.

   We found it imperative that a clinician in a leadership role, such as a chief nursing officer or medical director, is engaged at the outset to shepherd the tool's implementation and champion its use. 31 This leader's tasks included setting the expectation for the use of predictive analytics monitoring in clinical care, mobilizing resources, and identifying boots-on-the-ground leadership. We empowered cross-disciplinary leaders at the level of individual units to lead by example, utilizing the tool in practice and encouraging their peers to do the same.

   Buy-in from physicians and advanced practice providers was important for optimal interdisciplinary adoption and was inspired by strong leadership support. 32 33 We engaged and educated the interprofessional team in sessions together, whenever possible, to galvanize collaboration. 34 35 We encouraged the clinicians to use the predictive analytic tool in their individual practice and, collectively, as a means toward efficient communication about a patient's risk before decompensation. We promoted team discussions to clarify practice goals across disciplines and to establish communication standards, such as when a rising risk should be reported. 36 We supported nursing leadership in setting standards for nursing communication and documentation of the scores.

   In this way, the predictive analytics monitoring score became a framework or lexicon for the necessary communication among the levels of caregivers about a patient's status. 37 38 Its use can enhance teamwork 39 by empowering everyone at the bedside to identify patients at rising risk before decompensation occurs. An example is shown in the Fig. 1 and Table 1 .

4. To ensure relevance, the education must be tailored to the clinical role and hospital culture

   Since there was a standard message to the education—early warning can lead to proactive care—everyone received the same overview and introduction to the display and its relevance to the patient's future risk. Thereafter, the education approach focused on assisting clinicians to interpret the predictive analytics within the context of their specific role 40 and the patient population of the unit. For physicians and advanced practice providers, we emphasized the scientific foundations along with the practical use of the tool in triaging and managing patients. For nurses and respiratory therapists, we emphasized hands-on sessions to demonstrate its relevance to their care provision.

   Institutional and unit hierarchies, cultures, and norms informed the depth, frequency, and focus of our educational interactions with clinicians. 10 41 We noted a difference in the level, type, and timing of education requested at the academic center and the regional hospital. In the former, emphasis was placed on the scientific foundations, the published literature, and real-time case review in cross-disciplinary sessions. For both, we used recurring interactions with the material presented in a variety of ways—lectures, demonstrations, and online learning—to amplify comprehension. Regardless of hospital culture or level of a clinician's practice, education that centered on case-based reviews was very effective. 42

5. To lead to clinical action, the information must integrate into clinical care

   Successful implementation must inform on how to translate an early warning of deterioration into preemptive action. Earlier implementations of predictive analytics monitoring showed improved outcomes simply by presenting risk scores to clinicians with no stipulation as to how they should respond. 5 12 43 44 Yet, in early qualitative studies, 11 13 45 clinicians recognized that a prescribed pathway for response to rising scores had the potential to augment broader use, increasing integration within the care environment.

   We taught that patient assessment based on a rising risk trend is the first proactive step. We developed an interprofessional framework that codified methods to translate the risk calculation into action, standardizing its use in clinical workflows across disciplines. We collaborated with leaders of the interprofessional teams to develop response protocols that integrated predictive monitoring into clinical decision-making. We found it important to tailor the protocol to a unit's acuity level. In one ICU, a prespecified increase in the score led to initiation of previously ordered “as needed” interventions or a page to the provider on call. In contrast, an acute care floor used that same increase to start a rapid response team protocol.

   Differences in hospital culture and clinical care teams informed the mechanisms used to integrate predictive analytics monitoring into clinical care. At the academic institution, the interprofessional clinical care team members routinely shared information during rounds and collaborated on goals for care. It was natural, then, for the team to review a particular patient's risk score trajectory together and for everyone to contribute observations about the patient's stability. The provider and nursing teams at the regional hospital, on the other hand, did not routinely round together but usually communicated when a specific clinical need arose. There, the provider and rapid response team leadership agreed on a level of increase in the score that would trigger nursing to notify them both.

   Integrating the predictive analytic into care, including documentation of the risk scores within the EHR and discussion of the patient's risk trends during rounds and at handoff of care, gave more opportunity for clinicians to evaluate a patient's changing clinical risk.

6. To promote sustainability, there should be periodic support interactions after formal implementation

   Repeated, spaced exposures to material enhance uptake and consolidate learning. 46 47 We found that when implementing Artificial Intelligence-based predictive tools, the key inflection point for long-term sustainable adoption occurred in the months after the initial implementation. It was often initially difficult for clinicians at all levels to credit the tool's predictive capability and the concept of proactive evaluation based on the estimated risk of a future event. Thus, periodic interactions were part of our postimplementation strategy and included continuing education with actual patient case review, reinforcement of concepts, training new users, and refinement of integration into the workflows. 48 Our schedule included bimonthly sessions in the first month, monthly for at least 2 months and then quarterly. During support sessions at each institution, clinicians provided patient cases of deterioration and together we identified missed opportunities for earlier evaluation and treatment had the clinicians capitalized on the early warning. We found that their understanding of the tool's capability was augmented by examining such instances. We assisted them in developing these cases as additional educational materials to identify and address knowledge gaps among staff.

   Likewise, clinicians' ideas about how to integrate the analytics into standard care practices increased with use. Our recommendations for placement of large monitors that can be easily seen and integration into the EHR were implemented differently at each institution at the outset. At the regional hospital, smaller monitors were used and EHR integration was postponed due to an expected EHR upgrade. The academic hospital installed a single monitor in several larger units with several wings. Over the ensuing months, clinical champions at each hospital expressed interest in changing their implementation footprint to more closely follow the initial design. Accordingly, at the academic hospital, a large wall monitor was installed in each wing of those units and, at the regional hospital, efforts were undertaken to install larger monitors and to effect EHR integration of the scores.

   In the first year after implementation, the clinicians at each hospital expanded on our recommendations. At the regional hospital, clinical informaticists began examining ways to include the predictive analytic scores into other clinical dashboards. The ICU and ward champions at the academic institution developed educational programs aimed specifically at rotating medical trainees and placed monitors in their workrooms to increase awareness. Such efforts to expand the tool's integration beyond the initial implementation targets contributed to sustainability of the innovation.

Conclusion

We conclude that the ingredients of successful implementation of predictive analytics monitoring include understandable science, widespread buy-in, tailored education across disciplines, minimal workflow disruption, integration into standard clinical care, and continuing education in the months following implementation. Each relates to several of the taxonomic elements of implementation success proposed by Proctor and coworkers. 14 Doubtless, as the architects of this field continue to develop new applications of Artificial Intelligence to clinical care, additional considerations will arise and can build on these principles.

Clinical Relevance Statement

The implementation of clinical decision support using big data and Artificial Intelligence (AI) is becoming increasingly common. We offer a real-world case study in which we addressed the challenges inherent in implementing AI-based monitoring systems.

Multiple Choice Questions

1. Which of the following is a principle for real-world implementation of predictive analytics for use at the bedside?

   1. to ensure accuracy of the analytic, information must be typed in by the clinician.

   2. to ensure universal understanding, all clinicians should receive the same education.

   3. to maximize buy-in, the education should be focused toward the medical provider.

   4. to promote trust, the science must be understandable.

   Correct Answer: The correct answer is option d. The other five principles are as follows: (1) to enhance uptake, the workflow should not be impacted greatly; (2) to maximize buy-in, engagement at all levels is important; (3) to ensure relevance, the education must be tailored to the clinical role and hospital culture; (4) to lead to clinical action, the information must integrate into clinical care; (5) to promote sustainability, there should be periodic and support interactions after formal implementation.

2. Repeated education support interactions following implementation of predictive analytic monitoring are important because:

   1. clinicians will need reinforcement about which laboratory values and vital signs they need to enter.

   2. clinicians' understanding of the potential for proactive care increases with use of the predictive tool.

   3. it is most efficient to engage physicians after all other clinical staff has been educated and is using the tool.

   4. it is important to ensure that the tool continues to be used exactly as it was at the time of initial implementation.

   Correct Answer: The correct answer is option b. It is a paradigm shift for clinicians to contemplate a patient's risk for imminent critical events often in the absence of obvious signs and symptoms, so understanding and trust are augmented by reviewing current patient situations and, importantly, recent past critical events using the documented early warning provided by the computed analytics. Clinicians do not need to enter any additional data in this analytic. Engaging physicians at the outset in planning the collaborative use of the tool is the goal. We found that sustainability (continued use of the tool) is enhanced as clinicians find new ways to incorporate the risk prediction score into other aspects of their clinical workflow.