Evaluation of User-Interface Alert Displays for Clinical Decision Support Systems for Sepsis

Abstract

Background

Hospitals are increasingly turning to clinical decision support systems for sepsis, a life-threatening illness, to provide patient-specific assessments and recommendations to aid in evidence-based clinical decision-making. Lack of guidelines on how to present alerts has impeded optimization of alerts, specifically, effective ways to differentiate alerts while highlighting important pieces of information to create a universal standard for health care providers.

Objective

To gain insight into clinical decision support systems–based alerts, specifically targeting nursing interventions for sepsis, with a focus on behaviors associated with and perceptions of alerts, as well as visual preferences.

Methods

An interactive survey to display a novel user interface for clinical decision support systems for sepsis was developed and then administered to members of the nursing staff.

Results

A total of 43 nurses participated in 2 interactive survey sessions. Participants preferred alerts that were based on an established treatment protocol, were presented in a pop-up format, and addressed the patient’s clinical condition rather than regulatory guidelines.

Conclusions

The results can be used in future research to optimize electronic medical record alerting and clinical practice workflow to support the efficient, effective, and timely delivery of high-quality care to patients with sepsis. The research also may advance the knowledge base of what information health care providers want and need to improve the health and safety of their patients.

Sepsis, a life-threatening disease state characterized by organ dysfunction due to a dysregulated host response to infection,¹ affects more than 1 million persons in the United States every year.² Sepsis represents a rapidly growing problem in terms of the number of patients who have the condition, the complexity of the cases, and the clinical outcomes, including increased risk for complications and death, longer courses of treatment, and long-term cognitive impairment and functional disability.³ Sepsis is a marked economic problem; it accounted for more than $20 billion of total US hospital costs in 2011.⁴ Sepsis is present or develops in approximately 1 of every 23 hospital admissions⁵ and accounts for nearly half of all hospital deaths.⁶ The prevalence and stress of sepsis underscore the importance and potential impact of clinical decision support (CDS) in the clinical environment.

Awareness of sepsis is low; in many patients, sepsis is not diagnosed at an early stage when aggressive treatment might reverse the course of infection.¹ Early recognition and response can temper the inflammatory response and improve patient outcomes,⁷ but failure to initiate appropriate therapy remains strongly correlated with increased morbidity and mortality.⁸ For every 1-hour delay in antibiotic treatment of a patient with septic shock, survival decreases incrementally. A central unresolved challenge is timely and consistent implementation of diagnostic and therapeutic sepsis guidelines.⁹ Health care organizations are increasingly turning to CDS systems for sepsis, which provide clinicians with patient-specific assessments and recommendations to promote recognition and aid in evidence-based clinical decision-making.^10,11

Currently, clinicians must identify sepsis through a process of intense searching in a patient’s electronic medical record (EMR), clinical judgment, and assessment of the patient, searching for information on a patient’s vital signs and clinical laboratory values to determine the presence of new organ-system dysfunction. Screening tools, including paper-based algorithms, protocols, and computerized sepsis CDS systems, have been used extensively and are relatively effective in the intensive care unit,^11–13 the emergency department,^14,15 and medical and surgical units.^16–18 Evidence-based CDS solutions currently exist; these are moderately successful in predictive accuracy, improved communication, and appropriate therapeutic and diagnostic interventions.^{11,13,18–23} However, these tools emphasize venue-centric, localized definitions of sepsis, a situation that, in addition to constantly changing definitions of and guidelines for treatment of sepsis, affects the tools’ diagnostic usefulness and broad adoption.²⁴ Because of the challenges of health information technology, current sepsis alerting systems include variability and availability of reliable electronic data, making monitoring of these alerts tedious and time-consuming.^22,23

Standardization in acquiring and integrating data into CDS alert interfaces is lacking, and thus implementing CDS systems effectively without inducing alert fatigue remains a challenge.^25–28 Little consensus exists on how alerts and warnings should be generated and displayed to the user to optimize response.²⁹ Features important in improving CDS effectiveness include “support presented at the time of the decision, computer-based support, support that included a recommendation rather than just an assessment, and automatic provision of decision support as part of workflow.”³⁰ However, implementation of CDS systems has generated unintended consequences. including the elimination or shifting of human roles, difficulty in keeping content current, and inappropriate content.³¹ Lack of guidelines on how to present alerts to health care providers has impeded optimization of alerts, specifically, the most effective ways to differentiate alerts while highlighting important pieces of information without adding noise, to create a universal standard.²⁹

For efficient use of CDS to enhance the reliability of data, effective user interfaces must be designed properly for medical systems in line with human factors engineering techniques. From the perspective of human factors engineering, our objective in this study was to determine the best way to provide a sepsis alert to improve decision-making in the dynamic, fragmented health care work environment,³² to design and formally evaluate user interfaces used to display CDS output for sepsis, and to discover and evaluate how alternative human-computer interaction models might affect nurses’ perception of CDS systems. We hypothesized that soliciting nurses’ perceptions of sepsis CDS systems, behaviors toward and perceptions of alerts, and visual preferences for alerts would yield insight into CDS-based alerts specifically targeting nursing interventions for sepsis to improve current and future design and implementation of CDS systems for sepsis.

Methods

Study Setting

Christiana Care Health System (Christiana Care), headquartered in Wilmington, Delaware, is one of the largest health care providers in the United States; it ranks 22nd in the nation for hospital admissions (53 621 annually). A not-for-profit, nonsectarian health system, Christiana Care includes 2 major teaching hospitals with more than 1100 patient beds. The system is home to Delaware’s only level I trauma center, the only center to offer this level of care between Philadelphia and Baltimore, and features a level 3 neonatal intensive care unit, the only hospital in the state with delivery services that offers this level of care for newborns. This study was approved by the Christiana Care institutional review board.

Study Participants

We used members of existing nursing councils as participants: the Nursing Quality and Safety Council and the Nursing Technology Council. The Nursing Quality and Safety Council is a system-wide shared governance council that provides planned, systematic, and collaborative approaches to oversee and direct quality and safety related to nursing and interdisciplinary care of patients. The council’s scope encompasses quality and safety initiatives throughout the department of nursing. The mission of the Nursing Technology Council is to promote nursing excellence with new computer technologies by evaluating and providing feedback on new applications and to assist with design, integration, education, and implementation of electronic data in order to provide exceptional service and create a culture of safety.

Survey Design and Administration

We developed a survey based on the display of novel user interfaces for sepsis alerts of CDS systems specific to staff nurses. Survey questions were not validated because of the study design, which called for a highly interactive survey with questions to boost engagement with participants. Participants were asked to answer questions in 5 domains: general perception of CDS systems; risk parameters (prioritizing information provided in an alert); alert word taxonomy (which signal words best indicated the highest level of alert severity); vital signs, context, and recommendations (which vital signs are the most valuable elements of an alert for predicting severe sepsis); and evaluation of novel interface designs (information design display and location of information). A list of sepsis-alert criteria was used to obtain feedback for displays designed to support clinical decision-making on sepsis by clinicians who administer diagnostic and therapeutic interventions. This information was developed by using elements included in existing sepsis CDS systems¹⁶ and in our health system. The survey was administered in 2 parts. A pilot (paper survey) was used to collect feedback and make adjustments, and subsequent electronic surveys were administered in person to engage participants and provide the ability to answer questions and provide education on the design of novel user interfaces. For example, participants were educated on recently mandated Centers for Medicare and Medicaid requirements that include financial punishments if specific actions within specified time frames are not met.^33–35 Of note, the survey was administered in the fall of 2015, before the release of new consensus definitions of sepsis.

Study data were collected and managed by using REDCap electronic data capture tools hosted at Christiana Care.³⁶ REDCap (research electronic data capture) is a secure, web-based application designed to support data acquisition for research studies, providing an intuitive interface for validated data entry, audit trails for tracking data manipulation and export procedures, automated export procedures for seamless data downloads to common statistical packages, and procedures for importing data from external sources.

Statistical Analysis

We used Stata/IC, version 11.0, software (StataCorp) to perform statistical analyses. Standard descriptive measures (frequencies, means, medians, and standard deviations) were computed for survey responses. Qualitative data on participants’ feedback were analyzed by the research team to identify trends in nurses’ perceptions.

Results

Demographics

A total of 43 nurses participated in 2 interactive survey sessions (100% of Nursing Technology and Quality and Patient Safety council members). The nurses were a diverse group of staff members in terms of education, experience, and service (Table 1).

Table 1.

Demographic characteristics of participants (N = 43)

Characteristics	Valuesa
Sex
Female	33 (77)
Male	6 (14)
Declined to answer	4 (9)
Age, range (mean), y	23–67 (41)
Education
Associate degree of nursing	6 (14)
Bachelor’s in nursing	29 (67)
Advanced degree	7 (16)
Other	1 (2)
Years of experience (in general)
<1	2 (5)
1–2	2 (5)
3–5	9 (21)
6–10	9 (21)
11–15	6 (14)
16–20	3 (7)
>20	12 (28)
Years of experience (on current unit)
<1	6 (14)
1–2	6 (14)
3–5	10 (23)
6–10	8 (19)
11–15	10 (23)
16–20	1 (2)
>20	2 (5)
Unit type
Medical	16 (37)
Surgical	8 (19)
Intensive care	4 (9)
Emergency department	3 (7)
Pediatrics	3 (7)
Operating room/postanesthesia care	4 (9)
Administrative	5 (12)

^aUnless indicated otherwise, all values are No. (%). Because of rounding, not all percentages total 100.

Perception of CDS Systems

Nurses were asked to rate current CDS systems, given their experience with the CDS system currently offered within our clinical information system, including the EMR (Table 2). This part of the survey included general attitudes about alerts, factors that influence behaviors associated with alerts, and visual preferences.

Table 2.

Nurses’ (N = 43) perception of clinical decision support (CDS) systems^a

Perception	Never, No. (%)	Rarely, No. (%)	Sometimes, No. (%)	Most of the time, No. (%)	Always, No. (%)	Declined to answer, No. (%)
CDS helps me take better care of my patients.	0	0	14 (32)	16 (37)	9 (21)	4 (9)
CDS reminds me of something I had forgotten about.	1 (2)	3 (7)	19 (44)	10 (23)	6 (14)	4 (9)
I feel relieved when I get an alert.	2 (5)	5 (12)	19 (44)	8 (19)	5 (12)	4 (9)
I feel empowered when I get an alert.	2 (5)	9 (21)	15 (35)	9 (21)	4 (9)	4 (9)
I feel grateful when I get an alert.	0	3 (7)	20 (47)	10 (23)	6 (14)	4 (9)
I feel annoyed when I get an alert.	4 (9)	16 (37)	16 (37)	2 (5)	1(2)	4 (9)
I look up a patient’s information as a result of receiving an alert.	0	2 (5)	11 (26)	11 (26)	13 (30)	6 (14)
I trust the information provided in an alert.	1 (2)	0	7 (16)	23 (53)	8 (19)	4 (9)

^aLikert scales required participants to select their level of agreement (never, rarely, sometimes, most of the time, always) with the provided statement.

Risk Parameters

Nurses were asked to prioritize information provided in an alert by ranking 4 risk parameters for why an alert should occur (be sounded). Their aggregate ranked order from most important to least important was severity of condition, probability of harm occurring, complexity (presence of factors that increase patient complexity), and detectability (ease of recognizing a worsening condition). Consensus was reached for this ranked order.

Alert Word Taxonomy

Signal words are used to indicate severity. In order to assess language, participants were asked which signal words best indicated the highest level of alert severity. In ranked order from most severe to least severe, summarized responses were critical, danger, high risk, warning, caution, and notice.

Vital Signs, Context, Recommendations

Nurses were asked which vital signs were the most valuable elements of an alert for predicting severe sepsis. The elements chosen as the most valuable were temperature, heart rate, blood pressure, white blood cell count, respiratory rate, band count, and lactate level (Table 3).

Table 3.

Nurses’ (N = 43) perception of most valuable elements of sepsis alerts

Vital sign	No. (%)
Temperature	34 (79)
Heart rate	33 (77)
Blood pressure	32 (74)
White blood cell count	31 (72)
Lactate	27 (63)
Respiratory rate	22 (51)
Band count	17 (40)
Oxygen saturation by pulse oximetry	14 (33)
New oxygen requirement	14 (33)
Platelet count	6 (14)
Creatinine level	5 (12)
Bilirubin level	4 (9)

Among 39 nurses, 34 (87%) would like to have the interventions provided in the alert based on an established treatment protocol (eg, administer antibiotics, obtain samples blood cultures). This percentage increased once nurses were educated on Centers for Medicare & Medicaid Services requirements to meet guidelines at specific times (31 of 34 nurses; 91%). If technologically possible, 33 of 36 nurses (92%) would like these measures automated if a patient met CMS criteria for severe sepsis. Although nurses viewed CMS requirements as valuable, they thought they would be more motivated if an alert addressed the patient’s clinical condition in order to provide safe and effective treatment to meet Christiana Care priorities.

Alert Evaluation

Nurses were provided with different display options for severe sepsis alerts. Among 37 participants, 31 (84%) preferred pop-up alerts to a handoff page or a summary page (static pages in the EMR), discussion in interdisciplinary rounds, or nursing flow sheets. Nurses were presented with nurse-specific user interfaces for sepsis alerts. The first interface was the currently used St. John Sepsis Advisory (Cerner). The interface includes basic patient information, a sepsis advisory indicating the type of vital signs of concern and physician discussion points, and values of criteria for systemic inflammatory response syndrome.

The second interface was a mock alert that mimicked the current Cerner medication alerts in terms of design and function. Cerner discern alerts are designed by using a standard framework. This alert provides additional details, including the definition of septic shock, abnormal values (with reference range values indicated), and recommended actions.

The third interface was a second mock alert design with a similar Cerner framework. Unlike the second interface, this alert provides information specific to systemic inflammation and organ failure individually, again providing abnormal values with reference range values providing context. Nurses were asked to compare the alerts and evaluate each for 6 domains: usefulness, the ability to facilitate detecting critical information, the ability to accomplish a task, sufficiency of information for making a prescribing decision, user friendliness, and the ease with which you could arrive at a decision (Table 4). The evaluation included the use of a slider bar, assigning a value of 0 to 100.

Table 4.

Interface evaluation^a

	Interface 1	Interface 2	Interface 3
Usefulness of the displayed alert	34 (23.50, 50)	15 (5, 32)	33.50 (18.25, 50)
Ability of the interface to facilitate detecting critical information	31 (18.50, 50)	21 (6, 31)	31 (15, 44)
Ability to accomplish a task	40 (25.50, 50)	21 (6.50, 33)	39.50 (24.25, 50)
Sufficiency of information for making a prescribing decision	34 (27, 50)	19 (7, 32)	43 (26, 59.50)
User friendliness of the interface	43 (22.50, 62.50)	15.50 (5, 33.25)	36 (19.50, 46.50)
Ease with which you could arrive at a decision	36 (24.50, 50)	19 (6.50, 30.50)	36 (19, 50)

^aScale 0–100, with 0 = poor, 100 = excellent. All values are median (interquartile range).

Discussion

The nursing councils provided a representative sample of nursing staff at Christiana Care and a somewhat representative sample of nurses in the United States (mean age 50 years, 89% female, 61% with a bachelor of science in nursing).³⁷ Although the accuracy and implementation of CDS systems have been evaluated, little evidence is available on how nurses use or experience a CDS system. Understanding use of a CDS system and nurses’ perceptions of CDS systems is critical in understanding the effect of these systems on clinical care for sepsis. In our study, participants did not have strong positive or negative impressions related to CDS systems used in clinical practice. In evaluations of general perception, most participants selected sometimes (the scale’s midrange point) for every question. This finding indicates that nurses are neither satisfied nor dissatisfied with current CDS tools.

Evaluations of risk parameters and alert word taxonomy provided unique insights into nurses’ needs and preferences. Responses for risk parameters were not surprising; severity and probability are traditionally the dimensions of a risk assessment. The alert word taxonomy provided surprising results; the word critical was selected as perceived highest severity. Traditional signal words include danger, warning, and caution. Although not traditionally considered a signal word, critical may have been selected as highest severity because it is familiar in nursing terminology (eg, critical care).

In the assessment of perceptions and behaviors in response to a sepsis-specific CDS system, participants indicated that 2 diagnostic tests were highly valuable: lactate level and white blood cell bandemia count, both identified as predictive markers for poor outcome in patients with sepsis. Understanding what health care providers think are key indicators, and thus what they might expect in a predictive model, is critical in developing trigger tools to detect indications of sepsis and CDS systems with strong clinical usefulness. Predictive modeling encompasses a variety of statistical techniques to analyze current and historical facts to make predictions about future or otherwise unknown events. Predictive models integrate methods that are part of the health system’s engineering (eg, optimization) and derive methods from related professions (eg, computer science) to analyze and predict patterns and outcomes. Although predictive models for sepsis-related outcomes can be the basis for designing alerts specific to nursing interventions, nurses’ clinical intuition and knowledge offer unique insights based on years of experience and training.

We strongly suggest that CDS systems provide a recommendation, not solely an assessment. An effective CDS system must be relevant to persons who can act on the information, and the design of recommendations with a CDS system plays an essential role.^38–43 A CDS system that uses an active interaction model, such as generating clinical alerts or reminders based on clinicians’ data entry and interaction, is the most effective in improving clinical practice.^39,44–46 In our study, the nurse preference to receive recommendations in the alert was refreshing, indicating they would adopt suggested interventions. In alert motivation, the finding that priorities of the patient, unit, and hospital were placed ahead of federal mandates and accreditation, which are most often more influential to physicians who may experience consequences of not meeting guidelines, was not surprising. Nurses focus on what is perceived to be safe and effective treatment and on what is best for the patient and therefore are generally less worried about meeting regulatory requirements.

In the comparison of different alert user interfaces, participants responded in extremes. For example, some participants thought the designs were not useful at all, whereas others thought the designs were extremely useful. Participants responded similarly in all 6 domains. This result may reflect similarities in design, limitations in analysis due to minimal experience with the designs, or an indifference to the information. Participants indicated that their main priority when provided with a CDS system was to share the information provided by the system with physicians who were able to order both diagnostic and therapeutic interventions. Therefore, the participants’ responses may reflect that the alert is simply a tool to identify an at-risk patient in order to relay information to providers and that the specifics of the alert are not as critical to this conversation because, ultimately, the goal is to bring attention to a specific patient. The majority of participants indicated that alerts direct them to information in a patient’s chart, suggesting that they may prefer to view data on specific vital signs and laboratory values in the EMR rather than in an alert. Last, nurses preferred that a CDS system for sepsis be provided in a traditional pop-up alert. Prototypes and pilot alerts should be evaluated by using novel metrics for predicting inappropriate alerts and responses, building off the traditional framework for evaluating alerts.⁴⁷

The many studies done to evaluate the effect of CDS systems have indicated several benefits, including improved interprofessional communication, increased access to best-practice information, and more consistent quality of care.⁴⁰ However, fewer published studies^41–43 have been assessments of attitudes and perception toward clinical alerts and CDS systems. Feblowitz et al⁴⁴ found that among health care providers, alert acceptance was 38.1% (interquartile ratio, 14.8%–54.4%), indicating the mass complexities of alerts. Studies on alarm fatigue indicated that most nurses (~95%) considered that false alarms reduced trust in alerts, disrupted clinical care, and desensitized members of the nursing staff.⁴² Specifically, for alerting systems for sepsis, 39% of clinicians and 48% of nurses thought that sepsis-specific earning warning system alerts provided new information about the patient, and 33% of clinicians and 40% of nurses thought the alerts were helpful.⁴³ The goal of our research was to proactively evaluate design of CDS systems to ensure that future systems are properly integrated into nurses’ workflow and properly customized to address key clinical elements.

The issues underlying nurses’ support or rejection of CDS technology must be examined more carefully to better understand adoption of CDS systems and better design optimal alerting tools.⁴⁵ Our results begin to delineate how nurses could use CDS systems in clinical practice and the factors that influence use of the systems. Research^46–48 suggests that several factors, including nurses’ experience, workflow, and integration of EMRs, and organizational characteristics influence use and acceptance of CDS systems. The accuracy and usability of these CDS tools can markedly effect the sustainability of the use of CDS systems. Furthermore, because nurses represent the largest group of care providers in the health care system, acceptance by nurses is associated with the overall acceptance of CDS systems. Perhaps most important is the understanding that having a single alert design is not appropriate. Nursing staff require different information than do physicians and may prefer to have the information provided through different means. This concept of user-centered design is a central tenet of the TURF design model for health care information systems⁴⁹; a comprehensive alert system requires personalization of alert design and content for different groups of health care providers.

Limitations

Our research sample was limited to the members of 2 nursing councils at a single-center, introducing bias related to the commitment, understanding, and experience of the council members with the CDS system. The nursing councils included participants with an interest and perhaps strength in the use of technology, leading to a self-selection bias, and yet we found little consensus in the participants’ responses. Many of the survey questions provided limited options for selection, as opposed to open-ended questions, a situation that may have biased nurses’ response by guiding the participants to select answers. However, we think this bias was limited because of the interactive survey design and open discussion among participants.

Limited time and experience with novel user interfaces may reflect perceptions that would develop over time and with use. The interface designs were introduced in concept, and our results might have been different if the designs had been introduced into direct patient care and clinical workflow. Additionally, our results could have also been biased, because CDS models were introduced as a concept instead of introduced into direct patient care and clinical workflow. Despite these limitations, our findings suggest that more research is needed in the design and development of CDS systems for sepsis for nurses. Future usability testing will influence perception during simulated use of CDS systems, improving the understanding of the clinical usefulness these systems may provide.

Conclusions

Our results provide a basis for future research to optimize EMR alerts and clinical practice workflow to support efficient, effective, and timely delivery of high-quality care to patients with sepsis. When the EMR is not used to its full advantage, errors in timely diagnosis and care may occur for a complex, fast-moving condition such as sepsis. By providing information for timely patient-focused care and early intervention, improved CDS systems can help improve mortality rates, decrease organ failure, and reduce use of health care resources (eg, hospital length of stay, long-term care). We think our findings can form a basis for and advance our knowledge of what information providers want and need to improve the health and safety of patients.