The Nature and Variability of Automated Practice Alerts Derived from Electronic Health Records in a U.S. Nationwide Critical Care Research Network

Abstract

Rationale: The nature, variability, and extent of early warning clinical practice alerts derived from automated query of electronic health records (e-alerts) currently used in acute care settings for clinical care or research is unknown.

Objectives: To describe e-alerts in current use in acute care settings at medical centers participating in a nationwide critical care research network.

Methods: We surveyed investigators at 38 institutions involved in the National Institutes of Health–funded Clinical Trials Network for the Prevention and Early Treatment of Acute Lung Injury (PETAL) for quantitative and qualitative analysis.

Measurements and Main Results: Thirty sites completed the survey (79% response rate). All sites used electronic health record systems. Epic Systems was used at 56% of sites; the others used alternate commercially available vendors or homegrown systems. Respondents at 57% of sites represented in this survey used e-alerts. All but 1 of these 17 sites used an e-alert for early detection of sepsis-related syndromes, and 35% used an e-alert for pneumonia. E-alerts were triggered by abnormal laboratory values (37%), vital signs (37%), or radiology reports (15%) and were used about equally for clinical decision support and research. Only 59% of sites with e-alerts have evaluated them either for accuracy or for validity.

Conclusions: A majority of the research network sites participating in this survey use e-alerts for early notification of potential threats to hospitalized patients; however, there was significant variability in the nature of e-alerts between institutions. Use of one common electronic health record vendor at more than half of the participating sites suggests that it may be possible to standardize e-alerts across multiple sites in research networks, particularly among sites using the same medical record platform.

The rapid implementation of electronic health records (EHR) in the last decade has allowed the application of automated electronic alerts (e-alerts) to improve the processes of care by providing clinical decision support, facilitating recruitment in clinical trials, and improving the reporting of quality measurements (1–4). Outside of acute care settings (emergency medicine, trauma care, prehospital emergency care, acute care surgery, critical care, urgent care, and short-term inpatient stabilization) (5), e-alerts have been widely used (6, 7) and have improved enrollment in clinical trials (8).

In acute care settings, e-alert use has been limited to single-center reports (2, 9, 10). With increasing use of EHR (11) and frequent vital sign and laboratory monitoring in acute care settings (12), e-alerts can serve as an essential tool for clinical research and collaboration. Our aim in this study was to describe the use and variability of EHR-generated e-alerts in 38 hospital sites within a U.S.-based nationwide critical care research network to understand the feasibility of e-alert standardization in multicenter research.

Methods

Survey Development and Administration

We generated a survey questionnaire using a mixed methods approach (13) for the Clinical Trials Network for the Prevention and Early Treatment of Acute Lung Injury (PETAL). PETAL is funded by the NHLBI to develop and conduct randomized, controlled clinical trials to prevent or treat acute respiratory distress syndrome (ARDS) (14).

The survey was developed in two phases (Figure 1). The first phase collected general institutional information and identified EHR vendors. The second phase aimed to describe the nature and variability of e-alerts at individual sites using a quantitative and qualitative approach. The survey used 18 questions and is reproduced in the online supplement.

Figure 1.

Use of electronic alerts in Clinical Trials Network for the Prevention and Early Treatment of Acute Lung Injury. At present, the network has 47 hospital sites; at the time of the survey there were 38 sites.

Investigators at institutions who responded to the first phase of the survey, as well as a representative of one additional site that was identified during this phase, were contacted for the second phase. For the second phase, we contacted the principle investigators for each hospital site to identify local EHR experts. These experts then completed a semistructured telephone interview with detailed questions about the nature, purpose, and implementation of the e-alerts at their institution. We followed up in 5 business days to increase response rate and then followed up again with a phone call at 2 weeks post initial contact for clarifications.

Data Collection and Analysis

Research Electronic Data Capture (REDCap) system was used to send the survey and store survey data (15). We summarized categorical responses using frequencies and percentages. Stata 14 (StataCorp, College Station, TX) was used for statistical analysis.

Results

During the first phase of the survey, 38 PETAL sites were contacted, including 37 original sites and 1 additional site that we identified while conducting the survey. Of these, 32 sites (84%) responded. All respondents reported using EHR. Twenty-six were tertiary academic centers and six were community hospitals (Figure 1). We then followed up with the 32 sites for completion of the second phase of the survey. Thirty of the 38 PETAL sites (79%) completed phase two. Among those 30 respondents, 17 (57%) used e-alerts at the time of the survey.

Within our cohort of 30 responding institutions, 56% used Epic (Epic Systems Corporation, Verona, WI) as their EHR and 28% used homegrown institutional-specific health information systems. Although sites with locally developed EHR used e-alerts more frequently than those with commercial EHR, this difference was not statistically significant (77 vs. 48%, P = 0.13).

The most frequent acute care e-alerts were designed for early identification of systemic inflammatory response syndrome, sepsis, severe sepsis, or septic shock as a combined group (16 of 17 sites, 94%). Pneumonia alerts were the next most used (6 of 17, 35%). Alerts for ARDS (1 of 17, 6%) and pancreatitis (1 of 17, 6%) were infrequent.

Institutions used e-alerts equally for clinical decision support and research; 7 of 17 sites (41%) used e-alerts for both clinical decision support and research, 5 sites (29%) used e-alerts only for clinical decision support, and the 5 remaining sites (29%) used them only for research (Table 1). Clinical process improvement was a goal for 8 of 17 sites (47%). E-alerts were most commonly generated for patients at emergency department presentation (15 of 17 sites, 88%).

Table 1.

Conditions with electronic alerts and the goals of use in the Clinical Trials Network for the Prevention and Early Treatment of Acute Lung Injury

Condition or Goal	No.
Conditions with electronic alerts
Sepsis syndromes (including SIRS)	16
Pneumonia	6
ARDS	1
Acute pancreatitis	1
Other	8
Goals of electronic alerts
Research	12
Clinical decision support	12
Clinical process improvement	8
Other	2

Definition of abbreviations: ARDS = acute respiratory distress syndrome; SIRS = systemic inflammatory response syndrome.

Seventeen sites had e-alerts.

E-alerts used discrete variables, predominantly laboratory values (37%) or vital signs (37%) (Table 2). Query cycle time for e-alerts was less than 1 hour in 68% of alerts, including those with continuous cycling. Alert notification targeted multiple team members, including the research team at 65% of sites and the primary treating physician or nurse at 59%. Because alerts were generated in the emergency department or intensive care unit (ICU), only a small number of alerts notified rapid response team members.

Table 2.

Functional details of electronic alerts

Functional Details	Percentage of Responses
Variables included in e-alerts (total responses = 40)
Laboratory results	37
Vital signs	37
Radiology reports	15
Ventilator settings	8
Diagnostic codes	3
Location of e-alert operation (total responses = 24)
Within-hospital EHR	71
Separate server	29
Notification target (total responses = 35)
Research team	31
Primary treatment team physician	29
Primary treating nurse	29
Rapid response team member	6
Other	6
Action from e-alert (total responses = 22)
Change to clinical treatment	50
Enrollment in clinical trial	36
Other	14

Definition of abbreviations: e-alert = electronic alert; EHR = electronic health records.

Survey answers from institutions with active e-alerts. Some network sites had more than one e-alert. E-alert variables were factors used to derive the rules of the individual e-alert (single or multiple variables). Notification analysis included primary target, method, content, and action.

The content of the e-alerts included patient demographics and diagnosis at 69% of sites, clinical treatment guidelines at 65% of sites, and clinical trial enrollment reminders at 47% of sites. Changes to clinical treatments included initiation of bundled order sets (e.g., sepsis bundle), inpatient admission, antibiotic recommendations, or changes to ventilator settings. Qualitatively, half of the respondents reported that it would be possible to generalize their institution-specific e-alert across to other institutions or other EHR platforms, whereas a third of them did not know whether this was possible. Only one site, with a homegrown EHR, indicated no possibility of generalization. More than half the sites, 10 of 17 (59%), had tested the accuracy or validity of their e-alerts at their institution.

Discussion

Our three aims with this study were to understand the nature and variability of e-alerts, to understand the feasibility of e-alert standardization, and to develop a framework for electronic collaborations in multicenter research. To our knowledge, this is the first attempt to identify and describe electronic alerting across a large, U.S.-based research network. We found that all sites in the PETAL network use an EHR system, with a majority of sites using e-alerts in acute care settings. In our study, more than half of our sites reported using e-alerts for clinical research. This is an area of growing interest. E-alerts may have the potential for research recruitment, automated data collection, compliance detection, or identification of adverse events, because they are able to “alert” physicians about the patient’s eligibility and facilitate secure messaging to the trial coordinators (8). In our study, e-alerts were used for both clinical decision support and research. However, there was substantial heterogeneity within the specifics of the e-alert parameters, which may limit collaboration and generalization. In addition, there are limited data available on the use of automation or alerting for clinical trial recruitment.

In our study sites, e-alerts were used most frequently to identify the systemic inflammatory response syndrome and other sepsis syndromes. Severe sepsis is the 11th leading cause of death and costs the health care system $20 billion annually (16, 17). The prevalence of sepsis syndromes in the ICU is approximately 10%, with mortality ranging from 25 to 46% (18–20). Clinical trials on the effectiveness of e-alerts have shown mixed results (21–26). High sensitivity with poor specificity has been noted, and this may be a reason for decreased efficacy (27). The effectiveness has been described to be specifically worse in an ICU environment (24, 26, 27). A small majority (59%) of our sites had tested the effectiveness of their alerts. When data were available, sites reported high false-positive rates (15–20%) and low false-negative rates. This was most likely related to e-alerts being designed to be highly sensitive for screening.

An important issue that stands out in our study is the paucity of e-alerts related to ARDS. Only 1 of 17 sites (6%) had an ARDS e-alert. We postulate that this may be related to the complexity of interpreting chest radiograph reports with natural language processing, difficulty deriving ventilator settings from EHRs, or difficulty in accurately determining the beginning of acute lung injury or worsening of ARDS (e.g., initiation of inhaled nitric oxide as a marker for severe ARDS), which has been shown in other studies (28–30). However, Blum and colleagues developed automated alerting for the early identification of acute lung injury, and this resulted in significant reduction in the tidal volume administered compared with control subjects (31). Furthermore, alerts for other issues related to mechanical ventilation, such as safe tidal volume administration and weaning, have been used in the past (10, 32), suggesting that it is possible to develop e-alerts for ARDS.

Limitations

Our study is limited by the fact that it was directed at a cohort of institutions involved in a large U.S.-based clinical trials network. Therefore, bias in the favor of larger, academic institutions is possible. The use of e-alerts may differ in other settings. Another limitation is the use of a mixed-methods survey. Because the data were qualitative, it is possible that information on individualized e-alerts designed for specific locations or patient/disease populations within an institution may not have been identified. Finally, this was an investigation of the use of e-alerts and does not have the capability to determine the impact of e-alerts on the clinical care or research.

Conclusions

Despite limitations in study design, our study provides a useful analysis of the current use of e-alerts across a large clinical trials network in the United States. We found that e-alerts were being generated in the emergency room, ICU, and inpatient hospital settings, and there was ability to direct them to both health care staff and research team members. In addition, the query recycle time in most centers was short (<1 h), which should help in instituting time-sensitive real-time alerts.

We postulate that increase in use of standardized e-alerts across multicenter critical care research networks will lead to improved enrollment in clinical trials. The increased prevalence of EHR use and the market share of certain large EHR systems may lead to greater standardization of e-alerts across multisite acute care research networks, thus improving trial recruitment and reducing enrollment costs across differing clinical and research settings.