Does integrating nonurgent, clinically significant radiology alerts within the electronic health record impact closed-loop communication and follow-up?

Abstract

Objective To assess whether integrating critical result management software—Alert Notification of Critical Results (ANCR)—with an electronic health record (EHR)-based results management application impacts closed-loop communication and follow-up of nonurgent, clinically significant radiology results by primary care providers (PCPs).

Materials and Methods This institutional review board-approved study was conducted at a large academic medical center. Postintervention, PCPs could acknowledge nonurgent, clinically significant ANCR-generated alerts (“alerts”) within ANCR or the EHR. Primary outcome was the proportion of alerts acknowledged via EHR over a 24-month postintervention. Chart abstractions for a random sample of alerts 12 months preintervention and 24 months postintervention were reviewed, and the follow-up rate of actionable alerts (eg, performing follow-up imaging, administering antibiotics) was estimated. Pre- and postintervention rates were compared using the Fisher exact test. Postintervention follow-up rate was compared for EHR-acknowledged alerts vs ANCR.

Results Five thousand nine hundred and thirty-one alerts were acknowledged by 171 PCPs, with 100% acknowledgement (consistent with expected ANCR functionality). PCPs acknowledged 16% (688 of 4428) of postintervention alerts in the EHR, with the remaining in ANCR. Follow-up was documented for 85 of 90 (94%; 95% CI, 88%-98%) preintervention and 79 of 84 (94%; 95% CI, 87%-97%) postintervention alerts (P > .99). Postintervention, 11 of 14 (79%; 95% CI, 52%-92%) alerts were acknowledged via EHR and 68 of 70 (97%; 95% CI, 90%-99%) in ANCR had follow-up (P = .03).

Conclusions Integrating ANCR and EHR provides an additional workflow for acknowledging nonurgent, clinically significant results without significant change in rates of closed-loop communication or follow-up of alerts.

BACKGROUND AND SIGNIFICANCE

Rapid increase in electronic health record (EHR) use has been spurred by the Health Information Technology for Economic and Clinical Health act of 2009, which sought to improve the quality, safety, and efficiency of US health care through universal adoption of interoperable EHRs.^1–3 Part of the envisioned efficiency gains from EHR use may be realized through the integration of workflows, enabling providers to use a single system or a group of interoperable, seamlessly integrated applications to accomplish their work rather than logging into multiple systems or using paper-based approaches.

The Alert Notification of Critical Results (ANCR) software was developed and implemented at our institution to facilitate closed-loop communication of critical test results, including nonurgent, clinically significant results.⁴ Alerts are initiated by radiologists who use clinical judgment to select 1 of 3 predefined levels of alert urgency as defined by institutional policy (see supplemental online appendices).^5,6 In contrast to systems that flag an entire report as abnormal,^7,8 ANCR highlights the specific concerning finding for the alert recipient. Referring providers are actively notified by pager for life-threatening and urgent findings (eg, tension pneumothoraces and intra-abdominal abscesses), and by pager or email for nonurgent, clinically significant findings (eg, pulmonary nodules and solid renal masses).⁴ All alerts must be acknowledged within a prespecified timeframe that is enforced by an institutional escalation policy for clinically significant results.⁹ ANCR was successfully adopted and associated with increased adherence to this policy while reducing workflow interruptions.¹⁰

A results management application was developed as part of our EHR by our enterprise information systems group to facilitate laboratory test results tracking. With rule-based designation of result abnormality, it allows providers to easily identify, sort, and acknowledge results within their queue.¹¹ While the application's rules can identify radiology reports with abnormal Breast Imaging-Reporting and Data System classifications, the vast majority of radiology reports cannot be evaluated. This application is widely used by primary care providers (PCPs) to review, acknowledge, and track test results.¹² Although ANCR is considered part of a patient's medical record, ANCR-generated alerts are not available to providers within the EHR.

Prior to this study, ANCR was the institution's method for communicating and acknowledging most critical imaging results, while the EHR-based results management application was used for tracking and acknowledging laboratory test results. After integration of the 2 applications (the intervention), PCPs could view and acknowledge nonurgent, clinically significant ANCR-generated alerts from their EHR queues (“standard” workflow for essentially all PCPs at our institution) or from within ANCR itself. Radiology reports were designated with the same level of urgency in ANCR and in the EHR, and acknowledgement in 1 system was reflected in the other. We hypothesized that integrating ANCR with the EHR would provide an alternative method for acknowledging nonurgent, clinically significant imaging results, and that integration would not adversely impact rates of closed-loop communication or follow-up of these imaging results.

Objective

Our objective was to assess whether integrating critical radiology result management software (ANCR) with an EHR-based results management application impacted rates of closed-loop communication and follow-up of nonurgent, clinically significant radiology results by PCPs.

METHODS

Study Design

This institutional review board-approved study was performed at a 793-bed tertiary academic medical center with a radiology department performing over 600 000 examinations annually. Nonurgent, clinically significant alerts received by 171 PCPs working at 13 affiliated outpatient practices during a 12-month preintervention and 24-month postintervention period were included in the study. Inclusion and exclusion criteria for the study cohort are summarized in figure 1 and are discussed as the components of the system are described.

Figure 1:

Study Cohort. ANCR, Alert Notification of Critical Results software; PCP, primary care provider; EHR, electronic health record.

Alert Notification of Critical Results

This web-based system that was developed at our institution to facilitate closed-loop communication of critical test results, including nonurgent, clinically significant results, has been previously described.⁴ When a radiologist identifies a critical result, they categorize it as immediately life-threatening (Level 1), urgent (Level 2), or nonurgent, clinically significant (Level 3), as per institutional policy (see supplemental online appendices).^5,6 The referring provider is notified via pager for Level 1 and 2 alerts and via pager or email for Level 3 alerts. The referring provider for an outpatient examination is the provider who placed the order, while the referring provider for inpatient and emergency department (ED) examinations is the provider currently caring for the patient. If the patient has been discharged from the hospital or ED, the responsibility reverts to the ordering provider. In general, referring providers return pages for Level 1 and 2 alerts, and the radiologist acknowledges the alert on their behalf. Referring providers can opt for the same workflow for Level 3 alerts, or they can follow a secure link in the ANCR notification email to review alert details and acknowledge the alert within the ANCR on any tethered or mobile device using a web browser. Level 3 alerts must be acknowledged within 15 days and an escalation policy that holds both the referring provider and radiologist accountable for overdue alerts enforces this requirement. To ensure acknowledgement in the EHR was an option for the referring provider, Level 1 and 2 alerts were excluded from the study cohort.

Results Manager, Electronic Health Record

The results management application developed by our enterprise information systems group to facilitate laboratory test results tracking is accessible within our web-based EHR and has also been previously described.¹¹ Providers are only required to acknowledge results with the highest level of urgency—acknowledgment of test results with lower levels of urgency can be used for record keeping but is not required. The results management application was designed for and is used primarily by PCPs; it is variably used by specialists. Inpatient and ED providers use other systems dedicated to those workflows.¹³ This study was limited to alerts generated on outpatient radiology examinations ordered by PCPs. As PCPs ordered the examinations, they received the alerts for nonurgent, clinically significant results and could acknowledge these alerts in the EHR.

ANCR-EHR Integration

The radiology department and enterprise information systems group collaborated to modify and create web services for transfer of data between ANCR and the EHR. When an ANCR alert is created, a web service notifies the EHR, which searches its archives for a radiology examination with the same accession number. Level 1 and 2 ANCR alerts flag test results with the highest urgency level (!!!), and Level 3 (nonurgent, clinically significant) ANCR alerts flag test results as second-highest urgency (!!). The ANCR alert text appears in the “Alerts and Guidelines” section of the results details screen (figure 2). If the provider acknowledges the critical result in the EHR, the acknowledgment is communicated to ANCR via a web service and the alert becomes automatically acknowledged in ANCR. Conversely, if the alert is acknowledged in ANCR, ANCR uses a web service to acknowledge the alert in the EHR. Primary care providers were notified of ANCR-EHR integration via email although no specific additional training was provided.

Figure 2:

Electronic Health Record Results Manager Test Result Details Screen. (a) After integration, radiology reports are flagged with the degree of urgency of associated Alert Notification of Critical Results software (ANCR) alerts; (b) When primary care providers use this button to acknowledge radiology reports, acknowledgment is reflected in ANCR; (c) ANCR alert text appears in the alerts and guidelines box.

Outcome Measures

System Adoption

System adoption, our primary outcome, was measured as the proportion of nonurgent, clinically significant (Level 3) ANCR-generated alerts acknowledged in the EHR. Adoption was measured for each PCP and for all PCPs in aggregate.

Alert Follow-up (Actionable and Actioned Alerts)

We identified nonurgent, clinically significant ANCR alerts generated for 171 PCPs from May 2011 to April 2012 (preintervention) and June 2012 to May 2014 (postintervention). Our secondary outcome was the rate of alert follow-up defined as the proportion of actionable ANCR-generated alerts that were acted upon appropriately (“actioned”). A multidisciplinary team developed criteria for actionable and nonactionable ANCR alerts (table 1).

Table 1.

Criteria for actionable alerts

Actionable alert	Relevant actiona
Intervention recommended (biopsy, aspiration, surgical procedure)	Intervention performed
Imaging recommended (same modality, different modality, comparison with priors)	Imaging performed (any modality, if addresses critical finding)
Clinical assessment recommended (specific history or physical question, referral to specialist, laboratory test results, follow-up on specimen required)	Documentation of clinical assessment, consult note, laboratory test results, documentation of result review
Description of condition with obvious treatment (infection, fracture, etc)	Antibiotics, splinting/casting for fracture
Possible new malignancy with no specific recommendations	Imaging, biopsy, surgical procedure, laboratory test results, consult, chemotherapy or radiation

Nonactionable alert
Description of finding without mention of a condition that could harm a patient and no clear action to be taken
Normal findings, return to normal care

^aIn all cases, an alert was considered acted upon if a provider's note mentioned the alert and why it was not acted upon or that the patient refused further action.

Using these criteria, a radiologist with 3 years of experience (SDO) categorized a random sample of alerts as actionable or not, created chart abstractions containing relevant follow-up actions and/or notes, and categorized actionable alerts as having been actioned or not. A radiologist with 6 years of experience (VAS) and an internist with 10 years of experience (AKD) then independently reviewed the alerts and abstractions, and any disagreements between the 3 physicians were deliberated until reaching consensus. All physicians were blinded to the method and timing of alert acknowledgement. If the timing of the alert was such that the action should occur after the time of record review (such as an alert from 6 months prior to record review recommended computed tomography in 1 year), the alert was removed from the actionable alert cohort.

Power and Sample Size Calculation. In a review of 50 random alerts in the preintervention cohort by 1 radiologist (SDO), 47 (94%) acknowledged alerts were actionable and 44 (89%) actionable alerts were actioned. Therefore, to detect an increase in the rate of actionable alerts that were actioned from 89% to 99% (a 10% absolute increase), with 80% power and α level of .05, a sample size of 182 acknowledged alerts would be needed. To be conservative, we randomly sampled 200 acknowledged alerts for review; patient characteristics for the study cohort and sample are presented in table 2.

Table 2.

Patient demographics

Characteristics	Study Cohort (n = 4805)	Sample Size of Acknowledged Alerts (n = 199)
Age, mean (SD), y	57 (15)	57 (15)
Female, No. (%)	3556 (74)	151 (76)

Time to Acknowledgement

Time to acknowledgement was measured as the interval between alert generation by the radiologist and acknowledgement by the PCP. The Wilcoxon rank sum test was used to compare time to acknowledgement for alerts in the pre- and postintegration periods as well as between ANCR-acknowledged and EHR-acknowledged alerts in the postintegration period.

Statistical Analysis

Alert follow-up, as defined above, was compared for alerts in the pre- and postintervention periods using the Fisher exact test, and the Cochran-Armitage trend test was used to evaluate for a trend across the 3 years in the study. To determine the maximum theoretical impact of the intervention, we compared follow-up rates for alerts acknowledged in the EHR vs ANCR in the postintervention period. Two-sided P values <0.05 were considered statistically significant. System adoption was analyzed over time using a P-type statistical process control chart.

RESULTS

Study Cohort

Of 100 672 alerts sent via ANCR in the 12 months preimplementation and 24 months postimplementation, 6841 (6.8%) were sent to PCPs. After excluding alerts not in the EHR, alerts generated on inpatients and ED patients, 6440 alerts remained. The 5931 Level 3 (nonurgent, clinically significant) alerts, comprising 92% of alerts generated for PCPs on outpatient examinations, formed the study cohort (figure 1). The 171 PCPs received 1 to 200 alerts each, a median of 41 alerts (interquartile range [IQR], 19-69).

System Adoption

The EHR was used to acknowledge 15.5% (688 of 4428) of alerts generated in ANCR in the postintervention period (table 3). A statistical process control chart (figure 3) demonstrated that the month-to-month variation in this proportion was not statistically significant.¹⁴ Providers acknowledged 0% to 100% of their alerts in the EHR (median, 4% [IQR 0-15]).

Table 3.

Acknowledgement of ANCR-generated alerts

Cohort	No. of Total Alerts	Alerts Acknowledged in ANCR, No. (%)	Alerts Acknowledged in EHR Results Manager, No. (%)	Total Acknowledged Alerts, No. (%)
Preintervention	1503	1503 (100)	0 (0)	1503 (100)
Postintervention	4428	3740 (84)	688 (16)	4428 (100)
Year 1	1979	1669 (84)	310 (16)	1979 (100)
Year 2	2449	2071 (85)	378 (15)	2449 (100)

Abbreviations: ANCR, Alert Notification of Critical Results software; EHR, electronic health record.

Figure 3:

Proportion of ANCR Alerts Acknowledged in the Electronic Health Record. ANCR, Alert Notification of Critical Results software; UCL, upper control limit; LCL, lower control limit; EHR, electronic health record.

Alert Follow-up

Using a random sample and by consensus review, 90% (90 of 100; 95% CI, 83%-94%) of alerts preintervention and 84% (84 of 100; 95% CI, 76%-90%) of alerts postintervention were actionable (P = .29; table 4). PCPs acted on 94% (85 of 90; 95% CI, 88%-98%) of actionable alerts preintervention and 94% (79 of 84; 95% CI, 87%-97%) postintervention (P > .99). In the postintervention period, PCPs acted on 79% (11 of 14; 95% CI, 52%-92%) of actionable alerts acknowledged in the EHR and 97% (68 of 70; 95% CI, 90%-99%) of actionable alerts acknowledged in ANCR (P = .03).

Table 4.

Estimates of actionable and actioned alerts^a

Cohort	Sample No. of Alerts	No. of Actionable Alerts	Actionable Alerts, Proportionb (95%CI)	No. of Actioned Actionable Alerts	Actioned Alerts, Proportionb (95% CI)
Preintervention	100	90	90 (83–94)	85	94 (88–98)
Postintervention	100	84	84 (76–90)	79	94 (87–97)
Year 1	41	34	83 (69–91)	31	91 (77–97)
Year 2	59	50	85 (73–92)	48	96 (87–99)

^aNo significant difference between the proportion of actionable alerts acted upon pre- and postintervention (P > .99 for pre- vs postintervention; P = .79 for trend from preintervention to Year 1 postintervention to Year 2 postintervention).

^bConfidence intervals are presented as percentages.

Time to Acknowledgement

The median time to acknowledgement was 0.6 hours (IQR 0.03-20.5 h) preintervention and 0.6 hours (IQR 0.03-6.9 h) postintervention (P = .003). The median time to alert acknowledgement was 7 hours (IQR 1.1-46.4 h) in the EHR and 0.3 hours (IQR 0.02-3.3 h) in ANCR (P < .001).

DISCUSSION

Integrating a critical radiology results management system (ANCR) with an EHR-based results management application provided an alternate method for managing critical results but was not associated with a change in the rate of closed-loop communication or the proportion of actionable alerts acted upon by PCPs. Although adoption of the integrated ANCR-EHR system was immediate and sustained, PCPs continued to acknowledge the majority of critical radiology results in ANCR postintervention. Compared with the EHR, PCPs acknowledged alerts in a significantly shorter duration of time when using ANCR. Nevertheless, the follow-up rate for actionable alerts acknowledged via the EHR and ANCR were still similar.

We have several potential explanations for these findings. First, although PCPs have established workflows that can take advantage of ANCR-EHR integration (eg, reviewing test results in the EHR-based results management application while seeing patients in clinic) and may prefer notification in the EHR over email notification,¹⁵ this is not a consistent preference.¹³ Second, ANCR relies on “active” external notifications (eg, notifications are sent to the referring provider’s pager or network email inbox) as opposed to “passive” internal EHR notifications (eg, providers must access the EHR to view and acknowledge alerts). Providers may not accept a system that requires them to actively seek test results, especially if that system does not fit with their workflow.¹³ Third, ANCR offers a mobile workflow—referring providers can open a secure hyperlink from a notification email that directs them to view and acknowledge an alert. Fourth, ANCR enforces the enterprise policy of 100% closed-loop communication of critical results⁹ by automatically reminding radiologists and referring providers of outstanding alerts until they are acknowledged. Finally, providers can launch the computerized provider order entry application from the alert detail page in ANCR, facilitating ordering of follow-up imaging.

Our study builds on the findings of prior studies. We describe the follow-up of nonurgent but clinically significant (Level 3) radiology results, and include all alerts with this level of urgency rather than concentrating on unexpected findings of possible malignancy. The follow-up rate we observed is higher than in other studies of abnormal findings^15,16 and follow-up recommendations^17,18 in radiology reports, but it is consistent with other critical result alert systems.^8,7 Additionally, our findings confirm that results management applications work best when designed to support enterprise-wide policies. A recent study at our institution found that PCPs acknowledged no more than 78% of the Level 3 (nonurgent but clinically significant) alerts via the EHR,¹² whereas the rate of acknowledgement via ANCR at baseline and postintervention was consistently 100%.

We acknowledge limitations of this study. First, it was performed at 1 academic center with a single combination of critical radiology results software and EHR-based results management application. However, multiple critical result communication systems have been built external to local EHRs,^8,19–21 and many EHRs have test result management tools^11,22; the benefits of integrating these systems has not been thoroughly investigated. Second, although ANCR alerts could be sent by radiologists at our community hospital as well as those at our main hospital, the intervention only integrated alerts generated at the main hospital with the EHR. PCPs were required to continue to acknowledge alerts from the community hospital via ANCR and may have been reluctant to use parallel workflows, preventing greater adoption of ANCR-generated alert acknowledgement in the EHR. Additionally, notifications of critical results from the 2 hospitals were identical, so, even if they were willing to use 2 workflows, PCPs did not have an easy way to determine which results they could acknowledge in the EHR. Next, there was wide variation in the proportion of ANCR-generated alerts each PCP acknowledged in the EHR, raising the question of whether there was variation in the impact of the intervention on alert follow-up by each PCP. As alerts were distributed amongst a large number of PCPs, with most providers each receiving less than 1% of all alerts, the clustering effect would have been minimal. Therefore, we chose to use a simple model that did not include the PCP as a covariate. Some of the variation in adoption may have been due to lack of training, and adoption rate may have been improved with additional training. Finally, this study uses pre-postintervention analysis rather than a randomized trial design, so we cannot control for changes in follow-up of actionable alerts due to factors other than our intervention. However, as no other interventions targeting follow-up of nonurgent clinically significant radiology findings were introduced at our institution during the study period, and the proportion of ANCR-generated alerts acknowledged in the EHR was constant in the 2 years postintervention, it is unlikely that such factors significantly affected our findings.

Conclusion

Integration of ANCR-generated alerts into the outpatient EHR provided an alternate method for nonurgent, clinically significant results management and did not adversely impact the rate of closed-loop communication or nonurgent, clinically significant radiology results that were appropriately acted upon by PCPs.

CONTRIBUTORS

All authors contributed sufficiently to this work to be included as authors, and all those who are qualified to be authors are listed as authors.

FUNDING

This work was partially supported by grants from the Controlled Risk Insurance Company Risk Management Foundation and the Boston Area Research Training Program in Biomedical Informatics (National Library of Medicine grant T15LM007092).

COMPETING INTERESTS

None.

Supplementary Material

Supplementary material is available online at http://jamia.oxfordjournals.org/.