Four-Year Impact of an Alert Notification System on Closed-Loop Communication of Critical Test Results

Abstract

OBJECTIVE

One of the patient safety goals proposed by the Joint Commission urges hospitals to develop a policy for communicating critical test results and to measure adherence to that policy. We evaluated the impact of an alert notification system on policy adherence for communicating critical imaging test results to referring providers and assessed system adoption over the first 4 years after implementation.

MATERIALS AND METHODS

This study was performed in a 753-bed academic medical center. The intervention, an automated alert notification system for critical results, was implemented in January 2010. The primary outcome was adherence to institutional policy for timely closed-loop communication of critical imaging results, and the secondary outcome was system adoption. Policy adherence was determined through manual review of a random sample of radiology reports from the first 4 years after the intervention (n = 37,604) compared with baseline outcomes 1 year before the intervention (n = 9430). Adoption was evaluated by quantifying the use of the system overall and the proportion of alerts that used noninterruptive communication as a percentage of all reports generated by 320 radiologists (n = 1,538,059). A statistical analysis of the trend at 6-month intervals over 4 years was performed using a chi-square trend test.

RESULTS

Adherence to the policy increased from 91.3% before the intervention to 95.0% after the intervention (p < 0.0001). There was a ninefold increase in the critical results communicated via the system (chi-square trend test, p < 0.0001). During the first 4 years after the intervention, 41,445 alerts (41% of the total number of alerts) used the system's non-interruptive process for communicating less urgent critical results, which was substantially unchanged over the 4 years postintervention, thus reducing unnecessary paging interruptions.

CONCLUSION

An automated alert notification system for communicating critical imaging results was successfully adopted and was associated with increased adherence to institutional policy for communicating critical test results and with reduced workflow interruptions.

Timely communication among care team members is of utmost importance in patient management, particularly when involving critical test results. An unnecessary delay can jeopardize patient safety by hampering further testing or can lead to errors in management [1–6]. Communication failures increase the risk of adverse patient events [7–9] and account for significant malpractice risk and claims [10]. Delayed communication of critical test results also creates significant anxiety for patients, particularly when awaiting the results of tests for potentially life-threatening conditions. Therefore, the Joint Commission has recognized communication of critical test results as a national patient safety goal [11].

We established an enterprise-wide policy for the communication of critical test results to specifically address a potential breakdown in the patient care process that can lead to primary care claims in Massachusetts: the nontransmittal of diagnostic test results [10, 12, 13]. For imaging studies, the policy provides guidelines and procedures for notifying the attending provider who is primarily involved in a patient's care and who can take action when a critical imaging result is identified by a radiologist. A “critical imaging result” is defined as the following: new or unexpected radiologic findings that could result in mortality or significant morbidity without appropriate follow-up or interpretations differing from a previously communicated preliminary interpretation [14]. Based on the recommendations of the Massachusetts Coalition for Patient Safety [15], the policy includes three categories of urgency: red, orange, and yellow alerts. Alert levels are determined on the basis of the urgency with which the radiologist believes the critical results should be communicated and addressed; for example, tension pneumothorax on chest radiography, pancreatic necrosis on abdominopelvic CT, and a new incidental pulmonary nodule on chest CT are examples of red, orange, and yellow alerts, respectively. Alerts have corresponding notification time parameters: Red alerts are urgently life-threatening, requiring documented closed-loop communication within 60 minutes, whereas orange and yellow alerts require communication within 3 hours and 15 days, respectively. The manner of communication varies as well; red alerts are urgent and must be synchronous and interruptive, requiring an in-person or telephone conversation between the radiologist and the licensed independent practitioner caring for the patient. Communication of other results that are important but are less time-sensitive (e.g., yellow alerts) may be asynchronous and noninterruptive, but documentation of closed-loop communication is still required.

A quality improvement initiative was undertaken after the policy was established. A manual review of a random sample of radiology reports for a previous study [14] showed that the percentage of critical imaging results adherent to the policy increased from 28.6% before the initiative to 90.4% over a 4-year period. However, the authors of that study reported that a portion of critical results were still not adherent to the policy, perhaps because of the manual, heavily analog processes required to communicate critical test results. Additionally, policy adherence required inefficient workflow interruptions because a paging system was being used by radiologists and ordering providers for all alert levels at that time. Therefore, we developed and implemented an automated system named the “Alert Notification of Critical Results” or “ANCR” to further facilitate imaging critical test result notification, documentation, management, and communication among providers [16].

The objective of this study was to evaluate the impact of the automated system (ANCR) on adherence to the institutional policy for communicating critical imaging test results and to assess the system's adoption for the first 4 years after implementation.

Materials and Methods

Study Setting and Population

The ANCR system was designed and implemented at a 753-bed urban adult tertiary referral academic medical center. A waiver of informed consent for medical record review was granted by our institutional review board for this HIPAA-compliant study. All reports from radiology studies completed in the emergency department, inpatient hospital, and outpatient clinics from January 2009 through December 2013 were included. Multiple imaging tests resulting in one radiology report (e.g., abdomen and pelvis CT, report addenda) were treated as single reports.

Intervention

The ANCR system was implemented in January 2010. The software code and its functional and technical specifications are publicly available [16]. Briefly, ANCR is embedded in the workflow of radiologists and referring providers through integration with multiple systems including the PACS, paging and e-mail systems, and electronic medical record. It is closely integrated with a radiology quality analytics system [17, 18]. It allows radiologists to communicate critical findings through synchronous mechanisms (e.g., paging) or asynchronously (i.e., secure and HIPAA-compliant e-mail); the latter was specifically instituted for yellow alerts that account for the majority of critical results at our institution. ANCR allows secure, web-enabled acknowledgment by ordering providers of the alerts received, and the alerts and acknowledgments are accessible from tethered (e.g., PC or Macintosh [Apple Inc.]) or mobile (e.g., iPhone [Apple Inc.], iPad [Apple Inc.], Android [Google]) devices. Additionally, all alerts are tracked and are auditable to ensure findings are acknowledged in a timely fashion and that the communication loop is closed.

Study Outcome Measures

The primary outcome was ANCR's impact on radiologists’ adherence to the policy, and the secondary outcome was ANCR's adoption during the first 4 years after implementation, including utilization of noninterruptive ANCR communication for yellow alerts.

Adherence to the policy was measured by the percentage of radiology reports with critical results that were adherent to the policy provisions for both timely and closed-loop documentation for communicating critical results. Documentation was defined as complete if it included the name of the ordering provider who was contacted and the date and time the critical results were communicated. Timeliness was defined by the specific alert level (e.g., within 60 minutes for red alerts) as defined by the policy [12].

Adoption was defined as the proportion of all finalized radiology reports that used ANCR. In addition, we calculated the proportion of finalized radiology reports using ANCR in relation to all finalized radiology reports containing documentation of communication to the referring provider or care team. “Communicated radiology reports” were defined as those reports containing documentation that radiologists communicated findings with another clinician. Although not all results communicated to a clinician are expected to be critical results, we hypothesized that the usefulness of ANCR for communicating critical results would also result in an increase in ANCR use for all communicated radiology reports.

The use of noninterruptive ANCR communication, which is allowed by our institutional policy for yellow alerts only, was defined as the total proportion of finalized radiology reports each year using ANCR secure e-mail compared with those using the paging system for yellow-level alerts. For ANCR alerts sent through both secure e-mail and paging, the modality used for sending the initial communication was counted.

Data Collection and Statistical Analysis

To determine the percentage of critical results adherent to the policy, we analyzed records from the bimonthly review being conducted by the heads of each radiology section in which all radiology reports for one randomly selected day every other month were assessed for critical results documented in the report [14]. We continued this quality initiative and reviewed approximately 9600 reports per year beginning in January 2009. With 10% of reports expected to contain critical results [14], this process provided 90% power at a two-tailed alpha value of 0.05 to detect a 4% increase in the proportion of reports adherent to the policy due to the intervention. All reviewers were instructed regarding the policy and the review process. Each report was assigned to one of the four following categories: category 1, report contained a critical result and was adherent to the policy requirements; category 2, report contained a critical result and was not adherent to policy requirements; category 3, report contained a critical result but communication was not documented; and category 4, report did not contain a critical result [14]. Reports in categories 2 and 3 were deemed nonadherent to the policy. The preimplementation percentage of policy adherence in the sampled reports was compared with adherence postimplementation. A comparison of proportions was performed using a two-sided chi-square analysis.

We also determined policy adherence for all reports that used ANCR to communicate findings during the first 4 years after implementation. We queried the ANCR database and captured the alert level and information about the communication (radiolo-gist's name, ordering provider's name, and date and time of communication). A chi-square trend test was used to analyze the proportion of critical results that adhered to the policy over time.

To analyze ANCR adoption, we recorded and counted all unique radiology report accession numbers in 6-month time periods as the denominator and all distinct ANCR alerts as the numerator. ANCR adoption was also calculated as a proportion of all findings that radiologists communicated with another clinician. The numerator was the total number of all distinct ANCR alerts, and the denominator was all unique reports that contained findings communicated by radiologists with another clinician. All reports containing documentation of communication by a radiologist with another physician were retrieved using a publicly available document retrieval toolkit, Information From Searching Content With an Ontology-Utilizing Toolkit (iSCOUT) [19]; iSCOUT has successfully been used for retrieving radiology reports containing critical results [20–22]. A chi-square trend test was used to analyze ANCR adoption as a proportion of all unique radiology reports and as a proportion of all unique reports that contained documented communication between physicians over time.

To determine adoption of noninterruptive ANCR communication, we counted all ANCR yellow-level alerts that used secure e-mail each year and those that used the paging system for communication.

Results

A total of 1,914,044 radiology reports, generated by 320 radiologists, were available for analysis during the 1 year before (n = 375,985) and the first 4 years after (n = 1,538,059) ANCR implementation.

Impact of ANCR System on Adherence to the Critical Results Policy

A total of 9430 reports were randomly sampled and manually reviewed before ANCR implementation and 37,604 reports after ANCR implementation. Table 1 illustrates a comparison of adherence to the policy for critical imaging results identified during the reviews before and after system implementation. All critical results including those not communicated through ANCR are shown. Adherence to the policy increased from 91.3% before ANCR implementation in 2009 to 95.0% after implementation (p < 0.0001).

TABLE 1.

Comparison of Adherence to Policy for Communication of Critical Test Results Before and After Implementation of Alert Notification of Critical Results (ANCR) System

Year	No. of Reports With Critical Results		No. of Reports With Noncritical Results	% of Sampled Reports With Critical Results (No./Total No.)	% of Reports With Critical Results Adherent to Policy (No./Total No.)
	Adherent to Policy	Nonadherent to Policy
2009 (before ANCR)	901	86	8443	10.5 (987/9430)	91.3 (901/987)
2010–2013 (after ANCR)	4688	247	32,669	13.1 (4935/37,604)	95.0a (4688/4935)

^aChi-square test, p < 0.0001.

The percentage of ANCR-communicated radiology reports adherent to the policy, by alert level and in total, are displayed in Table 2. The majority of alerts were for yellow-level findings, comprising 68.6% of all ANCR alerts from 2010 through 2013. Adherence to the policy was 90.7% in 2010, with 7843 alerts. Beginning in 2011, policy adherence for all reports with critical results communicated through ANCR increased to 97.9% (chi-square trend test, p < 0.0001).

TABLE 2.

Adherence to Policy for Communication of Critical Test Results for All Results Communicated Using Alert Notification of Critical Results (ANCR) System

Year	Total No. of Radiology Reports	Red-Level Alerts		Orange-Level Alerts		Yellow-Level Alerts		Total No. of ANCR Alerts (% of Total)	Total % of Reports Adherent to Policya
		No. (% of Total)	% Adherent to Policy	No. (% of Total)	% Adherent to Policy	No. (% of Total)	% Adherent to Policy
2010	363,332	178 (0.1)	96.1	1819 (0.5)	92.5	5846 (1.6)	89.8	7843 (2.2)	90.7
2011	382,983	542 (0.1)	97.2	9816 (2.6)	97.6	18,126 (4.7)	98.0	28,484 (7.4)	97.9
2012	388,721	278 (0.1)	98.6	10,176 (2.6)	96.9	21,202 (5.5)	98.7	31,656 (8.1)	98.1
2013	403,023	298 (0.1)	96.0	8582 (2.1)	94.9	24,540 (6.1)	98.0	33,420 (8.3)	97.9
Total	1,538,059	1296 (0.1)	97.4	30,393 (2.0)	96.9	69,714 (4.5)	97.3	101,403 (6.6)	97.1

^aChi-square trend test, p < 0.0001.

Adoption of ANCR System

ANCR was used for critical result communication in 101,403 radiology reports in the postintervention period, and ANCR alerts were communicated to 4323 ordering and attending providers. In addition to an absolute increase in ANCR use across all radiology reports, Figure 1 also shows the increase in ANCR use among reports with documented communication with another clinician, whereas the proportion of all reports with any documented communication remained relatively constant throughout the postintervention period. The ANCR-usage proportion reached 0.81 for all communicated reports in the second half of 2011, only 18 months after implementation. This proportion remained relatively unchanged through 2013.

Fig. 1.

Adoption of Alert Notification of Critical Results (ANCR) system over first 4 years after implementation. Jan = January, Dec = December.

A and B, Graphs show proportion of radiology reports in which ANCR was used to communicate critical results as compared with radiology reports with documented communication (A) and as compared with all radiology reports (B).

Of 69,714 yellow-level alerts, 41,445 were communicated using noninterruptive communication (i.e., secure HIPAA-compliant e-mail), with 28,269 using interruptive communication (i.e., paging) (Fig. 2). The use of noninterruptive methods for communicating yellow-level alerts remained substantially unchanged over 4 years (chi-square trend test, p = 0.15). Noninterruptive communication accounted for 60% of all yellow-level alerts and 41% of all ANCR alerts over 4 years.

Fig. 2.

Bar graph shows mode of communication for yellow-level alerts communicated using Alert Notification of Critical Results (ANCR) system over first 4 years after implementation. Yellow-level alert is defined as result that is not immediately life-threatening or urgent. Communication of yellow alerts may be asynchronous and noninterruptive, but documentation of closed-loop communication is still required.

There was a ninefold increase in the use of ANCR as a proportion of all unique radiology reports (chi-square trend test, p < 0.0001) and a sevenfold increase in ANCR adoption as a proportion of all reports with any documented communication (chi-square trend test, p < 0.0001) comparing the last 6 months of the study with the first 6 months after ANCR implementation.

Discussion

The use of an automated system to facilitate and track closed-loop communication of critical imaging results increased ninefold within the first 4 years after implementation, reaching 9% of all finalized radiology reports and more than 80% of finalized reports with documented communication (whether or not the communication was for critical results). The adoption of ANCR increased rapidly in the first 18 months after implementation and was subsequently sustained, providing evidence of the acceptance of the system and incorporation into radiologist workflow. ANCR use was associated with significantly increased adherence to the institution's policy for communicating critical results, reaching 95.0% for all finalized radiology reports containing critical results (whether or not ANCR was used to communicate the critical results). In the subset of finalized reports in which ANCR was used to communicate critical results, adherence to the policy, with referring providers using ANCR to acknowledge result receipt, was more than 98%. The 2% of reports that were nonadherent to the policy were considered nonadherent primarily because the closed-loop communication occurred beyond the time frame stipulated by our policy. Because ANCR records the time stamp when the alert is created and when it is acknowledged, verbal acknowledgments that were not documented in ANCR until beyond the time frame stipulated by our policy are marked as late and thus are considered nonadherent to institutional policy. Our findings suggest systems enabling communication of critical test results can be embedded in physician workflow to significantly improve closed-loop communication of critical results, with inherent relevance for improving patient safety. We have made ANCR software code and its functional and technical specifications publicly available [16].

Our implementation appears to have overcome a number of known barriers to instituting new clinical information systems in multisetting health care institutions, including new or increased workload for clinicians, unfavorable workflow issues, and changes in communication patterns [23, 24]. We believe that three major factors positively influenced adoption at our institution [1]: first, substantial institutional and executive support for the establishment, monitoring, and enforcement of the policy for the communication of critical test results; second, integration of ANCR within the existing physician workflow, both for radiologists and for referring providers including secure web-enabled use of mobile devices to acknowledge alerts [25–27]; and, third, ANCR's capability to enable secure HIPAA-compliant e-mail (asynchronous and noninterruptive communication) to replace the more time-consuming, intrusive, and interruptive person-to-person or telephone communication among providers enabled through the institution's paging system.

With respect to workflow, the ANCR system is also integrated with multiple clinical systems so the amount of data input required by the radiologist and the receiving care provider is reduced, thus facilitating result communication for both groups. In addition, significant feedback was elicited from end users including radiologists, ordering providers, and clinical administrators during system design to improve workflow efficiency (e.g., minimize duplication of effort when alerts are acknowledged).

An additional important workflow enhancement is the ability to communicate less urgent but nevertheless critical results asynchronously through noninterruptive communication. The majority of critical results communicated through ANCR are yellow-level alerts (68.6%), and the majority of these (60%) are now communicated through web-enabled secure, HIPAA-compliant e-mail communication that can be acknowledged using a mobile device such as a smart telephone or a tablet. Interruptive communication (e.g., telephone, pager) unnecessarily contributes to inefficiency in the workplace and may disrupt patient care [28, 29]. More importantly, noninterruptive notification is preferred by radiologists and referring providers for communicating less urgent findings.

Establishing the policy and the ANCR system for critical imaging results communication involved a number of challenges. No established guidelines exist for classifying imaging results as “critical,” unlike for laboratory tests where an outlier value beyond the range of acceptable values is deemed abnormal—making them easily amenable to automated alerts [30–32]. The unstructured textual reports used in radiology do not lend themselves easily to automated processing for detecting critical results for various reasons. For any single imaging examination, multiple findings are often enumerated and each can be described in various ways by different radiologists. More importantly, some abnormal imaging findings that are deemed critical in specific cases are not necessarily critical in other clinical settings. For instance, newly discovered intracranial hemorrhage on CT in a patient presenting with a severe headache to the emergency department constitutes a critical result by our policy. However, stable intracranial hemorrhage in the same patient on a follow-up CT study as an inpatient is no longer a critical result by our policy. Thus, according to our policy, potentially critical imaging findings (e.g., intracranial hemorrhage on CT) may or may not be critical results depending on the clinical presentation of the patient. Therefore, automated detection and communication of critical imaging test results from textual radiology reports is a challenging task [5, 8, 33, 34].

In the 4 years since ANCR implementation, documented timely closed-loop communication of critical results increased from 90.7% in 2010 to 97.9% in 2013 for ANCR-communicated test results. Considering that we currently have more than 400,000 radiology reports annually, resulting in approximately 40,000 reports with critical results, this 7.2% improvement represents a large number of reports and thus patients with critical results (approximately two patients each day).

We acknowledge several limitations of this study. This study used a pre-post design rather than a randomized clinical trial. It carries the limitations inherent in such a design, primarily posed by any secular trends affecting adoption of the policy. However, providers (both radiologists and ordering providers) would be exposed to ANCR in various settings (e.g., outpatient, inpatient), and it would be difficult to randomize to a control group without exposure. In addition, it may not be possible to generalize our results to other institutions with different practice settings and policies. However, the baseline adherence to policy at our institution was 91.3%. Thus, institutions with a lower baseline performance would potentially have greater benefit from such a notification system. We did not evaluate timely performance of follow-up actions required to manage critical results, which would evaluate the impact of ANCR on reducing harm to patients and increasing the quality of care.

Our primary outcome measure was timely closed-loop communication of critical results between providers rather than patient harm from suboptimal communication of critical results, which is the ultimate patient safety goal but is an outcome beyond the scope of our study. Finally, we did not evaluate the direct role of patients in receiving the alerts, a role that may be an important element in improving communication hand-offs among caregivers, particularly in transitions of care, such as after discharge from the emergency department [35–37].

Conclusion

We documented adoption of an automated alert notification system for critical imaging results embedded in clinical workflow at our institution. Implementation of this system was made possible through substantial institutional and executive support for the creation, monitoring, and enforcement of a policy for communication of critical test results and the use of secure noninterruptive communication tools for notification and acknowledgment of critical results using both tethered and mobile devices such as smart phones and tablets. Adoption of the system was associated with a significant increase in timely closed-loop communication of critical results as defined by our institution's policy and consistent with a national Joint Commission patient safety goal. Future studies are needed to assess whether improved timely closed-loop communication of critical imaging results reduces patient harm.