Impact of a national QI programme on reducing electronic health record notifications to clinicians

Abstract

Background

Emerging evidence suggests electronic health record (EHR)-related information overload is a risk to patient safety. In the US Department of Veterans Affairs (VA), EHR-based ‘inbox’ notifications originally intended for communicating important clinical information are now cited by 70% of primary care practitioners (PCPs) to be of unmanageable volume. We evaluated the impact of a national, multicomponent, quality improvement (QI) programme to reduce low-value EHR notifications.

Methods

The programme involved three steps: (1) accessing daily PCP notification load data at all 148 facilities operated nationally by the VA; (2) standardising and restricting mandatory notification types at all facilities to a recommended list; and (3) hands-on training for all PCPs on customising and processing notifications more effectively. Designated leaders at each of VA’s 18 regional networks led programme implementation using a nationally developed toolkit. Each network supervised technical requirements and data collection, ensuring consistency. Coaching calls and emails allowed the national team to address implementation challenges and monitor effects. We analysed notification load and mandatory notifications preintervention (March 2017) and immediately postintervention (June–July 2017) to assess programme impact.

Results

Median number of mandatory notification types at each facility decreased significantly from 15 (IQR: 13–19) to 10 (IQR: 10–11) preintervention to postintervention, respectively (P<0.001). Mean daily notifications per PCP decreased significantly from 128 (SEM=4) to 116 (SEM=4; P<0.001). Heterogeneity in implementation across sites led to differences in observed programme impact, including potentially beneficial carryover effects.

Conclusions

Based on prior estimates on time to process notifications, a national QI programme potentially saved 1.5 hours per week per PCP to enable higher value work. The number of daily notifications remained high, suggesting the need for additional multifaceted interventions and protected clinical time to help manage them. Nevertheless, our project suggests feasibility of using large-scale ‘de-implementation’ interventions to reduce unintended safety or efficiency consequences of well-intended electronic communication systems.

Background

Inefficiencies from electronic health record (EHR) use have led to burnout and dissatisfaction among clinicians.^1–4 In the Department of Veterans Affairs (VA), EHR-based ‘inbox’ notifications originally intended for communicating important clinical information are now cited by 70% of primary care practitioners (PCPs) to be of unmanageable volume.⁵ Alarmingly, 30% of PCPs also report missing abnormal test results due to information overload from notifications.⁶ There are >50 types of notifications (eg, test results, referral-related information, medication refills, messages and orders).⁷ PCPs may receive low-value or redundant notifications (eg, multiple messages for the same laboratory panel) depending on how they customise their notification settings.⁸

Additionally, each VA facility can choose how many and which types of notifications should be ‘mandatory’ for clinicians to receive. The VA health system uses VistA, an internally developed EHR platform that integrates 200 applications, with most facilities using a unique VistA version.⁹ Much of VistA’s software was developed by programmers at the local level, allowing for facility flexibility yet also providing interoperability for clinical and administrative data sharing across the VA.¹⁰ Types of clinical services available, workflows, level of care acuity and clinical leadership structure vary between facilities.¹¹ Thus, while there is a standard set of notification types in VistA, how they are used individually has always been locally determined, resulting in high variability in numbers and types of notifications received.¹² No nationally developed guidance existed on how to measure or reduce them.

Solutions to decrease notification burden could potentially include restricting ‘mandatory’ notification types to those deemed high-priority and training PCPs¹³ to customise their EHR settings to filter out certain notifications.¹⁴ Based on this rationale, the VA implemented a national, multicomponent, quality improvement (QI) programme to reduce low-value EHR notifications. We evaluated the impact of this intervention on PCP notifications.

Methods

The programme involved three steps: (1) accessing daily PCP notification load data at all 148 facilities operated nationally by the VA; (2) standardising and restricting mandatory notification types at all facilities to a recommended list of 10 derived from evidence and expert consensus, with flexibility to include up to two additional types; and (3) hands-on training for all PCPs on customising (eg, turning a notification type off) and processing notifications more effectively. We pilot-tested the standardising and training aspects of the programme in two Veterans Integrated Service Networks (VISNs), which comprised 16 facilities before scaling up the entire programme for national implementation. The recommended notification list included six notifications that were determined to be mandatory based on prior review of evidence and consensus at the national level for their importance in ‘closing the loop’ to ensure test result follow-up.^{15 16} Four additional notifications were recommended to be considered as mandatory based on expert consensus that emerged after pilot-testing the programme. These 10 notification types and their rationale are described in Table 1. Additionally, the facilities were given flexibility to include up to two more types for a maximum of 12 mandatory notification types.

Table 1.

List of mandatory and recommended notifications and rationale for inclusion

Notification type	Rationale for inclusion
Mandatory notification types
Abnormal imaging result, needs attention	Notification often communicates new unexpected findings.
Anatomical pathology results	Previous national Veterans Affairs recommendation
Critical lab result: action	Many lab results that are critical, emergent or urgent are communicated this way.
Imaging request cancel/hold	Used to communicate information when important tests are not done or completed
Imaging results amended	Amended findings might lead to new treatment decisions or changes in management or diagnosis.
Orderer-flagged results	Clinicians commonly ‘flag’ tests when initially ordering them for themselves or to other clinicians to ensure they receive a message when tests results are available. The notification must be turned on to receive these notifications.
Additionally recommended notifications to be considered mandatory
Consult request cancelled/hold	Commonly used communication method for status of referrals and consultations
Consult/request resolution	Commonly used communication method for information exchange between primary care practitioners and specialists
Order requires electronic signature	Used widely to remind clinicians to sign pending orders
Outpatient non-renewable prescription renewal	Notification for medications that can only be ordered by a clinician, for example, for controlled substances

The Chief Medical Officer, Chief Health Informatics Officer (CHIO) or a primary care leader in each of VA’s 18 regional VISNs led the programme implementation using a nationally-developed toolkit containing (1) recommendations for mandatory notifications; (2) computer code to access local notification data; (3) multimedia materials to train PCPs; and (4) a sample implementation timeline. There was some heterogeneity in methods to implement the programme because certain VISN programme leads let their facility leadership determine their own list of mandatory notifications often through consensus, whereas others used a top-down approach to decide that list for the entire VISN.

While the QI programme had many features of a ‘top-down’ approach, several decisions were entrusted to the PCPs to ensure patient safety.¹⁷ For example, turning notifications off involved several steps. While a notification was made non-mandatory at the facility level through a technical switch, PCPs had the flexibility to adjust their notification settings themselves and were prompted to go into their settings and decide if they wanted to leave the notification on (ie, the previous default) or turn it off (which they could not do before). This enabled PCPs to make the final choice according to their unique workflows and preferences for receiving clinical information, and to implement a new communication pathway before a specific notification was turned off.

Programme implementation was synchronised nationally (March–June 2017) using VA’s robust hierarchical infrastructure and leadership support to carry out operational changes. Each VISN programme lead supervised technical requirements and data collection, ensuring consistency. The national executive team, which consisted of a physician executive sponsor and two VISN CHIOs, closely monitored programme data and led frequent coaching calls with the VISN programme leads to enable sharing of best practices to overcome implementation challenges. These coaching calls, in addition to daily email contact, allowed the executive team to address operational challenges and to monitor the field for potential benefits and unintended consequences of the programme. The team also frequently queried for any feedback on outcomes such as PCP satisfaction and patient safety related to any aspect of implementation. While a pre–post questionnaire to assess educational impact was developed and used during pilot testing, it was made optional during implementation due to workload concerns. We analysed baseline notification load and mandatory notifications preimplementation (January–March 2017) and immediately postintervention (June–July 2017) to assess the impact of the QI Programme.

Results

The median number of mandatory notification types at each facility decreased significantly from 15 (IQR: 13–19) to 10 (IQR: 10–11) preintervention to postintervention, respectively (P<0.001; figure 1). Similarly, the mean number of daily notifications per PCP decreased significantly by 12, from 128 (SEM=4) to 116 (SEM=4; P<0.001) (figure 2), and the median of the change in mean number of daily notifications per PCP decreased by 10, from 122 to 112 (P<0.001). There was significant dose–response relationship between the number of notifications PCPs received preintervention and the change in those notifications preintervention to postintervention (Pearson’s r=0.57, P<0.001). Reduction in the number of mandatory notification types set at the facility level did not significantly impact the change in PCP notification load (P=0.64).

Figure 1.

Number of notification types set as mandatory at each facility by Veterans Integrated Service Network (VISN). VA, Veterans Affairs.

Figure 2.

Change in mean notifications per provider per day at each facility from preintervention to postintervention. VA, Veterans Affairs.

Outcomes of training

Across all 148 VA facilities, 97% (n=8411) of PCPs received training by the end of the project period. Several (11 of 18) VISNs initially reported PCP training rates <80% and were contacted by the executive team to develop new plans to increase training rates to >80%. While no VISN chose to administer the educational impact questionnaire, additional communication with these VISNs revealed that the rigour of measuring how PCPs were trained was quite variable. Training rates improved to target levels when VISNs with low initial training rates implemented more rigorous methods to measure training. For example, one VISN monitored training compliance through its QI programme tracking software (ExtraView), which triggered emails to remind all PCPs to complete training and provided leadership with data to address any residual training issues. Another VISN created a mandatory online course in VA’s employee learning system, including a training video.

Additionally, feedback from programme implementation leads during the pilot phase cited the strategy of active face-to-face training with coaching to make changes in real time as most effective to empower PCPs to customise their EHR settings. Not surprisingly, during implementation, VISN programme leads found that PCPs using these active methods reported increased satisfaction with the ability to make choices on notifications and with customising their workflow.

Additional assessment of programme impact

We assessed programme impact by querying facilities that experienced changes in notification burden that were greater than expected compared with results from pilot testing. We found that these sites made additional configuration changes or implemented additional QI projects relating to notifications, which likely amplified programme impact. One VISN reported uncovering new avenues to increase efficiency in electronic communications, particularly between call centres and PCP care teams. This was because their local measurement efforts revealed that the call centres generated a substantial number of notifications that PCPs deemed to be of low-value, such as patient no-show messages that may have been better triaged to other members of the care team.

While we did not specifically find unintended consequences, programme implementation led to other ‘carryover’¹⁸ effects (ie, where exposure to one condition ‘carries over’ to another). One VISN reported that they undertook a review of all radiology-related notifications and unexpectedly found a backlog of open radiology study requests still in a ‘pending’ status. This prompted process mapping and a more streamlined process for managing radiology-related notifications across the VISN. Another VISN reported developing an alternative pathway for important communication between consultants and PCPs and eliminating certain low-value notifications.

Discussion

Implementing a national, multicomponent, QI programme involving standardisation, measurement and training successfully reduced EHR notifications to PCPs by 9.4%. Based on prior estimates of 85 seconds to process each notification,¹⁷ this programme potentially saved 1.5 hours per week per PCP, which could be used for higher value work. Programme impact appeared to be achieved by reducing certain types, versus just the sheer number of mandatory notifications, underscoring the complexity of addressing notification burden. Given a ‘dose-response’ effect, PCPs with the highest notification loads likely received the greatest benefit.

Similar to results from other large-scale QI programmes,¹⁹ the heterogeneity in implementation across 148 VA sites likely led to high variability in observed programme impact. Over 50% of VISNs found it hard to deliver training within the initial project timeline. Several programme leads reported lack of time and resources to support training efforts because they directly competed with providing patient care. Leadership engagement was key and some VISNs directed more resources to overcome these barriers, including providing more active methods of training (eg, additional in-person training sessions) and more time.²⁰ These findings underscore the importance of investing resources in periodic refresher training to optimise EHR use.

We did not find a significant correlation between the reduction in number of mandatory notification types and PCP notification load. The mandatory notifications recommendations were based on their potential impact on safety, rather than baseline quantities of each particular notification type typically generated. There is a wide range in the daily number of notifications generated for PCPs depending on notification type.^{17 21} In addition to the VISN/facility-level carryover effect described earlier, it may be possible that the programme also had a carryover effect at the PCP level leading to a reduction in different types of notifications at different sites, not just the mandatory ones. For instance, feeling more empowered post-training, PCPs could have chosen to turn off certain additional notifications that they believed to be of lower value, beyond those determined as mandatory notifications. Conversely, new types of notifications may have been enabled by PCPs fearing missing important information and could explain why some facilities had higher notifications postintervention. While preimplementation pilots at test sites did not support this, subsequent interventions to reduce notifications need to address such potential consequences. This is understandable because clinicians experience a tension between wanting to see clinically relevant notifications and needing to keep notification volume manageable, which is challenging to navigate without specific coaching and encouragement by peers and supervisors. Competing QI initiatives may also have led to increased notifications, such as through sending PCPs information related to newly discovered unresolved issues. In some other facilities, there were no significant changes, which could result from less rigorous training (eg, disseminating educational materials via email vs hands-on coaching).

Despite some benefits, the average number of notifications that a PCP received per day (which decreased to 116) remains too high to manage. Prior work suggests more protected time be given to PCPs to handle this workload.^{1 5 22} The scale of national implementation also limited our understanding of the impact of the programme beyond the sheer notification volume. While most VISNs opted to restrict mandatory notifications to the proposed 10, because of the complexity of abstracting data on specific types of notifications, we were unable to capture more granular data regarding which PCPs decided to make changes to their notification settings, or which types of notifications they decided to turn off. We also were not able to correlate how and which intervention component led to changes in the types and quantities of messages received. The programme also had local carryover effects, which we were not able to measure systematically. These pressing questions merit further research to fully understand the complexity of use of EHR-based communication systems.²³

In conclusion, we found that a national, multicomponent, QI programme involving standardisation, measurement and training successfully reduced EHR notifications to PCPs by nearly 10%. This programme potentially saved 1.5 hours per week per PCP, which could enable them to spend on higher value work. Our findings highlight the feasibility of using large-scale ‘de-implementation’ interventions to reduce unintended safety or efficiency consequences of well-intended electronic communication systems.