Evaluating alert fatigue over time to EHR-based clinical trial alerts: findings from a randomized controlled study

Abstract

Objective

Inadequate participant recruitment is a major problem facing clinical research. Recent studies have demonstrated that electronic health record (EHR)-based, point-of-care, clinical trial alerts (CTA) can improve participant recruitment to certain clinical research studies. Despite their promise, much remains to be learned about the use of CTAs. Our objective was to study whether repeated exposure to such alerts leads to declining user responsiveness and to characterize its extent if present to better inform future CTA deployments.

Methods

During a 36-week study period, we systematically documented the response patterns of 178 physician users randomized to receive CTAs for an ongoing clinical trial. Data were collected on: (1) response rates to the CTA; and (2) referral rates per physician, per time unit. Variables of interest were offset by the log of the total number of alerts received by that physician during that time period, in a Poisson regression.

Results

Response rates demonstrated a significant downward trend across time, with response rates decreasing by 2.7% for each advancing time period, significantly different from zero (flat) (p<0.0001). Even after 36 weeks, response rates remained in the 30%–40% range. Subgroup analyses revealed differences between community-based versus university-based physicians (p=0.0489).

Discussion

CTA responsiveness declined gradually over prolonged exposure, although it remained reasonably high even after 36 weeks of exposure. There were also notable differences between community-based versus university-based users.

Conclusions

These findings add to the limited literature on this form of EHR-based alert fatigue and should help inform future tailoring, deployment, and further study of CTAs.

Background and significance

Clinical trials are essential to the advancement of medicine, and research participant recruitment is critical to successful trial conduct. Unfortunately, difficulties achieving recruitment goals are common, and failure to meet such goals can impede the development and evaluation of new medical therapies.^{1 2} It is well recognized that physicians often play a vital role in the recruitment of participants for certain trials. However, barriers including time constraints, unfamiliarity with available trials, and difficulty referring patients to trials, often make it challenging to recruit during routine practice.^3–6 Consequently, most clinicians do not engage in traditional recruitment activities and recruitment rates suffer.^{4 6}

The increasing availability of electronic health records (EHRs) presents an opportunity to address the issue of inadequate recruitment for clinical trials by leveraging the information and decision support resources often built into such systems. Indeed, recent studies of EHR-based, point-of-care, clinical trial alerts (CTA) have demonstrated they have the potential to improve recruitment rates when applied to clinical trials.^7–9 Despite their promise and the fact that they have been well tolerated in recent studies, CTAs like any point-of-care alert do have the potential for misuse, and further study is needed to better understand and inform their appropriate, widespread use.¹⁰ One important but poorly understood aspect relates to the performance characteristics of such alerts, particularly the issue of clinician responsiveness to alerts over time, and the implications of such phenomena on alert design and deployment decisions.

It is well recognized that when clinicians are exposed to too many clinical decision support (CDS) alerts they may eventually stop responding to them. This phenomenon is often called alert fatigue.^{11 12} While definitions vary and empirical evidence as to its cause is limited, alert fatigue is generally thought to result from one or more distinct but closely related factors. One such factor is declining clinician responsiveness to alerts as the number of simultaneous alerts increases. This is thought to be related to issues such as alert irrelevance and cognitive overload, and has also been referred to as ‘alert overload.’^{13 14} A second factor relates to declining clinician responsiveness to a particular type of alert as the clinician is repeatedly exposed to that alert over a period of time, gradually becoming ‘fatigued’ or desensitized to it.¹⁵ Few studies have explored this latter issue of ‘fatigue’ due to repeated exposure to alerts over time,¹⁶ and it is this latter aspect of alert fatigue that motivated and was the focus of the current study.

Although the purpose of CTAs is to provide decision support for trial recruitment rather than for clinical care, we hypothesized that this phenomenon of alert fatigue over time would be a factor in the usage patterns of CTAs, as also noted for CDS alerts, and would present itself as a gradual reduction in response rates over time. As the nature and extent of this issue and the overall performance characteristics of CTAs are not fully understood, the current study was performed among physician subjects as a planned part of a recently conducted randomized controlled intervention study of a CTA.¹⁷

Methods

We performed a cluster-randomized controlled study of a CTA intervention across three health system environments that share a common, commercial ambulatory EHR (GE Centricity EMR). The study of the CTA and the clinical trial to which it was applied were approved by our institutional review board. The associated clinical trial involving patients with insulin resistance after a recent stroke event was registered in Clinical Trials.gov (NCT00091949).

Subjects involved in this 36-week study of the CTA intervention in 2009 (the first, pre-cross-over phase of a larger study on the impact of this CTA intervention) included 178 physicians who were randomly divided into equal groups within their specialties (ie, neurologists, n=26; family medicine physicians, n=35; general internists, n=46; internal medicine-pediatrics specialists, n=8; and internal medicine house staff, n=63). All neurologists were university-based practitioners, while the other generalist physicians practiced either in university-based or community-based settings. Prior to CTA activation, all physicians were encouraged via traditional means (eg, discussion at staff meetings, email blasts, flyers) to recruit patients to the trial.

Upon CTA activation, intervention physicians seeing eligible patients were presented with on-screen CTAs that suggested they consider and discuss trial recruitment with the patient, and click an on-screen button to send a secure referral message to the trial coordinator if appropriate (figure 1). The physicians involved in this study did not receive any incentives for their participation in these recruitment efforts. It is worth noting that the results of the phase of the underlying CTA intervention study during which this analysis of responses took place revealed a significant 20-fold increase in referrals (p<0.0002) and a ninefold increase in enrollments (p<0.006).¹⁷ The design of the CTA intervention itself has also previously been reported and involved a minimal amount of novel programming as the built-in tools and resources of the particular EHR system were used.¹⁸ This was a similar approach to that employed with another popular EHR system upon which we have previously reported.¹⁹

Figure 1.

Screen shot of the clinical trial alert used in the randomized controlled trial that was the basis for the current study.

Additional data were also collected during the same period via direct query of the EHR-fed enterprise data warehouse for the two types of events in which we were interested: (1) responses that indicated interaction with the CTA (ie, taking action by responding to at least one of the questions posed in the alert); and (2) flags (referrals) sent to the study coordinator via positive responses to both questions and then processing of the CTA. In addition, data on the total number of alerts triggered were extracted as the denominator value for determination of a ratio of (1) response rate and (2) referral rate per physician, per time unit.

In our analyses, the dependent variables ‘responses’ and ‘referrals’ were offset by the log of the total number of alerts received by that physician during the same period, in a Poisson regression. Correlations across time were modeled as a ‘spatial power’ process (equivalent to an autoregressive 1 process). While CTAs could have triggered more than once for a given patient–physician pair if the initial CTA was ignored and if the patient returned during the study period and was still eligible for the alert, any such subsequent alerts were not included in our analyses. We used 2-week time periods for our analyses, for a total of 18 time periods over this 36-week study. Results are presented in numeric and graphical form along with the relevant p values to indicate the significance of the findings.

Results

During the 36-week CTA exposure period, 915 total alerts were triggered for 178 physicians in the intervention arm of the associated randomized controlled study of the CTA. Eight alerts were discarded because they represented second alerts for patients who had earlier received an alert during the study period that was not responded to, leaving 907 eligible alerts for analysis.

During the initial time period, the response rate to CTAs was about 50% among all users but dropped significantly over time by 2.7% for each advancing 2-week period, and that trend was significantly different from zero (flat) (p<0.0001; figure 2). Notably, there was still a 35% response rate at the 36th week of exposure.

Figure 2.

Physician response rates to clinical trial alerts (CTAs) are plotted at 2-week intervals over the 36-week study. The solid line tracks response rates at each time point. The dashed line represents the linear regression line through each time point. Response rates declined at a rate of 2.7% per 2-week time period (p<0.0001).

Subgroup analyses of response rate changes over time reveal that there are no significant differences between subspecialists (neurologists in this case) and generalists, with both groups trending downward to a similar extent, whether or not community generalists are included in the analyses. However, there is a greater response rate drop-off among all community-based providers compared with all university-based physicians (p=0.0489). Further, that drop-off is stronger when university-based subspecialists (neurologists) are removed and the analyses are restricted to generalists in both types of settings (p=0.0146).

Referrals rates started at about 33% and, although they fluctuated, declined to about 9% by the end of the study period. Of note, generalists made fewer referrals than subspecialists (p<0.0001), and community-based physicians made fewer referrals than university-based physicians (p=0.006). The decline in referrals rates over time was more pronounced than the decline in response rates noted above. Specifically, there was a significant 4.9% decrease in referral rates per time period (p=0.0294) (figure 3).

Figure 3.

Physician-generated referral rates using clinical trial alerts (CTAs) are plotted at 2-week intervals over the 36-week study. The solid line tracks referrals rates at each time point. The dashed line represents the linear regression line through each time point. Referral rates declined at a rate of 4.9% per 2-week time period (p=0.0294).

While absolute referral rates differed between groups, subgroup analyses of physician-generated CTA referral rates revealed no significant differences in rate declines between: (1) subspecialists versus generalists; (2) community-based versus university-based physicians; and (3) community-based versus university-based generalists only (ie, excluding neurologists).

Discussion

The use of EHR-based CTAs has been demonstrated to increase participant recruitment rates to clinical trials, and is a promising approach for overcoming the major problem of inadequate and slow participant recruitment.^{7–9 17} Because such an approach will necessarily be employed in the context of complex and varied clinical care environments, information on the performance characteristics and response patterns among different groups of potential end-users is needed to inform its application and use.

These findings add to our understanding of how such alerts for clinical trials operate in real-world implementations by demonstrating empirically how and to what extent the rates of responses to such alerts decline across a variety of settings and end-users. Notably, they reveal, as hypothesized, that responses to point-of-care CTAs decline over time, although not as severely as anticipated, at least with regard to response rates.

Indeed, overall response rates to this series of alerts was initially high at 50% and remained reasonably high at 35% even after 36 months of exposure, compared to CDS alerts which tend to have 4%–51% response rates.¹² While the fall in response rate suggests alert fatigue over time, the fact that a substantial proportion of the alerts were still being responded to at 36 weeks suggests that such a duration of use may still provide benefit. However, the finding that referral rates declined more quickly and more precipitously over time than response rates suggests there might be a point after which use of a CTA might not be worth even the minimal disruption they cause.¹⁰ In addition, the differences seen among community-based versus university-based physicians suggest that future CTA deployments should be tailored to a particular setting (ie, shorter in community-based settings and longer in university-based settings) in order to maximize benefit while avoiding excess fatigue. Additionally, as noted with some CDS alerts, tailoring of the alert's operating characteristics (eg, increasing specificity such that they trigger less often) might also affect response patterns and ultimately effectiveness, particularly in practice settings or specialties where response rates fall more rapidly.^{13 20}

While the design of this study does not allow for definitive determination of the reasons for the declines noted, the difference between the response rate decline (2.7% per time period) and the referral rate decline (4.9% per time period) might reflect the fact that the act of CTA referral requires more effort than a simple response, and therefore causes more fatigability over time. However, this difference could also suggest the presence of other factors such as the possibility that declines in referrals reflect a drop in the available pool of eligible or interested candidates rather than alert fatigue. However, the population of potentially eligible participants (ie, patients with a recent stroke) remained relatively constant during the study, making this less likely. Nevertheless, it is probable that the reasons for the declines were multi-factorial, reflecting the combined influence of alert fatigue and other factors. Additional studies, including qualitative studies to assess physician-user perceptions, are ongoing and should help clarify other reasons for the declines noted.

Comparison of physician response patterns over time and apparent alert fatigue with those when similar CDS approaches are employed for clinical use would be useful. Unfortunately, data on such changes over time in CDS response rates appear to be lacking in the published literature. As noted above, plentiful circumstantial evidence of this aspect of alert fatigue in many studies reveals less than ideal average rates of response to CDS interventions,^{11 12} with some studies commenting on the common behavior of overriding alerts,²⁰ and still others addressing changes that can increase average response rates by improving the usability or appropriateness of alerts.¹³ However, although this form of alert fatigue over time undoubtedly exists, there has been surprisingly little empirical evidence of it, or data to characterize the nature of the phenomenon. Our study appears to be among the first to empirically demonstrate this aspect of alert fatigue by tracking changes in clinician response to alerts over time. Therefore, we believe it has implications beyond recruitment using CTAs, and that such an approach to measuring responses over time can help advance understanding of alert fatigue in general. We also believe that the methodology employed here could be used to evaluate and refine the design and application of decision support alerts in the future.

Although the randomized study design and multi-user, multi-environment setting strengthen these findings and advance our understanding of CTA usage, this study has some limitations. These findings were derived from a single study of CTAs employed in a single trial of patients with recent stroke. Whether these findings would differ if the CTA were applied to another type of trial or in different settings remains to be determined. Also, while the CTA approach has been demonstrated to be effective using multiple EHR platforms,^{7–9 17} this study employed a single EHR and these findings might differ with the use of another EHR. Furthermore, this alert was employed in a setting where other alerts were rarely triggered. Another factor possibly impacting response rates over time is the threshold setting (ie, sensitivity vs specificity) for a given alert. Whether the findings of this study would differ if there were multiple or more frequent alerts is not known but is possible given that multiple simultaneous alerts are a commonly cited factor leading to alert fatigue as noted above, and should be studied.

Conclusion

Physician response rates to CTAs started and remained relatively high even after a period of use, although they gradually but significantly declined over time. While overall response rates were lower among generalists than subspecialists, the rates of decline in CTA responses and referrals varied significantly only between university-based versus community-based physicians, and not between generalists versus subspecialists. These data also suggest that alert fatigue over time is likely a factor that must be taken into account when CTAs are employed.

While it is currently unclear how the nature and degree of alert fatigue for CTAs compares to that of other types of CDS alerts, this study has implications for the implementation and management of such alerts. The methodology used here also appears to have implications for studies into the relative impact of alert fatigue across a range of decision support alert interventions. Overall, these findings offer much-needed empirical data about the performance characteristics of CTAs, data that should help inform the tailoring and application of CTAs in real-world environments in order to overcome the major research challenge of improving and accelerating participant recruitment.