Leading a Horse to Water: Using Automated Reminders to Increase Use of Online Decision Support

Abstract

Online knowledge resources are generally underused despite their potentially beneficial effect on patient care. We are interested in ways to increase use of context-specific links, called infobuttons, from clinical information systems to such resources. We used log files to examine the impact of classroom instruction on infobutton use to pinpoint a particular context where infobuttons appeared to be underused by particular users. We sent users e-mail suggestions about infobuttons and reviewed log files to examine the impact of the messages. Log files showed that usage by medical students and housestaff following classroom instruction was initially high but then tapered off. In particular, we found that infobuttons were infrequently used in an outpatient, but not inpatient, medication application. We sent 552 suggestion e-mails to 371 users after log files showed that they used other resources in that context. Fifty-two of these users used infobuttons within 31 days of receiving the e-mail, which represents 25.9% of recipients who used any resource during the same period. Users responded best when the e-mail was sent within 5 days of using the resource. Traditional instruction had a limited sustained effect on improving the use of infobuttons. An automated method for sending targeted suggestions complemented the traditional approach.

Introduction

Despite the generally accepted belief that the use of evidence-based knowledge resources will improve clinician decision-making and patient outcomes, information resources are routinely underutilized during the course of clinical care. For example, Ely and colleagues found that physicians in clinical practice sought answers to questions only 45% of the time and found answers only 34% of the time; when they did seek answers, they used computer resources only 18% of the time.¹

Infobuttons are context-specific links that can be inserted into clinical information systems to provide links to resources that are likely to be relevant to the users of those systems.² Several studies have shown that, when they are used, infobuttons and similar context-specific links are found to be useful and have a beneficial impact on patient care.^2,3,4,5

Infobuttons are used less than their potential impact would seem to warrant. Rosenbloom and colleagues found that their PC-POETS application (which provides infobutton-like access to online resources) was used only 3.8% of the time at Vanderbilt Medical Center.³ Del Fiol and colleagues found that their University of Utah Infobutton Manager was used at Intermountain Healthcare hospitals less than twice a month by those users who used it at all.⁴ Maviglia and colleagues at Harvard University found a similar rate of usage of their KnowledgeLink system at Partners Health Care System,⁵ as have we with the Columbia University Infobutton Manager at New York Presbyterian Hospital (NYPH).⁶

One strategy for increasing use of infobuttons, or any other resource, is to include them in educational activities that train clinicians to use appropriate information resources in the course of patient care. Indeed, this is the method primarily used at LDS Hospital (G. Del Fiol, personal communication) and Partners Health Care (S. Maviglia, personal communication). Training on the use of PC-POETS is similar at Vanderbilt and then reinforced at biweekly “pizza meetings” and on rounds with members of the development team (T. Rosenbloom, personal communication). Training on the Columbia Infobutton Manager is included in classroom-based sessions for housestaff and medical students prior to their use of clinical information systems at NYPH.

In this paper, we report the impact of NYPH training on infobutton use. We instituted a novel, automated approach to educating users about infobuttons, namely, e-mail reminders sent to users who had recently used a knowledge resource other than infobuttons while using a clinical information system, and report on the impact of that intervention.

Methods

Traditional Educational Interventions

Third-year medical students and new housestaff are provided with orientation, through lectures and demonstration, to NYPH clinical systems prior to starting work in clinical areas. Demonstration of infobuttons is included for both groups. Medical students are also given practical exercises that involve using the systems, including infobuttons.

Impact of Traditional Interventions

The clinical systems at NYPH record in log files all user activities, including use of infobuttons and other online knowledge resources.⁶ We examined the log files before and after the medical student and housestaff orientations in 2006 and 2007 to see if there was any correlation between the training and the use of infobuttons, and the duration of that correlation. We also examined the log files to identify situations where other online information resources were used.

Automated Reminders

Based on patterns detected in the log file analysis, we identified a particular clinical application context in which infobuttons seemed to be underutilized when compared with other resources. We then used log files to identify users who (a) had used another resource in that context and (b) had not used infobuttons in that context within the previous two months. An automated system was created to periodically review the log files to identify these users and (as soon as practical) send an e-mail reminder that infobuttons were available in that context. The system ran daily, for intermittent periods, from February 2006 to February 2007. A sample message is shown in Figure 1.

Figure 1.

Sample e-mail reminder

Impact of Automated Reminders

We reviewed the log files to determine which e-mail recipients subsequently used infobuttons (IB) and/or other health resources (HR) in the specified application context (outpatient medication ordering and review) and the time duration between sending the e-mail and observing such use.

Results

Traditional Educational Interventions

Approximately 155 medical students and 400 house staff received orientation to the NYPH clinical systems, through live demonstrations, during sessions in June 2006, with similar numbers in June 2007. Medical students were also given hands-on exercises to allow them to use the system features, including infobuttons.

Impact of Traditional Interventions

Log files showed that use of IBs, but not the use of other HRs, increased in June 2006 and June 2007, with subsequent tapering to baseline levels (see Figure 2). When we considered only housestaff and students who used either IBs or HRs for the first time in June or July of 2006 or 2007, we found 490 such new users. As shown in Figure 3, their use of IBs peaked in July of their first year and tapered thereafter.

Figure 2.

Monthly use of infobuttons (IB; grey) and other health resources (HR; black), by month for 2006 (dotted lines) and 2007 (solid lines). HR use has been fairly constant; fluctuations correspond to month length; increased IB use in June and July corresponds to training periods for new housestaff and students.

Figure 3.

Use of Infobuttons by 255 housestaff (black) and 235 students (grey) in 2006 (dotted lines) and 2007 (solid lines) who were first time users in the respective year.

When we examined particular clinical application contexts, we noted that while infobutton use was relatively high while reviewing inpatient medication orders (when compared to other health resources), the reverse relationship was seen when users were writing or reviewing outpatient medication orders (see Figure 4). We therefore chose this latter application context as the focus for our intervention.

Figure 4.

Monthly use of infobuttons (IB; grey) and other health resources (HR; black), in inpatient (IP; dotted lines) medication and outpatient (OP; solid lines) medication contexts, by month (2006 and 2007 combined). For IP users (dotted lines), IB is consistently higher than HR, while the reverse is true for OP users (solid lines).

Automated Reminders

We generated 552 messages to 371 users (80 attendings, 189 housestaff, 29 nurses, 24 students, 49 other/unknown). Because we only considered IBs in the prior two months, 173 users got two messages and one got three messages. There was an average of 89.9 days between duplicates. Due to the lag time between user activity and availability of log files, 351 messages (63.6%) were sent two days after they used an outpatient medication HR, while 74 (13.4%) were sent in less than two days, 37 (6.7%) were sent within three to seven days, and 90 (16.3%) were sent later.

Impact of Automated Reminders

Overall, 111 (20.1%) of recipients eventually used an IB while in the outpatient medication context. However, only 52 (9.4%; including 26 housestaff and 18 attendings) did so within one month of receiving the message. We refer to these users as the early responders.

Not all users of the system will use it every day (due to changing clinical duties, for example), so we cannot know who had an opportunity to use IBs. However, if we consider only those who used either HRs or IBs in the outpatient context within one month of receiving their message, we find a new denominator of 201. Among these users, IB use within one month was 25.9%. By way of comparison, we identified 525 users who used HRs before using IBs in the outpatient medication context in 2006 and 2007, but did not receive e-mail reminders, and identified the time until they first used an IB (if at all) in that context. Figure 5 shows cumulative new use of IBs during the first month following message receipt, compared to use of IBs by first-time health resource users who did not receive an e-mail. The divergence between the groups reached a significant level (P<0.05 by the Yates corrected Chi-square method) at Day 4 and remained significant through day 31 (P<0.001).

Figure 5.

Cumulative first use, by day, of infobuttons among 552 recipients of reminder messages (grey) versus 525 first-time users of health resources (black).

When we examined the effect of the delay between HR use and message transmission on the success rate of new IB use, we found a strong inverse association, with 48 of the early responders responding to messages received within two days of previous HR use. There was no early response to the 361 messages sent more than five days after HR use.

Early response to an e-mail also correlated with recurrent use of IBs. Nine early responders used IBs more than seven times in the ensuing months (range 8 to 97 times) while only one of the 500 other recipients went on to use IBs more than seven times.

Discussion

Although evidenced-based knowledge is recognized as a valuable adjunct to good health care decision making, attempts to get practitioners to use it has been difficult. Traditional education efforts have shown poor results, including those related to infobuttons, as demonstrated at other institutions. Our own data clearly show that students and housestaff will use infobuttons briefly after formal training, but their usage quickly decays.

We could have chosen to simply send blanket e-mails to all WebCIS users, hoping to reach those most likely to be receptive to our particular suggested use. Most recipients of such messages will not have had a recent relevant information-need and so, not only will the messages be useless, they may inure users to later, relevant messages.

Instead, our method targeted those users who were recently in a situation where they demonstrated a need for information that could be addressed by an IB, but did not choose to use an IB. This method is somewhat similar to context-specific suggestions used by online merchants (“other people who bought this also bought…”) since we attempt to target users who will be most receptive to the message, rather than randomly inundate all users.

Despite being targeted, the messages were deliberately impersonal. While a message that said, “We noticed that you used a health resource other than infobuttons yesterday…” might have garnered more attention, we were concerned about the potentially negative reaction to such a “Big Brother” message. Instead, when users did respond to the message (12 responses to 522 messages), they were invariably positive, usually expressing thanks for infobuttons.

We used a quasi-experimental approach in this study for a number of practical reasons. Not the least of these was that formal enrollment of subjects in a randomized trial might have lead to inadvertent increased awareness of infobuttons, possibly masking the impact of the intervention. The disadvantage of our approach is that our ability to interpret the impact of the intervention is limited, since we used a convenience sample of HR users who did not receive our e-mails as a comparison group. We cannot, therefore, ascribe the difference between the two groups to our intervention. However, we can state that there was a rapid and significant association between our intervention and the adoption of IBs (as shown in Figure 5). The strong temporal association supports, although it does not prove, our hypothesis that such e-mails would have a positive impact..

We have reason to be encouraged by our results. First, despite extensive, detailed log file data, we actually know very little about the users. For example, after having sought information while using the outpatient medication application, we do not know when, if at all, these users will have the need or opportunity to do so again. While some will use the application daily (at least during weekdays), some may use it weekly or may rotate out of an outpatient setting for months at a time, or may even leave the institution. When we adjust our estimate by only considering those users who show any information-seeking activity in the outpatient context, the response to the e-mails appears to be 25.9%. We believe that this is a respectable response for passive, spam-like e-mail, especially when we don’t even know which users read their e-mail.

Second, while the response to the messages seems slow – perhaps so slow that the influence of the messages may have worn off (especially after two weeks, as shown in Figure 5), there does appear to be a real impact – more than two-fold – when compared to HR users who did not receive messages (as also shown in Figure 5).

Third, the effectiveness of the messages for stimulating initial use of IBs appears to be related to the lag time between use of an HR and sending an e-mail. The experimental system developed in this project suffered several technical setbacks that caused intermittent delays in sending the messages. This proved to be useful from an investigative standpoint, since it created random delays that allowed us to observe the effect of those delays. However, a more reliable system should result in more early responders – especially if we can reduce the lag time attributable to the delay in obtaining daily log file extracts. An even more immediate approach, such as a system that monitors HR use in real time, could produce even more timely messages.

Fourth, while the impact of the messages was low overall, their impact was high on a small number of users. Seventeen percent of early responders went on to become frequent users, ranging from two to ten uses per month for two to ten months. For these users, the messages seemed to be the catalyst to increased usage. Improving the timeliness of messages should lead to an increased percentage of early responders which may in turn lead to a proportional increased number of frequent users.

Finally, the use of an automated e-mail reminder system is a fairly inexpensive approach – essentially cost-free once it is in place. This makes for a very favorable cost-benefit ratio, especially when compared to more resource-intense methods such as classroom teaching and “pizza rounds”.

Reminder systems have long been used as a form of clinical decision support. We have used reminders to remind users not about a decision they should make, but about a decision support resource they might use to help with their decisions. We thus consider our approach a kind of “meta-reminder” that sends relevant messages to recipients who may be in the best state of mind to receive them: those who have demonstrated their need to obtain relevant information in a specific clinical context. Our method appears to work and might be even more influential if messages could be sent in a more timely way. The approach complements the traditional classroom-based approach and could be applied to many systems and contexts where active alerts might not be possible, but where access to log files and e-mail addresses is available.

Conclusions

We have used log files to identify users who have demonstrated a context-specific need for information and sent e-mails to those users to educate them about an available option for future needs. This approach has a limited but definite impact on subsequent information seeking behavior for those users. A similar approach could be applied to a variety of educational interventions to affect clinician behavior.