Pediatricians’ Responses to Printed Clinical Reminders: Does Highlighting Prompts Improve Responsiveness?

Abstract

Objective

Physicians typically respond to roughly half of the clinical decision support prompts they receive. This study was designed to test the hypothesis that selectively highlighting prompts in yellow would improve physicians’ responsiveness.

Study Design

conducted a randomized controlled trial using the Child Health Improvement through Computer Automation clinical decision support system in four urban primary care pediatric clinics. Half of a set of electronic prompts of interest was highlighted in yellow when presented to physicians in two clinics. The other half of the prompts was highlighted when presented to physicians in the other two clinics. Analyses compared physician responsiveness to the two randomized sets of prompts: highlighted versus not highlighted. Additionally, several prompts deemed “high-priority” were highlighted during the entire study period in all clinics. Physician response rates to the high-priority highlighted prompts were compared to response rates for those prompts from the year before the study period, when they were not highlighted.

Results

Physicians did not respond to prompts that were highlighted at higher rates than prompts that were not highlighted (62% and 61% respectively, OR=1.056, p=0.259, ns). Similarly, physicians were no more likely to respond to high-priority prompts that were highlighted, compared to the year prior when the prompts were not highlighted (59% and 59%, respectively, χ²=0.067, p=0.796, ns).

Conclusions

Highlighting reminder prompts did not increase physicians’ responsiveness. We provide possible explanations as to why highlighting did not improve responsiveness and offer alternate strategies to evaluate for increasing physician responsiveness to prompts.

INTRODUCTION

As clinical decision support systems (CDSS) in health care have advanced, research has increasingly focused on the use of these systems across a variety of health care settings as well as how they are being used. Some argue that CDSS is the optimal means of ensuring that evidence based care guidelines are immediately available to clinicians at the point of care.^1,2 Physicians in a variety of settings have expressed interest in, and a need for, decision support that helps them to care for their patients better.^2–5

The use of decision support prompts and reminders has been studied across a variety of health care settings and health care issues.^2,6,7 Prompts can improve health care provider compliance with guidelines^1,2,8–10 and, ultimately, improve health care quality and outcomes for patients.^6,7,10,11 They have been found to improve delivery of anticipatory guidance in pediatric offices,^9,12 reduce inappropriate antibiotic prescribing,¹¹ and increase post-surgery antibiotic administration to reduce post-operative infections.¹

However, other researchers have found that, although sentiments towards clinical decision support are typically positive,^1,3 the rates at which clinicians adopt and rely on clinician decision support prompts and reminders vary and can be quite low.^1,3,13 The earliest research on CDSS from the 1970s by McDonald and colleagues indicated that only half of prompts are responded to or acted on in a clinical decision support system.¹⁴ More recently, we found that this response rate has remained steady approximately three decades later.¹⁵ The low likelihood of physician response is a substantial impediment to the effective use of clinical decision support systems.

Several challenges to the adoption and use of clinical decision support prompts have been identified. These vary from costs, to electronic infrastructure, to the setting itself.¹⁶ Another reason that has been offered is “alert fatigue.”^17,18 Alert fatigue occurs when clinicians encounter a large number of alerts or the same alerts many times. As a result they become desensitized to the information, finding it uninformative and no longer notice it or choose to ignore it.^17–20 What is largely unknown, however, is how to successfully increase physician response rates to CDSS prompts.

This study was designed to test the hypothesis that highlighting prompts in a CDSS can increase rates of physician response.

MATERIALS AND METHODS

The Child Health Improvement through Computer Automation (CHICA) System

The CHICA system is a CDSS which was implemented in 2004 and has been used by a variety of pediatric health care providers continuously since its implementation.^8,9,21 Currently, the CHICA system operates in four outpatient pediatric clinics in Indianapolis. CHICA has captured data from over 255,000 encounters with more than 37,500 unique patients.

Data are captured by CHICA through two means.⁹ The first source of data is the 20-item pre-screener form (PSF), a paper form with 20 yes/no questions that families complete upon arrival to the clinic while awaiting their appointment. The items presented on an individual patient’s PSF are electronically generated by an algorithm using the child’s age and other demographic data, data captured at other prior encounters, and data contained elsewhere within the child’s electronic medical record. Once completed, the form is scanned to capture the parents’ answers as coded data. Examples of questions include, “Does [child’s name] always wear a helmet when riding her bike or tricycle?” and “Do you feel safe in your home?”

The other CHICA data source is the physician worksheet (PWS). This paper form is completed by the physician during the encounter with the patient. It includes up to six prompts generated both by responses provided on the PSF by families, as well as age-specific general care guidelines. Each PWS prompt alerts the physician to possible interventions. Each prompt has up to six check boxes through which the physician can document assessments or actions taken in response to the reminder. The number of prompts appropriate for most encounters exceeds six so CHICA uses a prioritization scheme based on expected value to select the six highest priority prompts to print on the PWS.²²

Prior research with the CHICA system has examined human and system errors²³; successes of clinical interventions, such as a parental smoking cessation system²⁴; clinical guideline evaluation^8,21; chronic condition management²⁵; developmental milestones and mental health outcomes^26,27; and prioritization strategies of preventive care reminders.²⁸

Design and Sample

With the hypothesis that highlighting the prompts would increase physician responsiveness, we programmed CHICA to print certain prompts with yellow highlighting over the alert (Figure 1). To select prompts for study inclusion, we ordered the CHICA system’s PWS prompts by how frequently they were printed for physicians and how frequently physicians responded to them. The prompts were matched in pairs with similar priority, frequency of printing, and response rates. The prompts we identified for randomization between clinics were also those that we were comfortable randomizing to being either color highlighted or not. Additionally, “high-priority” prompts were identified by the investigators as being so critical that they were always highlighted during the intervention. Specifically, we determined that it was unethical to randomize these high-priority prompts to being either highlighted or not highlighted given our hypothesis that highlighting would increase physician response to prompts and our concern that highlighting could potentially decrease responses to (less salient) prompts that were not highlighted. This trial ran from 05/16/2012 to 08/14/2012.

Figure 1.

Example of yellow highlighted prompt seen by physician on Physician Worksheet (PWS).

Seven PWS prompts of interest were included for randomization among clinics. The Supplemental Table lists the prompts, how they were triggered to appear to physicians on the PWS, and the age range of the child targeted by the prompt. Pairs of clinics were matched on size by number of providers. One of each clinic pair was randomly by coin flip assigned to receive one set of highlighted prompts. The other clinic was given the other set of highlighted prompts. This design is perfectly balanced because each clinic group served as a control for the other. Physicians did not see any prompts that were highlighted only some of the time; if a prompt was highlighted in that clinic, it was highlighted for the entire study duration at that clinic (and never highlighted at the other two control clinic sites). Figure 1 shows an example of a highlighted prompt on the PWS.

Additionally, four “high-priority” PWS prompts were highlighted every time they appeared throughout the intervention period. These included one prompt pertaining to concerns of possible abuse of the patient, one about concerns of possible domestic violence in the patient’s household, and two dealing with adolescent depression and suicide. For analysis purposes, we compared physicians’ responses to these prompts when highlighted compared to their responses to these same prompts in the year prior to the study period.

By nature of implementing a visual-based intervention (i.e., color-highlighting), study personnel were therefore not blinded to the study design. However, data were extracted automatically by the CHICA system to avoid any bias in interpretation.

Statistical Analyses

For the randomized control trial, we used chi-square (χ²) analysis and binary logistic regression. We controlled the regression for patient sex, age, insurance status, race, as well as the position of the prompt on the PWS, because our previous work has shown these influence the response rates of physicians.²⁹ We used Bonferroni correction to establish a cutoff for statistical significance for the eight separate comparisons conducted for these analyses (0.05/8 = 0.00625).

To compare response rates to the high-priority prompts before and during the intervention period, we used chi-square (χ²) analysis to determine if responsiveness to a given prompt changed over time.

The dependent variable in all analyses was whether the physician responded to the prompt by checking any box signifying that they saw the prompt and did or did not take action.

The Indiana University Institutional Review Board approved this study.

RESULTS

Overall, randomized prompts were printed 2,237 times during the study period. Physicians did not respond to prompts that were highlighted at significantly different rates than those that were not highlighted (OR = 1.056, CI = 0.956–1.167, p = 0.259, ns, χ² = 0.3, p = 0.58, ns). Differences in physicians’ responses to prompts when highlighted compared to when not highlighted were not significant in six comparisons (p > 0.00625). When highlighted, physicians responded to the burn injury prompt for 6 month – 6 year olds 44% of the time compared to when not highlighted, where they responded 66% of the time; i.e., response decreased when the prompt was highlighted (χ² = 31.5609, p < 0.001). On the other hand, when highlighted, physicians responded to the prompt concerning teeth brushing for younger children 72% of the time, compared to 58% of the time when not highlighted (χ² = 16.4218, p < 0.001). Table 1 presents the counts of how frequently prompts were presented to physicians and how frequently physicians responded to the prompts as a function of whether they were highlighted. Figure 2 depicts the rates of physicians’ responses to the randomized prompts as a function of highlighting (prompts not presented to physicians at least 15 times during the study period are not depicted in Figure 2). A post-hoc power calculation³⁰ based on the overall sample size of highlighted versus not highlighted prompts indicates that this study was 80% powered to detect an overall absolute difference in response rate of about 5%, both for the randomized and the high-priority prompts included in this study..

Table 1.

Counts of how frequently the randomized prompts were presented to physicians and how frequently physicians responded to them as a function of whether they were highlighted.

	Highlighted			Not Highlighted			χ2	p-value
	Total # Times Presented	# Times Responded To by Physician	Response Rate (%)	Total # Times Presented	# Times Responded To by Physician	Response Rate (%)
Overall (All Randomized Prompts)	1,076	672	62	1,161	712	61	0.3	0.58
Advise to Brush Teeth (Older Children)	164	112	68	283	183	65	0.61	0.44
Advise to Brush Teeth (Younger Children)	487	352	72	289	168	58	16.42	<0.001*
Alcohol High Risk	3	0	0	3	1	33	1.2	0.27
Alcohol Low Risk	13	6	46	6	3	50	0.02	0.88
Drugs High Risk	4	3	75	2	0	0	3	0.08
Drugs Low Risk	0	0	n/a	1	1	100	n/a	n/a
Injury Burns from Fire 6m - 6y	222	97	44	473	313	66	31.56	<0.001*
Injury Burns from Fire 6y - 12y	183	102	56	104	43	41	5.49	0.02
Overall (All High-Priority Prompts)	292	171	59	661	393	59	0.07	0.8
Household Abuse Concerns	9	8	89	42	29		1.47	0.23
Adolescent Depression or Suicide Concerns	169	104	62	352	207	59	0.35	0.55
Adolescent Depression or Suicide Follow Up	18	11	61	20	17	85	2.79	0.1
Domestic Violence Concerns	96	48	50	247	140	57	1.25	0.26

^*Indicates statistical significance accounting for Bonferonni Correction (0.05/8 = 0.00625).

Figure 2.

Proportions of physicians’ responses to prompts as a function of whether they were color higlighted. (Data from prompts not presented to physicians at least 15 times total during the study period have been omitted from this graph.)

*Indicates statistical significance accounting for Bonferonni Correction (0.05/8 = 0.00625).

Similar to the RCT, analyses of the high-priority prompts also revealed that, overall, they were not more likely to be answered when highlighted than in the year prior when they were not (χ² = 0.067, p = 0.796, ns). None of the four prompts selected for study inclusion produced response differences that attained statistical significance (p > .05). Table 1 and Figure 2 also present these high-priority prompt data.

DISCUSSION

Overall, highlighting prompts did not increase physicians’ responsiveness to them. This lack of an effect held both for our RCT between clinics, as well as our before-after analysis of high-priority reminder prompts. The study was 80% powered to find differences in responsiveness of approximately5%. These findings suggest that highlighting is not an effective strategy to increase the rates with which physicians attend to reminder prompts.

We offer several possible explanations for why highlighting did not impact physicians’ responses to reminder prompts. First, it could be the case that highlighting a prompt is simply not a strong enough indicator to render a prompt more perceptually salient. In the context of alert fatigue, highlighting might not be a strong enough cue to overcome fatigue. Other explanations for lack of physician responsiveness to prompts includes that they disagree with the content of the reminder, they need to address more pressing issues with the patient, or they think the data on which the reminder is based are incorrect. Signal detection theory speaks to this phenomenon,^31,32 and refers to the capacity to discriminate between environmental input that does (known as a stimulus or signal) or does not (known as noise) provide useful information or require a response.³¹ The lack of highlighting’s efficacy in this study, therefore, could be conceptualized as a failure to render the prompts salient enough to reach a perceptual threshold wherein physicians acknowledge them as a stimulus or signal among the “noise” present in a clinical encounter. In an instance in which there is no response when a stimulus or signal is present (in this study, a prompt), signal detection theorists refer to it as a “miss.” Therefore, highlighting prompts in this study was unable to convert these misses into “hits.”

Another possible explanation, tied to principles of operant conditioning³³ and human motivation,³⁴ is that when there are no outcomes--either intrinsically or extrinsically--tied to one’s actions or failures to act, individuals may be unmotivated to act. In the case of reminder prompts, it could be that physicians notice the prompts (i.e., they are perceptually salient). However, if physicians are not either intrinsically motivated or have no extrinsic motivations, such as avoiding consequences or attaining incentives, this could render it difficult to change their behavior to respond to prompts.

There are some limitations to the study that warrant consideration. The clinic sites are concentrated in an urban pediatric outpatient setting, so generalizability to other settings is cautioned; however, we do not have any reason to suspect that color highlighting prompts would be more effective in other settings. This study was also conducted using paper-based prompts; it is possible that highlighting on a computer screen might make a difference, although we have no evidence to support that this would be the case. Further, we did not ask physicians why they did or did not respond to prompts (highlighted or not). Identifying physicians’ reasons for responding or not responding would be a useful next step. Factors impacting responsiveness could include their perceived level of knowledge or training about a given issue, the perceived “actionability” of the prompt, and/or their belief in the effectiveness of what they may say to a family about the issue. Additionally, we did not control for provider demographic information in our analyses. Given that our study design was perfectly balanced with each clinic pair serving as the other pair’s control, any demographic differences should be irrelevant unless the interaction between each prompt and each physician happened to be exactly equal in magnitude and opposite in direction as the hypothesized effect of the color-highlighting. Lastly, although this study was sufficiently powered overall, one could argue that that it was potentially under-powered with respect to specific prompts. Power to detect a change in each specific prompt category is lower both because of the smaller number of times each of these was printed and because of the Bonferroni correction for multiple comparisons

Although highlighting the prompts did not increase physicians’ responsiveness to them in this study, it offers ideas for future studies to improve responsiveness. We recommend that some sort of outcome be tied to failure to respond to prompts. For example, use of a “hard stop” in an electronic CDSS could prohibit the physician from advancing in the electronic system without clicking a checkbox to signify that she at least saw the prompt.³⁵ Alternately, if the physician attempts to advance in the electronic system without clicking to acknowledge the prompt, the system could present a pop-up box that reviews the physician’s decision to ignore the prompt that they would have to confirm. In a paper-based system like CHICA, physicians could be given regular feedback concerning how many and what types of prompts they ignored. There could also be asynchronous feedback structures such that physicians receive daily summaries of patients for whom they ignored reminder prompts, to provide an opportunity to attend to those decisions when they have finished seeing patients for the day during a, perhaps, less hectic timeframe.

CONCLUSION

We hypothesized that color highlighting reminder prompts would improve physician responsiveness to them, but this hypothesis was not supported. We encourage investigators to evaluate other strategies to increase physicians’ response rates to reminder prompts.

What’s New.

This study tested the hypothesis that selectively color highlighting prompts in yellow would improve physicians’ responsiveness. Color highlighting reminder prompts does not appear to be an effective strategy to increase physicians’ responsiveness to clinical prompts.

Abbreviations

CDSS

Clinical Decision Support Systems

CHICA

Child Health Improvement through Computer Automation