You Can Lead a Horse to Water: Physicians’ Responses to Clinical Reminders

Abstract

Background:

Meaningful use of health information technology (HIT) requires the use of clinical decision support systems (CDSS). However, the effectiveness of CDSS depends on physician compliance with clinical reminders which is known to be highly variable. Our objective was to evaluate physician adherence to clinical reminders from a CDSS designed to maximize features known to improve practice.

Methods:

We evaluated physicians’ compliance with clinical reminders generated by the Child Health Improvement through Computer Automation (CHICA) system, a pediatric CDSS that generates scannable paper forms that are completed by patients, staff and physicians during routine care. The forms provide tailored reminders and collect coded clinical data during routine care. We examined CHICA’s database to assess the rates of response by patients and physicians to questions and reminders generated by the system. Results showed that while patients answered, on average, 60.6% of 1,351,896 questions generated by the system over 5 years, physicians responded to only 42.9% of 343,949 alerts and reminders over the same period of time. Response rates appeared to be inversely related to both the complexity and sensitivity of the topic.

Discussion:

Poor physician adherence to clinical reminders in this optimized system reduces effectiveness of the system and poses some liability issues. Strategies to alert physicians to the reminders of highest import are needed.

Introduction

The Health Information Technology for Economic and Clinical Health (HITECH) Act of the American Recovery and Reinvestment Act of 2009 (ARRA) (Pub. L. 111–5), was enacted on February 17, 2009. Under HITECH, physicians will be eligible for $40,000 to $65,000 for showing that they are meaningfully using health information technology. The interim final rule defining “meaningful use” includes implementing clinical decision support tools to facilitate disease management. Likewise, certification criteria for meaningful use require that HIT systems automatically and electronically generate and indicate in real-time, alerts and care suggestions based upon clinical decision support rules and evidence grade.¹

Indeed, clinical decision support systems (CDSS) can improve care. In a critical review of 100 clinical trials of CDSS compared to usual care controls,² 64% of systems improved practitioner performance, especially with preventive care (76%). However, the effectiveness of clinical decision support is highly dependent on physician adherence to clinical reminders. Agrawal and Mayo-Smith showed that physician adherence to reminders can be highly variable.³ Adherence by reminder varied from 67%–97%, and by physician varied from 29%–100%. Factors impeding adherence include lack of coordination between nurses and providers; reminders outside routine workflow, difficult data acquisition; and poor interface usability. Factors facilitating adherence include limiting the number of reminders at a site; and integration of reminders into workflow.⁴

Kawamoto, et al. conducted an analysis of 88 randomized controlled trials of CDSS to identify features predicting success.⁵ Four features, found together in 36% of systems, were identified: (1) integrating CDSS into clinician workflow, (2) CDSS that made recommendations rather than assessments alone, (3) decision support at the time and place of decision making, and (4) computer assessment of eligibility for services. Of systems that had all four features, 94% improved practice. We developed a CDSS for pediatric primary care that incorporates these features. Although the system appears to improve the quality of care, we sought to evaluate how adherent clinicians were to the recommendations provided by the system.

Methods

CHICA

CHICA (Child Health Improvement through Computer Automation) is a CDSS that has been operating in a high volume academic pediatric primary care clinic at Wishard Memorial Hospital in Indianapolis, IN, since November 2004.⁶ In October 2009, the system was installed in a second, smaller community health center. CHICA incorporates clinical decision support for pediatric guidelines in the form of dynamic risk factor assessment questionnaires for parents and reminders to physicians. CHICA uses adaptive turnaround document (ATD) technology.^{7, 8} ATDs are computer generated paper forms that are optically scanned to capture structured data. ATDs are used to generate two tailored, scannable paper forms: the Pre-Screening Form (PSF) and the Physician Work Sheet (PWS).

To determine what information needs to be printed on each ATD, CHICA employs a library of computer readable rules (Arden Syntax) that evaluate the underlying electronic medical records (Regenstrief Medical Record System and CHICA databases). CHICA also uses a global prioritization scheme to ensure the most important content is printed.⁹

When a patient registers through the electronic appointment system, a standard HL7 message is sent to the CHICA system. This triggers CHICA to request a download of medical information from the Regenstrief medical record system (RMRS).¹⁰ CHICA uses this information to generate the PSF. The PSF has two sections: The top section has a space to record vital signs, and the bottom section has the 20 most important yes/no questions for the parent to answer in a particular visit in order to guide preventive care or disease management. The questions are selected by a set of rules encoded in Arden Syntax Medical Logic Modules (MLMs).^11–13

The PSF questions are completed by the parents in the waiting room. Then the nurse registering the patient enters the vital sign information and scans the PSF. After the PSF is scanned, the patient’s information is sent to CHICA and analyzed along with the existing patient record in RMRS. MLMs are applied to generate the PWS, the worksheet that the physicians complete during the encounter. The PWS contains three sections: (1) vital sign data transferred from the PSF, including calculations such as height and weight percentiles and body mass index; (2) an area for the physician to write free text notes, assessment and plan; and (3) a section with 6 guideline based reminders. Each reminder has a “stem” which explains the reason for the prompt and up to 6 “leaves” with check boxes for the physician to document his or her response to the prompt. (Figure)

Figure.

Three typical prompts from the PWS. The “stems” show information obtained from the PSF and the check boxes provide responses for the physician.

At the same time the PWS is printed, a Just-In-Time (JIT) handout can be generated for the physician to share with the families. A JIT handout is an informational sheet that is tailored to the patient needs and generated with certain PWS prompts. It can be used as a counseling aid to the physician, improving physicians’ self-efficiency and effectiveness as a counselor. JITs can be used for everything from asthma action plans¹⁴ to smoking cessation tools.¹⁵

The PWS is scanned into the computer after the encounter. Coded data are extracted from the form and stored along with a TIFF image of the PWS in the CHICA database. CHICA uses a commercial software package, Teleforms (Cardiff, Vista, CA), to interpret and verify the handwritten and checkbox responses. CHICA is currently used by 20 attending physicians, 30 resident physicians, and a variety of medical students on their outpatient pediatrics rotation. The positive and negative predictive value of the scanning to capture data entered by the physicians was 99.3% and 98.9% respectively, significantly better than more traditional sources of physician generated data such as ICD-9 billing codes.¹⁶ CHICA 1.0 was developed with C# and SQL Server (Microsoft). A new open source version of CHICA has been built on the open source OpenMRS medical record platform, using JAVA and a numerous open source tools.^17–19

CHICA PSF is known to identify frequent issues requiring physician attention. Family responses to CHICA questions addressing maternal depression symptoms, firearms in the home, domestic violence risk, hot water heater adjustments to avoid burns, sleep position to prevent sudden infant death syndrome, family history of deafness (a risk factor for congenital deafness), sickle cell disease, environmental tobacco smoke exposure, concerns regarding child abuse, smoke detector use, risk factors for tuberculosis, and adolescent psychosocial issues uncovered risk factors in 11.3% of responses,⁸ alerting providers to needed preventive care counseling and intervention. Because of CHICA’s ability to identify problems needing the clinician’s attention, it became especially important to evaluate clinicians’ responses to the alerts and reminders generated.

Measuring Responses to Clinical Alerts and Reminders

At the time a PSF or PWS document is generated, the content of the forms – questions asked of families or prompts to physicians – is stored. Likewise, when the forms are scanned into the system, the responses to these questions and prompts are saved. In order to determine the rate at which families and physicians respond to these questions and prompts, we extracted these data and calculated rates at which the PSF questions were answered and PWS prompts received responses. We counted any box checked as a response. We also looked for specific reminders to see if any patterns emerged suggesting why some questions or prompts might be addressed more than others. Mean ratios of responses recorded to questions or prompts printed were calculated along with 95% confidence intervals.

To confirm that a failure of the physician to check a box on a reminder indicated that the issue was not addressed, we reviewed a sample of charts for which the prompts were not answered to see if the issue was addressed elsewhere in the physician’s note.

Results

Between November 2004 and March 2010, CHICA was used to support 87,916 pediatric visits for over 40,000 patients. During those visits, families were asked 1,351,896 questions on the PSF, and physicians were given 343,949 alerts and reminders.

Families answered 60.6% [95% CI: 60.5%, 60.6%] of questions presented to them. Straight forward, factual questions such as developmental milestones, safe sleep position, low birth weight, or introduction of cow’s milk to the diet were answered at rates of 85%–90%. More complex medical history questions such as whether the child might be exposed to lead or tuberculosis, whether the child’s birth weight was low, the parent’s cholesterol level, or whether the family’s water source came from a well or municipal source had response rates from 8%–59%. Potentially sensitive questions such as the presence of smokers in the home, guns in the home, and domestic violence or child abuse also had lower response rates (36%–69%), but these were not the least frequently answered questions.

Physicians responded to only 42.9% [95% CI: 42.7%, 43.0%] of the prompts presented to them. Some prompts represented routine care such as anticipatory guidance or developmental screening. However, several were designed to alert the physician to a high risk situation identified by the family such as domestic violence or adolescent depression. The table shows response rates for a representative sample of these prompts.

Table.

Response rates of selected physician prompts. Only “stems” of the prompt are shown, not the check-box responses.

Prompt	Rate	95% CI
“Milestones to eval today. Check if passed.”	70%	69.4–70.3%
“This mother had some breast feeding problems. How is it going now?”	65%	57.9–71.3%
“<child>’s parent expressed concern that <child> may be maltreated. If you suspect, reporting is mandatory.”	53%	47.6–59.3%
“<child> reportedly has asthma or symptoms suggesting asthma. Please evaluate and/or treat (see attached ASTHMA ACTION PLAN):”	50%	47.8–51.8%
“** ATTENTION ** <child> is at risk for suicide or depression: <reported symptom> Explore symptoms and consequences. Identify social support. Ask about suicide.”	48%	45.0–51.5%
“<child>’s parent reported being abused or feeling unsafe on <date>. If you can speak confidentially and confirm risk of domestic violence, assess for child abuse and advise parent:”	38%	34.2–41.7%

A review of 20 charts from prompts in the table that were not answered confirmed that these issues were not addressed elsewhere in the note.

Discussion

Despite designing the CHICA system to include key characteristics identified in the literature as maximizing physician adherence to clinical reminders, we found that physician responses to CHICA prompts were relatively low. In fact, our rates of physician’s adherence to reminders are remarkably similar to the earliest reports of CDSS in 1976.²⁰ Lin and colleagues found that physicians reported ignoring hypertension treatment recommendations because the data on which the recommendation was based were flawed, the recommendation was not a clinical priority for the visit; or patients were non-adherent.²¹ However, these reasons seem unlikely in our situation.

Based on an examination of the prompts with higher versus lower adherence, it seems that physicians are most likely to respond to prompts for care with which they are familiar. Developmental screening and breast feeding counseling are topics quite familiar to virtually all pediatricians. Child abuse, while less common, is an area in which pediatricians are typically well trained. By contrast, depression and domestic violence are topics pediatricians (and most generalist physicians) may feel unprepared to address. It has been documented that physicians facing domestic violence feel lack of comfort, fear of offending, powerlessness, loss of control, and time constrained.²²

The last, time constraints, may explain the 50% response rate to the asthma prompt. It seems unlikely that pediatricians would feel unprepared to diagnose or treat possible asthma. It is also possible that six prompts per encounter are too many. We don’t believe that vague wording of prompts is a problem because the check box responses are clear.

In contrast to physicians’ relatively low response to clinical reminders, families are generally willing to answer questions that can inform the clinical encounter. This, in fact, allowed CHICA to generate many of the prompts to which physicians seemed reluctant to respond. It is certainly unlikely that discussions of domestic violence and depression would otherwise come up in a well child visit. For this reason, despite low response rates from clinicians, our data (under review) show improved smoking cessation and asthma detection with CHICA reminders.

An obvious question raised by these findings is whether CHICA creates a legal problem for its users. Although we know of no precedent for this, it does seem possible that such a situation could be created. However, to stop screening for important health issues also seems unreasonable. Therefore, it is critical to draw the clinician’s attention to the most critical reminders. We have developed a process for doing this by highlighting in yellow those prompts deemed to be of highest risk. CHICA already produces handouts and additional screening instruments to aid pediatricians in dealing with situations they may not be familiar with. We are also developing a system for monitoring and alerting (by phone call) physicians who miss one of these prompts.