Measuring the Effects of Reminders for Outpatient Influenza Immunizations at the Point of Clinical Opportunity

Abstract

Objective: To evaluate the influence of computer-based reminders about influenza vaccination on the behavior of individual clinicians at each clinical opportunity.

Design: The authors conducted a prospective study of clinicians' influenza vaccination behavior over four years. Approximately one half of the clinicians in an internal medicine clinic used a computer-based patient record system (CPR users) that generated computer-based reminders. The other clinicians used traditional paper records (PR users).

Measurements: Each nonacute visit by a patient eligible for an influenza vaccination was considered an opportunity for intervention. Patients who had contraindications for vaccination were excluded. Compliance with the guideline was defined as documentation that a clinician ordered the vaccine, counseled the patient about the vaccine, offered the vaccine to a patient who declined it, or verified that the patient had received the vaccine elsewhere. The authors calculated the proportion of opportunities on which each clinician documented action in the CPR and PR user groups.

Results: The CPR and PR user groups had different baseline compliance rates (40.1 and 27.9 per cent, respectively; P < 0.05). Both rates remained stable during a two-year baseline period (P = 0.34 and P = 0.47, respectively). The compliance rates in the CPR user group increased 78 per cent from baseline (P < 0.001), whereas the rates for the PR user group did not change significantly (P = 0.18).

Conclusions: Clinicians who used a CPR with reminders had higher rates of documentation of compliance with influenza-vaccination guidelines than did those who used a paper record. Measurements of individual clinician behavior at the point of each clinical opportunity can provide precise evaluation of interventions that are designed to improve compliance with guidelines.

Attention to preventive health services is increasing as evidence for their efficacy accumulates.¹ One of the clearest guidelines that reflects widespread consensus among clinicians is the recommendation to administer influenza vaccines to adults 65 years of age or older.^2,3,4

Yet vaccination rates range between 45 and 58 per cent.⁵ Although health care providers have good in-Donald⁶ wrote in 1976 the “nonperfectibility of man” has prevented health care professionals from acting reliably on their own intentions.

Providers often inadvertently overlook effective preventive therapy for millions of people—an error of omission that contributes to tens of thousands of influenza-related deaths.⁷ A cohort study that measured adjusted outcomes of immunized compared with nonimmunized patients over a three-year period demonstrated that immunized patients have approximately half the deaths, half the hospitalization rates, and half the hospital costs of nonimmunized patients.⁵ Other studies showed similar efficacy rates.⁸ In a Canadian study, 80 per cent of patients 65 years of age or older who were hospitalized for influenza-associated respiratory conditions had had at least one visit with a health care provider prior to the admission and thus in many cases had presented the health care system with an opportunity to administer the influenza vaccine.⁹

Managed care health plans often require provider organizations to report how well their practitioners comply with clinical guidelines.¹⁰ Methods for reporting and auditing guideline compliance rates, however, are not clearly or consistently defined.

For most public health studies of influenza vaccination rates, either administrative billing data⁵ or consumer questionnaires^11,12 are used to determine the rates of immunization. Although the denominators used in different methods of calculating the compliance rates vary, the numerator represents the number of consumers or patients who have received (or believe they have received) an influenza vaccine during the past influenza season. This method of calculating vaccination rates produces a valid rate for a defined population, but it does not permit investigators to study the effectiveness of an intervention designed to improve individual clinicians' compliance rates. For example, offering immunizations at churches, senior centers, or immunization clinics may increase vaccination rates even though providers may take no specific action. Increased media attention to influenza deaths during a given year may motivate patients to seek vaccinations from their providers. A more precise measure of clinicians' actions at each clinical opportunity would help researchers understand what tools help clinicians comply with their own intentions to administer preventive health services.

Several interventions have been developed to address the problem of underutilization of influenza vaccination in adults and children. Placing reminder notices on the front of the paper medical record is labor intensive and is subject to the same “nonperfectibility” problem.¹³ Similarly, mailing reminder postcards or letters has not produced consistently good results.^14,15 Because health care providers can favorably influence patients' beliefs about and acceptance of influenza vaccination, reminding clinicians when they are seeing an eligible patient has additional benefits.^16,17

Computer-based reminder systems have been shown to increase clinicians' rates of compliance with guidelines, especially in preventive health maintenance interventions.^{18,19,20,21,22,23,24} Although the methods used to measure rates of compliance vary across the studies, interventions using computer-based reminders generally use the entering of an influenza vaccination order as the satisfying action for the guideline. When we initially distributed individual compliance reports to our clinicians, using the metric by which only vaccination orders are equated with guideline compliance, they questioned the validity of the report. They argued that there are patient conditions and preferences that are not under the control of the clinician (e.g., patient declines the vaccine, patient received the vaccine elsewhere, patient is allergic to or has other contraindications for the vaccine). One of the reasons that these extenuating patient conditions are not typically considered in compliance measurements is that they are difficult to ascertain through review of the paper medical record. To overcome the limitations that paper-based records impose on practical measurement of influenza vaccination guideline compliance, we used a computer-based patient record system to capture these situations in a coded format and incorporated these considerations in our measure of compliance with influenza vaccination guidelines.

Methods

We conducted this study in a large internal medicine clinic at an academic center. Approximately one half of the clinicians had been using a computer-based patient record system (EpicCare, developed by Epic Systems, Madison, Wisconsin) as their primary record-keeping tool for almost two years (CPR users). The other clinicians in the clinic used traditional paper records (PR users). We defined the influenza season as October 1 through January 31 and referred to the season by the year in which it ended (in January). The CPR system was installed in July 1996, prior to the 1997 influenza season. Rule-based clinical reminders were developed and implemented in September 1997, just before the 1998 influenza season. The reminders appear when the clinician opens the chart of a patient who is eligible for a recommended intervention (Figures ▶ and ▶).

Figure 1.

Display of clinical reminders. When a patient's chart is opened, the system presents relevant alerts on the patient. Rules can be written to consider the patient's problems, medications, allergies, procedures, health maintenance, and laboratory test results.

Figure 2.

Influenza vaccination order set. If the provider agrees with the recommended action, clicking “yes” on the screen shown in ▶ brings up a dynamically configured set of actions with defaults preselected (indicated by check marks). If the clinician accepts these actions, the system orders an influenza vaccination, associates the proper diagnosis with the order, and provides a handout for the patient that includes relevant patient instructions.

Before implementing the clinical reminder system, we distributed a baseline survey of clinicians' attitudes about recommendations regarding preventive health services to all 34 clinicians in the clinic; 23 (68 per cent) returned it. One question was whether they agreed with the recommendation to offer influenza vaccination to adults 65 years of age or older who do not have contraindications to the vaccine.

We used scheduling data to identify all patients 65 years of age and older who had one or more nonacute clinic visits during the influenza seasons of each year from 1995 through 1998. We divided the pool of patient visits into those who saw a CPR user and those who saw a PR user. Patients who had received vaccinations in September (i.e., prior to the defined influenza season) were excluded, as were patients who were allergic to a vaccine component or who had another contraindication. There were no statistically significant differences in the distribution of chronic medical conditions (congestive heart failure, pulmonary diseases, diabetes, coronary artery disease, or stroke) among the patients of the two provider groups. Data were abstracted from the medical records and entered into Paradox version 4.5; they were then imported to SPSS version 7.0 for statistical analysis.

We used the PR as the source document during the prestudy period for both groups; we used the CPR as the source document for the CPR user group and the PR as the source for the PR user group during the study period.

We assessed what actions clinicians took when they were presented with an opportunity to follow the influenza guideline, by measuring the rate at which each clinician offered eligible patients an influenza vaccination. Each visit by a patient eligible for an influenza vaccination was considered an opportunity for the clinician to comply with the guideline by performing one of the following four actions: ordering the vaccine; documenting that counseling was performed; documenting that the vaccine was offered, but the patient declined; or documenting that the patient had already received the vaccine elsewhere. After a patient received or declined the vaccine, he or she was no longer considered an eligible patient for the remainder of that year's influenza season. If only counseling was offered during a visit or if a time-limited contraindication was present, the patient remained eligible for vaccination.

Results

We attempted to abstract 100 per cent of the records of all eligible patients for each study year. The numbers of patient visits in each study group are summarized in ▶. Despite multiple attempts, however, we could not locate the records or find documentation for a portion of the patient visits. ▶ summarizes the number of patient records or encounter documents that we were not able to retrieve.

Table 1.

Eligible Patients Seen by Clinicians in Computer-based Patient Record (CPR) Group and Those in Paper-based Records (PR) Group

	CPR Group			PR Group
	No. of Visits	No. of Patients	Average Visits per Patient	No. of Visits	No. of Patients	Average Visits per Patient
1995	292	182	1.6	283	165	1.7
1996	372	221	1.7	303	173	1.7
1997	422	264	1.6	445	251	1.8
1998	450	314	1.4	550	315	1.7

Table 2.

Missing Encounter Documents in Computer-based Patient Record (CPR) Group and Paper-based Records (PR) Group

	CPR Group			PR Group
	Patients with Missing Records (% of total patients)	Visits Affected (% of total visits)	Visits Remaining for Review	Patients with Missing Records (% of total patients)	Visits Affected (% of total visits)	Visits Remaining for Review
1995	50 (27.5)	78 (26.6)	215	51 (30.9)	84 (29.7)	199
1996	43 (19.5)	92 (24.7)	280	45 (26.0)	88 (29.0)	215
1997	0 (0)	0 (0)	422	66 (26.3)	138 (31.0)	307
1998	0 (0)	0 (0)	450	88 (27.9)	154 (28.0)	396

Of the 23 participants (13 CPR users and 10 PR users) who returned the preventive services survey, 95 per cent either agreed or strongly agreed with the recommendation that adults 65 years of age or older who had no contraindications should be offered an influenza vaccination during the influenza season. The influenza guideline compliance rates are summarized in ▶ and in ▶.

Table 3.

Rates of Compliance with Influenza Vaccination Guidelines in Computer-based Patient Record (CPR) Group and Paper-based Records (PR) Group

	CPR Group		PR Group
	Patient Visits with Actions Satisfying Guideline	% of Visits in Compliance with Guideline	Patient Visits with Actions Satisfying Guideline	% of Visits in Compliance with Guideline
1995	67	40.1	41	27.9
1996	84	38.7	43	28.5
1997	142	60.9	74	37.0
1998	165	68.2	82	30.6

Figure 3.

Rates of compliance with influenza vaccination guidelines among clinicians using a computer-based patient record system (solid black squares indicating intervention group) and those using traditional paper-based records (open triangles indicating control group).

There were no statistically significant changes from the first (1995) to the second (1996) year of baseline measurements for either the CPR users or the PR users (P = 0.36 and P = 0.47, respectively). Both groups increased their compliance rates in 1997 compared with 1996 (59 per cent, P < 0.001, and 30 per cent, P < 0.02, respectively). In 1998, the CPR user group's compliance rate increased again (P < 0.02), for a total increase of 78 per cent (P < 0.001 by the Kruskal-Wallis test) over the 1996 rate prior to the introduction of a CPR. The PR user group's compliance rate dropped by 17 per cent (P < 0.03) in 1998 compared with 1997. The final PR user group compliance rate in 1998 was similar to the baseline in 1996 (P = 0.18).

Discussion

To evaluate an intervention designed to encourage administration of influenza vaccines for eligible patients, we developed a metric to assess clinicians' behavior at each opportunity to offer the vaccine. We refined existing methods of measuring responses to computer-based reminders by capturing patient conditions and preferences that affect appropriate clinician behavior in following the influenza-vaccination guideline. In contrast to those based on population data alone, this measure allowed us to quantify more precisely changes in the behavior of individual clinicians that were attributable to computer-based reminders. We believe this metric for assessing clinicians' behaviors at each clinical opportunity can be applied to other clinical guidelines.

Because of the personal time required to learn to use a CPR, the CPR user group was made up of clinicians who volunteered to use a CPR as part of a pilot project. Consequently, our results may not generalize to the broad population of all clinicians. Although the baseline compliance rates of the CPR user group were higher than those of the PR user group, each group's rates were stable during the two years prior to the implementation of computer-based reminders. The stability of the rates gave us confidence that, without any other external changes, the clinician group's compliance rates would remain approximately the same from year to year.

The statistically significant increases in compliance rates for both groups that occurred between 1996 and 1997 are interesting. During the 1997 influenza season, for reasons unrelated to the study, clinical memoranda informed all clinicians that the major health plans had requested reports on the practice's compliance with preventive health services, such as administration of the influenza vaccine. The practice had also set up a table in the building lobby to promote the clinic and remind people about influenza vaccinations. The public media ran stories about how severe influenza infections were expected to be that season. In contrast, because of an unseasonably warm winter in 1998, media attention to influenza vaccination waned. Although we do not have a definitive explanation for why the PR user group's compliance rate increased in 1997, it is notable that the level of increase was only half that of the CPR user group, and that whatever effect led to the increase in 1997 disappeared in 1998, while the CPR user group's rate continued to increase. We interpret the magnified positive increase among the CPR user group as a positive reflection of how use of a CPR can improve the degree to which clinicians act on their own intentions to follow guidelines. We interpret the continued rise in compliance rates, at a time when compliance by the PR user group returned to baseline rates, as evidence that computer-based reminders can further enhance the compliance rates.

Notice that this study measured documentation of actions related to influenza vaccination guidelines and does not report the actual percentage of patients who received the vaccine in either the CPR or the PR group. Only by directly observing or recording actual clinician behavior, neither of which we performed in this study, would we be able to determine whether CPR users and PR users interact differently with patients or whether that interaction affects clinical outcomes.

We argue that an increase in documentation is in and of itself an improvement in practice. The fact that the documented behaviors of both groups did change in 1997 indicates that both groups can change their documentation rates. Others have demonstrated that CPRs can improve documentation behavior as well.²⁵ In addition, documentation is rising in importance both for quality reporting and for reimbursement compliance.

Conducting a study using a CPR system also had logistical advantages over paper chart reviews. For example, during medical record abstraction, we could not locate 20 to 31 per cent of the paper medical record documents. In contrast, there were no missing CPR documents. These benefits should extend to routine measurement of quality information, which is vastly easier with computer-based records.

As more clinical guidelines view each encounter as an opportunity to provide preventive health services or promote healthy behavior, providers will need effective and efficient methods to take advantage of each opportunity to enhance patient care. For example, health plans now request that clinicians not only document the smoking history of patients but also counsel patients who smoke about the health risks of smoking at each opportunity. The CPR is a valuable tool whose reminders can help clinicians act on these opportunities. Through the use of coded entries, it also automates the capture of precise, accurate information that helps quality personnel as well as researchers evaluate the effects of interventions designed to help improve compliance with guidelines, at every point of opportunity.