Abstract
OBJECTIVE
To compare the effectiveness of three interventions designed to improve the pneumococcal vaccination rate.
DESIGN
A prospective controlled trial.
SETTING
Department of Veterans Affairs ambulatory care clinic.
PATIENTS/PARTICIPANTS
There were 3,502 outpatients with scheduled visits divided into three clinic teams (A, B, or C).
INTERVENTIONS
During a 12-week period, each clinic team received one intervention: (A) nurse standing orders with comparative feedback as well as patient and clinician reminders; (B) nurse standing orders with compliance reminders as well as patient and clinician reminders; and (C) patient and clinician reminders alone. Team A nurses (comparative feedback group) received information on their vaccine rates relative to those of team B nurses. Team B nurses (compliance reminders group) received reminders to vaccinate but no information on vaccine rates.
MEASUREMENTS AND MAIN RESULTS
Team A nurses assessed more patients than team B nurses (39% vs 34%, p = .009). However, vaccination rates per total patient population were similar (22% vs 25%, p = .09). The vaccination rates for both team A and team B were significantly higher than the 5% vaccination rate for team C (p < .001).
CONCLUSIONS
Nurse-initiated vaccine protocols raised vaccination rates substantially more than a physician and patient reminder system. The nurse-initiated protocol with comparative feedback modestly improved the assessment rate compared with the protocol with compliance reminders, but overall vaccination rates were similar.
Keywords: pneumococcal vaccine, vaccinations, standing orders, reminders, feedback
Streptococcus pneumoniae, the most common cause of community-acquired pneumonia, is responsible for 40,000 deaths per year.1 Increasing incidence of pneumococcal resistance to antibiotics has focused attention on disease prevention. A meta-analysis of nine randomized controlled trials demonstrated that the pneumococcal vaccine is effective in preventing pneumococcal infection.1,2 However, the 1993 National Health Interview Survey reported that only 28% of high-risk individuals, including persons aged 65 years or older, had ever received the pneumococcal vaccine.3
A variety of methods have been used to improve pneumococcal vaccination rates but with varying degrees of success. Effective methods have included patient and provider education, physician and patient reminders, written orders prepared by pharmacists for physicians to sign, and standing orders for vaccine administration by nurses.1,4–16 However, among these methods, it remains unclear which is the most effective to increase vaccination rates in adult outpatients.
We built on previous studies to investigate the effectiveness of three comprehensive methods to better deliver pneumococcal vaccines to our patients: (A) nurse standing orders with comparative feedback combined with a patient and clinician reminder system; (B) nurse standing orders with compliance reminders combined with a patient and clinician reminder system; and (C) a patient and clinician reminder system alone. Peer-comparison feedback has proved useful in improving preventive health care measures such as childhood immunizations and cancer screening, but has not been previously evaluated for pneumococcal vaccine.1,4,5,17,18 To test which method was most effective, we evaluated the three in a prospective controlled study in a Department of Veterans Affairs (VA) outpatient clinic.
METHODS
Institution and Patients
The West Los Angeles VA General Medicine ambulatory clinic provides comprehensive general medical care for 12,000 patients. More than 90% of the patients that visit the clinic are men. According to a recent survey of VA patients, they were relatively older (36.5% aged 65 years and older), poorer (average income $13,300), and more likely to be homeless (10.5%) than patients of a capitated group practice.19
The clinic is staffed by three health care firms, or teams. New patients are preassigned according to the last two digits of their social security number to a primary care physician within one of three teams. The teams practice in geographically separate areas, although all are in a single outpatient facility. Each team is staffed by separate internal medicine attending physicians (n= 14 to18), internal medicine residents (n= 10 to 17), nurse practitioners/physician assistants (n= 2), registered nurses (n= 2), licensed vocational nurses (n= 1), and clerks (n= 2). Interns account for approximately 50% of all internal medicine residents. Approximately 75% of the clinical attending physicians are general internists, while 25% are subspecialists who act as generalists within the clinic. As is typical of a tertiary care teaching facility, the attending physicians range in level of training from clinical instructors to full-time professors. Teaching attending physicians (almost all general internists) share a common room when overseeing the resident clinics. Patients share a common area while waiting for appointments. As well, nursing staff and clerks occasionally cover for each other, on a limited basis.
Study Design
During the 12-week period of June 16 to September 5, 1997, we performed a prospective trial to compare three methods to increase the pneumococcal vaccine rate in a VA ambulatory care clinic. All patients with regularly scheduled appointments with a primary care provider (attending physician, resident physician, nurse practitioner, or physician assistant) were included in this study. Walk-in patients seen on an unscheduled basis (i.e., urgent care patients) were excluded because such patients are assessed by and referred to specific urgent care providers by a separate nursing triage system that was outside of our control.
Primary care providers from all three medical teams received the same baseline intervention of two reminders: (1) a health maintenance checklist requesting that the clinician assess the patient’s pneumococcal vaccine eligibility and (2) a computerized summary of the patient’s vaccine status. These were given to the clinician at each patient visit. In addition, patients from each team received reminders in the form of posters and flyers advertising the benefit and importance of the pneumococcal vaccine. The posters were placed in the vital-signs check-in room, and the educational flyers were placed on the check-in clerk’s desk. All protocols were instituted at the onset of the study.
Two teams were selected for further intervention by random assignment. Team A nurses were given standing orders to vaccinate all eligible patients with the pneumococcal vaccine and provided with comparative feedback on their vaccination rates; team B nurses were given standing orders to vaccinate all eligible patients with the pneumococcal vaccine and provided with compliance reminders to vaccinate. Team C was the control group. The nurse standing orders protocol consisted of an eligibility checklist and vaccine prescription. Feedback consisted of comparisons between team A and team B nurse vaccine assessments and vaccinations. Compliance reminders consisted of messages encouraging the clerks to continue to hand out the forms to patients on initial check-in, and the nurses to continue to assess and provide vaccines to eligible patients. Comparative feedback or compliance reminders were sent by preformatted electronic mail twice monthly to team A and team B nurses.
On presentation to either team A or team B, the patients were asked by the clerks to complete a brief questionnaire on previous vaccine status. Eligibility for pneumococcal vaccination was defined using previously accepted criteria.1 On vital-signs check-in, the nurse would then assess patient eligibility for vaccination by reviewing the questionnaire with the patient and checking the computerized records for evidence that the patient had not been previously vaccinated. The vaccine would then be offered.
Performance Measures
Performance measures included (1) the number and proportion of patients assessed for eligibility by the nursing staff; (2) total number of vaccines ordered in each team per patient population; and (3) the sustainability of the intervention, as described by the number of patients assessed and the number of eligible patients receiving nurse-prescribed pneumococcal vaccines every 2 weeks.
Data Collection and Analysis
Data on vaccine status were collected using the completed checklists and the computerized vaccination database at the West Los Angeles VA. Patient responses to vaccine recommendations were recorded on the protocol sheet. Forms were collected weekly by the project coordinator (DCR).
We determined prevalence and incidence data by obtaining information from the computerized records and outpatient charts. To ensure the validity of this method, we performed a comprehensive chart review of all written records for a sample of patients. This demonstrated a low yield for detecting vaccinations not documented by either the computer or outpatient record (1 of 51; 2%).
Data were entered into an EXCEL database. Comparisons between groups were made by χ2analysis.
RESULTS
During the 12-week study period, 3,502 patients were seen in regularly scheduled appointments in the three medical clinics (team A, 1,101 patients; team B, 1,221 patients; team C, 1,180 patients).Table 1 shows that team A nurses (nurse standing orders with comparative feedback) assessed a higher fraction of patients than did team B nurses (nurse standing orders with compliance reminders) (39% vs 34%, p= .009). The spillover effect to nurses in team C (control team) was nominal, with only 15 (1%) of 1,180 patients being assessed for pneumococcal vaccination eligibility.
Table 1.
Comparison of Team Vaccination Data
| Vaccination Data | Team ANurse Feedback | Team BNurse Reminder | Team CControl | Total |
|---|---|---|---|---|
| Primary care patients, n | 1,101 | 1,221 | 1,180 | 3,502 |
| Patients assessed for eligibility by nursing staff, n(%) | 428 (39) * | 412 (34) * | 15 (1) | 855 |
| Patients determined eligible by nursing staff, n(%) | 259 (61) † | 320 (78) † | 11 | 590 |
| Refusals by eligible patients, n(%) | 81 (31) ‡ | 58 (18) ‡ | 0 | 139 |
| Vaccines ordered by nursing staff, n | 189 | 272 | 11 | 472 |
| Vaccines ordered by primary care providers, n | 53 | 33 | 42 | 128 |
| Total vaccines given by team, n(%) | 242 (22) § | 305 (25) § | 53 (5) § | 600 |
Team A vs team B, p =.009.
Team A vs team B, p <.001.
Team A vs team B, p =.001.
Number of patients receiving vaccine per number of primary care patients: team A vs team C (control) p <.001; team B vs team C (control) p <.001.
The number of nurse-assessed patients determined to be eligible for vaccination was lower for team A than for team B (61% vs 78%, p < .001). Refusal rates for patients determined eligible for vaccination were higher for team A than team B (31% vs 18%, p= .001). The accuracy rate of nurses’ assessment of patient eligibility was very high: team A nurses incorrectly categorized only 11 patients (3%) and team B, 10 patients (2%) as eligible (p= .47).
The overall vaccination rate (Table 1) for team A and team B was similar (22% vs 25%, p= .09). Both intervention teams had substantially higher rates than did control team C, in which the pneumococcal vaccination rate was approximately 5% (p < .001). Primary care providers from teams A, B, and C ordered pneumococcal vaccinations in 3% to 5% of all patients.
During the 12-week study period, 600 vaccines were ordered: 472 (79%) by the nurses, and 128 (21%) by the primary care providers. This contrasts with the 12-week prestudy period when only 72 pneumococcal vaccines were ordered by primary care providers. Focusing on the sustainability of the nursing-intervention protocols (Fig. 1), the initial surge in the vaccination rate diminished to a relatively steady rate of between 20 and 30 vaccinations per week (total for both intervention teams) after week 8. Even this lower rate exceeded the observed rates of vaccinations ordered by physicians either during the study or prestudy periods.
FIGURE 1.
Sustainability of the nursing intervention.
DISCUSSION
We found that nurse standing orders were more effective for improving the pneumococcal vaccination rate than were clinician and patient reminders, and that nurse standing orders were sustainable over time. Our study underestimated the true vaccination rate of eligible patients because the denominator included all patients regardless of prior vaccination status or eligibility for vaccine. In a review of 120 patients seen during the study period, we found that 39% did not need to receive the pneumococcal vaccine, based on previously established guidelines.1 Thus, in actuality, we estimate that teams A and B vaccinated 36% and 41% of eligible patients, respectively, whereas team C vaccinated only 7% of eligible patients. Although even the best of these rates was not optimal, the success of the nursing intervention represented a dramatic improvement over both the control team (in which physicians were required to order vaccines) and the preintervention rate of 2%.
Our study’s results may not seem surprising, given the recent statement of the Advisory Committee on Immunization Practices (ACIP) that standing orders are the most effective method for increasing pneumococcal vaccination rates.1 Yet, we found only one other study that compared standing orders with other pneumococcal vaccination strategies in the ambulatory setting. That study, by Herman et al., 6 was a prospectively controlled trial in a public urban teaching hospital but differed from our study in some important respects. First, the physician and patient interventions were passive, involving only education and informational materials. The present study used active interventions including patient-specific physician reminders provided at each patient visit. Second, while the study by Herman et al. evaluated the effect of combining nurse standing orders with physician education, our study assessed the combination of nurse standing orders with several other methods to increase the rate of vaccination. Our study represented an aggressive attempt to improve the vaccination rate by combining nurse standing orders with compliance reminders and comparative feedback, patient education, and timely patient-specific clinician reminders, all methods that would likely be found in contemporary ambulatory settings.1,4,5,8–10,13–15 Third, Herman and colleagues looked only at patients aged 65 years or older, whereas our study was more complex and thus better able to assess the impact on the rate of vaccination for all eligible patients. Fourth, in the Herman study, a research assistant provided the nurses with the initial evaluation of patient vaccination status. In the present study, the nurses performed all the assessments for patient vaccination eligibility. Fifth, Herman et al. suggested that improving the pneumococcal vaccination rate was linked to improving the influenza vaccination rate. Our study demonstrated that pneumococcal vaccination rates could be increased during the summer when influenza was not a concern. Thus, the present study is the first to test the ACIP statement in a comprehensive manner, and the results lend solid support for the benefit of nurse standing orders in raising the pneumococcal vaccination rate.
The nurse-initiated protocols raised the vaccination rates substantially. We were surprised, however, that the impact of aggressive clinician reminders and education on physician, nurse-practitioner, and physician-assistant prescribing was nominal. The total number of vaccines given by primary care providers (physicians, nurse-practitioners, and physician assistants) rose by only a factor of about 2 from the prestudy period to the study period (72 to 128 vaccines), with a pneumococcal vaccination rate during the study period about 3.3 per team per week. This suggests that even more vigorous educational and reminder efforts directed at primary care providers would be needed to raise the vaccination rate to the sustainable level observed in the nurse standing order teams (10–15 vaccinations per team per week, after 8 weeks).
The failure to increase the vaccination rate in team C could, in part, have been due to the fact that the clerks in the control team did not actively distribute patient flyers, as they were asked to do for the intervention teams, or did not receive electronic messages or feedback that their nurse colleagues received. Previous studies have demonstrated that organizational efforts involving the active participation of clerks and nurses can result in improved vaccination rates in the ambulatory setting.1,4–6,20 We did not assess clerk compliance and cannot state with any certainty what effect clerk participation had on our results. However, as noted above, clinicians were presented with reminders at every visit, and it is not clear that the extra effort by clerks would have improved vaccination rates substantially.
Overall, though, the determining factor for improving the pneumococcal vaccination rate was nurse evaluation of eligibility and vaccine status. Although the provision of comparative data increased the assessment rate of patients for vaccine eligibility, it did not improve the vaccination rate. This difference between the rate of assessment and the rate of vaccination was largely attributable to differences in eligibility and in the patients’ acceptance rates for being vaccinated, rather than to the system used to encourage vaccination. We note that even though our intervention was generally successful, the nurses assessed less than 40% of all patients. We speculate that the lower than expected assessment rate was due to two important factors. First, as the study progressed, nurses began to “prescreen” patients to save time. As can be seen in Figure 1, by week 8 the assessment rate approached the actual vaccination rate, suggesting that nurses were not filling out forms for patients who refused the vaccine or who were not eligible. This would tend to lower the documented assessment rate, which was based on tabulating completed forms. Second, the nurses reported that as their workload increased, their ability and willingness to inquire about the patient’s vaccine status and eligibility diminished. This suggests that further improvement of assessment and vaccination rates might be limited by other competing demands on nursing time. Nonetheless, the vaccination rate of the nursing staff was still far superior to that of the physician staff.
Our study has certain limitations. The protocols were implemented at a single institution, and the results may not be generalizable to other settings. The study lasted only 3 months, and the long-term effect (i.e., sustainability) might have fallen over the ensuing months. Also, given our design methodology, clustering may have occurred. In this study, clustering refers to the possibility that a nurse or group of nurses may have affected the results of the study through extraordinary efforts to vaccinate. In order to address this issue, we randomly sampled eight nursing assessment forms per week, per nursing intervention team, over the 12-week study. Thus, a total of 192 forms were reviewed.Table 2 illustrates the distribution of patient assessments and total vaccines ordered by nursing staff in each firm. These data do not show a substantial clustering effect.
Table 2.
Distribution of Patient Assessments and Total Vaccines Ordered by Nursing Staff
| Team | Individual | Assessments, % | Total VaccinesOrdered, % |
|---|---|---|---|
| A (nurse feedback) | RN-A1 * | 39.5 | 44 |
| RN-A2 * | 25.0 | 23 | |
| LVN † | 18.8 | 17.5 | |
| Cross-coverage ‡ | 16.7 | 15.8 | |
| B (nurse reminder) | RN-B1 * | 35.4 | 33.3 |
| RN-B2 * | 20.8 | 22.2 | |
| LVN † | 29.2 | 34.7 | |
| Cross-coverage ‡ | 12.5 | 9.7 |
RN indicates registered nurse. Two RNs worked full-time for team A (A1, A2), and two worked full-time for team B (B1, B2).
LVN indicates licensed vocational nurse.
Cross-coverage refers to temporary assignment of RNs and LVNs from other teams or from other clinic areas.
Another limitation of this study is that nurses may have discussed comparative vaccination rates with each other, which may have diluted the effects of feedback and reminders. Finally, although we reviewed the information on each assessment form, we did not review patient medical records to assess the accuracy of the assessment form. Even so, it is unlikely that substantial error was made because three criteria accounted for 90% of eligible patients: age 65 or older (46%), diabetes (25%), and chronic cardiopulmonary disease (19%).
In conclusion, at our institution, nurse standing orders raised vaccination rates substantially more than a clinician and patient reminder system. In a systematic review of clinical guidelines, Grimshaw and Russel concluded that the provision of patient-specific reminders at the time of consultation was the most effective strategy to improve performance.21 Though the present study used that strategy for physicians, we found that a nurse-initiated intervention was far more effective. Furthermore, nurse standing orders with comparative feedback only modestly improved the assessment rate relative to nurse standing orders with compliance reminders, but did not improve the overall vaccination rate. As a final note, because institution-specific processes of care may have affected the success and overall magnitude of the intervention, further studies are warranted to confirm our findings.
Acknowledgments
The authors thank Carrie Hutchinson, RN, and the other administrative support staff of the Primary Ambulatory Care Center for their help with the study; Mr. Andrew Lanto, MA, and to Martin Lee, PhD, for their assistance with the statistical analysis; and we acknowledge the contribution of the anonymous reviewers.
REFERENCES
- 1.Centers for Disease Control and Prevention. Prevention of Pneumococcal Disease: Recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR. 1997;46(RR-8):1–24. [PubMed] [Google Scholar]
- 2.Fine MJ, Smith MA, Carson CA, et al. Efficacy of pneumococcal vaccination in adults: a meta-analysis of randomized controlled trials. Arch Intern Med. 1994;154:2666–77. doi: 10.1001/archinte.1994.00420230051007. [DOI] [PubMed] [Google Scholar]
- 3.Centers for Disease Control and Prevention. Influenza and pneumococcal vaccination coverage levels among persons aged ≥65 years—United States, 1973–1993. MMWR. 1995;44:506–7. 513–5. [PubMed] [Google Scholar]
- 4.Gyorkos TW, Tannenbaum TN, Abrahamowicz M, et al. Evaluation of the effectiveness of immunization delivery methods. Can J Public Health. 1994;85:S14–30. [PubMed] [Google Scholar]
- 5.Fiebach N, Beckett W, et al. Prevention of respiratory infections in adults: influenza and pneumococcal vaccines. Arch Intern Med. 1994;154:2545–57. [PubMed] [Google Scholar]
- 6.Herman CJ, Speroff T, Cebul RD, et al. Improving compliance with immunization in the older adult: results of a randomized cohort study. J Am Geriatr Soc. 1994;42:1154–9. doi: 10.1111/j.1532-5415.1994.tb06981.x. [DOI] [PubMed] [Google Scholar]
- 7.Klein RS, Adachi N, et al. An effective hospital-based pneumococcal immunization program. Arch Intern Med. 1986;146:327–9. [PubMed] [Google Scholar]
- 8.Siebers MJ, Hunt VB, et al. Increasing the pneumococcal vaccination rate of elderly patients in a general internal medicine clinic. J Am Geriatr Soc. 1985;33:175–8. doi: 10.1111/j.1532-5415.1985.tb04888.x. [DOI] [PubMed] [Google Scholar]
- 9.Turner RC, Waivers LL, O’brien K, et al. The effect of patient-carried reminder cards on the performance of health maintenance measures. Arch Intern Med. 1990;150:645–7. [PubMed] [Google Scholar]
- 10.Becker DM, Gomez EB, Kaiser DL, et al. Improving preventive care at a medical clinic: how can the patient help? Am J Prev Med. 1989;5:353–9. [PubMed] [Google Scholar]
- 11.Bloom HG, Bloom JS, Krasnoff L, Frank AD, et al. Increased utilization of influenza and pneumococcal vaccines in an elderly hospitalized population. J Am Geriatr Soc. 1988;36:897–901. doi: 10.1111/j.1532-5415.1988.tb05782.x. [DOI] [PubMed] [Google Scholar]
- 12.Clancy CM, Gelfman D, Poses RM, et al. A strategy to improve the utilization of pneumococcal vaccine. J Gen Intern Med. 1992;7:14–8. doi: 10.1007/BF02599095. [DOI] [PubMed] [Google Scholar]
- 13.Shea S, Dumouchel W, Bahamonde L, et al. A meta-analysis of 16 randomize-controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc. 1996;3:399–409. doi: 10.1136/jamia.1996.97084513. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Cohen DI, Littenberg B, Wetzel C, Neuhauser DB, et al. Improving physician compliance with preventive medicine guidelines. Med Care. 1982;20(10):1040–5. doi: 10.1097/00005650-198210000-00006. [DOI] [PubMed] [Google Scholar]
- 15.Cheney C, Ramsdell JW, et al. Effect of medical records’ checklists on implementation of periodic health measures. Am J Med. 1987;83:129–36. doi: 10.1016/0002-9343(87)90507-9. [DOI] [PubMed] [Google Scholar]
- 16.Morton MR, Spruill WJ, Cooper JW, et al. Pharmacist impact on pneumococcal vaccination rate. Am J Hosp Phar. 1988;45(1):73. Letter. [PubMed] [Google Scholar]
- 17.Balas EA, Boren SA, Brown GD, Ewigman BG, Mitchell JA, Perkoff GT, et al. Effect of physician profiling on utilization: meta-analysis of randomized clinical trials. J Gen Intern Med. 1996;11:584–90. doi: 10.1007/BF02599025. [DOI] [PubMed] [Google Scholar]
- 18.LeBaron CW, Baughman AL, Dini EF, Maes E, Dietz V, Bernier R, et al. Impact of measurement and feedback on vaccination coverage in public clinics, 1988–1994. JAMA. 1997;277(8):631–5. [PubMed] [Google Scholar]
- 19.Peabody JW, Luck J, et al. How far down the managed care road? A comparison of primary care outpatient services in a Veterans Affairs medical center and a capitated multispecialty group practice. Arch Intern Med. 1998;158:2291–8. doi: 10.1001/archinte.158.21.2291. [DOI] [PubMed] [Google Scholar]
- 20.Ratner ER, Fedson DS, et al. Influenza and pneumococcal immunization in medical clinics, 1978–1980. Arch Intern Med. 1983;143:2066–9. [PubMed] [Google Scholar]
- 21.Grimshaw JM, Russel IT, et al. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342(8883):1317–22. doi: 10.1016/0140-6736(93)92244-n. [DOI] [PubMed] [Google Scholar]