Limited Impact of a Multicenter Intervention To Improve the Quality and Efficiency of Pneumonia Care

Abstract

Study objectives

To evaluate the impact of a multifactorial intervention to improve the quality, efficiency, and patient understanding of care for community-acquired pneumonia.

Design

Times series cohort study.

Setting

Four academic health centers in the New York City metropolitan area.

Patients or participants

All consecutive adults hospitalized for pneumonia during a 5-month period before (n = 1,013) and after (n = 1,081) implementation of an inpatient quality improvement (QI) initiative.

Interventions

A multidisciplinary team of opinion leaders developed evidence-based treatment guidelines and critical pathways, conducted educational sessions with physicians, distributed pocket reminder cards, promoted standardized orders, and developed bilingual patient education materials.

Measurements and results

The average age was 71.4 years, and 44.1% of cases were low risk, 36.8% were moderate risk, and 19.2% were high risk. The preintervention and postintervention groups were well matched on age, sex, race, nursing home residence, pneumonia severity, initial presentation, and most major comorbidities. The intervention increased the use of guideline-recommended antimicrobial therapy from 78.1 to 83.4% (p = 0.003). There was also a borderline decrease in the proportion of patients being discharged prior to becoming clinically stable, from 27.0 to 23.5% (p = 0.06). However, there were no improvements in the other targeted indicators, including time to first dose of antibiotics, proportion receiving antibiotics within 8 h, timely switch to oral antibiotics, timely discharge, length of stay, or patient education outcomes.

Conclusions

This real-world QI program was able to improve modestly on some quality indicators, but not effect resource use or patient knowledge of their disease. Changing physician and organizational behavior in academic health centers will require the development and implementation of more intensive, system-oriented strategies.

Nearly one million patients are hospitalized with community-acquired pneumonia each year in the United States, at an annual cost of $9 billion.^1,2 It is the most common infectious cause of death, the sixth-leading cause of death overall, and has an average mortality rate of 14%.^3,4 Previous work^5–8 has demonstrated substantial variations in the quality of inpatient pneumonia care between hospitals and regions on several performance measures, including antibiotic selection, use of blood cultures, and oxygenation assessment, among others. There are also unexplained geographic and local differences in the resource utilization outcomes of hospital admission rates and length of hospital stay.^9–11

A growing number of processes of pneumonia care have been associated with risk-adjusted mortality rates, and have been proposed as valid indicators of the quality of inpatient care. These include the following: antibiotic selection,^12,13 time to first dose of antibiotics,⁵ performance of blood cultures,⁵ assessment of oxygenation,⁵ and discharge prior to clinical stability.¹⁴ Appropriate and timely antibiotic use have also been associated with shorter lengths of stay.¹⁵ Many of these processes of care have been adopted as national indicators for assessing the quality of pneumonia care among physicians, hospitals, and regions.^7,8,16

The American Thoracic Society (ATS) and Infectious Diseases Society of America (IDSA) have published national practice guidelines on the management of pneumonia.^17,18 However, knowledge and familiarity with guidelines is quite variable,¹⁹ and the publication of national guidelines, while useful, are passive, educational strategies that by themselves are unlikely to change behavior.^20,21

In order to overcome some of the limitations of these more passive attempts to improve pneumonia management, several more active approaches have been undertaken. Most of the published interventions in this field have primarily focused on one aspect of management, such as the admission decision,^22–25 length of stay,^{26 –28} were aimed at lower risk patients,^26,27,29 or were limited to single institutions.^{22,26,28 –30} Several of these studies^22,23 also had the added benefit of supplemental research nurses or other staff dedicated to implementing the intervention, circumstances that may not be easily replicable or sustainable in real-world practice where resources are much more limited While these studies have produced favorable improvements in some aspects of care, no multicenter study has sought to improve a broad spectrum of clinical quality, resource use, and patient education outcomes.

The purpose of this multicenter study was to evaluate an evidence-based, multifactorial intervention to improve the quality and efficiency of inpatient pneumonia care and patient understanding of their disease using the standard quality improvement (QI) tools and resources available to the most hospitals in the United States. As such, this study sought to rigorously evaluate the effectiveness of a real-world QI initiative similar to those commonly undertaken in many US hospitals, not to measure the efficacy of an optimal, high-intensity, research intervention.

Materials and Methods

Study Population and Sites

Prior to the intervention, we identified a consecutive cohort of patients hospitalized with community-acquired pneumonia between December 1, 1999, and April 30, 2000, at four academic health centers in the New York City metropolitan area. We collected postintervention data on a matching consecutive cohort of pneumonia patients admitted from November 1, 2000, to March 31, 2001. We identified both cohorts by querying the administrative discharge records of each hospital to identify all adults with a principal or secondary discharge diagnosis of pneumonia using a previously published algorithm (International Classification of Diseases, Ninth Revision codes 480.X, 481.X, 482.X, 483.X, 485.X, 486.X, 487.X, 507.X, 510.X, and 511.X).³¹ Trained research nurses reviewed the medical records of the cases identified by the screening algorithm to determine final eligibility.

Study inclusion criteria were as follows: (1) age ≥ 18 years; (2) signs or symptoms of acute pneumonia (fever, cough, chills, sputum, chest pain, dyspnea, respiratory distress, abnormal lung examination, or elevated WBC count); and (3) radiographic evidence of pneumonia (defined as pneumonia, infiltrates, consolidation, opacities, densities, or effusion [or similar phrases] on the official radiology report). If the official radiology report was not available, we used the attending physician’s interpretation of the chest radiograph. Patients were excluded if they were hospitalized in the previous 10 days, were immunosuppressed (chemotherapy, long-term prednisone > 20 mg/d), or received parenteral antibiotics for > 24 h prior to hospital admission, or had tuberculous pneumonia, fungal pneumonia, or a lung abscess.

QI Intervention

Each hospital formed a local multidisciplinary QI team with representatives of all pertinent stakeholders, including “opinion leader” physicians (pulmonary, infectious diseases, emergency and general internal medicine), nurses, respiratory therapists, and pharmacists, among others. All QI teams operated with full institutional support, and leaders from the existing QI programs of each hospital were involved in every team. In addition, a study-wide steering committee composed of representatives from each of the four local QI committees provided the overall direction and coordination of the main intervention objectives, implementation strategies, and their evaluation.

Both the overall study committee and each of the four individual hospital QI teams reviewed the preintervention performance data and shared ideas about process improvement goals and strategies. The intervention targeted four key processes: two national quality indicators associated with lower risk-adjusted mortality (time to first dose of antibiotics, and initial antibiotic selection^5,7,8) and two efficiency markers (switch to oral antibiotics within 24 h of becoming stable and discharge within 24 h of becoming stable³²). We were also interested in making sure that efforts to reduce length of stay would not increase the proportion of patients being discharged “sicker and quicker.” Discharge prior to becoming stable has been associated with higher risk-adjusted rates of death and readmission and slower return to usual activities.¹⁴ Finally, we sought to improve patient understanding of their pneumonia care, and what to expect when they went home.

Information about the preintervention performance of each hospital on these key processes of care was reviewed in detail by all four QI teams and disseminated to target physicians as part of the provider education strategy. Based on these key objectives, the overall steering committee and the QI team of each hospital developed and implemented a local pneumonia treatment guideline designed to improve the quality and consistency of inpatient care. All sites did a series of educational lectures and dissemination sessions for attending physicians, housestaff, and nurses, and distributed pocket cards that highlighted current gaps in performance, antibiotic selection/timing recommendations, and stability criteria (for antibiotic switch and discharge decisions.) In addition, three of the sites that favored critical pathway approaches created a critical pathway for pneumonia. All QI teams made specific recommendations regarding initial admission orders (two hospitals used paper forms, and two hospital used computer order-entry systems). While these recommendations and order sets were encouraged, they were not mandatory.

To improve the patient’s understanding of his or her illness, we used a bilingual patient education handout that we developed in a previous study²⁹ that advises patients about what to expect and how to take care of their pneumonia once home. We also produced a pneumonia videotape that was available on the educational television channel of the hospital. The educational materials were to be distributed by nurses, social workers, and case managers.

Data Collection, Measurements, and Definitions

Trained research nurses abstracted detailed clinical information from the hospital medical record using computerized data collection instruments. Information on sociodemographic characteristics, initial pneumonia severity, comorbid conditions, vital signs, mental status, ability to eat, physical examination findings, laboratory results, and chest radiograph findings was collected on hospital admission. Pneumonia severity was assessed using the pneumonia severity index (PSI), a well-validated disease severity classification based on age, sex, nursing home residence, five comorbid illnesses, vital signs on admission, mental status, and seven laboratory and chest radiograph findings from presentation.³³ Class I patients have the least severe disease, and class V patients have the most severe disease. The PSI is a robust predictor of a full range of 30-day outcomes, including mortality, readmissions, and return to usual activities.^10,33–36

Processes of Care

The time to first dose of antibiotics was defined as the interval between the initial time the patient presented to the emergency department (ED) and the actual administration of the first dose of antibiotics. Consistent with ATS and IDSA guidelines, first-line antimicrobial therapy was defined as an antibiotic regimen in the first 24 h that covered both typical and atypical organisms.^17,18 These types of first-line regimens have been associated with lower risk-adjusted mortality rates. Based on the ATS and IDSA guidelines, we defined first-line initial antibiotic regimens as use of any of the following in the first 24 h of presentation: (1) respiratory fluoroquinolone, (2) advanced macrolide, or (3) cephalosporin (second, third, or fourth generation) and doxycycline or tetracycline or erythromycin.^17,18

Efficient care was defined as being switched to oral antibiotics within 24 h of stability and discharged within 24 h of stability. Patients were considered to be clinically stable when their temperature, heart rate, respiratory rate, oxygenation, ability to eat, and mental status met the following stability criteria for at least two consecutive periods at least 8 h apart.³² The stability cut point for temperature was ≤ 37.8°C (100°F), for heart rate was ≤ 100 beats per minute, for respiratory rate was ≤ 24 breaths/min, and for room oxygenation was ≥ 90% or Pao₂ ≥ 60 mm Hg.^14,32 Patients who used home oxygen prior to hospital admission were not considered to have unstable oxygenation on discharge. Patients who were able to eat (or had resumed long-term tube feeding) were counted as having stable eating status. Mental status was considered stable if the patient was either normal or, for those with chronic dementia, was back to baseline.

Clinical Outcomes

Deaths and ICU use during admission were ascertained by chart review. All patients who were discharged to home and spoke English or Spanish (or had an appropriate surrogate caregiver) received a standard telephone follow-up call 4 to 6 weeks after discharge to ascertain the patient education outcomes.

Statistical Analysis

Means ± SDs are presented for normal data, and medians with interquartile ranges are shown for nonnormal data. We screened for any differences in patient characteristics in the preintervention and postintervention cohorts using t tests for normally distributed continuous variables, Wilcoxon rank-sum tests for nonnormally distributed continuous variables, and the χ² test for categorical data. We used similar two-sample tests to compare differences in process and outcome variables between the preintervention and postintervention cohorts. All analyses used two-tailed significance levels of p < 0.05, and were conducted with SAS statistical software 8.0 (SAS Institute; Cary, NC).

Results

Using our screening algorithm, we identified 5,490 potential cases with a principal or secondary discharge diagnosis of pneumonia across both study periods. We were able to review 5,427 of these charts (98.9%) to determine eligibility. Of these, 2,109 cases met study inclusion criteria, and 2,094 cases were successfully abstracted (99.3%; 1,013 preintervention and 1,081 postintervention).

The characteristics of patients in the preintervention and postintervention cohorts are displayed in Table 1. The average age was 71.4 years; 54.6% were women, and 63.6% were white. According to the PSI, 44.1% were low risk (class I–III), 36.8% were moderate risk (class IV), and 19.2% were high risk (class V). There were no differences in age, sex, race, nursing home residence, initial PSI score, vital signs on presentation, or most major comorbidities between the preintervention and postintervention groups. Though both groups were very closely matched on the vast majority of sociodemographic and clinical characteristics, the postintervention group had modestly higher rates of Medicaid insurance, congestive heart failure, dementia, HIV, and substance use, and fewer uninsured patients.

Table 1.

Characteristics of Preintervention and Postintervention Cohorts^*

Characteristics	Preintervention (n = 1,013)	Postintervention (n = 1,081)	p Value
Age, yr	71.7 ± 17.1	71.0 ± 18.4	0.40
Female gender	55.0	54.4	0.75
Race
White	64.1	62.9	0.48
Black	15.5	15.8
Hispanic	11.2	13.4
Other	9.2	7.9
Insurance
Medicare	54.4	49.1	0.0001
Medicaid	9.0	11.4	0.07
Medicare/Medicaid	14.4	16.7	0.14
Commercial	15.4	18.4	0.08
Uninsured	4.6	2.9	0.01
Other/missing	2.2	1.6	0.31
Nursing home resident	12.3	12.0	0.78
PSI
Low (class I–III)	45.0	43.2
Moderate (class IV)	36.6	36.9	0.59
High (class V)	18.4	19.9
Diabetes mellitus	24.7	22.9	0.34
COPD	20.3	20.2	0.93
Asthma	11.3	12.5	0.38
Prior stroke	14.5	14.5	0.98
Hypertension	45.1	52.9	0.36
Coronary artery disease	33.4	31.4	0.33
Congestive heart failure	20.3	25.6	0.004
Active cancer	4.8	4.9	0.94
Renal insufficiency	9.7	10.3	0.64
Liver disease	2.3	2.2	0.94
Dementia	17.1	21.4	0.01
HIV disease	2.8	4.9	0.01
Alcohol dependence	3.6	9.4	0.0001
IV drug use	1.8	4.1	0.002
Initial presentation
Temperature, °C	37.6 ± 1.2	37.5 ± 1.3	0.09
Heart rate, beats/min	99.6 ± 21.7	98.4 ± 21.7	0.21
Systolic BP, mm Hg	134.2 ± 28.6	132.7 ± 26.5	0.23
Diastolic BP, mm Hg	71.9 ± 17.1	71.4 ± 16.0	0.52
Respiratory rate, breaths/min	23.9 ± 7.4	23.4 ± 7.0	0.13
O2 saturation, %	92.8 ± 6.3	92.9 ± 6.3	0.69
WBC count	13,278 ± 7,897	13,264 ± 7,380	0.96

^*Data are presented as mean ± SD or %; n = 2,094.

The effects of the intervention on a variety of performance measures and outcomes are displayed in Table 2. Improvements were seen in two of the quality measures, but none of the efficiency indicators. Overall, 5 patients never received any antibiotics during their hospital stay, and 61 patients received their first dose after the initial 24 h. The intervention increased the use of guideline-recommended antimicrobial therapy from 78.1 to 83.4% (p = 0.003), a difference that appeared to largely increase over the course of the intervention (Fig 1). The most common regimen was a respiratory fluoroquinolone alone (39.6%), fluoroquinolone and a nonmacrolide (15.7%), and a cephalosporin/macrolide combination (11.6%). Overall, 63.8% of initial regimens included a fluoroquinolone and 24.4% included a macrolide. Modifying the definition of first-line therapy to include fluoroquinolones or the combination of a cephalosporin plus a macrolide or doxycycline produced a similar intervention effect (67.2% before intervention vs 71.5% after intervention, p = 0.03). However, there was no improvement in the time to first dose of antibiotics or the proportion of patients receiving the first dose within 8 h (or 4 h) of arrival. Nor were there any changes in the blood culture measures.

Table 2.

Effects of Intervention on Quality-of-Care Outcomes^*

Performance Measures	Preintervention (n = 1,013)	Postintervention (n = 1,081)	p Value
Quality-of-care measures
First-line initial antibiotic therapy	78.1	83.4	0.003
First dose of antibiotics ≤ 4 h	57.1	59.5	0.29
First dose of antibiotics ≤ 8 h	73.6	74.0	0.82
Time to first dose of antibiotics, h	4 (2–7)	4 (2–7)	0.25
Blood cultures obtained	86.2	86.3	0.94
Blood cultures before antibiotics	78.2	80.6	0.18
Days to clinical stability	3 (2–5)	3 (1–5)	0.51
Discharged prior to stability	27.0	23.5	0.06
Inpatient mortality	10.9	11.3	0.76
ICU/coronary care unit stay	14.3	11.7	0.07
Resource use measures
Length of stay, d	6 (4–10)	6 (4–10)	0.83
Switch to oral antibiotics ≤ 24 h of stability	57.7	60.7	0.24
Discharge ≤ 24 h of stability	29.7	26.3	0.13
Days from stability to discharge	3 (1–7)	4 (2–7)	0.07
Patient education measures
Knew what medicines to take on discharge	85.2	85.3	0.96
Took all antibiotic pills at home	80.9	84.1	0.29
Told about medication side effects	44.7	43.1	0.69
Told danger signals of worsening	38.7	33.9	0.17
Knew how long it would take to feel better	32.4	30.4	0.45

^*Data are presented as % or median (25 to 75th percentile).

Figure 1.

Effect of intervention on first-line antibiotic therapy over time. The arrow divides the preintervention from the postintervention periods. There was also a significant reduction in the monthly variation on this measure (preintervention vs postintervention, p < 0.003)

While patients during both periods took a median of 3 days to become clinically stable, the proportion of patients being discharged prior to attaining clinical stability after the intervention decreased from 27.0 to 23.5%, although this was of borderline significance (p = 0.06). There were no improvements in any of the targeted resource use outcomes including length of stay (median, 6 days; interquartile range, 4 to 10 days), timely switch to oral antibiotics, and timely discharge once clinically stable.

We successfully interviewed 326 of 576 patients (63.2%) or proxies eligible for the follow-up survey during the preintervention period, and 461 of 649 patients (71.0%) after the intervention. Twelve percent of interviews were done in Spanish, and one third were done with proxies. There was no effect of the intervention on the patient education outcomes we targeted, including knowing what medications to take on discharge, taking all of the antibiotic pills at home, being told about medication side effects, being told about danger signals, or knowing how long it would take to feel better (Table 2).

While there were a few small baseline differences in performance among the four participating hospitals, the effects of the intervention were largely consistent across sites. Since many of the components of the QI strategy were rolled out over time, we also looked for, but did not find, any evidence that there was a trend toward greater improvements over time during the intervention period.

Discussion

In this multicenter effectiveness trial, we sought to rigorously evaluate the impact of a multifaceted, evidence-based intervention that employed many commonly used strategies for improving quality and reducing the costs of inpatient care. We assembled a multidisciplinary QI team of opinion leaders to identify the best evidence from the literature, collected local clinical data on key performance measures, fed back these data to the teams, identified opportunities for improvement, and implemented real-world strategies for changing physician behavior. Specific techniques included the development and dissemination of hospital-specific, evidence-based practice guidelines and critical pathways, educational sessions with attending physicians and houseofficers, distribution of pocket reminder cards, and use of standardized orders sets and bilingual patient education materials.

The good news of the study was that we were able to demonstrate modest improvements in two measures of the quality of care—increases in the use of guideline-recommended antibiotics and lower rates of discharge prior to clinical stability—two process of care associated with lower risk of death. The bad news was that there were no improvements on the three key resource use outcomes that were targeted (length of stay, timely switch to oral antibiotics, and timely discharge) or the patient education indicators we assessed. If anything, there was a trend toward slower time from stability to discharge.

We think this mixed result can be viewed as an example of what can and cannot readily be accomplished with typical QI efforts in academic health centers. Our two areas of success (antibiotic selection and discharge prior to stability) shared two common elements. First, they are both process measures for which there was high quality evidence linking performance of that indicator to risk-adjusted mortality rates.^12–14 Physicians may be more likely to change their behavior when the evidence strongly suggests that this will directly benefit their patients. Second, both of these processes of care are directly and independently controlled by physicians. The other aspects of care that we sought to influence but did not succeed in changing (time to first dose of antibiotics, timely switch to oral antibiotics, timely discharge, length of stay, and patient knowledge) are all dependent on a more complex series of interactions among several individuals and systems over which an individual physician may have less control. For example, several upstream processes must be accomplished in a timely fashion for a patient to receive antibiotics within a few hours of presentation, including timely triage, initial physician evaluation, chest radiograph performed and reviewed, diagnosis of pneumonia made, antibiotics ordered, and antibiotics administered. It was very difficult to expedite all of these processes for patients specifically with pneumonia when there were many other competing demands in the large, busy EDs of the hospitals we studied. Similarly, discharging patients in a timely fashion is a complex process with multiple determinants beyond just an evaluation of pneumonia stability, including psychosocial, home support, and other discharge planning issues. While we tried to address on a hospital-specific basis ways of overcoming system-level barriers to improving performance and efficiency, these were clearly challenging problems.

Of note, we found no difference in some of the secondary quality measures that we assessed but did not expect to change because we did not directly target them with strategies for change such as performance and timing of blood cultures, time to stability, and mortality. We performed some secondary analyses (data not shown) to try to investigate whether our failure to improve in certain areas was because there was a lag time while the full intervention was rolled out. Unfortunately, a post hoc time series analysis did not pick up a late benefit to support the “delayed institutional learning curve” hypothesis.

The results of other attempts to improve the quality and efficiency of inpatient pneumonia management have been mixed. The only randomized controlled trial²³ found that hospitals in Canada randomized to have a research staff help implement a pneumonia critical pathway had higher rates of recommended antibiotic use, fewer days of IV antibiotics, and shorter lengths of stay. A statewide pneumonia QI initiative found overall decreases in time to initial dose of antibiotics and length of stay; however, there was significant variation among sites with some hospitals having improved their performance, some remaining unchanged, and others worsening.³⁷ Implementation of a health system-wide pneumonia guideline was associated with lower mortality rates compared to non-guideline hospitals in Utah.³⁸ Use of a pneumonia guideline in an health maintenance organization urgent-care setting increased use of recommended antibiotics and reduced length of stay.²⁵ Use of a pneumonia clinical pathway in one ED resulted in decreased time to initial antibiotic treatment.³⁰ A QI intervention focused on very small, rural hospitals demonstrated improvements in timely administration of the first dose of antibiotics, although no change in length of stay.³⁹

A recent study⁴⁰ of a multifaceted QI intervention similar to ours among 12 nonacademic hospitals reported mixed results. They found improvements in the proportion of patients receiving antibiotics within 4 h, but not within 8 h. Overall length of stay significantly declined, although this was due to a drop in length of stay after conversion to oral antibiotics, without any change in the time to switch to oral antibiotics. Several other pneumonia intervention studies were not able to effect timely discharge or hospital length of stay^26–28,39 or time to first dose of antibiotics.²⁸ Our previous, single hospital study²⁹ demonstrated a significant decrease in the time from stability to switch to oral antibiotics and number of days of IV therapy and increase in patient education outcomes, but no reduction in length of stay. There is also likely a negative publication bias, whereby interventions lacking an impact are less likely to be submitted and/or published.

It appears that the interventions reported in the literature that have been the most successful have tended to be ones based in single institutions, small hospitals, or nonacademic health centers. It may be that changing physician and organizational behavior in large, academic medical centers is especially challenging due to the larger number of physicians, residents, and nurses, complex working processes, and greater fragmentation of care.

Another potential explanation for some of our negative results was that our implementation strategies, although well intentioned, were not well executed. However, we did employ widely used organizational approaches to improving hospital quality that are generally believed to be effective, especially when used in combination.^41–43 The fact that we did achieve some significant, though modest improvements also makes the “poor strategy” explanation less likely. However, we did not have staff who were primarily focused on promoting implementation of all guideline elements on a patient-by-patient basis—a more resource intensive strategy that has worked elsewhere.

Several methodologic limitations are worth acknowledging. We used a prospective, time series design with no concurrent control or randomization. However, preintervention and postintervention patients appeared to have been well matched on most key patient characteristics, including sociodemographics, pneumonia severity, and most comorbidities. While secular trends are always a concern with designs such as ours, they would not readily explain the mixed results we found. Our results may not be generalizable to other settings because all participating sites were academic health centers in one metropolitan area. However, the process of care performance levels of the participating institutions were comparable to those reported in a recent nationally representative sample of Medicare inpatients.⁷ The predominant use of once a day antibiotic agents (new fluoroquinolones and azithromycin) may partly explain our inability to change the proportion of patients being switched to an oral antibiotic within 24 h of reaching stability, since the fastest this switch can be made is 24 h.

In conclusion, this real-world QI intervention was able to improve modestly on some quality measures, but not effect resource use or patient knowledge of their disease. As such it represents a cautionary tale of what may and may not be achieved by the typical QI initiatives that are undertaken in many hospitals nationwide. Our data also suggest that changing physician and organizational behavior in academic health centers will require the development and implementation of more intensive, system-oriented approaches.

Abbreviations

ATS

American Thoracic Society

ED

emergency department

IDSA

Infectious Diseases Society of America

PSI

pneumonia severity illness

QI

quality improvement