Does Quality Improvement Work? Evaluation of the Organ Donation Breakthrough Collaborative

Abstract

Objective

The Organ Donation Breakthrough Collaborative is a quality improvement initiative to encourage adoption of “best practices” for identifying potential donors and obtaining consent for deceased organ donation. We evaluate the impact of the first phase on organ donation rates.

Setting

We study donation rates in the 95 hospitals that participated in the first phase and a control group of 125 hospitals.

Design

We use a controlled pre/post design. The preperiod is the year before the start of the Collaborative (September 2002 to August 2003), the postperiod is the final 6 months of the first phase (March 2004 to August 2004).

Data

We use administrative data from the Organ Procurement and Transplantation Network to compute the conversion rate in each hospital group and time period. The conversion rate is the proportion of eligible donors who became actual donors.

Principal Findings

Preperiod conversion rates in Collaborative and control hospitals were similar: 52 and 51 percent, respectively. In the postperiod, the conversion rate increased to 60 percent among Collaborative hospitals and remained at 51 percent among control hospitals. The relative change was 8 percentage points (95 percent confidence interval: 2–13: p<.001).

Conclusions

Our findings suggest that the Breakthrough Collaborative led to an increase in donation rates at participating hospitals.

Every year there are thousands of decedents who could donate organs but do not (Sheehy et al. 2003). Many families of potential donors refuse consent, others are never asked in the first place. According to a recent estimate, the number of life years lost as a result of the failure to transplant all suitable organs is comparable with the years lost as a result of suicide, homicide, or perinatal conditions (Schnitzler et al. 2005). Frustration with the growing imbalance between organ demand and supply has led policy makers to contemplate new strategies for promoting organ donation, including financial incentives for donor families and use of a “presumed consent” regime (Howard, in press). A recent Institute of Medicine report recommended against these policies (Committee on Increasing Rates of Organ Donation 2006).1 The report cited the Organ Donation Breakthrough Collaborative, a nationwide quality improvement initiative, as evidence that it is possible to increase donation rates within the confines of the current system.

While initial findings from the Breakthrough Collaborative are promising, neither the Breakthrough Collaborative nor other large scale quality improvement programs to promote donation (Burris and Jacobs 1996; Plessen et al. 1997; Bozzi et al. 2004; Tokalak et al. 2005) have been rigorously evaluated. This study fills the gap on the effectiveness of quality improvement in organ procurement by estimating the impact of the Organ Donation Breakthrough Collaborative on donation rates. Our evaluation is limited to the first phase of the Collaborative, which took place from September 2003 to August 2004 and involved 95 hospitals in 43 organ procurement organizations (OPOs).

BACKGROUND

The Organ Donation Breakthrough Collaborative is an ongoing, hospital-based quality improvement initiative funded by the Division of Transplantation in the Health Resources and Services Administration and developed in collaboration with the Institute for Healthcare Improvement and Quality Reality Checks Inc. Before the Collaborative, the Division of Transplantation retained the Lewin Group to identify best practices for organ donation. The results of this study were used to develop a “Change Package” recommending steps hospitals and OPOs can take to increase donation rates.

All 58 OPOs were invited to participate in the Collaborative. OPOs are the regional entities that oversee the consent and organ removal process. OPOs have exclusive territories. Most roughly correspond to state boundaries, but some OPOs operate in multiple states and some states have multiple OPOs.2

At the time of the invitation, each OPO was provided with the conversion rates of the 300 hospitals in the United States with the largest numbers of potential donors. The conversion rate is the proportion of potential donors from whom organs are recovered. OPOs were asked to invite hospitals whose conversion rates needed improvement. Hospitals were free to accept or decline the invitation. Because the Collaborative was not designed as a research study, OPOs did not have to follow a formal protocol when selecting invitees or record which hospitals declined to participate or volunteered to participate without being asked.

Forty-three OPOs and 95 hospitals agreed to participate. Typical of hospitals with the greatest numbers of potential donors, most were large, urban facilities. Of the 95 participating hospitals, 84 (88 percent) were ranked in the top 300 hospitals in terms of the total number of eligible donors in 2002. Despite the Collaborative's admonition to OPOs to enroll hospitals that needed to improve, conversion rates in 2002 among participating hospitals and nonparticipating hospitals in the top 300 were similar (52 percent). Some hospitals with low conversion rates declined to participate; some hospitals with high conversion rates enrolled to achieve even higher rates.

The intervention, which began in September of 2003, consists primarily of a series of “learning sessions,” attended by teams of hospital and OPO staff, dissemination of printed materials (the “Change Package”), and feedback on donation rates (Organ Transplantation Change Package 2006). Some recommendations are fairly general (e.g., “Involve senior leadership to get requests”), but others are direct and specific (e.g., “Track consent rates of all requestors and consistently deploy effective staff accordingly”). Although the purpose of the Collaborative is to spread “best practices,” there is a conscious effort to avoid being overly prescriptive and an emphasis on tailoring practices toward local institutions and capabilities.3

METHODS

Study Design

We use a controlled pre/post study design. The preperiod is the year before the start of the Breakthrough Collaborative, September 1, 2002 to August 31, 2003. In the baseline analysis, the “post”-period is the final 6 months of the first phase of the Breakthrough Collaborative, March 1, 2004 to August 31, 2004. The first phase was ongoing during this time. Using a later postperiod would be problematic because the second phase began immediately after the first. During the second phase, OPOs began to involve many nonparticipating hospitals in Collaborative activities.

Data

Data on actual organ donors (decedents from whom at least one solid organ was recovered) and the number of eligible (or potential) donors by hospital by month were obtained from the Organ Procurement and Transplantation Network (OPTN). Eligible donors are decedents below age 70 who were declared dead based on neurological criteria (i.e., brain death) and who do not have a history of cancer or HIV. In 1998, the Health Care Financing Administration required hospitals participating in Medicare to notify their local OPOs of all in-hospital deaths (or imminent deaths). OPOs determine which decedents are classified as eligible donors, subject to the criteria described above. Because OPOs are required to report information on actual and eligible donors to the OPTN, the data include the universe of actual and identified eligible donors in the United States.4 Data on historical trends in the number of organ donors were obtained from the OPTN website (OPTN 2006).

Sample Construction

The intervention group consists of the 95 hospitals that participated in the first phase of the Breakthrough Collaborative (“the Collaborative”). These hospitals were located in 43 OPOs, which also participated in the first phase.

The control group consists of hospitals meeting three criteria: (1) the hospital is located in the region of an OPO that did not participate in the Collaborative, (2) the hospital had at least one actual donor in the preperiod, (3) the hospital had at least four eligible donors in the preperiod (all but one of the 95 hospitals participating in the Collaborative had at least four eligible donors5). One-hundred-and-twenty-five hospitals met all three criteria. We also compared Collaborative hospitals with the subset of the 125 control hospitals that participated in the second phase of the Breakthrough Collaborative (N=19).

Outcome Variable

For each group of hospitals (Collaborative and control), we calculated “conversion rates” by period. The conversion rate is simply the number of actual eligible donors in the hospital group divided by the number of total eligible donors in the group. Note that the outcome variable is not the average of conversion rates across hospitals, which would give the same weight to a hospital with few eligible donors as to a hospital with a large number of eligible donors.

Statistical Analysis

We examined differences in the characteristics of Collaborative and control hospitals using t-tests for continuous variables and χ² tests for dichotomous variables. We estimated the “treatment effect” of the Collaborative on the conversion rate by subtracting the pre/postdifference in the conversion rate among control hospitals from the pre/postdifference in the conversion rate among Collaborative hospitals (i.e., the “difference-in-difference” estimator). To compute 95 percent confidence intervals, we estimated a logistic regression where the eligible donor was the unit of analysis and the dependent variable was equal to one if the eligible donor became an actual donor. Independent variables include a binary variable for period (pre versus post), a binary variable for group (Collaborative versus control), an interaction term between period and group, and a random effect for hospital to control for clustering at the hospital level. The random effects are hospital-specific intercept terms with an expected value of 0. Because the data on eligible donors are reported on an aggregated basis, we were unable to include donor characteristics as covariates. Using the coefficients from the logistic regression, we (1) computed predicted conversion rates by period and by hospital group, (2) constructed the difference-in-difference estimator using these predicted values, and (3) used the delta method (Oehlert 1992) to compute 95 percent confidence intervals around the observed difference-in-difference estimate. The delta method is used for computing standard errors and confidence intervals for nonlinear combinations of parameters. We estimated a separate logistic model to compare conversion rates in Collaborative hospitals and the 19 control hospitals that participated in the second phase of the Breakthrough Collaborative.

To examine the impact of the Collaborative on trends in conversion rates, we estimated the likelihood of donation as a function of a time variable, a postperiod indicator variable, and an interaction between time and period. For this analysis, we defined the postperiod as the entire first phase of the Collaborative, September 1, 2003 to August 31, 2004. The model included a random effect for hospital.

RESULTS

Table 1 displays hospital characteristics by group. Compared with the 125 control hospitals, Collaborative hospitals were more likely to be members of the Council of Teaching Hospitals (p<.001) and offer transplant services (p<.001). The average conversion rate among Collaborative hospitals did not differ from the average conversion rate among the control hospitals (p=.27) but Collaborative hospitals had larger numbers of eligible donors in the preperiod (p<.001). Collaborative and control hospitals were located in OPOs with similar average conversion rates (p=.17) but Collaborative hospitals were located in OPOs with greater numbers of eligible donors (p<.001), mirroring the hospital-level differences. Collaborative hospitals were similar in most respects to the subset of 19 control hospitals that participated in the second phase of the Breakthrough Collaborative, except that Collaborative hospitals were located in OPOs with larger numbers of eligible donors (p=.004).

Table 1.

Hospital Characteristics by Group

		Control Hospitals
	Collaborative Hospitals (N=95)	All (N=125)		BC 2 (N=19)
	Mean (SD)	Mean (SD)	p-Value	Mean (SD)	p-Value
Teaching hospital, N (%)	46 (48)	16 (13)	<.001	9 (47)	.93
Transplant hospital, N (%)	55 (58)	24 (19)	<.001	13 (68)	.40
Preperiod conversion rate	0.52 (0.16)	0.49 (0.22)	.27	0.52 (0.10)	.99
Preperiod eligible donors	24 (14)	14 (13)	<.001	27 (16)	.51
OPO-level preperiod conversion rate	0.54 (0.09)	0.53 (0.08)	.17	0.52 (0.06)	.25
OPO-level preperiod eligible donors	239 (144)	177 (57)	<.001	141 (53)	.004

BC 2, control hospitals that participated in the second phase of the Breakthrough Collaborative; OPO, organ procurement organization; SD, standard deviation.

Figure 1 shows the smoothed trend in the conversion rate among Collaborative and control hospitals. The conversion rate among Collaborative hospitals increased during the first phase of the Breakthrough Collaborative and continued increasing during the second phase. The conversion rate among control hospitals was fairly constant during the first phase but increased steeply during the second phase.

Figure 1.

Smoothed Trend in the Conversion Rate among Hospitals Participating in the First Phase of the Breakthrough Collaborative and Control Hospitals.

Table 2 displays the main finding from our study: the pre/postdifference in conversion rates. In the preperiod, there were 102 actual donors per month in Collaborative hospitals (note that this is a total, not a hospital-level average) and 195 eligible donors per month. The conversion rate was 52 percent (=102/195). There were 77 actual donors and 151 eligible donors per month in the control hospitals. The conversion rate, 51 percent (=77/151), is not statistically different from the preperiod conversion rate in Collaborative hospitals (p=.68). In the postperiod, the conversion rate among Collaborative hospitals was 60 percent and 51 percent among control hospitals (p<.001). The difference in the change in conversion rates between Collaborative and control hospitals is 8 percentage points (p<.001). The difference in the change (or the “difference-in-difference”) provides an estimate of the impact of the Breakthrough Collaborative on the conversion rate.

Table 2.

Pre- and Postperiod Conversion Rates in Collaborative and Control Hospitals

	Collaborative Hospitals	Control Hospitals	Difference [95% CI]	p-Value
Preperiod (9/1/2002–8/31/2003)
Actual donors per month	102	77
Eligible donors per month	195	151
Conversion rate	0.52	0.51	0.01 [−0.03 to 0.05]	.68
Postperiod (3/1/2004–8/31/2004)
Actual donors per month	118	80
Eligible donors per month	196	156
Conversion rate	0.60	0.51	0.09 [0.04–0.13]	<.001
Change	0.08	<.01	0.08 [0.02–0.13]	<.001

Estimates of the impact of the Breakthrough Collaborative were moderately sensitive to the time period used as the postperiod in the analysis. When we define the postperiod as the final 3 months of the first phase of the Breakthrough Collaborative, instead of the final 6 months, the difference in the change in conversion rates between Collaborative and control hospitals is 9 percentage points (p=.01).

Table 3 displays the same analysis but restricts the control group to the 19 hospitals that participated in the second phase of the Breakthrough Collaborative. Conversion rates were similar in the preperiod, 52 versus 50.5 percent (p=.63), but differed in the postperiod, 60 versus 51.8 percent (p=.045). The difference in the change is 7 percentage points, but it is not significantly different from zero at conventional levels (p=.12). The coefficient on the Collaborative/postperiod interaction term in the logistic regression is borderline significant (p=.057). When we define the postperiod as the final 3 months of the first phase of the Breakthrough Collaborative, instead of the final 6 months, the difference in the change in conversion rates is 11 percentage points (p=.052).

Table 3.

Pre- and Postperiod Conversion Rates in Collaborative and Control Hospitals That Participated in the Second Phase of the Breakthrough Collaborative

	Collaborative Hospitals	BC 2 Hospitals	Difference [95% CI]	p-Value
Preperiod (9/1/2002–8/31/2003)
Actual donors per month	102	22
Eligible donors per month	195	43
Conversion rate	0.52	0.50	0.02 [−0.05 to 0.08]	0.63
Postperiod (3/1/2004–8/31/2004)
Actual donors per month	118	24
Eligible donors per month	196	47
Conversion rate	0.60	0.52	0.08 [0.00–0.16]	0.045
Change	0.08	0.01	0.07 [−0.02 to 0.15]	0.12

BC 2, Control hospitals that participated in the second phase of the Breakthrough Collaborative.

The impact of the Collaborative varied at the hospital level. Of the 95 participating hospitals, 64 (67 percent) experienced an increase in conversion rates and 31 (33 percent) experienced a decrease. Of the 125 control hospitals, 55 (44 percent) experienced an increase and 70 (56 percent) experienced a decrease. Among Collaborative hospitals, the average change in conversion rates and 25th and 75th percentiles are 8, −4, and 21 percentage points, respectively. The comparable figures for control hospitals are 0, −20, and 22 percentage points. The t-statistic associated with the null hypothesis that the change in conversion rates was similar across Collaborative and control hospitals is −1.959 (p=.051).

Estimates of the impact of the Collaborative on trends in donation rates in Collaborative hospitals were sensitive to time period. In the baseline model, the coefficient on the interaction of time with period was near zero, 0.008 (p=.26). When we used a longer preperiod, January 2002 to August 2003, the coefficient was 0.027 (p=.044). Among control hospitals, the trend in donation rates was negative in both the preperiod, regardless of how it was defined, and the postperiod.

Table 4 shows trends in the number of organ donors nationally, including donors from Collaborative hospitals (which account for a large share of all donors), nonparticipating hospitals located in participating OPOs, and nonparticipating hospitals in nonparticipating OPOs. These figures also include both “eligible” and “ineligible” donors. From 1991 to 2003, the annual increase in the number of organ donors from the prior year was never greater than 7.5 percent, and never greater than 4.3 percent from 1999 to 2003. In 2004, the first year of the Breakthrough Collaborative, the number of organ donors increased by 10.7 percent. The number of donors increased by 6.2 percent between 2004 and 2005.

Table 4.

Change and Percent Change in the Number of Organ Donors

Year	Donors	Change	Percent Change (%)
1990	4,509
1991	4,526	17	0.4
1992	4,520	−6	−0.1
1993	4,861	341	7.5
1994	5,099	238	4.9
1995	5,362	263	5.2
1996	5,416	54	1.0
1997	5,478	62	1.1
1998	5,793	315	5.8
1999	5,824	31	0.5
2000	5,985	161	2.8
2001	6,080	95	1.6
2002	6,190	110	1.8
2003	6,458	268	4.3
2004	7,149	691	10.7
2005	7,594	445	6.2

COMMENT

Using a controlled pre/post study design, we compared conversion rates between hospitals that participated in the first phase of the Organ Donation Breakthrough Collaborative and control hospitals. The relative increase in the conversion rate in Collaborative hospitals between the preperiod and the final months of the first phase was large—8 percentage points—and statistically significant. The relative increase in the conversion rate between Collaborative hospitals and the 19 control hospitals that participated in the second phase of the Breakthrough Collaborative was also large −7 to 11 percentage points—but the significance varied depending on whether the postperiod was the final 6 or final 3 months of the first phase. Nationwide, there were large increases in the total number of organ donors during the first phase. The conversion rate was rising in Collaborative hospitals before the start of the Collaborative. It is unclear whether the conversion rate would have continued to increase in the absence of the Collaborative. The decline in the conversion rate during the first months of the Collaborative suggests that it would have not. Based on the relative change in conversion rates, we conclude that the Breakthrough Collaborative led to an increase in donation rates at participating hospitals.

It remains to be seen whether the success of the Breakthrough Collaborative is sustainable or will dissipate over time. The experience to date is encouraging. It is also unclear whether quality improvement can yield additional gains in the conversion rate or whether policy makers should consider more fundamental changes to the organ procurement system.

This study adds to the growing body of literature on quality improvement efforts in health care (Berwick 1989; Laffel and Blumenthal 1989; Blumenthal and Kilo 1998; Shortell, Bennet, and Byck 1998). Continuous quality improvement is widely used in hospitals and other large health care organizations, but is often difficult to evaluate in a rigorous manner (Wilson, Berwick, and Cleary 2003; Shojania and Grimshaw 2005). The incremental roll-out of the Breakthrough Collaborative facilitated evaluation using concurrent controls.

Our results are subject to a number of caveats. Collaborative hospitals differed from control hospitals along a number of dimensions. These differences will not impart bias to our estimates as long as they are related only to the level of conversion rates, not the rate of change. The comparison of Collaborative hospitals and the subset of control hospitals that participated in the second phase, which are similar to Collaborative hospitals in most respects, suggests that bias due to differences in time-invariant hospital characteristics is minimal or nonexistent.

Internal validity may be compromised by regression-to-the-mean. If hospitals that experienced a random, temporary decline in conversion rates were more likely to participate in the Breakthrough Collaborative, then conversion rates in these hospitals would have increased as donations rebounded to normal levels, giving the false impression that the Collaborative was a success. However, two observations are inconsistent with regression-to-the-mean as an explanation for the observed effect. First, the conversion rate was slightly higher in Collaborative versus control hospitals during the preperiod. Second, the conversion rate increased in control hospitals once the Breakthrough Collaborative intervention was applied on a much wider scale during the second phase.

The external validity of the study may be limited by self-selection of hospitals into the first phase of the Collaborative. The decision to participate in a quality improvement program is revealing (Wilson, Berwick, and Cleary 2003; Westphal, Gulati, and Shortell 1997), and we can speculate that administrators and staff at participating hospitals are more open to change and have more favorable attitudes toward organ donation than administrators and staff at non-Collaborative hospitals. Also, Collaborative hospitals have large numbers of eligible donors, and so the potential returns to investing in efforts to increase donation are higher.

Because all of the Collaborative hospitals participated fully in all aspects of the intervention, we are unable to identify the contribution of specific components of the Collaborative to the increase in donation rates. Informal conversations with transplant program and organ procurement personnel indicate that the Breakthrough Collaborative succeeded mainly by better focusing the mission of OPOs and providing a structural mechanism (O'Brien, Shortell, and Hughes 1995) for dissemination of best practices. It is difficult for organizations to change unless they are aware of variations in processes or outcomes (Jha and Schneider 2004).

Also, the Collaborative increased the commitment of hospital staff and administrators to organ donation, consistent with past research emphasizing the importance of leadership and physician buy-in to the success of continuous quality improvement efforts in health care (Shortell, Bennet, and Byck 1998; Bradley et al. 2001) Traditionally, hospital administrators and physicians viewed organ donation efforts warily based on the costs and potential for requestors to upset decedents' families. The “culture” of a hospital is critical to the success of organ procurement efforts. Hospitals that are oppositional or neutral to this effort or whose leadership fails to integrate it into the central mission of the hospital will not obtain consent for organ donation except from the most committed of donor-eligible patient families.

We have not undertaken a formal cost–benefit analysis of the Breakthrough Collaborative, but preliminary calculations suggest that the program was cost-effective. Applying our estimate of the increase in conversion rates, 8.0 percentage points, to the number of eligible donors per month in the postperiod, 196, we calculate that the Collaborative yielded an increase of 15.7 actual donors per month. Assuming that a typical donor is associated with a gain of 30.8 life years to recipients (Schnitzler et al. 2005),6 the Collaborative led to a gain of over 5,700 life years annually. The budget of the Collaborative is only $3 million per year. These calculations probably understate the benefits of the Collaborative because they do not account for (1) increases in donation rates among non-Collaborative hospitals, (2) increases in donation rates during the second phase, (3) increases in the number of organs obtained from each donor, and (4) increases in donation rates among decedents who are not “eligible” donors (e.g., decedents who were declared dead based on cardiac, rather than neurological, criteria).