Developing a Benchmarking Process in Perfusion: A Report of the Perfusion Downunder Collaboration

Abstract:

Improving and understanding clinical practice is an appropriate goal for the perfusion community. The Perfusion Downunder Collaboration has established a multi-center perfusion focused database aimed at achieving these goals through the development of quantitative quality indicators for clinical improvement through benchmarking. Data were collected using the Perfusion Downunder Collaboration database from procedures performed in eight Australian and New Zealand cardiac centers between March 2007 and February 2011. At the Perfusion Downunder Meeting in 2010, it was agreed by consensus, to report quality indicators (QI) for glucose level, arterial outlet temperature, and pCO₂ management during cardiopulmonary bypass. The values chosen for each QI were: blood glucose ≥4 mmol/L and ≤10 mmol/L; arterial outlet temperature ≤37°C; and arterial blood gas pCO₂ ≥ 35 and ≤45 mmHg. The QI data were used to derive benchmarks using the Achievable Benchmark of Care (ABC™) methodology to identify the incidence of QIs at the best performing centers. Five thousand four hundred and sixty-five procedures were evaluated to derive QI and benchmark data. The incidence of the blood glucose QI ranged from 37–96% of procedures, with a benchmark value of 90%. The arterial outlet temperature QI occurred in 16–98% of procedures with the benchmark of 94%; while the arterial pCO₂ QI occurred in 21–91%, with the benchmark value of 80%. We have derived QIs and benchmark calculations for the management of several key aspects of cardiopulmonary bypass to provide a platform for improving the quality of perfusion practice.

Improving and understanding perfusion practice is an appropriate goal for the perfusion community. Steps required to achieve this goal include the prospective collection and reporting of data that defines our practice, promotion of research and development in perfusion, the use of continuous quality improvement processes based on principles of evidence based medicine, adoption of best practice guidelines, and development of perfusion practice benchmarks. The Perfusion Downunder Collaboration (PDUC) has established a multi-center perfusion focused database aimed at helping the Australian and New Zealand perfusion community to achieve these goals (1). Benchmarks provide a foundation for quality improvement, and have been adopted in cardiac surgery by professional organizations such as the Society of Thoracic Surgeons and the Australasian Society of Cardiac Thoracic Surgeons (ANZSCTS). The ANZSCTS published their early observations with their dataset and goals of developing performance indicators and benchmarks in 2004 (2); this early work has led to the development of the AUSCORE risk prediction model (3) and most recently an Australasian risk prediction model for aortic valve surgery (4).

What is Benchmarking?

Benchmarking involves using a structured method of measurement to identify standards of practice excellence. The purpose of which in the healthcare setting is to improve practice and quality. Benchmarking is a continuous process by which we can measure and compare our practice to those with best standards “leaders” (5,6). There are different measures that can be related to structure, process, or outcome (7); we will focus in this paper on those which measure compliance with processes of care. These typically may reflect recommended practices, for example in a post myocardial infarct population, administration of various therapies could be measured such as thrombolytic therapy, use of beta blockers, or aspirin (5). In the domain of cardiopulmonary bypass (CPB) we may wish to look at process of care measures such the adherence to the conduct of bypass within an agreed physiological range.

What Quality Indicators Should We Use?

The choice and definition of benchmarks should be based on the following principles: they should be clinically relevant; the required information should be routinely (or at least readily) collected; and they should have the potential to be the focus for quality improvement initiatives. The development of collaborative datasets, such as the PDUC dataset or the International Consortium for Evidence Based Perfusion dataset will assist in providing one important aspect for this process, the generation of relevant and concurrently valid clinical practice data with tightly controlled data definitions. The aim of this report is to define three quantitative quality indicators for perfusion practice and provide benchmark data as a platform for clinical practice improvement.

METHODS

Data Collection

Data were collected using the PDUC Database (PDUCD) as previously described (1). The PDUCD dataset is divided into six categories: demography, clinical (medical history, etc.), perfusion (CPB related data), procedural (surgical data), quality control (related to conduct of CPB), and outcome data. The database was designed to suit multiple institutions’ ability to access electronic data collection at the point of care. With the approval of local ethics committees,1 data were collected from procedures performed in eight Australian and New Zealand cardiac centers between March 2007 and February 2011 (see Appendix 1). A site coordinator was appointed from within each center with the responsibility of installation, configuration, and coordination of the database. The dataset for QI and benchmarking was generated from procedures performed in patients ≥18 years of age undergoing isolated on pump coronary artery bypass graft (CABG), isolated valve repair and/or replacement, and valve/CABG procedures. Both first time and redo procedures were included. Electronic data variables were collected every 20 or 30 seconds during CPB in seven centers using the Data Management System (Stockert, Munich, Germany), and every minute at one center using the JOCAP XL (MAQUET, Rastatt, Germany). Blood gas, hemoglobin, and glucose data were collected electronically with intermittent sampling. A subset of variables was selected to facilitate comparison with published ANZSCTS database data, which reported data collected from 8061 procedures in six cardiac surgical centers in Victoria, Australia (8). Continuous physiological and perfusion variables are not collected in the ANZSCTS database. Clinical data definitions were the same as those reported by ANZSCTS database, except for recent myocardial infarction (MI), which is defined in the PDUC dataset as an MI within 90 days prior to surgery compared with 21 days in the ANZSCTS dataset. The PDUC data are reported for periods March 2007–April 2008, May 2008–April 2009, May 2009–April 2010, and May 2010–February 2011, while ANZSCTS data is reported for financial years (July 1–June 30) 2007–2008, 2008–2009, and 2009–2010.

Quality Indicators and Benchmarks

At the Perfusion Downunder Meeting in 2010,2 it was agreed by consensus amongst the attendees (see appendix 2) to report QIs for glucose, arterial outlet temperature, and pCO2 management. The values chosen for each QI were the number of procedures in which:

1. The blood glucose was ≥4 mmol/L and ≤10 mmol/L

2. The arterial outlet temperature was ≤37°C and

3. The arterial blood gas pCO₂ was ≥35 and ≤45 mmHg.

These values were chosen based upon published guidelines and regional practices (9,10). A QI was reported when any of the above conditions weremet for an individual procedure, so the best performing centers reported the highest incidence of the QI.

The QI data were used to derive benchmarks using the Achievable Benchmark of Care (ABC™TM) methodology, which ranks sites’ performance according to the incidence of the QI (5,6). For each quality indicator the Bayesian adjusted performance fraction (APF) for each site was calculated using the following formula:

$$\text{APF}=(x+1)/(d+2),$$

where x is the actual number of procedures in which the conditions of the quality indicator were met, and d is the total number of eligible procedures. Sites with small numbers of eligible procedures have the potential to skew the calculated benchmark (for example a site with 10 procedures and 100% incidence of the QI would artificially inflate the benchmark in comparison to sites with much larger numbers of procedures); the adjusted performance fraction is used to reduce this effect. This technique allows for inclusion of all sites regardless of the number of procedures, because as the number of eligible procedures (d) increases, the adjusted performance fraction and the unadjusted mathematical percentage tend to the same number.

After ranking the sites according to performance for each QI (based on the APF), the benchmarks were defined by including the total number of eligible procedures from the highest ranked performing site(s) so that at least 10%of the total dataset was represented. The benchmarks were then calculated from this subset according to the formula:

Benchmark = number of procedures with the quality indicator/ total number of procedures. (See Table 3 for an example of how the benchmark is calculated using this method).

Table 3.

Glucose benchmark calculation.

QI: Blood Glucose Concentration ≥4 or ≤10 mmol/L
Site	x (%)	d	APF	Rank
1	399 (95.9)	416	.957	1
2	1113 (87.7)	1269	.877	2
3	408 (86.8)	468	.864	3
4	779 (37.3)	2089	.373	8
5	683 (79.7)	857	.795	6
6	211 (82.1)	257	.819	4
7	49 (81.7)	60	.806	5
8	22 (64.7)	34	.639	7
Benchmark calculation: (399 + 1113)/(416 + 1269) = 89.7%

x is the number of procedures in which the quality indicator, blood glucose concentration ≥4 or ≤10 mmol/L, occurs, d is the number of eligible procedures at each site, APF is the adjusted performance factor, and rank is the ranking of performance from best to worst for the QI based upon the APF. The benchmark calculation combines the QI for sites 1 and 2, so that the benchmark was represented by at least 10% of the eligible procedures.

RESULTS

The PDUCD has procedural data on 7385 cases, 5465 (74%) were eligible for developing QIs and benchmarks. Isolated coronary artery bypass graft procedure (CABG) was the most commonly performed procedure (60%), followed by isolated valve procedure (25%), and combined valve and CABG (15%). Patient demographic and risk factor data are shown in Table 1 with a comparison to data reported from the ANZSCTS database (8). Postoperative outcomes are reported in Table 2.

Table 1.

Demographic and risk factors data.

	PDUC 2007–08	ANZSCTS 2007–08*	PDUC 2008–09	ANZSCTS 2008–09	PDUC 2009–10	ANZSCTS 2009–10*	PDUC 2010–11	PDUC Total
Number of patients	1191	2629	1286	2692	1530	2740	1458	5465
	%	%	%	%	%	%	%	%
Risk factors
Current smoker	16	14	11	15	14	14	15	14
Diabetes	28	29	29	30	27	30	28	28
Hypertension	68	71	64	72	68	73	68	67
Cerebrovascular disease	9	13	10	13	10	14	10	10
Family history of heart disease	35		40		34		36	36
Hypercholesterolemia	63		63		65		62	63
Previous cardiac intervention	17	19	17	21	19	21	18	18
Congestive heart failure	25	25	16	21	13	22	15	16
MI before surgery†	34	20	27	20	25	20	26	28
Male	74	75‡	74	70	74	72	73	74
Age > 60	68	72	71	72	71	72	72	71
EuroSCORE	5.9		6.4		6.1		6.4	6.2

^*Based on the ANZSCTS (Australia and New Zealand Society of Cardiothoracic surgeons) Cardiac surgery in Victorian public hospitals 2009–10 public report.

^†Myocardial infarction in <21 days (ANZSCTS) or <90 days (PDUC)

^‡Approximate.

Table 2.

Postoperative outcomes.

	PDUC 2007–08	PDUC 2008–09	PDUC 2009–10	PDUC 2010–11	PDUC Total
	%	%	%	%	%
Stroke	1.6	1.1	1.8	1.7	1.6
New renal failure	2.6	2.0	2.1	2.5	2.3
Myocardial infarction	2.2	1.7	1.8	1.0	1.6
Reoperation	7.6	4.6	5.5	7.1	6.1
Ventilation >24 hrs	11.3	13.8	15.7	15.7	14.2
30-day mortality	2.7	3.4	1.4	2.4	2.4

Quality Indicator and Benchmark Data

The minimum and maximum blood glucose concentrations during CPB for each site are shown in Figure 1. The benchmark calculation for the glucose QI is shown in Table 3, in which the number of procedures in which the QI is achieved is reported for each site (x (%)), the total number of procedures (d), and the APF. A benchmark value was calculated for the glucose QI of 89.7%.

Figure 1.

Box plot of minimum (white box) and maximum (grey box) blood glucose values during cardiopulmonary bypass for each participating site. The box indicates the interquartile range, error bars indicate the 95% confidence interval; o indicates the outlier values outside of 95% confidence interval; and * indicates the outlier values more than three times the interquartile range. Horizontal lines indicate targets of 4 and 10 mmol/L.

The total times (cumulative) that arterial blood outlet temperature exceeded 37°C during CPB for each site are displayed in Figure 2. A benchmark value of 93.8% was calculated for achieving the arterial temperature QI (Table 4).

Figure 2.

Box plot of arterial outlet temperature >37°C during cardiopulmonary bypass for each participating site. The box indicates the interquartile range; error bars indicate the 95% confidence interval; o indicates the outlier values outside of 95% confidence interval; and * indicates outlier values more than three times the interquartile range.

Table 4.

Arterial outlet temperature benchmark calculation.

QI: Arterial Outlet Temperature ≤37°C
Site	x (%)	d	APF	Rank
1	1016 (24.2)	417	.243	6
2	197 (15.5)	1273	.156	8
3	96 (20.5)	469	.206	7
4	1959 (93.6)	2093	.936	2
5	798 (93.1)	857	.930	3
6	113 (43.1)	262	.432	4
7	60 (100)	60	.984	1
8	11 (32.4)	34	.333	5
Benchmark calculation: (60 + 1959)/(60 + 2093) = 93.8%

x is the number of procedures in which the quality indicator, arterial outlet temperature ≤37°C, occurs, d is the number of eligible procedures at each site, APF is the adjusted performance factor, and rank is the ranking of performance from best to worst for the QI based upon the APF. The benchmark calculation combined sites 4 and 7.

The minimum and maximum arterial blood gas pCO₂ during CPB for each site are shown in Figure 3. A benchmark value of 79.6% of procedures was calculated for achieving the pCO₂ QI (Table 5). Figure 4 displays the overall site performance in terms of compliance with each benchmark, demonstrating site 7 to have the overall highest benchmark achievement overall, and to be the best performing site on the temperature QI.

Figure 3.

Box plot of minimum and maximum arterial blood gas pCO₂ during cardiopulmonary bypass for each participating site. The box indicates the interquartile range; error bars indicate the 95% confidence interval; o indicates outlier values outside of 95% confidence interval; and * indicates outlier values more than three times the interquartile range. Horizontal lines indicate targets of 35 and 45 mmHg.

Table 5.

CO₂ management benchmark calculation.

QI: Arterial pCO2 ≥35 or ≤45 mmHg
Site	x (%)	d	APF	Rank
1	378 (90.6)	417	.904	1
2	967 (76)	1272	.760	2
3	333 (71)	469	.709	3
4	792 (48)	1651	.480	5
5	277 (32.3)	857	.323	6
6	80 (31.1)	257	.312	7
7	39 (66.1)	59	.656	4
8	13 (20.6)	34	.222	8
Benchmark calculation: (378 + 967)/(417 + 1272) = 79.6%

x is the number of procedures in which the quality indicator, arterial pCO2 is ≥35 mmHg or ≤45 mmHg, occurs, d is the number of eligible procedures at each site, APF is the adjusted performance factor, and rank is the ranking of performance from best to worst for the QI based upon the APF. The benchmark calculation combined sites 1 and 2.

Figure 4.

Cumulative percentage of the occurrence of each quality indicator for each participating site (Solid shading, pCO2 QI; white, arterial outlet temperature QI; and grey, glucose QI).

DISCUSSION

We have defined three QIs for CPB, reported benchmarks for their rate of occurrence, and reported the variation in relation to these benchmarks that occurs within the participating sites of the PDUC. These results provide a resource for quality improvement initiatives. The data for patient risk factors and outcomes highlight the opportunity to report risk adjusted outcomes in the future. Benchmarks of quality indicators are not direct measures of outcome, however, it is reported that when processes of care are met there is a general association with improved outcomes. It follows that sites achieving standards of excellence as identified by benchmark performance may have better outcomes than those who do not (5,6). The demographic data collected in the PDUC dataset were comparable to those reported by ANZSCTS for data from Victorian hospitals for similar time periods (8), demonstrating the maintenance of concurrent validity for the PDUC dataset, which we had previously reported (1).

The frequency of the QI for blood glucose concentration (blood glucose ≥4 mmol/L and ≤10 mmol/L) ranged from 37.3–95.9% of cases. We identified a benchmark value of 89.7%. Glucose management during CPB has been the focus of numerous studies as hyperglycemia during CPB has been shown to be an independent predictor of morbidity and mortality (11). Similarly, the risks of hypoglycemia are well recognized (12). Guidelines for the management of perioperative blood glucose management during CPB have been reported; for example Shann et al. (9) recommended the following in the maintenance of euglycemia: The clinical team should maintain perioperative blood glucose concentration within an institution’s normal clinical range in all patients, including non-diabetic subjects (Class 1, Level B). More recently The Society of Thoracic Surgeons published practice guidelines specifically addressing the management of blood glucose during adult CPB recommends, in both diabetic and non-diabetic patients, that blood glucose levels should be maintained ≤180 mg/dL (10 mmol), while recognizing that hypoglycemia must be avoided (10). In our collaboration, one site achieved a value less than this benchmark and two sites were close. DioDato et al. (13) reported their regions comparison to the recommendation published by Shann et al. (9). They reported a regional practice with respect to the maintenance of euglycemia of 55.1% (range 3.2–76.9%) of cases recording a maximum glucose ≤200 mg/dL. The overall practice within the PDUC registry of 67.2%(range 37.3–95.9%) occurrence of the glucose QI suggests a lower percentage of cases are not maintained within this range, however similar to Northern New England experience we report a wide range in the expression of the QI. Of note, DioDato et al. (13) reported practice from 2004 and early 2005, while the PDUC has reported contemporary data (2007–2011) and both hypoglycemia and hyperglycemia.

The PDUC QI for arterial outlet temperature not exceeding 37°C produced a benchmark of 93.8%. Shann and co-authors (9) made the following recommendation in relation to the avoidance of hyperthermia during CPB: “Limiting arterial line temperature to 37°C might be useful for avoiding cerebral hyperthermia (Class IIa, Level B)”. The avoidance of hyperthermia was supported by the review by Grigore et al. (14), albeit with their recognition of the lack of randomized controlled data evaluating the adverse effects of hyperthermia following CPB. Our quality indicator suggested overall nearly 61% of cases maintained arterial outlet temperature ≤37°C, with a range of 15.5–98.4% of cases. This compared with DioDato’s report in which temperatures maintained ≤37°C for 23.4% (1.5–83.2), that is, highest blood temperature reached by the arterial inflow during re-warming was in excess of 37°C in nearly 67% of the cohorts’ reported cases (13). This was more recently investigated in the same region by Warren et al. (15), who reported a regional quality improvement project looking at reducing the variation in arterial inflow temperatures. They reported, after their improvement projects, a reduction from 90% of procedures having arterial inflow temperatures >37°C to 69% (i.e., temperature maintained below 37°C for only 10–31% of cases). The PDUC data is unique in that it uses the continuous data acquisition capabilities of the current heart lung machine data management systems to report continuous temperature output from the arterial outlet temperature port of the oxygenator, unlike the reports of DioDato et al. and Warren et al. who rely on manually recorded temperature data. The manual reporting of temperature has previously been shown to be less accurate than electronic temperature measurements (16).

The arterial outlet temperature results were not influenced by changes in oxygenators used at each site during the period of data reporting. We did not include a confirmation of the accuracy of arterial outlet measurements as an evaluation of all devices used by collaborating centers has not been reported to date. Warren et al., in their report, did evaluate the inaccuracy of the arterial outlet temperature probes; however, they have not reported these findings in detail to date. Interestingly, while they reported a range of temperature variation from different sites they only corrected data from one site (15).

The benchmark for procedures in which the arterial pCO₂ was maintained between 35 and 45 mmHg was 79.6%, with overall incidence ranging from 20.6–90.6%. Adoption of alpha stat blood gas management in adults undergoing CPB under mild hypothermia and tepid bypass is widely practiced, and the following recommendation that “The clinical team should manage adult patients undergoing moderate hypothermic CPB with alpha stat pH management (Class I, Level A)” was published (9). Measurement of pCO₂, in the range of 35–45 mmHg, allows monitoring of adherence to alpha stat management practices. Continuous online blood gas monitoring was only used at two sites (sites 1 and 6), with oxygenator exhaust gas capnography being variably used at sites 2, 3, 4, and 8. Both DioDato et al. (13) and the PDUC reported 100% adoption of the recommendation of alpha stat blood gas management; however, there are no reported comparisons of compliance to this blood gas management regime previously in the literature.

The cumulative performance of the sites in terms of compliance with each of these three benchmarks indicates that while some sites may be leaders for one particular QI, they may not perform as well across all practice areas, highlighting the potential for improvement of practice through sharing of collective experience, a fundamental principle of benchmarking. Further investigation of the barriers that have been encountered and the practices that have been found to be beneficial in the performance observed at each institution is required.

Limitations

The PDUC has developed since 2007, however, we are still limited by the relatively small number of contributing sites. In addition, as with any registry accumulated data there will be some variation in the accuracy of the reported information, and the lack of specific detailed knowledge of each site’s management protocols may account for variation in the incidence of the reported QIs. We recognize that there can be improvement in monitoring and data collection within our collaboration and we have introduced an audit (17) as one mechanism to improve data quality. Our benchmark data has been created using data from CABG, valve, and valve/CABG procedures; in the future procedure-specific benchmark data will be able to be derived. Another area of consideration relates to the interpretation of the arterial outlet temperature data. We have presented raw data from participating sites, however, we recognize that there are limitations associated with the accuracy of these arterial temperature measurements (18). Industry could assist in this area by improving the accuracy of the arterial outlet blood temperature through the hardware by providing built-in temperature off-set capability similar to that currently offered for pressure transducer measurements.

CONCLUSIONS

The PDUC has combined prospective data collection with a structured benchmarking process. We have defined process of care QIs and used these to calculate benchmarks for the management of blood glucose, arterial outlet temperature, and blood gas management during CPB. These benchmarks provide a baseline for the implementation of multicenter continuous quality improvement processes for perfusion practice.

APPENDIX 1: PARTICIPATING CENTERS

Ashford Hospital, Adelaide, South Australia; Auckland City Hospital, Auckland, New Zealand; Cabrini Hospital, Melbourne, Australia; Flinders Medical Center, Adelaide, South Australia; Flinders Private Hospital, Adelaide, South Australia; Royal Hobart Hospital, Hobart, Tasmania; Royal Perth Hospital, Perth, Western Australia; Westmead Hospital, Sydney, New South Wales

APPENDIX 2: ATTENDEES 2010

• Agbulos, Grace, Westmead Hospital

• Anderson, Bruce, Auckland City Hospital

• Arnold-Barron, Susan, Wellington Hospital

• Baker, Rob, Flinders Medical Centre

• Bayly, Marcus, St. George Hospital

• Bean, Misty, Auckland City Hospital

• Bennetts, Jayme, Flinders Medical Center

• Bhana, Jack, Waikato District Health Board

• Carpenter, Rowan, Austin Hospital

• Cifuentes, Oscar, Redcliff Anaesthetic Services

• Couyant, Melanie, Royal Melbourne Hospital

• DiBenedetto, Vita, Cardiopulmonary Associates

• Fenton, Carmel, Royal Hobart Hospital

• Grieve, Robert, Auckland City Hospital

• Habib, Imran, Gleneagles JPMC Sdn Bhd

• Kruger, Cornelis, Auckland City Hospital

• Letis, Angela, Austin Hospital

• Maines, Hazel, Obex Limited

• Markou, Mark, Flinders Medical Center

• McDonald, Charles, The Prince Charles Hospital

• McDonald, Michael, Perfusion Services, Melbourne

• McKee, Andrew, Auckland City Hospital

• McMillan, Darryl, Royal Northshore Hospital

• Merry, Alan, Auckland City Hospital

• Middleton, Neil, Auckland City Hospital

• Miller, Russell, Christchurch Hospital

• Miraziz, Ray, Westmead Hospital

• Mitchell, Simon, Auckland City Hospital

• Morley, Christopher, Geelong Hospital

• Morris, Katherine, Auckland City Hospital

• Nand, Parma, Auckland City Hospital

• Newland, Richard, Flinders Medical Center

• Orison, Kim, Westmead Hospital

• Ottens, Jane, Ashford Community Hospital

• Potger, Kieron, Royal North Shore Hospital

• Poullis, Mike, Liverpool Hospital

• Raymond, Paul, The Prince Charles Hospital

• Rumball, Elizabeth, Starship Hospital, Auckland City Hospital

• Sidebotham, David, Auckland City Hospital

• Spiess, Bruce, University Virginia

• Tuble, Sigrid, Flinders Medical Center

• Underwood, Jessica, Perfusion Services Pty Ltd

• van Uden, Rachel, Auckland City Hospital

• Vanden Berg, Jon, Waikato District Health Board

• Varghese, Sara, Canberra Hospital

• Walsh, Graham, Wellington Hospital

• Willcox, Tim, Auckland City Hospital

• Zazulak, Carla, Mater Childrens Hospital