Abstract
Objective.
We sought to develop a method to adjust for case mix diversity and allow comparison of adverse outcome rates among practitioners in pediatric and congenital cardiac catheterization.
Patients and Methods.
A single institutional database that captured demographic and procedural data was used to identify patient and procedural characteristics associated with adverse events (AE) and any high severity event classified as preventable or possibly preventable (P/PP). Diagnostic and procedural risk groups were created and indicators of hemodynamic vulnerability were defined. Expected event rates by the practitioners adjusting for case mix were calculated. Standardized adverse event ratios (SAER), defined as the observed rate divided by the expected rate for each practitioner were calculated with 95% confidence intervals.
Results.
The database included 1727 hemodynamic (30%) and interventional (70%) cases performed by seven practitioners in 18 months. During 147 cases, at least one P/PP AE occurred; among the seven practitioners observed, event rates ranged from 3.2 to 14.2%. In multivariable analysis, risk factors for all P/PP events included highest procedure risk group (odds ratio [OR] 2.1 for group 2, and 2.8 for group 3, relative to group 1, P = .001 and P < .001, respectively) and weight less than 4 kg (OR 2.8, P < .001). High severity P/PP events occurred in 67 cases with rates ranging from 2.0 to 6.6% by the practitioners. For these events, risk factors included: highest procedure risk group (OR 4.5 for group 2, and 4.9 for group 3, both P < .001) and an indicator of hemodynamic vulnerability (OR 1.8, P = .026). For the seven practitioners, the SAER ranged from 0.41 to 1.32 for any P/PP AE and from 0.69 to 1.44 for P/PP high severity events. In this cohort, we did not identify any statistically significant performance differences.
Conclusion.
Despite wide variations in case mix complexity in pediatric and congenital cardiac catheterization, this study demonstrates a method for risk adjustment which allows equitable comparisons among practitioners at a single institution.
Keywords: Adverse Events, Risk Adjustment, Performance Measure
Introduction
Many pediatric cardiology catheterization labs maintain institutionally developed databases to monitor patient demographics and complication rates for internal assessment and reporting. Rates of reported complications range from 9 to 24%, with major events occurring in 1 to 4% of cases and mortality rates of less than 1% in the past decade.1–10 Comparisons among practitioners and institutions have been limited by lack of uniformity among outcome definitions and small sample sizes. Furthermore, all prior efforts to study pediatric catheterization outcomes fail to consider differences in case mix and patient acuity.
We sought to develop a method for performance assessment which would allow equitable comparisons among pediatric and congenital cardiac catheterization operators. As demonstrated by previous outcome assessment methods for comparisons of practitioners and institutions, risk adjustment is necessary to account for variations in case mix.11,12 We therefore began by developing models to adjust for patient and procedural characteristics associated with higher event rates. Additionally, we decided that rates of unavoidable events may not be important when evaluating physician performance; categories of preventability were therefore created. Using a single institutional catheterization database, we present our method for physician performance assessment and reporting.
Methods
A single institutional database was used to identify patient and procedural characteristics associated with any event, any high severity event, and events classified as preventable or possibly preventable (P/PP). Expected event rates accounting for case mix were calculated by the practitioners adjusting for statistically significant risk factors identified in logistic regression models. Finally, standardized adverse event ratios (SAER) were determined by dividing the observed event rate by the expected rate for each practitioner. We obtained Institutional Review Board approval to perform this analysis on data collected prospectively between January 1, 2004 and June 25, 2005 for quality improvement in the cardiac catheterization laboratory at Children’s Hospital Boston.
Practitioners
Each case was performed by at least one of seven board certified pediatric cardiologists with advanced training in pediatric/congenital cardiac catheterization. Although in some circumstances two interventional cardiologists may have participated in a case, all cases were assigned to a primary attending interventional cardiologist. Two operators had less than 2 years of attending level experience when this study started and continued to receive supervision with difficult cases during the study interval. All seven operators participated in the data collection and reviewed the final manuscript, providing written permission for submission and publication in a peer reviewed journal.
Data Collection
All hemodynamic and interventional cases performed in the cardiac catheterization lab were included in the database. Primary electrophysiology and ablation cases were excluded. Patient and procedural characteristics, including adverse events (AE) occurring during the case, were entered at the time of the case by the assisting cardiology fellow. Both the attending staff and the first author reviewed each case for accuracy and completeness. Late AE were reported by the attending cardiologist at the time of identification.
Patient information collected included age, weight, and diagnosis (no structural heart disease, heart transplant, pulmonary hypertension, isolated defect, complex defect with a single ventricle, or complex defect with two ventricles). The status of the case was classified as elective (outpatient, inpatient, same day admission) or nonelective (unscheduled case or emergently performed). The following groups broadly described all cases: biopsy with or without coronary angiography, hemodynamic only, or interventional. We recorded any and all interventions performed during the case.
Adverse events were defined as any anticipated or unanticipated event for which avoidable injury (e.g., the percutaneous needle entry into a vessel for access was not avoidable) could have or did occur, potentially or definitely as a consequence of performing the catheterization procedure. Each event was categorized according to the following definitions:
Severity Level 1—none: No harm, no change in condition, may have required monitoring to assess for potential change in condition with no intervention indicated.
Severity Level 2—minor: Transient change in condition, not life threatening, condition returns to baseline, required monitoring, required minor intervention such as holding a medication or obtaining lab test.
Severity Level 3—moderate: Transient change in condition may be life threatening if not treated, condition returns to baseline, required monitoring, required intervention such as reversal agent, additional medication, transfer to the intensive care unit for monitoring, or moderate transcatheter intervention to correct condition.
Severity Level 4—major: Change in condition, life threatening if not treated, change in condition may be permanent, may have required an intensive care unit admission or emergent re-admit to hospital, may have required invasive monitoring, required interventions such as electrical cardioversion or unanticipated intubation or required major invasive procedures or transcatheter interventions to correct condition.
Severity Level 5—catastrophic: Any death and emergent surgery or heart lung bypass support to prevent death with failure to wean from bypass support.
For purposes of understanding performance, each event was also categorized according to the following definitions, to assess whether or not the event was avoidable:
-
Preventable: Events in which a definite breech of standard technique was identified; necessary precautions were not taken; event was preventable by modification of technique or care.
Examples:- Low severity: End hole power injection with no harm to the patient.
- High severity: Recatheterization for missed diagnosis or ineffective intervention.
-
Possibly Preventable: Events in which a definite breech of standard technique was not identified but may have occurred; necessary precautions may not have been taken; the event may have been preventable by modification of technique or care.
Examples:- Low severity: Groin hematoma possibly related to inadequate holding technique.
- High severity: Regurgitation postvalvotomy requiring surgery.
-
Not Preventable: Events in which no obvious breech of standard technique occurred; necessary precautions were taken; no clearly known alteration in method or care exists to prevent the event.
Examples:- Low severity: Pulmonary edema following successful pulmonary angioplasty.
- High severity: Ventricular arrhythmia induced by appropriate catheter manipulation requiring cardiopulmonary resuscitation and electrical cardioversion.
Preliminary designations for each of these two definitions were made by two operators and discrepancies were resolved according to consensus opinion among all the practitioners.
Hemodynamic Vulnerability Indicator
To account for hemodynamic factors that could increase risk for AE, patients were classified as being hemodynamically vulnerable if any of the following were present: main pulmonary artery systolic pressure greater than systemic systolic pressure, or mean pulmonary artery pressure greater than 30 mm Hg, right ventricle systolic pressure greater than systemic, systemic ventricle end diastolic pressure greater than 20 mm Hg, cardiac index less than 2 L/min/M2, systemic arterial saturation less than 75%, mixed venous saturation less than 50%, and/or case performed on heart lung bypass support. Data were retrospectively extracted from the hemodynamic summary in the catheterization report. Values that were missing were assumed to be normal.
Development of Procedure Type and Diagnosis Risk Groups
In order to develop procedure type risk groups, we convened a panel of interventional cardiologists from six pediatric institutions; the goal was to group procedures with similar risk of a high severity event.13 Each of the 11 participating interventional cardiologists categorized procedure types into groups from lowest to highest anticipated risk of a higher severity AE, choosing to create either five or six groups. Based on the panel’s assessment, six independent groups of risk were considered to exist. Procedure types near the cutoff values between groups were reviewed and changes in groupings made based on consensus opinion. Once these final groups had been created, the percentage of patients experiencing any AE, and any high severity AE (level 3, 4, or 5) were computed. By inspection, the rates of events in groups 1 and 2, 3 and 4, and 5 and 6 appeared similar. Similarly, odds ratios for events were similar in these groups, with overlapping confidence limits; therefore, these groups were not considered to be unique and were collapsed with three final groups of risk. When multiple procedure types were performed in a single case, the case was assigned to the group corresponding to the procedure type with the greatest risk. Procedure type risk groups are listed in Table 1. Within each group, types of procedures are ordered by most to least frequently occurring in this data set.
Table 1.
Procedure Type Risk Groups
| Risk Group 1 |
| RV biopsy elective post-transplant ≥10 kg |
| Hemodynamic catheterization |
| Other procedures: bronchoscopy, drains, echo, TEE |
| Coil occlusion/device/systemic arterial collaterals |
| BA/proximal LPA or RPA/dilation <8 ATM |
| Device closure/ASD |
| Device closure/fenestration |
| Other intended hemodynamic alteration/ionotropes |
| BA/aorta/dilation <8 ATM |
| Coil occlusion/veno-veno collaterals |
| Stent redilation/proximal LPA or RPA |
| Coil occlusion/PDA |
| BA/RV to PA conduit |
| Device closure/PFO |
| Interventional techniques / trans-septal puncture |
| Valvuloplasty/pulmonary valve ≥1 mo age |
| RV biopsy elective post-transplant <10 kg or on ionotropes |
| Invasive procedure/Elective chest tube pericardiocentesis |
| Stent redilation/RV to PA conduit |
| Invasive procedure/pericardiocentesis |
| Device closure/PDA |
| Atrial septostomy BAS |
| Stent placement/systemic vein |
| Diagnostic with EPS |
| RV biopsy diagnostic ≥10 kg |
| Stent redilation/aorta |
| BA/systemic vein/dilation <8 ATM |
| BA/RVOT s/p surgery (no conduit) |
| Atrial septostomy static balloon dilation |
| Device closure/venous collateral |
| Invasive procedure/central line placement |
| BA/native RVOT |
| RV biopsy diagnostic <10 kg or on ionotropes |
| Coil occlusion/LSVC |
| Stent redilation/systemic vein |
| Interventional techniques/snare foreign body |
| Stent redilation/intracardiac/atria |
| Ultrasound/IVUS |
| Stent redilation/systemic artery not aorta |
| Risk Group 2 |
| Stent placement/proximal LPA or RPA |
| BA/proximal LPA or RPA/dilation ≥8 ATM or CB |
| BA/lobar segment LPA RPA/dilation <8 ATM/<4 vessels |
| Valvuloplasty/aorta ≥1 mo age |
| Stent placement/RV to PA conduit |
| Stent placement/aorta |
| Valvuloplasty pulmonary <1 mo age |
| Stent placement/lobar segment LPA or RPA |
| Stent placement/intracardiac/atria |
| Interventional techniques/atherectomy catheter |
| Stent redilation/lobar segment LPA or RPA |
| Device closure/baffle leak |
| Interventional techniques/recanulization of occluded vessels |
| Interventional techniques/recanulization of jailed vessel in stent |
| BA/systemic artery (not aorta)/dilation <8 ATM |
| BA/aorta/dilation ≥8 ATM or CB |
| Coil occlusion/systemic shunt |
| BA/systemic vein/dilation ≥8 ATM or CB |
| BA/systemic shunt/dilation <8 ATM |
| BA/systemic shunt/dilation ≥8 ATM or CB |
| BA/systemic artery (not aorta)/dilation ≥8 ATM or CB |
| Stent placement/systemic artery (not aorta) |
| Stent redilation/pulmonary vein |
| Stent placement/native RVOT |
| Valvuloplasty tricuspid |
| Coil/coronary fistula |
| Stent placement/RVOT s/p surgery (no conduit) |
| Atrial dilation and stent/diagnosis not single ventricle |
| Risk Group 3 |
| BA/lobar segment LPA RPA/≥8 ATM or CB/<4 vessels |
| BA or stent/pulmonary vein and <3 vessels |
| BA/lobar segment LPA or RPA and ≥4 vessels |
| Any interventional catheterization within 72 h of surgery |
| Valvuloplasty mitral ≥1 y age |
| Valvuloplasty aorta <1 mo age |
| Device closure/VSD/1 device or ≥1 y age |
| Atrial dilation and stent/diagnosis single ventricle <1 y age |
| Device closure/perivalvar leak |
| Device closure/VSD/> 1 device or <1 y age |
| Interventional techniques/atretic valve perforation |
| Valvuloplasty mitral <1 y age |
| Stent placement/systemic shunt |
| BA or stent/pulmonary vein and ≥3 vessels |
| Stent placement/intracardiac/ventricular |
| Any diagnostic catheterization within 72 h of surgery |
| Stent redilation/intracardiac/ventricular |
| Atrial dilation and stent/diagnosis single ventricle ≥1 y age |
Procedure types within each risk group listed in order of most common to least commonly performed procedure type in this data set.
RV, right ventricle; TEE, transesophageal echocardiogram; LPA, left pulmonary artery; RPA, right pulmonary artery; ATM, atmospheres; ASD, atrial septal defect; PDA, patent ductus arteriosus; PA, pulmonary artery; PFO, patent foramen ovale; BA, balloon angioplasty; BAS, balloon atrial septostomy; EPS, electrophysiology study; RVOT, right ventricular outflow tract; LSVC, left superior vena cava; IVUS, intravascular ultrasound; CB, Cutting Balloon™ (Boston Scientific, San Diego CA); VSD, ventricular septal defect; ECMO, heart lungs bypass support.
The panel of interventional cardiologists grouped diagnosis types into either two or three groups based on anticipated risk of a higher severity AE. When three groups were considered, only one diagnosis had a mean score greater than the cut point for the highest risk group; therefore, the panel chose to limit the groups to a low- and a high-risk group. Diagnosis types near the cutoff values for the two groups were reviewed and changes in groupings made based on consensus opinion. Diagnosis risk groups are listed in Table 2 and types of diagnosis within each group are ordered by most to least frequently occurring in this data set.
Table 2.
Diagnosis Type Risk Groups
| Risk Group 1 |
| No structural heart disease, s/p OHT |
| Isolated ASD or PFO |
| Isolated PDA |
| Isolated pulmonary valve obstruction |
| Pulmonary hypertension with intracardiac shunt |
| No structural heart disease |
| Two ventricles s/p repair with ASD or VSD |
| Isolated VSD(s) |
| Combination defect PFO/ASD/VSD/PDA |
| Combination defect PFO/ASD and VSD |
| Isolated systemic venous obstruction |
| Combination defect PFO/ASD and PDA |
| Risk Group 2 |
| Two ventricles s/p intracardiac repair with right-sided abnormality |
| Single ventricle s/p Fontan |
| Single ventricle s/p BDG |
| Single ventricle s/p shunt or balanced circulation |
| Two ventricles with right-sided abnormality |
| Two ventricles with left-sided abnormality |
| Two ventricles s/p intracardiac repair with left-sided abnormality |
| Two ventricles with RVOTO and VSD or ASD |
| Two ventricles s/p repair with RVOTO and VSD or ASD |
| Isolated systemic arterial obstruction |
| Two ventricles s/p repair with right and left-sided abnormality |
| Isolated aortic valve |
| No structural heart disease and cardiomyopathy |
| Pulmonary hypertension without intracardiac shunt |
| Isolated pulmonary venous abnormality |
| Two ventricles on PGE |
| Isolated pulmonary arterial obstruction |
| Single ventricle on PGE |
| Isolated pulmonary venous obstruction |
| Isolated mitral valve obstruction |
| Two ventricles s/p atrial switch |
| Two ventricles s/p repair with LVOTO and VSD or ASD |
| Two ventricles with LVOTO and VSD or ASD |
Diagnosis types within each risk group listed in order of most common to least common diagnosis type in this data set.
OHT, orthotopic heart transplant; ASD, atrial septal defect; VSD, ventricular septal defect; PDA, patent ductus arteriosus; PFO, patent foramen ovale; BDG, bidirectional Glenn; PGE, prostaglandin infusion; LVOTO, left ventricle outflow tract obstruction; RVOTO, right ventricle outflow tract obstruction.
Statistical Analysis
Patient and procedural characteristics for cases were tabulated by performing physicians (designated by letters A through G) and reported as percentages of cases performed. Logistic regression was used to identify patient and procedural characteristics associated with: (1) any AE; (2) any high severity (level 3, 4, or 5) AE; (3) any P/PP AE; and (4) any high severity (level 3, 4, or 5) P/PP AE. Odds ratios with 95% confidence intervals were calculated. The following variables, significant in univariate analysis at the P < .10 level of significance, were entered into multivariable models in a stepwise manner: admission type, age, weight, diagnosis, diagnosis risk group, indicator of hemodynamic vulnerability, type of case, and highest procedure risk group. Only variables which were both statistically significant at the 0.05 level and increased the area under the receiver–operator characteristic (ROC) curve, a measure of the model’s ability to discriminate between patients who experienced an event and those who did not, were retained in the final models.
For each interventional cardiologist, observed AE rates were calculated by dividing the number of cases performed, in which at least one event occurred, by the total number of cases performed by the practitioner (unadjusted AE rate). The final multivariable models were used to predict the probability of AE occurrence for each case in the data set based on the final characteristics retained in the model. The expected AE rate was calculated by summing the probabilities of an AE (generated from the logistic regression model) for all cases performed by a practitioner and dividing by the total number of cases performed by the physician. A SAER was generated by dividing the observed AE rate by the expected AE rate; 95% confidence intervals were calculated.14 Thus, an SAER equal to one signifies an observed event rate equal to the expected; a ratio greater than one signifies that the observed rate is greater than the expected rate.
Results
Case Mix Characteristics
Seven practitioners performed 1727 catheterization cases in an 18-month period at a single academic tertiary referral institution. Wide variations in case mix characteristics by the practitioners (A to G) are summarized in Table 3.
Table 3.
Case Mix Characteristics by the Practitioners
| Total | A | B | C | D | E | F | G | |
|---|---|---|---|---|---|---|---|---|
| Total cases | 1727 | 183 | 275 | 373 | 404 | 202 | 167 | 123 |
| Admission type | ||||||||
| Elective—Outpatient | 414 (24%) | 9 (5%) | 26 (9%) | 23 (6%) | 257 (64%) | 59 (29%) | 19 (11%) | 21 (17%) |
| Elective—Same day admit | 725 (42%) | 63 (34%) | 129 (47%) | 231 (62%) | 39 (10%) | 67 (33%) | 130 (78%) | 66 (54%) |
| Elective—Inpatient | 240 (14%) | 53 (29%) | 45 (16%) | 56 (15%) | 29 (7%) | 25 (12%) | 7 (4%) | 25 (20%) |
| Nonelective (emergent) | 348 (20%) | 58 (32%) | 75 (27%) | 63 (17%) | 79 (20%) | 51 (25%) | 11 (7%) | 11 (9%) |
| Age group (years) | ||||||||
| Less than 1 | 352 (20%) | 76 (42%) | 83 (30%) | 70 (19%) | 58 (14%) | 46 (23%) | 0 | 19 (15%) |
| 1 to 10 | 684 (40%) | 76 (42%) | 131 (48%) | 169 (45%) | 154 (38%) | 82 (41%) | 1 (1%) | 71 (58%) |
| Greater than or equal to 11 | 691 (40%) | 31 (17%) | 61 (22%) | 134 (36%) | 192 (48%) | 74 (37%) | 166 (99%) | 33 (27%) |
| Weight (kilograms)* | ||||||||
| Less than 4 | 120 (7%) | 25 (13%) | 26 (10%) | 23 (6%) | 20 (5%) | 21 (11%) | 0 | 5 (4%) |
| 4 to 9.9 | 321 (19%) | 73 (40%) | 71 (26%) | 69 (19%) | 53 (14%) | 35 (18%) | 0 | 20 (16%) |
| Greater than or equal to 10 | 1247 (74%) | 83 (46%) | 174 (64%) | 280 (75%) | 312 (81%) | 134 (71%) | 166 (100%) | 98 (80%) |
| Diagnosis | ||||||||
| Isolated defect | 367 (21%) | 58 (32%) | 44 (16%) | 94 (25%) | 23 (6%) | 28 (14%) | 89 (53%) | 31 (25%) |
| No structural heart disease | 61 (4%) | 8 (4%) | 8 (3%) | 6 (2%) | 25 (6%) | 6 (3%) | 6 (4%) | 2 (2%) |
| Heart transplant | 353 (20%) | 2 (1%) | 16 (6%) | 0 | 285 (71%) | 50 (25%) | 0 | 0 |
| Pulmonary hypertension | 58 (3%) | 10 (5%) | 7 (3%) | 12 (3%) | 3(1%) | 9 (4%) | 15 (9%) | 2 (2%) |
| Complex defect with two ventricles | 542 (31%) | 51 (28%) | 100 (36%) | 204 (55%) | 29 (7%) | 65 (32%) | 42 (25%) | 51 (41%) |
| Complex defect with one ventricle | 346 (20%) | 54 (30%) | 100 (36%) | 57 (15%) | 39 (10%) | 44 (22%) | 15 (9%) | 37 (30%) |
| Diagnosis risk group | ||||||||
| 1 | 674 (39%) | 43 (24%) | 60 (22%) | 60 (16%) | 312 (77%) | 82 (41%) | 91 (54%) | 26 (21%) |
| 2 | 1053 (61%) | 140 (76%) | 215 (78%) | 313 (84%) | 92 (23%) | 120 (59%) | 76 (46%) | 97 (79%) |
| Indicator of hemodynamic vulnerability† | ||||||||
| Yes | 653 (38%) | 92 (50%) | 126 (46%) | 174 (47%) | 72 (18%) | 83 (41%) | 63 (38%) | 43 (35%) |
| No | 1074 (62%) | 91 (50%) | 149 (54%) | 199 (53%) | 332 (82%) | 119 (59%) | 104 (62%) | 80 (65%) |
| Type of case | ||||||||
| Biopsy | 300 (17%) | 1 (1%) | 14 (5%) | 0 | 239 (59%) | 46 (23%) | 0 | 0 |
| Hemodynamic | 363 (21%) | 50 (27%) | 72 (26%) | 47 (13%) | 48 (12%) | 57 (28%) | 50 (30%) | 39 (32%) |
| Intervention | 1064 (62%) | 132 (72%) | 189 (69%) | 326 (87%) | 117 (29%) | 99 (49%) | 117 (70%) | 84 (68%) |
| Procedure type risk group | ||||||||
| 1 | 1254 (73%) | 123 (67%) | 201 (73%) | 156 (42%) | 384 (95%) | 164 (81%) | 134 (80%) | 92 (75%) |
| 2 | 265 (15%) | 25 (14%) | 44 (16%) | 115 (31%) | 16 (4%) | 17 (8%) | 26 (16%) | 22 (18%) |
| 3 | 208 (12%) | 35 (19%) | 30 (11%) | 102 (27%) | 4 (1%) | 21 (10%) | 7 (4%) | 9 (7%) |
Value missing for 39 cases.
Right ventricular systolic pressure greater than systemic, main pulmonary artery systolic pressure greater than systemic or mean pulmonary artery pressure greater than 30 mm Hg, systemic ventricle end diastolic pressure greater than 20 mm Hg, cardiac index less than 2 L/min/M2, systemic arterial saturation less than 75%, mixed venous saturation less than 50%, and/or case performed with heart lung bypass support.
Adverse Events
A total of 405 AE occurred. The event was classified as P/PP in 161 of the 405 events recorded. Rhythm disturbances were the most common event classified as not preventable, 79 or of 244 (32%); AE are listed by severity and preventability in Table 4.
Table 4.
Adverse Events by Severity and Preventability
| Total Events N = 405 | Number of P/PP Events N = 147 | |
|---|---|---|
| Severity level 1 | ||
| Other catheterization related events (minor vessel trauma/staining) | 4 | 3 |
| Respiratory acidosis PaCO2>45 | 2 | 1 |
| Air embolus | 2 | 2 |
| Vessel tear | 1 | |
| Operator error (end hole power injection) | 1 | 1 |
| Airway obstruction | 1 | |
| Heart block | 1 | |
| Pneumothorax | 1 | 1 |
| Device thrombus | 1 | 1 |
| Other dilation related event | 1 | |
| Balloon rupture | 1 | |
| Severity level 2 | ||
| Hypotension (intervention ionotropes) | 27 | 2 |
| Heart block | 20 | |
| Pulmonary edema | 18 | |
| Confined vascular tear | 15 | 3 |
| Air embolus | 13 | 12 |
| Coil malposition | 13 | 10 |
| Respiratory distress/Airway obstruction/Respiratory acidosis | 12 | 6 |
| Atrial arrhythmia | 10 | |
| Coil embolization | 8 | 8 |
| Local hematoma groin (pain or required work-up/intervention) | 8 | 8 |
| Ventricular arrhythmia | 8 | |
| Re-bleed after case | 6 | |
| Pulse loss (requiring intervention) | 5 | |
| Device malposition | 5 | 4 |
| Balloon rupture | 5 | 1 |
| Wire fracture | 4 | 4 |
| Metabolic acidosis | 4 | |
| Stent malposition | 4 | 4 |
| Thrombosis-vessel, conduit, or stent | 3 | 3 |
| Operator error related to access | 3 | 3 |
| Hemothorax related to access | 3 | 3 |
| Other catheterization related events | 3 | 2 |
| Bleeding via endotracheal tube | 2 | 1 |
| Vessel tear due to catheter manipulation | 2 | 2 |
| Bradycardia (sinus) | 2 | |
| Heart perforation | 2 | 1 |
| Operator medication error | 2 | 2 |
| Pulmonary hypertensive crisis | 2 | |
| Vessel aneurysm | 2 | 1 |
| Apnea | 1 | |
| Nerve damage Horners after neck access | 1 | 1 |
| Abdominal pain | 1 | |
| Local hematoma neck | 1 | |
| CNS slow to arouse postcatheterization | 1 | 1 |
| Renal insufficiency or failure | 1 | |
| Fever | 1 | |
| GI bleed (quaic positive stools) | 1 | |
| VSD/Fistula after biopsy resolved | 1 | |
| Infection (single positive blood culture) | 1 | 1 |
| Other dilation related event | 1 | |
| Severity level 3 | ||
| Atrial arrhythmia | 21 | |
| Apnea, airway obstruction, respiratory acidosis | 21 | 10 |
| Intravascular tear with flow obstruction | 8 | 2 |
| Hypotension | 7 | 1 |
| Confined tear | 4 | 1 |
| Coronary vasospasm | 4 | 2 |
| Device malposition | 4 | 4 |
| Heart block | 4 | |
| Stent embolization | 4 | 3 |
| Stent malposition | 4 | 4 |
| Balloon rupture with fragment | 3 | 1 |
| Valvar regurgitation (greater than one grade predilation) | 3 | |
| Bradycardia (sinus) | 3 | 1 |
| Hemothorax related to access attempts | 3 | 3 |
| CNS event seizure | 2 | |
| Pseudoaneurysm requiring intervention | 2 | |
| Tachyarrhythmia | 2 | |
| Vessel thrombosis | 2 | |
| Vessel aneurysm | 1 | |
| Retroperitoneal bleed related to access | 1 | 1 |
| Fistula coronary after biopsy | 1 | |
| Local hematoma groin developed abscess | 1 | 1 |
| Local hematoma neck required prolonged intubation | 1 | 1 |
| Air embolus | 1 | 1 |
| Ventricular arrhythmia | 1 | |
| Groin AV fistula | 1 | |
| Device embolization | 1 | 1 |
| Stent thrombosis | 1 | 1 |
| Bronchus compression (stent related) | 1 | 1 |
| Severity level 4 | ||
| Heart perforation | 7 | 6 |
| Asystole (cardiac arrest) | 5 | 1 |
| Heart block | 4 | |
| Ventricular arrhythmia | 4 | |
| Stent embolization | 3 | |
| Unconfined vascular tear | 3 | 1 |
| Recatheterization for missed diagnosis or ineffective intervention | 2 | 2 |
| Regurgitation postvalvotomy requiring surgery | 2 | 1 |
| Atrial arrhythmia | 2 | |
| Thrombosis-vessel or conduit | 2 | 1 |
| Apnea or airway obstruction | 2 | 2 |
| Hypotension | 1 | |
| Pulse loss resulting in fasciotomy | 1 | 1 |
| Air embolus | 1 | 1 |
| Coronary artery thrombosis | 1 | 1 |
| Hypothermia resulting in Vfib arrest | 1 | 1 |
| Endocarditis | 1 | 1 |
| Renal insufficiency or failure | 1 | |
| Tricuspid valve damage requiring surgery | 1 | 1 |
| Unintended extubation | 1 | 1 |
| Vessel damage due to catheter manipulation | 1 | 1 |
| Severity level 5 | ||
| Asystole (cardiac arrest) | 3 | 3 |
| Aspiration | 1 | 1 |
| Coil embolization | 1 | 1 |
| Multi-organ failure | 1 | |
| Unconfined vascular tear | 1 | |
P/PP, preventable or possibly preventable; CNS, central nervous system; GI, gastrointestinal; VSD, ventricular septal defect; AV, arteriovenous.
At least one AE occurred at 324 of 1727 cases (19%); at least one potentially avoidable event classified as P/PP occurred during 147 (9%) cases. Among the 155 cases in which a high severity (level 3, 4, or 5) event occurred, 67 cases (4%) had an event classified as P/PP.
Development of Risk Adjustment Models
Risk Factors for any AE
Many patient and procedural characteristics were associated with the occurrence of any AE and high severity AE. However, in a multivariable model highest procedure type risk group, type of case (intervention vs. other), and indicator of hemodynamic vulnerability were associated with any event, area under the ROC curve 0.724. For high severity level 3, 4, or 5 events the important variables included highest procedure type risk group, type of case, weight, and indicator of hemodynamic vulnerability, area under the ROC curve 0.754.
Risk Factors for P/PP Events
All patient and procedural characteristics except diagnosis were associated with the occurrence of any P/PP event (Table 5). However, only highest procedure risk group and weight remained predictors of the outcome in multivariate analysis (area under the ROC curve 0.657) (Table 6). Similarly, most patient and procedural characteristics explored demonstrated a statistically significant association with the occurrence of a high severity (level 3, 4, or 5) P/PP event (Table 5). In the multivariable model, highest procedural risk group and indicator of hemodynamic vulnerability remained predictors for the outcome (area under the ROC curve 0.741) (Table 6).
Table 5.
Associations between Patient and Procedural Characteristics and Preventable or Possibly Preventable (P/PP) Adverse Events (n = 1727)
| Any Severity (level 1–5) P/PP Adverse Events | High Severity (level 3–5) P/PP Adverse Events | ||||
|---|---|---|---|---|---|
| Patient and Procedural Characteristics | N | n (%) | OR (95% CI) | n (%) | OR (95% CI) |
| Admission type | |||||
| Elective | 1379 | 103 (7%) | 1.0 | 44 (3%) | 1.0 |
| Nonelective (emergent) | 348 | 44 (13%) | 1.8 (1.2, 2.6) | 23 (7%) | 2.1 (1.3, 3.6) |
| Age group (years) | |||||
| Greater than or equal to 11 | 691 | 40 (6%) | 1.0 | 18 (3%) | 1.0 |
| 1 to 10 | 684 | 55 (8%) | 1.4 (0.9, 2.2) | 27 (4%) | 1.5 (0.8, 2.8) |
| Less than 1 | 352 | 52 (15%) | 2.8 (1.8, 4.4) | 22 (6%) | 2.5 (1.3, 4.7) |
| Weight (kilograms)* | |||||
| Greater than or equal to 10 | 1247 | 86 (7%) | 1.0 | 40 (3%) | 1.0 |
| 4 to 9.9 | 321 | 37 (12%) | 1.8 (1.2, 2.6) | 14 (4%) | 1.4 (0.7, 2.6) |
| Less than 4 | 120 | 24 (20%) | 3.4 (2.1, 5.6) | 13 (11%) | 3.7 (1.9, 7.1) |
| Diagnosis | |||||
| Isolated defect | 367 | 33 (9%) | 1.0 | 16 (4%) | 1.0 |
| No structural heart disease | 61 | 1 (2%) | 0.2 (0.02, 1.3) | 0 (0%) | — |
| Heart transplant | 353 | 13 (4%) | 0.4 (0.2, 0.7) | 6 (2%) | 0.4 (0.1, 1.0) |
| Pulmonary hypertension | 58 | 4 (7%) | 0.8 (0.3, 2.2) | 1 (2%) | 0.4 (0.05, 3.0) |
| Complex defect with two ventricles | 542 | 51 (9%) | 1.1 (0.7, 1.7) | 24 (4%) | 1.0 (0.5, 1.9) |
| Complex defect with one ventricle | 346 | 45 (13%) | 1.5 (0.9, 2.4) | 20 (6%) | 1.3 (0.7, 2.6) |
| Diagnosis risk group | |||||
| 1 | 674 | 39 (6%) | 1.0 | 14 (2%) | 1.0 |
| 2 | 1053 | 108 (10%) | 1.9 (1.3, 2.7) | 53 (5%) | 2.5 (1.4, 4.5) |
| Indicator of hemodynamic vulnerability | |||||
| No | 1074 | 72 (7%) | 1.0 | 26 (2%) | 1.0 |
| Yes† | 653 | 75 (11%) | 1.8 (1.3, 2.5) | 41 (6%) | 2.7 (1.6, 4.5) |
| Type of case | |||||
| Biopsy | 300 | 9 (3%) | 1.0 | 4 (1%) | 1.0 |
| Hemodynamic | 363 | 18 (5%) | 1.7 (0.7, 3.8) | 4 (1%) | 0.8 (0.2, 3.3) |
| Intervention | 1064 | 120 (11%) | 4.1 (2.1, 8.2) | 59 (6%) | 4.3 (1.6, 12.1) |
| Procedure type risk group | |||||
| 1 | 1254 | 76 (6%) | 1.0 | 23 (2%) | 1.0 |
| 2 | 265 | 35 (13%) | 2.4 (1.5, 3.6) | 22 (8%) | 4.8 (2.7, 8.8) |
| 3 | 208 | 36 (17%) | 3.2 (2.1, 5.0) | 22 (11%) | 6.3 (3.5, 11.6) |
Value missing for 39 cases.
Right ventricle systolic pressure greater than systemic, main pulmonary artery systolic pressure greater than systemic or mean pulmonary artery pressure greater than 30 mm Hg, systemic ventricle end diastolic pressure greater than 20 mm Hg, cardiac index less than 2 L/min/M2, systemic arterial saturation less than 75%, mixed venous saturation less than 50%, and/or case performed with heart lung bypass support.
Table 6.
Risk Adjustment Models
| Patient and Procedural Characteristics | Any P/PP* Adverse Event | ||
|---|---|---|---|
| Odds ratio | (95% CI) | P value | |
| Procedure type risk group | |||
| 1 | 1.0 | — | — |
| 2 | 2.1 | (1.4, 3.3) | .001 |
| 3 | 2.8 | (1.8, 4.3) | <.001 |
| Weight (kilograms) | |||
| Greater than or equal to 10 | 1.0 | — | — |
| 4 to 9.9 | 1.6 | (1.0, 2.4) | .032 |
| Less than 4 | 2.8 | (1.7, 4.7) | <.001 |
| Patient and Procedural Characteristics | P/PP* Severity Level 3, 4, or 5 Adverse Events | ||
| Odds ratio | (95% CI) | P value | |
| Procedure type risk group | |||
| 1 | 1.0 | — | — |
| 2 | 4.5 | (2.5, 8.3) | <.001 |
| 3 | 4.9 | (2.6, 9.3) | <.001 |
| Indicator of hemodynamic vulnerability | |||
| No | 1.0 | — | — |
| Yes | 1.8 | (1.1, 3.1) | .026 |
P/PP, preventable or possibly preventable.
Risk Adjusted Event Rates by the Practitioners
Risk adjusted event rates are shown in Table 7. We observed P/PP AE rates ranging from 3.2 to 14.2% among the seven practitioners. The SAER ranged from 0.41 to 1.32; the value 1.0 was included in the 95% confidence intervals for all practitioners. Among high severity (level 3, 4 or 5) P/PP events, rates ranged from 2.0 to 6.6%. Standardized event ratios ranged from 0.69 to 1.44, with 95% confidence intervals including 1.0 for all practitioners. Thus, none of the operators demonstrated a statistically significant difference in performance compared to the average for this group. In a similar analysis including all AE regardless of preventability, the SAER ranged from 0.86 to 1.22 for any AE and 0.93 to 1.05 for high severity events; the value 1.0 was included in the 95% confidence interval for all seven practitioners.
Table 7.
Risk-adjusted Comparison of Physician Outcomes
| Any Preventable or Possibly Preventable (P/PP) Event | ||||
|---|---|---|---|---|
| Operator | Observed Event Rate | Expected* Rate for Case Mix | SAER† | 95% Confidence Interval |
| A | 14.2% | 10.8% | 1.32 | (0.86, 1.93) |
| B | 10.9% | 8.9% | 1.22 | (0.82, 1.74) |
| C | 10.7% | 10.8% | 1.00 | (0.71, 1.34) |
| D | 4.5% | 6.3% | 0.71 | (0.42, 1.12) |
| E | 7.9% | 8.3% | 0.96 | (0.55, 1.56) |
| F | 7.8% | 6.5% | 1.20 | (0.64, 2.06) |
| G | 3.2% | 7.8% | 0.41 | (0.11, 1.06) |
| Total | 8.5% | 8.5% | 1.00 | |
| Any P/PP Severity Level 3, 4, or 5 Events | ||||
| Operator | Observed Event Rate | Expected* Rate for Case Mix | SAER† | 95% Confidence Interval |
| A | 6.6% | 4.6% | 1.44 | (0.74, 2.51) |
| B | 5.1% | 3.9% | 1.30 | (0.71, 2.18) |
| C | 4.8% | 6.2% | 0.79 | (0.46, 1.24) |
| D | 2.0% | 2.0% | 0.99 | (0.46, 1.94) |
| E | 3.0% | 3.4% | 0.87 | (0.32, 1.89) |
| F | 3.6% | 3.3% | 1.08 | (0.39, 2.35) |
| G | 2.4% | 3.5% | 0.69 | (0.14, 2.02) |
| Total | 3.9% | 3.9% | 1.00 | |
The final multivariable models for each outcome were used to predict the probability of adverse event occurrence for each case in the data set based on the final characteristics retained in the model. The expected adverse event rate was calculated by summing the probabilities of an adverse event (generated from the logistic regression model) for all cases performed by a practitioner and dividing by the total number of cases performed by the physician.
A standardized adverse event ratio (SAER) was generated by dividing the observed adverse event rate by the expected adverse event rate.
Discussion
Given the extremely heterogeneous nature of practice in interventional pediatric cardiology, standardized assessment of outcomes must account for differences in patients treated and procedures performed. Using a single institutional experience, we have developed a method at our institution to evaluate and compare outcomes among pediatric and congenital interventional catheterization practitioners. First, we classified clinically relevant events based on their severity. We chose to only consider events classified as P/PP in our assessment of performance and incorporated this designation in our assessment of events. Additionally, we used new patient and procedural variables, allowing the categorization of various procedure types and diagnoses in risk groups. Then, we developed risk adjustment models which performed reasonably well. The first model for any P/PP event included procedure risk group and weight (area under the ROC curve 0.657), while the other model for P/PP severity level 3, 4, or 5 events included procedure risk group and indicator of hemodynamic vulnerability (area under the ROC curve 0.741). Finally, standardized event ratios were calculated as the observed event rates divided by the expected rates adjusted for the practitioner’s case mix. Since the final performance measurements were focused on P/PP events, examination of risk adjusted event rates allows individual practitioners to assess their potential for performance improvement. But, to assess whether performance differences existed among the many AE classified as not preventable, or preventable cause not identified, we performed a similar analysis for all events and all high severity AE and did not detect any performance outliers.
Most pediatric cardiology catheterization labs maintain databases to monitor patient demographics and complications. However, no method for comparing the existing databases has been established, nor has such comparison been attempted. Some institutions have published reports on complication rates.1–10 Depending on the definitions, complication rates range from 9 to 24%, with major events occurring in 1 to 4% of cases, and mortality rates of less than 1% in the last decade. Younger age has been universally associated with a higher complication rate. Interventional cases appear to have higher complication rates when compared to diagnostic cases. Specific interventions such as balloon angioplasty have been associated with higher rates of both major and minor complications. However, due to lack of uniformity among outcome definitions, small sample sizes, and case mix differences, the potential for generalization is limited.
In pediatrics, an ongoing quest for quality measures is actively being sought.15,16 However, attempts to report and compare institutional outcomes such as AE in pediatric cardiac catheterization have not been attempted. The reasons for this, while complex, are most likely united by concerns that the results will be misinterpreted due to case mix differences among practitioners and institutions. When comparing procedural outcomes, risk adjustment is necessary to allow for differences in case complexity and severity of illness.11,12 In this study, we have shown that despite wide variations in case mix, a method can be developed to allow equitable comparison of performance among physicians performing catheterization cases in children and adults with congenital heart disease.
This method, however, is preliminary, with a number of inherent limitations. First, although the current model performed reasonably well in a single institutional data set, areas under the ROC curve ranged from 0.65 to 0.74; thus, other important variables may not yet have been identified. Furthermore, initial procedure and diagnosis type risk groups and the indicator of hemodynamic vulnerability may not have optimum predictive power. In addition, one of the greatest limitations of our preliminary method for evaluating outcomes by the practitioners is the lack of a procedural efficacy measure. In an interventional-based procedure, success or failure to complete the objectives must be considered when evaluating the outcome, as well as the occurrence of preventable AE. False conclusions regarding physician performance may be made, for example, if the objectives of the procedure were not accomplished due to concern for event risk, or if interventions were not performed according to evidence-based methods, placing the patient at greater risk for injury. In order to assess procedural outcomes, we must develop a measure to assess procedural efficacy. At present, few measurement tools exist.17
Future Directions
To develop a universally applicable performance assessment measurement in the field of pediatric and congenital cardiac catheterization, we must first agree upon uniform definitions for outcomes, develop and validate risk adjustment tools, and establish benchmark rates applicable to catheterization labs with varying volume and patient acuity. The Congenital Cardiac Catheterization Outcomes Project participants are committed to these objectives. Six participating institutions (Children’s Hospital Boston, Morgan Stanley Children’s Hospital of New York Presbyterian, Cincinnati Children’s Hospital, Columbus Children’s Hospital, St. Louis Children’s Hospital, and Rady Children’s Hospital and Health Center San Diego) began collecting data using a web-based entry tool in February 2007. Patient and procedural characteristics, as well as occurrence of AE, are recorded for all hemodynamic and interventional cases performed at the institutions. In the coming years, with the combined expertise of the Congenital Cardiac Catheterization Outcomes Project investigators, this group will refine and improve the currently presented method by incorporating a measure of efficacy, and utilizing the multi-institutional data set for further analysis and measurement development.
Conclusion
Despite wide variations in case mix complexity, this study demonstrates a feasible method for risk adjustment which at a single institution allows equitable comparisons among practitioners performing pediatric and congenital cardiac catheterization procedures. Multicenter collaboration will allow us to refine and improve methods for practitioners and institutions seeking outcome measurement tools in pediatric cardiology.
Acknowledgments
Data analysis support was provided by Gary Piercey and supported by funding provided by the Hill Family Outcomes Research Initiative Fund.
Footnotes
Conflict of interest: None.
References
- 1.Agnoletti G, Bonnet C, Boudjemline Y, et al. Complications of paediatric interventional catheterization: an analysis of risk factors. Cardiol Young. 2005; 15:402–408. [DOI] [PubMed] [Google Scholar]
- 2.Rhodes JF, Asnes JD, Blaufox AD, Sommer RJ. Impact of low body weight on frequency of pediatric cardiac catheterization complications. Am J Cardiol. 2000;86:1275–1278, A9. [DOI] [PubMed] [Google Scholar]
- 3.Tavli V, Kayhan B, Okur FF, Kirman M, Tekdogan M. Complications of pediatric cardiac catheterization: 18-month study. Turk J Pediatr. 2000;42:294–297. [PubMed] [Google Scholar]
- 4.Zeevi B, Berant M, Fogelman R, Galit B, Blieden L. Acute complications in the current era of therapeutic cardiac catheterization for congenital heart disease. Cardiol Young. 1999;9:266–272. [DOI] [PubMed] [Google Scholar]
- 5.Vitiello R, McCrindle BW, Nykanen D, Freedom RM, Benson LN. Complications associated with pediatric cardiac catheterization. J Am Coll Cardiol. 1998;32:1433–1440. [DOI] [PubMed] [Google Scholar]
- 6.Love BA, Rosenberg HC, Nykannen D, Li MD. The safety of pediatric cardiac catheterization in an adult hospital setting. Can J Cardiol. 1992;8:347–350. [PubMed] [Google Scholar]
- 7.Cassidy SC, Schmidt KG, Van Hare GF, Stanger P, Teitel DF. Complications of pediatric cardiac catheterization: a 3-year study. J Am Coll Cardiol. 1992; 19:1285–1293. [DOI] [PubMed] [Google Scholar]
- 8.Fellows KE, Radtke W, Keane JF, Lock JE. Acute complications of catheter therapy for congenital heart disease. Am J Cardiol. 1987;60:679–683. [DOI] [PubMed] [Google Scholar]
- 9.Cohn HE, Freed MD, Hellenbrand WF, Fyler DC. Complications and mortality associated with cardiac catheterization in infants under one year: a prospective study. Pediatr Cardiol. 1985;6:123–131. [DOI] [PubMed] [Google Scholar]
- 10.Kennedy JW. Complications associated with cardiac catheterization and angiography. Cathet Cardiovasc Diagn. 1982;8:5–11. [DOI] [PubMed] [Google Scholar]
- 11.Richardson D, Tarnow-Mordi WO, Lee SK. Risk adjustment for quality improvement. Pediatrics. 1999;103:255–265. [PubMed] [Google Scholar]
- 12.Shwartz MAA, Iezzoni LI. Comparing outcomes across providers In: Iezzoni LI, ed. Risk Adjustment for Measuring Healthcare Outcomes, 2nd edn Chicago, Ill: Health Administration Press; 1997: 471–516. [Google Scholar]
- 13.Jenkins KJ, Gauvreau K, Newburger JW, Spray TL, Moller JH, Iezzoni LI. Consensus-based method for risk adjustment for surgery for congenital heart disease. J Thorac Cardiovasc Surg. 2002;123:110–118. [DOI] [PubMed] [Google Scholar]
- 14.Breslow NE, Day NE. Statistical Methods in Cancer Research, Volume II—The Design and Analysis of Cohort Studies. New York: Oxford University Press; 1987. [PubMed] [Google Scholar]
- 15.Forrest CB, Shipman SA, Dougherty D, Miller MR. Outcomes research in pediatric settings: recent trends and future directions. Pediatrics. 2003;111: 171–178. [DOI] [PubMed] [Google Scholar]
- 16.Dougherty D, Simpson L. Measuring the quality of children’s health care: a prerequisite to action. Pediatrics. 2004;113:185–198. [PubMed] [Google Scholar]
- 17.Landon BE, Normand SL, Blumenthal D, Daley J. Physician clinical performance assessment: prospects and barriers. JAMA. 2003;290:1183–1189. [DOI] [PubMed] [Google Scholar]