Pediatric/Congenital Cardiac Catheterization Quality: An Analysis of Existing Metrics

Abstract

Objectives/Introduction:

Enumerate and categorize quality metrics relevant to the pediatric/congenital cardiac catheterization laboratory(PCCL).

Methods:

Potential metrics were evaluated by 1) a systematic review of the peer-reviewed literature, 2) a review of metrics from organizations interested in quality improvement, patient safety, and/or PCCL programs, and 3) a survey of US PCCL cardiologists. Collected metrics were grouped on two dimensions: 1) US Institute of Medicine domains, and 2) Donabedian structure/process/outcome framework. Survey responses were dichotomized between favorable and unfavorable responses, then compared within and between categories.

Results:

In the systematic review, 6 metrics were identified (from 9 publications), all focused on safety either as an outcome (adverse events (AE), mortality, and failure to rescue (FTR) along with radiation exposure) or as a structure (procedure volume or operator experience). Four organizations measure quality metrics of PCCL programs, of which only one publicly reports data. For the survey, 229 cardiologists from 118 hospital-programs responded (66% of individuals and 72% of hospital-programs). The highest favorable ratings were for safety metrics (p<0.001), of which major AE, FTR, and procedure-specific AE had the highest ratings. Of respondents, 67% stated that current risk adjustment were not effective. Favorability ratings for hospital characteristics, PCCL lab characteristics, and quality improvement processes were significantly lower than for safety and less consistent within categories.

Conclusions:

There is a limited number of PCCL quality metrics, primarily focused on safety. Confidence in current risk adjustment methodology is low. The knowledge gaps identified should guide future research in the development of new quality metrics.

Condensed abstract:

Quality metrics for the pediatric congenital interventional cardiology (PCCL) laboratory were evaluated through a systematic review of 1) the scientific literature, 2) organizations measuring/reporting quality, and 3) a survey of US PCCL cardiologists to help identify current areas of consensus and knowledge gaps. Published literature and quality organizations focus on measures of safety, but there is limited confidence in current risk adjustment methods. Major knowledge gaps are 1) the dearth of hospital and program characteristics associated with quality and 2) quality metrics evaluating domains other than safety (e.g. efficacy or efficiency). These findings should guide future research and quality improvement efforts.

INTRODUCTION

Cardiac catheterization procedures remain the gold standard for hemodynamic evaluation in young patients with congenital and acquired heart disease, as well as treatment for an increasing number of conditions. At the same time, pediatric/congenital cardiac catheterization laboratory (PCCL) procedures represent a critical period in the lives of these patients. The risk of morbidity and mortality during PCCL catheterization is second only to the cardiac operating room, with reported risks of major adverse events (AE) between 10 and 11%(1, 2). Measurement of the performance of PCCL programs in a stringent and consistent fashion is a crucial step towards improving outcomes. To our knowledge, a systematic evaluation of current quality metrics in PCCL has not been performed to date.

A conceptual framework dividing potential quality metrics into domains facilitates their study by allowing investigators to identify areas of consensus and knowledge gaps. Two frameworks are widely accepted: the first by the US Institute of Medicine (IOM) divides quality into six domains (effectiveness, safety, efficiency, timeliness, equity, and patient-centeredness) based on content, and the second, developed by Avedis Donabedian, divides them into structures, processes, and outcomes based on where they fit into the process of healthcare delivery. The two can be combined into a two-axis schema that provides a useful comprehensive framework which has demonstrated utility in other fields(3).

We sought to comprehensively evaluate how PCCL quality is currently measured, describing: 1) what performance measures are evaluated in the scientific literature, 2) how quality and performance are measured and reported by interested organizations, and 3) how practicing interventional cardiologists view performance measures. Consensus between the scientific literature, quality organizations, and practitioners would not necessarily constitute proof of the validity of various quality metrics. At the same time, identifying schisms and/or knowledge gaps, however, would be useful to direct priorities for future research.

METHODS:

Study design

Three sources were utilized to capture potential quality metrics: 1) a systematic review of the peer-reviewed literature, 2) a review of measures used by organizations involved in the measurement of quality and quality improvement, and 3) a survey of United States PCCL cardiologists. The study was reviewed by the Institutional Review Board of The Children’s Hospital of Philadelphia and was declared to be exempt from review as per the Common Rule. The research was performed in a manner consistent with good research practices.

Systematic review

We searched Medline and Embase for English language articles and abstracts from 1/1/1995 through 12/30/2019 using a set of pre-determined search terms (Table 1) for quality metrics for PCCL programs. Studies were included if the design of the study compared a metric between different centers to measure relative quality of care. The procedures performed at PCCL programs and those specializing in coronary artery and adult-onset structural heart disease are qualitatively different, so searches were limited to studies performed in children (age 0–18 years). To optimize sensitivity, potential measures were included regardless of study design and validation. Reference lists of included papers were also reviewed to identify potentially relevant manuscripts. Data abstraction was limited to the information presented in the publication and no authors were contacted. Database queries were performed by two members of the study staff (MLO and IA) indepdently. The results from parallel searches were compared and differences were settled through consensus discussion.

Table 1:

Search Terms for Systematic Review

Interventional cardiology OR heart catheterization AND One of the following: Quality Quality indicator Safety Effectiveness Efficiency Metric Performance measure

^*Limited to Child (birth to 18 years)

Review of cardiology and quality improvement organizations

Suggested quality metrics from organizations that measure and/or report healthcare quality and/or specifically address quality of PCCL programs were reviewed to identify the quality metrics used by each organization. The organizations included were identified based on previous publications(3) or because of their prominence in measuring/reporting quality performance of pediatric/congenital cardiology and interventional cardiology programs. Quality metrics for each source were described using the same schema used for the systematic review. In addition, the process by which data about quality were disseminated to member institutions and/or the public was reviewed. For the purpose of presentation, these strategies were divided between 1) public reporting, 2) reporting to member institutions with anonymization of the results from other programs, and 3) combination of component scores within a larger score without reporting of component data (without presenting the granular data directly).

Survey of practicing interventional cardiologists

A study instrument was distributed to all physicians identified in a census of United States PCCL cardiologists performed by the Society for Cardiac Angiography and Intervention Pediatric/Congenital Section in 2019. The study instrument was designed by the authors to evaluate agreement about quality metrics for PCCL programs: specifically outcome metrics and characteristics of hospitals, PCCL programs, and practitioners that might represent high quality. For each metric, respondents were asked to use a 5-point Likert scale to rate the degree to which they agreed that a specific metric reflected the quality of a PCCL program. One additional question directly asked respondents to rate the current state of risk stratification for adverse events using a Likert scale (very poor, poor, neutral, good, very good). No formal focus group or field testing were performed during survey development.

The instrument was distributed electronically using the REDCap survey distribution system. Electronic mail reminders were sent to encourage participation. No other incentives (financial or otherwise) were applied. Responses to survey questions were automatically anonymized, but whether a specific individual responded was known. These data enabled calculation of the response rate and permitted a limited description of the respondents (e.g. number of centers and geographic spread).

Statistical Analysis

For the systematic review, all identified measures are enumerated. Each metric was categorized by both IOM and Donabedian framework. All categorizations were made by two members of the study team (MLO and IA) with disagreements resolved through consensus discussion. No meta-analytic techniques were applied. The metrics used by organizations are presented in a similar framework. Agreement between reporting organizations and systemic review is presented and evaluated qualitatively (Central Illustration).

Central Illustration: Quality Metrics in Pediatric/Congenital Catheterization.

Results from systematic review of the medical literature, review of organizations reporting quality, and a survey of US pediatric/congenital interventional cardiologists are presented. Quality metrics are divided into the six domains (rows) identified by the US Institute of Medicine. The scientific literature about quality metrics is focused almost entirely on safety metrics. Pooled adverse events (AE) and failure to rescue have been studied, but further research is necessary to determine the optimal measure of safety. IMPACT®, CCISC, and C3PO registries report raw and risk adjusted AE to members. Survey data demonstrates high-levels of interest in safety metrics with particular interest in procedure-specific AE as a possible metric. At the same time confidence in current risk-adjustment models is low. For effectiveness, technical success has been reported for individual procedures, but their use as quality metrics has not been evaluated. Development of risk adjustment models specifically for each procedure-specific metric of technical success is necessary. IMPACT® reports measures of technical success for a subset of procedures, and survey data demonstrate that there is interest in procedure-specific technical success as a metric for procedural success. In terms of efficiency, economic cost of procedures has been studied as an outcome, with large magnitude variation demonstrated between centers. However, it has not been evaluated as a quality metric. Metrics combining efficacy and cost (i.e. value) are more complete measures of efficiency but have not been studied. There remain significant knowledge gaps and lack of consensus about measures of timeliness, equity, and patient-centeredness. These can only be addressed by additional research.

For survey data, descriptive data were calculated for respondents to demonstrate the generalizability of the sample (response rate, center characteristics, and geographic distribution). Results were summarized using standard descriptive statistics. To evaluate consensus, the results were dichotomized into favorable (strongly agree and agree) and non-favorable (neutral, disagree, and strongly disagree). Proportions of favorable vs. unfavorable ratings were compared between measures using Chi-squared test.

RESULTS:

Systematic review of quality metrics

A total of 6 metrics were identified in 9 manuscripts(4–12) and 1 abstract(13). The metrics proposed or evaluated in these manuscripts are presented in Table 2. In terms of IOM quality domains, 100% of the identified quality metrics measure patient safety. There are no metrics that address effectiveness, efficiency, timeliness, equity, or patient-centeredness (Central Illustration). In terms of the Donabedian framework, 67% (4/6) metrics are outcome metrics with the remaining 33% (2/6) addressing structures.

Table 2:

Framework of performance measures by Institute of Medicine domains (rows) and Donabedian (columns) schemae from systematic review

IOM Quality Domain		Donabedian Framework
		Structure	Process	Outcome
	Timeliness	(none)	(none)	(none)
	Effectiveness	(none)	(none)	(none)
	Efficiency	(none)	(none)	(none)
	Safety	Center volume Operator volume/experience	(none)	Mortality Adverse event rate Failure to rescue Radiation exposure
	Patient centeredness	(none)	(none)	(none)
	Equity	(none)	(none)	(none)

Metrics used by quality-measuring organizations

The organizations sampled are summarized in Table 3. Of these, 22% (4/18) propose or measure quality metrics from PCCL programs. Joint recommendations from the American Academy of Pediatrics, American College of Cardiology, and Society for Cardiac Angiography and Intervention(14, 15) define minimum requirements for a PCCL program, but do not provide criteria by which to discriminate between programs. All of these recommendations are based on consensus without any basis in observational or trial data. American College of Cardiology guidelines for management of adults with congenital heart disease do not address quality metrics for PCCL programs(16, 17).

Table 3:

Organizations measuring quality of pediatric/congenital cardiac catheterization programs and metrics measured by each

ORGANIZATION	QUALITY METRICS:
Agency for Healthcare Research and Quality (AHRQ) www.qualityindicators.ahrq.gov	None
American Academy of Pediatrics/American College of Cardiology/Society for Cardiac Angiography and Intervention Consensus Recommendations(14)	STRUCTURE Directed by board certified pediatric cardiologist with additional fellowship (or other experience) in interventional cardiac catheterization Each case staffed by a board certified or board eligible pediatric cardiologist AND one pediatric nurse with expertise in airway/sedation Biplane laboratory Physiologic recording system, crash cart, and supply of catheters Appropriate ICU (not specifically pediatric cardiac intensive care unit) In house surgical back-up
	PROCESS Prospective data collection about outcomes Systematic review of outcomes and case complications
	OUTCOME Acknowledge mortality and adverse event rates but no standards
American College of Cardiology Adult Congenital and Pediatric Cardiology Section (ACC ACPC) Practice Guidelines (16, 17)	None
Center for Medicare and Medicaid Services (CMS) cms.gov	None
Children’s Hospital Association childrenshospitals.org	None
Congenital Cardiovascular Interventional Study Consortium (CCISC) Risk Registry	STRUCTURE
	None
	PROCESS
	None
	OUTCOME Adverse event rates stratified by pre-procedural risk Benchmark radiation exposure
Congenital Cardiac Catheterization Project on Outcomes (C3PO)	STRUCTURE
	None
	PROCESS
	None
	OUTCOME Case-mix adjusted adverse event rates Benchmark radiation exposure
IMproving Pediatric And Congenital Treatment (IMPACT®) Registry	STRUCTURE:
	None
	PROCESS Percentage of cases where all three radiation exposure measurements (fluoroscopy time, skin dose, and dose area product) were reported
	OUTCOME 1Observed (Raw) adverse events (stratified by age group)* 2Technical success for specific procedures# Device in place at end of procedure (ASD closure, PDA closure) Non-trivial residual shunt (ASD closure, PDA closure) Residual gradient (coarctation stent, coarctation angioplasty, balloon pulmonary valvuloplasty) Peak to peak gradient <=40mmHg and regurgitation less than 3+ (aortic valvuloplasty) Ratio of diameter of stented vessel and pre-intervention distal vessel (pulmonary artery stent angioplasty- stratified by single and bi-ventricular patients) 3Procedure specific adverse events All (transcatheter pulmonary valve replacement) Device embolization for specific procedures (ASD closure, PDA closure)
	* Risk adjusted adverse event rate is listed as a “test metric”
Healthcare Effectivness Data and Infromation Set (HEDIS) ncqa.org	None
Health Resources and Services Administration Child Health Bureau hrsa.gov	None
Institute for Healthcare Improvement (IHI) www.ihi.org	None
Institute of Medicine/National Academy of Medicine nam.edu	None
The Joint Commission www.jointcommission.org/performance_measurement.aspx	None
Leapfrog Group leapfroggroup.org	None
National Committee for Quality Assurance (NCQA) www.ncqa.org	None
National Quality Forum (NQF) www.qualityforum.org	None
National Quality Measures Clearinghouse (NQMC) www.qualitymeasures.ahrq.gov	None
Patient Centered Outcomes Research Institute (PCORI) pcori.org	None
RAND www.rand.org/health	None
US News and World Report	STRUCTURE Active fellowship program (overall program) Reporting data to IMPACT® Registry Procedure-specific annual case volume (low, medium, or high) Diagnostic cardiac catheterization Endomyocardial biopsy Balloon angioplasty Balloon valvuloplasty Pulmonary stent implantation Coarctation stent implantation Transcatheter occlusion of cardiac shunt Transcatheter placement of stented pulmonary valve Stent redilation Transcatheter occlusion of atrial septal defects Transcatheter occlusion of ventricular septal defects Transcatheter occlusion of patent ductus arteriosus Aortic/pulmonary catheter-based valvuloplasty
	PROCESS Engaging in procedural safety procedures (time out, pre-procedure briefings, and implementation of hand-off procedure)
	OUTCOME None

All measured metrics address concerns about safety except for those identified with a #, which address effectiveness.

US News and World Report’s scoring of pediatric cardiology and cardiovascular surgery programs includes several potential quality metrics(18). The overall reputation score of the program and the presence of an active fellowship program are included as is the presence of a “diagnostic and/or interventional catheterization program.” Participation in the IMProving Adult and Congenital Treatment (IMPACT®) Registry also adds to the center’s score (along with other registries). The majority of the score is based on the volume of diagnostic and other specific trans-catheter procedures, which are divided into low, medium, and high volumes. Programs also gain points for having procedural safety procedures (programs for time outs, pre-procedure briefings, and hand-off procedures), a potential process measure. In contrast to cardiac surgery (where mortality rates form a significant portion of the score), cardiac catheterization programs are not judged for either safety or technical success outcomes.

The IMPACT® Registry(19), Congenital Cardiac Catheterization Project on Outcomes collaborative (C3PO)(20), Congenital Cardiovascular Interventional Study Consortium Risk Registry (CCSIC) provide member institutions quality metrics for their own use (presenting data for the individual program in comparison to other member programs) (Central Illustration). C3PO reports both case-mix adjusted adverse event rates and radiation data(4) benchmarked to other member institutions. CCISC also reports adverse event rates but instead of a single case-mix adjusted metric, they provide observed results stratified by a risk prediction score derived for either pediatric(21, 22) or adult(23) PCCL patients as well as radiation exposure data(13, 24). The IMPACT® Registry reports observed (i.e. not risk adjusted) adverse event rates for specific age groups as well as technical success rates for ASD closure, PDA closure, aortic and pulmonary balloon valvuloplasty, coarctation procedures, and pulmonary artery stent angioplasty. Procedure specific observed AE are reported for transcatheter pulmonary valve replacement, while device embolization is reported for ASD closure and PDA closure. Risk adjusted adverse event rate is also now presented as a test metric. No structure measures are evaluated or presented. One process measure is reported for radiation, which is the percentage of cases in which fluoroscopy time, skin dose, and dose area product are reported.

In terms of reporting, none of the organizations directly reports all of their findings publicly. US News and World Report’s total scores are available and the summation of catheterization volumes is made available on their website as part of their center scorecard for Pediatric Cardiology and Heart Surgery. Results from C3PO, IMPACT®, and CCISC are not publicly available, with member institutions able to review their center’s data against other institutions (with the specific other institutions anonymized).

Survey results

The study instrument was distributed to 229 cardiologists from 118 hospital programs in 38 of 50 US states. Of these, 145 individuals from 85 programs in 31 states responded. This represents a response rate of 66% in terms of individuals and 72% in terms of programs. Of those responding 92% (134/145) were willing to provide information about their individual and institutional experience. The median hospital catheterization volume reported was 500 cases/year (range: 100–1700 and IQR: 300–800) for these programs. The median individual physician experience was 11 years (range: 1–42, IQR: 6–20).

Survey results are depicted in Table 4 (with favorability ratings summarized in Supplementary Table 1). Within each of the five categories there were significant differences in agreement between different measures (p<0.001 for each).

Table 4.

Results of Survey of United States Pediatric/Congenital Interventional Cardiologists

	Strongly disagree	Disagree	Neutral	Agree	Strongly agree	Missing
SAFETY
Adverse events	3% (5)	10% (15)	15% (21)	49% (71)	19% (28)	3% (5)
Mortality	5% (7)	9% (13)	10% (15)	45% (65)	28% (40)	3% (5)
Failure to rescue	2% (3)	6% (8)	10% (14)	46% (66)	34% (49)	3% (5)
Procedure specific AE	1% (2)	2% (3)	5% (7)	61% (89)	26% (38)	4% (6)
	Very poor	Poor	Neutral	Good	Very good	Missing
CURRENT RISK ADJUSTMENT	3% (5)	32% (46)	32% (46)	30% (43)	3% (5)	3% (5)
OTHER OUTCOMES
Procedure specific technical success rate	1% (2)	2% (3)	6% (9)	55% (80)	31% (45)	4% (6)
Femoral artery pulse loss	5% (7)	12% (18)	25% (36)	41% (60)	12% (17)	5% (7)
Radiation exposure	1% (1)	9% (13)	14% (21)	51% (74)	20% (29)	5% (7)
Patient satisfaction scores	3% (5)	16% (23)	33% (48)	35% (51)	8% (12)	4% (6)
HOSPITAL CHARACTERISTICS
Pediatric CT operative volume	3% (4)	14% (21)	21% (30)	45% (65)	12% (17)	6% (8)
ECMO program	1% (1)	5% (7)	8% (12)	43% (62)	38% (55)	6% (8)
Heart transplant program	7% (10)	20% (29)	26% (38)	34% (49)	8% (11)	6% (8)
Dedicated pediatric cardiac or CT surgical intensive care unit	3% (4)	9% (13)	10% (15)	43% (63)	28% (41)	6% (9)
Dedicated cardiac preparatory and recovery unit	3% (5)	13% (19)	21% (31)	37% (54)	19% (28)	6% (8)
PCCL LAB CHARACTERISTICS
Lab annual catheterization volume	3% (5)	16% (23)	19% (27)	41% (60)	14% (21)	6% (9)
Operator annual catheterization volume	4% (6)	13% (19)	22% (32)	41% (59)	14% (20)	6% (9)
Procedure specific annual volume	3% (5)	8% (12)	19% (27)	44% (64)	19% (27)	7% (10)
Individual operator experience (years)	4% (6)	13% (19)	22% (32)	41% (59)	14% (20)	6% (9)
Dedicated pediatric cardiac anesthesia	1% (1)	3% (4)	11% (16)	51% (74)	29% (42)	6% (8)
Participation in device trials	6% (8)	26% (37)	30% (43)	28% (41)	6% (8)	6% (8)
Average number of nurses and/or technologists staffing each case	2% (3)	12% (17)	30% (43)	48% (69)	3% (5)	6% (8)
Use of a scrub person in addition to trainee assistants	5% (7)	20% (29)	37% (53)	27% (39)	5% (7)	7% (10)
Specific ratio of registered nurses to technologists on staff	3% (4)	31% (45)	39% (56)	19% (27)	3% (4)	3% (4)
Presence of categorical cardiology fellows	10% (14)	30% (44)	45% (65)	8% (11)	2% (3)	6% (8)
Presence of advanced PCCL interventional fellow	7% (10)	30% (44)	38% (55)	17% (25)	2% (3)	6% (8)
QUALITY IMPROVEMENT PROCESSES
C3PO participation	6% (9)	19% (28)	34% (50)	32% (47)	2% (3)	6% (8)
IMPACT® participation	9% (13)	12% (17)	23% (34)	39% (57)	10% (15)	6% (9)
CCISC participation	8% (11)	13% (19)	40% (58)	32% (46)	2% (3)	6% (8)
Radiation exposure monitoring program	1% (1)	1% (2)	8% (12)	66% (95)	18% (26)	6% (9)
Femoral artery pulse loss program	1% (2)	4% (6)	28% (40)	52% (75)	10% (14)	6% (8)

In terms of safety metrics, procedure-specific AE, FTR, and pooled AE had the highest favorability ratings (87%, 80%, and 73% respectively). These were significantly higher than all AE and mortality (See Supplementary Table 2 for results of pairwise comparisons). There was no significant difference in the favorability ratings between FTR and major AE (p=0.39) and between major AE and procedure specific AE (p=0.14). Procedure specific AE had a significantly higher favorable rating than FTR (p=0.02). Current risk adjustment was judged to be poor or very poor by 35% of respondents (51/145). This was significantly worse than any of the individual safety metrics (p<0.001 for all comparisons).

For other outcome metrics, there were significant differences in respondent agreement about their representativeness (p<0.001). Procedure specific technical success (90%) had the highest favorable ratings. It was significantly higher than radiation exposure (75%), femoral artery pulse loss (56%), and patient satisfaction scores (46%) (p<0.001 for all, Supplementary Table 2). Radiation exposure had higher favorable ratings than femoral artery pulse loss and patient satisfaction scores as well (p=0.001).

In terms of hospital characteristics, an extracorporeal membrane oxygenation (ECMO) program (85%) and dedicated pediatric cardiac intensive care unit (76%) had the highest favorable ratings with no significant difference between them (p=0.06). Their favorable ratings were significantly higher than those for annual CT OR volume (60%), heart transplant program (44%), and a dedicated cardiac preparatory and recovery unit (60%) (Supplementary Table 3).

In terms of lab characteristic, dedicated pediatric anesthesia (85%) had the highest favorable rating, which was significantly higher than other ratings (p<0.0001). Other lab characteristics had lower favorable ratings (Supplementary Table 4).

Radiation exposure monitoring programs had the highest favorable rating (90%) amongst quality improvement process measures. This was significantly higher than participation in registries and femoral artery pulse loss monitoring (p<0.001) which had lower favorable ratings (Supplementary Table 5).

Discussion

The current study provides a multidimensional evaluation of established and potential metrics of quality in the PCCL, combining a review of the peer-reviewed literature and organizations studying healthcare quality as well as the views of the practicing interventional cardiologists, to determine areas of consensus and areas where there is persistent disagreement. The study demonstrates that previously the exclusive focus of the field has been on measures of safety. Effectiveness, efficiency, timeliness, equity, and patient centeredness have not received the same attention. There is even less consensus regarding structures and processes. Further evaluation of these hospital-level factors is contingent on identifying valid and practically useful outcome metrics. Addressing these knowledge gaps should be priorities for future research.

Safety has been the subject of all published studies of PCCL quality metrics, the common domain reported by all three physician-organized organizations measuring quality (IMPACT®, C3PO, and CCISC), and the category of quality metric that practicing PCCL cardiologists reported the highest favorability ratings. However, precisely how to measure safety is not settled. Despite extensive research into risk adjustment (11, 25–27) and pre-procedural risk prediction(21–23), the majority of respondents to our survey were not confident that current risk adjustment is effective. Though there are high favorability ratings for individual safety metrics, there is not consensus about the optimal outcome, and research suggests that these metrics may reflect different aspects of quality. Specifically, in a recent study using the IMPACT® Registry, the correlation of rankings derived from case-mix adjusted AE and FTR were significantly different(7). Empirically determining the optimal outcome metric is challenging and requires one or more hospital-level characteristics that are indicative of quality independent of case-mix. As demonstrated in our systematic review, there is a dearth of these metrics with only annual procedure volume and operator experience offered as potential measures of quality. A possible solution to this would be to evaluate potential outcome metrics for safety against a panel of potential structure and process metrics to determine if there is coherence.

For other aspects of quality, there is clearly room for the development of new metrics. Some domains may not be as immediately germane to PCCL procedures as in other fields within medicine. For example, timeliness in the treatment of acute conditions such as myocardial infarction or cerebrovascular accident in the elderly is clearly important. Its importance is less relevant for most congenital conditions with some notable exceptions (e.g. balloon atrial septostomy in transposition of the great arteries). Other domains, however, have more obvious immediate application.

Our current survey demonstrates community interest in procedure-specific effectiveness measures. The IMPACT® registry reports technical success for a panel of core procedures as a metric for effectiveness to member institutions (blinded to other centers), but to date the utility of these as quality metrics between centers has not been evaluated. An important difference between potential effectiveness measures and the existent safety measures is that it is unlikely that there will be a single metric that spans the entire range of procedures for effectiveness. Several issues should be considered as new effectiveness measures are generated. Measuring effectiveness for procedures where technical success is nearly 100%(e.g. device closure of patent ductus arteriosus)(2) is unlikely to be useful. Instead, metrics should focus on procedures or contexts with significant variability in outcomes between centers. Design of metrics for technical success will have to be procedure specific. In balloon aortic valvuloplasty, for example, the competing outcomes of gradient reduction and development of aortic insufficiency will need to be considered(28–31). It is also important to recognize that the factors influencing the risk of AE and technical failure may differ(32), and that separate novel risk adjustment models will be necessary for each technical effectiveness metric. Additionally, current metrics (and the systems used to track them) are restricted to short-term outcomes, and there may be value in expanding the time horizon for outcomes (e.g. tracking the durability of transcatheter valves instead of immediate technical success). Finally, because the number of different procedure types performed in PCCL labs is large, measures of effectiveness will have to be similarly numerous or metrics will have to be restricted to a subset of procedures indicative of program quality.

Efficiency measures were not seen in any of the current study measures. However, across medicine, there is increasing interest in providing high-value care both by increasing the quality of care and reducing costs. Relative to their prevalence, congenital cardiac conditions consume a disproportionate amount of spending in children(33). To dates, metrics that combine efficacy (either in clinical measures or patient-reported outcomes) and economic impact have not been developed. Previous research has focused on cost alone as an outcome as a means of comparing operative and transcatheter treatments(34–37). Cost, however, has not been routinely used as a quality metric between centers. The presence of large magnitude variation between hospitals in the cost of transcathteter procedures(38) implies large variation in efficiency of care and the potential for improvement. As with other high-cost aspects of healthcare, congenital cardiology and PCCL procedures are likely targets for external pressure from both insurance payers and governmental agencies.

Evaluating equity and patient-centeredness are challenging using traditional data and analytic methodologies, and novel combinations of data sources (e.g. clinical data from registries with socioeconomic status from census data) and/or methodologies (qualitative research and patient-reported outcomes) are necessary to move forward.

For each new quality metric(s), the pattern of the metric’s distribution is also critically important. Several of the metrics identified in our survey were uncontroversial with very high favorability ratings, but also were also nearly universally present (e.g. an ECMO program or radiation monitoring). These measures define a metaphorical “floor” and identify only a small number of programs that fail to meet the standard. These metrics do not meaningfully differentiate quality between the majority of centers. This kind of discernment requires quality metrics with a broader range of values between centers. These metrics are likely to be more divisive and will need to be supported by empirical validation. Ultimately “grading” institutions will have to incorporate metrics across these domains, which may be complicated. Determining the weight given to each domain is not a purely mathematical question but should reflect the relative importance of metrics to PCCL cardiologists, cardiologists, patients, and their families.

In discussing how to measure PCCL quality, it is important to acknowledge the accelerating demand for transparency and public reporting. The current review found that no physician-organized PCCL registry reports center quality metrics. US News and World Report was the only group to report any aspect of PCCL, and the only aspects reported are the volume of certain procedures. This is in contrast to cardiothoracic surgery, where ratings are based on both volume and risk-adjusted mortality. In light of the fact that mandated public reporting appears to be inevitable, establishing valid and useful performance measures with appropriate case-mix adjustment is imperative. Appropriately determining how best to account for 1) the broad range of procedures and diagnoses that are part of PCCL practice 2) the range of physiologic derangement and non-cardiac covariates, and 3) the tricky attributability of adverse outcomes in patients who pass between PCCL and cardiac surgical therapy is vital to insure that publicly reported data are an accurate representation of programmatic quality.

The path forward to developing quality metrics is challenging. To date, there has been a dearth of clinical trials in pediatric cardiology and even fewer in PCCL cardiology(39), which is reflected in the limited evidence base for current guidelines for PCCL interventions(40, 41). Previously, large administrative datasets such as the Pediatric Health Information Systems Database have lacked important clinical data, while clinical registries such as IMPACT® have lacked data about hospital-level characteristics and data use agreements have prohibited combining hospital and lab characteristics with their clinical data. While individual centers can describe innovations in their own practice, it has not been possible to study how their performance compares to other centers. Progress is dependent on assembling a group of centers willing to share data about their outcomes and practices. Identifying and evaluating metrics in a smaller sample of motivated centers may be necessary to build sufficient momentum to evaluate metrics in larger national registries like IMPACT®. The effectiveness of this kind of approach has been demonstrated in the pediatric cardiac intensive care unit, where collaboration of different programs was associated with significant reductions in mortality and major complications in post-operative patients not seen in hospitals outside the collaborative(42).

There are several limitations that we acknowledge. First, the absence of evidence for a performance measure does not imply that the performance measure is invalid, only that there is a knowledge gap that needs to be addressed. The schemae used to classify quality metrics are well established but may not describe all possible quality metrics. The survey was limited in terms of the breadth and number of characteristics and consequently not comprehensive. It was also developed without field or focus group testing. Favorability within the survey is not proof of validity, and establishing the validity of various quality metrics is beyond the scope of this project. As noted, future research is needed to evaluate these issues. Finally, this review has focused on measures of PCCL programs as whole (and including technologists, nurses, and other physicians) rather than on individual physicians. Outcomes that are informative about hospital quality are likely also informative about individual physicians, but further research is necessary to disentangle the relative contributions of individual physicians from their teams.

CONCLUSIONS

We conclude that there is some consensus in the PCCL community that measures of safety (risk of adverse events) are an important measure of quality in the PCCL, reflected both in published literature, organizations monitoring quality, and in the opinions of practitioners. There remains significant concern that risk-adjustment and the applicability of current AE measures are imperfect. There is far less consensus about structures and processes that are associated with good quality. These knowledge gaps should guide future research. Empirical evaluation of these quality metrics will require a novel approach combining data about patient and procedure characteristics, hospital and practitioner characteristics, and outcomes – data that are not available in current data sources.

Clinical Perspectives

What is Known?

Defining valid quality metrics are a foundational step in outcomes research and quality improvement. Agreement about how to measure quality is a necessary first step towards studying how to improve it. To date, a systematic evaluation of quality metrics has not been performed for pediatric congenital cardiac catheterization laboratory(PCCL) programs.

What is New?

The current study provides a multi-dimensional evaluation of quality metrics for PCCL programs through a systematic review of the scientific literature, a review of quality organizations, and a survey of US PCCL cardiologists. It demonstrates that the focus of previous research has been on measures of safety, while also showing that there is limited confidence in current risk adjustment models. There are important knowledge gaps in 1) hospital and program characteristics that are associated with quality and 2) quality metrics evaluating domains other than safety (e.g. effectiveness and efficiency).

What is Next?

Research and quality improvement should focus on addressing these knowledge gaps. Specifically, it is important to systematically evaluate current metrics for safety against a panel of hospital quality metrics to determine which is the optimal metric. Identification of quality metrics for other domains should occur in parallel. Accomplishing these goals requires access to case- and hospital-level data from a large number of programs.

Abbreviations:

AE

Adverse events

ECMO

Extracorporeal membrane oxygenation

FTR

Failure to rescue

C3PO

Congenital Cardiac Catheterization Project on Outcomes collaborative

CCISC

Congenital Cardiovascular Interventional Study Consortium Risk Registry

IOM

Institute of Medicine

IMPACT®

Improving Adult and Congenital Treatment Registry

PCCL

pediatric congenital cardiac catheterization laboratory

US

United States of America