Development of a Congenital Heart Surgery Composite Quality Metric: Part 1—Conceptual Framework

Abstract

Background.

Current pediatric and congenital heart surgery quality measures focus on operative mortality, and numerous stakeholders are interested in more comprehensive measures. This report describes the background, rationale, and conceptual framework related to the development of the first composite quality metric in the field.

Methods.

A multidisciplinary panel reviewed methodology and framework related to quality measurement and several composite quality measures across adult cardiac surgery and other fields. The panel subsequently developed methodology and selected measures for a congenital heart surgery composite measure and reviewed potential advantages and limitations. Individual measures considered for potential inclusion in the composite were reviewed within the context of Donabedian’s triad and the Institute of Medicine quality domains. Decisions were made through group consensus.

Results.

The final composite measure selected is comprised of two domains: (1) a mortality domain (operative mortality) and (2) a morbidity domain (the 6 major complications endorsed by The Society of Thoracic Surgeons and Congenital Heart Surgeons Society plus cardiac arrest, and postoperative length of stay). Potential advantages include the more comprehensive view of quality compared with mortality alone and improvements in discrimination of hospital performance through increasing the number of end points. Potential limitations include the lack of longer term outcomes and challenges related to case-mix adjustment.

Conclusions.

We have applied and adapted conceptual framework and methodology related to composite quality measures across other fields to congenital heart surgery. The composite quality metric created is inclusive of both morbidity and mortality, and expands our view of quality in this patient population.

Measures of quality of care are important to numerousstakeholders, including physicians, patients and families, administrators, payers, and policy makers. Across medical and surgical specialties, quality metrics are used in a variety of efforts that are geared toward improving quality of care and outcomes and reducing variation across hospitals. These activities include benchmarking and feedback to providers and hospitals to inform quality improvement initiatives, public reporting, designation of centers of excellence, and pay for performance programs. In understanding the use and further development of quality metrics in the field of congenital heart surgery, it is useful to first consider the overall framework and concepts related to quality measurement.

Defining Quality

Although quality metrics are now widely used as the basis for many initiatives, defining quality can be challenging. Quality is an abstract construct that cannot be directly measured but which may be estimated based on various observable characteristics or end points [1]. The overt, observable manifestations of quality may be intuitively apparent at the extremes but often are difficult to quantify. For example, observers may agree that a penthouse suite is high quality and that a studio apartment is of lower quality. However, the exact variables that allow this classification to be made, and their relative importance, can be more complicated to delineate [1].

Donabedian’s Triad

In health care, quality is often characterized by a combination of structure, process, and outcome measures, as first proposed by Donabedian [2].

Structure

Structure measures refer to physical facilities (eg, a dedicated cardiac intensive care unit), staffing (eg, 24/7 in-house coverage, nurse staffing ratios), technologies, or other characteristics (eg, surgical volume) inherent to the provider or hospital. Structure measures are most useful when outcomes data are not available or are unreliable [1]. It is important to note that their relationship with outcome measures may be variable (eg, the volume-outcome relationship in congenital heart surgery is most robust for high complexity procedures) [3]. In addition, some structure measures may not be readily modifiable by providers, limiting their usefulness in quality improvement initiatives [1]. Finally, the use of certain structure measures in isolation could lead to unintended negative consequences (eg, loosening appropriateness criteria for echocardiograms to meet a volume threshold) and in certain cases could also be reflective of lower overall quality (eg, a higher volume of interventional catheterizations reflective of a high rate of postsurgical residual lesions).

Process

Process measures refer to the way care is delivered or adherence to certain care practices (ideally evidence based). Similar to structure measures, they can be useful when outcome data are not available and have the advantage of being actionable by providers in quality improvement initiatives [1]. However, their use can also be limited by several factors, including the limited evidence in many cases to support the association of various practices with patient outcomes (a key issue in the field of congenital heart surgery) or inadvertent creation of incentives to perform unnecessary testing or interventions to comply with the measures.

Outcome

Outcome measures represent the “bottom line” or final product of care delivery and are the most obvious and intuitive measures of quality. They are generally the most important metric to patients and currently the most widely used type of quality measures [1, 4]. Their use can be limited, however, when the event rate is low or when sample sizes are small [5]. Other important considerations include the costs associated with detailed data collection and analytics related to the use of outcome metrics [6].

Institute of Medicine Quality Domains

In addition to Donabedian’s triad of structure, process, and outcome, the Institute of Medicine has further delineated six domains of quality [7]:

Safe: Avoiding harm to patients from the care that is intended to help them.

Effective: Providing services based on scientific knowledge to all who could benefit and refraining from providing services to those not likely to benefit (avoiding underuse and misuse, respectively).

Patient-centered: Providing care that is respectful of and responsive to individual patient preferences, needs, and values and ensuring that patient values guide all clinical decisions.

Timely: Reducing waits and sometimes harmful delays for both those who receive and those who give care.

Efficient: Avoiding waste, including waste of equipment, supplies, ideas, and energy.

Equitable: Providing care that does not vary in quality because of personal characteristics such as sex, ethnicity, geographic location, and socioeconomic status.

Quality Measurement in Congenital Heart Surgery

Important Advances to Date

Several important advances over the past three decades have supported the application of the quality framework described in the preceding sections and development of quality measures in the field of congenital heart surgery. First, the development of clinical registries has allowed the capture of widespread, standardized, validated data across congenital heart centers. The Society of Thoracic Surgeons Congenital Heart Surgery Database (STS-CHSD) is the largest in this specialty worldwide, and it now includes data from 119 centers and more than 425,000 operations [8]. It is estimated that more than 90% of US congenital heart programs participate in this registry [9]. These efforts have been critical in ensuring accurate assessments of quality, because it has been shown that the use of administrative or billing data sets can lead to discrepancies in case ascertainment and outcomes assessment in the congenital heart surgical population [10].

A second related important advance has been the development of uniform nomenclature (the International Pediatric and Congenital Cardiac Code) for the vast number of congenital heart disease diagnoses and procedures and uniform definitions for other variables such as patient characteristics and complications by the Nomenclature Working Group of International Society for Nomenclature of Pediatric and Congenital Heart Disease and the Multi-societal Database Committee [11]. This work has been critical in facilitating uniform data capture across centers.

Finally, a third important advance has been the development and refinement of methodologies to account for differences in case-mix across centers when assessing quality. This is particularly important in congenital heart surgery because of the heterogeneity of congenital heart disease and known wide variation in case-mix across centers. The current mortality risk models used in the STS-CHSD include adjustment for both differences in the type of congenital heart procedure performed and differences in patient factors (age, weight, prior cardiothoracic operation, prematurity, non-cardiac anatomic anomalies, chromosomal abnormalities/genetic syndromes, and a variety of STS-defined preoperative risk factors) [12]. Further refinements are under way.

Current Quality Measures

The most widely used quality metric in pediatric and congenital heart surgery, as in most other surgical fields, is mortality. The STS-CHSD currently provides bench-marking feedback reports to centers about case-mix adjusted operative mortality and offers voluntary public reporting of this metric [8]. Mortality metrics are also used by certain US states, other clinical registries, and quality collaborative in the field, payers, and the lay press, including US News & World Report and Consumer Reports, who all evaluate and report on hospital quality in congenital heart surgery. These initiatives also report on a nonuniform variety of other types of structure, process, and outcome metrics [13–17]. The STS-CHSD mortality metric along with a mortality metric that can be reported by using administrative data are both endorsed by the National Quality Forum [15].

Limitations

The primary limitation of the widely used metric focused on operative mortality is its unidimensional nature. Although mortality is certainly an important outcome, early survival after congenital heart surgery has dramatically improved over the past several decades, and current data demonstrate that the overall operative mortality rate among centers participating in the STS-CHSD is 3.0% [8]. Thus, metrics focused solely on operative mortality are applicable to less than 95% of patients undergoing congenital heart surgery and do not take into account important morbidities survivors may experience. Recent studies suggest that up to 30% of patients can experience major complications after congenital heart surgery [18]. Parents, providers, and other stakeholders are interested in a child not only surviving surgery but thriving once the child is discharged home free of major morbidities [4]. From a health system and policy perspective, children with major morbidities are also more likely to consume greater health care resources [19]. For these reasons, incorporation of morbidity as a component in quality assessment is important and addresses the Institute of Medicine principles of patient-centered and safe care (in addition to effective care). An additional limitation of the use of mortality alone is that it has relatively limited ability to discriminate hospital performance because of the low overall event rate in the current era [8]. Further, it can be difficult to make sense of the multitude of other reported metrics aside from mortality or to integrate them into a comprehensive view of quality. Other important limitations related to the use of short-term procedural mortality are discussed further in subsequent sections.

Composite Quality Measures

Composite measures combine information across multiple domains of quality and can address many of the limitations associated with the use of individual quality metrics [1, 20]. Composite quality measures have been developed and are used across multiple medical and surgical specialties, including several in adult cardiothoracic surgery, and are used by payers, professional societies, and others in efforts to assess and improve quality [1, 20–22]. Composite metrics can allow a more comprehensive assessment of quality through combining information across multiple quality domains. Specifically with regard to outcomes, they can incorporate both morbidity and mortality end points, thereby addressing the qualitative limitations of quality assessment limited to mortality. Quantitatively, composite measures effectively increase the number of end points, thus increasing the ability to discriminate hospital performance compared with using mortality alone [1, 20].

Framework and Development of a Congenital Composite Quality Measure

Initial Steps

To support the development of a composite quality measure in congenital heart surgery, a multidisciplinary team of investigators comprised of congenital heart surgeons, pediatric cardiologists, and experts in health services research and quality measurement methodology successfully applied for funding from the National Heart, Lung, and Blood Institute (R01HL122261; Dr Pasquali principal investigator). The team included representatives from the STS-CHSD Taskforce, and the STS Quality Measurement Task Force. In addition to composite measure development and evaluation, subsequent aims of the grant will also assess the relationship between various aspects of quality and health care costs (in accordance with the Institute of Medicine’s domain of efficient health care), through combining clinical data from the STS-CHSD with resource use data from the Children’s Hospital Association.

The 13 member multidisciplinary team reviewed methodology and framework related to the development of several other composite measures within adult cardiac surgery and across other fields and subsequently developed a conceptual framework and methodology toward developing a congenital heart surgery composite measure. Decisions were made through iterative discussion and group consensus.

Measure Selection

Measures were considered for inclusion in the composite and reviewed within the context of Donabedian’s triad and the Institute of Medicine domains [2, 7]. We primarily considered the set of 21 measures that had been previously reviewed and endorsed by the STS and the Congenital Heart Surgeons Society (CHSS) [23].

STRUCTURE MEASURES.

The primary structure measure considered for inclusion was center surgical volume because this has been the most widely studied in the field. Although many studies have now demonstrated a relationship between center volume and outcomes, it is also known that this relationship is variable and primarily applicable to higher complexity operations [3, 24]. Even for these higher complexity operations, one study demonstrated that Norwood volume explained only a minority (14%) of the observed between-center variation in Norwood outcomes [24]. For these reasons, center surgical volume was not included in the proposed composite measure, and it was decided to focus more directly on outcome measures. Future analyses are planned to further understand contemporary volume–outcome associations as well as the association between other structure measures and patient outcomes [23].

PROCESS MEASURES.

Process measures were also considered for inclusion in the composite, but ultimately they were excluded because of limited evidence about the relationship of most care practices and processes with outcomes in pediatric and congenital heart surgery. Similar to structure measures, the ongoing collection of data within the STS-CHSD about several of the STS-and CHSS-endorsed process measures (multidisciplinary preoperative planning conference, quality assurance/ improvement conference, availability of transesophageal echocardiography, timing and type of preoperative antibiotics, use of expanded preoperative and postoperative time-out or debrief) and other multicenter efforts may allow for further refinement, analyses, and understanding of these measures in the future [23].

OUTCOME MEASURES.

Ultimately, the multidisciplinary team elected to focus on outcome measures because recent analyses confirm these are of greatest interest and importance to patients and families [4] and because of the previously described limitations of current structure and process measures. The specific outcome measures selected and rationale are described below.

FINAL COMPOSITE.

The panel elected to include both a mortality domain and a morbidity domain (Figure 1).

Fig 1.

Congenital heart surgery composite quality metric. (LOS = length of stay.)

• Mortality domain: The mortality domain included the measure of operative mortality. In the STS-CHSD, operative mortality is defined as any death occurring in hospital, and any deaths occurring after discharge within 30 days of the operation. Further, in-hospital deaths include deaths in the hospital performing the operation, or in another acute care facility to which the patient is transferred, or in a long-term care facility up to 6 months after transfer [8, 12]. It was acknowledged that an understanding of longer-term disease-based (rather than procedure-based) outcomes would be ideal, and that ongoing efforts to capture this information may allow for incorporation of such metrics in the future [25].

• Morbidity domain: The morbidity domain included two main components: (1) major complications and (2) postoperative length of stay (LOS).

  • Major complications: The set of major complications (Table 1) previously endorsed by the STS and CHSS were included: renal failure requiring dialysis, neurologic deficit persisting at discharge, arrhythmia requiring permanent pacemaker, paralyzed diaphragm/phrenic nerve injury, mechanical circulatory support, and unplanned re-intervention (includes surgical or catheter-based unplanned interventions, and cardiac and noncardiac procedures) [23]. These complications were selected by the STS and CHSS based on review of available literature about durable impact of these complications on longer term outcomes and the availability of clear definitions in the STS-CHSD [23]. Other analyses have shown that each of these complications is associated with substantial increases in resource use as well [19].
    In addition to these major complications recognized by STS and CHSS, the investigator panel also considered the inclusion of other complications, including cardiac arrest. Previously, there had been concern about certain aspects of the definition of cardiac arrest in the STS-CHSD and uniform coding. However, the definition has been revised over time to align more with accepted definitions used across other registries, and a review by the investigative panel of cardiac arrest rates in the STS-CHSD compared with other registries collecting detailed information on cardiac arrest showed similar event rates (data not shown). In addition, there are ongoing efforts in the field specifically geared toward reducing cardiac arrest and variation across centers, including those led by the American Heart Association and the Pediatric Cardiac Critical Care Consortium, emphasizing the importance of this complication [26, 27]. Finally, the panel reviewed prior work demonstrating the impact of cardiac arrest on outcomes as well as resource use [19, 26, 27]. For these reasons, the panel decided to include cardiac arrest in the list of composite complications (Table 1).
    The panel also reviewed other complications for inclusion. Many, such as mediastinitis, were captured in large part by other codes already included (eg, mediastinitis requiring reintervention would be captured by the reintervention codes). In other cases, there were relatively limited data about durable long-term impact.
    Overall, the panel acknowledged that similar to the mortality domain, there was a need to move beyond short-term outcomes alone to better inform future iterations of the composite measure. Longer term outcomes data could allow more complete assessment of the current subset of complications included (eg, unplanned reinterventions that occur in subsequent hospitalizations) and better inform which complications have the greatest impact on longer term outcomes such as quality of life, survival, and functional status.
  • Postoperative length of stay: The second component or measure included in the morbidity domain consisted of postoperative LOS. LOS was included for several reasons. It is an additional measure of morbidity and may capture other components of quality of care and the influence of other complications aside from the major complications described above. In addition, LOS is known to vary widely across centers and substantially affect resource use [19, 28]. The impact of LOS on resource use persists even after adjusting for complications, suggesting that there may be unique or additive important information conveyed by LOS [19]. Both complications and LOS were included in a previously developed STS-CHSD statistical model that defined the morbidity risk associated with different surgical procedures [18]. Finally, prior studies have shown that prolonged LOS has a deleterious impact on longer term neurodevelopmental outcomes, likely reflective of LOS serving as a marker of overall postoperative morbidity [29]. The group also considered metrics associated with LOS such as duration of mechanical ventilation; however, not all STS-CHSD centers submit complete data for this variable. Other organizations focused primarily on care in the intensive care unit collect more detailed information on mechanical ventilation (such as the Pediatric Cardiac Critical Care Consortium) and provide a platform for understanding detailed case-mix adjusted mechanical ventilation data at participating sites [16].

Table 1.

STS-CHSD Codes for Complications Included in the Composite Measure

STS-CHSD Code	Complication
230	Renal failure, acute renal failure; acute renal failure requiring dialysis at the time of hospital discharge
223	Renal failure, acute renal failure; acute renal failure requiring temporary dialysis with the need for dialysis not present at hospital discharge
224	Renal failure, acute renal failure; acute renal failure requiring temporary hemofiltration with the need for dialysis not present at hospital discharge
320	Neurologic deficit; neurologic deficit persisting at discharge
74	Arrhythmia necessitating pacemaker, permanent pacemaker
40	Postoperative/postprocedural mechanical circulatory support
300	Paralyzed diaphragm (possible phrenic nerve injury)
22	Unplanned cardiac reoperation during the postoperative or postprocedural time period, exclusive of reoperation for bleeding
24	Unplanned interventional cardiovascular catheterization procedure during the postoperative or postprocedural time period
26	Unplanned noncardiac reoperation during the postoperative or postprocedural time period
240	Bleeding, requiring reoperation
30	Cardiac arrest, cardiac arrest during or after procedure

STS-CHSD = The Society of Thoracic Surgeons Congenital Heart Surgery Database.

Patient Population

The composite quality measure will initially be developed for planned application in the population of patients undergoing any congenital cardiovascular operation with or without bypass, similar to the population for which operative mortality is currently reported in the STS-CHSD. Recent analyses have also highlighted the potential advantages of focusing on more homogeneous subsets of patients in quality assessment, such as those undergoing the STS benchmark operations or certain disease base cohorts [30]. Future work may explore the application of the composite measure in these populations.

Case-Mix Adjustment

Adjustment for differences in case-mix across centers remains a critical issue in quality assessment regardless of the use of individual quality measures or composite measures. This issue is particularly challenging in the field of congenital heart surgery because of wide heterogeneity of disease and substantial variation across centers in patient population and complexity. The STS-CHSD risk models have been updated and refined in recent years to include more detailed adjustment for procedure type and patient characteristics and morbidities [12]. Further improvements are under way, and it is anticipated that over time the most up-to-date and refined STS-CHSD risk models will be used for case-mix adjustment with the composite measure.

Reporting and Interpretation

It is anticipated that the composite measure and individual components will be included in STS-CHSD feedback reports to participating centers and once further refined and evaluated may also be publicly reported on the STS website for centers participating in this program in the future. As with other congenital heart surgery metrics reported in the United States and other countries, the composite measure will be reported at the program level rather than the surgeon level in recognition of the impact of the entire cardiac team on quality of care and outcomes.

Similar to individual quality measures, appropriate interpretation is also critical for composite measures. In general, the methods used allow one to understand how a center is performing in the context of what would be expected for the center’s specific case-mix, but they are not intended to be used to directly compare or rank centers one against another [30]. This important topic is discussed further in Part 2 of this report [31].

Comment

We describe the background, rationale, and framework related to the development of the first composite quality measure in the field of pediatric and congenital heart surgery. The measure includes both a mortality domain (operative mortality) and morbidity domain (major complications and postoperative LOS). It can provide a more comprehensive view of quality than mortality alone and can improve our ability to discriminate hospital performance through effectively increasing the number of end points. Further efforts related to longer term outcomes assessment and improved case-mix adjustment are needed to refine and improve future iterations of the composite measure. Part 2 of this report describes the statistical methods used to develop this initial composite measure and several reporting considerations [31].