Quality Indicators in Pediatric Orthopaedic Surgery: A Systematic Review

Abstract

Background

The ability to measure health system quality has become a priority for governments, the private sector, and the public. Quality indicators (QIs) refer to clear, measurable items related to outcomes. The use of QIs can initiate local quality improvement and track changes in quality over time as interventions are implemented.

Questions/purposes

We identified existing evidence-based indicators of quality pediatric orthopaedic care and evaluated published QIs that may be applicable to pediatric orthopaedic care.

Search Strategy

Using five standard search engines we searched the literature using terms such as “quality indicators,” “orthopaedic surgery,” and “pediatric.” Study selection was performed in a stepwise manner, first by title, then abstract, and then full-text review. Of the 604 citations identified, 13 articles were selected for inclusion. Eight papers included only pediatric patients.

Results

The most commonly reported indicator was mortality followed by postoperative complications. Reoperation and readmission rates were also reported along with patient-centered QIs, although with less frequency.

Conclusion

Although mortality and postoperative complications were the most frequently reported QIs, concern for their applicability was raised because of their relative infrequency in pediatrics. Patient-centered QIs appear to be the most useful tools reported, although their use is somewhat limited in the published literature. Although there are benefits and drawbacks to all reported QIs, patient-centered and surgeon-defined outcomes along with cost-effectiveness have important roles in evaluating the quality of pediatric orthopaedic care.

Introduction

Health system improvement has become a priority for governments as well as the public. In 2004, unintentional and intentional injuries were estimated to cost Canadians $19.8 billion with almost 10% of this a result of falls in patients younger than 19 years of age [38]. Although the economic burden of injury is documented, it is still unclear as to what impact quality of care has on the overall health of patients. Currently, almost one-third of children with time-sensitive orthopaedic problems will have a problem with diagnosis, treatment, or delay in treatment before arriving at a tertiary pediatric center [37]. Various national and regional bodies in Canada and the United States now mandate reporting on health system quality [1, 18, 23, 32]. This has created a pressing need for published information on best practices to study and report on health system quality.

The assessment of surgical performance has evolved considerably over the past decades through the reporting of quality indicators. Quality indicators (QIs) refer to well-defined, structural, and process measures that are associated with outcomes. Although adult surgery has been the initial focus of public reporting efforts, pediatric surgery is increasingly included [6, 15, 27, 34]. Some efforts have been made in congenital heart surgery [20], juvenile arthritis [3, 13] and trauma [29, 40] to examine outcomes. The Agency for Healthcare Research and Quality has commissioned a report on pediatric QIs and although postoperative hemorrhage and hematomas were included, no specific orthopaedic indicators were investigated [1]. Postoperative infection is believed to be important; however, standard hospital-based surveillance systems often fail in their ability to detect postdischarge surgical site infections [25]. There is a movement to implement a pilot project to assess mortality and perioperative morbidity for inpatient surgical procedures in children through the administrative database Children’s National Surgical Quality Improvement Program [27]. However, this program and others are not designed to address pediatric surgical care specifically and have reported challenges and barriers to implementation [35].

There is increasing evidence that the evaluation of pediatric surgical programs for quality is more complicated than the simple comparison of mortality rates or case volume, and currently, well-defined metrics specific to the pediatric population are scarce in the medical literature as a whole [31, 36]. Although the many unique qualities of pediatric surgical care provide challenges to the development of a quality improvement program, QIs can provide an opportunity to initiate local quality improvement and track changes in quality over time as interventions are implemented [4, 22]. Research focused on developing, evaluating, or improving QIs relevant to the pediatric population is critical.

The aim of this systematic review is to assess the existing evidence-based indicators of quality of care in pediatric orthopaedics. First, we aim to identify published QI through a survey of the literature. Second, we evaluate existing QIs by reporting on their use in pediatric orthopaedic care. Third, we make recommendations for the future of QIs in pediatric orthopaedic care.

Search Strategy and Criteria

A search of the literature using the MEDLINE, Cochrane Database of Systematic Reviews, CENTRAL (Cochrane Central Register of Controlled Trials), and the Journal of Pediatric Orthopedics databases was performed using key words and MeSH headings. Relevant articles were included in this review if they described or used any quality-of-care indicator in pediatric orthopaedic surgery or any orthopaedic surgical specialty. Articles that did not describe or use a quality-of-care indicator in their outcome measures or did not include any pediatric patients or patients specific to an orthopaedic specialty were excluded from this review. There were no restrictions based on language, publication date, or study design. The search was conducted in April 2011 and a comprehensive list of studies was compiled in bibliographic software.

The MEDLINE search strategy was: “quality indicator$.mp. or Quality Indicators, Health Care/ “Quality of Health Care”/ or Quality Assurance, Health Care/ or “Outcome and Process Assessment (Health Care)”/ or quality measure$.mp. or Quality Indicators, Health Care/ or performance improvement.mp. or best practice.mp. AND Orthopedic Procedures/ or Orthopedics/ or orthopaedic$.mp.”

Two of us (CB, AK) critically appraised the studies and independently extracted the data from each study. Study selection was performed in a stepwise manner, first by title, then abstract, and then full-text review. Bibliographies of the identified articles were studied to include any missed articles.

The search strategy generated 604 citations (Fig. 1). Thirteen articles met our inclusion criteria and are reviewed in this article (Table 1) [2, 9–11, 14, 16, 17, 21, 24, 26, 30, 33, 34]. Eight papers specifically looked at pediatric patients and included no adults. Two papers were specifically focused on orthopaedics alone and the remaining 11 papers included some orthopaedics patients plus patients from at least one other specialty. Most study designs were either a systematic review or survey (five of 13 articles) or a retrospective review of administrative databases (five of 13 articles). Only one paper was a prospective intervention study evaluating the impact of the World Health Organization Safety Checklist on major complications.

Fig. 1.

The study selection trial flow diagram is shown.

Table 1.

Literature review for quality indicators in pediatric orthopaedics

Reference	Study design	Objectives	Quality indicators identified	Comment
Beal et al. [2]*‡	Systematic review	To collect healthcare quality measures for child health	396 measures identified from 19 healthcare quality measure sets	Measures of healthcare quality for children are not as well developed as those for adults Most measures fell within the effectiveness domain and few within the safety domain
Chambers and Clarke [9]‡	Retrospective review of hospital administrative data	To assess the feasibility of measuring readmission rates as a health service indicator	Readmission rate	Readmission requires gender and age adjustments Authors cautioned against using readmission rates to compare between specialties
Cox and Clarke [10]*†	Retrospective chart review of all nonaccidental orthopaedic patients younger than age 15 years	To determine the population statistics and success and failure of conservative and surgical orthopaedic treatment	Reoperations and readmissions Salvage operative procedures Length of stay Mortality rate	Audit of one hospital without comparison to other hospitals or the literature Mortality rate was zero
Croft et al. [11]‡	Retrospective review of hospital administrative data	To review trends in wait times and the impact on wait list initiatives from five military hospitals in the United Kingdom	Wait times using an aggregated monthly “weighted mean” to compare between hospitals	Surgical wait times were 50% higher than medical wait times Wait list initiatives in orthopaedics had only a temporary impact on reducing wait times
Gallagher [14]*‡	Descriptive	To illustrate the process for statistically based clinical pathway development	Identified a five-step pathway that includes: - Assessment of literature and existing practice parameters - Defining patient population - Resource evaluation -Quality functions (performance and outcome indicators) - Service design and development	Not a specific evaluation of quality indicators but targeted the process for development of practical clinical pathway
Haynes et al. [16]‡	Prospective intervention	To evaluate the mortality and complication rate changes following the implementation of the World Health Organization Surgical Safety Checklist	Major complications (death, acute renal failure, bleeding requiring the transfusion, cardiac arrest, coma, deep vein thrombosis, myocardial infarction, unplanned intubation, ventilator use for 48 hours or more, pneumonia, pulmonary embolism, stroke, major disruption of wound, infection of surgical site, sepsis, septic shock, the systemic inflammatory response syndrome, URTT, vascular graft failure)	After introduction of checklist, postoperative complication rates in hospital death rates dropped by 36% In-hospital death rate dropped from 1.5% to 0.8% Surgical site infection rate dropped significantly
Idvall et al. [17]‡	Survey	To compare patient and nurse assessments of the quality of care in postoperative pain management	The Strategic and Clinical Quality Indicators in Postoperative Pain Management questionnaire which encompasses four subscales: communication, action, trust, and environment	Notable differences in patients’ and nurses’ reports
Lanford et al. [21]*	Survey	To assess patient and family perception of care among Shriners Hospitals for Children (SHC)	Used the Pickler Survey, which measured dimensions of respect for patient values, coordination of care, information and education, and physical comfort/pain management, emotional support/alleviation of fear and anxiety, involvement of family and friends and discharge planning/continuity and transition	Each SHC developed internal mechanisms for improvement so systemwide comparisons were not possible
McDonald et al. [23]*	1) Systematic literature review and expert panelists survey 2) Retrospective review	To refine previous quality indicators produced by the Agency for Healthcare Research Quality (AHRQ) for a pediatric population and compare the national rates for each indicator	18 pediatric quality indicators identified (accidental puncture/laceration, decubitus ulcer, foreign body left in procedure, iatrogenic pneumothorax in neonates and at-risk neonates, pediatric heart surgery mortality rate, pediatric heart surgery volume, postoperative hemorrhage and hematoma, postoperative respiratory failure, postoperative sepsis, postoperative would dehiscence, selected infection, transfusion reaction, asthma admission rate, diabetes short-term complications, gastroenteritis admission rate, perforated appendix admission rate, urinary tract infection admission rate)	General pediatric surgical complication rates (not developed or compared specifically with rates in pediatric orthopaedics)
Moore [25]†	Short communication	To report on outcomes between teaching and nonteaching hospitals	Mortality rate Complication rate	Teaching hospitals have better outcomes than nonteaching hospitals Teaching hospitals with orthopaedic residency programs have the strongest results
Pike et al. [29]*	Expert panelists survey for quality indicator development using modified Delphi approach	To develop injury indicators to monitor injury prevalence and prevention in Canadian children	34 indicators were identified for usefulness and ability to prompt action across seven domains: - Indicators that span all domains (mortality rate, years life lost, hospital separations rate) - Overall health service implications (diagnosis specific hospital separations, injury severity hospital admission, length of stay) - Motor vehicle injury (cost of injuries, crash rate, intersection crash rate, rural roadways, drunk driving, speed, young drivers, graduated driver licensing, child restraints, unrestrained injuries, children restrain laws) - Sport, recreation, leisure (bicycle helmet laws, cost of injuries, percent sport specific injuries, CSA standard playgrounds, pool fencing legislation - Other policy (window guard bylaw, hot water tap temperature provincial laws) - Violence (violent crime rate, inflicted childhood neurotrauma rate, suicide prevention) - Trauma care, quality, and outcomes (access to pediatric Level I trauma care, appropriate use of pediatric Level I trauma care, quality of trauma system, prehospital transport time, presence of coordinated pediatric trauma system)	Injury indicators have overlap with pediatric orthopaedics Identified indicators based on usefulness and ability to prompt action to reduce injury Further research is needed to determine the use of these indicators
Scanlon et al. [32]*	Retrospective chart review	They sought to establish rates for each pediatric quality indicator, investigated the accuracy of these indicators and determined whether a complication was preventable or not	Accidental puncture or laceration Infection caused by medical care Foreign body left in after a procedure Postoperative hemorrhage or hematoma Postoperative respiratory failure Postoperative sepsis Decubitus ulcer	Retrospectively reviewed number of events to determine if the event occurred at admission or was preventable, nonpreventable, or uncertain
Sedman et al. [33]	Retrospective review of hospital administrative data	To apply patient safety indicators developed by the AHRQ to the National Association of Children’s Hospitals and Related Institutions (NACHRI)	11 indicators were reviewed (complications of anesthesia, death in low mortality DRGs, decubitus ulcer, failure to rescue, foreign body left in during procedure, iatrogenic pneumothorax, infection attributable to medical care, postoperative hemorrhage or hematoma, postoperative pulmonary embolus or venous thrombosis, postoperative wound dehiscence, and accidental puncture/laceration)	Indicators were risk-adjusted by age, gender, and AHRQ comorbidity groups: - Foreign body left in during procedure, iatrogenic pneumothorax, infection attributable to medical care, decubitus ulcer, and venous thrombosis were deemed appropriate for pediatric care and amenable to system changes - Failure to rescue and death in low mortality DRGs were inaccurate for the pediatric population

* Pediatric subjects only; ^†orthopaedic specialty only; ^‡mixed specialties; AHRQ = Agency for Healthcare Research and Quality; DRG = diagnosis-related group; URTT = unexpected return to theatre; SHC = Shriners Hospitals for Children.

Results

The most common indicator cited was mortality (six of 13 articles). Of these six articles, four of the papers specifically looked at pediatric patients.

Postoperative complications were the second most commonly cited QIs in our review (five of 13 articles); however, further study and validation were recommended. Only one of these papers was specifically looking at orthopaedic surgery and this paper was a short communication reporting that teaching hospitals have better outcomes than nonteaching hospitals [26]. The need for risk adjustment was reported along with the difficulty and added complexity in performing this in pediatric patients.

Reoperation and readmission rates were the next most frequently reported QIs [9, 10, 16] along with patient-centered QIs [2, 17, 21]. The Chambers and Clarke and Haynes et al. articles were neither pediatric nor orthopaedic-specific so it is difficult to delineate what the reoperation or readmission rates would be for these specific populations. However, Cox and Clarke reported that 42 of 398 children (12.1% of fractures) were readmitted on a further 47 occasions in connection with their original injury [10]. Beal et al. used the Institute of Medicine national healthcare quality report framework that includes a separate patient-perspective domain [2]. In addition, Beal et al. report that almost one-third (32.1%) of measures within the healthcare quality domain were related to patient-centeredness. The vast network of Shriners Hospitals for Children treats orthopaedic problems and burns and began a systemwide outcome management effort to validate their quality of care using a patient and family perception of care inpatient survey [21]. No pediatric subjects were included in the Idvall report [17].

Discussion

In a recent Policy Statement, the American Academy of Pediatrics (AAP) summarized efforts on quality measurement and reinforced that current efforts often failed to recognize the unique needs of children and their families [39]. The AAP advocates the use of pediatric measurement data, including public reporting on validated pediatric measures that are appropriately constructed for quality improvement [39]. The Canadian Pediatric Society (CPS) has also made quality pediatric care a priority [8], albeit to a lesser extent than the AAP. In a 2009 Position Statement that broadly addressed the role of pediatricians in the health needs of youth, the CPS stated that children and youth who are hospitalized should have access to quality, specialized pediatric expertise that meets their needs [8]. Following these position statements, we conducted a systematic review to identify and evaluate existing published QIs in pediatric orthopaedic care.

Our review is limited by a number of factors. First, we did not search using Google Scholar. One major advantage of Medline is that it is readily updated not only with printed literature, but also with literature that has been presented online in an early version before print publication by various journals [12]. Second, our search did not include a review of all published and unpublished literature and did not include databases that are not peer-reviewed [6, 12]. A scoping review is a consideration for the future. Third, the strength of our recommendations is based on the strength of the results. The articles identified in this search strategy were largely retrospective reviews of administrative databases, other systematic reviews, or surveys. Health record databases are largely administrative and retrospective. This can be problematic because the data depend on medical record coders, whose major focus is billing [34]. One large national database in Canada is the Canadian Hospitals Injury Reporting and Prevention Program (CHIRPP), a computerized information system that collects and analyzes data on injuries at participating hospitals since 1990 [7]. Although more than 80% of the more than 1.5 million records in the CHIRPP data set capture pre-event information on pediatric injuries, the data that are collected are mainly used to establish injury prevention guidelines and do not contain any measures of quality of care or patient-specific outcomes [7].

Mortality, postoperative complications, and readmission rates and reoperation rates were the most commonly reported QIs in pediatric orthopaedics despite the fact that they have been adapted from the adult literature and are less than ideal in the pediatric population. For example, despite the use of prophylactic antibiotics a minimum of 60 minutes before incision in adult surgery, this may not be appropriate as a quality measure in pediatric surgery because prophylactic antibiotics are often not given as a result of the lack of supporting evidence [16]. In the papers that cited mortality, most authors concluded mortality was not a useful indicator. Although experts in this field rank mortality as a potentially very useful indicator and one that is also very likely to prompt action for improving quality of care, this is one QI that is not particularly sensitive to pediatric use because of its relative rarity in children’s hospitals and because death is not often linked to a preventable error or complication. In a 1-year review of all pediatric orthopaedic patients, Cox and Clarke reported a zero mortality rate [10]. The article by Sedman et al. on 1.92 million discharges across 4 years of data from children’s hospitals in 31 states reported that the majority of mortality cases were very complex cases for which death was a nonpreventable rather than a preventable complication [33].

Many unique qualities of pediatric surgical care provide challenges to the development of a quality improvement program. There is marked heterogeneity in outcomes of surgical procedures in children, some resulting from the physiological changes in function, which are a part of normal development and growth [2]. As well, a positive relationship between volume and outcome has been reported for various orthopaedic procedures [31, 36]. However, most procedures in pediatrics have much lower volume than adult procedures and mortality rates are so low that statistical analysis and modeling is challenging if not impossible [9]. In addition, limitations exist in applying this type of information without risk adjustment [36]. Risk adjustment involves accounting for patients’ characteristics that influence quality measures but are not under the control of the provider [19]. Risk adjustment is also difficult in pediatrics.

Cost-effectiveness was notably absent as an identified measure of quality in our review. Cost-effectiveness analysis (CEA) provides an important tool by which policymakers may assess and potentially increase the return on healthcare investments [5]. Although CEA is most useful for the evaluation of new technologies such as in knee and hip arthroplasties, there may be a place for CEA in pediatrics surgery [5, 28].

Our review of the literature looking for specific QIs that may be applicable to pediatric orthopaedics highlights that although there appears to be a trend toward performing patient-centered outcomes, there is a long way to go to be able to reliably evaluate the quality of pediatric surgical programs. Patient-centered outcomes capture the impact of a patient’s overall well-being much better than traditional, surgeon-defined outcomes such as mortality or infection rates [28]. Generic health utilities have limited value in pediatric quality of care evaluation because of they are ill-equipped to account for the developmental changes resulting from normal growth and many are only appropriate for older children.

Although there are benefits and drawbacks to all reported QIs, patient-centered and surgeon-defined outcomes along with cost-effectiveness have important roles in evaluating the quality of pediatric orthopaedic care. When we look specifically at the subspecialty of pediatric orthopaedics, there is a dearth of reliable or applicable measures to address quality of care on the whole. The next step in quality measurement will require targeting specific areas for further literature review to identify any available clinical practice guidelines or evidence to be able to begin to develop some disease-specific measures that can then be assessed for feasibility. Pediatric orthopaedic care cannot be improved until we can better measure quality and identify areas to target for improvement. If there is to be a commitment to providing the highest quality and safest health care for infants, children, adolescents, and young adults, quality measurement must be made a priority.