Perils and Opportunities of Comparative Performance Measurement

Guillén and colleagues conducted a systematic review of studies performed in high-income countries on the effect of loss to follow-up at on neurodevelopmental outcomes for extremely preterm infants (1). This study showed substantial variation in neurodevelopmental impairment (NDI) rates between cohorts, with the worst rates reported mostly from studies in the United States (US). The authors demonstrated that this can be partly explained by higher loss to follow-up of healthy patients among US cohorts.

These findings raise several important questions:

1. What are the reasons for the large variation in NDI?

2. Why are follow-up rates in the US lower than those in other high-income countries?

3. What are the implications for comparative performance measurement?

What are the reasons for the large variation in NDI?

For the purpose of policy–setting or quality improvement, comparative measurement of outcomes often serves to spotlight actionable differences in quality of healthcare delivery. In their systematic review, Guillén et al. found NDI rates varying from 12.4 to 57.5%. Since most US cohorts fall within the upper end of this range, a superficial assessment might imply that the US healthcare system does a poor job at caring for these infants. While US cohorts experienced significantly greater loss to follow-up of healthier infants, this component accounted for only 38% of the variation in NDI rates. Therefore, additional factors contribute to higher NDI rates among US cohorts. Higher NDI rates may result from differences in structures of care delivery at the hospital or healthcare system level (e.g., care integration, regionalization, or patient-to-nurse ratios); care processes (e.g., use of potentially harmful soy-based intravenous lipid emulsions in the US; non-adherence to care guidelines); or culture of care delivery (e.g., safety culture) (2-7).

Beyond these factors, NDI rates reflect the complex interactions between the patient's (and his parents’) biology, as well as the socioeconomic context in which healthcare is delivered (8). Since biology and socioeconomic context are not usually under the control of healthcare providers, they need to be adequately accounted for in statistical analyses before variation between countries can be attributed to differences in quality of care.

The four major sources of variation in outcomes are risk, chance, true differences in care, and bias (see Table 1) (9). Differences in quality of care originate from differences in personnel, practices, resources, technology, organizational structure, and culture. In order to attribute differences in outcomes to differences in quality of care, comparative analysis of performance must account for risk and chance.

Table 1.

Strategies to enhance usefulness of comparative performance assessments

Factors	Strategies for Mitigation
Risk	Statistical adjustment for -Illness severity -Socioeconomic factors
Chance	Control limits around parameter estimate to separate random from non-random variation Increase sample size
Quality of care	Focus on processes of care Composite indicators
Bias	Definition of homogeneous sample Standardized measurement of process or outcome Repeated sampling

In performance measurement, chance is addressed by allowing absolute or relative results to vary within a predefined range, such as a 95 or 99.9% control limit. Chance is influenced by sample size, with smaller samples having a higher probability for an outlier result. Indeed, some of the single center and smaller studies have among the lowest (10-13) and highest (14) rates of NDI. Funnel plots could highlight whether individual studies truly have outlier NDI rates or whether these fall within random variation among the group.(15)

Risk is a characteristic that elevates the probability of an adverse outcome but is beyond the control of the agent responsible for the outcome (9). Comparative performance assessments should account for such risks. In neonatal intensive care, low gestational age or birth weight, male gender, and plurality have been shown to be associated with higher mortality rates. Additional factors commonly predicting outcome include birth at a perinatal center (16-18); cesarean section, which appears to confer a survival benefit to very low birth weight (VLBW) infants who are also small for gestational age (19;20); and the existence of congenital anomalies (21;22). Guillén and colleagues’ study is an important addition to this literature and highlights the importance of accounting for attrition rates in comparative outcome assessments.

The relative contributions of socioeconomic and biological influences (e.g., race, ethnicity, poverty, education, social support, and health system functionality) to NDI can be difficult to measure and are often excluded from international comparisons. However, these factors may be highly influential and help to identify high-risk groups to target through health policy. We think that in future studies every effort should be made to collect this data in a standardized fashion.

The methods used to define outcomes may also account for variation in NDI rates, even beyond those demonstrated by Guillén et al. Despite the authors’ attempts at defining a homogeneous sample for comparison and statistical adjustments for clinical risk, residual heterogeneity across studies may bias results. For example, NDI rates at high-risk perinatal centers tend to be higher than population-based rates. Cohorts with more inclusive definitions of NDI (e.g, those including seizures) may have increased rates. Selection bias may exist with regard to continuing care in the face of predictable adverse neurologic outcome. Thus, it is unclear whether NDI rates reported in US studies are spuriously higher than in other high-income countries, or whether they result from systematic differences in quality of care or socioeconomic context.

Why are follow-up rates in the US lower than those in other high-income countries?

Multiple factors, including patient and family characteristics, healthcare system structure, and social environment, affect whether families keep their follow-up appointments more than one year post hospital discharge. Families must weigh whether the value of the visit exceeds the burdens associated with it. American families face several barriers, which may negatively influence this value equation. For example, social support systems, such as paid job-protected parental leave, are much less generous in the US than in most European countries (zero weeks compared to sixty eight weeks in Sweden) (23). US employment conditions may also be less forgiving, such that keeping appointments may result in wage losses, and endanger employment or chances for promotion.

Parents in the US may also be more worried about expenses associated with follow-up visits, such as, transportation, or co-payments. US health insurance coverage tends to be less comprehensive for speech and occupational therapy. Even when covered, families face higher out-of-pocket costs. These barriers limit healthcare utilization and may result in greater loss to follow-up.

In addition, labor mobility is higher in the US, compared to many European countries, and families may relocate to different geographic areas (24). Depending on how follow-up is ascertained in a given cohort, patients may be lost to follow-up even if they access care at a new location. Again, financial considerations may influence families’ decision to move as they seek social support, higher wages, or more comprehensive health insurance. The impact of health expenditures on household financial health was illustrated in a study by Himmelstein and colleagues, who showed that these accounted for nearly half of personal bankruptcies.(25)

Guillén and colleagues show that infants lost to follow-up have less NDI. Is it possible, that lower US follow-up rates merely reflect more efficient healthcare resource use? Do US families underuse resources, or do other countries, because of more comprehensive social support and insurance coverage, overuse resources? While we cannot definitively answer these questions, the literature suggests a benefit of early education programs for high-risk infants of the kind initiated and tracked through follow-up programs (26). In this regard, loss to follow-up remains concerning.

What are the implications for comparative performance measurement?

Comparative performance measurement has become ubiquitous. We use it to decide which movie to watch, which restaurant to visit, and which schools to choose for our children. In recent years, comparative performance measurement has become increasingly common in healthcare (27-30). Frequently, comparisons attempt to promote improvements in the quality of healthcare delivery through the use of monetary or reputational incentives (31-33). While users of this information (e.g. consumers, health plans, and policy-makers) may appreciate information on provider, hospital, and healthcare system performance, they tend to be less sensitive to the methodological weaknesses of approaches to measurement. Further, their desire for information may outpace measurement science. However, for those being evaluated, the stakes are high, and they demand that measurement be valid and reliable (34). The tension between the desire for information by users and the demand for accuracy by providers is not easily reconciled.

NDI rates are attractive for judging performance of neonatal intensive care. They represent the bottom-line of what the neonatal community, families, and policy-makers care about. However, for the reasons mentioned above, international comparisons of NDI rates should be interpreted with caution. Much of the differences in NDI rates between the US and other high-income countries may be explained by systematic bias and confounding.

Yet, it is possible that there are real differences in the care that US infants receive, which promote higher NDI rates. Given the current imperfect state of knowledge regarding potential socioeconomic confounding, we recommend that comparisons of NDI rates between countries be viewed as a starting point for further inquiry and testable hypotheses. These hypotheses should be studied using a data collection framework that includes clinical and contextual measures.

We recognize that such a research agenda will take many years to bear fruit. In the meantime, the following approaches may improve the value and policy relevance of international comparisons (Table 1). First, as done in this systematic review, international comparisons of NDI rates should always use statistical methods that account for risk and chance (35;36). Attrition rates could be handled in a manner similar to a missing data analysis, with the potential to impute values based on the characteristics of those lost to follow-up (8). Analogous to methods used in decision science, the effect of potential biases on NDI rates could be investigated using sensitivity analysis. For example, in simulation models follow-up rates in laggard cohorts could be set to 100%, or be allowed to vary within a defined range.

Another way to improve comparative performance measurement is to focus less on one outcome measure and instead, develop a dashboard of measures relevant to families, clinicians, and policy-makers. In related work, we are working to develop a composite indicator of neonatal intensive care quality for VLBW infants, the Baby-MONITOR, based on data items collected by members of the California Perinatal Quality Care Collaborative and the Vermont Oxford Network (37). A mix of process and outcome measures, such as those used in the Baby-MONITOR, might provide deeper insights into reasons for performance variation (38). Given the influence that multiple disciplines have on neonatal outcomes, such a scorecard should be expanded longitudinally to include perinatal and post-neonatal care. Measurement should reflect the multidisciplinary nature of neonatal care by assessing integration of care and coordination among disciplines. A multi-dimensional evaluation of the perinatal medical home may provide users with more actionable information on how to best serve our patients.

In summary, international comparisons can result in oversimplification of policy messages. Nevertheless, such comparisons, if done well, offer the opportunity to learn from existing differences and generate hypotheses for improvement that would be otherwise remain hidden.

Abbreviations

NDI

neurodevelopmental impairment

NICU

neonatal intensive care unit

VLBW

very low birth weight