Metrics That Matter

Abstract

Increasingly, performance metrics are seen as key components for accurately measuring and improving health care value. Disappointment in the ability of chosen metrics to meet these goals is exemplified in a recent Institute of Medicine report that argues for a consensus-building process to determine a simplified set of reliable metrics. Overall health care goals should be defined and then metrics to measure these goals should be considered. If appropriate data for the identified goals are not available, they should be developed. We use examples from our work in the Veterans Health Administration (VHA) on validating waiting time and mental health metrics to highlight other key issues for metric selection and implementation. First, we focus on the need for specification and predictive validation of metrics. Second, we discuss strategies to maintain the fidelity of the data used in performance metrics over time. These strategies include using appropriate incentives and data sources, using composite metrics, and ongoing monitoring. Finally, we discuss the VA’s leadership in developing performance metrics through a planned upgrade in its electronic medical record system to collect more comprehensive VHA and non-VHA data, increasing the ability to comprehensively measure outcomes.

Enhancing value—i.e., improving outcomes as efficiently as possible—has gained salience across health systems. To maximize value, it must be measured.¹ The complexity of accurately measuring value has led to a proliferation of quality and performance metrics, often leading to confusion, duplication, and provider reporting burden.^2,3 To address these issues, a recent report from The Institute of Medicine (IOM) recommends stakeholders participate in a consensus-building process that identifies key health care goals focusing on outcomes that enhance value.^2,3 Once the key goals have been identified, metrics that measure these goals can be developed. Often, the IOM could not find existing measures that comprehensively captured the desired goals, resulting in a call for the development of comprehensive data and metrics in areas with identified gaps.^2,3

We expand upon the IOM’s findings on the need to simplify metrics through a consensus- building process that identifies core comprehensive metrics. Our experience working with performance metrics in the Veterans Health Administration (VHA) highlights other equally important challenges in the metric selection process. Key lessons discussed below include: specification and predictive validation, attention to implementation challenges to ensure data fidelity, and the ongoing evolution of data to comprehensively measure outcomes.

VALIDATION OF METRICS

The IOM recommended that health systems measure 15 core metrics, including access to care. This is a domain in which the VHA has extensive experience. For over 15 years, Congress has requested that the VHA respond to concerns that visit waiting times are too long.⁴ These concerns peaked in 2014 with accusations that delays in access to VHA care resulted in preventable deaths.⁵ Throughout this long history, the VHA has employed a variety of strategies to measure and report on waiting times, learning some important lessons about the feasibility of validating and implementing performance metrics along the way.

The IOM committee argued that the best measure of access to care is unmet care need, assessed by surveys such as the Medical Expenditure Panel Survey or the VHA’s Survey of Healthcare Experiences of Patients (SHEP).^6,7 Collection of survey data is costly and difficult to obtain in real time because of the effort required to contact patients and obtain their responses. Survey sampling frames also raise concerns of selection bias.^8–10 For example, only veterans who have used VHA services are surveyed in SHEP, limiting information about access for non-VHA users.⁷ The ability to find and measure the entire population of patients that suffer from unmet health care needs in a timely matter is an important example of gaps in the metric landscape that the IOM identified.

Recognizing the limitations of survey-based access measures, our work has validated access metrics that can be constructed with routinely collected administrative data. This validation work has focused on both specification and predictive validity. Even conceptually compelling metrics can be useless in practice if they fail to measure the desired concept because the data underlying the metrics are misunderstood or inaccurate (i.e., specification validity).¹¹ Demonstrating specification validity, our work shows that several types of waiting time metrics (e.g., number of days until next available appointment, number of days between appointment request and completed appointment) predict veterans’ self-reported access. The relationship is seen across multiple outcomes self-reported in SHEP including patients’ ability to obtain timely appointments, access to treatments and specialists, and overall satisfaction with VHA care.⁸

Predictive validation relates a potential performance metric to a conceptually related outcome in historical data.¹² If the metric is associated with the outcome as expected, users can have greater confidence that the metric truly reflects the intended concept. For example, if hospitals rated most highly for infection control processes also had high rates of post-surgical infections after controlling for patient characteristics, we might question the validity of the rating metric. In our work the same access metrics that demonstrated specification validity also demonstrated predictive validity by predicting clinical outcomes such as glycated hemoglobin levels, preventable hospitalization and mortality. These clinical outcomes are not likely to be related to metrics that fail to measure access and unmet health care needs.^8,13–18

Two key lessons emerged from this validation work. First, different access metrics predicted satisfaction for different patient populations. The best metric for new patients (based on the date the patient first contacted the clinic) is not the same as the best metric for returning patients (based on the date of service requested by the patient). These population differences may not be clear in the IOM recommended consensus-building process that determines the original metrics, highlighting the need for validation. Second, when predicting patient-level outcomes for validation purposes, appropriate methods are needed to control for confounders such as patient-level health status. Without taking these confounders into account, true relationships between metrics and key outcomes can be obscured (e.g., longer waits predict improved health instead of poorer health). A common strategy to address this problem is to compute facility-level average metrics that exclude the individual patient whose outcome is being assessed.^8,13–18 This is a similar approach to using an instrumental variable analysis that allows consistent estimation of relationships between explanatory variables and outcomes even when key explanatory variables are correlated with the error terms in the models.

MAINTAINING DATA FIDELITY OF METRICS

Even metrics based on initially appropriate and accurate data can fail over time if incentives to appear successful affect the fidelity of data generation or metric definition. The IOM recommends assessing the threats to metric validity that incentives can create.^2,3 We agree and argue that strong implementation design requires policymakers to anticipate potential unintended strategic responses from those whose performance is being measured and design metrics to minimize the likelihood of such responses.

The VHA access crisis in 2014 highlighted the importance of this point and how it differs from validation. Our waiting time studies validated routinely collected access metrics by confirming that they predicted a wide variety of patient outcomes. Upon adoption of the metrics, the provision of bonuses to senior management for meeting access metric targets led to widespread concerns that access metrics were being manipulated and reported VHA waits from some facilities did not reflect actual patient experience.^19–21 The controversy resulted in the resignation of several senior officials including former Secretary Shinseki, the passage of the Veterans Access Choice and Accountability Act, and the current reorganization of the VHA that focuses on veteran-centric outcomes.^5,22–25

The VHA experience that performance incentives can encourage behavior that can bias metrics is not unique. Roth et al. (2012) examined the effect of the HEDIS quality metric to avoid antibiotic prescribing for acute bronchitis on overall antibiotic prescribing. Antibiotic prescribing for acute bronchitis decreased sharply. However, overall antibiotic prescribing only decreased slightly because of a significant shift in diagnosis coding from acute to unspecified bronchitis.²⁶

One strategy to reduce this danger is to use information from independent data sources. For example, as a measure of hospital quality, post-hospitalization mortality rates can be computed from the National Death Index instead of relying on hospitals to report mortality rates.²⁷ Models would need to be appropriately adjusted for case mix and there would be a lag in the ability to obtain these data, but we believe the advantages of using an independent data source outweigh its potential disadvantages.^13–18,27

Where it is not possible to rely on independent data, it is important to minimize the role that subjective judgments can play in the data-generating process. The most controversial metric in the VHA access crisis required schedulers to determine and enter the date on which a patient wanted to be seen for follow-up care.²⁸ Though this process worked reasonably well most of the time, VHA audits found that some schedulers were instructed by supervisors not to enter the dates requested by patients.²⁰ An alternative way to measure waiting time is to calculate the duration between the date that the provider determines the patient should receive care and the date when he or she actually does. VHA physicians could enter these data directly into the electronic medical record and the data could be automatically extracted from the system. This process minimizes the role of discretion on the part of the scheduler.

The recommendation of the IOM to combine several measures into a composite metric is another important strategy to maintain data fidelity.^2,3 Composites can condense an overwhelming number of metrics while simultaneously making incentives less responsive to the manipulation of individual measures. For example, the VHA Office of Mental Health combines over 30 individual metrics into three composites to track mental health care performance across facilities and over time.²⁹ In addition, the VHA clinical operations uses a scheduling composite metric that measures fidelity to scheduling policy.³⁰ Overall, the composite metrics are less easily manipulated than individual metrics but composites can hide data fidelity issues in specific individual metrics. This concern highlights the importance of validating all the individual metrics in the composite.

Finally, although careful validation and implementation can reduce the risk of unintended consequences, it is typically not possible to anticipate all strategic responses to new metrics and incentives. Consequently, ongoing monitoring is advisable to detect potential degradation of data fidelity over time. One way to accomplish this is to track multiple measures that have historically been highly correlated while incentives are linked to only a few. For example, the VHA is currently tracking the difference between two measures of waiting times for new patients, even though only one measure is publically reported.³⁰ Historically, these two measures have been closely related, so divergence between them either nationally or at a particular facility could indicate strategic changes in data generation processes.

WHAT’S NEXT: DEVELOPING COMPREHENSIVE DATA TO FILL METRIC GAPS

The IOM report argues there is insufficient data to measure many of the core metrics. The identification of these gaps allows stakeholders to work on developing appropriate data to comprehensively measure outcomes.³ The VHA is a leader in this regard with current plans to enhance its electronic medical record system. In concordance with the IOM’s recommendation the new system is being designed with generation of performance metrics as a fundamental feature from the beginning instead of trying to compute metrics from whatever data were recorded after implementation. This approach will improve the collection of key data elements that measure health care value and increase the ease with which health record data can be shared between VHA and non-VHA systems.³¹

Access to non-VHA data is crucial to improving the comprehensiveness of outcome measures because veteran enrollees rely heavily on non-VHA care. In the 2013 Survey of Veteran Enrollees Health and Reliance Upon VA, only 30 % of respondents reported they relied solely on the VA to meet all their health care needs. Over half of the respondents reported they were also enrolled in Medicare, and almost 30 % reported they were enrolled in another individual or group health insurance plan.³² Many outcomes, particularly acute ones such as heart attack and stroke, are more likely to be recorded in non-VHA records (because the nearest facility at the time of the event was not a VHA facility).^33,34 More comprehensive outcome measures better reflect the true health status of the patient population, which should be the focus of performance metrics.² Access to non-VHA data would also strengthen the fidelity of performance measures by supplying more independently collected data. These data could be used directly in reported metrics or indirectly in alternative measures computed for fidelity monitoring purposes.

CONCLUSION

The recent emphasis on value in health care has joined with advances in health data systems to fuel a rapid expansion in development and use of quantitative performance metrics. The IOM report correctly highlights the need to simplify the landscape of performance metrics by defining key health care goals through a consensus-building process and then obtaining needed data to measure these goals. In order to increase the chances that metrics will measure the agreed-upon goals, policymakers and managers should validate metrics before using them and employ implementation strategies that minimize unintended consequences. These strategies include maximizing the use of independent data sources, maintaining multiple related measures for monitoring purposes, and employing composite measures. Continuing evolution of electronic medical record systems should improve the design of performance metrics by providing more comprehensive data for outcome measurement.

Compliance with Ethical Standards

Conflict of Interest

The authors report no conflicts of interest.