Quality measurement

Summary

The Centers for Medicare and Medicaid Services (CMS) is shifting from volume-based to value-based reimbursement of health care services. Measuring the value of health care requires measurement of quality and cost. We provide an overview of quality measurement and review a well-known and widely used conceptual model for assessing quality: structure, process, and outcome. We highlight the advantages and disadvantages of using these types of metrics. We then use this conceptual model to describe prominent CMS programs such as the Physician Quality Reporting System, Physician Compare Web site, and the Medicare Shared Savings Plan. We highlight 2 recent trends: the increasing use of outcome measures to supplement process measures and the public reporting of quality.

Proposed over 40 years ago, Dr. Avedis Donabedian's¹ conceptual model of health care remains useful today. A strength of this model is that it provides a common language for describing how to measure quality. Dr. Donabedian divided health care into 3 dimensions: outcomes, processes, and structure. Outcomes of care are what happen to patients. Processes of care are the actions that clinicians perform on behalf of patients to produce better outcomes. Structures of care are characteristics that change how likely a process of care will occur. We review each dimension in turn, underscoring the advantages and disadvantages of each, to create a foundation for analyzing the goals of current and future quality measurement programs.

Structure

Examples of structure are equipment (such as a magnetic resonance image scanner), people (such as possessing board certification), or policies. Structure can be relatively easy to measure. US News and Report uses a structural score to rank medical centers.² In its most recent rankings, the structural score for the combined specialties of neurology and neurosurgery consisted of 10 metrics including designations of centers of excellence (e.g., for epilepsy or Alzheimer disease), availability of intensivists, nurse-to-patient ratios, availability of advanced technology, and patient volume.

A drawback of structure is that strong links between structure and outcome have been rarely studied and identified, either within or outside the field of neurology. The most rigorous study design—the randomized controlled trial—compares an intervention arm vs usual care. It can be difficult to conduct a randomized study involving structure. One of the few examples is that randomized controlled trials have consistently shown that stroke units are associated with better outcomes.³ Observational studies have shown positive links between neurologists (a type of structure) and outcomes. However, because patients are not assigned at random, there remains the possibility of selection bias. For example, patients seen by neurologists may be different enough from patients seen by non-neurologists to prevent a fair comparison between the 2 groups.

Process

Processes of care are the actions physicians perform on behalf of their patients such as taking a history, performing a physical examination, ordering tests, prescribing medications, and counseling patients. Processes of care are currently the dominant metric for measuring quality of care. It is the dimension of care that physician and other providers can most directly control. It is also the most intuitive measure that can be developed from the medical literature. Because most randomized controlled trials are evaluating a process of care (delivery of a medication), it is fairly straightforward to develop a process measure to describe that care.

Using process measures is not without its drawbacks. The process–outcome link is assumed to be present, but the magnitude of that link requires investigation. A large study of elderly adults showed a clear link between performing processes of care and lowered mortality.⁴ Yet in other studies, the process–outcome link may not be as large as expected. A recently reported pay-for-performance program in a region of England⁵ and a large study of Medicare hospitals⁶ showed that performance of quality measures was associated with only a modest 1% absolute difference in 1-year mortality rates. Even worse, a study of 400,000 surgical patients found no association between 6 process measures of infection prevention and the development of an infection.⁷

Another potential limitation of process measure is that physicians can self-report whether a patient should be excluded. Some exclusions, such as comfort care, can only be identified by a physician and not through administrative data. However, there are concerns that exclusions may be too frequently applied. A surgical study recently showed that quality of care was lower among patients who had exclusions applied to them, though it was not clear whether all such patients should have been excluded.⁸

Outcomes

The overall objective of health care is to improve the health of patients, and the health of patients is assessed through outcome measures. A historical classification of outcomes consists of the 5 Ds: death, disease, disability (functional status), discomfort (quality of life), and dissatisfaction. Some outcomes, such as mortality, are relatively easily ascertained and widely available, though not necessarily the most relevant for chronic neurologic conditions. Other outcomes, such as quality of life for patients with a particular condition, are captured through scales that require extensive psychometric testing to demonstrate their validity and reliability before they can be widely used. There are other methodologic issues to consider when using outcome measurements, even when the outcome is as easily ascertained and as widely accepted as mortality. We focus on 2 of them: risk adjustment and gaming.

Outcome analysis requires adjustment for the level of risk because health care providers manage different groups of patients, each with a different risk of developing the outcome of interest. For example, tertiary care hospitals manage some of the most severely ill patients, who have the highest risk of death. A fair comparison of inpatient mortality rates would require an adjustment for disease severity. However, a variable for disease severity is typically unavailable; instead, a weighted count of the number of comorbidities is often used to estimate the sickness of a patient.⁹ Even when a variable is available, it is not always clear how it should be used. For example, mortality rates are higher among older patients, so inpatient mortality rates should adjust for the age of patients. But there are numerous ways to analyze age: as a linear variable, as categories, and as interactions with other variables. The lack of consensus on how to perform risk adjustment is best demonstrated in a recent study of inpatient mortality in Massachusetts.¹⁰ Four vendors applied their proprietary risk adjustment methods to the same state database of inpatient clinical data, but showed a surprisingly high level of disagreement on which hospitals were considered the best performing ones. Some of the most well-known studies of quality of care stated that they deliberately focused on process measures instead of outcome measures simply because the former was less reliant on risk adjustment.¹¹

Gaming refers to improving scores of a quality measure by a method other than improving the quality of care. Outcome scores can be improved by changing patient populations or overriding patient preferences. For example, inpatient stroke mortality rates could be lowered through aggressive use of life-sustaining interventions such as intubations and enteric feeding, even if patients did not prefer such therapy.¹²

The current era of process measures and the American Academy of Neurology response

The most widely known system is the Physician Quality Reporting System (PQRS) incentive program of CMS.¹³ In the basic option of the PQRS program, a provider picks 3 measures from a current menu of around 300. Many of the initial quality measures were geared toward primary care providers. At that time, the American Academy of Neurology (AAN) leadership recognized that if the AAN did not participate in developing PQRS quality measures, either relatively few relevant measures would be developed for neurologists or such measures would be developed by non-neurologists. To address this concern, the AAN created the Quality Measurement and Reporting (QMR) Subcommittee, whose initial charge was to develop neurology-relevant measures for adoption in quality measurement programs, particularly PQRS. We describe our methodology in previous publications.¹⁴ QMR leveraged considerable expertise from the AAN guideline subcommittees to perform systematic reviews of the literature and to identify efficacious processes of neurologic care.

Yet it is also clear that QMR and the AAN need to better inform AAN members about PQRS and other similar quality measurement programs. In the most recent data available, in 2011 only 21% of all neurologists participated in PQRS (Gina Gjorvad, AAN, personal communication, 2013). Failure to participate in PQRS in a given year will result in a penalty 2 years later, so it is imperative that neurologists familiarize themselves with the PQRS program as soon as possible. For example, for physicians who do not participate in PQRS in 2013, CMS will hold back 1.5% of Medicare Part B billing in 2015. We have covered some of the details of how to participate in the PQRS program in a previous publication.¹⁵

CMS is changing what it measures and how it reports quality of care

Recently enacted CMS programs show that the current era of using privately reported process measures will be replaced by an era with greater public reporting and greater use of outcome measures.

Public reporting may promote better quality of care through 2 mechanisms: by encouraging consumers to choose higher quality providers and spur internal quality improvement activities among providers.¹⁶ An example of public reporting is the Hospital Compare Web site,¹⁷ a CMS initiative to disseminate data about quality of care delivered by hospitals. In addition, there is a Physician Compare Web site, which focuses on individual providers instead of hospitals (see http://www.medicare.gov/physiciancompare/search.html).¹⁸ This Web site highlights whether physicians participate in e-prescribing, the PQRS program, or the electronic health record incentive program (also known as Meaningful Use). Beginning in 2014, CMS started displaying performance of physicians who participate in certain options of the PQRS program on the Physician Compare Web site.

An example of a CMS program that uses new types of metrics is the Medicare Shared Savings Plan (MSSP), which was implemented in 2012. It establishes 33 metrics for evaluating Accountable Care Organizations (table).¹⁹ About half the metrics are processes and half are outcomes. Some of the outcome metrics are physiologic measures and readmissions. There are also metrics on patient-reported experience, satisfaction, and other ratings of health care. In the past, patient ratings were collected among patients after hospital discharge, but not so commonly in the outpatient setting. It is thought that patient ratings address a dimension of care not well-captured by traditional quality measurement. Neurologists may have trepidation about the use of patient satisfaction metrics because they care for some of the most severely disabled patients, who may give lower ratings of all health care providers who treat them.

Table.

Thirty-three metrics for evaluating accountable care organizations¹⁹

The future effect of CMS programs and the AAN response

The AAN leadership has supported active engagement in the quality measurement movement in health care. Some actions are straightforward. For example, the AAN, like other physician advocacy organizations, has consistently recommended that CMS harmonize quality measures across the various programs to lessen provider burden. In addition, the AAN provides feedback when government agencies propose the implementation of any new neurology-specific measure.

However, the direction the AAN should take in other areas is far less clear. Measuring neurology-specific patient outcomes requires tracking patients over time, as is done with registries. The AAN is considering whether to develop its own registry or work with existing ones.

Next, while the AAN is well-situated to obtain the perspectives of neurologists, it is less experienced and positioned to obtain perspectives from patients themselves. As is stands, patient-reported outcomes are used without risk adjustment. If neurologists believe that such ratings may be biased against our specialty, we need to perform research on whether persons with neurologic conditions rate providers differently from persons without neurologic conditions.

The AAN publicly reports neurologists' board certification status on its membership Web site, but it does not report on other markers of physician quality. If Physician Compare becomes more heavily used, the AAN may consider whether to promote its own Web site as a competing source of referrals. However, it is not clear what data should be reported.

At the AAN, the QMR Subcommittee within the Practice Committee and the Payment Policy Subcommittee within the Medical Economics and Management Committee monitor how CMS measures and pays for health care. To understand a complex program such as MSSP, such subcommittees will need to work closely together to leverage expertise in both quality measurement and reimbursement strategies. Many members of AAN leadership practice in academic medical centers, but CMS quality measurement programs have a greater effect on physicians practicing outside of academic medical centers because they rely less on revenue from research and teaching. We encourage them to become involved with organizations such as the AAN so that their voice is heard in this rapidly evolving landscape of quality measurement.

STUDY FUNDING

No targeted funding reported.

DISCLOSURES

E. Cheng has received funding for travel from the American Academy of Neurology and receives research support from the NIH/National Institute of Neurological Disorders and Stroke, the Department of Veterans Affairs, the National Multiple Sclerosis Society, and the American Heart Association. A. Sanders serves on the editorial board of the Journal of Neurology and Psychology; has reviewed for the NIH/NIA, the Center for Medicare and Medicaid Innovation (CMMI), the Patient-Centered Outcomes Research Institute (PCORI), and the Alzheimer's Association; has received honoraria for serving on peer-review panels from the CMMI and PCORI; is a member of a federal advisory committee (MEDCAC); and is a member of the MAC committee of the AMA Physician Consortium for Practice Improvement. A.B. Cohen receives publishing royalties for The Strokes pocketcards, Brain Hemorrhage and Trauma pocketcards (Borm Bruckmeier Publishing; 2011, 2012), and Resuscitation (mobile app) (EM Gladiators LLC, 2012); and receives research support from the Center for Integration of Medicine and Innovative Technology. C.T. Bever Jr. has received funding for travel from the Paralyzed Veterans of America and the Chesapeake Health Education Foundation; is author on a pending patent on a method for producing hematogenous stem cells for cell replacement and gene therapy (royalties received from Abraxis); receives publishing royalties from Principles of Ambulatory Medicine (Lippincott, Williams & Wilkins, 2007); and receives research support from the Department of Veterans Affairs and the National Multiple Sclerosis Society. Full disclosure form information provided by the authors is available with the full text of this article at Neurology.org/cp.