Patient-Reported Outcome-Based Performance Measures in APMs: Current Use, Implementation Barriers, and Principles to Succeed

Abstract

Objective:

Patient-reported outcome (PRO)-based performance measures (PRO-PMs) offer opportunities to aggregate survey data into a reliable and valid assessment of performance at the entity-level (e.g. clinician, hospital, accountable care organization). Our objective was to address the existing literature gap regarding the implementation barriers, current use, and principles for PRO-PMs to succeed.

Methods:

As quality measurement experts, we first highlighted key principles of PRO-PMs and how alternative payment models (APMs) may be integral in promoting more widespread use. In May 2023, we reviewed the Centers for Medicare & Medicaid Services (CMS) Measures Inventory Tool (CMIT) for active PRO-PM usage within CMS programs. We finally present principles to prioritize as part PRO-PMs succeeding within APMs.

Results:

We identified five implementation barriers to PRO-PM use: original development of instrument, response rate sufficiency, provider burden, hesitancy regarding fairness, and attribution of desired outcomes. There existed 54 instances of active PRO-PM usage across CMS programs, including 46 unique PRO-PMs within 14 CMS programs. Five principles to prioritize as part of greater PRO-PM development and incorporation within APMs include: 1) clinical salience, 2) adequate sample size, 3) meaningful range of performance among measured entities and the ability to detect performance change in a reasonable timeframe, 4) equity focus, and 5) appropriate risk adjustment.

Conclusions:

Identified barriers and principles to prioritize should be considered during PRO-PM development and implementation phases to link available and novel measures to payment programs while ensuring provider and stakeholder engagement.

Précis:

PRO-PM development and implementation within alternative payment models may benefit from considering identified barriers and principles to prioritize while ensuring provider and stakeholder engagement.

Introduction

The Centers for Medicare & Medicaid Services (CMS) has sought to advance quality measurement away from traditional process measures toward outcome measures, including those using patient-reported outcomes (PROs), to advance patient-centered health care quality improvement.¹ In 2017, CMS launched the Meaningful Measures initiative to identify priority areas for quality measurement and improvement,² and in 2021, the CMS Innovation Center published a renewed vision focused on five objectives to support CMS’s strategic vision and priorities that center on achieving equitable outcomes through high-quality, affordable, person-centered care.³ Since this, PROs have largely been implemented in test cases or in traditional quality measurement programs, but not alternative payment models (APMs). Advancing measurement within APMs is essential given the goals set forth by the Health Care Payment Learning & Action Network,⁴ aiming for 100% of Traditional Medicare and Medicare Advantage payments to be tied to quality and value through two-sided risk APMs by 2030. Within current programs or APMs, the incorporation of patient and caregiver perspectives through patient-reported outcome measures (PROMs) is often a strong measure of total health and applicable to broad populations.⁵ Aside from PROMs, a greater emphasis has more recently been placed on PRO-based performance measures (PRO-PMs) by CMS to capture the patient voice and drive quality measurement toward patient-centered care at the level of the provider.^6–8 One promising but potentially underused avenue for PRO-PMs has been incorporation within APMs. As quality measurement experts, we highlight key principles of PRO-PMs and how APMs can be integral in overcoming current implementation barriers.

PROs, PROMs, and PRO-PMs

Increasingly being used for outcome measurement, a PRO is defined as information about the status of a patient’s health condition, health behavior, or experience with healthcare that comes directly from the patient, without interpretation by a provider or anyone else.⁹ A PROM provides a way to collect information directly from respondents, typically by using an instrument, tool, or single-item measure. A PRO-PM is a way to aggregate the collected data into a reliable and valid measure of performance at the entity level (e.g. clinician, hospital, accountable care organization).⁶ For example, depressive symptoms may be considered a PRO, while the Patient Health Questionnaire – 9 (PHQ-9) would serve as the PROM to capture the respondent’s symptoms. A PRO-PM, such as National Quality Forum (NQF) measure #0711 (Depression Remission at Six Months), may then be used to assess entity performance by capturing the proportion of patients with a diagnosis of major depression or dysthymia (PHQ-9 score >9) with a follow-up score <5 at 6 months.¹⁰

Increased development and implementation of PRO-PMs offers benefits from the vantage point of several stakeholders. Patients may benefit from more widespread PRO-PM usage as patient-centered outcomes of interest are increasingly captured and made an object of providers’ focus. Clinicians can use PRO-PMs to evaluate quality improvement interventions and to measure and improve quality for an array of conditions for which traditional outcome measures (e.g. mortality) may be less relevant or available. Hospitals and health care systems can incorporate PRO-PMs to assess provider performance across and beyond traditional settings, including a patient’s home. Finally, at a greater level, health care plans can use PRO-PMs to determine or tier value-based purchasing initiatives across broader populations than current disease- or utilization-focused quality measures.

PRO-PM Implementation Barriers

Despite these attractive features and increasing availability, five major barriers exist to more widespread PRO-PM implementation. First, development of new instruments is time-consuming and expensive, and so current PRO-PMs tend to rely on existing PROMs that may have been developed decades ago. Implementation as a PRO-PM may not have been considered during original development. Second, measured entities must ensure sufficient response rates to reach the representative sample size necessary for valid and reliable scoring. Provider-level differences in volume may play a role, as some providers may more readily achieve the required sample and response rate. PRO survey respondents are also known to vary in comparison to non-respondents,¹¹ which could bias provider-level scores. Third, implementation of PRO-PMs potentially creates additional burden to providers to collect and submit data.^12,13 Unlike claims-based measures, implementing a new PROM or PRO-PM requires additional costs, labor, and/or changes to established workflows in order to collect and report data. This barrier depends on the method of data collection used – for example, a PRO-PM using data captured electronically or by integration with EHR systems will pose different challenges than patient surveys delivered by telephone or mail. Some providers may be better equipped than others to collect data through a given distribution mechanism. Fourth, hesitancy among providers may exist regarding PRO-PM implementation because measures may be perceived as ‘unfair’ or to reflect aspects of quality beyond a given provider or group’s control. This would include system- and patient case mix-level risk factors that affect the comparability of measure scores across entities and time. Fifth, implementation of more general PRO-PMs, such as those addressing Health-Related Quality of Life (HRQoL), offer both clinical and population health benefits, but often makes detecting changes in outcomes of interest more challenging. Particularly, meaningful differences that can be attributed to quality of care may be difficult to detect and these outcomes may change only gradually. Many influences (such as social risk factors) may also contribute to overall well-being or quality life, limiting the ability to isolate desired outcomes with the provision of quality care.

APMs as a Venue for PRO-PMs

Overcoming the aforementioned barriers in several ways, APMs offer a substantial opportunity for PRO-PM implementation because they incentivize providers to prioritize the outcomes that matter most to their patients. In traditional fee-for-service models, providers are primarily incentivized to perform as many services as possible, regardless of whether the services improve patient outcomes, and those models are generally more specific to individual conditions or services. In contrast, under APMs, providers’ efforts more naturally align with PRO-PMs given their focus on patient-centered quality measurement. Uniquely, APMs also offer the opportunity to implement and test quality measures via the CMS Innovation Center without significant regulatory burden or delays, with voluntary model participants potentially in a better position to pioneer the implementation of PRO-PMs.

APMs currently housed within CMS’s Quality Payment Program are largely divided into two categories: Advanced APMs and Merit-based Incentive Payment System (MIPS) APMs.¹⁴ Advanced APMs require entities to: 1) use certified electronic health record technology, 2) provide payment for covered professional services based on quality measures comparable to those used in the MIPS Quality performance category, and 3) either be a Medical Home Model expanded under CMS Innovation Center authority or require participants to bear a significant financial risk.¹⁴

The APM Performance Pathway (APP) is a MIPS reporting and scoring pathway for eligible clinicians who also participate in MIPS APMs.¹⁵ When reporting through the APP, clinicians can participate at three levels: individual, group, APM entity (e.g. accountable care organization [ACO]). With data aggregated at the level of an ACO, for example, PRO-PMs offer great promise in driving meaningful quality change given the assurance of sufficient volumes required and the integration of clinical networks providing whole person care able to overcome concerns of attribution as opposed to measurement of a single specialty office-based clinician. Aside from traditional MIPS and the APP, CMS has also recently developed the MIPS Value Pathway (MVP) framework which began voluntary reporting in the 2023 performance year and has a tentatively planned transition to mandatory reporting by 2028 with the sunsetting of the traditional MIPS Program. Each developed MVP includes a subset of measures and activities related to a specialty or medical condition (potentially amenable to more specific PRO-PMs), and allows reporting clinicians to participate as an individual, a group, a subgroup, or an APM entity.

Current Landscape of PRO-PMs

To determine the current use of PRO-PMs in CMS programs, we reviewed the CMS Measures Inventory Tool (CMIT)¹⁶ as of May 2023. We identified 54 instances of active PRO-PM usage across CMS programs. This included 46 unique PRO-PMs within 14 CMS programs, with 15 endorsed by the National Quality Forum (NQF),¹⁷ the Consensus-Based Entity (CBE) that oversaw endorsement, four with the CBE endorsement removed, and the remaining 35 not endorsed by the CBE (Table 1). PRO-PMs to date have primarily been introduced in the “hospital: outpatient department,” “ambulatory: office-based care,” and “hospital: inpatient acute care facility” settings, yet the majority of other care settings have few, if any, PRO-PMs that are endorsed by the CBE (Table 2).

Table 1.

PRO-PM use within CMS programs

CMS Program	Number of PRO-PMs in Use	Example PRO-PMs
Ambulatory Surgical Center Quality Reporting	1	Cataracts: improvement in patient’s visual function within 90 days following cataract surgery
Home Health Quality Reporting	1	CAHPS Home Health
Home Health Value Based Purchasing	1	CAHPS Home Health
Hospice Quality Reporting	1	CAHPS Hospice
Hospital Inpatient Quality Reporting	2	Hospital CAHPS Patient-reported outcomes following elective primary total hip and/or total knee arthroplasty; hospital-level performance measure
Hospital Outpatient Quality Reporting	2	Cataracts: improvement in patient’s visual function within 90 days following cataract surgery CAHPS Outpatient and Ambulatory Surgery
Marketplace Quality Rating System	9	Access to care Access to information Care coordination Flu vaccinations for adults ages 18 to 64 Plan administration Rating of all health care Rating of health plan Rating of specialist Rating of personal doctor
Medicaid: Adult Core Set	3	National Core Indicators Survey Flu vaccinations for adults ages 18 to 64 CAHPS Health Plan Survey 5.0 H, Adult Version (Medicaid)
Medicaid: Child Core Set	1	CAHPS Health Plan Survey 5.1 H, Child Version including Medicaid and Children with Chronic Conditions Supplemental Items
Medicare Part C Star Rating	7	Annual flu vaccine Care coordination Customer service Getting needed care Getting appointments and care quickly Rating of health care quality Rating of health plan
Medicare Part D Star Rating	2	Getting needed prescription drugs Rating of drug plan
Medicare Shared Savings Program	1	CAHPS for MIPS clinician/group
Merit-based Incentive Payment System	22	Cataracts: improvement in patient’s visual function within 90 days following cataract surgery Cataracts: patient satisfaction within 90 days following cataract surgery Functional status change for patients with knee impairments Functional status change for patients with hip impairments Functional status change for patients with lower leg, foot or ankle impairments Functional status change for patients with low back impairments Functional status change for patients with shoulder impairments Functional status change for patients with elbow, wrist or hand impairments CAHPS for MIPS clinician/group Varicose vein treatment with saphenous ablation: outcome survey Back pain after lumbar discectomy/laminectomy Back pain after lumbar fusion Leg pain after lumbar discectomy/laminectomy Functional status after lumbar fusion Functional status after lumbar discectomy/laminectomy Functional status change for patients with neck impairments Leg pain after lumbar fusion Functional status after primary total knee replacement Urinary symptoms score change 6–12 months after diagnosis of benign prostatic hyperplasia Psoriasis – improvement in patient-reported itch severity Dermatitis – improvement in patient-reported itch severity Person-Centered Primary Care Measure PRO-PM
Prospective Payment System-Exempt Cancer Hospital Quality Reporting	1	Hospital CAHPS

Note: Normal font – not Consensus-Based Entity (CBE) endorsed; Italicized – CBE endorsement removed; Bold – CBE endorsed

Abbreviations: CAHPS – Consumer Assessment of Healthcare Providers and Systems; CMS – Centers for Medicare & Medicaid Services; MIPS – Merit-based Incentive Payment System; PRO-PM – Patient-Reported Outcome-based Performance Measure

Table 2.

PRO-PM use across healthcare settings within CMS programs

Setting	Number of PRO-PMs in Use	Number of PRO-PMs currently endorsed by CBE
Ambulatory: office-based care	18	1
Ambulatory: surgery center	3	0
Home health	4	2
Hospice	1	1
Hospital: inpatient acute care facility	8	2
Hospital: outpatient department	21	10
Inpatient rehabilitation facility	2	0
Other	4	2
Pharmacy	2	0
Skilled nursing facility/nursing home	9	1

Abbreviations: CBE – Consensus-Based Entity; CMS – Centers for Medicare & Medicaid Services; PRO-PM – Patient-Reported Outcome-based Performance Measure

Note – A single PRO-PM may be used by CMS across multiple programs and within multiple settings.

To date, PRO-PMs actively implemented within CMS programs are predominantly directed towards public reporting of quality performance on Care Compare sites to assist Medicare beneficiaries in choosing between healthcare options or healthcare and drug plans. With 22 PRO-PMs actively in use, the MIPS program provides evidence of nascent interest in PRO-PM implementation in APMs. Two common PRO-PM categories that are promising for continued APM integration are procedure-specific PRO-PMs and patient experience PRO-PMs.

Procedure-specific PRO-PMs

The use of procedure-specific PRO-PMs offers benefit in that clinical interactions and anticipated trajectories around the time of procedure are often limited to singular or few providers. Fewer external factors are likely to influence results and the timing for measurement can be tied to a specific event. All of this allows for more reliable attribution of patient outcomes to a single provider after a procedure and the creation of a relatively common denominator in comparison to conditions with more heterogeneity such as diabetes mellitus or chest pain. Counter to procedure-specific PRO-PM assessment, diabetes mellitus management offers a greater potential for variability in practice, thereby limiting reliable attribution of performance scores to the measure/concept being assessed. For example, disease heterogeneity in the case of diabetes mellitus may result in adverse selection incentives within an APM (i.e. choosing healthier diabetic patients) or could make PRO-PM assessment difficult given that most PROMs are not developed or validated for the entire spectrum of a disease. Additionally, diabetes mellitus management may be spread between primary care physicians and endocrinologists with unclear boundaries, creating difficulty attributing patient outcome measurement to either provider. Finally, there is often not a discrete point of provider intervention or anticipated clinical trajectory of the patient with diabetes mellitus as compared to many procedures.

Patient experience PRO-PMs

Patient experience measures make up a large proportion of existing general non-condition-specific PRO-PMs. These measures, including the Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys, have limitations due to their time-intensive requirements as well as the substantial lag in feeding data back to providers, impacting their ability to respond and change practice. However, CAHPS surveys are generally widely accepted, capture critically important domains, and are important to monitor as additional PROs are released to ensure no increased patient burden and that overall payment models do not have a negative impact on patient’s experience of care. Patient experience PRO-PMs may be a category to further consider for linkage to payment, particularly when monitoring quality of care.

PRO-PM Recommendations for Success in APMs

PRO-PMs offer potential within value-based payment initiatives to promote patient-centeredness and lower costs.^6,18,19 Currently, there is relatively little guidance for measure development efforts targeting PRO-PMs. We present five principles to prioritize as part of greater PRO-PM development and incorporation within APMs.

Clinical salience

It is critical that PRO-PMs be clinically salient to be prioritized for care transformation and linkage to payment. Collection of data for PRO-PMs today remains an added burden in the clinical setting, though this is intended to become easier with time. In order to make the effort meaningful to both patients and providers, the PRO-PM would capture a relevant clinical outcome for the patient, that cannot be measured easily or at all through other means, while also being useful at the point of clinical care to drive decision-making. Variable symptoms, priorities, and trajectories across conditions all limit the ability to have one standard means for assessment of clinical salience. As an example, a PRO-PM may be most salient for quality-of-life assessment before/after a total hip or knee arthroplasty (THA/TKA) given that the procedure is most frequently performed with the goal of pain relief and functional improvement. However, after a myocardial infarction, mortality may instead be better suited than a PRO-PM for quality measurement of care delivered. For PRO-PMs with a broader clinical focus (e.g. quality of life) or a longer term outcome, APMs are particularly well-suited and aligned with these characteristics given inherent limitations of traditional quality measurement in attributing collaborative and longitudinal care to a single provider or single setting. Prioritizing clinical salience is essential to ensure patient buy-in and investment in the capture of these data.

Adequate sample size

An adequate sample size and response rate are essential to consider moving forward any measure into payment models. Given that the average ACO is attributed approximately 20,000 beneficiaries,²⁰ PRO-PMs implemented within APMs should have a reasonable and achievable minimum response count with demonstrated validity, reliability, and an acceptable response rate that is representative of the population served. This principle supports prioritization of PRO-PMs that are used for high-volume procedures/conditions (per entity) or applicable to a wide range of patients in order to reliably assess performance and link to payment.

Maximizing PRO-PM response rates requires not only optimizing the logistics of data collection but ensuring both patients and providers are engaged in the value of the data. This may be best achieved by having the PROM data available at the point of care and integrating it in shared decision-making. Consideration should be given to the mechanism of survey administration (timing and tool), and the overall presentation and wording of a survey to improve response rates. Strategies to mitigate patient burden may include: using brief items written in plain language, providing multiple modalities for completion, and incorporating more tailored questionnaires using algorithms within computer adaptive tests.^19,21,22 For example, if a patient reports severe or extreme difficulty with getting out of the bath, then asking them about running at a later point in the PROM (e.g. Hip Disability and Osteoarthritis Outcome Score [HOOS]) is likely of limited utility and may lead to respondent frustration. Finally, it may also benefit providers to have a period of time to ramp up data collection by initially using structural or process measures to credit providers for adequate data collection prior to utilizing PRO-PM performance results in payment.

Meaningful performance range and detection timeframe

Measures used in payment programs will ideally have two features – meaningful range of performance among measured entities and a reasonable timeframe for detecting performance changes. A scenario in which all entities perform well on a PRO-PM and detection of meaningful difference takes several years would be considered lower priority for incorporation in payment models as the penalties or rewards of the APM would not be temporally aligned with the implemented quality improvement process or intervention. Both concepts first require key stakeholders to place the PRO-PM responses into context of the APM’s goals. Determining a meaningful performance range requires an assessment of the distribution of response values, with some desire for a spread distribution, but without there being a clear gold standard for determining what is universally appropriate. Second, the desired detection timeframe is also at the discretion of the data evaluators, as some APMs may be more willing to have a longer time horizon, while others may desire a shorter time period to detect differences between group PRO-PM responses.

Equity focus

Equity should be considered at each stage of PRO-PM development and implementation, tracking of response rates, and reporting of stratified results. The collection, reporting, and use of PRO-PMs for quality improvement should monitor social risk factors and the unique challenges faced by vulnerable populations, which affect both patients’ outcomes and their reporting or ability to report on PROs, so that implementation of a PRO-PM does not inadvertently exacerbate inequities in care faced by different groups. Specifically, measure selection should take into consideration any existing evidence of disparities in the selected outcome. Disparities-sensitive measures can be prioritized for use within APMs to improve equity as they may reduce selection bias from differential access to care procedures or settings. For example, if collecting PRO data on all patients with knee osteoarthritis, performance assessment could identify patients with high pain/low function scores who have not had surgery and could be attributed to entities (e.g. providers) by social risk factors. Additional measures can be chosen for use to focus on areas of known disparities in conditions, outcomes or experience. Once in use, PRO-PMs should be monitored both for inclusion of patients with social risk factors and disparities in outcomes. Oriented towards equity, implementing PRO-PMs requires thoughtful consideration of all potential biases, including biased provider assessments. Collection and examination of the relationship between responses/response rates and responder and provider demographics should be performed to monitor for unintended consequences.

Appropriate risk adjustment

Finally, appropriate risk adjustment is essential and should be considered in tandem with statistical approaches to address nonresponse bias. In particular, patients who are sicker or had worse outcomes may be less likely to respond than patients with better outcomes, suggesting that providers who are most diligent in following up with those patients may end up with a worse score that does not fairly reflect their performance relative to their peers. APMs may be particularly beneficial for PRO-PM implementation given the inclusion of data across several settings. Statistical methods, such as inverse probability weights, can be applied to address issues of missing data. Use of these techniques and risk adjustment models should be made transparent and prioritized to account for providers caring for sicker patients who may be less likely to respond to a PROM.^23–25

Conclusion

PRO-PMs offer the ability to measure performance at the entity-level in a patient-centered manner. Significant consideration must be taken during PRO-PM development and implementation phases to link available and novel measures to payment programs. Several principles should be prioritized when developing PRO-PMs and tying their measurement to payment: clinical salience, adequate sample size, meaningful range of performance among measured entities and the ability to detect performance change in a reasonable timeframe, an orientation towards equity, and appropriate risk adjustment. A phased and iterative PRO-PM implementation strategy that overcomes several potential uptake barriers should be considered to ensure provider and stakeholder engagement.

Highlights:

1. What is already known about the topic?

   1. Patient-Reported Outcome-based Performance Measures (PRO-PMs) offer a way to collect patient-centered information and aggregate the collected data into a reliable and valid measure of performance at the entity level (e.g. clinician, hospital, accountable care organization), yet have been potentially underused within the promising avenue of APMs.
2. What does the paper add to existing knowledge?

   1. We identified 54 instances of active PRO-PM usage across CMS programs as well as five principles to prioritize as part of greater PRO-PM development and incorporation within APMs: 1) clinical salience, 2) adequate sample size, 3) meaningful range of performance among measured entities and the ability to detect performance change in a reasonable timeframe, 4) equity focus, and 5) appropriate risk adjustment.
3. What insights does the paper provide for informing healthcare-related decision making?

   1. A phased and iterative PRO-PM implementation strategy that overcomes identified barriers should be considered to ensure provider and stakeholder engagement and ultimately inform and improve healthcare-related decision making by allowing reliable outcome comparisons across entities.