Challenges and Opportunities in Using Patient-Reported Outcomes in Quality Measurement in Rheumatology

Summary

Use of Patient-reported outcome measures (PROs) in rheumatology research is widespread, but use of PRO data to evaluate the quality of rheumatologic care delivered is less well established. This article reviews the use of PROs in assessing healthcare quality, and highlights challenges and opportunities specific to their use in rheumatology quality measurement. We first explore other countries’ experiences collecting and evaluating national PRO data to assess quality of care. We describe the current use of PROs as quality measures in rheumatology, and frame an agenda for future work supporting development of meaningful quality measures based on PROs.

Introduction

Quality measures provide important insight into variability or problems within structures of care, processes of care, or outcomes of care (1-3). Patient-reported outcomes (PROs) provide valuable information on patients’ health-related quality of life, and can be used to facilitate shared decision-making in the clinical setting, for comparative effectiveness research, for adverse event reporting, and in quality assessment (1,2,4-6). However, use of PRO measures as indicators of health care quality and accountability is a new—and growing—area in the United States.

Following passage of the Patient Protection and Affordable Care Act (ACA) in 2010, there has been a growing emphasis on improving performance and accountability of healthcare systems and individuals (7-9). Very recent legislation, the Medicare Access and CHIP Reauthorization Act (MACRA) of 2015, supports a shift in physician reimbursement via a Merit-Based Incentive Payment System (MIPS), in which physicians and systems will be judged and reimbursed partly on the basis of the quality of care they provide. Appropriate selection of measures that define “quality”—particularly measures that matter to beneficiaries of care—will be critical to the success of MIPS(8,10).

Given increased recognition that patient engagement and inclusion of the patient's voice are critical to the success of a high quality, affordable health system (4,7,8,11), incorporating measures that reflect the patient's direct report about how they feel and function into measures that evaluate quality of care is essential. However, there are several challenges to using PRO measures to assess performance and accountability (7,12-16), and how best to do this in rheumatology has yet to be defined.

In this paper, we discuss the role of structure, process, and outcome measures of healthcare quality using PROs, review European countries’ experiences collecting and evaluating national PRO data to assess quality of care, describe the current use of PROs as quality measures in rheumatology, and frame an agenda for future work supporting development of meaningful quality measures based on PROs.

Structure, Process, and Outcome: PRO Measures as Indicators of Health Care Quality

Our ability to understand the quality of health care, defined by the Institute of Medicine as “the degree to which health care services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge (17),” is fundamentally linked to how we define and measure quality. Quality measures that use PROs can address health care structures, processes, and outcomes, and there are important strengths and limitations to measuring each of these categories.

PRO outcome measures attempt to evaluate the ultimate impact of care provided, and thus are sought after metrics of health care quality (1,18). Outcomes can be measured at the individual level or aggregated by provider, practice, institution, organization, or region. Aggregating PRO outcomes data at the level of the health care system could theoretically identify poor performers and makes it possible for individuals to compare performance between health systems, driving accountability. However, with each level of aggregation, information about the processes and environments of care that contributed to a high or low score may become more difficult to identify.

While outcome-based quality measures are preferred by the Centers for Medicare and Medicaid Services, they provide limited information about the processes of care that lead to an outcome. PRO outcome measures might therefore tell us what needs to be improved, not how to do so. By contrast, process measures using PROs (e.g. was a PRO completed and scored, or shared with a patient) may be more actionable, and as such, more conducive to iterative quality improvement strategies (19). However, process measures may not map well to outcomes. This often reflects the presence of unmeasured factors that affect outcomes, such as socioeconomic determinants of health, which can be difficult to account for but are also often more difficult to change. The complex relationship between processes and outcomes does not invalidate process measures, rather it indicates the value in measuring both to understand quality and drive quality improvement. Thus, PROs are critically important in that they bring the patient's voice to the fore, but inherently problematic in that we don't measure or fully understand all the potentially modifiable processes of care that impact these outcomes.

The movement to use PROs as metrics of health care quality is underway. The National Quality Forum recently defined a new category of performance measures “based on PRO data aggregated for an accountable health care entity” called PRO performance measures (PRO-PMs), and delineated a pathway for their endorsement (4,7). While use of PRO-PMs has gained some initial traction in oncology and mental health(4), evidence directly linking collection of PRO measures to improvements in provider performance is conflicting or lacking (12,15,16,20). Additionally, we understand little about the relationship between PROs and the processes of care that modulate them.

National Health Systems and PRO Measurement: Experiences and lessons learned from Sweden and England

Sweden

National Quality Registers (NQRs), population level clinical quality databases, have existed in Sweden since 1975. With the creation of NQRs, the infrastructure to collect population level data to better understand the connection between health care processes and disease was created (21-23). Data from NQRs are aggregated and publicly reported for use in benchmarking and to develop guidelines and patient information. The majority (87%) of NQRs collect PRO data, and about 20% of these report using patient-reported data for local quality improvement work (21). PRO data from NQRs have been used to support patient-centered continuous process improvement, but participation in NQRs is voluntary and hospitals are not remunerated on the basis of PRO measure data.

The experience of the Swedish Rheumatology Quality Registry, created in 1995, highlights several important practical challenges and successes in using PRO data to improve care of patients with RA (22,23). Between 1995 and 2009, iterative improvement cycles were used to facilitate widespread use of the quality register, with 29,000 patients from 60 clinics registered by 2009 (22). Initially, paper forms were used for data collection, and data collected were redundant with data from the health record. Providers were frustrated with the time required to complete information for the register that detracted from patient care. With improvements in technology and work-flow, the process was streamlined: patients complete an electronic questionnaire assessing self-reported health (pain, global health, daily function, and tender and swollen joints). This information is then fed to the clinician, updated during the encounter, and fed back to the patient. Thus, the physician and the patient see and analyze changes in disease activity together. Patient and provider satisfaction have improved, as have providers’ understanding of patients’ actual medication-taking behaviors, since patients have been more forthcoming about medication adherence while reviewing trends in disease activity(22).

Data on PROs are aggregated at the provider, clinic, and hospital level, and are publicly reported (23). Thus data are used to understand which hospitals in Sweden have the lowest or highest PRO disease activity measures, which hospitals have the greatest decrease in PRO disease activity measures over defined time periods, and how certain PRO measures change over time(23). Data are also aggregated regionally to stimulate comparisons and create and open conversation about health care cost, though no data yet support the efficacy of this strategy.

Because of the breadth and depth of PRO data collected, Sweden is uniquely positioned to understand how PRO measures change over time, and specifically, how much variation in measurement is related to patient or health system factors, or reflects natural variation—and thus, the extent to which these measures are valid indicators of health system performance. It is critical that this kind of work be done in a system where evaluation is separate from remuneration. However, lessons about aggregate PRO data may be specific to the Swedish health care system and population--and less applicable to countries with lower literacy levels, greater racial and ethnic diversity, and multiple languages spoken.

England

In contrast to Sweden, where formation of NQRs and collection of PROs are voluntary, gathering PRO data in England is a government mandate (8,10). In 2010, the National Health Service (NHS) unveiled an Outcomes Framework outlining a performance model emphasizing health outcomes, in which PRO measures are used to facilitate comparison of providers, improve accountability, and motivate improvements in quality (4,7,8,11). PROs are also being used to help align patient and physician goals of care and treatment plans, such as the use of serial PRO measures in patients considering hip replacement (8). Engaging in shared decision-making and providing value-aligned care are processes that indicate high-quality care. However, a provider's aggregated PRO measure doesn't tell us whether he or she provides value aligned care.

Several unique challenges have arisen in England as a result of widespread mandatory implementation of PRO data collection. For example, the logistic burden and cost of creating the information technology infrastructure to support PRO data collection and analysis is significant. Furthermore, different patient populations may be less responsive to web-based questionnaires, such as those who are elderly (24). Capturing a broad spectrum of patients with a wide range of PRO responses is critical to ensuring measure validity, thus including both the very sickest and most vulnerable patients—as well as patients who are most healthy but do not interact with the health care system—will be key(8).

Because England is using PRO data to make determinations about the quality of care with financial repercussions, there have been specific challenges related to the ability (or failure) to attribute patient-reported outcomes to quality of care provided (8,11). Difficulties understanding when PRO variability reflects true change and when it reflects normal variation, understanding how best to use case-mix adjustment, how to define the reporting period, how to aggregate and report the data to different stakeholders, how to decide what constitutes ‘unacceptable performance’, and how to avoid misuse or misinterpretation of data indicate are all additional challenges. Therefore, the work to date suggests that far more experience with these measures is needed before they create value for patients and to the NHS. (8,25).

Quality Registries in Rheumatology: The US Experience with PROs as quality measures

Based on our systematic review of the literature (detailed in Appendix 1), evidence supporting the use of PRO measures in rheumatic disease care in the United States is scarce. However, the experience at Geisinger Health System demonstrates that collection of PROs in a busy rheumatology clinic is feasible, is associated with high-quality processes of care, and may improve outcomes (26).

In a recent evaluation of an EHR optimized for rheumatology practice, Newman and colleagues tracked 14 clinicians, nearly 6,700 patients, and data from almost 20,000 encounters over a two-year period within the Geisinger Health System in eastern Pennsylvania. The EHR, Rheum-PACER, captures, aggregates, and displays patient-reported measures of disease activity, physical function, and pain. While the primary aim of the study was to evaluate the impact of the software on physician productivity and efficiency, the authors also reported a modest but significant correlation (r=0.59) between physicians’ use of the software and rheumatoid arthritis (RA) patients’ disease control, defined as Clinical Disease Activity Index (CDAI), a composite outcome measure which includes a patient-reported component, of ≤10. Additionally, they show a small (3%) but significant trend of increasing numbers of patients with controlled disease (CDAI ≤10) over time. Use of the EHR was associated with process improvements (chart review and documentation time decreased and productivity increased); patient adherence, activation, and satisfaction scores were high at baseline and did not change.

Building infrastructure to collect PRO data from a large and diverse network of rheumatology practices in the US will help create large datasets that will yield important information about the opportunities and challenges of using PROs for quality measurement in clinical settings. The American College of Rheumatology (ACR) national registry, Rheumatology Informatics System for Effectiveness (RISE), the VA Rheumatoid Arthritis registry (VARA), and quality networks like the Pediatric Rheumatology Clinical Outcomes and Improvement Network (PR-COIN) have made progress in building this infrastructure and accumulating data on PROs will continue to support and inform future quality measurement efforts.

National Quality Forum Endorsed Quality Measures in Rheumatology

The National Quality Forum (NQF) now endorses many musculoskeletal quality measures; while nine measures included in the NQF Quality Positioning System (27) address self-reported changes in basic mobility, these are all intended to assess rehabilitation following injury, surgery, or facility admission. There are four NQF-endorsed RA quality measures, of which, two include PROs: an annual recorded measure of disease activity, and an annual recorded measure of physical function. All disease activity measures include a patient-reported component. Some include only PROs (Routine Assessment of Patient Index Data 3 (RAPID3), Patient Activity Scale (PAS, PASII), while others incorporate physician-reported information such as tender and swollen joint counts (CDAI), and laboratory data (Simplified Disease Activity Index (SDAI) and Disease Activity Score (DAS). Functional status measures require administration of a validated tool, such as the Multi-dimensional Health Assessment Questionnaire (MDHAQ), HAQII, or Patient Reported Outcomes Measurement Information System (PROMIS) Physical Function forms (PF-10a, PF-20, or CAT).

Both NQF-endorsed quality measures—documenting disease activity and physical function annually in RA patients—incorporate PROs. These measures calculate the proportion of patients seen by a given clinician in a fixed time period who have the measure recorded as a score in the electronic health record and thus reflect structure (whether the particular system is structured to facilitate collection of these data) and process (whether providers collect these data) of care. Additionally, both of these measures require that the resulting disease activity or functional status score appear in the EHR, so that while the disease activity metric and the physical function metric are not outcome measures, they do allow aggregation of outcomes by practice and provider, and RISE allows national benchmarking of scores against other practices. For example, all of the disease activity measures have validated cut-points for remission, low, moderate and high disease activity (28,29). This allows aggregate benchmarking by the registry regardless of which measure Clinicians use.

Existing US Guidance for Use of PROs in Performance Measurement

As we move towards a new funding model in the healthcare system in the United States where accountability and performance are evaluated by both measures of process and outcome, ensuring that PRO measures used to evaluate performance and quality are relevant to the population, reliable, valid, interpretable, culturally and linguistically appropriate and understandable, and are not burdensome is critical. The National Quality Forum endorses measures that meet these standards, and has established a pathway for PRO performance measure endorsement (7,13). A Technical Expert Panel (TEP) assembled from the Physician Consortium for Performance Improvement, expanded on NQF guidelines and outlines nine best practices (30). Recommendations from NQF and the TEP are summarized in Table 1. A hypothetical PRO measure that could be used as a performance measure in rheumatology, conforming to NQF recommendations and the TEP best practices is also presented in Table 1.

Table 1.

Best practices for developing and evaluating proposed patient-reported outcome performance measures (PRO-PMs),

Adapted from recommendations from Basch E, Spertus J, Dudley RA, Wu A, Chuahan C, Cohen P, et al. Methods for Developing Patient-Reported Outcome-Based Performance Measures (PRO-PMs). Value Health 2015;18:493-504 and National Quality Forum. Patient Reported Outcomes (PROs) in Performance Measurement. National Quality Forum, ed. qualityforumorgProjectsn-rPatient-Reported_OutcomesPatient-Reported_Outcomesaspx 2013. Available at: http://www.qualityforum.org/Publications/2012/12/Patient-Reported_Outcomes_in_Performance_Measurement.aspx. Accessed September 28, 2015 with an example of a hypothetical rheumatology performance measure; with permission.
Best Practices (TEP)	NQF Measure Evaluation criteria	Hypothetical Example of future PRO-PM for Rheumatology
1. Describe a rationale for measuring the outcome (What is knowledge gap? Does use of a patient-reported outcome specifically address the gap? Are patients the most appropriate source of information?)	Evidence Performance Gap Impact - Importance to Measure and Report (evidence of value to patient/person, amenable to change)	Rationale: Published cross sectional data suggest high-rates of symptomatic fatigue in RA patients. This represents a symptomatic and potentially modifiable problem - recent data suggest that feedback from wearable devices may improve physical activity level and reduce RA-associated fatigue.
2. Describe and justify the intended context of use (How will information inform change in practice to improve performance in the intended setting?)	Evidence Performance Gap Impact - Importance to Measure and Report Comparison to Related or Competing Measures	Purpose: to understand whether there are variable rates of RA-associated fatigue between practices that might be improved through feedback of rates to providers.
3. Measure should have data to support its meaningfulness and importance to patients as well as adequate psychometric properties in the setting in which it will be used	Reliability and Validity / Scientific Acceptability of Measure Properties	The PROMIS-Fatigue CAT instrument has been tested in RA patients in clinical trial settings, but is not yet widely used in the clinical setting.
4. Measure should have demonstrated sensitivity to change and yield of clinically actionable information in the setting in which it will be used	Feasibility Usability and Use	Studies to assess the sensitivity of PROMIS-Fatigue CAT to clinically meaningful changes in RA patients are underway.
5. Implementation strategy for measure should exist in the setting in which it will be used	Feasibility	In this hypothetical example, PROMIS-Fatigue CAT would be integrated into the electronic medical record for all practices participating in the Rheumatology RISE registry, an existing quality-reporting network. The measure would be administered, at baseline and every 3 months for 12 months.
6. An analysis plan should be determined in advance that includes a risk-adjustment strategy (if appropriate), approach to missing data, and power calculation	Reliability and Validity / Scientific Acceptability of Measure Properties	The proportion of patients with a PROMIS-fatigue score >50 (US population mean) at any time during follow up will be measured. The mean PROMIS-fatigue score for patients in each ACO at baseline and in each rolling 3-month period will be measured. Mean within-person change in each ACO will be calculated. Sample size will need to be determined based on existing data. Exploratory risk adjustment for age, race/ethnicity, insurance status, comorbidities, baseline quality of life, etc. will be performed.
7. A framework should exist to identify and interpret clinically meaningful results	Reliability and Validity / Scientific Acceptability of Measure Properties	Significant work understanding the responsiveness and minimally important difference (and relevant reporting time period) of the PROMIS-fatigue CAT is needed prior to undertaking this work.
8. An approach to reporting, sharing, and disseminating results should be planned and described	Usability and Use	Results will be fed-back to practices with a planned strategy to address recognition and management of RA-associated fatigue for sites with the lowest performance.
9. An evidence-based approach should be taken to assess the impact of the measure	Comparison to Related and Competing Measures	Regular follow-up assessments of all practices to understand trends and undertake continuous process improvement strategies locally. Until clear validity and responsiveness (and impact) of the measure is established, systems should not be evaluated on the basis of the metric.

Future Directions

Challenges

• There remain significant challenges with implementation of PROs. The US is a large, racially-ethnically, linguistically, and culturally diverse country, and has many electronic health records (EHRs) that don't yet communicate with each other, making widespread implementation of PROs (for any purpose) challenging. However, multilingual tools, such as many PROMIS measures, are becoming available, and some EHR vendors, such as Epic systems, are beginning to include functionality that enables more seamless collection and recording of PROs in the EHR. Moreover, technological innovations, such as the RISE registry, are aggregating data across practices with different EHRs, thereby offering a solution to some of the interoperability issues that been barriers to PRO data aggregation.

• Using PROs to assess provider quality will continue to pose challenges. Rheumatologic diseases have a long trajectory, making it potentially difficult to attribute quality of care based on PROs to a single provider or even health system over a short time period. Evidence supporting a relationship between PRO data collection and improved provider performance is not strong – but the need to improve actual patient-centered care is.

• More data is needed to understand whether changes in PRO outcome measures reflect changes in healthcare process and environment, to ensure measures are valid and reliable, and to understand whether case-mix adjustment is appropriate.

• More work is needed to understand how PRO information should be summarized and presented to various stakeholders (patients/purchasers of health care, physicians, and accountable care organizations).

• Much planning is needed to address competing priorities of different stakeholders when PROs are used for different purposes (patients, providers, accountable care organizations, and benchmarking organizations).

Opportunities

• Building infrastructure to develop widespread PRO collection with harmonized measures is valuable not just for quality improvement, but for comparative effectiveness research as well.

• Routinely measuring PROs may help shift rheumatology care towards being even more symptom driven and better align patient and provider goals.

• Aggregating PROs at the population level may help us elucidate disparities in health. At the bedside, the role of PROs might be to facilitate incorporating the patient's voice, helping patients monitor their own progress, ensuring that key symptoms are addressed.

Recommendations and Future Directions

• Decide on the measures, with input from patients about what is most important to them, but also with input from experts about what is reliable, valid, and responsive. The ACR's RA quality measures that incorporate PROs, disease activity and functional status, are examples of measures developed with at least some patient input.

• Develop and test a viable implementation strategy and formally test both measure implementation and its effect on downstream outcomes. Implementation of the RA PROs in EHRs around the country, formal testing of the feasibility, validity and reliability of these early implementation strategies, and subsequent scaling of efforts to the national registry (RISE) is one example of development and testing of a PRO process measure.

• Continue foundational work necessary for use of PRO outcome performance measures. Decide on the appropriate level of aggregation (at the population or health system level rather than individual provider). Develop clear and consistent clinical and administrative definitions of patients who represent the population of interest, the reporting period, and the period at risk. Perform extensive study to understand the relationship between elements of care (structure, process, outcome) and PROs to clarify outcome attribution and case mix adjustment. Use balancing measures to monitor unintended consequences at the health system level.

• Interpret aggregated PRO measures thoughtfully - if PRO measures are consistently low in safety-net settings, this could reflect poor quality care or could represent a population of vulnerable patients with more comorbidities or barriers to care who need access to more substantial resources.

• Develop adequate information technology infrastructure to capture PRO data from diverse rheumatology practices across the United States to understand measure variability in urban and rural settings, from safety net, private practice, academic, and Veterans Affairs health systems—and in patients with different rheumatic diseases.

• Think broadly and creatively about the role of PROs in clinical care –the most effective use of PROs in clinical care may be not performance measurement at the individual physician or even practice level, but to assess how rheumatologists are caring for patients at a population level, and to elucidate disparities in health. At the bedside, the role of PROs might be to facilitate incorporating the patient's voice, helping patients monitor their own progress, and ensure that key symptoms are addressed.

As rheumatology develops clinical practice guidelines and quality measures (9,31-35), PROs should be considered and evaluated as candidate measures. Moving forward, standardized and routine collection of patient-reported outcome measures will likely be required. Being able to evaluate this information across different rheumatologic diseases and across healthcare systems will allow for richer understanding of the relationship between a patient's health-related quality of life and the quality of care rheumatologic care received, and ideally, will promote development of novel strategies to address gaps that exist.

Key Points.

• The healthcare landscape in the United States is likely shifting to a model in which health systems will be reimbursed for the quality of care they provide, and developing valid, responsive, and meaningful patient-centered measures is key.

• How best to incorporate PRO measures in assessments of healthcare quality in rheumatology is underexplored.

• Experiences with widespread use of PROs in Sweden and England, and in smaller health systems within the US, provide us with valuable lessons about challenges and opportunities in using PROs to assess quality.

• Major challenges include: developing sufficient information technology infrastructure to collect data from diverse medical records and diverse patients; need for better understanding of PRO reliability, validity, and responsiveness; determining that PROS are responsive to changes in the healthcare environment; clarifying the role of case-mix adjustment; understanding how measures should be summarized and reported to stakeholders.

• Major opportunities include: provision of more value-aligned care; collection of data that can serve multiple purposes (comparative effectiveness research in addition to quality assessment); assessment of the impact of rheumatologic care, from the patients perspective, at the population level; ability to elucidate disparities in care.

Appendix: Systematic Review

We performed a systematic review of the literature to assess current practices, guidelines, and evidence for use of PRO measures in quality assessment in rheumatology. Specifically, we focused on how PROs were used in the clinical setting to promote quality improvement strategies, whether and how PRO data were aggregated, and whether these data were used to assess quality of care delivered.

With the assistance of a professional librarian, we searched three electronic databases (MEDLINE, Web of Science, GoogleScholar), from database inception to September 2015; MeSH terms and strategy are listed below (Table 2). We evaluated grey literature, including proceedings from major rheumatology meetings (American College of Rheumatology (ACR) and the European League Against Rheumatism (EULAR) from 2010-2015), and conducted hand searches of reference lists of retrieved articles.

Table 2.

MESH terms used:

Search strategy	Search terms
1	“Patient-Reported Outcome” OR “Patient Reported Outcomes” OR “Patient Reported Outcome Measure” OR “Patient Reported Outcome Measures” OR “PRO-PM” OR (“Self Report”[MeSH] AND “Outcome Assessment (Health Care)”[MeSH])
2	“Guideline Adherence”[MeSH] OR “Outcome and Process Assessment (Health Care)”[MeSH] OR “Benchmarking”[MeSH] OR “Quality Improvement”[MeSH] OR “Quality Indicators, Health Care”[MeSH Terms] OR “quality indicator” OR “quality indicators” OR “quality measure” OR “quality measures” OR “performance indicator” OR “performance measure” OR “performance measures” OR “NQF OR “NHS”
3	“Rheumatic Diseases”[MeSH] OR “Spondylarthropathies”[MeSH] OR “Lupus Erythematosus, Systemic”[MeSH] OR “Scleroderma, Systemic”[MeSH] OR “Vasculitis”[MeSH] OR “Myositis”[MeSH] OR “Mixed Connective Tissue Disease”[MeSH] OR “rheumatic disease” OR “lupus” OR “scleroderma” OR “vasculitis” OR “myositis” OR “ankylosing spondylitis” OR “psoriatic arthritis” OR “reactive arthritis” OR “rheumatoid arthritis” OR “juvenile arthritis”
Medline Web of Science Google Scholar	1 and 2 1 and 2 and 3

Studies evaluating use of PROs in a clinical trial or in comparative effectiveness research were excluded as were articles describing use of PROs in evaluating orthopedic management of musculoskeletal conditions such as total joint arthroplasty for osteoarthritis.

Of 534 titles identified by the literature search, 1 publication was identified that described initiatives in which PROs were explicitly used or studied in quality initiatives related to rheumatic disease care in the US (26). An additional case from Sweden (23) was identified by reviewing reference lists of articles related to PRO measures and quality initiatives not specific to rheumatic disease. 24 abstracts from available ACR and EULAR proceedings related to PROs, and none specifically addressed quality indicators or quality measures.

Both studies identified focused on disease activity measure, pain, and physical function PROs for patients with RA. In both studies, PRO data were collected and fed-back to patients and providers in real time, PRO data were used for improvement at level of individual (decision to escalate therapy) and aggregated at level of provider to evaluate relationship between collection of PRO data and health outcomes (disease activity). In the Swedish study (23), PRO data were also aggregated by clinic and used for both national benchmarking and identification of opportunities for improvement, but there was no explicit discussion of whether this strategy led to improvements in quality of care.

Our systematic review demonstrates the dearth of published literature evaluating PRO measures as candidate tools to evaluate quality of care in the rheumatic diseases, particularly in the United States.