The Patient-centered Outcomes Research Institute’s Role in Advancing Methods for Patient-centered Outcomes Research

Currently, there is wide-spread recognition that much of health care is insufficiently grounded in evidence as to what works for whom under what circumstances. An important reason for this is that research efforts often do not compare alternative options of health care nor are they sufficiently focused on outcomes that matter to patients.1,2 In addition, the methods to conduct research designed to inform decision makers about what works for whom and the infrastructure to support these efforts require further development.3,4 The Patient-centered Outcomes Research Institute (PCORI) was established to fund patient-centered comparative effectiveness research (CER) to assist patients, clinicians, payers, and policy makers in making informed health decisions.5 Its mission also includes an active program to develop and improve CER methods and to invest in critical infrastructure within which effective and efficient health care research can be conducted and integrated with a learning health care system.3

This article focuses on PCORI’s mission relative to CER methods development and improvement. We describe PCORI’s legislatively mandated Methodology Committee and its major initial product, the Methodology Report; PCORI’s current slate of CER methods projects; and finally, some initial thoughts about future areas where further methods development is needed.

PCORI AND PCORI’S METHODOLOGY COMMITTEE

A majority of PCORI’s funding is dedicated to studies in 5 priority areas identified in the National Priorities Research Agenda.6 These include research focused on: (1) investigating clinical comparative effectiveness between different treatment options; (2) improving health care systems; (3) addressing disparities in health; and (4) communicating and disseminating research evidence. A fifth priority identified in the Agenda is focused on accelerating patient-centered CER and includes an emphasis on research methods used in the conduct of this type of research. The articulation of this priority is an explicit recognition that methodological improvements in patient-centered CER will benefit all stakeholders, including researchers, policy makers, clinicians, patients, and caregivers making health care decisions. PCORI was mandated by Congress to fund research through a contract mechanism rather than a grant. This enables PCORI’s programmatic staff to work closely with awardees throughout the postaward stage.

PCORI’s founding legislation established a 17-member Methodology Committee, whose charge is “to develop and improve the science and methods of comparative clinical effectiveness research” and to produce “methodological standards for research.”7 These standards are intended to support the conduct of methodologically robust CER that is responsive to the needs of patients and other stakeholders. Rigorous research methods support findings that can be marshaled to directly improve patients’ health care outcomes.8

On July 23, 2012 the Methodology Committee released its first draft Methodology Report for public comment, with a final version approved by PCORI’s Board of Governors on November 18, 2013.9 The Report contains the first set of recommended standards for the conduct of patient-centered CER. All applicants for PCORI funding awards are required to adhere to these standards.7 The Report describes the rationale behind the creation of standards for patient centeredness, for prioritizing topics for research, for choosing a study design, and for designing, conducting, and reporting patient-centered CER. Importantly, the Report also highlights areas where further methodological work is needed to provide necessary guidance to investigators conducting patient-centered CER.

PCORI’S CER METHODS PROGRAM: ADDRESSING METHODOLOGICAL GAPS

PCORI’s Methods Program is building a research portfolio to address the gaps in the field of patient-centered CER identified in the Methodology Report. The first iteration of the Methods Program Funding Announcement is largely based on the research areas and gaps identified by the Methodology Committee and articulates 6 broad research areas of programmatic interest.10 These are: (1) research into patient engagement strategies and measurement, including ethical issues related to participation in research; (2) methods to identify patient-centered and patient-reported outcomes (PROs) and measure them appropriately; (3) methods for research topic generation and prioritization; (4) the improvement of analytic methods of importance to PCOR, including causal inference, heterogeneity of treatment effects, and missing data; (5) methods related to cluster-randomized trials, registries, and studies of devices and diagnostics; and (6) methods related to the capture and use of data collected from multiple data sources across multiple health care systems. In September 2013, PCORI awarded the first 19 projects under the Methods Program. The Methods Program portfolio now comprises 41 projects in a range of methods areas. A list and description of the projects can be found in Table 1.

TABLE 1.

PCORI-funded Methods Projects

Methods for Improving the Patient Centeredness of CER

One set of standards described in the Methodology Report provides guidance on patient centeredness. PCORI is committed to funding research that answers questions that are important to patients, so that they may make treatment decisions based on evidence that reflect their individual preferences. To help accomplish this, PCORI requires patient and stakeholder engagement in all aspects of research. However, methods for selecting, recruiting, and engaging patients and stakeholders in patient-centered CER are in their infancy and require further development and evaluation. Indeed, recent systematic reviews of patient and stakeholder engagement in the conduct and dissemination of research have articulated a number of evidence gaps related to establishing effective engagement practices.11–13 Specifically, these systematic reviews highlight gaps related to methods for selecting and recruiting patients and stakeholders to participate in the design and conduct of research,11 research that compares the impact of different methods of engagement on the relevance and applicability of research findings,11,13 as well as research that develops user-friendly tools and training materials for patient and stakeholder engagement that can be used in patient-centered CER.13

To address these outstanding questions, the PCORI Methods Program is funding methods projects that focus on patient and stakeholder engagement in research as well as projects that aim to improve the methods for developing and implementing patient-centered outcomes. Methods projects within this category cover a variety of topics including the impact of patient engagement strategies, the involvement of patients and stakeholders in generating and prioritizing topics for research,14 and determinations of how best to elicit information from patients concerning what outcomes matter most to them. Within our current portfolio, 1 study is comparing 2 methods for recruiting and engaging minority patients and stakeholders in research (Table 1, row 23). Investigators from another study are developing methods and tools for engaging patients in diverse communities in Colorado to translate evidence-based guidelines into messaging that resonates with their communities (Table 1, row 24). A third study is developing a set of best practices for engaging patients and stakeholders in the development of patient decision aids (Table 1, row 26).

Regarding the engagement of patients and stakeholders in generating prioritized research agendas, PCORI is especially interested in methods to quantitatively assess research priorities, such as value of information. Addressing this area of interest, 1 study will implement a structured approach to prioritizing cancer trials using multistakeholder panels (including patients and payers), testing assumptions that including these key stakeholders early on in a prioritization process helps with the process of discarding less meritorious ideas (Table 1, row 38). Another study will use value of information techniques to determine whether additional research on treatments for coronary heart disease is needed (Table 1, row 8). Studies such as these can assist PCORI in identifying priorities for future patient-centered CER.15

Given the lack of standards available for selecting appropriate patient-centered outcomes16 and for eliciting patient’s values and preferences and incorporating them into patient-centered CER and clinical care, the CER Methods Program is also funding several projects in these areas. One study is developing portable quality of life measures to improve their validity and interpretability in a wide range of patient-centered outcomes studies (Table 1, row 31). Another study will develop methods to measure patient-centered communication in the field of oncology (Table 1, row 29), where the importance of effective communication and information transfer is increasingly recognized but where there is still often a significant disconnect between patients and clinicians regarding the goals of cancer care.17 A third study is assessing methods for patient goal setting and attainment. These methods allow for trade-offs when patients and their clinicians are managing multiple comorbidities and allows for patients’ preferences to be expressed in their choice of treatments. The study focuses particularly on dementia patients and their families (Table 1, row 28). A final study is working on improving the measurement of PROs through the development of tailored Patient-reported Outcomes Measurement Information System condition-specific impact assessments that integrate generic and condition-specific PRO measures (Table 1, row 30).

Analytic Methods for CER

A large section of the Methodology Report is focused on improving analytic methods.9 As was discussed during a recent Institute of Medicine workshop on Observational Studies in a Learning Health System, the use of large observational datasets, as well as the increasing number of datasets that include diverse data sources such as electronic health records and administrative and claims data, will benefit from advanced methods to improve causal inference and to handle missing data.18 In addition, as CER seeks to make inferences about the best treatments for patients with different characteristics, there is a need for improved methods to study heterogeneity of treatment effect.18,19

Several studies in the methods portfolio focus on improving techniques that account for unmeasured confounding using high-density propensity scoring, targeted maximum likelihood estimation, and machine learning techniques in large observational datasets, including datasets where there are few outcome events. One study is developing a specialized toolkit to provide researchers access to advanced analytic techniques, such as inverse probability weighting of marginal structural models and the parametric g-formula approach, to account for time-dependent confounding (Table 1, row 4). A second study is comparing the results and patient populations from randomized controlled trials and observational studies for the same health condition to assess and refine statistical methods for causal inference (Table 1, row 8). Another study is evaluating propensity score methods with the goal of determining which are optimal for detecting and estimating treatment-effect modification (Table 1, row 9). A fourth study is investigating the properties of risk adjustment and instrumental variables when treatment effects are heterogeneous across >1 outcome (Table 1, row 16). Finally, several studies are developing or improving methods to address missing data. One study uses global sensitivity analysis to account for missing data in clinical trials (Table 1, row 20), whereas several studies are investigating imputation methods in observational data sets (Table 1, rows 21 and 22), including missing time-varying covariates.

Methods to Improve Study Designs and the Quality of Data in CER

In addition to the general interest in improving analytic methods, the CER Methods Program is also seeking to fund projects that aim to improve specific study designs and data quality. One study seeks to improve methods in systematic reviews and meta-analysis by exploring the incorporation of data from multiple data sources, including observational and clinical trial data (Table 1, row 12). Another study investigates methods to improve the collection and analysis of PROs in registries. It is examining the different methodological assumptions made to account for the collection of PROs at defined time intervals and will address the impact of these assumptions on CER results (Table 1, row 36). A third study is developing new statistical methods and a software tool for designing adaptive enrichment trials. The software will compare different study designs and will recommend those that have the best performance for answering a specific research question (Table 1, row 11).

Finally, multiple data sources are now available to support patient-centered CER but the quality of these data vary and its use is impeded by a lack of a set of agreed-upon quality measures.20 One study aims to generate empirical evidence about the effects of including patients and stakeholders in decisions about reporting and storing data, and will seek to develop recommendations for reporting data quality results in a standard format, a development that will help with the transparency of data quality across CER studies (Table 1, row 41).

DISCUSSION

An important component of PCORI’s mission is to support the development of both innovative methodological approaches and refinements of existing methods that will improve the rigor, reliability, and validity of patient-centered CER studies. The current portfolio of projects funded through PCORI’s CER Methods Program reflects PCORI’s commitment to a wide range of areas from improving methods for patient engagement in research to improving analytic methods and challenges related to study designs.

However, there are a number of challenges to establishing a successful portfolio of methods projects. These challenges include attracting diverse and innovative methods proposals and defining the metrics for success.

Because the methods gaps identified in the Methodology Report are wide ranging, to attract research to address those gaps, PCORI must engage multidisciplinary researchers from diverse research communities. In addition, PCORI must ensure that it selects and funds methods projects that have high usability, and therefore high uptake, in the field of CER. This can be challenging to assess as many projects aimed at methods development focus on one particular clinical condition. Applicants are encouraged to consider the usability of their study and to describe how the methods developed through their proposal might be applied in a different population or clinical setting. Determining the right level and scope of stakeholder engagement in projects that aim to develop or improve methods, and how it impacts the relevance of the project is another key consideration. Currently, PCORI staff identify a slate of applications for funding based on the scores they obtain during merit review, programmatic balance and fit, and PCORI’s strategic priorities. Balancing is largely aimed at ensuring that a diversity of methods’ areas and issues are represented in the portfolio.

A foundational challenge for any funder is how to define and measure the success of the funded portfolio. The metrics required for such an evaluation are particularly complex to define in the area of methodological research. One metric is whether the research gaps identified by the Methodology Committee report are being addressed. Another is whether successful methods are being disseminated to the appropriate users including other methodological researchers and CER investigators. PCORI will continue to explore what successful dissemination of its methodological findings might require, including the awarded research teams’ ability to make their research products and deliverables available in the form of open-source software, webinars, applications, or other platforms.

Moving forward, the CER Methods Program will continue to work to identify relevant methods gaps in the field of patient-centered CER and to refine its funding announcement to focus on those areas. The program has already expanded its funding announcement to include methods related to human subjects’ protections as well as methods to develop Patient-reported Outcomes Measurement Information System measures. In addition, PCORI anticipates learning of other gaps through several new funding initiatives including the establishment of the National Patient-Centered Clinical Research Network (PCORnet) and the funding of several large pragmatic CER trials. PCORnet is a $102 million initiative that is a significant step towards supporting the development of a learning health care system in the United States.3,21 Indeed PCORnet aims to significantly improve the nation’s capacity to conduct patient-centered CER efficiently, by creating a large, highly representative clinical research network for conducting clinical outcomes research. In addition to remaining practical and cultural challenges in PCORnet,22 there are also a number of methodological innovations that will need to be developed, piloted, accepted, and implemented for such a system to be successful. Possible areas where additional methods work may be needed include methods for matching patient electronic health record and claims data to obtain longitudinal patient histories suitable for research purposes, methods for distributed data analysis,23 methods for developing PRO instruments that can be used in clinical practice by patients and clinicians without disrupting workflow,24 methods for data linkage that protect patient privacy, and methods for determining appropriate real-world decision maker thresholds of evidence and how to handle uncertainty.

Pragmatic CER trials are trials that are designed to address practical comparative questions faced by patients and clinicians, that are inclusive of diverse patient populations, and that are conducted in real-world clinical settings.25–27 To mimic clinical care, these trials often impose few inclusion and exclusion criteria and also often impose few restrictions in terms of how the interventions are delivered. Although these design features may help to improve external validity, they can threaten internal validity and therefore, it will be important to conduct additional methodological work alongside ongoing PCTs to understand the appropriateness of this trade-off in different situations. Other areas where innovative methods works is needed is in developing appropriate informed consent platforms that can be readily integrated with clinical care, in operationalizing Bayesian-type adaptive mechanisms in a less controlled environment, and in defining “standard practice” or “usual care” as one of the comparator arms in this type of research.

These and undoubtedly many other methods-related questions will arise as the field of patient-centered CER continues to develop and as the United States continues to work toward instituting a real-world, dynamic learning health care system. Addressing these methods challenges will require both conceptual thinking and practical research and likely consensus building among stakeholders regarding the most appropriate approaches to adopt in different situations. The PCORI Methods Program is committed to funding methods projects that aim to discover, develop, and disseminate research methods that achieve these important goal and looks forward to fostering ongoing partnerships with investigators to ensure the success of the program.