Model for a Patient‐Centered Comparative Effectiveness Research Center

Abstract

This special report describes the systematic approach the University of Pittsburgh and the University of Pittsburgh Medical Center (UPMC) undertook in creating an infrastructure for comparative effectiveness and patient‐centered outcomes research resources. We specifically highlight the administrative structure, communication and training opportunities, stakeholder engagement resources, and support services offered.

Introduction

Comparative effectiveness research (CER) is defined by the Institute of Medicine (IOM) as the generation and synthesis of evidence comparing benefits and harms for prevention, diagnosis, treatment, monitoring, or improvement to delivery of care for the purposes of making informed decisions to improve health care at the individual and population levels.1 CER can also be described as research that aims to determine, “which treatment works best, for whom, and under which circumstances.”2 By either definition, effectively conducting CER requires multidisciplinary research using numerous types of study designs, such as observational studies, randomized trials, analysis of registries and electronic health records, systematic reviews and meta‐analyses. Patient involvement may enhance CER. For example, the IOM lists measuring outcomes that are important to patients, as well as informing a specific clinical decision from the patient perspective as CER characteristics.3 The necessary expertise for this research is substantial and diverse, and unlikely to exist in an already collaborating group. Additional infrastructure, such as the initiation of an organizational unit devoted to motivating, facilitating and collaborating CER, is therefore necessary on both national and local levels. A number of funding efforts have been devoted to building such infrastructure. Most notably, the American Recovery and Reinvestment Act (ARRA) of 2009 earmarked $1.1 billion for CER.

Building on ARRA and repeated calls for an independent CER institute,4 the Patient Protection and Affordable Care Act established the Patient‐Centered Outcomes Research Institute (PCORI), which is an independent, federally funded institute dedicated to advancing the CER agenda. PCORI defined patient‐centered outcomes research (PCOR) as addressing the questions of what a patient should expect given his or her characteristics, conditions, and preferences, what treatment options are available (with what harms and benefits), what can be done to improve outcomes, and how clinicians and healthcare systems can best facilitate that improvement.5 PCORI has become the major funding source for patient‐centered CER (PC‐CER), with $150 million a year for each of the 8 years, 2012 through 2019.

While PCORI and associated funding create opportunities, they also produce substantial new challenges that need to be the part of building infrastructure for PC‐CER. First, PCORI emphasizes involving patients and stakeholders from the beginning of the proposal development phase through conducting the study and disseminating the results. Thus, patients and stakeholders must be active members of the investigative team, which is atypical compared to the investigator‐driven approach of the traditional clinical research enterprise. In addition, all research must follow the standards within the PCORI Methodology Report.6 This report details 47 minimum standards for conducting PC‐CER. These standards cover both cross‐cutting methods and approaches specific to certain study designs and analysis methods. While these requirements and standards promote quality research that focuses on the patient and patient‐centered outcomes, they further complicate the need for multidisciplinary collaborations and PC‐CER infrastructure. This special report details the efforts at the University of Pittsburgh and the University of Pittsburgh Medical Center (UPMC) to develop a research core and data center, that provide the needed resources for facilitating high quality PC‐CER.

Approach

Objectives

Lesson 1: Core funding quickly built an infrastructure that proactively sought additional funding.

The Comparative Effectiveness Research Core (CERC) was established in 2011 to support PC‐CER at the University of Pittsburgh and UPMC, which have a unique collaborative clinical and academic model. UPMC oversees all clinical activity, including that from a consolidated physicians’ practice plan. The University of Pittsburgh leads the overall institution's academic activities, particularly faculty‐based research.7 Interest in developing this infrastructure stemmed from (1) the desire to promote collaborative PC‐CER across the University, and (2) the availability of new funding sources, such as PCORI. Prior to founding the core, the University performed a detailed inventory of ongoing PC‐CER. The generated project portfolio was intentionally broad, showcasing current PC‐CER work and enabling the CERC leadership to learn about existing resources and expertise. The CERC is currently housed in the University of Pittsburgh Health Policy Institute (http://www.healthpolicyinstitute.pitt.edu/) and also previously served as a core in the Clinical and Translational Science Institute (CTSI). This paper is written from the latter perspective. Its overall goal is to coordinate PC‐CER activities across the six Schools of Health Sciences at the University of Pittsburgh (School of Dental Medicine, School of Medicine, School of Nursing, School of Pharmacy, Graduate School of Public Health and School of Health and Rehabilitation Sciences) and UPMC. The CERC provides a clearinghouse for university and UPMC researchers to learn about PC‐CER, obtain training in PC‐CER methodology, collaborate with established PC‐CER methodologists, receive guidance on stakeholder involvement and utilize a data center. Initial capital for the CERC was provided by the University's Senior Vice Chancellor for the Health Sciences and the CTSI for a minimum of 4 years. Funding provided salary support for a Director at 10%, Data Center Director at 20%, program manager at 100%, biostatistician at 10%, systems engineer at 100%, and administrative support at 50%. Hardware, software, licenses, and data were purchased for under $130,000 to create and sustain the data center. The funding created an infrastructure that could provide services through the CERC without charging researchers. Consequently, all levels of researchers from all of the Schools of the Health Sciences and UPMC had equal access to PC‐CER services and resources.

The CERC is impartial to specific clinical or methodological priorities within the fields of PC‐CER. It is centered on methods and directed by a statistician. An alternative would have been to build the CERC around a clinical area of expertise, such as comparative effectiveness in cardiovascular disease. We decided not to focus on a specific disease or condition, but rather concentrate on broad PC‐CER opportunities. The possible positive result of this decision was that all researchers at the University and UPMC felt included and were not threatened by the impartial methodological perspective of the CERC. Potentially negative aspects were the perceived lack of a specific strength and conceivably, the same core funding level targeted at a specific condition could have provided more in‐depth assistance. Now that our PC‐CER portfolio has grown, this is no longer a drawback. However, not having a specific stakeholder community connected to one disease or condition is a continual challenge, which we will address in more detail below.

Organization

The CERC organizational structure is displayed in Figure 1. The main component of this structure is an administrative core that interfaces with existing resources at the University and UPMC. The CERC structure seeks to build synergistically on existing institutional strengths, including other CTSI cores, additional data centers at the University, ongoing clinical and methodological research, complementary educational programs, and medical center collaboration to optimize use of associated resources. None of the associated resources directly report to the CERC. The CERC aims to be the administrative center of a wheel that connects all PC‐CER institutional resource spokes. Figure 1 is not a matrix or hierarchical model, but simply an organizational depiction.

Figure 1.

Organizational Structure of the University of Pittsburgh and UPMC Model for a Patient‐Centered Comparative Effectiveness Research Center.

Another major component of the CERC is a Health Insurance Portability and Accountability Act of 1996 (HIPAA) compliant data center. The Comparative Effectiveness Research Core Data Center (CERCDC) is a University‐wide resource designed to facilitate research using large public health and clinical datasets containing sensitive health information. It offers three general services: data storage, data access/analysis, and consulting. The CERDC recognizes the essential role of large, population‐based data sets in high‐quality PC‐CER. Unfortunately, there are substantial barriers to the conduct of this research, particularly the need for high‐throughput computing and data security procedures adherent to federal standards.

Lesson 2: Establishing a PC‐CER focused data center centralizes data analysis, facilitates new funding opportunities and fosters collaboration between entities.

The CERCDC is composed of hardware, software, and human resources that together provide a powerful, secure analytic, and storage platform to conduct health services research compliant with state and federal security regulations. The computing infrastructure consists of multiple servers dedicated exclusively to the analysis and storage of large clinical and administrative datasets. The CERCDC contains both anonymized and nonanonymized data, depending on the rules and regulations of the data owner (e.g., the Centers for Medicare and Medicaid Services or UPMC). Users access these services via a secure remote link, allowing the researchers to directly manage and analyze sensitive data directly in a secure computing environment, maximizing both operability and security.

The database includes a subset of information from across the health system through UPMC. UPMC extracts the data requested from their databases and is then loaded into a University of Pittsburgh “UPMC” database, which is then queryable along with all the other sites. Users could access UPMC data without going through the CERC, as UPMC has ultimate control over the data. However, CTSI and the CERC serve as intermediaries between researchers and UPMC in order to facilitate access.

The CERCDC is administered by a faculty director, a project manager and a systems engineer. All CERCDC activities, such as prioritization and approval processes, are overseen by an advisory committee with representatives from each of the six Schools of the Health Sciences at the University of Pittsburgh.

Several considerations were central to the decision to initiate the CERCDC. First, in terms of data security and data use agreements, government agencies such as Medicare consider all University‐affiliated researchers to belong to a single pool, requiring that investigators abide by similar data security standards. Keeping all of the data under one roof ensures that individual researchers are not at risk due to the potential negligence of others. Second, the large costs of storing and analyzing these data sets can be prohibitive for junior investigators who are interested in beginning a public health research career, or senior investigators looking to take their research in a new direction. By creating a shared resource, the CERCDC distributes those costs over a large number of researchers. Third, CERDC staff develops familiarity with multiple datasets, which is helpful to new PC‐CER researchers with limited experience working with large datasets. The shared experience with these data reduces the start‐up costs involved, facilitating new avenues of research for University investigators.

With the success of the CERCDC, it became a University cost center in 2013. Cost centers are self‐supporting business enterprise activities that charge other University departments for services. The total annual budget for the CERCDC is approximately $225,000. The charges for individual studies range from approximately $5,000 to $25,000 per year, depending on the number of users accessing the data, the level of computing power needed, and the amount of data storage needed. For junior investigators without funding seeking access to the CERCDC, low‐cost options are available and the charges are typically borne out of Departmental budgets.

Leadership and staffing

Other key aspects of the CERC include a Director and senior faculty with ties to national activities. Having CERC leadership with external connections is invaluable. The participation of these faculty in national efforts, such as PCORI review panels and workshops; Institute of Medicine workshops and committees; and funding agency conferences; raises the national visibility of the CERC, and also allows the CERC to learn from others in the field to stay at the cutting edge of PC‐CER methodological and policy issues.

Lesson 3: Core members should volunteer for national PC‐CER activities for education and visibility.

Internally, we found value in senior faculty having ties to executive‐level activities at the University and UPMC. This characteristic allows the CERC representation at decision‐making levels within the University. The resulting two‐way communication guarantees that the University and UPMC leadership were continually informed of the CERC's progress.

Necessary personnel include both staff and supporting faculty. With initial funding, the CERC hired a fulltime project manager who handles CERC day‐to‐day activities. This includes monitoring PC‐CER funding opportunities, organizing workshops and presentations, scheduling individual researcher meetings, drafting communication and educational material, facilitating collaborations, assisting with proposal reviews, and monitoring the budget. With centralized administration, we are responsive and organized, which is key to our success.

CERC activities

The following aims describe the specific activities of the CERC.

Aim 1. Support high‐quality PC‐CER across the University through infrastructure support; training; collaborations; and strategic coordination of responses to funding opportunities.

Aim 2. Promote the University's PC‐CER externally to increase funding opportunities.

Aim 3. Develop new statistical and methodological approaches to advance the science of PC‐CER.

Aim 4. Expand the pool of researchers trained in PC‐CER.

Aim 5. Demonstrate the translation of PC‐CER via dissemination and implementation into actions that effectively reach the patients, thus directly impacting clinical care.

To accomplish these aims, the CERC conducts a number of activities to reach the 2,993 potential investigators across the Schools of the Health Sciences and UPMC. We employ an open door policy by assisting anyone related to the University and UPMC during workshops and individual meetings. Since 2011, the CERC conducted 54 training workshops on PC‐CER funding opportunities and review criteria, PC‐CER methodology and stakeholder engagement. The CERC Director also presented seminars to a number of different departments and schools across the University. Those presentations were tailored to the discipline of the department and provided a tutorial on PC‐CER, funding opportunities, and CERC services. They allowed the CERC to identify interested researchers and increase the research network. See Table 1 for specific presentation titles.

Table 1.

List of comparative effectiveness research Core (CERC) workshop and seminar titles.

CERC Workshop and Seminar Titles Listed Alphabetically

A Successful PCORI Application: An Insider's Perspective on Methods Comparative Effectiveness Research

Comparative Effectiveness Research Core (CERC)

Comparative Effectiveness Research Core Stakeholder Engagement Forum

Comparative Effectiveness Research (CER), Patient‐Centered Outcomes Research (PCOR), Funding (PCORI), and all that Jazz From a Statistician's Perspective

Comparative Effectiveness Research and Precision Medicine: Promises and Challenges

Comparative Effectiveness and Patient‐Centered Outcomes Research

Decoding the PCORI Review Process

Incorporating Stakeholder Engagement into PCORI Applications, Including the Grant Development Phase

May 2012 PCORI Proposal Preparation Seminar

Methodology Reports for Patient‐Centered Outcomes Research

Opportunities at the Patient‐Centered Outcomes Research Institute (PCORI)

Patient‐Centered Outcomes Research Institute (PCORI) Funding

Patient‐Centered Outcomes Research Institute (PCORI) Grant Reviews and New Funding Opportunities Workshop

Patient‐Centered Outcomes Research Institute (PCORI) Informational Session

Patient‐Centered Outcomes Research Institute (PCORI): Targeted Funding Opportunities

Qualitative Methods and Stakeholder Engagement in PCORI Proposals

Stakeholder Engagement and Involvement

The Power and Pitfalls of Evidence‐Based Medicine…and the Promise of Comparative Effectiveness Research

Update on the Comparative Effectiveness Research Core (CERC)

Lesson 4: Workshops are not just educational, but also interactive, which facilitates project collaborations.

The CERC also maintains an e‐mail distribution list that includes over 300 investigators at the University, UPMC, and affiliates. The distribution list is used to communicate timely and concise notices of PC‐CER funding, CERC workshops, and other PC‐CER educational opportunities about once every 2 weeks.

Lesson 5: Meaningful, concise, and timely information sharing keeps investigators engaged and cognizant of funding opportunities.

Members of the CERC also engage in teaching PC‐CER‐focused topics through collaboration with the Institute for Clinical Research Education (ICRE). The ICRE is the home for the University of Pittsburgh's premier clinical and translational research training programs.8 Several members of the CERC coteach courses on an introduction to CER and CER study design. The ICRE offers both a track and certificate in CER, both of which are directed by a member of the CERC. The CERC also collaborates with the ICRE on developing training grant applications and other educational efforts.

In addition to group‐based educational activities previously mentioned, the CERC individually met with around 150 investigators since 2011 free of charge. Investigators from all of the Schools of the Health Sciences and UPMC used our services, with the majority from the School of Medicine. These meetings typically focus on discussing new project proposals, guiding investigators toward more PC‐CER‐focused objectives and improving the associated methodological approaches. Whenever possible, investigators are referred to additional resources available throughout the University, UPMC and community. This resulted in 96 proposal submissions and 23 funded projects.

The CERC provides formal reviews of project proposals. A biostatistician with PCORI review experience leads the proposal review. To best assist investigators in revising their funding proposals, we developed a template that specifically focuses on the key components of PC‐CER. Review questions range from the general concept, “Does the research plan meet the definition of PC‐CER?” to more detailed questions, such as, “Does the application actively involve stakeholders in terms of both developing the research plan and in terms of executing the subsequent approach?” and “Is there a clear research plan with sufficiently rigorous methods? Do those methods meet the standards in the PCORI Methodology Report?” The CERC then provides a letter of support to highlight the available CERC and institutional resources.

Lesson 6: Providing researchers with stakeholder resources and an engagement roadmap encourages stakeholder participation at the earliest project phases.

The CERC developed a University of Pittsburgh and UPMC Stakeholder Engagement Resource Guide. The CERC compiled relevant resources that serve as practical guidelines and helpful examples of effective stakeholder engagement. Though many resources are unique to the University and UPMC, it is hopefully helpful to other entities. The main contents include: stakeholder definitions and requirements by funding agency; stakeholder involvement methods, documentation of stakeholders in proposals and budgets, stakeholder Institutional Review Board (IRB) requirements, and University of Pittsburgh and UPMC‐specific resources. Example stakeholder biosketches, letters of support, letters of agreement, stipend policies and budget justifications are included in the appendices. The guide is available to all researchers on the CERC webpage: (http://www.publichealth.pitt.edu/Portals/0/BIOSTAT/StakeholderEngagementGuide.pdf).

Primary Barriers

Stakeholders

The largest challenge for researchers in learning to conduct PC‐CER, particularly in response to PCORI opportunities, is how to identify and collaborate with stakeholders. We partially overcame this challenge by constructing the Stakeholder Engagement Resource Guide, mentioned above. We are fortunate that internal University and UPMC researchers are interested in the issue of stakeholder engagement and are available for consultation. We also leveraged local resources for contacting patients for research. For example, we directed researchers to the CTSI Research Participant Registry to identify, contact, and engage stakeholders. The Research Participant Registry is a voluntary database of individuals willing to consider participation in research studies.9 Successful University and UPMC PCORI proposals serve as examples of stakeholder engagement best practices in theory and content. If the CERC was created to focus on PC‐CER in a specific disease or condition, we most likely would have initially been more organized and successful on the stakeholder front.

Methodology

A major challenge researchers have is understanding and implementing pragmatic trial, quasi‐experimental, and observational study designs and analyses. For example, many are familiar with experimental studies with randomization at the patient level, but not with cluster randomized trials where randomization occurs at the clinic level. Many researchers are unfamiliar with observational data sources, such as registries, electronic medical records, or administrative data. Researchers may not be familiar with methods for causal inference in observational data, such as instrumental variables or propensity scores. The CERC continues to work on training researchers and engaging methodologists in PC‐CER through workshops and individual meetings to strengthen our collaborative capacity.

Reputation and validity

PC‐CER is a new type of research and its major funder, PCORI, is new. We have continual challenges in making clear the legitimacy of this research and the funding. For example, faculty may be fixated on National Institutes of Health (NIH) funding because of its importance to faculty development and promotion. University contracting officers must become familiar with the PCORI contracting process. IRB issues, such as stakeholder training, need to be addressed. Grants staff should become familiar with the PCORI submission process. Over time, and with a concerted effort at advertising and communication, we have overcome some of these legitimacy and implementation barriers.

Conclusion

PC‐CER components are integral to many recent funding opportunities. The University of Pittsburgh and UPMC established the CERC by building off of existing resources and creating new ones, such as the CERCDC, to further PC‐CER institutional goals. Barriers continue to be identified and addressed at an institutional level. Establishment of a dedicated research and data center has fostered a community of researchers across our schools and health system. The resulting funding to‐date, including multiple PCORI and AHRQ awards totaling over $40 million, supports this approach.