A National Strategy to Develop Pragmatic Clinical Trials Infrastructure

Abstract

An important challenge in comparative effectiveness research is the lack of infrastructure to support pragmatic clinical trials, which compare interventions in usual practice settings and subjects. These trials present challenges that differ from those of classical efficacy trials, which are conducted under ideal circumstances, in patients selected for their suitability, and with highly controlled protocols. In 2012, we launched a 1‐year learning network to identify high‐priority pragmatic clinical trials and to deploy research infrastructure through the NIH Clinical and Translational Science Awards Consortium that could be used to launch and sustain them. The network and infrastructure were initiated as a learning ground and shared resource for investigators and communities interested in developing pragmatic clinical trials. We followed a three‐stage process of developing the network, prioritizing proposed trials, and implementing learning exercises that culminated in a 1‐day network meeting at the end of the year. The year‐long project resulted in five recommendations related to developing the network, enhancing community engagement, addressing regulatory challenges, advancing information technology, and developing research methods. The recommendations can be implemented within 24 months and are designed to lead toward a sustained national infrastructure for pragmatic trials.

Background

An important methodological focus in comparative effectiveness research (CER) is the advancement of pragmatic clinical trials (PCTs),1 which overlap conceptually with practical trials,2 large simple trials,3 and effectiveness trials. The approach strives to generate results that are relevant to patients and clinicians by comparing interventions in participants and settings representative of usual care. Pragmatic trials and trial infrastructure have been supported through initiatives at NIH,4, 5 the Centers for Medicare & Medicaid Services,6 the Agency for Healthcare Research and Quality (AHRQ),7 the Patient‐Centered Outcomes Research Institute (PCORI),8 Practice‐Based Research Networks (PBRNs),9 and community‐based participatory research initiatives across the Federal government.10

A challenge for these initiatives has been the lack of sustained research infrastructure to support them. For this reason, the NIH Clinical and Translational Science Awards (CTSA) Consortium launched an effort in 2012 to advance such an infrastructure. The proposed PCT infrastructure (PCTi) would attempt to align CTSA and other research institutions, their partners, and the NIH National Center for Advancing Translational Sciences on a common goal of developing and deploying an infrastructure to support high‐priority PCTs. Such an infrastructure might also serve as a learning ground and shared resource for investigators and communities interested in developing PCTs.

Approach

We undertook a three‐stage process to build infrastructure that could support PCTs within the CTSA Consortium: developing a researcher network; prioritizing proposed trials; and implementing learning exercises that would culminate in a 1‐day workshop.

Developing the network

We held a 1‐day meeting to launch a national network of thought leaders on infrastructure for PCTs. A 32‐member planning committee (Table 1) drew representatives from 17 CTSAs and other organizations with expertise in stakeholder and community engagement (Supplement: Community and Stakeholder Engagement Experience of Coauthors), quantitative and qualitative research methods, informatics, CER, dissemination and implementation research, and a range of clinical and health systems topics relevant to developing a strategic plan for development of PCT research infrastructure. The planning committee began by reviewing existing definitions of PCTs.11, 12 These definitions place PCTs on a continuum with explanatory trials but with a focus on comparisons of clinical, community and health systems interventions. This led to the following definitions:

• A pragmatic clinical trial is a prospective comparison of a community‐, clinical‐, or system‐level intervention and a relevant comparator in participants who are similar to those affected by the condition(s) under study and in settings that are similar to those in which the condition is typically treated.

• A pragmatic clinical trial infrastructure includes the resources, systems, and processes needed to prioritize, conduct, and use the results of PCTs.

Table 1.

PCTi workshop planning committee

Name	Affiliation
Sergio Aguilar‐Gaxiola	University of California, Davis, California
Syed Ahmed	Medical College of Wisconsin, Milwaukee, Wisconsin
Arthur Blank	Albert Einstein College of Medicine, New York, New York
Kathleen Brady	Medical University of South Carolina, Charleston, South Carolina
Tim Carey	The University of North Carolina, Chapel Hill, North Carolina
Thomas Concannon	Tufts Clinical and Translational Science Institute, Tufts University, Boston, Massachusetts
Linda Cottler	University of Florida, Gainesville, Florida
Rowena Dolor	Duke University, Durham, North Carolina
Milton Eder	University of Chicago, Chicago, Illinois
Alecia Fair	Clinical and Translational Science Awards Consortium Coordinating Center, Vanderbilt University, Nashville, Tennessee
Carol Ferrans	University of Illinois, Chicago, Illinois
Rosemarie Filart	National Center for Advancing Translational Sciences, NIH, Bethesda, Maryland
Dan Ford	Johns Hopkins University, Baltimore, Maryland
Mark Helfand	Oregon Health & Science University, Portland, Oregon
William Hersh	Oregon Health & Science University, Portland, Oregon
Larry Kessler	University of Washington, Seattle, Washington
Jerry Krishnan	University of Illinois, Chicago, Illinois
Colleen Lawrence	Clinical and Translational Science Awards Consortium Coordinating Center, Vanderbilt University, Nashville, Tennessee
Paul Meissner	Albert Einstein College of Medicine, New York, New York
Lloyd Michener	Duke University, Durham, North Carolina
Peter Neumann	Tufts University, Boston, Massachusetts
Iris Obrams	National Center for Advancing Translational Science, NIH, Bethesda, Maryland
Wilson Pace	University of Colorado, Denver, Colorado
Harold Pincus	Columbia University, New York, New York
Al Richmond	The University of North Carolina, Chapel Hill, North Carolina
Gary Rosenthal	University of Iowa, Iowa City, Iowa
Joel Saltz	Emory University, Atlanta, Georgia
Andrea Sawczuk	National Center for Advancing Translational Science, NIH, Bethesda, Maryland
Harry Selker	Tufts University, Boston, Massachusetts
Jodi Segal	Johns Hopkins University, Baltimore, Maryland
Jonathan Tobin	The Rockefeller University, New York, New York
Sean Tunis	Center for Medical Technology Policy, Baltimore, Maryland

Three goals that emerged were to: (1) generate a set of recommendations for building a PCTi; (2) catalyze PCT research projects within CTSA‐affiliated research consortia; and (3) evaluate the performance of this initiative through an iterative learning network approach. Thereby, the intent was to learn by doing; to learn how to build infrastructure from the experience of launching actual trials.

Prioritizing proposed trials

To accomplish these goals, the planning committee solicited PCT proposals to serve as illustrative “use cases” for a learning network of CTSA members and partners. A solicitation was issued to members of the CTSA community in July 2012; 19 proposals were submitted by investigators and reviewed by a subgroup of the planning committee by early September. The proposals were reviewed on eight primary and five secondary criteria (Figure 1). Primary review criteria were considered “must‐have” features of each use case. Secondary criteria were considered “desired” features. Five illustrative use cases were selected (Table 2), representing a range of interventions in inpatient and outpatient settings, varied study designs, and both pediatric and adult populations. Summaries of the use cases are presented below and more detailed descriptions can be found on the CTSA Consortium Web site.13

Figure 1.

Proposal review criteria.

Table 2.

Use case investigators and proposed pragmatic clinical trials

Principal investigator & affiliation	Study design	Population	Intervention vs. comparator	Outcomes	Setting
Nicotine replacement for hospitalized smokers (NICHOLS) study
Jerry Krishnan, University of Illinois Chicago	Clustered RCT	Adult	Counseling vs. counseling + NRT	Biochemically confirmed 24‐week abstinence	Inpatient
The case of translating and implementing evidence‐based childhood asthma interventions in federally qualified health centers
Anne Markus, George Washington University	Matched cohort	Pediatric	Primary care‐based counseling vs. home‐based environmental remediation vs. counseling + remediation	Reduced symptom days, inappropriate utilization, and days absent from school.	Outpatient
Community‐acquired methicillin‐resistant staphylococcus aureus bacteria treatment and transmission prevention
Jonathan Tobin, Rockefeller University	RCT	Adult	Usual care vs. usual care + decolonization in the home	Clinical response + healthcare utilization	Outpatient–PBRNs
Chronic pain self‐management
Hayden Bosworth, Duke University	Clustered RCT	Adult	Decision support with and without nurse telephone support	Total dose of narcotics prescribed	Outpatient
Multimodal intervention to reduce the risk for major adverse cardiovascular events
Karl Hammermeister, University of Colorado Denver	RCT	Adult	Patient education by PharmD	Improved guideline concordant care	Outpatient

RCT = randomized clinical trial; NRT = nicotine replacement therapy; PBRN= practice‐based research network; PharmD = doctor of pharmacy.

The nicotine replacement for hospitalized smokers (NICHOLS) study

The objectives of the proposed research are to compare the effectiveness and cost‐effectiveness of counseling plus nicotine replacement therapy (NRT) versus counseling alone in hospitalized smokers. The investigators propose a cluster‐randomized design, using hospitals (N = 50) as the unit of randomization. The investigators hypothesize that counseling plus NRT will result in higher abstinence rates, better health status, and lower rates of acute care utilization post discharge than will counseling alone among hospitalized smokers. Counseling is to be delivered by an experienced, guideline‐based telephone counseling service at the point of care in hospitals and in the outpatient setting. Medications will be provided during the hospitalization and then prescribed for use after hospital discharge for 12 weeks; follow‐up would continue for an additional 12 weeks. To mimic clinical practice, medications after discharge will not be funded by the study and will be administered by clinicians, not research staff. Biochemically verified 7‐day abstinence from smoking at 24 weeks will be the primary endpoint, assessed by intention‐to‐treat. Secondary outcomes will include patient‐reported health status, acute care utilization over 24 weeks, total and mean costs, and incremental cost‐effectiveness ratios from the health system, insurer, and societal perspectives. Subgroup analyses will examine effects by race, gender, age, reason for hospital admission, and income.

The case of translating and implementing evidence‐based childhood asthma interventions in federally qualified health centers (FQHCs)

The proposed study is designed to: (1) evaluate the decisions that FQHC leaders make to deploy and sustain evidence‐based childhood asthma interventions; (2) evaluate the effectiveness of interventions once integrated into FQHC routine practice; and (3) communicate and disseminate findings to other FQHCs across the United States. The investigators plan to select three “intervention” and three matched (on organizational characteristics and pediatric populations) “comparison” FQHCs that jointly represent a variation in capacity, experience with asthma, resources in the community, and local policy. The comparison interventions will consist of: (1) asthma counseling tailored to the environmental triggers to which a child is sensitive, an intervention used by the National Cooperative Inner City Asthma Study; (2) home‐based environmental remediation, an intervention used by the Inner City Asthma Study; and (3) a “hybrid” counseling and remediation intervention. All three were previously found to be efficacious in reducing symptom days, inappropriate utilization of services, and days absent from school. However, previous studies did not engage primary care practices as a matter of protocol, thereby leaving out an important element in understanding how the interventions may be implemented in real‐world clinical settings.

Community‐acquired methicillin‐resistant Staphylococcus aureus bacteria (CA‐MRSA) treatment and transmission prevention

The purpose of this proposed research is to compare clinical and community‐based interventions among patients presenting with skin and soft tissue infections (SSTIs) and diagnosed with CA‐MRSA who are seen in primary care settings. The interventions to be compared are: (1) standard CDC guideline‐directed care, including incision, drainage, and oral antibiotics (usual care), and (2) usual care with decolonization interventions conducted in the home setting, including hygiene education, twice daily intranasal mupirocin and daily chlorhexidine body washes (usual plus experimental care). Patients will be enrolled through outreach to PBRNs, their community health centers, primary care practices, and primary care clinicians. Follow‐up at 3, 6, and 12 months after baseline will measure short‐ and long‐term clinical response, and identify primary care, emergency department, inpatient, laboratory, and pharmacy utilization for SSTIs.

Chronic pain self‐management

The proposed research is designed to compare promotion of lifestyle guidelines that have been shown to reduce pain, including breathing relaxation and distraction, as well as to improve patient–provider communication, patient social support, and medication adherence. The investigators propose a 2‐year, cluster‐randomized controlled trial with interventions at both the provider and patient levels. Primary care providers will be randomized to have access to a decision support interface or usual care without the decision support interface. Within each participating provider's primary care panel, patients with chronic pain will be randomized to receive either a nurse telephone intervention focused on self‐management, or usual primary care. The patient will be the unit of analysis, and the primary outcome will be the total dose of narcotic medications prescribed over the study period. The investigators hypothesize that narcotic medication prescriptions will be lower for patients whose providers have access to the decision support interface and who receive the nurse telephone intervention.

Multimodal intervention to reduce the risk for major adverse cardiovascular events: enhancing blood pressure and cholesterol guideline concordance

The purpose of the proposed research is to improve prescription guideline concordance in patients at risk for major adverse cardiovascular events. The Investigator proposes to conduct a randomized pragmatic trial of intervention and usual care clinics caring for approximately 2,800 nonconcordant patients. The intervention is to consist of monthly PharmD‐led meetings with enrolled patients, presentation of an educational curriculum, and evaluation of patient‐specific medication issues including side effects, adherence, and guideline concordance. At the conclusion of each session, the PharmD will either modify prescriptions or inform the primary care provider of proposed modifications to prescriptions, to achieve guideline concordance. Investigators will meet monthly with clinic directors and medical directors to evaluate clinic‐ and study‐level concordance data. Outcomes of interest include: patient‐, care provider‐, and practice‐level factors associated with successful implementation of the intervention; sustainability of the intervention; and the incremental costs of intervention.

Implementing learning exercises

The planning committee worked for 2 months with principal investigators of the five selected proposals and developed “use cases” that were presented at a 1‐day, in‐person learning network meeting in November 2012. We paired investigators with a member of the planning committee to address 13 questions (Figure 2) that explore the infrastructure strengths and needs of individual PCTs. Investigators redeveloped their proposals into “use cases” that were distributed to individuals who had registered online for the 1‐day learning network meeting. The meeting was attended by 97 individuals representing 40 CTSA institutions, the NIH, Clinical Research Organizations, community leaders, the Institute of Medicine (IOM), and stakeholder organizations. The use cases were presented in early sessions.

Figure 2.

Use case review questions.

Most of the meeting was devoted to studying the cases' infrastructure capacities and needs. Participants and investigators discussed strategies to develop and sustain the proposed research programs, following a semistructured format addressing a range of infrastructure domains: (1) dissemination and implementation research; (2) informatics and information technology; (3) CER methods; (4) community and stakeholder engagement; and (5) evaluation of research.

The discussions yielded practical solutions to support the launch and conduct of the proposed studies. Each workshop participant initially stayed with the discussion surrounding one of the five use cases, and later participated in a plenary meeting headed by small roundtables of experts representing the five infrastructure domains. The individual challenges and solutions that applied to one use case were compared and synthesized with those of the others. Finally, the recommendations were vetted and redrafted in an iterative process over 3 months of discussion with our planning committee's infrastructure experts.

Recommendations

Five recommendations grew out of this process (Table 3): develop a sustained PCT learning network, establish standing infrastructure for community and stakeholder engagement, ease regulatory challenges, advance two domains in information technology, and improve research methods to study heterogeneity in PCTs. The recommendations and brief discussions are:

Table 3.

Recommendations for infrastructure to support pragmatic clinical trials

Domain	Description
1. Learning network	• Partner with NIH ICs, AHRQ, and PCORI to coordinate PCTi infrastructure components and test with use cases • Develop and document PCTs • Share investigator learning opportunities
2. Community and stakeholder engagement	• Establish a standing structure for routinely engaging communities, practices, and stakeholders in trial development, implementation, and dissemination activities
3. Regulatory challenges	• Modify IRB process to support joint approvals for effectiveness studies through a central or deemed IRB • Develop strategies to streamline multiinstitutional contracting process
4. Information technology	• Implement secure, standards‐based, interoperable information systems across sites/institutions • Develop a comprehensive dictionary of data elements across platforms
5. Research methods	• Focus methods work on study design and analytical approaches that help measure and interpret treatment, site, and patient heterogeneity

NIH IC = Institutes and Centers of the National Institutes of Health; AHRQ = Agency for Healthcare Research & Quality; PCORI = Patient‐Centered Outcomes Research Institute; PCT = pragmatic clinical trial; IRB = Institutional Review Board.

Develop a sustained PCT learning network

The role of CTSAs is not to conduct trials, but rather to support their conduct by others, by facilitating efficient development and implementation. In this role, CTSAs and the Consortium should work with the NIH Institutes and Centers, the NIH Collaboratory initiative, NIH Common Fund initiatives, AHRQ, and PCORI to stimulate and share process learning through the conduct of high‐priority trials. The CTSA‐supported activities could include consultation with trial investigators, documentation of trials, and development of investigator learning opportunities. This activity would require the redirection of resources within CTSAs, could leverage other trial infrastructure, and might depend on new resources from Federal and other funders.

Establish standing infrastructure to engage with communities and stakeholders

Community14 and stakeholder engagement15 are valued functions of individual CTSAs and the Consortium. Investigators planning PCTs need community and stakeholder (patients, caregivers, providers, funders, employers, policy makers, product makers, etc.) input and advice regarding issues of study topic and design, acceptable methods for participant recruitment and participation, appropriate choice of study outcomes, and methods for dissemination of results. Assessing research priorities and obtaining feedback on research populations and the most important study outcomes currently is conducted project by project, often taking 6 months or more. Yet, this input is needed in advance of research planning, and investigators are often confronted by short proposal timelines of 6 to 8 weeks, which makes an authentic process of this type very challenging. Because engagement of communities and stakeholders is relatively new, trialists are still learning how to make this process efficient and effective. While some activities may need to engage communities and other stakeholder de novo, use of standing stakeholder groups could be a more efficient alternative. These structures may help to establish a culture of engagement among researchers, long‐term bidirectional relationships between researchers and stakeholders, and timely advising on research. The downsides of such an approach may include high time and resource costs, increased stakeholder fatigue,16 inability to include members representing every condition and disease group of interest, and a risk of paradoxically creating the perception by researchers that standing structures alone can provide sufficient support for research proposals and projects.

An area to be explored is whether such functions could, or should, be shared across several CTSAs, such as engaging with condition‐focused advocacy organizations and stakeholders for prevalent conditions. For example, the COPD Outcomes‐Based Network for Clinical Effectiveness & Research Translation (CONCERT) has developed a national stakeholder consortium for collaborative activities to support CER in COPD, including patient advocacy groups, professional organizations representing providers, research organizations, and others.17 Representatives of national stakeholder groups have contributed to CONCERT prioritization activities, preparation and submission of applications, recruitment into studies, and technical assistance to stakeholder groups seeking COPD CER experts. The availability of a “standing” stakeholder consortium has proved to be efficient for CONCERT investigators and stakeholders, and could serve as a paradigm for research in other conditions. These experiences are consistent with the recent IOM recommendation that CTSAs “build partnerships” with communities and other stakeholders to support engagement “across the research spectrum.”18

This activity would require resource support and substantial coordination from CTSAs, institutional review boards, and oversight agencies.

Ease regulatory challenges

Pragmatic trials that involve interventions applied with minimal research oversight in heterogeneous sites may be faced with multiple and expensive approvals for review of informed consent policies, without appreciable improvement in subject safety.19 Some proposed modifications to the “Common Rule” for review of informed consent seek to address this challenge,20, 21, 22, 23 and research organizations should be active in these discussions. Through their regulatory and governance processes, CTSAs are currently working on efforts to develop centralized IRBs or to “deem” an IRB for multisite trials. While such activities are possible under current human subject regulations, they are not yet widespread.24 In addition, to reduce burdens associated with contracting across institutions, CTSAs should identify and share policies and tools that improve contracting efficiency. These activities would require significant new coordination and resources.

Develop standard data formats for common conditions and implement secure, standards‐based, interoperable information systems

Researchers need consistent ways of describing populations, conditions, interventions, and outcomes in electronic data. They also need tools for data generation, capture, and management, including infrastructure to catalog interventions and their implementation and methods for efficient data collection. A robust, standards‐based informatics infrastructure has the potential to reduce PCT costs and improve PCT quality. While a long‐term goal for such an infrastructure has been articulated by the CTSA Consortium,1 PCORI25 and the IOM,26 a shorter‐term goal should include standardizing on key variables across conditions and insuring that such standards are used in electronic health record (EHR) platforms. Tools for using operational EHR data should be identified, standardized, and made available to all CTSA Consortium members and disseminated more broadly. For operational clinical data, attention must be paid to the limitations of using such data while adhering to emerging best practices for their use.27 The Consortium should also provide targeted consultations to investigators as research is developed. Such a learning environment, if documented and disseminated, would benefit multiple members of the consortium, and it could be the basis for sharing resource‐saving information system innovations across institutions. These activities would require substantial redirection or new allocation of resources, and/or collaboration with ongoing related efforts.

Improve methods to study treatment, site, and patient heterogeneity in PCTs

By their nature, pragmatic trials enroll participants and sites with relatively open inclusion and fewer exclusion criteria than the typical efficacy trial, leading to a heterogeneous sample. In addition, some types of PCT allow modifications to the intervention protocol, leading to heterogeneity of the intervention. Unless addressed in the study design, these issues could result in significant challenges for interpretation of PCTs. The CTSA Consortium should coordinate among its many committees and should engage with the PCORI Methods Committee,24 AHRQ Effective Healthcare Program,28, 29 and others to establish common standards for the study of treatment, site, and participant heterogeneity. These activities would require redirection of resources from CTSAs and new allocations from Federal and other funders.

Discussion

There is a growing national interest in research that reflects usual care and usual care settings. The rationale is that the results of research should be relevant to patients and clinicians, and they should be useable in everyday practice. Real‐world settings typically have greater complexity and heterogeneity than traditional trial environments. To improve generalizability, these trials depend on larger enrollments and more diverse research participants. The infrastructure needed to conduct these studies will depend on coordinated and material support from funders, research institutions, and individual investigators.

We recommend practical infrastructure changes that can be deployed to assist investigators, stakeholders, and communities in the development and launch of PCTs within 24 months. Although these recommendations were developed for CTSAs and the CTSA Consortium, they could be adapted by any public or private entity with an interest in PCTs and their infrastructure. Some proposed changes could be accomplished with existing CTSA infrastructure, requiring coordination of existing personnel and resources. Other changes, such as an infrastructure for a sustained learning network, may require redirection of existing resources or new resources. New information technology innovation will require sustained attention and resources. New methods for interpreting treatment, site, and patient heterogeneity will also require redirected or new resources.

We launched this effort in the CTSA Consortium on the assumption that efficient infrastructure changes could be developed, evaluated, and validated through case‐based learning, and we focused our attention on immediate needs for the next 2 years. However, we envision a national infrastructure that can support and sustain PCTs well into the future. The return on this investment of attention, coordination, and resources should be to the benefit of the healthcare system and the public's health.

Conflict of Interest

The authors have no competing interests that bear on the content of this manuscript.

Author Contributions

Drs. Concannon and Carey took primary responsibility for conceiving and writing the manuscript, obtaining contributions from coauthors and managing reviews and government clearances. All coauthors contributed to writing the manuscript.