Pragmatic Clinical Trials Offer Unique Opportunities for Disseminating, Implementing, and Sustaining Evidence-Based Practices Into Clinical Care: Proceedings of a Workshop

Abstract

The National Institutes of Health (NIH) Health Care Systems (HCS) Research Collaboratory hosted a workshop to explore challenges and strategies for the dissemination, implementation, and sustainability of findings from pragmatic clinical trials (PCTs) embedded in HCS. PCTs are designed to assess the impact of interventions delivered in usual or real-world conditions and leverage existing infrastructure to answer important clinical questions. The goal of the workshop was to discuss strategies for conducting impactful future PCTs that bridge the gap between evidence, practice, and policy. This paper summarizes presentations about how to design and conduct PCTs embedded in HCS and use dissemination and implementation strategies during the planning and conduct of projects, emphasizing the ever-changing world of care delivery and the need for pragmatic trial operations to adapt at various levels of operation.

Background

The National Institutes of Health (NIH) Health Care Systems (HCS) Research Collaboratory, initiated in 2012 by the NIH Common Fund, launched a set of pragmatic clinical trials (PCTs) in partnership with various types of US HCS (Table 1). These trials have served as case examples of many issues related to pragmatic research, including strategies for designing trials in partnership with HCS that included planning for dissemination and implementation (D&I) and sustainability of interventions based on research findings. On May 24, 2017, NIH hosted a workshop to explore challenges and strategies for D&I and sustainability of findings from PCTs embedded in HCS. PCTs are designed to assess the impact of interventions delivered in usual or real-world conditions and leverage existing infrastructure to answer important clinical questions. The goal of the workshop was to discuss strategies for researchers, clinical partners and HCS leaders conducting future pragmatic trials that can demonstrably improve patient and population health and readily bridge the gap between evidence, practice, and policy. The video can be viewed on the NIH website.¹

Table 1:

NIH Collaboratory Demonstration Project Pragmatic Clinical Trials

Project Title & Project Leaders	Project Goals	Health Care Systems
Strategies and Opportunities to Stop Colon Cancer in Priority Populations (STOP CRC) Gloria Coronado, PhD, Beverly Green, MD, MPH, Kaiser Permanente Northwest	Improve the rates of colorectal cancer screening by mailing fecal immunochemical tests to patients.	26 Federally Qualified Health Centers in 3 US states (Oregon, California, and Washington)
Pain Program for Active Coping & Training (PPACT) Lynn DeBar, PhD, MPH Kaiser Permanente Washington	Help patients adopt self-management skills for chronic pain, limit use of opioid medications, and identify factors amenable to treatment in the primary care setting.	Primary care clinics affiliated with 3 US-based HCS (Kaiser Permanente Georgia, Kaiser Permanente Northwest, Kaiser Permanente Hawaii)
Time to Reduce Mortality in End-Stage Renal Disease (TiME) Laura Dember, MD University of Pennsylvania	Determine whether increasing the duration of hemodialysis sessions reduces mortality and hospitalization rates for patients receiving maintenance hemodialysis care.	266 outpatient dialysis units owned by two US-based dialysis provider organizations (DaVita, Inc.; Fresenius Medical Care – North America)
Active Bathing To Eliminate Infection (ABATE Infection) Trial Susan Huang, MD, MPH University of California Irvine School of Medicine	Determine if using antiseptic bathing for hospitalized patients, plus nasal ointments for patients harboring methicillin-resistant Staphylococcus aureus (MRSA), reduces multidrug-resistant organisms and bbloodstream infections.	53 Hospital Corporation of America hospitals in the US, including all inpatient medical, surgical, step-down, and oncology units
Lumbar Imaging with Reporting of Epidemiology (LIRE) Trial Jeffrey Jarvik, MD, MPH University of Washington	Determine if inserting epidemiologic benchmarks into lumbar spine imaging reports reduces subsequent spine-related tests and treatments.	100 clinics in 4 US-based HCS (Kaiser Permanente Northern California, Kaiser Permanente Washington, Mayo Clinic HCS Henry Ford Health System)
Suicide Prevention Outreach Trial (SPOT) Gregory Simon, MD, MPH Kaiser Permanente Washington	Compare outcomes in patients who receive care-management or online skills training for suicide prevention versus usual care.	4 US-based HCS(Kaiser Permanente Washington, HealthPartners, Kaiser Permanente Colorado, Kaiser Permanente Northwest)
Improving Chronic Disease Management With PIECES (ICD-Pieces) Miguel Vazquez, MD University of Texas Southwestern	Improve care for patients with chronic kidney disease, diabetes, and hypertension by using a novel technology platform (PIECES) that uses the EHR to identify patients and assign practice facilitators within primary care practices or community medical homes.	4 distinct US-based HCS (Parkland Health and Hospital System, VA North Texas, Texas Health Resources and ProHealth Physicians)
Trauma Survivors Outcomes and Support (TSOS) Douglas Zatzick, MD University of Washington	Coordinate care and improve outcomes for trauma survivors with post-traumatic stress disorder and comorbidity.	25 US-based Level I trauma centers
Pragmatic Trial of Video Education in Nursing Homes (PROVEN) Vincent Mor, PhD, Susan Mitchell, MD, MPH, Angelo Volandes, MD Brown University	Determine if showing advance care planning videos in nursing homes affects the rates of resident hospitalization.	2 US-based nursing home HCS (Genesis, Pruitt Health) (260 nursing homes)

This review summarizes the workshop discussions, including a keynote address and three moderated panel presentations about how to design and conduct PCTs embedded in HCS to use D&I strategies (Table 2). Each panel session described case examples from the Collaboratory’s PCTs to illustrate how these strategies were addressed in various settings during the planning and implementation phases of research, emphasizing the ever-changing world of care delivery and the need for pragmatic trial operations to adapt at various levels of operation. The workshop also included an interactive role play session, where a researcher pitched a strategy to a panel of HCS leaders to implement an intervention successfully tested in a previous PCT. After the panel presentations, the moderators invited some discussants to share perspectives and reflect on what they learned in the workshop.

Table 2:

NIH Workshop Presenters and Discussants

Josephine P. Briggs, MD

Director, National Center for Complementary and Integrative Health

National Institutes of Health, DHHS

Richard Hodes, MD

Director, National Institute on Aging

National Institutes of Health, DHHS

Catherine M. Meyers, MD

Director, Office of Clinical & Regulatory Affairs

National Center for Complementary and Integrative Health

National Institutes of Health, DHHS

Wendy Weber, ND, PhD, MPH

Branch Chief, Clinical Research Branch

Division of Extramural Research

National Center for Complementary and Integrative Health

National Institutes of Health, DHHS

David Chambers, DPhil

Deputy Director for Implementation Science

Division of Cancer Control and Population Sciences

National Cancer Institute

National Institutes of Health, DHHS

Eric B. Larson, MD, MPH

Vice President, Research and Health Care Innovation

Kaiser Foundation Health Plan of Washington

Executive Director and Senior Investigator

Kaiser Permanente Washington Health Research Institute

Kaiser Permanente Washington

Lesley H. Curtis, PhD

Professor of Medicine

Director, Center for Population Health Sciences, Duke School of Medicine

Director, Center for Pragmatic Health Systems Research, Duke Clinical Research Institute

Amy Kilbourne, PhD, MPH

Director, VA Quality Enhancement Research Initiative (QUERI)

US Department of Veterans Affairs

Professor of Psychiatry

University of Michigan

Susan S. Huang, MD, MPH

Professor, Infectious Diseases

Medical Director, Epidemiology and Infection Prevention

University of California, Irvine School of Medicine

Douglas F. Zatzick, MD

Professor, Psychiatry and Behavioral Sciences

University of Washington School of Medicine

Harborview Medical Center

Gloria Coronado, PhD

Senior Investigator, Mitch Greenlick Endowed Scientist for Health Disparities

Kaiser Permanente Center for Health Research

Kaiser Permanente Northwest

Miguel Vazquez, MD

Clinical Chief Nephrology Division

Professor, Internal Medicine

University of Texas Southwestern Medical Center

John J. Warner, MD

Chief Executive Officer, University Hospitals and Clinics

University of Texas Southwestern Medical Center

Lynn DeBar, PhD, MPH

Senior Investigator

Kaiser Permanente Washington Health Research Institute

Kaiser Permanente Washington

Andrew Bertagnolli, PhD

Vice President, Behavioral Health Clinical Products

Optum United Health Group

Vincent Mor, PhD

Florence Pirce Professor of Community Health

Brown University

Senior Health Scientist, Center on Innovation

Providence Veterans Administration Medical Center

Kevin Daugherty Hook, MSN, MA, MS, CRNP

Vice President, Nursing Practice and Education

Genesis HealthCare

Gregory E. Simon, MD, MPH

Senior Scientific Investigator

Kaiser Permanente Washington Health Research Institute

Kaiser Permanente Washington

Susan Mullaney, MHA

President, Kaiser Permanente Washington

Edward J. Septimus, MD

Vice President, Research and Infectious Diseases Clinical Services Group

Hospital Corporation of America

Matt Hough, MD

Medical Director, Jackson Care Connect

Care Oregon

Laura M. Dember, MD

Professor of Medicine and Epidemiology

Perelman School of Medicine, University of Pennsylvania

Jerry Jarvik, MD, MPH

Professor, University of Washington School of Medicine

Patrick H. Luetmer, MD

Associate Dean for Clinical Systems Oversight, Mayo Clinic

Keynote Address

Dr. David Chambers, National Cancer Institute, offered definitions for key terms that would be discussed during the workshop:

• Implementation science is the study of methods to promote the integration of research findings and evidence into health care policy and practice.²

• Dissemination research studies the distribution of information and intervention materials to public health or clinical practice audiences. The intent is to understand how best to spread and sustain knowledge and the associated evidence-based interventions.

• Implementation research studies the use of strategies to adopt and integrate evidence-based health interventions into clinical and community settings to improve patient outcomes and benefit population health.³

Adoption and implementation of effective interventions into health care settings can be a very slow process with multiple challenges. Research has typically been assumed to follow the drug-development pathway, and the resulting interventions are expected to be automatically implemented in HCS following regulatory approval. The desired end-product of research is usually expected to be a publication in a high-impact journal, with evidence then gradually migrating into databases, guidelines, textbooks, health care, etc.—a notoriously slow path. Researchers can over-focus on rigorous design of their intervention and assume as long as it is efficacious, demand for the intervention will be substantial. In the reality of a health care setting, however, providers must be trained to deliver the intervention and choose to deliver it, and patients who could benefit must be able to receive it. D&I processes have not, in general, received much attention from biomedical scientists, though we know that simply announcing or providing evidence is insufficient to achieve the high degrees of impact and reach desired for full-scale implementation (e.g. Diffusion, dissemination and implementation).⁴

Dr. Chambers offered guidance for improving this situation. Maximizing impact and benefit of an intervention goes beyond considering efficacy and safety to require thinking about the multiple stages of dissemination and implementation. Russ Glasgow and colleagues defined the RE-AIM⁵ evaluation framework toward this end, which describes five elements or dimensions:

• Reach to the target population

• Effectiveness of the intervention

• Adoption by target staff, settings, or institutions

• Implementation in terms of consistency, costs, and adaptions made during delivery

• Maintenance (i.e., sustainability) of intervention effects in individuals and settings over time.

A second framework⁶ by Enola Proctor and colleagues contrasts two types of trials: (1) those focused on what intervention should be delivered and the resulting outcomes (e.g., effectiveness trials), and (2) those focused on a strategy that explains how an intervention can be effectively delivered, within a set of clinical and community practices, to a broad population (e.g., D&I trials). A third framework, the PRECIS-2⁷ tool, takes a middle ground between the two to aid researchers in thinking about designing and testing for D&I earlier in trials, not just at the end stage. The PRECIS-2 toolkit⁸ was developed to help researchers design trials that are pragmatic across nine domains: (1) eligibility - who is selected to participate in the trial; (2) recruitment - how participants are recruited into the trial; (3) setting - where the trial is conducted; (4) organization - what expertise and resources are needed to deliver the intervention; (5) delivery flexibility - how the intervention is delivered; (6) adherence flexibility - what measures are in place to make sure participants adhere to the intervention; (7) follow-up - how closely participants are followed-up; (8) primary outcome - how relevant is it to participants; and (9) primary analysis - to what extent all data all included.

Partnerships between researchers and HCS are a critical element at multiple levels of PCTs so that interventions are designed to fit the needs of a HCS. Today’s science is a “team sport” enabling interventions to be developed with scientific and clinical teams working together to think not only about supply of but demand for an intervention. Consistency of the intervention from development through implementation is often framed as ideal but poses challenges, since contexts and interventions interact, are dynamic, and will inevitably change; this dynamism is at the heart of the learning HCS.⁹ Foreseeing this, research teams can build more adaptive designs and a collection of decision points, and design their research questions not only for the science but also for the questions HCS will ask. Incorporating specific outcomes relevant to implementation (e.g., uptake, sustainability, acceptability, and costs) would likely improve HCS’ abilities to provide effective interventions with equity and timeliness, which could improve health outcomes at the population level.

Finally, given that HCS have concerns about study speed, researchers can consider including in their designs: 1) opportunities to make decisions to adapt or change trials, 2) time to assess the embedding of tools into the electronic health record (EHR) for deciding whether, and how well, the intervention works for both patients and providers, and 3) research questions that are not only scientifically interesting but also fit well with the HCS and the local setting. This integration of D&I concepts into intervention development and testing can hopefully bridge the gap between innovations and their application to drive patient and population health. These and other approaches are described more fully in the NIH Collaboratory’s on-line Living Textbook of Pragmatic Clinical Trials.¹⁰

Panel 1: Setting the Stage for Dissemination and Implementation

Panel 1 speakers described their PCTs (Table 1) and strategies to help prepare early for potential success of D&I. Dr. Susan Huang presented the Active Bathing to Eliminate (ABATE) Infection Trial and discussed how an earlier trial, the REDUCE MRSA Trial (Randomized Evaluation of Decolonization vs. Universal Clearance to Eliminate MRSA), helped her plan for D&I for ABATE.^11,12 When the results of the first trial were published, the study team received numerous requests for more information, including details about how to implement the protocol. The authors created a free, downloadable toolkit that described the protocol. Many of these materials were adapted later for the ABATE Infection Trial. However, the differences between settings (intensive care units vs general medical and surgical units) required changes to planning and readiness for dissemination. The trial materials included the scientific rationale, a training video, instructional handouts, a simple do’s and don’ts list, a one-page “Just in Time Training” guide for daily substitute staff, sets of frequently asked questions (FAQs) for stakeholders and staff, “Daily Staff Huddle” sheets on 14 topics, flyers, and quarterly staff and patient assessment forms. All materials were designed to be visually appealing, written in lay language, and tailored to target populations. To prepare for scale-up after trial completion, Dr. Huang recommended that researchers ensure all materials are translated into languages appropriate for their audiences, integrate them into the workflow, and expect high staff turnover. Feedback during the trial informed an FAQ that was a living document and included practical steps that people from many sites had asked about over the years. Dr. Huang also credited the Agency for Healthcare Research and Quality (AHRQ), one of the REDUCE MRSA Trial funders, for requiring her team to develop a business case within the toolkit that could be made to HCS executives about why they should adopt the intervention.¹³

Next, Dr. Doug Zatzick presented on benefits of developing and harnessing a partnership with a professional society, the American College of Surgeons Committee on Trauma (ACS-COT), during the design and rollout phase of his trial. Policy (including regulatory), guidelines, and resource considerations and products have emerged from this partnership. Dr. Doug Zatzick’s previous trial informed ACS policy on screening and intervention for alcohol-related disorders¹⁴ and led to a subsequent requirement for alcohol screening and brief intervention for US Level I and II trauma centers. The Trauma Survivors Outcomes and Support (TSOS) team is now developing a similar clinical best practice guideline for post-traumatic stress disorder screening and intervention for the ACS resources guide.^14,15 Beginning in the planning phase of the trial, the TSOS team incorporated implementation science frameworks and methods, which reduced study team resource/time investments. For example, TSOS used a novel Rapid Assessment Procedure, based on clinical ethnographic mixed methods, for making clinically relevant observations of trial rollout. In addition, they embedded researchers, front-line clinicians, and policy personnel in the implementation team during the planning stage―an approach that may facilitate sustainable implementation of trial results within HCS. This helped ensure all sites’ familiarity with screening and intervention requirements. The TSOS team has a policy summit scheduled with the ACS-COT in the final year of the trial that will allow accelerated integration of trial findings with national trauma center policy.

Dr. Gloria Coronado described how her team overcame initial resistance from HCS leaders when they pitched the idea for the Strategies and Opportunities to Stop Colorectal Cancer in Priority Populations (STOP CRC) trial.^16,17 Data showed that colorectal cancer (CRC) screening had poor uptake and had pronounced racial/ethnic disparities. Fortuitously, state and Federal policy changes impacting testing for CRC were taking place at the same time: the Affordable Care Act, Medicaid incentives in Oregon, and legislation passed in Oregon in 2014 and 2015 that increased coverage for CRC screening. Medicaid incentives raised the visibility of CRC screening and brought in additional partners from Medicaid health plans, which was “a game changer.” To address expected changes in context and policy, their implementation efforts also included forming a project-specific advisory board, working with a member of the Oregon state legislature who could advocate for state policy changes, and holding annual daylong stakeholder meetings. They also applied a common quality improvement strategy, Plan-Do-Study-Act (PDSA), to engage each of the participating clinics in refining, adapting, and embedding the intervention into standard workflows.¹⁸ Dr. Coronado recommended that the trial evaluation plan be flexible enough to allow health centers to roll out the intervention slightly differently or within different timeframes. In addition, the study team worked to overcome the challenge of burdening clinic staff in preparing and sending out mailings, obtaining support from Oregon’s Medicaid health plan and funding from the Centers for Disease Control and Prevention (CDC) to conduct a collaborative project with the University of Washington on development of a model for direct mailing. The result was a cobranded direct-mail cost-sharing program involving the participating Medicaid Health Plans and health centers. National and local policy developments raised the priority of the intervention and identified new study partners. Implementation was aligned with familiar clinic approaches, which helped facilitate co-learning and problem solving on the research and clinical sides. Such partnerships hold promise for long-term sustainability because of better alignment with health plans on incentives and cost sharing.

Panel 2: HCS Engagement: Partnership, Relationships, and Transparency

The second panel reviewed challenges relating to trial implementation in the context of HCS that undergo constant change and ways to create alignment and partnership between researchers and the system partners. Drs. Miguel Vazquez and John Warner reviewed several facilitators for D&I during the design phase of Improving Chronic Disease Management With PIECES (ICD-PIECES):¹⁹ 1) having a prior history of work together; 2) choosing disease(s) or condition(s) and research questions that are important to both the HCS partners and researchers; 3) approaching study design and planning collaboratively with system administration and clinical “champions” early on to develop trust; 4) having early discussions about everyone’s needs, wants, and potential benefits to understand which endpoints are important; 5) establishing expectations early that the team would conduct the research and the HCS would provide health care and advise researchers how to accomplish things in the various clinics; and 6) maximizing use of available resources. Since resources vary by system they advised that researchers must minimize disruptions to busy clinics, provide needed tools and resources to help the system succeed, establish ongoing communication and updates, and be adaptive and flexible. At the study maturation and completion stage, Dr. Vazquez advised creating long-lasting value by developing patient registries with information that could be used to inform future care and promote D&I early, including a planning process for the future so that benefits obtained from the study could be incorporated into learning HCS.

Dr. Lynn DeBar of Pain Program for Active Coping & Training (PPACT) and one of her system partners, Dr. Andrew Bertagnolli, explained that the research question PPACT addressed was brought to Dr. DeBar by system leaders as an important problem.²⁰ They learned that it was critical for researchers to convey how a project may help system leaders achieve their organizational goals and what tools and other products the trial could create and test that may later ease clinical workload. They also shared that they used a rapid assessment process to understand the dynamic systems and to anticipate and keep abreast of frequent changes in the “ecosystem.” Frontline staff and the study team gather frequently and communicate regularly with various stakeholders. Through engagement they found that it helped to explore whether anything could be removed from frontline staff workloads to make room for the trial. They also found it helpful to work with labor during the planning stages so that complications such as grievances do not delay or halt the timeline. They discussed the importance of working across multiple silos and dealing with inevitable change by engaging all stakeholders from the beginning and maintaining channels of communication. Engaging staff at all levels led to alignment and engaged frontline staff in problem solving, especially with questions like “How can you help us get this done?” An upside of change is that it “keeps researchers on their toes.” They reinforced the need for researchers to always have a trial synopsis and to address return-on-investment and costs of a trial with system leaders, as this reminds them of trial goals and the pathway to a sustainable intervention.

Dr. Vincent Mor of Pragmatic Trial of Video Education in Nursing Homes (PROVEN) explained that he had long-term working relationships with his system partners in the nursing home industry who found the proposed project and the team’s previous experience a positive draw.²¹ In addition, the study topic was something that could help them with certain system problems, particularly their need to lower hospitalization and rehospitalization rates. Joint planning from the outset has been extremely important. Their trial design supported full engagement and investment, which helps when increased efforts or changes needed to be made. To overcome challenges with system leadership transitions, he and the research team worked with the system to make the best new hire. Researchers also need to be prepared for HCS corporate restructuring of companies, facilities, and staff, and to have an approach for working with new system leaders and staff.

Panel 3: Designing for Sustainability

Panel 3 included four speakers who described their approach to designing and conducting a trial with sustainability in mind. Dr. Amy Kilbourne began by describing lessons about sustainability from the US Department of Veterans Affairs’ Quality Enhancement Research Initiative (VA QUERI). She reviewed potential sustainability barriers including limited external validity of interventions, persistent quality gaps across systems, competing demands on frontline providers, and misalignment between what researchers propose and organizational priority goals. Her recommendations included considering treatments that are more generalizable and simpler to adapt, designing with frontline providers in mind and better engaging with them, allowing for adaptation by providers and consumers, and thinking about using data and stories to enhance buy-in at all levels.

Dr. Laura Dember discussed aspects of the TiME Trial (Time to Reduce Mortality in End-Stage Renal Disease) design that were viewed as important for sustainability outside of the trial setting.²² The investigators and HCS partners selected a question that was viewed as clinically important, that all partners wanted answered, and that was of interest to a major payer and regulator for US dialysis care: the Center for Medicare & Medicaid Services. It was also important for sustainability that they used system resources to collect data: multiple EHR platforms were involved, with data obtained through routine clinical care rather than research activities. The team used a relatively simple trial design to facilitate intervention rollout, if trial results were positive, by using nonrestrictive eligibility criteria, maintaining usual physician autonomy, anticipating sustained implementation post-trial, and employing oversight and financial pressures already existing in the dialysis setting. Buy-in of stakeholders―including system executives, frontline clinicians, and patients―at all levels was important. One challenge in implementing the trial in the real-world setting was aiming to determine effectiveness of an intervention for which efficacy had not been established.

Drs. Jerry Jarvik and Patrick Luetmer described lessons for planning for sustainability of the Lumbar Imaging with Reporting of Epidemiology (LIRE) program.²³ They emphasized the usefulness of the trial’s stepped-wedge randomization scheme, which assures that all sites eventually receive the intervention. Other lessons included keeping the intervention as simple as possible, minimizing burden on health-system partners (e.g., in outcome data collection), and budgeting for change. They also noted that “change is the only constant” in sustaining a pragmatic trial. These trials, unlike explanatory trials, rely on clinical staff time, workflows, data, and IT systems controlled by a HCS, and although researchers provide support for such resources, they do not control them. Tremendous market consolidation along with constant data changes and evolutions in technology, like replacement of systemwide EHR systems, have increased potential for flux in large HCS. The researchers found it was important to develop relationships within the HCS partners, understand how and where decisions are made, and leverage regional and site leadership structures. Whenever researchers need to effect change in a clinical system, they should build a multidisciplinary team (while remembering that teams are controlled by the practice). It is valuable to invest in onboarding new team members, including by communicating the big picture of why the study is being done (i.e., “to change health care in a positive way”) and why that person is important to the effort. Understanding what aspects of the intervention were being adhered to was important to know for future sustainability, so the researchers instituted special data polls to audit sites on how the intervention was being implemented. Since the LIRE intervention was mostly educational, the team tried fostering “generational sustainability” by making reports and other materials educational to anyone who would be exposed to them, from ordering clinicians to patients. They recommended thinking about explicitly engaging those who are going to be the next generation implementing the intervention, such as fellows and other trainees, and those in IT and informatics.

Dr. Greg Simon then presented on the Suicide Prevention Outreach Trial (SPOT) and described an important early step of advocating to system leaders that their interventions needed to be delivered by dedicated staff rather than being added to existing primary care provider’s workload.²⁴ They developed a staffing model that has been used across the study sites and can predict future needs if the intervention is scaled up. In selecting a research question, Dr. Simon opined that “the relationship is primary, the question is secondary.” Selection of the research question was driven by a number of factors, including the interests and priorities of the HCS; a commitment to choosing the question collectively, rather than individually or ad hoc; organizational equipoise, including the levels of importance, interest, and certainty about the topic; and cost. He also found, “The more complex the patients, the more important the focus.” In implementing more complex or behavioral interventions in rapidly changing environments with heterogeneous patients, an intervention protocol is important so that instances when it is not followed can be identified. Dr. Simon suggested that researchers should think about who their audience is, whether it is a journal editor or system leadership.

Panel 4: Swimming with the Sharks: Translation of Pragmatic Trial Results

The goal of this interactive session was to illustrate a strategy whereby researchers can approach HCS leaders and be persuasive about implementing results from pragmatic research. Researchers need to communicate effectively with HCS leaders to develop partnerships for pragmatic research within learning HCS, that is, to move the results of cost-effective, large-scale research studies on questions of major public health importance into clinical practice as quickly as possible.

Dr. Simon role-played as the researcher. He described the health problem, background on the trial, the question it answered, trial results, and rationale for implementing it into routine care at the system level. Very importantly, information presented included estimated cost for systemwide implementation. He also explained what screening tools would need to be embedded into the clinical workflow, the clinical staff needs, and the informatics needs, so requirements were clear.

After hearing the presentation, system executives indicated they needed a succinct targeted list of details about these requests to make an informed decision to move forward. They emphasized the need for brevity and clarity when researchers provide specifics on the following:

• The reasons for investing in the intervention and not others

• How the intervention is aligned with organizational priorities (the “short list”)

• Level of acceptability by the clinical team, for example, and impacts on providers, clinical teams and their workflows

• Value and benefits to the system, such as market growth and reputation

• Potential harms like liability issues

• Downstream implications

• Plans to sustain the intervention

• How the intervention aligns with payers and policymakers.

Panel 5. Stakeholder Reactions Panel Discussion

In this panel, a group of stakeholders representing HCS leadership, implementation science, and research sponsor perspectives responded to a set of questions. The focus was on identifying how pragmatic trials align with strategic goals for distinct stakeholders; options for future planning and implementation of studies; and tools and resources that are available for pragmatic research or can be created to facilitate results implementation.

Ms. Susan Mullaney underscored that pragmatic studies, by design and appropriate planning, can address many implementation aspects of an intervention, such that HCS can determine whether frontline staff are able to deliver it. Researchers should focus particularly on areas of high impact and priority for HCS leadership and carefully and efficiently articulate how their studies align with organizational goals.

Dr. John Warner noted that many HCS are interested in “tight collaborations” with the research community and aligning priorities for both. He advised that early communication and sharing of ideas for trials, and the sharing of preliminary data, encourage the development of a relationship around an idea. Other priorities include considering implementation, sustainability, and scale-up efforts, even at the design phase. Timing and timelines are critically important in the changing health care and reimbursement environments.

System leaders seek to improve health care and its outcomes through efficient delivery. Predictive risk modeling is especially a gap, and incorporating this as well as resource utilization in study goals is valuable to study leaders when they consider research priorities. Many systems have processes in place for researchers to communicate their ideas to leadership and resources for supporting “good ideas. Successful projects require close communication between all parties―including operations staff and those who deliver interventions. Understanding staff members’ workloads and workflows is critical for project success.

Dr. Kilbourne offered perspectives as both a health services investigator and research funder working in a constrained research environment. Pragmatic trial investigators must understand the priorities and needs of partnering systems, develop and refine tools and communication strategies at all phases of their trial, and have them ready for dissemination before trial completion. It is also critical to educate researchers on how to present their ideas, their “ask,” to system leaders or other stakeholders―including targeted talking points addressing priorities and return on investment at all levels. Since trials are focused on important questions of high impact, investigators should have an answer to provide at trial completion, not simply a message that more research is needed. The VA QUERI Program has fostered this type of environment, with short-, medium-, and long-term funding strategies for VA-affiliated investigators to establish “laboratories for implementation” that study both clinical effectiveness and impact as well as the effectiveness and sustainability of the implementation strategy to promote provider uptake of a clinical practice.

Approaches should be considered to (1) allow some additional space to conduct pragmatic trials that would allow more flexibility and minimal risk assignment if modification becomes necessary, and (2) understand how environmental changes and constraints end up influencing an intervention and vice versa―an area where more research and measurement are needed. Academic pragmatic trial researchers face many challenges that impact training and promotion, and new strategies are needed. The AHRQ has developed and prioritized a set of core competencies to guide the design, implementation, and evaluation of training programs for learning HCS researchers.²⁵

Dr. Catherine Meyers of the National Center for Complementary and Integrative Health noted strong interest for NIH to continue supporting the design and implementation of pragmatic trials within HCS. Several NIH agencies have been working in this research space, often in collaboration with other federal agencies such as the VA and AHRQ. Workshop input from system leaders had been very informative about how researchers, and perhaps other research funders, can align their priorities for future pragmatic trials and about the resources that would be most valuable to the research community. Articulating the research questions of highest priority within systems can markedly enhance researchers’ efforts to design and implement trials of high impact for health care operations and the patients they serve.

Summary

In summary, Drs. Chambers and Eric Larson identified important lessons from the workshop. NIH Collaboratory trials have created a unique learning opportunity for implementation science. Prominent workshop themes included aligning priorities, having bidirectional communication flow between research and practice, identifying enduring research questions while recognizing the rapidly changing environment of HCS, and, above all, maintaining strong relationships between partners. Each trial highlighted was an indicator of remarkable progress in the field over the past several years

The workshop also demonstrated that it is critically important to plan for D&I early in the life cycle of a PCT. Those plans must include allowing for the possibility of changes and local flexibility within the HCS. Pragmatic researchers and their partners must consider together the next step for moving an intervention from a test case to standard care. Such an approach is also important for research funders, in their solicitations for research proposals and internal project review processes. PCTs have emerged as a rich testing ground for D&I methods, in addition to their potential high impact for answering questions that improve health care. In theory, the close linkage of researchers with HCS staff who implement a PCT could help reduce the well-known implementation delay of research results.²⁶