EvidenceNOW: Balancing Primary Care Implementation and Implementation Research

Abstract

The mission of the Agency for Healthcare Research and Quality (AHRQ) is to generate knowledge about how America’s health care delivery system can provide high-quality care, and to ensure that health care professionals and systems understand and use this evidence. In 2015 AHRQ invested in the largest primary care research project in its history. EvidenceNOW is a $112 million effort to disseminate and implement patient-centered outcomes research evidence in more than 1,500 primary care practices and to study how quality-improvement support can build the capacity of primary care practices to understand and apply evidence.

EvidenceNOW comprises 7 implementation research grants, each funded to provide external quality-improvement support to primary care practices to implement evidence-based cardiovascular care and to conduct rigorous internal evaluations of their work. An independent, external evaluator was funded to conduct an overarching evaluation using harmonized outcome measures and pooled data. The design of EvidenceNOW required resolving tensions between implementation and implementation research goals.

EvidenceNOW is poised to develop a blueprint for how stakeholders can invest in strengthening the primary care delivery system and to offer a variety of resources and tools to improve the capacity of primary care to deliver evidence-based care. Federal agencies must maximize the value of research investments to show improvements in the lives and health of Americans and the timeliness of research results. Understanding the process and decisions of a federal agency in designing a large clinical practice transformation initiative may provide researchers, policy makers, and clinicians with insights into future implementation research, as well as improve responsiveness to funding announcements and the implementation of evidence in routine clinical care.

INTRODUCTION

The Agency for Healthcare Research and Quality (AHRQ) is committed to health services research in primary care and helping primary care practices use the latest evidence to improve care. In 2015, AHRQ launched EvidenceNOW: Advancing Heart Health in Primary Care, a multiyear pragmatic trial.¹ EvidenceNOW is designed to generate information about the effectiveness of external quality improvement support in helping small and medium-size primary care practices use findings from patient-centered outcomes research to improve the heart health of their patients. EvidenceNOW also is designed to determine what works best in developing practices’ capacity for ongoing improvement.² This $112 million initiative, one of the largest in AHRQ’s history, weaves together 3 goals: improving the delivery of primary care services, fulfilling AHRQ’s congressional mandate to accelerate the implementation of patient-centered outcomes research findings, and aligning with the Department of Health and Human Services’ Million Hearts and its focus on the ABCS of heart health: appropriate aspirin use, blood pressure control, cholesterol management, and smoking cessation support. EvidenceNOW comprises 7 implementation research grants, each funded to provide external quality-improvement support to approximately 250 primary care practices to implement evidence-based cardiovascular care and to conduct rigorous internal evaluations of their work. In addition, an independent, external evaluator was funded to conduct an overarching evaluation using harmonized outcome measures and pooled data. A technical support contract also was awarded to organize initiative-wide learning communities.

This article describes decisions and trade-offs made in designing EvidenceNOW. The principles underlying each decision were to maximize the possibility of implementing EvidenceNOW in real-world settings, to allow flexibility and local adaptation, and to maintain rigorous study design and methods to enhance the generalizability of findings.

IMPLEMENTATION RESEARCH GRANT DESIGN DECISIONS

Use of Cooperatives in Contiguous Geographic Regions

Because of the scope, breadth, and complexity of the project, applicants for the implementation research grants were encouraged to form regional cooperatives that would bring together the skills, experiences, and resources of academic researchers and local organizations (including primary care and quality improvement organizations, public health agencies and community-based organizations, public and private payers, and consumer/patient groups) to design and carry out their projects. AHRQ selected the term cooperative not only as a nod to the work of the agricultural extension service in supporting improvements in local farming communities but also to emphasize the importance of multiple organizations working together toward common goals. AHRQ recognized that cross-organizational collaboration requires extra time and resources, and that not requiring such collaboration might have been more efficient and nimble. AHRQ, however, decided to require cooperatives to incentivize the alignment of local resources and to contribute to long-term sustainability within communities.

AHRQ also recognized the importance of variation in the external contexts in which primary care practices operate. Many determinants of health care delivery are local, such as state policies (including Medicaid) and the scope of practice regulations. Implementation grant applicants were required to define a contiguous geographic region for their studies as a way to increase awareness of and responsiveness to local needs and changing conditions. One potential trade-off was that cooperatives not bound by geographic considerations might have been able to focus on specific issues across multiple regions, such as support for safety net clinics or nurse-led practices. Allowing both regional and national cooperatives could have created considerable (and undesired) overlap by geographic region or responsibility, however. Applicants were therefore encouraged to submit proposals that defined discrete, contiguous geographic regions as the settings for their studies.

Focus on Small and Medium-Size Primary Care Practices

Despite trends toward larger group practices, more than 88% of office visits occur in practices of 10 or fewer physicians.³ In addition, the majority of primary care practices in the United States employ 10 or fewer clinicians, and most primary care clinicians work in small and medium-size practices.⁴

AHRQ believed that larger practices or practices owned by large systems are more likely to have internal resources for quality improvement. In comparison, smaller practices have fewer resources for transforming and improving care.^5–8 External support resources and expertise are often needed for these practices to engage, initiate, and sustain quality improvement efforts.⁹ When provided with these types of support, smaller practices are potentially more able to adapt nimbly and improve care.¹⁰

AHRQ chose, therefore, to focus on small and medium-size primary care practices. AHRQ required each implementation grant applicant to develop a plan for the recruitment of independent primary care practices with 10 or fewer lead clinicians (defined as physicians, physician assistants, and nurse practitioners). Practices that were parts of larger systems could be included if the applicant documented that the system did not provide meaningful quality improvement support and the practice had the independence to engage in quality improvement activities.

Specific Intervention Strategies

AHRQ funded a series of research initiatives starting in 2010 to understand the challenges primary care practices faced as they transformed to new delivery models, as well as to help create an infrastructure to support transformation.¹¹ Several quality improvement interventions were identified as likely to be effective in translating patient-centered outcomes research evidence into practice, including practice facilitation, data feedback and benchmarking, health information technology support, local learning collaboratives, and expert consultation (Figure 1). A tension emerged in deciding whether AHRQ should mandate a specific intervention or combination of interventions to be used. A standardized intervention, if found to be effective, might provide the field with a clear picture of how to implement external quality improvement support with primary care practices.

Figure 1.

EvidenceNOW external quality improvement strategies.

A challenge in using a standardized intervention, however, was that the evidence to guide the selection of specific intervention components and their duration was limited. AHRQ therefore decided to allow applicants to propose their own multicomponent interventions and implementation strategies. Applicants were required to provide a detailed description of their proposed approach and the evidence supporting all intervention components. This variety across intervention strategies adds complexity to the national evaluation but makes it possible to conduct comparisons that will provide insights into which type of quality improvement interventions, in which contexts, and in what amounts and combinations produce what types of results.

Research Strategies

Similarly, AHRQ allowed applicants to define their research strategies, which included determining and describing their recruitment strategy and the duration and intensity of interventions provided. AHRQ required that all proposals detail feasible timelines that allowed for cooperative start-up, practice recruitment, and intervention delivery within a 3-year period. Establishing the duration of the initiative was a particularly difficult decision. Given the size and complexity of the project, and recognizing that practice change and resulting improvements in outcomes take time, AHRQ considered using a 4- to 5-year time frame for the project. AHRQ, however, also was keenly aware of the urgent need for high-quality evidence regarding the capacity of primary care to improve and implement evidence-based care: the health care system is changing rapidly and policy makers and health care leaders are demanding more information about the best ways to support primary care improvement.¹¹ After reviewing the literature and speaking with implementation and research leaders across the country, AHRQ balanced these competing reasons and selected a 3-year limit for the implementation research grants.

Evaluation

A critical element of EvidenceNOW is rigorous evaluations at both the cooperative level and the national level. Each regional cooperative was required to propose and conduct an internal evaluation of its local intervention. In addition, AHRQ funded an independent, overarching external evaluation to provide an analysis of findings across the implementation grantees. Decisions regarding the requirements for the evaluations were based on the overarching objectives of the initiative, the value of different types of data, and the feasibility of collecting specific types of data in real-world primary care practices.

Local Evaluations

Each implementation grantee was required to collect clinical outcome measures of ABCS (Table 1) at the practice level at baseline, every 3 months during the intervention, and 6 months after the intervention was completed. The specified measures of ABCS were selected to align with Million Hearts and to be consistent with measures used for CMS’s Physician Quality Reporting System. The decision to use measures already in place for quality reporting was intended to simplify data collection and connect EvidenceNOW to care processes relevant to primary care practice. AHRQ anticipated that clinical measures would continue to evolve in response to emerging evidence, and recognized that recommendations and guidelines, especially around hypertension control and cholesterol management, were likely to change during the study period. Implementation grant applicants were therefore required to develop plans that incorporated data harmonization during the early months of the initiative to address changes that might occur between publication of the funding announcement and the beginning of implementation. Applicants were free to include additional clinical process and outcome measures in their evaluations.

Table 1.

Harmonized ABCS Measures

Measure	Definition
A: Aspirin use (Source: CMS164v4)	Proportion of patients aged 18 years and older who were discharged alive for acute myocardial infarction, coronary artery bypass graft or percutaneous coronary interventions in the 12 months before the measurement period, or who had an active diagnosis of ischemic vascular disease during the measurement period, and who had documentation of use of aspirin or another antithrombotic during the measurement period
B: Blood pressure control (Source: CMS165v4)	Proportion of patients aged 18 to 85 years who had a diagnosis of hypertension and whose blood pressure was adequately controlled (<140/90 mm Hg) during the measurement period
C: Cholesterol management (Source: PQRS 438)	Proportion of the following patients—all considered at high risk of cardiovascular events—who were prescribed or were on statin therapy during the measurement period: Adults aged 21 years and older who were previously diagnosed with or currently have an active diagnosis of clinical atherosclerotic cardiovascular disease, or Adults aged 21 years and older with a fasting or direct LDL-C level (≥190 mg/dL), or Adults aged 40–75 years with a diagnosis of diabetes with a fasting or direct LDL-C level of 70–189 mg/dL
S: Smoking cessation (Source: CMS138v4)	Percentage of patients aged 18 years or older who were screened about tobacco use 1 or more times within 24 months and who received cessation counseling intervention if identified as a tobacco user

CMS = Centers for Medicare and Medicaid Services; EHR = electronic health record; LDL-C = low-density lipoprotein cholesterol; PQRS = Physician Quality Reporting System.

An important goal of EvidenceNOW, in addition to disseminating and implementing patient-centered outcomes research findings related to cardiovascular health, is to improve practices’ capacity to implement research evidence in the future. AHRQ therefore selected 2 measures of practice capacity for quality improvement to be used by all implementation grantees: the Change Process Capacity Questionnaire¹⁰ and a measurement of adaptive reserve.¹² These 2 instruments have been used in a variety of studies of primary care quality improvement research, and both showed promise as indicators of practices’ ability to engage in change. EvidenceNOW provides an opportunity to further test their validity.

Cooperatives were required, at a minimum, to survey practice capacity at the individual practice level at baseline, at the end of the intervention, and at least 6 months after the intervention. Other practice-level measures that were required are presented in Table 2. Applicants were encouraged to include additional measures (Table 3). In this way, AHRQ established a cross-cooperative baseline for use in the national evaluation and for use in comparative analyses while permitting diverse and targeted local evaluations.

Table 2.

Implementation Evaluation: Required Practice-Level Measures

Required Measures	Domain(s)
Change Process Capacity Questionnaire	Practice capacity
Measurement of adaptive reserve	Practice capacity
National Ambulatory Medical Care Survey Electronic Medical Records questionnaire (2010)	Internal context (includes practice organization, staffing, and patient population, and an assessment of the degree of EHR adoption of each practice and the ability of the practice to report quality measures)
Concurrent practice improvement initiatives	External context (for example, QIO/QINs, CMMI, CPCI, and TCPI)
Supporting strategies	External context (for example, pay-for-performance and public reporting initiatives)
Implementation and adaptation of comprehensive approach to quality improvement support	Possible aspects to address: acceptability, adoption, appropriateness, feasibility, fidelity, implementation costs, and sustainability
Intervention tracking	Specific strategies used with individual primary care practices (adaptation to local circumstances was allowed)

CMMI=Center for Medicare & Medicaid Innovation; CPCI=Comprehensive Primary Care Initiative; EHR=electronic health record; QIN=Quality Innovation Network; QIO=Quality Improvement Organization; TCPI=Transforming Clinical Practice Initiative.

Table 3.

Implementation Evaluation: Encouraged Additional Practice-Level Measures

Domains of measurement

Practice capacity, adaptive reserve

Leadership and organizational style

Quality improvement structures and processes

Team-ness

Staff satisfaction and burnout

AHRQ carefully considered requiring the collection of patient-level data to examine interventions’ effects on health outcomes and on racial, ethnic, and other health care disparities. AHRQ believed, however, that given the relatively short time frame for the initiative, the potential to document changes in cardiovascular health outcomes was small. Recognizing the costs associated with additional data collection, AHRQ chose not to require the collection of health outcomes data. This decision reflected the initiative’s focus on implementing already established patient-centered outcomes research evidence and on building practice capacity to incorporate evidence into care delivery.

EvidenceNOW was designed to be a large implementation effort and a research effort. Given expected variations in the intervention and implementation strategies, AHRQ decided to allow respondents to select research designs that best reflected the contexts, capabilities, and interests of the applicants. Each cooperative had to propose an appropriate form of control for secular trends and other factors unrelated to the intervention. Cluster randomization was encouraged but not required. Alternative study designs, such as interrupted time series, step-wedge, regression discontinuity, and other delayed intervention groups and matched cohort control were acceptable.

Cooperatives had to commit to participation in the separate national evaluation, agree to coordinate data element standardization and quality assurance, and share de-identified data with the national evaluator. By design, AHRQ anticipated data harmonization across the implementation research grantees after award, especially with regard to the ABCS. Although harmonization was important to ensure each cooperative could contribute to the national evaluation, AHRQ was sensitive to preserving the uniqueness of each cooperative’s specific aims and evaluation.

NATIONAL EVALUATION

AHRQ solicited grant proposals for a national evaluation at the same time that proposals were solicited for implementation research grants that included local evaluations. AHRQ considered using a contractual mechanism for the national evaluation but decided against it to ensure the real and perceived independence of the national evaluation, thereby seeking to maximize confidence in the results.

The goals of the national evaluation were to provide a summative evaluation of the findings from each individual cooperative; extract, examine, and rapidly report key themes and findings from across the implementation grantees; and evaluate the comparative effectiveness of the implementation strategies used by the different implementation grantees with particular attention to analysis of contextual and environmental factors and their influences. The national evaluation was designed to complement the implementation grantees’ evaluations and to require close collaboration with them.

Given the requirements for the local evaluations, the national evaluator could expect that they would be provided with de-identified data of the ABCS and practice capacity at regular intervals, as described above. AHRQ enabled the national evaluator to budget funds to compensate implementation grantees or primary care practices for efforts related to additional data collection for the overarching evaluation. This unique model of support for data collection meant that the implementation and evaluation grantees were interdependent, and each relied on the other to assist with obtaining practice data. This model also recognized that data collection for research purposes is a costly investment that often requires effort outside the routine work of care delivery.

AHRQ encouraged proposals for the national evaluation to use the Consolidated Framework for Implementation Research¹³ to capture and consider all important implementation domains. AHRQ also required that proposals for the national evaluation describe a plan for maintaining flexibility and allowing for the adaptation of the evaluation approach to continue meeting the aims and objectives of the project over time and in the context of a changing health care environment.

Because of the simultaneous nature of the grant solicitations, applicants for the national evaluation grant had to discuss how they would establish relationships with the to-be-determined implementation grantees, refine data collection approaches, and execute the evaluation within 4 years. AHRQ required that applicants consider a plan to minimize the burden placed on the implementation grantees in relation to the national evaluation. To facilitate communication and cooperation with the national evaluation, AHRQ also required that each implementation grantee dedicate a minimum of 20% full-time-equivalent senior evaluator and 40% full-time-equivalent research assistant in each year of their project to work with the national evaluator.

Although the national evaluation was designed to begin in parallel with the implementation work, AHRQ chose to fund the evaluation grant for an additional fourth year to maximize data analysis, interpretation, and dissemination.

TIMELY DISSEMINATION AND LEARNING COMMUNITIES

AHRQ recognized the rapidly changing landscape in primary care and the urgent need for knowledge about practice transformation. The funding opportunity announcements for both the implementation grantees and the national evaluator therefore included requirements that they commit to and plan for real-time dissemination of lessons learned. In establishing this requirement, AHRQ balanced the need and desire of researchers to hold findings until final analyses can be published in peer-reviewed journals with the needs of stakeholders to benefit from federal investments. AHRQ decided to ask all applicants to commit to timely dissemination of findings, given the need to inform major public and private initiatives while protecting their ability to publish findings in peer-reviewed journals.

In addition, AHRQ recognized the need for the implementation and evaluation grantees to communicate with one another regularly throughout the project period to create trust, collaborative relationships, and opportunities to learn from one another and solve problems together. AHRQ therefor funded a technical assistance center to create an initiative-wide learning community and assist AHRQ, the cooperatives, and the national evaluator in timely dissemination of lessons learned.

EVIDENCENOW IN ACTION

In May of 2015, AHRQ officially launched EvidenceNOW, awarding grants to 7 regional cooperatives and an independent national evaluation team (Figure 2). The cooperatives span 12 states, deliver health care in a range of metropolitan and rural settings, and serve diverse populations.¹ AHRQ established an EvidenceNOW Technical Assistance Center through a contract to Crosby Marketing in partnership with the MacColl Center for Health Care Innovation and Abt Associates, Inc.

Figure 2.

Geographic reach of EvidenceNOW.

ESCALATES = Evaluating System Change to Advance Learning and Take Evidence to Scale; TAC = Technical Assistance Center.

By January 2017, the 7 implementation grantees had enrolled more than 1,500 small to medium-size primary care practices with approximately 5,000 clinicians serving 8 million patients, launched their implementation interventions, and collected baseline ABCS and practice capacity data for enrolled primary care practices.¹ Of the 7 implementation research grantees, 4 are using step-wedge designs, and 3 are conducting cluster randomized trials with external control groups. Several grantees anticipate collecting patient-level data, and the national evaluation team is exploring whether cross-cooperative evaluation of patient-level data is possible.

CONCLUSION

EvidenceNOW is an implementation research initiative designed to generate evidence at the regional and national levels regarding interventions to help small and medium-size primary care practices translate evidence into practice and improve their delivery of care. It also will generate evidence about how external support services can be provided most effectively, taking into consideration practices’ internal characteristics and external environments.

AHRQ understands the need for implementation research that generates knowledge with real-world applicability in the quality improvement, clinical, and policy realms. The design decisions that were made, which required difficult trade-offs, reflect AHRQ’s commitment to both implementing the latest patient-centered outcomes research evidence in practice and supporting cutting-edge implementation research. The improvement support strategies being used by the EvidenceNOW cooperatives typically have been studied independently; this initiative will rigorously investigate multicomponent interventions and provide information about the strategies (or combinations of strategies) that are most effective given the circumstances and needs of specific primary care practices.

EvidenceNOW, through its explicit provision for local adaptation under a harmonized national umbrella, was designed to produce findings that will be widely applicable to most primary care practices across the country. It is AHRQ’s hope that EvidenceNOW will result in a blueprint for stakeholder investments in strengthening the primary care delivery system and offer a variety of evidence-based tools and resources to facilitate primary care improvement.

AHRQ is committed to maximizing the value of research investments to demonstrate improvements in the lives and health of Americans and to improving the timeliness of research results. Understanding the process and decisions of federal agencies such as AHRQ in designing a large clinical practice transformation initiative may provide researchers, policy makers, and clinicians with insights into the challenges of implementation research, as well as improve responsiveness to funding announcements and the implementation of evidence in routine clinical care.