Conducting Rapid, Relevant Research

Abstract

The lengthy and uncertain translation of research into clinical practice is well documented. Much of the current “gold standard” clinical research is slow, expensive, and lacks perceived relevance for practitioners and decision makers. In contrast, we summarize experiences conducting the My Own Health Report (MOHR) project to collect and address patient reported measures using principles of rapid, relevant pragmatic research. The methods used for rapid design and fielding of the MOHR project to improve attention to health behaviors and mental health are detailed. Within the multisite, pragmatic, implementation–focused MOHR study, we describe the four phases of the research and the key decisions made and actions taken within each. We provide concrete examples of how relevant research can be conducted transparently to rapidly provide information to practitioners. Data were collected and analyzed in 2013.

The multisite (seven research centers partnered with 18 clinics) cluster randomized pragmatic delayed intervention trial was conducted in less than 18 months from receipt of funding applications to completion of data collection. Phases that were especially accelerated included funding and review, and recruitment and implementation. Conducting complex studies rapidly and efficiently is a realistic goal. Key lessons learned for prevention research include: use of existing research networks; use of web-based assessment/feedback tools that are tailored to fit local needs; engaging relevant stakeholders early on and throughout the process to minimize need for redesign; and making pragmatic decisions that balance internal and external validity concerns rather than waiting for perfect solutions.

Introduction

Most research evidence based on “gold standard” trials has little immediate impact on practice or policy.^1–3 Such trials are often so focused on being rigorously internally valid that the results do not disseminate well into practice.^4,5 Traditional trials are also slow and expensive.^6–8 Most importantly, research often fails to respond to the questions of stakeholders who make decisions about health care, and who need rapid, actionable data to make decisions.^9–11

Riley et al.⁷ recently presented a conceptual model and set of key decisions to improve the speed and utility of health research. This model calls for ongoing stakeholder involvement, streamlining the grant announcement and review process, and planning for rapid dissemination, implementation, and analyses, resulting in faster availability of data for decision-making. Despite the need for alternative research administration, designs, and strategies,¹² as yet there are few examples of such rapid research.

This paper reviews the actions, challenges, and lessons learned in developing and conducting the My Own Health Report (MOHR) pragmatic trial, providing an example of a trial designed with the observations of Riley and colleagues⁷ as a guide. We report key decisions in the research phases of funding, collaborative planning, implementation, and analysis undertaken to balance research rigor with speed and relevance to practice. The primary study outcomes, which include the intervention reach, effectiveness, context, and costs will be reported separately. To our knowledge, this is the first report systematically describing the steps undertaken to design, implement, and analyze data from a rapid, complex pragmatic trial for healthcare change.

Methods

MOHR is a practice–level, cluster randomized pragmatic implementation study. The general methods, measures, and intervention components are described elsewhere.¹³ It is designed to develop rapid, actionable evidence around the use of patient-reported measures in patient care.¹³ The intervention uses the MOHR web–based patient assessment and feedback instrument that addresses ten domains of health behaviors and psychosocial issues. MOHR assesses a range of prevention and care issues including health behaviors, mental health issues, and substance abuse, as well as health–related quality of life and key demographic factors (Table 1). This automated assessment and feedback tool encourages collaborative goal setting between primary care providers and patients presenting for usual care, wellness, or chronic care visits. Appendices 1 and 2 present sample feedback reports for a patient and the practice team, respectively. Diverse patients and practice staff were involved in selecting and pilot testing the assessment items and feedback system.¹⁴ The primary outcome of the study was the establishment of collaboratively set action plans. The randomized delayed intervention sites provided a control for temporal trends and local context. The delayed intervention sites were paired with intervention sites and did not receive either the MOHR assessment and feedback tool or explicit goal setting support, but served as usual care controls for what was happening in these contexts given the rapidly changing healthcare environment. Diverse stakeholders (e.g., clinicians, behavioral scientists, public health researchers, and dissemination and implementation scientists) contributed to the measures, study design, and implementation.¹⁵

Table 1.

My Own Health Report measures for adult primary care

	Domain	Number of items and final measure (source)
F	Eating patterns	3 items: modified from Starting the Conversation (Adapted from Paxton AE, et al. Am J Prev Med 2011;40(1):67–71)
F	Physical activity	2 items: the Exercise Vital Sign (Sallis R. Br J Sports Med 2011;45(6):473–4)
F	Sleep	2 items: a. Adapted from BRFSS b. Neuro-QOL (Item PQSLP04)
F	Smoking/tobacco use	2 items: Tobacco Use Screener (Adapted from YRBSS Questionnaire)
SA	Risky drinking	1 item: Alcohol Use Screener (Smith PC, et al. J Gen Int Med 2009;24(7):783–8)
SA	Substance abuse	1 item: NIDA Quick Screen (Smith PC, et al. Arch Int Med 2010;170(13):1155–60)
MH	Stress	1 item: Distress Thermometer (Roth AJ, et al. Cancer 1998;15(82):1904–8)
MH	Anxiety and depression	4 items: Patient Health Questionnaire—Depression and Anxiety (Kroenke K, et al. Psychosomatics 2009;50(6):613–21)
G	Overall health status	1 item: BRFSS Questionnaire
G	Demographics

BRFSS, Behavioral Risk Factor Surveillance System; F, Framingham health behaviors; G, general; MH, mental health; NIDA, National Institute on Drug Abuse; QOL, quality of life; SA, substance abuse; YRBSS, Youth Risk Behavior Surveillance System

Setting and Participant Characteristics

MOHR collected data from over 2,700 patients in 18 primary care clinics associated with seven research centers (Table 2). Six clinic sites were urban, three suburban, and nine rural. The majority of clinic sites had patient–centered medical home (PCMH) status or were in the process of applying. The vast majority of clinics that we approached for participation took part in the project, although one research network that was initially interested declined participation owing to timing issues.

Table 2.

Participating practice characteristics

	State	Practice type	Setting	Patients seen annually	Provider FTEs		Rooming staff FTE			Patient ethnicity and race		Insurance type			PCMH status	Behavior change expertise
					Clinician	MLP	Nurse	MA	Non-clinical	Latino	Black	Medicare	Medicaid	Uninsured
Site1	VA	PBRN	Suburban	5,000	2	0	2	0	2	15%	10%	5%	0%	30%	No	None
Site2	VA	PRBN	Suburban	1,500	1	0	0	1	1	20%	10%	9%	0%	1%	No	None
Site3	VA	PRBN	Rural	2,500	1.2	0.4	0	2.6	4.4	1%	49%	12%	1%	49%	No	Some
Site4	VA	PRBN	Suburban	5,200	2	2	0	5	6	2%	18%	15%	2%	18%	No	None
Site5	VA	PRBN	Urban	4,440	4.9	0.5	7.3	0	5	2%	19%	24%	42%	17%	Yes	Some
Site6	VA	PRBN	Urban	4,770	5.3	0.5	7.3	0	5	1%	17%	26%	42%	17%	Yes	Some
Site7	CA	PRBN	Rural	3,500	5.1	0.4	3.1	6.7	5.2	3%	1%	13%	3%	1%	Applying	None
Site8	CA	PRBN	Rural	5,400	3.9	3.1	5.6	13.8	6.1	13%	2%	12%	13%	2%	Applying	A lot
Site9	VT	PRBN	Rural	9,500	3.3	1.7	2.5	2	9	1%	5%	13%	1%	5%	Yes	Some
Site10	VT	PRBN	Rural	10,000	5	0	3	5	6	1%	2%	15%	1%	2%	Yes	Some
Site11	NC	CPCRN	Rural	1,100	1.5	3	2	6	4	2%	60%	49%	2%	60%	Yes	Some
Site12	NC	CPCRN	Rural	7,500	2.5	1	2	4	4	40%	60%	10%	10%	70%	Yes	A lot
Site13	CA	CPCRN	Urban	2,040	1	0	0	7	5	75%	25%	5%	45%	50%	Applying	Some
Site14	CA	CPCRN	Urban	2,180	1	1	0	6	5	75%	25%	5%	45%	50%	Applying	Some
Site15	TX	CPCRN	Rural	4,800	1	1	0	4	2	48%	23%	2%	48%	23%	No	None
Site16	TX	CPCRN	Rural	3,800	1	1	0	3	3	23%	32%	2%	23%	32%	No	None
Site17	TX	CPCRN	Urban	2,800	1.7	1.3	0	5	14	82%	6%	1%	82%	6%	Yes	None
Site18	TX	CPCRN	Urban	2,800	1.8	1.8	0	5	7	80%	5%	1%	80%	5%	Yes	Some

FTE, full-time equivalent; PCMH, patient–centered medical home

There was variability in panel ethnicity/racial background with a few sites predominantly serving Latinos (approximately 80%) and most remaining clinics serving a small percentage of Latinos (≤23%). Two clinics served more than 50% African Americans. In one clinic, almost 50% of its clients were covered by Medicare, in contrast to most clinics where less than 15% received Medicare benefits. Although most sites (n=16) had electronic health record (EHR) capability, the diversity of EHR systems, limited time, and restricted funding did not allow for full MOHR integration with the diverse EHRs. Comparison of the race and ethnicity of actual MOHR participants to the general characteristics of the overall involved clinics indicated that there was good representation of ethnic and racial groups. The proportion of African Americans was higher among MOHR intervention participants than the overall clinic in four sites, equivalent (± 3%) in two sites, and lower in three sites. Latino ethnicity was higher in two sites, equivalent in four, and lower in three sites.

Study Phases

Funding and review of applications

MOHR was funded using supplemental funds from the National Cancer Institute (NCI), Agency for Healthcare Research and Quality (AHRQ), and NIH Office of Behavioral and Social Science Research. To accelerate the review and start-up process, project eligibility was restricted to currently funded research organizations that were part of one of two existing networks: the AHRQ–funded Practice Based Research Network (PBRN) program¹⁶ or the NCI Cancer Prevention and Control Research Network (CPCRN).¹⁹

Applicants completed a brief, structured application (three pages) that summarized their expertise, proposed key study issues and outcomes, and identified a matched set of proposed primary care practices. Internal review was conducted by staff from the funding organizations, supplemented by outside experts. Priority was given to applications proposing well-matched pairs of sites that varied across dimensions including geographical location, clinic type and size, patient population, urban versus rural setting, and level of EHR integration.

Collaborative planning and refinement

Study design was accelerated by agreement to use a set of common patient–report items and an automated assessment/feedback tool, which was eventually called the MOHR patient assessment and feedback system. The items in MOHR were based on prior work by the NIH to identify brief practical measures feasible for use in primary care.¹⁵ The resultant set of 17 items, covering ten different health behavior, mental health, and substance abuse domains (Table 1) are described by Estabrooks et al.¹⁵ and Glasgow and colleagues.¹⁷

A decentralized decision–making process was used in the collaborative trial planning process. Work groups varying in size from four to eight members were established for: assessment and feedback tool creation; patient-reported outcomes and assessment procedures; context assessment; cost and resources collection; and papers, publications, and public relations. These groups worked efficiently and made the major decisions and recommendations that were brought to overall project conference calls bimonthly.

Patient recruitment and implementation

Efforts were made to recruit a diverse set of primary care practices and patients. There were few exclusion criteria (e.g., we did not exclude patients based on health behavior profile or presence of existing conditions) and practices were encouraged to invite the widest possible variety of adult patients regardless of disease status who were coming for usual care, wellness, or chronic disease visits. Primary care practices had to agree to randomization to either early or delayed intervention, and to recruit 150–200 adult primary care patients within a 1-year period.

Analyses and reporting

The automated tool generated real-time reports on patient enrollment for each clinic. An initial summary data report summarized recruitment experiences and prevalence of the ten different health risks for practices.

Results

Time to Accomplish Key MOHR Activities

Table 3 summarizes the approximate time to complete each of the MOHR phases. The review and funding process took approximately 3 months and was conducted in two waves: one for CPCRN sites (four research sites and eight federally qualified health center clinics) and one for PBRN sites (three research sites and ten PBRN practices). The practices were not randomly selected, but rather resulted from a purposive, pragmatic selection of clinics from two federally funded stakeholder groups designed to be collectively diverse in terms of geography, practice characteristics, and patient profiles.

Table 3.

Key phases, decisions, and actions in the My Own Health Report study

My Own Health Report phases	Decisions and actions to increase speed and enhance relevance
Funding and review of applications (2–3 months)	• Restricted eligibility to existing research networks • Short response time and rapid review process • Internal review and use of supplement funding • Stipulated agreement to use common measures
Collaborative planning and refinement phases (3–4 months)	• Decision to use intermediate implementation outcomes rather than ultimate outcomes • Utilized existing measures where possible and built on prior electronic health record measures work • Conducted small scale, rapid tests of automated tool and items • Engaged stakeholders (clinicians and patients) at multiple points • Gave authority to subgroups to make decisions (e.g., context assessment, automated tool components, patient surveys, and cost collection) • Worked with each clinic to adapt My Own Health Report to their setting, clinic flow, and patients
Patient recruitment and implementation phases (9–10 months)	• Webinar rapid training and biweekly collaborative calls • Responsive “core steering committee” • Adaptations made based on real-time data including weekly feedback on reach and survey completion • Modified recruitment and follow-up procedures
Plans for analyses and reporting (Ongoing)	• Initially outlined table shells and prioritized analyses • First analyses focused on feedback to participating clinics on reach and patient health behaviors • Commitment to transparent reporting on adaptations, variations across sites, and time

Planning and refinement

Project planning, testing, and training on use of the MOHR assessment/feedback tool was accomplished in another 3–4 months. Previous experience with a paper–based pilot test of the MOHR items¹⁴ allowed for efficient creation of the MOHR web– based assessment and feedback tool.¹³ Programming of items, including follow-up questions if the initial screening item(s) were positive for four items (i.e., the Patient Health Questionnaire-2 for depression), were accomplished relatively quickly. It took somewhat longer to create and obtain feedback from both patients and practice staff on the prototype summary reports.

Work groups for the patient experience survey (to develop the patient experience items used as primary outcomes), context assessment (to develop mixed methods to assess key contextual factors), and cost and resources completed their tasks by the time the MOHR assessment/feedback tool was developed and piloted. In general, the larger steering committee endorsed the recommendations of these work groups with only minor refinements.

Recruitment and implementation

These activities were completed in approximately 10 months. Most sites experienced challenges to patient recruitment, consistent implementation of the feedback protocol, follow–up survey completion, or some combination. The general approach used to enhance quality was tracking and providing rapid, continuous feedback on progress, using collaborative semi–weekly learning calls to share successes and brainstorm responses to challenges, and if an initial implementation strategy still did not work, substituting or replacing it.

An example was one site that experienced particularly low response rates to mailed invitations to complete the MOHR assessment, despite using identical procedures as other sites. After consultation regarding options, it was possible to change to an existing call service that was part of the delivery organization. This change resulted in a ten-fold increase in participation. Most sites experienced challenges with obtaining high response rates to the post-visit survey sent 2–4 weeks following the visit and designed to assess patient experiences concerning health risk assessment and shared decision-making regarding identified risks. After reviewing initial data and consulting Dillman et al.,¹⁸ to increase survey return rates, each site supplemented their initial survey return procedures with an added modality (e.g., if initial survey was mailed, following up with a phone call for non-respondents), so that all sites made multiple contact attempts using different modalities.

Challenges Encountered

There were challenges in study implementation, both expected^19–20 and unexpected. We encouraged practices to recruit a large number of representative patients, not just a convenience sample or those they felt would most benefit. This created challenges for busy practices with multiple priorities, more so than just attempting to recruit an occasional, non-complex patient.

Some of the lengthiest delays were due to IRB review. Having seven different IRBs needing to review the protocol, and any changes, resulted in considerable delays. We attempted to address this challenge by having the coordinating center obtain approval early in the process, as a precedent for other IRBs. This was only partially successful, as no IRBs decided to cede review authority, and different decisions were made by different IRBs about which aspects of the study were exempt, expedited, or constituted research as opposed to quality improvement. Although most IRBs considered MOHR assessment and feedback as part of clinical care or a quality improvement process, one IRB required patient consent before patients could be approached. All IRBs except one required informed consent for the follow–up patient experience survey. Also, the IRBs had different requirements about what they would allow versus consider coercive in terms of follow-up contact and incentives for completing surveys.

The heterogeneity across sites and adoption of pragmatic principles that adapted the intervention to fit each setting required different approaches to the details of patient recruitment, MOHR implementation, and feedback procedures. Having to develop strategies that retained common elements, but were tailored to settings (e.g., if the assessment was administered over the web versus in-person, before versus during the visit, in English versus Spanish), proved time consuming. Part of the diversity resulted from the decision to include two care networks (i.e., PBRNs and FQHCs). This decision created greater diversity in the settings, samples, and contexts and increased confidence that the MOHR procedures can work in diverse settings, but it increased the complexity and length of the planning and implementation process.

Finally, it became necessary to replace the initial contractor engaged to create the web–based MOHR assessment/feedback tool when they could not meet the specifications required in a timely manner. Fortunately, all parties were willing to work together to make the transition as smooth as possible, making the delays and added costs as minimal as possible.

Discussion

This paper demonstrates that it is feasible to rapidly conduct complex, high–quality, pragmatic research that is relevant to stakeholders. Despite the complexity of issues addressed, MOHR was conducted in less than 18 months from concept to final data collection. Below, we summarize the primary reasons this was accomplished in a short time relative to traditional multisite trials. Two key influences were the rapid review process for funding and use of existing research networks. Compared to traditional review and funding mechanisms, these two decisions likely saved at least 12–18 months. Paired with a management team providing leadership in the implementation of pragmatic trials, selection of sites that had experience in collaborative multisite research, partnership research principles,^21–23 and working in real–world, non-academic primary care and community settings expedited the process.

MOHR might have proceeded even more quickly if we had restricted sites to only those using a common EHR platform, with shared data infrastructures, or who were at a common stage of PCMH implementation. Adapting the intervention and training, and automating a tool to fit varied clinic flows and implementation plans were challenging. Not attempting to integrate the MOHR into the various EHRs can be seen as a major limitation. Although we will pursue such integration in future research, we are not convinced that it is necessary to have all tools and guidance within an EHR as the most provider or patient-centered way to achieve high-quality implementation.

The use of patient experience outcomes (e.g., surveys about receipt of collaborative action planning) for our primary endpoint was another key factor in accomplishing the project efficiently. If we had required a biological outcome, this would have considerably increased time and dramatically increased sample size and expense, as discussed by Proctor et al.²⁴ and Glasgow and colleagues.²⁵ The decision to focus on many health risks rather than one or two in isolation, such as smoking cessation or depression, as is typically done substantially increased the relevance for primary care settings faced with all of these issues concurrently.²⁶ Further, only a subset of patients had any one risk factor.

MOHR was designed to see if a brief, low-cost procedure to address the complex array of health behavior, psychosocial, and substance abuse issues encountered in primary care could be implemented consistently in diverse real–world settings. Additional research is needed to clarify linkages among key implementation steps such as establishing an action plan²⁷ and follow-up contact²⁸ with behavior change or biological outcomes.

Finally, we used rapid learning principles^29–31 and stakeholder engagement³² processes to simultaneously address speed and relevance. MOHR adapted procedures based upon initial data and evolving issues—in particular, we had to adjust recruitment processes and survey procedures³³ to enhance participation rates. We think that such adaptations are important and that transparency is critical. Although MOHR could have been conducted even more quickly, we made several decisions to enhance relevance or rigor that resulted in modest delays. We recruited clinics interested in pragmatic trials and evaluating processes for ongoing use rather than just asking permission “to let us recruit 150 patients and not bother you.” The decision to translate the MOHR assessment and feedback tool and all measures and procedures into Spanish also took longer, but we judged this important for future use in an increasingly Hispanic patient caseload.

There have been other recent funding mechanisms to support rapid research.⁷ For example, the AHRQ Developing Evidence to Inform Decisions about Effectiveness network provides a quick review process by restricting applicants to a set of previously vetted research groups with demonstrated capacity; the National Institute of Diabetes Digestive and Kidney Diseases has recently used rapid review procedures to fund ongoing natural experiments and policy interventions related to obesity; and Patient–Centered Outcomes Research Institute funding mechanisms are experimenting with ways to enhance both the relevance (e.g., including patients and stakeholders in the review) and speed of review.³⁴ Although limiting funding to sites that are part of research networks would have been restrictive and possibly unrepresentative 10 years ago, today a vast array of PBRNs, community health centers, and rural networks exist, and many more are being established with the advent of accountable care organizations and other real– world learning healthcare systems.

Limitations of this project include that the pragmatic trial is admittedly not definitive. By focusing on multiple issues facing primary care, it is less likely to impact any one of the ten targets than if we had chosen to focus on only one. MOHR was not intended to produce practice transformation,³⁵ but rather to see if diverse practices could find a way to integrate screening and tailored action planning into their clinic flow and procedures. Finally, although we purposely included a wide range of clinics in different geographic areas, in rural, urban, and suburban settings and at different levels of EHR integration³⁶ and PCMH status, the study did not include integrated care plans such as Veterans Affairs or HMO settings, and thus is not generalizable to all primary care settings. Even with these limitations, we think it instructive to present a transparent real–world example of how valid multisite research can be conducted on complex issues in diverse settings relatively rapidly.

Recommendations for future research include: transparent reporting, especially about settings, inclusion and exclusion criteria, and adaptations made; differences across sites; and attempts to balance rigor with relevance, speed, and research efficiency. Remaining challenges related to addressing ten health behaviors, mental health, and substance abuse concerns concurrently include identification of optimal ways to help patients and staff prioritize goals. Long-term sustainability of enhanced patient assessment and counseling remains a question for future research.