Comparative effectiveness research in the “real” world: Lessons learned in a study of treatment-resistant hypertension

Abstract

Background

Comparative effectiveness research (CER) is vital to translate new efficacious diagnostic and therapeutic approaches into effectiveness in usual clinical practice settings. Studying the practice environment in which effectiveness protocols are implemented is necessary to identify the complex challenges that can limit translation of evidence. These issues were addressed in our NHLBI-funded R34, “Controlling Blood Pressure in Treatment-Resistant Hypertension (TRH): A Pilot Study”.

Methods

Qualitative methods were used in this cluster (clinic)-randomized, 4-arm pilot study of TRH in 8 diverse, community-based practices including: (i) Focus group discussions with practice staff and physicians (ii) conference calls with physicians (iii) discussions with research coordinators. Sources were summarized and analyzed by content analysis.

Results

Data segregated into categories representing facilitators of and barriers to research. Key facilitators included: (i) early success in controlling challenging TRH patients (ii), and (iii) reimbursement for study time and expenses. Barriers included: (i) time-consuming regulatory requirements (ii) limited training and research experience of some study coordinators and (iii) reluctance of some physicians to refer to Hypertension Specialists.

Conclusions

Qualitative assessment is valuable for identifying facilitators and barriers to CER. This information is important in designing and implementing CER to accelerate translation of clinical efficacy into effectiveness.

Introduction

Comparative effectiveness research (CER) and practical clinical trials (PCT) are vital tools to translate new drugs, devises and other treatment approaches from proven efficacy to effectiveness. ^1–3 Efficacy studies establish that a treatment produces the desired effect in the controlled environment of one or a few experienced, research-intensive sites. Subjects are typically recruited through a variety of approaches, both from within and outside the entity conducting the trial and subjects tend to be more demographically homogeneous. The research often involves a placebo. Effectiveness studies are conducted in the “real world” of a variety of busy clinical practices, with heterogeneous subjects that are more representative of the general population and may (PCT) or may not (CER) include a placebo.

If we are to translate efficacy into effectiveness and widespread adoption, we must understand the process by which the evidence produced in controlled research sites may vary from the desired effect in clinical practice. We must also appreciate the “adjustments” necessary to tailor efficacious interventions to specific populations or practice types to improve their rate of adoption. Published work in CER and PTC is relatively new, but a general consensus has formed that it must be representative of sites that provide care and the diverse spectrum of patients that can benefit. ³ This requires that we view this research as a collaborative effort, which includes input from practices in the design of the research and includes them in the presentation and publication of results. Specifically, it is important to involve staff from representative practice types, locations and with varying degrees of resource-intensity in the design and implementation of the approach and interpretation of results.^4,5 Studying the practice environment in which effectiveness protocols are conducted adds a layer of insight into the complexity of issues and challenges that can limit translation of evidence. This requires understanding the culture, resources and capacity of primary care practice in a variety of community settings. ⁶ For this reason, we included several qualitative methods to study the process of implementing a recent NHLBI-funded CER R34, “Controlling Blood Pressure in Treatment-Resistant Hypertension (TRH): A Pilot Study”. ⁷ The lessons we learned are important to consider in designing future research conducted in community-based practices.

Methods

OQUIN

The Outpatient QUality Improvement Network (OQUIN) is comprised of 210 diverse practice sites in South Carolina, North Carolina, Georgia, Alabama, Tennessee and Virginia. ⁸ These include individual and small group practices operating at a single site, multi-site physician and hospital owned practices, federally qualified health centers, rural health clinics, Veterans Affairs clinics, and free clinics. Each clinic signed a Business Associate Agreement (BAA) approved by the Institutional Review Board (IRB) at the Medical University of South Carolina. The BAA authorizes research use of EHR data de-identified by patient, provider and practice.

Various electronic health record systems (EHRS) are used by participating clinics, including GE Logician/Centricity, Practice Partner, eClinicalworks, MySys, Allscripts, Medinformatix, SSIMed, and Vista. Most EHRS store all data in a standard SQL-based database. With access to the database, typically a virtual practice network (VPN) link, the EHRS specialist develops queries to extract all discrete field data of interest. Data include age, race, sex, insurance status and insurer, medical visit dates, height, weight, vital signs, medications, laboratory data, diagnoses (ICD9[10]) and procedure (CPT) codes.

Practice settings

Eight community-based primary care practices of varying size, type and location were selected from OQUIN. These clinical sites volunteered to participate in the study and to be randomly assigned to one arm (two practices to each arm). Each clinic signed a memorandum of agreement which outlined the responsibilities of the Medical University of South Carolina and their clinic. All study personnel at each site successfully completed regulatory training requirements to participate in clinical research.

Patients

All patients provided written informed consent, approved by the Medical University of South Carolina prior to participation.

Inclusion criteria

Eligible patients included adults ≥18 years old with clinic blood pressure above JNC 7 goals, while prescribed three or more hypertensive medications including a diuretic prescribed at ≥50% of usually accepted maximum dose as recommended in the guidelines or in the FDA-approved medication information sheet if not included in the guidelines. The patient must be adherent with prescribed medications as assessed by history and review of prescribed medication bottles.

Exclusion criteria

Patients with estimated glomerular filtration rates <50 ml/min, drug or alcohol abuse, myocardial infarction or stroke in the previous 6 months, or life expectancy or serious medical illness expected to change health status of <6 months, were excluded. A list of potentially eligible TRH patients from each practice was generated electronically from the OQUIN database.

Study Design

This CER pilot project was developed to assess the preliminary effectiveness of four efficacious treatment approaches (four arms) for lowering blood pressure and controlling hypertension in patients with treatment resistant hypertension. The study was also designed to assess patient, physician, and practice staff perceptions and opinions of protocol effectiveness and acceptability with regard to time and resources. The qualitative information is intended to guide the design of a subsequent larger trial. To accomplish these dual objectives, a mixed-methods (quantitative and qualitative) design was used.

The eight community-based clinical practices were assigned to one of four interventions using a cluster(clinic)-randomized process: The four interventions include: a) adding an aldosterone antagonist, b) referring patients to a clinical hypertension specialist c) renin-guided therapeutics, and d) combining renin-guided therapeutics and referral to a clinical hypertension special.^9–11

Quantitative methods

Change in blood pressure and hypertension control are the primary quantitative outcomes. This report focuses on the qualitative methods and data prescribed below as the study has not yet concluded and blood pressure data cannot be accessed prior to that time.

Qualitative methods

We used an ethnographic approach to observe how the research was implemented in the various practices, sharing insights with practice staff as an iterative process to both improve the current study and plan for the next.⁷

Prior to the study, a conversation was held with each lead physician about the four arms and other methods proposed to understand how the research would be implemented. Focus group discussions were then held at each practice site to inform the entire staff about their assigned intervention (arm), review protocol requirements, and answer questions. Focus group questions included an assessment of how each practice would implement their arm of the study, attitudes toward using the BpTRU and their arm (randomized assignment), and any challenges they may anticipate. Each practice identified a research coordinator who was trained in all aspects of conducting the protocol and completing case report forms. Focus group discussions were summarized and “field notes” were kept on all contacts with practice staff including questions they asked, challenges posed with potential solutions, and progress in enrolling subjects and following the protocol. Contacts were made through email, phone calls, and site visits.

After practices received orientation, conference calls were scheduled with lead physicians to review study progress share lessons they learned in implementing the four protocols. We were particularly interested in shared decision-making among the lead physicians and how new ideas were disseminated (diffused) among the group.

Management and analysis of qualitative data

Detailed notes were summarized of focus group discussions, group teleconferences, and contacts with individual physicians and study coordinators. The notes were analyzed by content analysis.¹³

Results

Focus group discussions were held with all practices. Saturation (no new information) was attained by the fifth focus group. Summaries of focus groups and field notes from research monitoring activities were reviewed by the two lead investigators and a list of facilitators and barriers to implementing the research protocol in practice settings was generated. Items were placed under content categories: Facilitators of research, University/regulatory barriers, Practice-structural barriers, and Practice-clinical barriers. Figure 1 presents the Categories and observations under each.

Figure 1.

The facilitators and three groups of barriers to comparative effectiveness research are listed. These factors emerged in implementing the protocol on treatment resistant hypertension in diverse community-based practice settings.

Facilitators

As shown in Figure 1, the key facilitators to implementing research include the enthusiasm of the lead physicians and their staff in participating in the study, their interest in changing their practice flow to accommodate enrollment and data collection and offering ideas to simplify the implementation of the project. As patients with a long history of TRH were finally controlled, there was added enthusiasm for the protocol and requests for more studies. Staff identified simple changes to practice routine that improved efficiency of patient screening and enrollment. For example, when patients and staff went through the lengthy process of completing consent forms, only to learn that the BP fell below inclusion levels for enrollment in the study, we adjusted the protocol to use the BpTRU as a routine part of clinical management of all TRH patients. When the BpTRU value was elevated, this was followed by an explanation of the study, completion of IRB forms in patients with an interest in participating, and then a repeat BpTRU as written in the original protocol. We learned early that organizing the case report forms to include a sequential list of all activities for each visit at the top of the page, improved fidelity to the protocol and completion of required data fields.

Regulatory Barriers

All practices complained about the lengthy consent forms, but this gave us an opportunity to reinforce the importance of insuring that patients/subjects are fully informed and have the option of refusing to participate without penalty. This is especially critical in a clinical practice where patients and their physicians know and like one another. In some small towns, they may also be neighbors or belong to the same church. The fact that one or more patients in all practices refused to participate indicates that patients feel free to refuse to volunteer. CER and PCT conducted in a variety of community-based practices requires that research is conducted by practice staff trained in protection of human subjects and where local administrators understand their role and agree to be responsible for the collection and maintenance of data. This takes additional time on everyone’s part to have “off site” forms signed acknowledging that regulations have been explained and will be implemented as required. The study required full IRB review and took three months until final approval.¹⁴ As we learned to navigate the realities of real world practices and identified facilitators and barriers to implementation, we wanted to alter previously approved post-intervention surveys and focus group questions to capture important data. These revisions required another two months for IRB review.

Practice Structural Barriers

Practices varied in size, location, and research experience. All stated they had some experience in the past, although this was not required as it would have limited the “real world” of practices. All lead physicians had current Regulatory certification through the Collaborative Institutional Training Institute (CITI).¹⁵ We did not dictate who their office research coordinators would be or what changes in office flow or organization would be needed to implement the study. We were surprised when some practices selected Licensed Practical Nurses, and in one case, a Certified Nursing Assistant, to schedule patients for screening and follow-up appointments and to maintain the case report forms. While all research coordinators were motivated to adhere with all aspects of the protocol, those with less education required more training and monitoring by the academic staff. Some physicians also needed additional training to adhere to all components of their study arm.

Conducting the study in the natural environment of a busy practice setting presented important barriers to subject enrollment: (a) office settings operate around the needs of patients, not the priorities of research (b) EHRS do not always capture accurate information on medications and comorbid conditions. Heavy patient flow during flu season or the start of school often brings recruitment to a stop. One physician in a very rural community was so busy he had to schedule the research on the only half-day he was not scheduled to be in the practice. Another lead physician left the practice and a replacement from that practice had to be identified and trained. Inaccuracies in the EHR, primarily in the number of current medications, which is well recognized, produced lists of potential patients that, in fact, were not treatment-resistant. ^16,17

Practice Clinical Barriers

Finally, there were a set of barriers to conducting research in community settings that reflect the realities of practice and the physicians who work in these settings. All practices were surprised to learn that 30% or more of their TRH patients were being treated for “white coat effect.” One physician joked, “The BpTRU is making it more difficult to recruit.” Another added, “The BpTRU cured some of my patients.” All were concerned that some patients were misdiagnosed and treated using the standard office equipment for BP measurement. Two practices purchased an additional BpTRU to use with all patients with apparent hypertension.

Two of the arms (4 practices) required consultation or referral to a Certified Hypertension Specialist. We generated lists of current Specialists within 150 miles of their practice with contact information and included reimbursement for the consultation in our contract with the practice. The four physicians all agreed to and were trained in their arm of the study and all recognized the names of several specialists in their area. However, one physician refused to call or refer patients (“I know what I am doing and don’t need to refer”) and another avoided specialists close by in favor of someone in a city 100 miles distant (“I am more comfortable with someone in another city” (lower chance of losing patients). It was also noted that some enrolled patients were being treated with doses of medication substantially below 50% of the maximum recommended dose and were controlled with up-titration of ‘under-dosed’ antihypertensive medications.

Finally, several physicians noted that their TRH patients had not been returning for monthly appointments before they enrolled in the study due to heavy patient volume and concluded that a key change in practice policy was needed to improve their own adherence to national treatment standards. In fact, one unanticipated effect of the study was to make some lead physicians more aware of practice guidelines that had not been consistently followed in their practices. We obtained anecdotal evidence that participation in research improved adoption of national practice guidelines for TRH.

Discussion

The key findings are that conducting research in the real world setting of busy practices, while critical to determining effectiveness, requires additional time, effort and expense compared with conducting research in academic and other research-intensive settings. Although we encountered challenges that our pre-proposal planning and training sessions did not detect, the lessons learned and changes practices made in clinical policies to improve routine care of patients produced results that went beyond enrolling patients and collecting data. The results of this pilot study are critical to designing the larger, definitive study. A key early finding is that physicians and their staff became more aware and critical of their management of patients with hypertension through their participation in the study.

There are several caveats. First, although the study was conducted in practices of varying size, type, resources and geographic location throughout South Carolina, they are not representative of all primary care practices. For example, we did not include free clinics or the VA. Second, these findings are preliminary as the study is continuing and surveys and interviews with research subjects have not been completed. The patient’s perspective may be particularly helpful in guiding future CER.

We concur with Kessler and Glasgow (2011) that conducting research in the “messy, real-world, under-resourced public health setting” of community practices is critical to testing approaches that address the critical question: “will this work in our setting?”¹⁸ Our experience with CER in community practices mirrors experience of others with a few notable exceptions. For example, we found that including practice staff in the planning and implementation of the research, providing training and ongoing monitoring and being flexible in how the protocol is organized in the practice setting are critical to a successful project.^4,19,20 This requires working within the realities of a busy practice and ensuring they are fairly compensated for time and effort. We purposely selected a wide variety of practices to test our interventions in the “real” world. However, in selecting some practices that were under-resourced with little research experience, we underestimated the time and effort required to address regulatory and university administrative issues, along with needs for ongoing training and monitoring of practice staff.

Our pre-proposal planning did not identify referral to a hypertension specialist as a problem. Thus, we did not anticipate that adherence to two arms of the protocol would present a problem to some physicians who decided referral to a hypertension specialist outside their group was unnecessary or that they might lose patients. This represents a significant concern in practices often compete for patients and primary care is under-funded. It also underlines the importance of continuous monitoring to identify these problems to identify potential solutions or to revise the intervention in future trials.

One of the strengths of our study was integrating qualitative methods in key phases to inform the current and future larger-scale study. This allowed us to describe the context in which CER is conducted and led to some important lessons learned.

Learn about how each practice treats the condition under study

Do not make assumptions about standards of care in each practice. Having an accurate diagnosis, in our case TRH, reduces time in screening and enrolling unqualified patients. Conversely, understanding the processes of care and identifying barriers to implementation and fidelity may produce some of the more important findings about “how things work here.”

Add planning time and formative work before the study begins

Clinical trials conducted in an academic setting are typically well resourced with experienced clinicians and research coordinators. Clinicians who work in the community are focused on the realities of running a busy office, with fewer support staff to insulate them from competing needs of sick patients while conducting research. Each practice is different. Moreover, the existing staff members have a full slate of clinical responsibilities and obtaining significant relief from those activities to conduct research is challenging. Thus, existing staff must work more efficiently within the usual work day to accommodate research responsibilities. It is important to build in flexibility in how protocols are implemented, without jeopardizing the aims of the study. Work with practice staff to simplify the protocol as much as possible. Ask staff to review case report forms to make sure they are detailed enough to insure proper work flow, fidelity to the protocol, and completeness of data capture, while also keeping them simple and within practice workflow patterns.

Build in extra time and costs to train, re-train and monitor research coordinators and physicians

Some practices are very small with few support staff. Any change in personnel can require training of new personnel. Delays in starting the study after the initial training or interruptions due to heavy patient volumes may require re-training. The research team must be readily available to answer questions and work through minor protocol changes.

The R34 is ideal for pilot work but has limitations

Pilot studies aim to test feasibility of the protocol and to establish power for a larger study. CER, conducted in real community practices, requires considerable formative work to tailor the research to a variety of practices along with qualitative methods to determine if a protocol(s) will really work in some or all practices. The amount of funding may not be sufficient to learn all that is needed to design a complex, 4-arm, large-scale effectiveness study.

The study must be relevant to the needs of the practice, cover costs of practices to conduct the research, and acknowledge the hard work and creative input of the staff

The lead physicians all agreed that TRH was a challenge in their practices, a fact confirmed by our database. Each component of the medical protocol, in addition to focus groups, interviews, and meetings must all be seen as line items in the budget of each practice. A plan must be developed as part of the proposal to give recognition to practices and their staff through publications, presentations, and abstracts. CER is a team effort. The real experts in the “real” real world are those who live and work in it.

In summary, our ethnographic analysis of a CER pilot highlights the importance of understanding the mechanisms underlying how efficacious interventions are (or are not) implemented in various practice settings. We look forward to learning more about this process from the subjects themselves. The key lessons learned provide a foundation for improving the design, conduct and evaluation of comparative effectiveness research.