A Model for Incorporating Patient and Stakeholder Voices in a Learning Healthcare Network: Washington State’s Comparative Effectiveness Research Translation Network (CERTAIN)

Abstract

Objective

To describe the inaugural comparative effectiveness research (CER) cohort study of Washington State’s Comparative Effectiveness Research Translation Network (CERTAIN), which compares invasive to non-invasive treatments for peripheral artery disease; to focus on the patient-centeredness of this cohort study by describing it within the context of a newly published conceptual frameworks for patient-centered outcomes research (PCOR).

Study Design and Setting

The peripheral artery disease study was selected due to clinician-identified uncertainty in treatment selection and differences in desired outcomes between patients and clinicians. Patient-centeredness is achieved through the ‘Patient Voices Project’, a CERTAIN initiative through which patient-reported outcome (PRO) instruments are administered for research and clinical purposes, and a study-specific patient advisory group where patients are meaningfully engaged throughout the life cycle of the trial. A clinician-led research advisory panel follows in parallel.

Results

Primary outcomes are PRO instruments that measure function, health-related quality of life, and symptoms; the latter developed with input from patients. Input from the patient advisory group led to revised retention procedures, which now focus on short-term (3–6 months) follow-up. The research advisory panel is piloting a point-of-care, patient assessment checklist, there by returning study results to practice. The cohort study is aligned with the tenets of one of the new conceptual frameworks for conducting PCOR.

Conclusion

CERTAIN’s inaugural cohort study may serve as a useful model for conducting PCOR and creating a Learning Healthcare Network.

INTRODUCTION

Despite large volumes of research generated by investigators, healthcare’s stakeholders—patients, clinicians, policymakers—remain frustrated by the lack of adequate evidence to guide decisions. [1,2] Conventionally, investigators formulate research questions drawn from their own perspectives. However, this approach has too often meant that study results are not aligned with stakeholder needs. [3,4] Evidence from other fields suggests that involving stakeholders in evidence generation contributes to more successful project outcomes, [5] more rapid dissemination of findings,[6] and improved transparency of research organizations. [6,7] Further, patient stakeholders are the beneficiaries of these outcomes. [8–10] Despite growing recognition of the importance of stakeholder engagement, methods to achieve engagement (particularly for patients) are still being developed.

Patient-centered outcomes research (PCOR) is now a topic of national policy interest. [11] PCOR is related to comparative effectiveness research (CER) and keeps the interests and perspectives of patients, caregivers, and other stakeholders central to every component of the research process. The National Institutes of Health (NIH) National Center for Advancing Translational Sciences (NCATS) Clinical and Translational Science Awards (CTSAs) support both a CER Key Function Committee, whose mission is to build the fields of CER and PCOR by creating a learning community, [12] and a Community Engagement Key Function Committee, whose mission is to effectively engage communities and practices in the translational research process. [13] Jointly, these two Key Function Committees reflect current trends in integrating PCOR and stakeholder engagement.

Several investigator groups are addressing these gaps and proposing conceptual frameworks for stakeholder involvement in PCOR. [14–16] This emerging body of work suggests that stakeholders include not only patients, but also communities and organizations that have a direct interest in research outcomes and corresponding policy decisions. It suggests that ‘engagement’ means actively soliciting the experiences and values these stakeholders bring to the table, involving them throughout the project life-cycle. Acknowledging that engaging stakeholders is both challenging and time-consuming, these investigators recommend that these frameworks be tested in the field and that best practices be developed.

In this manuscript we describe a multisite, longitudinal, prospective, observational cohort study grounded in PCOR, and conducted by Washington State’s Comparative Effectiveness Research Translation Network (CERTAIN). We outline the ways in which patients and other stakeholders are being incorporated into all aspects of research in the context of a cohort study to compare invasive and non-invasive treatments for peripheral artery disease, and describe how results are being returned to practice. Finally, we test the patient-centeredness of this cohort study by describing it in the context of a newly published conceptual framework for patient-centered outcomes research (PCOR).

METHODS

Context

CERTAIN is a PCOR initiative that emerged from our team’s experience creating Washington State’s Surgical Care and Outcomes Assessment Program (SCOAP), a clinician-led performance surveillance and quality improvement (QI) initiative. [17] Trained SCOAP abstractors gather patient data from medical records for each surgical (vascular, general, spine, pediatric, oncologic) hospitalization. SCOAP is administered by Washington State’s Foundation for Healthcare Quality [18] and supported by grants from the NIH and Washington’s Life Sciences Discovery Fund [19]. Now in place for seven years, 55 of the 60 hospitals (92%) in Washington State currently participate.[20]

Leveraging from SCOAP, CERTAIN is a cross-disciplinary research initiative led by University of Washington investigators with expertise in CER, stakeholder engagement, technology assessment, health economics and policy, and improvement science. CERTAIN investigators are automating data abstraction from multiple electronic health records (EHRs) and evaluating data across care delivery sites, thereby advancing QI efforts and leveraging the QI registry to build a research network to design and conduct stakeholder-informed CER. As first use of the CERTAIN research initiative, CERTAIN investigators are conducting the cohort study of peripheral artery disease, described in more detail below.

Patient Voices Project

Patient stakeholder engagement mechanisms are included in CERTAIN’s ‘Patient Voices Project,’ which gathers patient-reported outcomes (PROs) using multiple modalities of communication, called the ‘Patient Survey Center’; leads clinical, condition-specific patient advisory groups; and provides a pathway to sustainability for community-based PRO gathering that returns PRO survey data to clinicians at the point of care, called ‘PROs in Practice’.

The Survey Center manages survey distribution and follow-up to collect PROs. Survey distribution methods, response data collection, and reporting are customized by project. The survey experience is personalized by applying patient contact preferences to survey modalities. Standard operations include four contact modalities (postal mail, electronic mail, Short Message Services [SMS], telephone) and four survey response formats (paper, web-based, live phone operator, automated phone). Detailed algorithms for patient follow-up are employed to maximize data capture; patients are contacted up to 13 times using various modes before being considered lost to follow-up. The Survey Center has contacted over 12,000 patients in its first two years of operation.

Within each SCOAP-defined clinical area, CERTAIN investigators create a patient advisory group, identifying patients with whom to collaborate in the research process. CERTAIN is particularly interested in recruiting patients affected by the disease under study. The reason for working with patients per se rather than patient advocates is that the goal of CERTAIN is to improve the care those very patients receive in a specific health system. Invitations are extended to patients whose physicians believe they may be interested in participating. Public outreach through a web interface (www.becertain.org) also invites patient stakeholders. Introductory materials are sent to potential participants; staff provide individual follow-up to nurture each relationship. Once patient stakeholders are on board, they participate for up to 12 months and are regularly engaged through a series of meetings, follow-up check-in times, and ongoing electronically generated requests for feedback. Data are collected about feedback provided by patient advisory groups, and the impact on study procedures evaluated.

PCOR Study in Peripheral Artery Disease

The 2007 TASC II consensus document estimates that 27 million individuals in Europe and North America are affected by peripheral artery disease, [21] with a US prevalence of 14.5% for those over 70 years of age. [22] Intermittent claudication is the most common presentation of lower-extremity peripheral artery disease; disease progression limits walking and decreases quality of life (QoL). Over $4.37 billion was spent on treatment in 2008, [23] but it is not clear whether more invasive interventions (e.g., open surgery, endovascular stenting) provide superior outcomes compared to conservative treatments (walking program, smoking cessation counseling, medications). Further, there is wide variation in the types of procedures used to treat Medicare beneficiaries. [24] Adding to this uncertainty is the fact that clinicians and patients offer different definitions of what constitutes improved outcomes. Clinicians focus on functional improvement; patients seek improved health-related QoL (HRQoL). [25]

To address this uncertainty, CERTAIN is conducting a multisite, longitudinal, prospective, observational cohort study of peripheral artery disease. [26] The intent of this study is to determine the effects of treatments for one type of peripheral artery disease, intermittent claudication, on outcomes of primary interest to patients - function and QoL - under conditions encountered in community practice. [27] While specific to patient involvement, Mullins’ conceptual framework [16] is ideal for describing the involvement of diverse stakeholders in PCOR (Figure 1); we adapted it to describe the patient-centeredness of this cohort study.

Figure 1. Mullins’ 10-Step Process for Enhancing CER through Continuous Patient Engagement¹⁸.

CER = Comparative Effectiveness Research; CERTAIN = Comparative Effectiveness Research Translation Network; PCOR = Patient-Centered Outcomes Research; PRO = Patient Reported Outcome

RESULTS

We first leveraged relationships with SCOAP sites and clinicians, as data collection mirrored that already taking place for vascular surgery. Relationships with sites providing endovascular or conservative (non-surgical) care were developed de novo. As intermittent claudication is treated by vascular surgeons, interventional radiologists, and cardiologists, all were engaged. Site and clinician engagement began in late 2010 and is ongoing. Recruitment has required concerted efforts on the part of physician leaders to engage their community-based colleagues. To date, fifteen sites are participating.

Topic selection and prioritization (Mullins’ framework, Steps 1 and 2) for the cohort study were completed while CERTAIN and formal PCOR were in their nascent stages. The decision to compare the effectiveness of three approaches to the treatment of claudication (surgical intervention, endovascular intervention, and conservative management) was based on the variation in treatments employed and uncertainty in the optimal strategy required to improve the lives of patients. Despite surgeons’ successful restoration of their patients' leg patency, those patients report no corresponding improvement in healthrelated QoL (HRQoL). [25] Together, CERTAIN investigators decided to evaluate function and HRQoL as primary study outcomes and to elicit this information directly from patients using PRO instruments.

Investigators spent several months developing the study protocol and formulating hypotheses to compare surgical, endovascular, and conservative care. (Step 3) Finalizing the three comparison groups, selecting specific PRO instruments with which to measure outcomes (Step 4), and developing the endpoint model to guide the study (Figure 2; Step 5) were completed concurrently. The endpoint model describes patient attributes, risk factors and co-existing conditions that lead to the signs and symptoms of claudication. Signs and symptoms, in turn, inform decisions about how to best intervene, both directly and through risk-adjusted clinical events. Both signs and symptoms, and types of interventions impact concepts most important to patients: walking ability, and HRQoL (both disease-specific and overall). (Figure 2)

Figure 2. CERTAIN Peripheral Artery Disease (Claudication) Study Endpoint Model.

EQ-5D = Euro-QoL-5D; SES = Socio-economic status

Detailed discussions took place to define inclusion/exclusion criteria; to identify important patient attributes, risk factors, co-morbid conditions, clinical events and follow-up; and to finalize important time points and secondary clinical outcomes. Adults experiencing claudication of infrainguinal origin were included. In the spirit of community-based care, investigators included patients with all levels of disease severity and any number of previous treatments, choosing to address these confounders during data analysis. [28,29] Relevant time-points at which to evaluate outcomes are pre-treatment (baseline), 30 days, and both 6 and 12 months after enrollment.

Investigators next selected the most appropriate PRO survey instruments by first identifying disease-relevant PRO domains (functional disability, HRQoL, symptoms). (Step 5) Selected instruments demonstrated content validity in a population with peripheral artery disease. Investigators reviewed additional measurement attributes of each instrument: reliability, responsiveness, meaningful interpretation of change, and burden to both administrators and respondents. [30] The Walking Impairment Questionnaire was selected to assess functional disability; [31] the Euro-Qol-5D (EQ-5D), preference-based HRQoL; [32] and the VascuQoL, disease-specific HRQoL. [33] The Walking Impairment Questionnaire consists of three domains – walking speed, walking distance, and stair climbing – and has been validated against treadmill tests in patients with peripheral artery disease. A change in each of the scores on these three domains is the outcome on which the study is powered (600 patients). Investigators from the PRO Core were unable to identify a claudication symptom instrument that met the selection criteria described above, so internal development was undertaken. Engaging the first ten patients who consented to participate in the pilot study to cognitively test the battery of the other three instruments, the PRO investigators conducted focus groups to elicit perceptions of disease symptoms. The result is a patient-derived five-item symptom instrument that will be validated during the study.

The statistical analysis plan was developed concurrently [28,29] (Step 6) This was a natural follow-on to the discussions described above. The biostatistician, a member of the CER Methods Core, guided deliberations and provided insight. The primary hypothesis is that, at 12-months post intervention, patients undergoing surgical and endovascular procedures will experience greater improvements in function, health-related quality of life, and claudication symptoms than patients in the conservatively managed cohort. Using 80% power to estimate a 15% reduction in walk speed, distance and stair climb over 12 months in the conservatively managed cohort, versus a 5–7% reduction in each domain in the surgical or endovascular cohorts, and a two-sided α of 0.05, requires a sample size of 200 patients in each arm. The analysis plan describes the longitudinal methods that will be employed to assess temporal changes in study outcomes: generalized estimating equations with robust inference, propensity scores or instrumental variables to address selection bias and confounding, multiple imputation to impute missing data from PRO survey instruments, and time-varying covariates to characterize patients who cross over from one cohort to another.

The Investigators finalized data collection methods (Step 7), which include medical record and telephone screening to identify patients, medical record review to abstract clinical data about the index hospitalization, and administration of patient survey instruments to collect baseline health history and PROs. Enrollment began in July 2011 and is ongoing. To date, over 5,500 patients have been screened and over 1,000 (18%) claudicants identified. Of these, 570 (57%) have failed screening procedures, primarily due to anatomic location of disease (aorto-iliac) or disease severity (limb threat, where surgery is required).

The Survey Center is employing all survey distribution and data collection methods as well as follow-up algorithms to maximize PRO data collection at all study-designated time points. Early on, because they realized that each survey response mode might not provide equivalent measures of the PRO data collected, investigators are currently conducting a ‘mode of administration’ sub-study to establish whether analytic adjustments will be warranted. This sub-study is block-randomizing 72 patients with leg pain who are not eligible to enroll in the peripheral artery disease study and administering to them the Walking Impairment Questionnaire and EQ-5D using each of the four modes of administration once within one week.

Although not explicitly listed as one of Mullins’ ten steps to transform CER into PCOR, recruitment and retention are important aspects of any PCOR study. We insert these between Steps 7 and 8. Over 420 of 1,000 claudicants (42%) have passed screening and have received study packets; over 200 of these have consented and are enrolled, the majority in the conservatively managed cohort. One-hundred eight-five of 1,000 (19%) declined to participate; general lack of interest and time concerns are most frequently cited as reasons. This is despite institutional review board-approved financial incentives offered for participation. Early feedback from patients in the patient advisory group, which first met in February 2012, indicated patients preferred outreach to take place during the early months (3–6) of recovery after treatment. During this timeframe, they are interested in what to expect from the recovery process and how soon they will return to “normal” function. At 9–12 months, these patients stated that claudication would no longer play a large role in their lives, and that information distributed at that time would likely be ignored. With this feedback, retention procedures were modified to include more detailed information about patient recovery at earlier stages, saving simpler outreach tasks (thank-you postcards, quick check-in phone calls) for later. As the patient advisory group continues to meet, investigators will engage members to identify strategies for interpretation, translation, and dissemination of study results, including use of the CERTAIN website. (Steps 8, 9, and 10)

Finally, the research advisory panel for the cohort study is comprised of vascular surgeons, interventional radiologists, and cardiologists. Because twelve-month data are now available for patients enrolled during the first year of the study (July-June 2012), research advisors are reviewing and interpreting these data and translating these back into clinical practice (Steps 8 and 9). Early findings indicate that physicians counsel patients on smoking cessation 35% of the time and record scores on the Rutherford Scale (the standard measure used to indicate claudication severity) 50% of the time. [34] Thus, investigators have created a checklist for clinician use at the point of care. The checklist lists risk and clinical factors to assess when making treatment decisions and provides basic recommendations for care. It also includes patient assessments of HRQoL. To develop the latter, investigators reviewed focus group transcripts of patients who participated in the development of the symptom measure. The checklist has now been fully vetted, is currently being piloted and will eventually be incorporated at the point of care, integrated into clinic workflow, delivered via the EHR, and made available at each patient visit to facilitate patient-provider interaction and to inform longitudinal patient-centered care. Employing methods from the field of improvement science, investigators will also formally evaluate whether use of this new checklist improves process and outcome quality measures for patients with claudication.

DISCUSSION

The inaugural CERTAIN cohort study in claudication is being conducted in a community-based, heterogeneous population; compares invasive to non-invasive treatments; and uses PRO instruments to assess primary outcomes that are meaningful to patients. In 2009, grantees receiving American Reinvestment and Recovery Act funds, [35] including CERTAIN investigators, were charged with conducting CER that involves several types of stakeholders. When the Patient Protection and Affordable Care Act was passed in 2010, [36] CERTAIN investigators further enhanced the patient-centeredness of the claudication study to include patients, and are now conducting PCOR. The study employs a dedicated PRO Survey Center as well as and patient and research advisory panels. The results of the CERTAIN claudication study will provide high-quality evidence to aid community-based clinical decision making and improve HRQoL outcomes for patients with claudication.

The structure of CERTAIN and the claudication study illustrate how patient engagement transforms CER into PCOR. It also illustrates the challenges inherent in engaging busy community practitioners and patients; the PCOR paradigm is new for both. Clinicians may not share investigators’ passion for creating the Learning Healthcare System. [37] Most patients are not accustomed to participating in clinical trials, and are certainly not familiar with their new role as stakeholders. Yet, in the CERTAIN claudication study, patient engagement has proven valuable in making iterative improvements in study execution. In future studies, earlier engagement of patients will be helpful in identifying research questions of greatest value to them.

Applying the Mullins’ framework, [16] albeit retrospectively, is a useful way to explain how patients and other stakeholders can be engaged from project inception to dissemination of results. Nevertheless, we could have applied other frameworks. Deverka’s Analytic Deliberative Model emphasizes the consideration of different points of view with the goal of reaching a reasoned decision, and often involves formalized procedures to ensure adequate exchanges of views. Deverka’s group applied their model to the Center for Comparative Effectiveness Research in Cancer Genomics project (CANCERGEN). [14] Concannon’s Six-Stage Model outlines activities that must be carried out by researchers at each stage of the research process, when surrounded by stakeholders. The model illustrates both a sequential flow from beginning to end of a project and the cyclical nature of that flow. [15] All three existing frameworks provide definitions of stakeholders and stakeholder engagement, span the CER life-cycle, and make CER more patient centered. [14–16] It is too early to tell whether investigators will agree on one, or will find multiple frameworks useful, depending on the research question at hand.

Taking sufficient time to develop a sustainable infrastructure is well worth the considerable investment. Efficiencies in conducting subsequent studies in other clinical disciplines will be realized now that the CERTAIN enterprise is fully functioning. We continue to engage additional ambulatory surgery centers and primary care practices to create non-interventional comparator cohorts for future studies. With payers and the Washington State Health Technology Assessment body, we are also discussing programs that involve coverage with evidence, either within or alongside research. [38] CERTAIN is positioned to answer important research questions in a scalable and sustainable way for patients across Washington State.

CONCLUSION

The CERTAIN initiative is a viable and valuable mechanism for conducting PCOR. The Patient Voices Project is enabling development of a community-based PRO registry that incorporates project-specific patient advisory groups, uses multiple methods for collecting PRO data, and employs mechanisms for incorporating PROs into clinical practice. Patient and research advisory panels are engaged in and guiding all aspects of research. Making use of CERTAIN's multiple components, investigators are building a meaningful infrastructure for conducting PCOR, while simultaneously creating a Learning Healthcare Network.