Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Sep 28.
Published in final edited form as: Med Care. 2018 Oct;56(Suppl 10 1):S58–S63. doi: 10.1097/MLR.0000000000000791

Building Meaningful Patient Engagement in Research: Case Study from ADVANCE Clinical Data Research Network

Nathaniel T Warren 1, James A Gaudino Jr 1,2, Sonja Likumahuwa-Ackman 1,2, Kristin Dickerson 3, Lynn Robbins 3, Kathy Norman 3, John Lind 3, Sele D’Amato 3, Perry Foley 1, Rachel Gold 1,4, Vance Bauer 1, Scott A Fields 1,2, Deborah J Cohen 2, Khaya D Clark 2, Jennifer E DeVoe 1,2
PMCID: PMC6136943  NIHMSID: NIHMS893404  PMID: 30074953

Abstract

Background

Strategies to engage patients to improve and enhance research and clinical care are increasingly being implemented in the United States, yet little is known about best practices for or the impacts of meaningful patient engagement.

Objective

We describe and reflect on our patient stakeholder groups, engagement framework, experiences and lessons learned in engaging patients in research, from generating proposal ideas to disseminating findings.

Setting

The ADVANCE (Accelerating Data Value Across a National Community Health Center Network) clinical data research network (CDRN) is the nation’s largest clinical data set on the safety net, with outpatient clinical data from 122 health systems (1,109 clinics) in 23 states.

Results

Patients stakeholders co-developed the ADVANCE engagement framework and its implementation in partnership with network leaders. In Phase I of ADVANCE, patients were involved with designing studies (input on primary outcome measures and methods) and usability testing (of the patient portal). In Phase II, the network is prioritizing research training, dissemination opportunities, an “ambassador” program to pair more experienced patient stakeholders with those less experienced, and evaluation of engagement activities and impacts.

Discussion

The ADVANCE framework for patient engagement has successfully involved a diverse group of patients in the design, implementation, and interpretation of comparative effectiveness research. Our experience and framework can be used by other organizations and research networks to support patient engagement activities.

Keywords: Consumer engagement, underserved populations, primary health care, patient centered outcomes research

“We are all patients – you have the whole community to draw from for your patient advisors. Look to the people who you respect. Reach out to those who inspire you personally to find individuals willing to meaningfully give back from their life experience, because these patients have important opinions on this work. Involve community leaders, scientists, nurses, doctors, clinic managers, educators. Ask for their recommendations. The individuals you find will be a joy to work with and will drive great improvements. Traditional medicine will only recognize breakthroughs after they have been thoroughly studied and documented, and I chafe at the pain and loss caused by that delay. Having a small voice in the world of medical research enables me to participate where benefit will eventually be brought to the greatest number of people.”

ADVANCE Patient Investigator Lynn Robbins

INTRODUCTION

As consumers of health care and stakeholders in clinical research, the importance of patients’ “real-world” insights has led to a burgeoning movement to engage patients in health care decision making, quality improvement initiatives and more recently, research. The body of literature speaks to a need to clearly define the scope of “meaningful patient engagement” and how it can be effectively implemented. Limited research exists around identifying and recruiting patients, equitable compensation processes and best practices in providing research guidance.1–8 Furthermore, knowledge gaps remain on building an effective engagement infrastructure to address power differentials for patients joining research teams, as well as providing and testing engagement tools, role descriptions and resources to aid in this re-visioning of patient-centered outcomes research (PCOR).9–10

Founded in 2001, OCHIN (Oregon Community Health Information Network) is a nonprofit, community-based health information technology collaborative that serves 97 health systems (Federally Qualified Health Centers, community health centers, critical access hospitals, and rural hospitals) in 18 states across the nation, linking 618 clinics with 4,927 providers serving >2.2 million patients 11–15 in one common electronic health record. OCHIN leads the ADVANCE (Accelerating Data Value Across a National Community Health Center Network) clinical data research network (CDRN) in partnership with Health Choice Network and Fenway Health. The network reaches 23 states and is the largest clinical data set on the safety net in the nation. It has a truly national footprint, and engages patients, clinicians, health systems, and researchers.

Community health centers (CHCs), which comprise the majority of the ADVANCE network, have a long history of engaging patients on their community-based governing boards.16 Before ADVANCE, OCHIN researchers had developed mechanisms (e.g., community retreats, focus groups, patient interviews, and project advisory councils) for engaging patients and communities for specific research projects. However, OCHIN had not explicitly structured research governance in a way that provided ongoing structure for direct patient engagement across projects and organizational services.17 Through ADVANCE, our network is able to build on the CHC tradition of patient engagement in a new era of patient-centered outcomes research.

Driven by a goal to fundamentally change the health research landscape, the Patient Centered Outcomes Research Institute (PCORI) funded several research networks, like ADVANCE, to enable big data comparative effectiveness research that is driven by patient priorities.18–20 In this paper, we describe and reflect on our patient stakeholder groups, our engagement framework and our experience in engaging patients and other stakeholders in research, from generating proposal ideas to disseminating findings.

DESCRIPTION OF ENGAGEMENT GROUPS AND ACTIVITIES

The ADVANCE network began Phase I in 2014 through an award from PCORI as a partner in their development of PCORnet: The National Patient-Centered Clinical Research Network. ADVANCE specifically contributes to this network as it brings vulnerable and diverse FQHC patient populations to PCORnet’s patient-centered outcomes research. PCORI defines PCORnet as “a large, highly representative, national ‘network of networks’ that collects data routinely gathered in a variety of healthcare settings, including hospitals, doctors’ offices, and community clinics. By engaging a variety of stakeholders – patients, families, providers, and researchers – PCORnet empowers individuals and organizations to use this data to answer practical questions that help patients, clinicians, and other stakeholders make informed healthcare decisions.”21

Phase II of ADVANCE through PCORnet seeks to expand on its Phase I goals of expanding our stakeholder engagement and community-academic partnerships; integrating outpatient, hospital, and community-level data into a single data management system; and build on our “community laboratory” of FQHCs jointly created by our patient, clinician and health system leader stakeholders. Our Phase II project summary describes this effort as such: “The mission of the ADVANCE CDRN—a team of organizations from around the country—is to learn about how to improve the health of safety net patients, including people living in poverty, with little or no insurance…In the past, vulnerable patient populations have not been included in many studies. As a network of safety net clinics, ADVANCE is able to help bridge this gap. Patients, caregivers, and clinicians are helping guide the ADVANCE research plan, develop study questions and materials, and share findings. By taking part in ADVANCE projects, people have an opportunity to improve care for themselves, their families, and their communities.”22

The ADVANCE CDRN engages patients and other stakeholders through four groups: the OCHIN Patient Engagement Panel (PEP), the Clinic and Patient Engagement Workgroup (CAPE), the Community Research Outreach and Dissemination Program (CROP-D) and finally, the ADVANCE Advisory Council (AAC), which oversees all research conducted using the ADVANCE network, includes members from each of the ADVANCE data partners as well as patient (2) and clinician (1) stakeholders. ADVANCE data partners include the following organizations: OCHIN, Legacy Health, Health Choice Network, Kaiser Permanente Northwest Center for Health Research, Fenway Health, CareOregon Medicaid Managed Care Plan, Oregon Health and Science University, and the Robert Graham Center.22

OCHIN Patient Engagement Panel (PEP)

Building on these early efforts, OCHIN established a PEP to include patient voices and perspectives. Appendix 1 describes the timeline of the PEP as part of the development of the OCHIN Practice-Based Research Network (PBRN) and ADVANCE research network. The details of the steps we took to establish the PEP and our lessons learned are described in a prior publication 23. Our founding PEP members co-developed the selection process of additional members, compensation policy, recruitment methodologies and health literacy review to materials. During ADVANCE Phase I, the work of PEP was generally expanded to support research capacity building and the ADVANCE Engagement Framework was developed (see Figure 1).

Figure 1.

Figure 1

Open in a new tab

ADVANCE Engagement Rubric

Currently, there are 18 active patient advisors on the PEP, representing diverse backgrounds and lived experiences across multiple states, urban/rural locations, races/ethnicities, ages, genders and sexual orientations. Patient advisors provide ongoing guidance to improve research recruitment, retention and transparency. Through ADVANCE, PEP members have increased their involvement as co-investigators directly advising on prepared and/or submitted proposals (from three to nine members advising on more than 22 projects). One member also successfully received two PCORI Pipeline to Proposals Tier One and Tier Two awards and has been building a separate patient-led collaborative research team, which now is poised to begin developing the team’s first comparative effectiveness research funding proposal(s).24

Clinic and Patient Engagement (CAPE) Workgroup

In June 2014 of ADVANCE Phase I, the Clinic and Patient Engagement (CAPE) Workgroup was created. The CAPE is a tactical group for identified operational, clinical and informatics staff from ADVANCE organizations (e.g. OCHIN; Health Choice Network; Fenway Health; Legacy; OHSU), OCHIN health systems clinical oversight, research team partners and select patients from the PEP. The workgroup provides feedback on an ongoing basis and learn about the challenges and barriers around increasing patient portal adoption rates, enhancing patient engagement (via research projects, patient outreach strategies and additional tools), building shared communication tools, and sharing best practices and success. Patient, clinician and health system leader advisors provide ongoing guidance to overcome barriers in expanding patient portal adoption and use, such as co-learnings around health literacy, technology access in the safety net, Spanish language resources, patient and clinic-facing instructional guides and patient portal workflow development.

Community Research Outreach and Dissemination Program (CROP-D)

ADVANCE network infrastructure funding has allowed OCHIN to build the Community Research Outreach and Dissemination Program (CROP-D) as part of its operations model in order to support and extend its engagement work with stakeholders. Established towards the end of ADVANCE Phase I, the CROP-D is tasked with: coordinating stakeholder engagement workgroups; developing protocols and tools to support member recruitment and participation in research projects; collecting data on member organizations’ research priorities and involvement; and disseminating research activities. The CROP-D is staffed from a variety of OCHIN departments and roles, including an Engagement Coordinator, a Site Principal Investigator, a Practice Facilitator, a Research Associate and an Account Manager.

Practice-Based Research Network (PBRN)

THE ADVANCE PBRN, established in 2007, is housed at OCHIN to maximize its independence and focus on community-based research. Members include network clinicians and operational staff, patient advisors and research leadership to provide evaluation and assessment for our research endeavors. Members review proposals, share models and methodologies, and disseminate findings.

ADVANCE Advisory Council (AAC)

Early in ADVANCE Phase I, the ADVANCE Advisory Council was created to review and approve ADVANCE-specific policies, serve as the review board for any potential proposal, project or data request utilizing ADVANCE, and develop and prioritize the ADVANCE research agenda. This model of shared decision-making is similar to that of the OCHIN and Health Choice Network community Boards of Directors. To ensure that ADVANCE is well integrated and attached to OCHIN’s formal governance structure, the Advisory Council is integrated directly into the organizational structure of the OCHIN Board of Directors, as encouraged by OCHIN’s bylaws for all projects of significance to OCHIN’s members. The Advisory Council includes representatives from each of the ADVANCE data partners (further described above), as well as patient and clinician representatives.

ADVANCE Engagement Framework

ADVANCE currently uses a three-level framework to guide and prioritize our strategies for engagement, jointly developed from our early PEP work and our patient, clinician and health system leader stakeholders (Figure 1). We actively recruit patients as investigators to advise and inform all aspects of ADVANCE (Engagement Level 1). We expanded the PEP, created a Clinic and Patient Engagement (CAPE) Workgroup, and recruited patient representatives for the ADVANCE Advisory Council and Practice Based Research Network (Engagement Level 2). To further increase our ability to reach, inform and engage more patients (Engagement Level 3), patient advisors offer ongoing guidance for the development and implementation of enhanced electronic tools for patients (e.g. patient portals, websites, social media). Each of these groups serves an important purpose in bringing the voices of our key patient stakeholders into institutional strategies and leadership structures, and OCHIN has a dedicated budget for patient engagement activities.

IMPACT OF ENGAGEMENT

Through the initial creation of our engagement groups and framework described above through ADVANCE Phase I, we developed the structure in which to involve a diverse group of patients in the design, implementation, and interpretation of comparative effectiveness research. As we were awarded Phase II funding, we were able to build further on our stakeholder-identified engagement priorities of training, dissemination and our ambassador program.

PHASE I ENGAGEMENT

Designing Studies

In the PEP and CAPE through Level 2 of our engagement framework, patient investigators reviewed and edited insurance support tools from a patient perspective, engaging in four project phases: proposal development; adapting study methods; understanding the context and tool testing and implementation.25 PEP guidance has resulted in a change of a proposal’s primary outcome measure following stakeholder investigator calls and proposal reviews, as well as guidance from presentation at the PEP. ADVANCE patient investigator guidance (through Level 1) resulted in changes to proposal methodology, such as the addition of domains and enhanced frequency of patient-reported outcome (PRO) measures to include pre- and post-treatment. Additionally, through this same study, patient investigator guidance on the level of burden that patients with the condition of study, shared by the patient investigator, might be willing to accept (from a five to a 40-minute PRO screening) supported a confirmation of retaining the initial length of the screening. This was augmented by the patient investigator stating that “those with a hard-to-treat disease would be more than willing to spend the extra time if they knew they were potentially contributing to better cures.”

Usability Testing

Patient advisors co-developed ADVANCE Engagement Level 3 activities. These included a clinical health survey (also assessing research interest) conducted across OCHIN clinics via patient portal; survey data included over 13,280 patient respondents, a response rate of 20% of all active patients (signed in at least once) on the portal. Patient advisors conducted rigorous usability testing, resulting in recommendations to make the survey more visible and its purpose more relevant for patients. Patient advisors have also guided layout and workflows for new patient-reported outcomes (PRO) tools such as SBIRT (Screening, Brief Intervention, Referral to Treatment)26, Patient Health Questionnaire (PHQ)27, and the Patient Reported Outcomes Measurement Information System (PROMIS)28. These advisors are contributing to the development of best practices to increase patient portal adoption and other patient engagement communication strategies (e.g. text messaging, social media). For example, in order to capture valid patient responses, a patient investigator advised on ADVANCE and the SBIRT tool within the patient portal. This specific guidance resulted in the removal of the tool’s scoring layout from the patient portal view and created a new best practice.

PHASE II ENGAGEMENT

Based on the feedback from ADVANCE’s community-academic partnerships and a proposal developed and approved by PEP members, the AAC-approved engagement priorities are as follows: training, ambassador program, dissemination, and evaluation. These priorities are shaping ADVANCE engagement activities throughout Phase II.

Training

This priority includes training resources for all stakeholders. In response, ADVANCE has purchased the CITI Program Training for Responsible Conduct of Research29 for utilization by all stakeholder groups. The CROP-D is currently working with the PEP to facilitate their completion of the CITI training module and certification. Through an environmental scan of the available research trainings recommended from PCORI, our stakeholders and research staff, we assessed several options and determined that CITI Training most closely aligned with our gaps in understanding, as well as our requirements for IRB and human subject’s protection processes. PEP patient leaders are also co-developing training resources) in partnership with OCHIN’s training subject matter experts and partner PCORnet CDRNs to be provided via web (websites, videos) and in person.

Ambassador Program

Our PEP and PBRN engagement structures are expanding their outreach through utilization of an ambassador program. Working with the Oregon Clinical and Translational Research Institute (OCTRI) and its framework for advocacy within community advisory boards, we have begun to recruit more PEP/PBRN members and co-investigators and broadened our outreach to patients in the larger communities in which OCHIN member sites serve. We recently used this peer-to-peer program to engage a non-ADVANCE affiliated patient with a specific opioid-treatment background into an ADVANCE research proposal as a new patient co-investigator. In this case, an established ADVANCE patient investigator co-led the successful recruitment effort for this role in collaboration with the network’s Engagement Coordinator.

This veteran patient investigator provided initial outreach for an introductory phone call between the prospective patient investigator and the research proposal’s principal investigator and team and continued support to build rapport and trust. Additionally, through the veteran patient investigator’s prior experiences around development of proposal materials (letter of support, biosketch), they were able to provide guidance in the new patient investigator’s completion of these items. While the proposal was unfortunately not funded, the inclusion of this new patient investigator, who shared personal experiences living with this condition, provided valuable guidance on the proposal’s methodology and recruitment strategies and powerfully strengthened the significance section of the final submission.

Dissemination

Within the engagement priority of enhanced dissemination, the ADVANCE team focused on efficient and effective ways to widely disseminate PBRN research findings. This is seen by our network stakeholders as a critical component for accelerating the translation of research into practice. Social media strategies, including podcasts and blogging, have the potential to augment the reach of research beyond traditional publication venues.30 The founding patient member of the PEP was a featured patient ambassador author on the PCORI national blog, where she described the influence that PCORI has had on the work of ADVANCE and her perspectives on clinical research. In the post, the patient member and caregiver talks about how she got involved in her local health system, then state policy, and most recently ADVANCE as a part of PCORnet.31 While development continues in these expanded dissemination efforts, we learned that they require thoughtful efforts - finding stories for blogs and podcasts that complement scholarly publications and using these social media posts to communicate findings about recently published papers to broad community and patient audiences.

Patient co-authors have advised, guided and been published on accepted manuscripts, including key reviews and contributions from PEP member co-authors with lived experience of the condition of study.32 Patient investigators have presented at national and international conferences and co-authored research proposals and manuscripts.33–34 ADVANCE patient and clinician leaders recently acted as co-presenters in the 2016 OCHIN Learning Forum (a national conference), with five PEP members and two PBRN steering committee members supporting five panels covering the impacts of the Affordable Care Act, stakeholder engagement, patient portal adoption and enhancement and the OpenNotes movement, which provides patient access to visit notes through their patient portal.35

Evaluation

The ADVANCE three-level engagement framework, developed with key input from many different stakeholders, serves as a guide for evaluating our patient engagement approaches. The network has been developing formal measures of engagement to track our activities and their impacts. Through an iterative process with the CROP-D and OCHIN research leadership, a number of process and Board-reported measures were determined to evaluate our progress with engagement efforts. These measures include: the number of OCHIN member organizations actively involved in research projects, tracking the involvement of stakeholder workgroups in research activities (e.g., the number of proposals and projects reviewed in partnership with stakeholder groups, the number of clinician and patient investigators named on research proposals and projects), and the centralization of tools and protocols for clinic and patient recruitment. Additionally, the CROP-D is responsible for reaching the engagement-related funder milestones for the ADVANCE network.

Our ADVANCE team and stakeholders continually look to refine our metrics on what constitutes "successful engagement” and how we approach this process differently based on the audience. Our stakeholder (e.g. patient, clinician, health system leader) engagement is evaluated within our ADVANCE team through both process measures (e.g. attendance; number of co-investigators) as well as outcome measures, such as documenting patient testimonials on the value of research engagement (Table 1), tracking patient guidance provided to aspects of network activities, and in what capacity feedback has impacted projects throughout the research lifecycle.

Table 1.
ADVANCE Patient Investigator Testimonials
Kay Dickerson "My focus is doing all I can to bring affordable healthcare to all poor and underserved people. As a Hurricane Katrina survivor, I have personally known too many people pass away due to inadequate healthcare. More people are now eligible for healthcare, but waiting for an open appointment still takes too long. I've heard many people on the bus and other public areas talk about lack of affordability and don't understand words and/or procedures and have many, many frustrations. Culturally appropriate healthcare would be a good place to start. Traditional healthcare workers need more recognition.”
PEP Founding Member, PCORI Ambassador, ADVANCE Patient Investigator and AAC Patient Representative
John Lind “I have been interested in understanding more about healthcare systems ever since I was wounded by shrapnel in Vietnam. I spent six months recovering from my injuries—half of the time in Vietnam and half of the time in Okinawa. During that time, I learned how hospital systems worked and how to interact with doctors. Since then, I cared for my aging parents, which led to a deeper interest in the medical field. I really wanted to understand how I could do more for them and help them. Most recently, I lost a girlfriend who passed away from a pulmonary embolism. Her loss stirred me to be more passionate about patient advocacy, and I began to explore patient advocate groups. In 2014, I joined OCHIN’s Patient Engagement Panel and the VA’s Veteran’s Engagement Group.”
ADVANCE Patient Investigator and AAC Patient Representative
Sele D’Amato “During my schooling and work history as a CNA, scrub technician, and patient transporter, I’ve realized that our system does not truly hear what patients are trying to say. Doctors and other health workers are not able to get to know the people they treat. In 2009, I began working with CareOregon as a member of the Community Health Engagement Advocates, which led to the OCHIN Patient Engagement Panel in 2014. I live with Type 2 Diabetes and my sister was recently diagnosed with Type 1 Diabetes. Each day, my sister and I tackle the challenges of day-to-day life that come with our diagnoses. We know all-too-well the risks and dangers of health problems getting out-of-control. My sister and I often say that “we want to manage our Diabetes instead of letting our Diabetes manage us.” This begins by helping clinic staff and doctors so they can help their patients. As an African-American woman representing patients whose voice has not been heard often enough, I’m also very interested in advocating for more effective care that can reach patients who represent diversity of race, ethnicity, socioeconomic status, language, geography, gender, and education.”
ADVANCE Patient Investigator
Kathy Norman “I believe that doctors are only able to do their jobs to their best when patients feel free and supported to have frank discussions. And, I believe this will require a lot of patient education especially in our older communities. An active patient voice just makes sense as a way to improve patient care.”
ADVANCE Patient Investigator
Open in a new tab

For the development of outcome measures, patient advisors are asked to complete quarterly surveys soliciting the value of research engagement, identify any barriers and make recommendations to enhance their roles within the ADVANCE network. The development of these metrics are influenced through the PCORI Engagement Rubric, which provides key examples of “successful engagement” activities through each step of the research lifecycle.36

DISCUSSION

ADVANCE has successfully involved a diverse group of patients in the design, implementation, and interpretation of our research, making progress in three levels of engagement. As noted by our patient investigators, we have seen the importance of continuing to expand and improve on this work. Our PEP founding member and co-author Kay Dickerson states, “My focus is doing all I can to bring affordable healthcare to all poor and underserved people. As a Hurricane Katrina survivor, I have personally known too many people pass away due to inadequate healthcare.” ADVANCE patient investigator and co-author Sele D’Amato further illustrates the need for this work, stating, “As an African-American woman representing patients whose voice has not been heard often enough, I’m also very interested in advocating for more effective care that can reach patients who represent diversity of race, ethnicity, socioeconomic status, language, geography, gender, and education.”

Through our Phase II priorities, we further expanded our engagement of stakeholders through our ambassador model, building on the diversity of the network’s patient advisor base. With enhanced training resources and research partner mentorship, we have able to build the skills, experience and expertise of our patient investigators, advisors and patient members of our PEP, CAPE and other engagement groups. Our co-developed engagement polices around compensation, patient investigator roles and PEP member responsibilities bring an enhanced level of respect to these positions and also streamline our network’s ability to extend these roles to project-specific engagement opportunities.

Future Work

Using our three engagement priorities of training, the ambassador program and dissemination, future directions in meaningful patient engagement for our network will include the development of mechanisms to engage more patients with minimal burden (e.g. brief online surveys), shared strategies to enhance working across a national collaborative, more consistent, relevant and ongoing training for patient co-investigators, and expanding towards more comprehensive and active collection of patient-reported outcomes via patient portals and other bi-directional communication methods to provide “real time” support for clinical care. These additional efforts may further increase patient engagement in research that impacts health care delivery and health outcomes.

Supplementary Material

Supplemental Data File _.doc_ .tif_ pdf_ etc._

Acknowledgments

Sources of Support:

This work was financially supported by the National Institutes of Health National Library of Medicine (grant no. RC4 1001482); the Patient-Centered Outcomes Research Institute (PCORI), PFA Cycle I Contract (2012), Health Systems; the Health Resources and Services Administration (grant no. UB2HA20235); and the Oregon Health & Science University Department of Family Medicine. This work was also supported through a PCORI Award (CDRN-1306–04716) for development of the National Patient-Centered Clinical Research Network, known as PCORnet. All phases of this study were also supported by PCORI Award (308). All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee or other participants in PCORnet.

Footnotes

Conflict of Interest: None declared.

References

  • 1.Westfall JM, VanVorst RF, Main DS, et al. Community-based participatory research in practice-based research networks. Annuals of Family Medicine. 2006;4(1):8–14. doi: 10.1370/afm.511. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Boote JD, Twiddy M, Baird W, et al. Supporting public involvement in research design and grant development: a case study of a public involvement award scheme managed by a National Institute for Health Research (NIHR) Research Design Service (RDS) Health Expectations. 2015;18(5):1481–93. doi: 10.1111/hex.12130. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Williams R, Shelley B, Sussman A. The marriage of community-based participatory research and practice-based research networks: Can it work? - A Research Involving Outpatient Settings Network (RIOS Net) study. Journal of the American Board of Family Medicine. 2009;22:428–35. doi: 10.3122/jabfm.2009.04.090060. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Domecq JP, Prutsky G, Elraiyah T, et al. Patient engagement in research: a systematic review. BMC Health Services Research. 2014;(1):1. doi: 10.1186/1472-6963-14-89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Mullins CD, Abdulhalim AM, Lavallee DC. Continuous patient engagement in comparative effectiveness research. JAMA. 2012;307(15):1587–1588. doi: 10.1001/jama.2012.442. [DOI] [PubMed] [Google Scholar]
  • 6.Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expectations. 2015;18(5):1151–1166. doi: 10.1111/hex.12090. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Frank L, Forsythe L, Ellis L, et al. Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Quality of Life Research. 2015;24(5):1033–1041. doi: 10.1007/s11136-014-0893-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583–1584. doi: 10.1001/jama.2012.500. [DOI] [PubMed] [Google Scholar]
  • 9.Marlett N, Shklarov S, Marshall D, et al. Building new roles and relationships in research: a model of patient engagement research. Quality of Life Research. 2015;24(5):1057–1067. doi: 10.1007/s11136-014-0845-y. [DOI] [PubMed] [Google Scholar]
  • 10.Duncan J. Building infrastructure to engage patients, families and members in research within a learning health organization. Journal of Patient Centered Research. 2015;2(2):133–134. [Google Scholar]
  • 11.Devoe JE, Sears A. The OCHIN community information network: bringing together community health centers, information technology, and data to support a patient-centered medical village. Journal of the American Board of Family Medicine. 2013;26:271–8. doi: 10.3122/jabfm.2013.03.120234. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.DeVoe JE, Gold R, Spofford M, et al. Developing a network of community health centers with a common electronic health record: description of the Safety Net West Practice-based Research Network (SNW-PBRN) Journal of the American Board of Family Medicine. 2011;24:597–604. doi: 10.3122/jabfm.2011.05.110052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.DeVoe JE, Likumahuwa S, Eiff MP, et al. Lessons learned and challenges ahead: report from the OCHIN Safety Net West practice-based research network (PBRN) Journal of the American Board of Family Medicine. 2012;25:560–4. doi: 10.3122/jabfm.2012.05.120141. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Krist AH, Green LA, Phillips RL, et al. Health information technology needs help from primary care researchers. Journal of the American Board of Family Medicine. 2015;28(3):306–310. doi: 10.3122/jabfm.2015.03.140246. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.DeVoe JE, Gold R, Cottrell E, et al. The ADVANCE network: accelerating data value across a national community health center network. JAMIA. 2014;21:591–595. doi: 10.1136/amiajnl-2014-002744. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Wright B, Martin GP. Mission, margin, and the role of consumer governance in decision-making at community health centers. Journal of Health Care for the Poor and Underserved. 2014;25:930–47. doi: 10.1353/hpu.2014.0107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Angier H, Wiggins N, Gregg J, et al. Increasing the relevance of research to underserved communities: lessons learned from a retreat to engage community health workers with researchers. Journal of Health Care for the Poor and Underserved. 2013;24:840–9. doi: 10.1353/hpu.2013.0086. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Selby J, Beal A, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. The Journal of the American Medical Association. 2012;307(15):1583–1584. doi: 10.1001/jama.2012.500. [DOI] [PubMed] [Google Scholar]
  • 19.Fleurence R, Curtis L, Platt R, et al. Launching PCORnet, a national patient-centered clinical research network. Journal of the American Medical Informatics Association. 2014;21(4):578–582. doi: 10.1136/amiajnl-2014-002747. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Fleurence R, Beal A, Sheridan S, et al. Patient-Powered Research Networks aim to improve patient care and health research. Health Affairs. 2014;33(7):1212–1219. doi: 10.1377/hlthaff.2014.0113. [DOI] [PubMed] [Google Scholar]
  • 21. [Accessed June 5, 2017];PCORnet [www.pcornet.org] 2017 Available at www.pcornet.org.
  • 22. [Accessed June 6, 2017];Accelerating Data Value Across a National Community Health Center Network (ADVANCE) [http://www.pcori.org/research-results/2015/accelerating-data-value-across-national-community-health-center-network] 2017 doi: 10.1136/amiajnl-2014-002744. Available at http://www.pcori.org/research-results/2015/accelerating-data-value-across-national-community-health-center-network. [DOI] [PMC free article] [PubMed]
  • 23.Arkind J, Likumahuwa-Ackman S, Warren N, et al. Lessons learned from developing a patient engagement panel: an OCHIN report. Journal of the American Board of Family Medicine. 2015;28(5):632–638. doi: 10.3122/jabfm.2015.05.150009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Henson-Apollonio V. [Accessed May 17, 2017];Setting the stage for patient involvement: connections patients with periodontal disease – tier II. 2015 Available at http://www.pcori.org/research-results/2016/setting-stage-patient-involvement-connecting-patients-periodontal-disease-tier.
  • 25.Yamauchi M, Carlson MJ, Wright BJ, et al. Does health insurance continuity among low-income adults impact their children’s insurance coverage? Maternal and Child Health Journal. 2013;17(2):248–255. doi: 10.1007/s10995-012-0968-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26. [Accessed June 29, 2017];SBIRT [http://www.sbirtoregon.org] 2017 Available at http://www.sbirtoregon.org.
  • 27. [Accessed June 29, 2017];Welcome to the Patient Health Questionnaire (PHQ) Screeners [http://www.phqscreeners.com] 2017 Available at http://www.phqscreeners.com.
  • 28. [Accessed June 29, 2017];PROMIS [http://www.phqscreeners.com] 2017 Available at http://www.phqscreeners.com.
  • 29. [Accessed May 10, 2016];CITI Program [www.citiprogram.org] 2016 Available at www.citiprogram.org.
  • 30.Hoang JK, McCall J, Dixon AF, et al. Using social media to share your radiology research: how effective is a blog post? Journal of the American College of Radiology. 2015;12(7):760–765. doi: 10.1016/j.jacr.2015.03.048. [DOI] [PubMed] [Google Scholar]
  • 31.Dickerson K. [Accessed May 9, 2016];Guiding research and policy to improve healthcare for everyone. 2014 Available at http://www.pcori.org/blog/guiding-research-and-policy-improve-healthcare-everyone.
  • 32.Nichols GA, McBurnie M, Paul L, et al. The high prevalence of diabetes in a large cohort of patients drawn from safety net clinics. Preventing Chronic Disease. 2016;13:160056. doi: 10.5888/pcd13.160056. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.DeVoe J, Angier H, Likumahuwa S, et al. Use of qualitative methods and user-centered design to develop customized health information technology tools within federally qualified health centers to keep children insured. The Journal of Ambulatory Care Management. 2014;37(2):148–154. doi: 10.1097/JAC.0000000000000016. [DOI] [PubMed] [Google Scholar]
  • 34.Angier H, Marino M, Sumic A, et al. Innovative methods for parents and clinics to create tools for kids' care (IMPACCT Kids' Care) study protocol. Contemporary Clinical Trials. 2015;44:159–163. doi: 10.1016/j.cct.2015.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.Leveille SG, Walker J, Ralston JD, et al. Evaluating the impact of patients' online access to doctors' visit notes: designing and executing the OpenNotes project. BMC Medical Informatics and Decision Making. 2012;12(1):1. doi: 10.1186/1472-6947-12-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. [Accessed October 25, 2016];PCORI Engagement Rubric [www.pcori.org/sites/default/files/Engagement-Rubric.pdf] 2016 Available at www.pcori.org/sites/default/files/Engagement-Rubric.pdf.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplemental Data File _.doc_ .tif_ pdf_ etc._
NIHMS893404-supplement-Supplemental_Data_File___doc___tif__pdf__etc__.docx (297.3KB, docx)

RESOURCES