IMPACCT Kids’ Care: a real-world example of stakeholder involvement in comparative effectiveness research

Abstract

The Patient-Centered Outcomes Research Institute has accelerated national conversations about the importance of actively engaging stakeholders in all aspects of comparative effectiveness research (CER). Other scientific disciplines have a history of stakeholder engagement yet few empirical examples exist of how these stakeholders can inform and enrich CER. Here we present a case study which includes the methods used to engage stakeholders, what we learned from them, and how we incorporated their ideas. We selected stakeholders from key groups, built relationships with them and collected their feedback through interviews, observation, and ongoing meetings during the 4 research process phases: proposal development, adapting study methods, understanding the context, and IT tool design and refinement.

BACKGROUND

Researchers and funding agencies [e.g., Patient-Centered Outcomes Research Institute (PCORI)] prioritize stakeholder engagement in all aspects of comparative effectiveness research (CER). Deverka and colleagues define stakeholders as “individuals, organizations or communities that have a direct interest in the process and outcomes of a project, research or policy endeavor” [1,2]. Community-based participatory research [3–5], action research [6–8], practice-based research [9–11], and user-centered design [12] have rich histories of incorporating a variety of engagement methods and techniques. Building on experiences from these disciplines, CER experts are developing and testing methods for engaging stakeholders to enrich the process, improve the relevance of research projects, and accelerate the implementation of findings [13–17].

In recent years, shared terminology and common definitions have been developed to identify categories of stakeholders and delineate roles [2,18,19]. For example, the “7Ps” framework was developed to identify key groups of people who might inform research: patients and the public, providers, purchasers, payers, policymakers, product makers, and principal investigators. Other researchers have developed similar stakeholder categories [1,20]. Different levels of stakeholder engagement have been defined, ranging from one-time feedback to long-term engagement as study co-investigators, [17] and a variety of methods have been used to elicit different perspectives [21]. Empirical examples from CER studies intentional about stakeholder engagement are needed to complement philosophical concepts and models.

Here, we describe stakeholder engagement in the PCORI-funded Innovative Methods for Parents And Clinics to Create Tools for Kids’ Care (IMPACCT Kids’ Care) project, which uses stakeholder-centered design processes to build, implement, and test the comparative effectiveness of health information technology (IT) tools to track and document patients’ insurance status in community health center (CHC) clinics. The study, itself, is described in detail elsewhere [22], but this case study provides a ‘real-world’ example of the methods used to identify, engage, and utilize feedback from diverse stakeholders for this CER project.

Study Setting

OCHIN was founded in 2001 by Oregon Community Health Centers (CHCs), originally as the Oregon Community Health Information Network but renamed when other states joined. OCHIN is a non-profit 501(c)(3) collaborative now serving CHCs in 20 states to develop and implement health IT. It also supports a practice-based research network (PBRN) and is an emerging community laboratory for CER [23–26]. OCHIN hosts a linked instance of the Epic© electronic health record (EHR) providing each CHC patient in the network with a single medical record number. The IMPACCT Kids’ Care project includes 8 OCHIN member CHCs from 3 clinic systems.

IDENTIFYING STAKEHOLDERS

We identified a diverse group of relevant stakeholders based on the 7Ps model described above: patients and the public, providers, payers, policymakers, and principal investigators. Patients and the public included patient partners, families, caregivers and community organizations. Providers included CHC leaders, managers and clinic staff (i.e., ‘end-users’ of the IT tools). Oregon Medicaid payers and policymakers were also key stakeholders for this study. Since we conducted the CER study, our team of researchers served as the principal investigators.

Existing Stakeholder Groups

Two stakeholder engagement groups for research exist at OCHIN, the PBRN and the Patient Engagement Panel (PEP). The PBRN, comprised of CHC clinicians and primary care researchers, met regularly to inform research proposals, develop new practice-based research methods, and help interpret results [25,26]. The PEP is comprised of patients who inform all OCHIN research projects and advise on organizational priorities [27]. We leveraged these groups, leaders of community organizations and state policymakers who have contributed to previous OCHIN studies or served on the PBRN [14,28–30] to provide feedback on the research phases listed below.

New Stakeholder Groups

In addition to existing relationships, we identified several new groups needing representation in this project: CHC staff, patients’ family members and caregivers. Working with managers at the CHCs we identified appropriate representatives from key staff involved in clinic health insurance processes. These ‘end-users’ were recruited to participate in user-centered design sessions and to serve on the project’s Clinic Advisory Group to advise the project team on all aspects of study methods and provide interpretation and dissemination of research results. During study interviews with patients’ family members and caregivers, we invited interviewees to participate in the project’s Family Advisory Panel; 3 (of 20) interviewees agreed to join. The Family Advisory Panel joined with the PEP to provide ongoing advice and guidance including monitoring the study’s progress and disseminating results. The challenges and lessons learned in developing the PEP (recruiting and retaining participants from underserved populations) are described elsewhere [27]. Figure 1 shows the existing and new stakeholder groups for this project.

Figure 1.

IMPACCT Kids’ Care project: existing and new stakeholder groups

COLLECTING AND USING STAKEHOLDER FEEDBACK

We engaged stakeholders during four early phases of the research process: 1) proposal development, 2) adapting study methods, 3) understanding the context, and 4) IT tool design and refinement. We collected feedback from stakeholders using a variety of qualitative methods including semi-structured interviews, advisory group meetings, observation, ‘Think Aloud’ usability testing, and group meetings. We used a qualitative data analysis strategy to analyze feedback received from stakeholders so we could incorporate it into our research project [31]. Stakeholders were modestly compensated for initial interviews and for their roles on advisory groups. However, they were not compensated for participation in IT tool testing activities. Our Institutional Review Board reviewed and approved this study. The methods we used to engage stakeholders, what we learned, and how we incorporated their ideas into each of the research process phases is described in detail below.

Proposal Development

The idea for developing IT tools within the EHR, to help clinics identify patients whose insurance was expired or nearing expiration, originated with a CHC staff member during an OCHIN PBRN meeting. This important idea was turned into a CER study and grant funding was sought. As the research team prepared the grant application, we consulted with the PEP, the PBRN, policymakers and leaders of community organizations who are collaborators on other OCHIN initiatives. Consultation occurred through committee meetings, one-on-one meetings with individual CHC patients and PBRN members, and by circulating key pieces of the proposal to these stakeholders for comment. We gained valuable insights from a diverse group which allowed us to shape the scope of the proposal. This process led to the in-depth involvement of 2 co-investigators: a patient (KD) and a policymaker (CG).

In addition to contributing ideas, helping with the proposal writing process, and serving as co-investigators, study advisors, and consultants, stakeholders guided development of compensation policies for patients. Although we assumed patients would like cash or gift cards as compensation, we learned that some CHC patients preferred receiving travel support (to attend conferences) or computers (to aid participation in teleconferences) that would help them become more actively engaged with the research team. We also learned that even modest financial payment might adversely impact annual tax returns and/or jeopardize eligibility for public programs (e.g., Medicaid, Social Security) for some patients.

Adapting Study Methods

After the IMPACCT Kids’ Care project was funded, we engaged CHC patients, clinicians, and staff in preliminary meetings to launch the study, review the timeline, and finalize study methods. Stakeholders provided invaluable feedback on the study methods. For example, when reviewing qualitative data collection methods, stakeholders advised our team to conduct interviews rather than focus groups with families, since they thought 1-on-1 interviews would yield more information and be less intimidating for participants. CHC leaders were also concerned about the staff burden of coordinating and scheduling focus groups and about reports of past experiences with low turn-out for these sessions. After consultation with qualitative research co-investigators, in lieu of the originally planned focus group, we modified our data collection design and conducted individual, semi-structured interviews to make the study more ‘stakeholder-friendly’ and relevant. This stakeholder-informed modification enabled us to collect rich data using rigorous methods while remaining flexible to meet the needs of research participants.

The original proposal called for comparing IMPACCT Kids’ Care IT tools in 2 intervention and 2 control clinics. However, we learned from discussions with CHC leaders that all clinics must utilize standardized, consistent workflows throughout the entire clinic system to manage insurance enrollment. Thus, instead of 2 clinics, we modified our implementation plan to accommodate all 4 clinics in the intervention clinic system. This change also strengthened our study by preventing contamination bias [32].

Understanding the Context

To understand the context of obtaining and maintaining public health insurance for children, we interviewed 20 family members of pediatric patients (11 interviews were conducted in Spanish; 9 interviews were conducted in English) and 31 clinic staff members (management, front desk and scheduling, insurance specialists, medical assistants, social workers, community health workers, and providers). We also held a focused advisory group meeting with policymakers from the State of Oregon, patient representatives, community leaders, and members of the OCHIN research team to exchange ideas and information about the project, state health insurance exchange, and the Affordable Care Act [33]. Based on what we learned from these interviews and meetings, we developed the IT tools and accompanying guides with a focus on the end-users (i.e., clinic staff).

IT Tool Design

Informed and guided by various stakeholder groups, OCHIN developed a set of IT tool prototypes. These prototypes were then shared with key stakeholders from the 3 participating clinic systems at a half-day retreat. During this retreat, CHC staff and families participated alongside members from OCHIN’s research team, including 4 community-based co-investigators, one of whom (TB) is OCHIN’s Chief Medical Informatics Officer and is directing the development of the IT tools. The goal of the retreat was for the research team to summarize findings from family and clinic staff interviews and initial stakeholder engagement activities, present possible tool options, and obtain critical feedback about how to improve the prototypes. We held dedicated IT tool review meetings with the Family Advisory Panel and Clinic Advisory Group to get their feedback on the tool prototypes. The research team also engaged stakeholders in Think Aloud exercises to further refine the tools. This user-centered design process involved key users ‘thinking aloud’ as they performed a set of specified tasks or reviewed a tool prototype [36]. Research team members visited 2 clinic sites and employed the Think Aloud process during interactions with 6 end-users of the IT tools [22].

Once prototypes were finalized, IT tools were built in the OCHIN Epic© system and activated for additional testing. In April 2014, to elicit feedback from end-users to fine-tune IT tools prior to full implementation, beta-testing was conducted across 4 sites with 16 stakeholders who represented a variety of roles in the clinic: insurance eligibility specialists, schedulers, and front desk staff. After final modifications, the tools were implemented and available for all staff to use at the 4 intervention sites in early June 2014. In mid-July 2014, about 6 weeks after the tools were activated and guided by the Morville Framework [37], we conducted usability trials with 24 end-users (clinic staff) at the 4 implementation sites. Based on these trials, we redesigned some of the IT tools for ease of use and modified language on several functions to enhance clarity (e.g., we expanded drop-down list options and added data fields).

We discovered during iterative testing cycles that the accuracy of some of the insurance coverage data provided by the State of Oregon was unclear. We met with our policymaker co-investigator and others from the State Medicaid office to resolve this and, as a result, the quality of the data has been improved and the IT tools were updated to be more reliable.

Key engagement methods, themes arising from our engagement activities and descriptions of stakeholder contributions are summarized in Table 1.

Table 1.

Descriptions of stakeholders, stakeholder engagement methods used, what we learned, and how we used stakeholder ideas throughout the project

Purpose	Stakeholders	Methods Used	What we learned and how we used stakeholder ideas
Proposal Development	13 stakeholders: clinic leaders, policymakers, patient representatives, community organization leaders	Advisory group meetings	CHC staff originated study idea Stronger proposal due to stakeholder comments Changed compensation for patient involvement
Adapting Study Methods	11 clinic leaders: clinicians, executives, managers	Advisory group meetings	Set timeline for study Modified data collection design from focus groups to 1-on-1 interviews Increased involvement from 2 to 4 clinics
Understanding the Context	58 stakeholders: families, clinic leaders, policy makers, patient representatives	Semi-structured interviews, advisory group meetings, observation	Confirmed clinic workflows from family and clinic perspectives Tools and tool implementation guides designed to focus on end-users (clinic staff)
IT Tool Design	39 stakeholders: clinic staff, clinic leaders, families, policy makers, patient representatives	Semi-structured interviews, advisory group meetings, observation, ‘Think Aloud’ usability testing, group meetings	Fine-tuned tool prototypes Post-implementation tool refinement for language and usability

DISCUSSION

Stakeholders bring different experiences, interests, and expertise to research studies which shape both the roles they play and the contributions they make to the process [1]. Engaging stakeholders in CER requires skills in aligning these interests and areas of expertise to the research project. Our study demonstrates effective engagement of different stakeholder groups ensuring that study design and interventions were truly clinic-, patient-, and family-centered. Beyond the clinical team, this case study also highlights the importance of engaging healthcare staff.

The method of inviting interviewees to participate in our Family Advisory Panel was particularly effective in engaging stakeholders from ‘hard-to-reach’ groups. Hard-to-reach, as defined by PCORI, is used as a general term for individuals or communities who are historically underrepresented in health care research and/or less likely to be involved in research because of differences or barriers that impede communication or collaboration with researchers [38]. These barriers include language, education, social class, ethnicity, race, culture, geography, physical or cognitive impairments, and other differences.

We leveraged previously cultivated relationships (e.g., PBRN clinicians, patient co-investigators, and policy-makers) and forged new relationships (e.g., clinic end-users) in the IMPACCT Kids’ Care project. Therefore, the IT tools we developed are highly relevant to the stakeholder groups involved but they may need to be modified for other clinics. Future efforts will use dissemination and implementation science methodology to study the best ways to utilize these tools in additional OCHIN sites and, through additional collaborations, possibly beyond OCHIN. We will also continue to involve stakeholders in the analysis, interpretation, and dissemination of project results.

The IMPACCT Kids’ Care project provides an example of how we involved stakeholders in CER proposal development, study method adaptation, context understanding, and IT tool design and refinement. Many of the relationships developed in this study are sustainable and have already influenced future studies.

Executive Summary.

Background

• Previous work by community-based participatory research, action research, practice-based research, and user-centered design has shown that stakeholder involvement can significantly shape the research process.

• The PCORI-funded IMPACCT Kids’ Care project uses stakeholder-centered design processes to build, implement, and test the comparative effectiveness of health information technology (IT) tools to track and document patients’ insurance status in community health center (CHC) clinics.

Discussion

• Stakeholders were selected from key groups (community clinics, patients’ families, and policymakers). We leveraged previous relationships and built new relationships to collect ongoing feedback through interviews, observation, and meetings during 4 phases of the research process: proposal development, study design, understanding the context, and tool design and refinement.

• Feedback and insights from stakeholders have bridged the academic and clinical worlds of our research by helping us understand clinic, family, and state policy workflows around insurance. Stakeholders have contributed to our knowledge about how IT tools can be used by clinics and patients’ families.

• The stakeholders for IMPACCT Kids Care continue to be engaged in the analysis, interpretation, and dissemination of the results.

Conclusion

• The IMPACCT Kids’ Care project provides an example of how we involved stakeholders in CER proposal development, study method adaptation, context understanding, and IT tool design and refinement.

Future Perspective.

As the field of stakeholder-engaged CER continues to mature, more real-world examples are needed to discuss methods, challenges and opportunities for meaningfully involving stakeholders in all aspects of research. CER will also continue to learn from other disciplines to sustainably and meaningfully engage stakeholders.