Qualitative Exploration of Engaging Patients as Advisors in a Program of Evidence Synthesis

Background:

There is an increasing expectation for research to involve patient stakeholders. Yet little guidance exists regarding patient-engaged research in evidence synthesis. Embedded in a learning health care system, the Veteran Affairs Evidence Synthesis Program (ESP) provides an ideal environment for exploring patient-engaged research in a program of evidence synthesis.

Objective:

The objective of this study was to explore views, barriers, resources, and perceived values of engaging patient advisors in a national program of evidence synthesis research.

Methods:

We conducted 10 qualitative interviews with ESP researchers and 2 focus groups with patient stakeholder informants. We queried for challenges to patient involvement, resources needed to overcome barriers, and perceived values of patient engagement. We analyzed qualitative data using applied thematic and matrix techniques.

Results:

Patient stakeholders and researchers expressed positive views on the potential role for patient engagement in the Veteran Affairs ESP. Possible contributions included topic prioritization, translating findings for lay audiences, and identifying clinically important outcomes relevant to patients. There were numerous barriers to patient involvement, which were more commonly noted by ESP researchers than by patient stakeholders. Although informants were able to articulate multiple values, we found a lack of clarity around measurable outcomes of patient involvement in systematic reviews.

Conclusions:

The research community increasingly seeks patient input. There are many perceived and actual barriers to seeking robust patient engagement in systematic reviews. This study outlines emerging practices that other evidence synthesis programs should consider, such as the careful selection of stakeholders; codeveloped expectations and goals; and adequate training and appropriate resources to ensure meaningful engagement.

Patients are increasingly valued for their contributions to research as scientific collaborators. Patients’ lived experiences of health conditions can improve research quality and relevance, and accelerate the adoption of findings into practice.1–4 Patient input into research design and dissemination has been explored most commonly in health services research and clinical trials.5,6 Notably, the Patient-Centered Outcomes Research Institute (PCORI) advocates engaging patients as stakeholders at multiple levels of trial development and conduct.7 However, there has been less focus on how best to engage patients in evidence synthesis.

Patient perspectives can optimize relevance, accessibility, and impact of systematic reviews.8,9 However, evidence synthesis presents slightly different opportunities and challenges to involving patient stakeholders. First, stakeholder involvement in the form of technical expertise is expected in evidence synthesis; yet, compared with other types of stakeholders, patients may offer different value at different stages of research.10 Second, evidence synthesis involves the collection of data across multiple studies examining the same question, which requires consideration of unfamiliar and highly technical concepts. Although methodologically important, this aspect of systematic reviews removes patient stakeholders from clinical scenarios directly relevant to their lived experience. Third, systematic reviews are often conducted by evidence synthesis scientists who may lack direct or ongoing connection to patients possessing the requisite expertise. This lack of connection to relevant patient populations may pose a barrier to timely identification of patient stakeholders. Although patients have contributed to systematic reviews for 2 decades, little practical guidance about how best to involve patient perspectives in systematic reviews exists, and there is little work on patient engagement in organizations that conduct systematic reviews.9

The Veteran Affairs (VA) national Evidence Synthesis Program (ESP) consists of a coordinating center and 4 geographically dispersed hubs. Embedded within a learning health care system, each hub generates at least 3 evidence syntheses annually in response to clinical and policy questions nominated by national VA offices. ESP products are routinely used to guide clinical and programmatic decisions on the local and national levels. In addition, each ESP project convenes a technical expert panel comprising both VA and non-VA national topical experts to provide input and feedback about protocol development and interpretation of findings. Thus, ESP clearly mandates stakeholder—although not specifically patient stakeholder—involvement. Further, some ESP hubs are colocated within VA research centers that have convened patient research advisory councils, called Veteran Engagement Groups (VEGs). VEGs include veterans and veteran caregivers who advise investigators on research. This structure presents unique opportunities for investigating perceptions about patients as collaborators in a national program of evidence synthesis research. Thus, we sought to explore views of systematic review researchers, patients, and caregivers on how to integrate patient stakeholders into a national evidence synthesis program.

METHODS

Design

We conducted and analyzed semi-structured qualitative interviews with ESP leaders (ie, directors, associate directors) and programmatic staff, and focus group discussions (FGDs) with established VEGs of patient stakeholders. As the primary goal of our evaluation was to inform ESP quality improvement efforts, this work was designated as a nonresearch operations project.

Participants

We purposively sampled key informants across 4 ESP hubs and the coordinating center. To solicit input from different positions within ESP, we sampled across organizational roles (eg, directors, research assistants). Key informants were recruited via e-mail. Interviews took place in July and August 2018. We conducted FGDs with established groups of veteran patients and caregivers. To identify participants, we contacted the faculty at ESP-affiliated VA research centers and asked if we could conduct FGDs with their veteran groups. The FGDs took place in August 2018.

Approach

To develop interview and FGD guides, we conducted a rapid scan of the peer-reviewed literature in Medline from inception to March 2018. The search yielded 2249 citations of which we retained 8 reports.8–15 A single investigator reviewed citations for studies of any design that addressed patient involvement in systematic review science. We developed narrative summaries of key findings of the identified reports and used these summaries to construct a list of common themes. We used these themes as domains from which to construct discussion guides (Table 1).

TABLE 1.

Focus Group Discussion Guide and Individual Interview Guide

All individual key informant interviews were conducted via telephone by 2 investigators with qualitative expertise (J.M.G., J.M.H.). Because this project was conducted to inform ESP process improvement, we chose an approach that provided high quality and timely information on quality improvement. Thus, rather than record interviews and transcribe them, we used other approaches to optimize rigor.16–18 One investigator took real-time detailed field notes focusing on verbatim statements during interviews and captured respondent input via structured interview summary templates. To ensure accuracy, we used several member-checking techniques19,20 to assess the validity of the data and findings, such as having a subset of key informants review their interview summaries for completeness and presenting key findings to informants for further clarification and validation. We aimed to conduct at least 12 interviews unless we reached thematic saturation before that threshold.21

Before starting FGDs, we conducted a brief overview of ESP and systematic review methods, assuming participants had little prior knowledge of evidence synthesis (Fig. 1). The FGDs were held in person and conducted by trained qualitative researchers (J.M.G., J.M.H.). For similar reasons as noted above, we did not record FGDs. We captured detailed notes in structured templates and circulated them to FGD participants to verify accuracy and completeness. Following prior work on sample size and thematic saturation, we aimed to conduct 3 FGDs but only conducted 2 due to time and logistical constraints.22

FIGURE 1.

Overview of the Veteran Affairs’s Evidence Synthesis Program and systematic review methods. *Cochrane, What are systematic reviews? (www.youtube.com/watch?time_continue=17&v=egJlW4vkb1Y), video, January 27, 2016. ESP indicates Evidence Synthesis Program.

We conducted an applied thematic analysis using both a priori codes structured from interview guides and emergent codes to develop a codebook.23 Two investigators (J.M.G., J.M.H.) applied codes to 2 interviews and 1 FGD summary. Codes were compared, a coding scheme was finalized through discussion, and then the coding scheme was applied to all notes by 1 team member (J.M.G.) using NVivo software (version 12, 2018; QSR International Pty Ltd). For data reduction, we used matrix analysis and categorized themes by key informant type (ie, patient/caregiver, ESP hub or coordinating center leadership, ESP staff) to compare findings.24 Before analysis, we redacted personal identifiers from summaries to protect informants’ identities.

RESULTS

We invited 3 individuals at each of the 4 hub sites plus 3 additional participants from the coordinating center for a total of 15 potential participants. Ten individuals agreed to participate (5 ESP leaders, 5 ESP staff), representing each hub and coordinating center. Experience with ESP ranged from ≤1 to 10 years. We also conducted 2 FGDs with VEGs. Reflecting the overall VA population, most VEG members were male (73%), white (81%), and served during the Vietnam War era (63%).

We organized our findings around interview guide domains: (1) potential roles for patient collaborators in an evidence synthesis program; (2) perceived barriers; (3) resources needed to overcome barriers; and (4) value of involving patient collaborators and how to measure that impact. Under each domain, we focused first on areas of concordance and then explored differences in themes by key informant type (ie, ESP staff, ESP leader, patient/caregivers). When warranted, we also explored how findings differed at the national ESP programmatic level and individual study level.

Patients’ Roles in ESP

Theme Concordance

Table 2 provides a matrix of potential patient roles in systematic reviews by key informant type. All stakeholder groups agreed that prioritizing topics for consideration at the national level was an area ripe for patient contribution. As 1 patient stated, “We are VERY INTERESTED in helping think through what is important to study.” There was high concordance between patients and researchers’ opinions regarding the central role patients could play in helping ESP contextualize, translate, and disseminate findings on individual evidence synthesis studies. Suggested ideas included disseminating findings through patient networks and social media, integrating patient narratives into evidence reports to contextualize findings, or having patients codevelop plain-language briefs. As one patient representative stated, “In plain talk … KISS … ‘keep it simple, stupid’ … we are mediators of simple, plain talk. We could do this for reviews.”

TABLE 2.

Thoughts on Patient Role in the Evidence Synthesis Program by Key Informant Type

Patient Theme Differences

Patients were willing to take on additional responsibilities and eager to be more deeply involved in reviews. Some expressed wanting to serve as coinvestigators, help screen citations, or interpret findings based on their lived experience. Patients recognized the potential limitations of their contributions due to the methodological focus of systematic review science. Moreover, patient and caregiver FGD members expressed a high degree of interest in nominating topics to be considered for ESP reviews. Patients proposed an online portal to garner solicitations from a broad base of patients and caregivers.

Research Staff and Leaders Theme Differences

The ESP research staff and leadership were more reluctant to open the nomination process to patients and others outside VA clinical operations due to 2 main concerns: (1) not having capacity at the national programmatic level to vet and prioritize additional nominations; and (2) needing to continue to demonstrate impact on the VA’s learning health care system. ESP researchers described having a ready consumer of their evidence reports because topics are nominated by operations and clinical stakeholders and represent pressing information needs with direct clinical and policy relevance. Researchers saw the optimal role of patients as strategic advisors, providing feedback on key, isolated aspects of reviews, such as outcome selection, and functioning similarly to technical expert panel members. As 1 ESP researcher stated, “picking outcomes that matter and clinically important effects … Patients could help with that.”

Barriers to Engaging Patients in ESP

Theme Concordance

All informants independently identified barriers related to seeking patient representativeness, researchers’ lack of skills in patient engagement, and scientific complexity of evidence synthesis as roadblocks to patient involvement in ESP. Yet, perceived barriers to engaging patients in ESP differed by key informant type (Table 3).

TABLE 3.

Perceived barriers to Patient Engagement in the Evidence Synthesis Program by Key Informant Type

Patient Theme Differences

Compared with ESP informants, patients identified far fewer barriers to engagement and endorsed time constraints to make meaningful contributions, perceived unwillingness of researchers to collaborate with patients, and patients’ historical mistrust of research as patient-specific barriers to engagement.

Research Staff and Leaders Theme Differences

ESP staff and leaders identified more barriers to patient involvement than did FGD patients and caregivers. When looking across barrier type, ESP staff defined barriers primarily related to timelines and logistics that would influence daily project operations. Research staff identified the inadequate time to engage patients in fast-paced project timelines, insufficient personnel to coordinate patient-engagement efforts, and logistical challenges in identifying patients, and uncertainty about research ethics oversight of patient involvement.

ESP leaders also identified these challenges but contributed additional barriers. They expressed concerns related to managing patient collaborator expectations and high potential costs (eg, staff time, patient compensation) associated with patient involvement. However, ESP leaders most often noted the need to manage potential bias that patients might bring to the process, explaining that it might be difficult for patients to remain objective when speaking from personal experience. As one researcher stated, “They [patients] can have VERY STRONG opinions that can be hard to manage … I have some trepidation because of this.”

Values of Patient Engagement

All informants articulated the perceived value of patient involvement. Values clustered into 3 areas of value to patient, organization, and science (Table 4), but there was low concordance across informant type. Yet, all informant had difficulty articulating key outcomes of patient engagement and ideas on measuring patient-engagement value.

TABLE 4.

Perceived Values of Patient Engagement in the Evidence Synthesis Program by Key Informant Type

Theme Concordance

The greatest areas of concordance and identification of values by informant type were related to the science of evidence synthesis. Patient involvement in systematic reviews was seen to improve scientific quality and relevance and speed dissemination of findings. In addition, all informants identified how patient engagement in ESP promotes the organizational mission of providing high-quality evidence synthesis to inform clinical policy. One ESP researcher stated, “participating in ESP demonstrates there is support to figure out what works best for veterans.” All informants agreed that patient participation could potentially increase ESP visibility. Both ESP leaders and patients and caregivers stated that patients value giving back, especially to those with shared experiences (eg, veterans).

Research Leaders Theme Differences

ESP leaders identified additional values of patient engagement and added that patient involvement promotes credibility and trust in science through enhanced transparency, and, as a byproduct, increases scientific impact. Further, many ESP hubs rely on volunteer trainee investigators to conduct reviews. Thus, ESP leaders endorsed patient involvement in the ESP as a way to foster patient-engaged research methods through early investigator exposure of effective strategies.

Resources Needed to Overcome Barriers to Patient Engagement

Resource needs responses coalesced into 3 categories that were consistent across informant type: (1) commitment to patient engagement as a core value; (2) human capital to fulfill the vision; and (3) processes to facilitate capacity.

Commitment

Commitment at the national programmatic level was expressed as 2 components: a cultural shift in norms towards engaging patients, and resources to execute the vision. Informants reported that organizational commitment would require additional funds and flexibility in project timelines, reporting templates, and metrics to measure patient engagement. At the local-hub level, commitment to patient engagement was operationalized as having a “spirit of innovation and creativity” to established local procedural operations and flexible ways of doing work that make sense for each project. ESP leaders expressed disdain for having an overly prescriptive “perfect protocol” to conduct patient engagement. As 1 ESP informant stated, “Things will not go right 100% of the time … Workshop ideas, try them and then have conversations.”

Human Capital

ESP leaders and staff also expressed the need for human capital resources in conducting meaningful patient involvement. A major personnel resource need was expertise in patient engagement. For many, patient engagement was a new method perceived to stretch systematic reviewers’ skillsets. The other major human capital was a stable group of patient collaborators possessing sufficient time to contribute to projects.

Facilitating Capacity

ESP informants expressed that establishing processes to facilitate the capacity to produce meaningful patient engagement was essential. Clear guidance on research ethics rules was seen as crucial to fostering the capacity to engage patient collaborators. ESP informants expressed needing procedures for identifying, vetting, and training patient collaborators. Ideas for identifying patient collaborators included partnering with veteran service organizations or disease advocacy groups, requesting use of established patient advisory groups, or asking frontline providers to nominate patients. Regardless of how patients were identified, ESP informants expressed a need for patient collaborators who had some understanding of systematic review methodology to foster authentic engagement. One ESP informant noted, “You do not want to make paraprofessionals but you want them to feel comfortable and prepared.” Last, ESP informants wanted guidance on how and when to engage patients in research to foster meaningful engagement. One ESP informant stated, “Do it with real intention … gain buy-in from local centers, develop process.”

CONCLUSIONS

We found that patient stakeholders and researchers alike expressed positive views on patient engagement in VA ESP. Potential contributions included assisting with topic prioritization, translating findings for lay audiences, and identifying clinically important outcomes relevant to patients. Barriers to patient involvement were primarily raised by research key informants rather than patient/caregivers. Evidence synthesis research teams cited concerns about time and financial costs required for meaningful patient involvement on tight project timelines. In addition, while informants were able to articulate multiple values, we found a lack of clarity around measurable outcomes of patient involvement in systematic reviews.

Much of the existing information about patient engagement with systematic review organizations has been generated by Cochrane, who is committed to partnering with patients.9 The 2018 Cochrane Colloquium was a “Patients Included” event organized around “Cochrane for all—better evidence for better health decisions.”25 Through Cochrane, patients have primarily been engaged as referees for specific reviews and during translation of report findings to plain-language summaries.9 Although we found interest in similar types of activities among patients and caregivers, they were also eager to be involved in task-oriented project activities. Our ESP research informants had significant concerns about how to do this in a sustainable and meaningful way. Cochrane has recently expanded opportunities for patients to engage in methods activities of reviews by inviting them to perform key steps such as citation screening.26 Other innovations are likely needed, such as software platforms for topic prioritization, to engage patients in systematic review processes in a meaningful yet cost-effective manner.

A novel finding from our work is the potential role for patients in earlier topic prioritization stages, which has not been commonly explored. Within the VA health care system, veteran input is highly valued, as exemplified by the widespread promotion of patient advisory groups within VA research centers to inform research activities. The veterans in our FGD felt they could offer added value in providing input around topic prioritization, whereas researchers raised concerns about the need for careful planning and weighing of input at this stage to avoid introducing bias. Balancing stakeholder input is not unique to patient stakeholders and has been a recognized challenge for other types of stakeholder involvement.13

Although researchers agree that patient input can aid evidence synthesis, measuring the success of patient involvement in systematic reviews requires clarification. Researcher informants hesitated to involve patients in program activities without clear directives on metrics to evaluate effectiveness, echoing a previously noted9,11 need for a delineated approach to assess impact. Potential domains for such assessment include relevance, accessibility, transparency, and research uptake.10

Researcher informants universally noted concerns over the additional resources entailed to conduct meaningful patient engagement. Specifically, there was a concern that increasing patient involvement in earlier stages of topic nomination and prioritization could raise demands and expectations for more products without the necessary growth in resources. This is a legitimate concern for ESP given fixed budgets and relatively inflexible timelines. The resource burden of patient engagement in evidence synthesis has been noted previously and includes time and financial resources.9,13,27 The importance of maintaining longitudinal relationships with patient stakeholders for optimal partnering is one example of this burden. Moreover, there is a need for clear guidance on optimal ways to engage with patients, including staff training on interactions with patient stakeholders, how to identify appropriate individuals, and how to manage potential perceptions of the impact on scientific integrity.9 PCORI, grappling with similar challenges, has developed training programs for patient peer-reviewers and resources to support investigators in patient engagement. These types of resources may be adapted to facilitate patient engagement in systematic review programs.

The VA ESP constitutes an ideal site for building evidence on patient-engaged research in systematic reviews. The ESP is a national program with 4 geographically dispersed hubs colocated within VA health services research centers, 3 of which have patient advisory groups. This structure offers a unique opportunity to test strategies and construct the evidence base on optimally engaging patients as advisors in a program of systematic review research. Opportunities exist within the ESP to pilot strategies, such as a partnership with existing patient advisory panels to cobuild plans for patient engagement in systematic reviews, which can be tested and refined.

Our study had several strengths, including multiple sources of data from patient FGD and individual interviews with both ESP staff and senior leaders across different sites, interview guides informed by the literature, a team approach to interviews, use of field notes in the form of interview summaries, and rigorous member checking. Despite these strengths, we note some limitations based on our design choices. First, we opted to not record interviews and FGDs. There are many valid reasons why recordings may not be feasible or necessary. Yet, not recording may open studies up to informant social desirability bias and researcher confirmatory bias.16,18,28 We tried to limit these by presenting questions in a neutral format, having someone external to the organization conduct the interviews, and implementing systematic and transparent coding processes facilitated by the state-of-the-science qualitative data analysis software. Next, our sample size was relatively small and not all targeted ESP researchers and patient advisory panels responded to our interview invitation request. We attempted to counteract any resulting potential bias by purposively sampling across the 4 centers. Finally, the results presented here should be interpreted cautiously as they only reflect opinions from one evidence synthesis program.

Early engagement of diverse stakeholders increases the likelihood that end-products will be relevant and useful to decision-makers, provides real-world context for evidence application, and can enhance end-user uptake.2,3,29 There is an increasing expectation for research to involve patient stakeholders. Thus, a shift toward patient engagement in the ESP, and other evidence synthesis programs is likely. There are many perceived and actual barriers to seeking robust patient engagement in systematic reviews. Our study provides information on emerging practices that other evidence synthesis programs need to consider when approaching how to foster patient engagement. Patient and research informants largely agreed that careful selection of stakeholders is critical. All parties should collaboratively outline goals, roles, and expectations. Research staff should provide adequate training and ensure appropriate resources, including patient-engagement expertise, to ensure meaningful engagement. Ongoing evaluation of engagement efforts is needed to assess the value this brings to both science and patient collaborators. Pilot testing and evaluating patient engagement may offer additional insights into processes, metrics, and outcomes. The considerations we report may serve as a foundation for building practical guidance about overcoming barriers to involving patients as meaningful collaborators in systematic review science.