Measuring the Impact of Patient Engagement in Health Research: An Exploratory Study Using Multiple Survey Tools

Abstract

Background

Studies report various ways in which patients are involved in research design and conduct. Limited studies explore the influence of patient engagement (PE) at each research stage in qualitative research from the perspectives of all stakeholders.

Methods

We established two small research groups, a Patient Researcher-Led Group and an Academic Researcher-Led Group. We recruited patient research partners (PRP; n = 5), researchers (n = 5), and clinicians (n = 4) to design and conduct qualitative research aimed at identifying candidate attributes related to patient preferences for tapering biologic treatments in inflammatory bowel disease. We administered surveys before starting, two months into, and post-project work. The surveys contained items from three PE evaluation tools. We assessed the two groups regarding the influence and impact each stakeholder had during the different research stages.

Results

PRPs had a moderate or a great deal of influence on the critical research activities across the research stages. They indicated moderate/very/extremely meaningful engagement and agreed/strongly agreed impact of PE. PRPs helped operationalize the research question; design the study and approach; develop study materials; recruit participants; and collect and interpret the data.

Conclusion

The three tools together provide deeper insight into the influence of PE at each research stage. Lessons learnt from this study suggest that PE can impact many aspects of research including the design, process, and approach in the context of qualitative research, increasing the patient-centeredness of the study. More comprehensive validated tools are required that work with a more diverse subject pool and in other contexts.

Introduction

Patient engagement (PE), has been defined as “active, meaningful and collaborative interaction between patients and researchers across all stages where decision-making is guided by patients’ contributions as partners recognizing their specific experiences, values, and expertise”.¹ Evidence supports PE as a way to improve research conduct including to improve the research question^2–5; study design^6,7; readability of study materials⁵; choice of study outcomes⁴; credibility of results (higher rates of enrollment and retention)^4,6,8–11; interpretation and communication of results^3,10,11; and influence on future research topics.¹² However, there is limited evidence of the influence of PE at each research stage and critical task in the context of qualitative research from the perspective of multiple stakeholders. This could be due to the many different factors that influence the causal chains leading to impact such as the skills, assumptions, values, and priorities of the stakeholders involved¹³ as well as the limitations in scope (e.g., measuring patient experience not impact on the research) and/or rigorous evaluation tools to measure PE (e.g., few measurement properties). There are also not many validated evaluation tools that are publicly available, informed by the literature and grounded in a theoretical or conceptual framework, inclusive of patient involvement in their development and reporting.^13–16 None provide a comprehensive picture of the impact across the whole research cycle.

We conducted an exploratory study to investigate the influence of patient engagement (PE) at each stage of research and critical task and self-perceived impact of PE using surveys administered at multiple timepoints. We examined how all stakeholders influenced and impacted the study from inception through completion.

Methods

Study design

We use the term “Patient Research Partner” (PRP) to describe our patients who joined the research groups on an equal basis with professional researchers. We collected survey data from a PRP-led group (PLG) and an academic researcher-led group (RLG). Each group comprised two PRPs, two clinicians, and two researchers in addition to a lead for a total of seven members in each group. Both groups designed and conducted a formative qualitative study to inform a preference elicitation study,¹⁷ specifically, a discrete choice experiment. Engagement of PRPs is recommended as good research practice^18,19 to generate a candidate list of the characteristics that are then considered for inclusion as attributes in the survey.²⁰ We provided the same question to both groups: “What factors or attributes are important to patients with inflammatory bowel disease (IBD) in considering treatment tapering of biologics?” to research independently for the seven-month duration of the project. We assessed how group members influenced and impacted the research at the different stages.

We recruited participants from the SPOR IMAGINE (Strategy for Patient-Oriented Research, Inflammation, Microbiome, and Alimentation: Gastro-Intestinal and Neuropsychiatric Effects) Network,^21,22 the Alberta Health Services Digestive Health Strategic Clinical Network,²³ and through professional contacts of the research team. Both these networks have trained PRPs, providing us opportunities to engage with patients immediately on the short-term project. We employed a maximum variation purposive sampling strategy to ensure a broad sample of PRPs. For our clinician and researcher group members and the two group leads, we used convenience sampling.²⁴

PRPs were eligible to participate if they were based in Canada, were currently taking or took some treatment for chronic digestive conditions such as IBD; had participated or currently participating in a health care initiative; and had received patient-oriented research (POR) training. The PRP lead had to have additional experience independently leading/facilitating all aspects of qualitative research activities.

Data collection

We administered three web-based survey tools (Table 1) at multiple timepoints: (1) Ways of Engaging ENgagement ACtivity Tool (WE-ENACT)—Patients and Stakeholders 3.0 Item Pool^4,25; (2) Public and Patient Engagement Evaluation Tool (PPEET-V2)²⁶; and (3) the Patient Engagement In Research Scale (PEIRS)-22.²⁷ We selected these tools after a rapid search of validated patient engagement self-reports that measure outcomes of patient–partner engagement in all phases of the research.^16,28 We initially considered one more additional measure—the Patients as Partners in research surveys²⁹ but this tool was dropped as it seemed more geared to identify behaviours that support productive partnerships and was quite lengthy.

Table 1.

Patient engagement measurement tools used in our surveys of project group stakeholders.

Tool	Original aim	Original description	Adaptations for this study
Ways of Engaging- ENgagement ACtivity Tool (WE-ENACT)—patients and stakeholders 3.0 item pool4,25	(1) To capture researchers’ experience with patient and other stakeholders’ engagement in research, (2) to describe the role of patients and other health care stakeholders in research projects, and (3) to describe engagement in research from the researcher point of view.	Designed to be flexible to apply to multiple time points in the project (annual intervals) by changing the time reference in each item. 28 open and closed-ended questions and takes 25–30 minutes to complete.	To capture stakeholder perspectives: how prepared stakeholders were to contribute to group work, what they did on the project, how they influenced each stage of the research and critical tasks from their perspective, and the challenges they faced working on the project.
The patient engagement in research scale-22 (PEIRS-22)27	To measure the degree of meaningful patient engagement in research projects from a patient perspective (to be completed by patients and family caregivers who work together with researchers on research projects).	Twenty-two items rated on a 5-point Likert scale and organized across seven domains: procedural requirements, convenience, contributions team environment and interaction, support, feel valued and benefits. Takes 3–7 minutes to complete.	To measure the quality or the degree of meaningful patient engagement in the project from all stakeholders.
The public and patient engagement evaluation tool (PPEET-Version 2)26	Includes three tools (1) an organization questionnaire to assess the organization’s capacity for, and culture of public and patient engagement; (2) a participant questionnaire to obtain participants’ assessments of key features of the engagement activity that they have participated in, and (3) a project questionnaire to assess the planning, execution and impact of the engagement activity after it has been completed.	The tool has not been developed for healthcare research. The original questionnaire includes 21 statements or questions. We used 19 closed and open-ended questions across four domains: communications and supports for participation, sharing your views and perspectives, impacts and influence of the engagement initiative, and final thoughts. We did not use two background questions as they were captured in the WE-ENACT tool. Takes 10–15 minutes to complete.	We used only the participant tool (one-time engagement) to capture all the stakeholders’ assessments of the key features of the engagement activity: how their participation in the project was supported, how they shared their views, how they influenced the research, their satisfaction with the engagement, and strengths.

No single tool had a set of measures for assessing the impact of PE by all stakeholders across multiple timepoints. We administered all the three tools to all the stakeholders in the two groups, not as originally developed. Vat et al. suggest that the impacts of PE can best be determined not by applying single indicators, but by a coherent set of measures.³⁰ Specifically, PPEET was developed in a non-health research context, PEIRS was developed to capture meaningful engagement of patient partners, and WE-ENACT was developed for direct inquiry of PE specifically within the Patient-Centered Outcomes Research Institute’s (PCORI) funded research teams (Table 1).

A screening survey was administered before the start of the project work; the second survey at two months into the project (early engagement); and the third survey at the end of the project (post-engagement; Fig. 1). The screening survey or baseline survey collected characteristics (demographics, experience, and knowledge about POR and training in POR, knowledge, and experience in qualitative research and training) to purposively place study participants in groups that were balanced to the extent possible. In addition to the items in the screening survey, items from the WE-ENACT were administered in the early engagement survey. The post-engagement survey contained items from all three tools.

Figure 1.

Timing of survey administration and tools used.

Data analysis

Descriptive statistics (frequencies, means) were used to summarize data. We used SPSS-26³¹ and Microsoft Excel for the analysis of the data. All tools were scored/coded in line with instructions/guidelines from developers. Each of the PEIRS Likert Scale responses was assigned a numeric value between 0 and 4 (strongly disagree to strongly agree) and a total score was calculated.³² A total score below 70.1 is considered “deficient”; between 70.1 and 82.7 is “moderately meaningful”; between 82.7 and 92.0 is “very meaningful”; and above 92.0 is “extremely meaningful” levels of engagement.²⁷ We converted the Likert-scale question responses to numeric values for the PPEET and the WE-ENACT items. Qualitative thematic analysis was conducted for all open-ended questions using NVivo-12.³³ We reviewed all the raw responses from the open-ended questions and then coded these responses using both deductive codes drawn from the research question and inductive codes generated by the data. We then grouped these initial codes into categories or themes.

Results

We present the results in ways that supported confidentiality and were relevant to the research question.

Study participants

Out of the twenty-nine eligible participants, fifteen declined participation due to workload issues and health concerns. We recruited fourteen participants including the leads as per our pre-determined sample size: (PRP; n = 5), researchers (n = 5), and clinicians (n = 4). Two PRPs, two researchers, and two clinicians were assigned to either a PLG or the RLG based on their demographics, years of qualitative experience, familiarity with each other, involvement in POR, etc. (Table 2).

Table 2.

Criteria/factors for purposive placement of participants in the two groups.

Criteria/factors for placement of participants	PLG	RLG
PRPs
1. 5+ years of qualitative experience 2. 5+ years of involvement in POR 3. Had led POR projects 4. Had qualitative training through their education 5. PACER trained	✓ ✓ ✓ ✓ ✓	✓ ✓ ✓ ✓ ✓
Researchers
1. Maximum qualitative research experience* 2. 5+ years of involvement in POR	✓ ✓	✓ ✓
Clinicians
Not doing collaborative work together currently	✓	✓

*Additional placement based on demographic characteristics. PRPs = patient research partners, PLG = patient researcher-led group, RLG = academic researcher-led group, POR = patient oriented research.

Of the fourteen participants, twelve had more than an undergraduate degree, ten self-identified as white women, and eleven were over thirty-five years old. Ten members reported more than a year of previous involvement in POR project tasks: from identifying research topics to disseminating research results. Three out of the five PRPs had completed the Patient and Community Engagement Research (PaCER) certificate training program^34,35 and the other two had research training backgrounds in qualitative and quantitative research. All researchers had qualitative research knowledge and experience. Only one had extensive inflammatory bowel disease (IBD) knowledge. Thirteen out of the fourteen participants remained until the end of the study. One PRP dropped out midway from the RLG as their needs and expectations were not met and one clinician did not complete the post-engagement survey.

WE-ENACT results: what did project group members do and how did they influence the qualitative study?

Members of the PLG and RLG described a wide range of activities in which they were involved during the course of the project (Table 3). PRPs in both groups were highly involved in the project; in all eleven critical activities in the PLG and ten critical tasks in the RLG. They shared their experience with IBD and biologics. Together with other group members, they operationalized the study question and co-designed the study and approach; participated in the literature review; developed or reviewed the study materials; recruited participants; collected data; and either analyzed the data or reviewed the results.

Table 3.

Group member involvement in project critical activities.

Critical research activities	Involvement in the project critical activities “Yes”
	PLG			RLG
	PRP	Clinician	Researcher	PRP	Clinician	Researcher
1. Getting to know your group	✓	✓	✓	✓	✓	✓
2. Deciding on how to work together	✓	✓	✓	✓	✓	✓
3. Helping the group understand what information is relevant to patients	✓	✓	✓	✓	✓	✓
4. Refining the group project question	✓	✓	✓	✓	✓	✓
5. Designing the study	✓	✓	✓	✓	✓	✓
6. Developing the study material	✓			✓		✓
7. Participating in the literature search	✓	✓	✓	✓	✓	✓
8. Training group members on how to recruit and work with patients	✓					✓
9. Finding patients to participate in the group project	✓		✓	✓		✓
10. Collecting data from the group project participants	✓			✓		✓
11. Analyzing data and/or reviewing results	✓	✓	✓	✓	✓	✓

PLG = patient researcher-led group, RLG = academic researcher-led group, PRP = patient research partner.

Clinicians were involved in seven critical tasks in both groups. Clinicians reported defining the study question, designing the study, reviewing the literature and the results besides sharing their experience treating IBD patients with biologics to help the group understand what information is relevant to patients. Researchers in the PLG were involved in nine critical tasks and in all the eleven tasks in the RLG. Researchers in the PLG were not involved in formal training and data collection.

Members in both groups perceived that their contributions influenced the research in meaningful ways at every project stage (Table 4). Their lived experience helped operationalize the study question and design a study that reflected and met the needs of patients affected by IBD, essentially increasing the patient-centeredness of the study design and approach. PRPs developed data collection instruments, collected data, and participated in data analysis and interpretation. This led to findings relevant to them and the community that will be impacted by these results. They created understandable and transparent study materials, which increased the accessibility of the study. They facilitated recruitment, data collection, and data analysis.

Table 4.

Group member role, influence, and impact during different project stages with illustrative quotes.

Project stages and critical research activities	What did group members do?	How did this make a difference in terms of influence and impact?
A) Launch stage
1. Getting to know your group	PRPs, researchers, and clinicians shared their backgrounds, living/lived experiences, biases and the purpose of engagement on the project.	Created collaborative teams “Our team was cohesive and had a high level of trust, we also understood each other’s strengths and interests.” PRP “I think it gave the team a better understanding of who we all are -- and who to look to for information at different times.” Researcher
2. Deciding on how to work together	PRPs, researchers, and clinicians in the PLG had specific roles and strategized on how to work together as a group, while RLG group members had no specific roles and volunteered to tasks as and when required.	Created collaborative teams “Everyone had opportunity to share their views on this subject, and presumably this made everyone feel like they were heard and the project reflected value to them.” Researcher Facilitated completion of the project on time “Allowed the team to focus on tasks and contribute meaningfully to the work, and allowed us to complete it on time” PRP
3. Helping the group understand what information is relevant to patients	PRPs shared their living/lived experience and clinicians shared their experience treating IBD patients with biologics.	Developed a more patient-centered study design and approach “(sharing the IBD experience) helped refine our recruitment criteria, informed literature review, influenced focus group design.” PRP Created understandable and transparent study materials “I think this has helped to shed light on the patient perspective and potentially tailor study materials in a patient-focused way.” PRP
B) Design stage
4. Refining the group project question	PRPs, researchers and clinicians decided the study question that included both the patient and the clinician perspective.	Developed understandable and transparent study materials “Doing so enabled us to develop project materials much more easily, as we had a specific goal to work towards” PRP Operationalized the project question “Ultimately agreed to a definition of the intent of the question.” PRP
5. Designing the study	PRPs, researchers and clinicians collaboratively determined the study design and approach including the inclusion-exclusion criteria, data collection approach, etc.	Developed a more patient-centred study design and approach “I was able to bring ideas about participatory action research, while other team members brought designs that they were more familiar with resulting in a study design that was collaborative and patient-focused.” PRP “We were able to include a broader range of individuals so there is a representative voice.” Researcher
6. Developing the study material	PRPs and researchers developed the study materials/reviewed the study materials.	Facilitated the data collection process “As a facilitator for the upcoming focus groups and interviews, it makes it easier to conduct these processes when I have been involved in planning for them.” PRP More understandable study materials “..in response to concerns in the group we tried to create wording which would not imply that the idea of tapering as a treatment was endorsed or supported by the group. I believe we reached an acceptable version that was sufficiently neutral in tone to satisfy concerns.” Researcher
7. Participating in the literature search (providing input on articles, adding references, etc.)	PRP, researchers, and clinicians either conducted the literature search and/or provided input on the literature and results of the search.	Helped deepen understanding of the topic that was helpful in designing the study, and in developing the data collection instruments “Reviewing the literature from the team helps deepen my understanding of the topic.” PRP
C) Data collection stage
8. Training group members on how to recruit and work with patients	Researcher trained a PRP on data collection.	Capacity building of PRP
9. Finding patients to participate in the group project	PRP and researchers either conducted the activity or supported recruitment by sharing recruitment flyers, suggesting venues.	Facilitated recruitment, PLG met recruitment targets RLG met clinician recruitment target, but not patient target “We were able to find enough participants that satisfied the requirements for diversity of perspective and data saturation.” PRP “Clinician recruitment was completed by designated group members. We needed more time to adjust patient recruitment strategies and recruit a sufficient number of patient participants.” Researcher
10. Collecting data from the group project participants	PRPs and researchers either collected the data or participated in discussions during data collection	Facilitated the data collection process “It helped to go over interview questions and how we were going to conduct patient interviews.” Researcher “Helped patients feel comfor and at ease to share information with us.” PRP
D) Data analysis stage
11. Analyzing or reviewing results	PRPs and researchers either conducted the analysis, or drafted or provided feedback on the code book, and discussed the final results along with the clinicians in their groups	Identified relevant codes, ensured context and statements included the patient perspective, succinct presentation of findings “Ensuring context of research and statements was appropriate.” PRP “This resulted in a succinct presentation of our study findings.” Researcher

PLG = patient researcher-led group, RLG = academic researcher-led group, PRP = patient research partner, IBD = inflammatory bowel disease.

The impact was determined by assessing the degree of perceived influence that stakeholders had on the activities (Table 5). In the PLG, the PRPs reported a moderate amount or great deal of influence on all eleven critical activities; the researchers reported a great deal of influence during the study design, data analysis, and literature review activities and none or a small amount of influence on the other eight critical tasks; the clinicians reported a moderate amount or great deal of influence during the literature review and helping the group understand what is relevant to patients, and a small or no influence on all the other tasks.

Table 5.

Numbers of PRPs, clinicians, and researchers in the two groups who indicated influence on the critical activities.

Critical activities	Perceived influence on critical activities by stakeholder and group
	PRPs				Clinicians				Researchers
	PLG (n = 3)		RLG (n = 1)		PLG (n = 2)		RLG (n = 1)		PLG (n = 2)		RLG (n = 3)
	N or SA	MA or GD	N or SA	MA or GD	N or SA	MA or GD	N or SA	MA or GD	N or SA	MA or GD	N or SA	MA or GD
1. Getting to know your group	0	3	1	0	2	0	0	1	1	1	0	3
2. Deciding on how to work together	0	3	1	0	2	0	0	1	1	1	0	3
3. Helping the group understand what information is relevant to patients	0	3	0	1	1	1	0	1	2	0	1	2
4. Refining the group project question	1	2	0	1	2	0	0	1	2	0	0	3
5. Designing the study	0	3	1	0	2	0	0	1	0	2	1	2
6. Developing the study material	0	3	1	0	2	0	1	0	2	0	0	3
7. Participating in the literature search	1	2	0	1	1	1	0	1	0	2	1	2
8. Training group members on how to recruit and work with patients	2	1	1	0	2	0	1	0	2	0	3	0
9. Finding patients	1	2	0	1	2	0	1	0	1	1	1	2
10. Collecting data	1	2	0	1	2	0	1	0	2	0	2	1
11. Analyzing or reviewing results	1	2	1	0	2	0	1	0	0	2	0	3

^*Post-engagement survey results. N = none; SA= small amount; MA= moderate amount; and GD= a great deal. Items scored on the Likert scale from none to a great deal. The two lowest and two highest Likert categories have been consolidated to summarize frequencies.

PLG = patient researcher-led group, RLG = academic researcher-led group, PRP = patient research partner.

In the RLG group, the PRP indicated a moderate amount or great deal of influence in refining the question, literature review, helping the group understand what information is relevant to patients, recruitment, and data collection. The researchers in this group indicated influencing most activities a moderate amount or a great deal, while the clinicians indicated influencing the project a moderate amount or a great deal in many of the early-stage activities.

PEIRS: level of meaningful engagement in the two project groups

Assessing meaningful engagement in each of the two groups involved looking at both the total scores of each member in a group and the mean group score. The total score of each group member in the PLG fell in either the extremely, or very meaningful level of engagement, while the total score of each member in the RLG fell in the very meaningful, moderately meaningful, and deficient levels of engagement (Table 6). The mean group score was between 82.7 and <92.0 in the PLG, indicating an overall very meaningful research project experience. The mean scores across all seven PEIR domains were above the cut-off points for deficiency (70.1) in this group. The mean group score in the RLG was above the cut-off point for deficient engagement (<70.1), indicating a moderately meaningful project experience. The mean score was deficient in two domains: team environment and interaction and benefits (Table 7). Members in this group indicated they did not feel a positive team environment and an inclusive team culture; nor did they feel that they personally benefited from their engagement.

Table 6.

Number of group members at the different levels of meaningful engagement by the two project groups.

Level of meaningful engagement	PLG (n = 7)	RLG (n = 5)
Extremely [>92.0 to 100]	2	0
Very [82.7 to <92.0]	5	1
Moderately [70.1 to <82.7]	0	2
Deficient [<70.1]	0	2

^*Post-engagement survey scores. The total score is the sum of all question sub-scores, ranging from 0 to 4, divided by 88 then multiplied by 100.

PLG = patient researcher-led group, RLG = academic researcher-led group.

Table 7.

Mean PEIR group scores of the two project groups.

	PEIRS 22 total score	Procedural requirements	Convenience	Contributions	Team environment & interaction	Support	Feel valued	Benefits
Total score	100	31.8	13.6	13.6	9.1	9.1	9.1	13.6
Cut off-point for deficiency	70.1	22.3	9. 6	9. 6	6.4	6.4	6.4	9. 6
PLG mean score	89	30	12	12	8	8	8	11
RLG mean score	73.2	24	10.6	9.6	6	6.6	7.4	9.2

^*Total domain scores below the cut-off point indicates deficiency in the specific domain.

PLG = patient researcher-led group, RLG = academic researcher-led group.

PPEET: stakeholder processes, outputs, and perceived impacts of engagement in the two project groups

Six group members in the PLG either agreed or strongly agreed with the items on the PPEET Likert-scale questions except for item B1 (I had a clear understanding of the purpose of the EP4 group work) as shown in Table 8. Three out of five members in the RLG were neutral or disagreed with item B1 as well. These members were also neutral or disagreed with five additional items on this scale. All PLG members and three RLG members strongly agreed or agreed that they are better informed about engagement in research as a result of their participation in this project. All PLG members and two RLG members were also satisfied with this engagement initiative. Similarly, all PLG and three RLG members indicated that this initiative was a good use of their time (Table 8).

Table 8.

Frequency of PLG and RLG responses of “strongly agreeable” and “agreeable” on the PPEET Likert scale items for the four domains.

Items	PLG (n = 7)			RLG (n = 5)
	SA	A	NAD	SA	A	NAD	D
B1. I had a clear understanding of the purpose of the EP4 group work.	1	4	2	0	1	3	1
B2. The supports I needed to participate were available (e.g., travel, child care, etc.).	3	4	0	1	2	2	0
B3. I had enough information to contribute to the topic being discussed.	4	3	0	0	5	0	0
B5. I was able to express my views freely.	4	3	0	3	2	0	0
B6. I feel that my views were heard.	4	3	0	2	2	1	0
B7. A wide range of views on the topics discussed was shared.	2	5	0	1	2	2	0
B8. The individuals participating in my group represented a broad range of perspectives on the topic.	3	4	0	0	2	2	1
B10. I think that our group work achieved its objectives.	3	4	0	1	1	1	2
B11. I am confident the input provided through this initiative will be used by the research study investigators.	2	4	1	1	1	2	1
B12. I think the input provided through this activity will make a difference to the work of the research study investigators.	1	5	1	0	2	2	1
B14. As a result of my participation in the EP4 group work, I am better informed about engagement in research.	4	2	1	2	1	2	0
B15. Overall, I was satisfied with this engagement initiative.	2	5	0	0	2	3	0
B16. This engagement initiative was a good use of my time.	2	5	0	0	3	2	0

^*SA= strong agree; A= agree, NAD= neither agree nor disagree; D= disagree, PLG = patient researcher-led group, RLG = academic researcher-led group.

Value, barriers, and improvements

As described, members from both groups indicated that they gained from this experience. Themes captured in the open-ended responses in the WE-ENACT and the PPEET tool were: opportunity to work collaboratively with a diverse and experienced team; opportunity to gain valuable experience in a virtual environment; and opportunity to learn about a unique topic. PRPs reported gaining new skills and/or improved their current skills, learnt more about IBD, and built new relationships. Researchers reported gaining new skills working in a virtual environment, learnt about IBD, and also about the value of PE in research.

Most stakeholders in the PLG reported that they felt trust, honesty, transparency, shared-learning, and give-and-take relationships somewhat or a great deal. There were mixed perspectives in the RLG with some indicating a great deal, some a little bit, and one participant indicating not at all in the early engagement survey. Sharing thoughts and ideas with the group was a salient example among respondents from both groups as it suggested comfort and willingness to engage in conversation. Having open lines of communication also made a positive difference in terms of trust, honesty, and transparency. Not having “assigned roles” on the project, “not feeling heard”, and “insufficient number of group members who were familiar with the topic” were some of the barriers to engagement reported in the surveys. Themes for improvement included: (1) needing a clearer picture of the study objective; (2) requiring more time to undertake a study of this nature; and (3) the study design.

Discussion

The three surveys administered pre-, early-, and post-engagement provided quantitative and qualitative data showing that active and meaningful collaboration between PRPs and researchers occurred in both groups. Quantitative findings revealed a moderate/great deal of influence of the PRPs on the critical research activities across the research process (WE-ENACT), moderate/very/extremely meaningful engagement (PEIRS) and agreed/strongly agreed impact of PE (PPEET). Group members gained valuable experience designing and conducting the two projects.

Although RLG members were neutral or dissatisfied with respect to some of the survey items (e.g., team environment), overall, our findings indicate that PRPs in both groups influenced and impacted the research question; the study design and approach; study materials; recruitment; data collection; and analysis and interpretation of qualitative research. The difference in PEIRS levels of meaningful engagement between the PLG and RLG could be attributed to sentiments revealed in the other tools such as the sense among RLG members that their group work did not achieve its objectives. Further, missing post-engagement data from two RLG members yields an incomplete assessment of meaningfulness for this group.

In our study, we measure the actual impact on all the research stages and not anticipated impacts of PE using three different tools. Our study is also unique in its emphasis on capturing engagement data from the perspectives of all group members at multiple points on the project timeline. There are limited quantitative studies and a few qualitative studies³⁴ that have studied PE longitudinally.

Our three evaluation tools captured the different aspects of PE that one tool could not have done alone. There are not many studies that have used two or three evaluation tools to measure PE in research in one study. Most studies use qualitative methods such as focus groups, semi-structured interviews, informal observation, and/or written surveys with open-ended text responses to formally assess the impact of engagement.³⁵ Bhati et al. used two tools, the PPEET and the WE-ENACT to assess patient experience and areas of involvement in three primary care research studies.³⁶ Thompson et al. used the PPEET participant questionnaire to evaluate the patient engagement processes within the project and the PEIR tool as a framework for the analyses.³⁷

In our study, the addition of the WE-ENACT tool allowed for a more nuanced understanding of PE through self-reports of group members’ roles and their self-perceived influence on the qualitative project activities. However, members of both groups preferred the PEIRS tool as an easier tool to fill out. These results emphasize the need for more validated tools¹⁰ or one good measure that is easy to complete and can empirically assess the impact of PE in research.³⁸ Developing one good measure, however, is challenging because of the different types of research conducted, different funder requirements, different stakeholders involved, different reasons for involvement, and different research contexts. The PPEET tool, for example, has been designed for and used primarily within health care institutions and less in the research context.³⁹ For these reasons, and others, mixed methods research designs, where both validated survey tools and qualitative data collection methods are used, may be the optimal approach in PE research.

We also ascertained the partnership needs for success such as having a common purpose, research interest, an engagement plan, etc.⁴⁰ Also evident in our findings is the need for time early in the process to build and sustain relationships, and for navigating challenges throughout the research, reiterating what is said in the literature.^6,40,41

Despite the insights gained from our findings, our study does have some limitations. We tried to match the groups as much as possible to facilitate comparison of the results but this was challenging given the small number of group members and the formation of the groups. Further, after the PRP withdrew from the RLG, one PRP remained, unbalancing the original two-researcher, two-clinician, and two-PRP dynamic established for both groups. This may have influenced the remaining PRP’s perception of significant impact on the critical research activities. A further potential limitation was the seven-month duration of the two group projects, which may not have provided sufficient time to develop trust and rapport, and created challenges for reaching the target sample size for one group. Additionally, the individual characteristics of group members, the limitations of having only two groups, and the specialized training of the PRPs^42,43 inhibit the generalizability of the results.

This research was conducted by a team with members from Alberta and British Columbia, Canada, and study participants from across Canada. The team used a POR approach to design and conduct the study. The study design, using multiple tools at multiple timelines, strengthened this research enormously. Another strength is the meaningful and active involvement of patients as research partners in both project groups, which differs from the traditional approach of patients as participants in researcher-led studies. Finally, our study fills a gap identified by a recent scoping review that did not find any studies that investigated the involvement of patients and public in inflammatory bowel disease (IBD) research.⁴⁴ Future research should examine the impact of PE in other contexts and study designs.

Conclusion

Few studies have measured PE in research with more than one evaluation tool and it is less common for studies to collect quantitative survey data to measure PE. The combination of three tools administered at multiple timepoints in our exploratory study offers a unique wealth of data about the influence and impact of PE in the context of qualitative research. Our findings highlight the need for validated, evidence-based, patient-involved tools that evaluate the impact of PE more comprehensively and in more diverse contexts throughout the research process.

Author contributions

Deborah A. Marshall conceptualized the study and led the design, conduct and analysis of this study, and the drafting of and revising of the article. Danielle C. Lavelle and Stirling Bryan conceptualized the study and led the design, conduct and analysis of this study, and helped revise the manuscript. Nitya Suryaprakash and Karis L. Barker participated in the design, coordination, data collection, conduct and analysis of the study, and in drafting and revising the manuscript. Paul Moayyedi contributed to the acquisition and interpretation of data and reviewed the manuscript critically. All other authors participated in the design, conduct and analysis of the study and reviewed the manuscript critically. All authors approved the final version to be published and agreed to be accountable for all aspects of the work.

Funding

Funding from the SPOR IMAGINE (Strategy for Patient-Oriented Research, Inflammation, Microbiome, and Alimentation: Gastro-Intestinal and Neuropsychiatric Effects) Network was used to conduct the study. The Network is supported by a grant from the Canadian Institute of Health Research (Funding Reference Number: 1715-000-001) with funding matched by McMaster University, University of Calgary, University of Alberta, Queen’s University, Dalhousie University, Montreal Heart Institute Research Centre, Takeda Pharmaceutical Company, Allergan Incorporated, Alberta Innovates, Research Manitoba, Crohn’s and Colitis Canada. The IMAGINE Network is sponsoring the open access fee.

Conflict of interest statement

Deborah A. Marshall discloses consulting fees from the Office for Health Economics, Novartis, and Analytica during the conduct of this study. She also received support from Illumina for travel expenses to attend a meeting. Nitya Suryaprakash and Karis L. Barker received reimbursement of expenses related to conference attendance from the SPOR IMAGINE Chronic Disease Network. All other authors declare no conflicts of interest relevant to the content of this article.

Data availability

The ethics approval for this study does not support the sharing of raw data.

Ethics approval

The study was approved by the respective research ethics boards of the University of Calgary [REB20-1563] and the University of British Columbia [H20-03385]. The authors certify that the study was performed in accordance with the ethical standards detailed in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Consent to participate

Informed consent was obtained from all individual participants included in the study.