Patient engagement partnerships in clinical trials (PEP-CT): protocol for the systematic development and testing of patient partner and investigator decision aids

Abstract

Introduction

Building capacity to improve sex/gender knowledge and strengthen patient engagement in clinical trials requires training and support. The overall goal of this 2-year project is to refine, translate and evaluate two web-based open-access patient and investigator decision aids aimed to improve patient engagement partnerships in clinical trials.

Methods and analysis

Two decision aids were designed in Phase 1 of this programme of research and this protocol describes a subsequent sequential phased approach to refine/translate (Phase 2A) and conduct alpha/usability (Phase 2B) and beta/field (Phase 3) testing. Decision aid development is guided by the International Patient Decision Aid Standards, User-Centred Design, Ottawa Decision-Support Framework and the Ottawa Model of Research Use. We have integrated patient-oriented research methods by engaging patient partners across all phases of our programme of research. Decision aids will first be refined and then translated to French (Phase 2A). Eight iterative cycles of semistructured interviews with 40 participants (20 patient partners and 20 investigators) will be conducted to determine usability (Phase 2B). A pragmatic pre/post pilot study design will then be implemented for field/beta testing using another purposive sample of 80 English-speaking and French-speaking participants (40 patients and 40 investigators). The samples are purposive to ensure an equal representation of English-speaking and French-speaking participants and an equal representation of men and women. Since sex and/or gender differences in utilisation and effectiveness of decision aids have not been previously reported, Phase 3 outcomes will be reported for the total sample and separately for men and women.

Ethics and dissemination

Ethics approval has been granted from the University of Toronto (41109, 28 September 2021). Informed consent will be obtained from participants. Dissemination will include co-authored publications, conference presentations, educational national public forums, fact sheets/newsletters, social media sharing and videos/webinars.

Strengths and limitations of this study.

• Purposive sampling to assist in recruiting participants of English-speaking and French-speaking patients and investigators who are representative of men and women from academic and community settings within urban and rural locations in Canada.

• Reliable and valid measures to assess acceptability and the core attributes of decision-making processes and decision choice to minimise information/measurement bias.

• Although the Decisional Conflict Scale is most widely used, there is potential for increased bias in patients with limited reading skills.

Introduction

Patient-oriented research (POR) aims to actively engage patients and their caregivers throughout the research process to better facilitate the prioritisation and research of patient-identified problems. Patient engagement in research has mostly been limited to providing feedback, rather than more active collaborative processes to design and governance, data analysis and knowledge dissemination.^{1 2} Patient engagement in research is influenced by three core values: developing best practices, improving research impact and quality and building strong relationships.³ Building genuine partnerships guided by collaboration, honesty and trust are important for all stakeholders; including patient partners, healthcare providers, researchers, industry and policy makers.³ Recommendations for developing patient partnerships suggest that the following are essential in building capacity for POR: (1) positive researcher attitudes grounded in shared goals and strong communication practices, (2) supportive institutional policies, (3) values of trust, respect and co-learning, (4) tools/resources for effective patient engagement, (5) necessary training for team members and (6) value for patient partners in all stages of research.⁴

A 2015 report commissioned by Clinical Trials Ontario provided information regarding patient engagement in clinical trials. One of the report’s recommendations was to offer resources/tools and best practices for patient engagement (eg, decision aids).⁵ Decision aids are interventions that increase knowledge surrounding expectations,⁶ advantages and disadvantages,⁶ and choices and outcomes.⁷ Users can develop skills in assessing uncertainties and highlight personal priorities using the advantages and disadvantages of participating^{7 8} in a particular decision.^{7 8} Research evidence supports the benefits of decision aids in increasing knowledge^9–11 and improving expectations concerning priorities and choice.¹⁰ A 2017 Cochrane Review found that patient decision aids reduce decisional conflict related to feeling uninformed and decrease the proportion of individuals who are passive in decision-making.¹¹ At a minimum, decision aids improve the: (1) quality of the decision-making processes, and (2) quality of the choice that is made.¹² However, potential sex and/or gender differences in utilisation and effectiveness of decision aids were not reported. Core attributes of the quality of decision-making processes include: (1) recognition that a decision needs to be made, (2) being informed about options and benefits/risks, (3) value clarity, (4) discussion about goals/preferences and (5) involvement in treatment decision-making.¹² The quality of choice is the extent to which end-users are informed and make decisions that reflect their goals and preferences.⁷ The decision aids developed in Phase 1 of this programme of research are not designed to assist patients in making treatment decisions; they are designed to assist patients and investigators in making decisions about engaging as/with patient partners in research.

In addition to limited sex and/or gender differences in utilisation and effectiveness of decision aids, there has been limited sex and gender reporting in clinical trial research.^13–15 There continues to be poor uptake of sex and gender in clinical trial research in Canada; of the trials published between January 2013 and July 2014, 6% (n=6) conducted a subgroup analysis of sex, 4% (n=4) reported sex-disaggregated data and none defined sex or gender or conducted a sex- or gender-based analysis.¹⁶ Many data collection instruments fail to incorporate variables associated with sex and gender, such as income and household responsibilities. The terms sex and gender also continue to be used interchangeably and incorrectly in research, suggesting a lack of understanding that these are distinct concepts.^{14 17}

Objectives

The overall goal of this 2-year project is to refine, translate and evaluate two web-based open-access patient and investigator decision aids designed in Phase 1 of this programme of research (online supplemental file 1). These decision aids aim to improve sex and gender knowledge and patient engagement partnerships in clinical trials, which we refer to as ‘PEP-CT’. The objectives for the next phases (2A, 2B and 3) are as follows: (1) to refine content and functionalities of the decision aids and translate them to French as the second most commonly spoken language in Canada (Phase 2A), (2) to further refine the patient partner and investigator decision aids through usability testing (Phase 2B) and (3) to assess patient and investigator decisional conflict related to PEP-CT (Phase 3). Secondary objectives of Phase 3 are to evaluate: (1) sex/gender and POR knowledge, and (2) patient and investigator acceptability and engagement with the respective decision aid. Additional exploratory objectives of Phase 3 are to: (1) conduct a formative evaluation of the use of the decision aids to assess the predisposing, enabling and reinforcing factors¹⁸ that may impact the ability of each aid to support informed decision-making and (2) evaluate adoption and impact (eg, uptake by end-users) of each decision aid.

Methods and analysis

Decision aid functionalities were developed through a scoping review with input from health charity and patient organisations, research administrators and industry via a full-day POR consultation workshop (Phase 1A) and results have previously been published.¹⁹ The two decision aids were developed for initial dissemination and feedback via a New/Early Investigator Training Day and disseminated for feedback in our Building Capacity for Patient-Oriented Research (POR) in Clinical Trials, Translating the Evidence into Practice, Policy and Outcomes (POR STEPP) Digital Health Project in Ontario (Phase 1B). Knowledge gained from the POR STEPP Digital Health Project will be used to further refine the decision aids and translate them to French (Phase 2A). Alpha/usability testing and further refinements of the decision aids will then take place via iterative cycles (Phase 2B), followed by beta/field testing of the patient and investigator decision aids for large-scale implementation (Phase 3).

Patient and public involvement

The PEP-CT patient partners will continue to collaborate throughout Phases 2 and 3 of this programme of research. In Phase 1, patient partners informed research priorities and identified search terms. They extracted and collated data from the scoping review and co-presented at the consultation workshop, New/Early Investigator POR Training Day and through online webinars and a conference presentation. Patient partners co-led the development of the Patient Decision Aid and co-authored the Phase 1 manuscript.¹⁹ Patient partners helped identify priorities for Phases 2 and 3 and will collaborate in alpha and beta testing and knowledge dissemination of the revised Patient and Investigator Decision Aids.

Phase 2A: refinement and translation

Procedures

Phase 2A focuses on refining and translating both the patient partner and investigator decision aids. Refinements identified in Phase 1 align with both the Integrate, Design, Assess and Share²⁰ and the WHO²¹ frameworks for disseminating and scaling up innovations. General refinements will include increasing font size, incorporating more white space and visuals/videos and ensuring language/content reflects diversity in race and ethnicity. Racial and ethnic diversity will be demonstrated using acceptable literacy, visuals/videos that promote equity, diversity and inclusion, and by translating both decision aids to French. Further refinements include the addition of a glossary, hover-over text, bookmarks, hyperlinks to existing resources, adapting print-friendly sections and ensuring access to decision aids on all devices (ie, iPAD). Hyperlinks to existing organisational information/resources will be added preceding My Decision to better guide individuals to find a patient partner or a clinical trial project. Usefulness of the investigator decision aid for researchers already interested in POR and patient partner training (eg, screening, data extraction) will be highlighted. Integration of broader language to demonstrate the application of the patient and investigator decision aids to research projects broader than clinical trials will be incorporated. The patient decision aid text will be reviewed to ensure a grade 5–6 reading level.²² We are currently completing all refinements and then both decision aids will be translated to French.

Phase 2B: alpha (usability) testing

Study design

A qualitative approach using semistructured, audiotaped interviews and user observation will be undertaken by a trained observer in iterative cycles to determine usability of the patient partner and investigator decision aids. The iterative rapid design in Phase 2B will focus on user performance (ease of use, efficiency, ease of learning and errors) and satisfaction with decision aid content and functionality (resources, web-based design).²³

Sample

A single coordinating centre will recruit a purposive sample of 20 English-speaking and French-speaking patients (men and women) and 20 English-speaking and French-speaking investigators (men and women) through Clinical Trials Ontario, Network of Women, Canadian Cancer Clinical Trials Network, Canadian Arthritis Patient Alliance, Cystic Fibrosis Canada, Canadian Skin Patient Alliance, Brain Tumour Foundation of Canada, Huntington Society of Canada, Sickle Cell Awareness Group of Ontario, Myeloma Canada and the SPOR Support Units and Chronic Disease Networks. Based on previous experience^24–26 and recommendations that usability testing by 3–5 users finds approximately 85% of interface usability problems,^{27 28} each usability cycle will include five end-users.

Eligibility criteria

Patient partners and investigators will be greater than 18 years of age and be fluent in either English or French. Access to a computer or another device with internet will also be mandatory.

Study setting

Participants will engage in one-on-one observation for 60–90 min via audio/video conferencing using ZOOM. Informed consent (online supplemental file 2) for participation and audiotaping will be done prior to the interview, along with completion of a Demographic and Clinical Information Form.

Procedures

Eight usability cycles in total are planned: 2 cycles of patients (English, men and women), 2 cycles of patients (French, men and women), 2 cycles of investigators (English, men and women) and 2 cycles of investigators (French, men and women). After completion of the first cycle for each group, changes will be made to the respective decision aid (ie, patient and investigator). The revised decision aids will then be evaluated in subsequent cycles. These iterations usually require 2–3 testing cycles with each end-user group until no further comments are identified.^{26 27 29} We will provide four testing cycles for each decision aid to accommodate for an equal number of men and women, and English-speaking and French-speaking end-users (figure 1). Travel reimbursement and compensation will be offered to participants based on time (ie, $50 for an estimated 2 hours to use each decision aid).

Figure 1.

Alpha and beta testing. PEP-CT, patient engagement partnerships in clinical trials; POR STEPP, Building Capacity for Patient-Oriented Research (POR) in Clinical Trials, Translating the Evidence into Practice, Policy and Outcomes.

Each participant will be provided with a brief explanation of the decision aid in their language and then asked to move through the required features: (1) Introduction (get facts on POR/PEP-CT), (2) My Priorities (where in the research process patient partners can be engaged, including levels of engagement), (3) Learn More (to plan, engage and evaluate PEP-CT (including sex/gender)), (4) My Readiness (comparing priorities with perceived benefits/risks) and (5) My Decision (decision and next steps, such as finding a patient partner or finding a clinical trial). We will employ a ‘think aloud’ approach³⁰ to gather insight into the way users move through the decision aids. Comments will be recorded, and the research coordinator will make field notes about any problems encountered on the Usability Testing Error and Efficiency Documentation Form. At the end of the session, participants will be asked to complete the System Usability Scale.^{31 32} Ten 5-point Likert questions will be scored to provide a point estimate of usability. In addition, four semistructured questions will be asked to determine users’ overall impression of the decision aid; what they liked and why, what could be improved, and if anything was missing.²⁶

Analysis

Quantitative data from the Demographic and Clinical Information Form, Usability Testing Error and Efficiency Documentation Form (adapted for use in this study)³³ and the System Usability Scale will be analysed using descriptive statistics in SAS V.9.2.³⁴ The 10 5-point Likert questions scores will be analysed to provide a point estimate of usability with a reported reliability of 0.85.³² The semistructured interview audiotapes will be transcribed and translated to English. Transcribed data will be managed and imported into NVivo.³⁵ Two members of the investigative team will use simple content analysis to obtain an understanding and develop codes after each iterative cycle. These codes will be used to generate themes. Disagreements about themes will be handled through consensus and a third member of the investigative team.^{36 37} Raw data will be revisited on a regular basis to ensure codes and resulting themes are grounded in the data.³⁸ In addition, we will collect information on sex and gender to provide recommendations on any sex and gender differences in usability, which could inform Phase 3 of this project.

Outcomes

This iterative user-centred design and the semistructured interview questions will assist us to understand the proposed requirements of each decision aid and whether each is suited to both men and women end-users. Patients and investigators will be able to assess the appropriateness and ease of use of each decision aid prior to the pragmatic pre/post pilot study (Phase 3).

Phase 3: beta (field) testing

Study design

Beta/field testing of the refined decision aids will be conducted using a pragmatic pre/post pilot study design. Field testing will be guided by the Ottawa Model of Research Use.³⁹ The Ottawa Model of Research Use provides a framework for evaluating knowledge translation innovation implementation using six key elements: (1) change agents and resources, (2) evidence-based innovation, (3) environmental barriers and facilitators, (4) awareness and skills/training, (5) adoption and (6) impact (eg, uptake by end-users). In the context of the decision aids, the potential adopters include patient partners and investigators interested in clinical trial research.

Sample

We will recruit 40 English-speaking and French-speaking patients (20 men, 20 women) and 40 English-speaking and French-speaking investigators (20 men, 20 women). Participants will be recruited through the same process as Phase 2B. The sample size was chosen based on Hertzog’s recommendation⁴⁰ of a minimum of 20–30 participants for single sample pre/post studies. In addition, based on Cochrane data⁴¹ and other recommendations,⁴² for a level of significance of alpha=0.05 (two-sided), power=0.80, a pre and post SD of 0.81 and a correlation between pre and/post scores of 0.80, a pre and post change of 0.34 in decisional conflict scores will be able to be detected with a sample size of 20 participants. This effect size can distinguish between being ready and not being ready to make a decision. However, in order to do a sex-based analysis with the same precision within each sex, 40 English-speaking and French-speaking patients (20 men and 20 women) and 40 English-speaking and French-speaking investigators (20 men and 20 women) will be recruited.

Eligibility criteria

Patients will be eligible if they have not participated in Phase 2B and have had no previous experience in participating as a patient partner on a research team. If patients have previously been a participant (ie, not a patient partner) in a clinical trial, they will be eligible to participate in this phase. Patient partners and investigators will be greater than 18 years of age. As the decision aids are web-based, access to a computer and/or other device with internet will be mandatory.

Procedures

Interested participants will contact the research coordinator by telephone or email to express their interest using a decision aid and participating in the study. Eligibility criteria will be confirmed, and informed consent obtained (online supplemental file 2). Participants will complete an online Demographic Form and baseline measures (ie, sex/gender and POR knowledge). Participants will then review either the patient or investigator web-based open-access decision aid. Engagement will be assessed using Google Analytics/Google Tag Manager (eg, event tracking and heatmap tools). Choice predisposition has been incorporated into the design features of each of the decision aids. After using their respective decision aid, patient partners and investigators will be asked to mark along a 5-point choice predisposition scale anchored by ‘engage’ or ‘not engage’ as a patient partner (patient decision aid) or with a patient partner (investigator decision aid). Response options in the centre indicate that the participant is ‘undecided’. Test–retest reliability of various iterations of the choice predisposition scale in various populations has exceeded 0.90, values and expectations have also been consistently correlated with choice predisposition.^{43 44} Decisional conflict will be assessed at post-test using the 16-item Decisional Conflict Scale,⁴⁵ which measures personal perceptions of: (1) uncertainty in choosing options, (2) modifiable factors contributing to uncertainty (eg, feeling uninformed or unclear about priorities/values) and (3) effective decision-making (eg, feeling the choice is informed, values-based and satisfied with the choice). Five subscales (informed, values clarity, support, uncertainty and effective decision) contribute to a total score that ranges from 0 (no decisional conflict) to 100 (extremely high decision conflict). Knowledge of sex/gender and POR will be assessed at pretest and post-test using two separate Sex/Gender and Patient-Oriented Research Knowledge Scales, one for patients and one for investigators, developed for use in this study. Based on Cochrane data,¹¹ the knowledge scale questions will be based on information contained in the decision aids. The proportion of correct responses will be converted to a percentage scale ranging from 0% (no correct responses) to 100% (all correct responses). Decision support acceptability will be assessed using a modified Acceptability E-Scale (AES)⁴⁶ at post-test only. Lastly, telephone interviews and a brief semistructured interview guide at 6 months will assist to assess predisposing, enabling and reinforcing factors of adoption and uptake in context of each decision aid based on the Ottawa Model of Research Use for evaluating knowledge translation innovation implementation and the WHO²¹ framework for disseminating and scaling up innovations. English and French interviews will be conducted by two team members experienced in conducting interviews. Field notes will be taken.

Analysis

The focus of the analyses is on descriptive statistics rather than formal tests of hypothesis (ie, we are not testing for statistical significance).⁴⁷ Since sex/gender differences in utilisation and effectiveness of decision aids have not been previously reported, we will report outcomes for the total sample and separately for men and women. Quantitative data will be analysed using version SAS V.9.2.³⁴ Since decisional conflict is our primary outcome, criteria for success will be defined as low decisional conflict (80% of scores <25). All differences between pretest and post-test knowledge scale scores will be assessed using McNemar’s test for binary variables and the paired t-test for (pseudo) continuous variables. We also anticipate high acceptability (80% of AES scores >24), and moderate to high post-test and 6-month engagement, defined as: (1) 80% of patients will access the PEP-CT Patient Partner decision aid at post-test and 6 months, (2) 80% of investigators will access the PEP-CT Investigator decision aid at post-test and 6 months, (3) end-users will access >80% or 4/5 functionalities (Introduction, My Priorities, Learn More, My Readiness and My Decision) at post-test and 50% of the functionalities at 6 months. Audiotapes and field notes from the interviews will be transcribed, translated and imported into NVivo.³⁵ Further analyses incorporating codes, content analysis and disagreement processes will be conducted in a similar way to phases Phase 2B. In addition, we will provide recommendations on any sex/gender differences that may impact knowledge translation.

Outcomes

Results of the pragmatic pre/post pilot study of the bilingual decision aids will establish the extent to which each decision aid is feasible in terms of implementation (acceptability, engagement and fidelity). Phase 3 field/beta testing will also enable us to finalise, disseminate and evaluate adoption of these open-access web-based innovations, with an anticipated planned end date of Spring 2023.

Ethics and dissemination

Ethics approval has been granted from the University of Toronto (41109, 28 September 2021). Informed consent (online supplemental file 2) will be obtained from all participants engaging in Phase 2B and Phase 3 of the study. We will disseminate knowledge of the decision aids through co-authored publications, conference presentation, educational national public forums, fact sheets/newsletters, social media sharing and videos/webinars.

Ethics statements

Patient consent for publication

Not applicable.