Discrete choice experiment (DCE) to quantify the influence of trial features on the decision to participate in cystic fibrosis (CF) clinical trials

Abstract

Introduction

Engaging people with cystic fibrosis (CF) in clinical trials is critical to improving outcomes for this fatal disease. Following extensive exploration of engagement in CF trials we believe six key concepts require a quantitative understanding of their influence in the current CF trials landscape including how controversial issues like placebos, washouts, stipend provision and location of trial visits are viewed by the CF community and how these might be modified depending on the type of medicine being investigated and the mechanism of access to the drug on trial completion.

Methods and analysis

We have designed and will administer an online discrete choice experiment to elicit and quantify preferences of people with CF for these trials’ attributes and estimate the relative importance of an attribute when choosing to participate in a trial. The cross-sectional data generated will be explored using conditional multinomial logit model. Mixed logit models such as the random-parameters logit and a latent class models will be used to explore preference heterogeneity. To determine the relative importance of an attribute, the difference between the attribute level with the highest preference weight and the level with the lowest preference weight will be calculated.

Ethics and dissemination

Imperial College London Joint Research Compliance Office has granted ethical approval for this study. Patient consent will be sought following full explanation. No identifying information will be collected. Dissemination will be via international conferences, peer-review publication and patient accessible forums. Major CF trials networks have agreed to incorporate our findings into their review process, meaning our results can realistically influence and optimise CF trial delivery.

PROSPERO registration number

CRD42020184886.

Strengths and limitations of this study.

• Discrete choice experiments (DCEs) bridge the gap between classical experimental methods investigating decision making which are constrained by practical and ethical difficulties, and descriptive methods which introduce numerous sources of bias and necessitate small sample sizes, posing questions about validity and generalisability.

• This DCE was developed following good research practices defined by International Society for Pharmacoeconomics and Outcomes Research guidelines and standard practice in preference studies and has a number of strengths derived from the use of best practices.

• There are some unavoidable potential sources of bias, including the selection bias seen with all online voluntary surveys, information bias from explanation of complex information within the survey, residual confounding bias and the differences between stated and real-world preferences.

• We have made every effort to address potential biases, including careful design of realistic scenarios, extensive pretesting with relevant stakeholders, advertising through multiple channels and analysis using advanced statistical techniques. Where potential bias cannot be mitigated, we acknowledge the source of bias.

• Our collaboration with the major cystic fibrosis (CF) trials networks in Europe and the UK mean our results can realistically influence and optimise CF trial delivery of clinical trials and enhance their acceptability to people with CF in Europe and beyond.

Introduction

Approximately 11 000 people in the UK have cystic fibrosis (CF).¹ Despite substantial progress, CF still carries a high symptom and treatment burden, and a significantly reduced life-expectancy.¹ Recent breakthroughs in modulator therapies offer substantial hope for improved health for nearly 90% of the CF adolescent and adult population.^{2 3} However, while hugely exciting, these breakthroughs do not represent a cure. Much work remains to identify treatments for those with rare mutations, to ensure that treatment starts as early as is safe and practicable, to optimise the choice of modulator therapies and regimen and improve traditional therapies targeting the downstream consequences of established disease.^{4 5} There are more than 100 drugs in the development pipeline,⁶ but without patient participation, clinical trials will not succeed. Therefore, engaging people with CF in research is essential to improve outcomes for this fatal disease.

Making research more patient-centred is a priority of research organisations such as the National Institute for Health Research.⁷ It has been shown to improve research quality, efficiency of enrolment and participant satisfaction and well-being^7–10 and may make trials accessible to a more diverse, therefore more representative, patient cohort.

Oncology leads the field in identifying ways to make research patient-centred.^11–13 The findings provide insight into the significance and complexity of the discipline and have some transferability to patients across all disease groups. However, the trials landscape in one disease may be very different from another disease. Factors such as number of competing trials, disease characteristics, predominant demographics of the patient population and available treatments appear to influence participant engagement in research.^14–18

Our team analysed data arising from the ‘Patient Experiences of Research Participation’ study.¹⁹ Participants with chronic obstructive pulmonary disease and obstructive sleep apnoea underwent semi-structured interviews after completing a trial at our site. A key finding was that even when interviewing only patients with respiratory disease, individual disease characteristics and cohort demographics appeared to influence patients’ interaction with research and their reasons for engaging in trials.¹⁹ This builds on our view that the factors influencing the decision to participate in trials cannot be reliably extrapolated from other disease groups and supported our decision to lead a CF specific exploration.

This exploration included literature review (Registered on PROSPERO, CRD42020184886), focus groups and a national Delphi study conducted with relevant stakeholders. These mixed method approaches built a rich picture of the breadth of factors that influence engagement in trials by people with CF and identified several potentially modifiable barriers/facilitators to participation.²⁰ Many of these would be simple to implement and have already led to practice changes locally and beyond. Others require a fundamental overhaul of trial design and delivery. Following this work, we identified six key concepts within the CF trial landscape which we believe require a more robust understanding of how they are perceived by people with CF and how they affect their decisions whether or not to participate in a trial:

• Does the type of medicine being investigated influence people’s willingness to participate?

• Are people with CF prepared to attend a site different from their usual clinical centre for trial visits, and how far are they prepared to travel?

• Does stipend provision influence the decision to take part in trials, and what level of stipend is optimal?

• Are drug washout periods acceptable to patients?

• Are placebo-controlled trials acceptable to patients?

• Does the mechanism of accessing the experimental medicine after the trial is finished influence the decision to participate, and what are the preferred methods of access?

We plan to use a discrete choice experiment (DCE) to quantify the effect of these key features on decisions to take part in trials. By gaining a quantitative understanding of how these factors interact and influence patients’ decisions to participate in trials, we hope to gain insight into these more controversial aspects of trial design and delivery. The European Clinical Trials Network (CTN) protocol review committee reviews all submissions and provides feedback to sponsors, which is mandatory for any trials being run through the network. CTN is supportive of our work and, as appropriate, will adjust their processes in light of our findings. The Clinical Trials Accelerator Platform (CTAP) is a UK based CF trials network, which has similarly agreed to incorporate our findings into their review processes. This means that our findings can realistically influence trial delivery across Europe and beyond to optimise the acceptability of clinical trials to people with CF.

Aims

Primary

• To elicit and quantify preferences of people with CF for clinical trials’ features.

• To estimate the relative importance of these features when choosing to participate in a clinical trial.

Secondary/exploratory

• To explore differences in preferences across subgroups of respondents (such as perceived disease severity and access to modulator therapy through clinic).

Methods

Methodological choice

Classical experimental methods investigating decision making are constrained by practical and ethical difficulties. Descriptive methods introduce numerous sources of bias and necessitate small sample sizes, posing questions about validity and generalisability. DCEs bridge this gap. They are based on the principle that products or services comprise multiple features, and that an individual’s choice of a product or service is a function of the utility of each feature.²¹ Thus, DCEs can be used to elicit preferences for features of a product or a service. Statistical testing of pooled results from a sample of participants allows researchers to investigate the relative value that participants place on multiple features when making complex decisions. DCEs have been used to elicit health preferences since before 1990 in a wide range of healthcare areas.^{21 22}

Our six questions can be isolated and defined as a feature, or attribute, with two or more fixed levels. We will use the DCE to quantify which attributes and levels exert the strongest influence on people with CF when deciding to accept/decline participation.

Study design

The first step of designing the DCE is defining attributes and levels that answer the research questions. Identifying the most appropriate attributes and levels is critical to ensuring the study’s validity and relevance.^{23 24} The attributes and levels should be representative of the range of scenarios that participants may encounter and sufficiently specific and distinguishable to minimise interactive effects. The team used mixed methods preliminary work and their extensive experience in CF trials to establish attributes and levels that mimic realistic scenarios and choices in the current or future trials landscape (table 1). The number of attributes⁶ and levels^2–4 represent a balanced set of options and are consistent with several high quality DCEs published in healthcare.²²

Table 1.

Showing the defined attributes and levels to be investigated in the discrete choice experiment

Technical attribute	Levels
Type of medicine	Modulators
	Mucolytics and anti-inflammatories
	New antibiotics
Where you attend your trial visits	Your usual clinical centre
	Any CF centre within 1.5 hours of your home
	Any CF centre within 3 hours of your home
	Any CF centre in the UK which is more than 3 hours away from your home
The stipend you may receive	No additional stipend
	£10 stipend per visit
	£30 stipend per visit
	£70 stipend per visit
What happens with your usual medicines	No washout period
	4-week washout of modulator
	4-week washout of non-modulator
Trial design	Placebo controlled
	Open label
Access to the medicine after the trial	No special access
	Priority access to later trials
	Open label extension phase
	Managed access programme

CF, cystic fibrosis.

These attributes and levels are then combined to create hypothetical choice tasks for participants (figure 1). As participants are never obliged to take part in clinical trials, the choice participants are offered is ‘trial A’ or ‘trial B’ or ‘no trial’ rather than requiring a forced response to participate.

Figure 1.

Figure showing the layout of an example discrete choice experiment choice to be presented to participants.

Once levels and attributes were defined, we developed a survey instrument. The instrument contains a standard introduction, an explanation to assist participants in understanding the attributes they are being asked to consider and an outline of the task. It describes the fixed features of the trial that remain constant such as the visit schedule and length of study, modified from three current large-scale CF trials. The text is punctuated by quiz questions to keep participants engaged and confirm their understanding of the complex concepts in the study. The instrument also includes screening questions to determine eligibility and questions to collect baseline demographic characteristics and other details such as location of clinical centre and access to modulator therapy.

RD and MB completed pretests with people with CF, purposively sampled to represent a range of trials experience. Subjects were asked to read the background information and compete the quiz questions to identify areas where the task was unclear or the concepts difficult to follow. Changes were incorporated after each test. The pretest instrument contained a set of choice tasks. We asked subjects to talk through their decisions so we could check that they could clearly identify the options and make a trade-off when considering each choice set. This allowed the study team to identify previously unnoticed confounders and refine the attributes and levels. Pretests 5 and 6 did not reveal any new concerns so pre-testing was ended after six participants.

The combination of levels used to define each profile, the set of profiles in each choice question, and the full set of choice questions in a DCE is known as the experimental design. These combinations of profiles must have statistical properties that allow estimation of the main-effect preference weights of interest. A full factorial design can be produced, in which every possible combination of level and attribute is presented to the participants. However, for many DCEs, this would result in hundreds of choice tasks. Considering the attributes and levels in our survey, a full factorial design would generate 1152 combinations for each alternative (4³×3²×2), resulting in a design that includes 1 327 104 questions. This would be impossible for participants to complete with any consideration- resulting in meaningless data. A fractional factorial experimental design overcomes this challenge by presenting participants with fewer choice tasks with specific combinations of attributes and levels.

The minimum number of tasks that can generate a well-balanced optimal design with six attributes with 4, 3 and 2 levels to be presented to a sample of 200 respondents is 72. The pilot testing found our target audience were able to engage well with 12 choice tasks. Thus, our final experimental design was based on presenting 12 choice tasks. Therefore, the experimental design was split into 6 blocks, comprising 12 choice tasks each. We used the commonly employed D-optimal algorithm with no prior assumptions as to how respondents would answer to construct a fractional factorial experimental design.^{25 26}

The final survey instrument was programmed into the internet-based data collection software, Qualtrics. Each participant will be randomised to receive one of the six blocks. Within each block, the questions will be randomly presented in different orders to minimise ordering effects and the impact of potential learning and fatigue. The survey will be administered online to allow participation across the whole of the UK. Qualtrics can be programmed to include a user-friendly interface with useful features including the ‘tool tip’ function, whereby participants can hover over the attributes and levels and receive a reminder description of what the options mean.

Sample size

Sample size calculations are challenging in DCEs because the models used to estimate preference weights simultaneously estimate multiple coefficients, and researchers often have little prior expectations of effect sizes. Most published choice experiments have sample sizes between 100 and 300 respondents.²⁷ However, minimum sample size depends on various criteria, including the complexity of the choice task and the desired precision of the results.²⁷ Considering the results from the pretest, the attributes and levels included, and the modelling approach planned, we assume a sample of 200 respondents will be sufficient to achieve the study’s aims.

Recruitment

We will advertise the survey over multiple social media platforms including Twitter, Facebook and Instagram. CTAP will advertise the survey in their newsletter. We will aim for a sample of 200 respondents, with no maximum sample size and no formal stratification. We will close the survey when we have recruited 200 people or at 8 weeks, whichever happens latest. If we over-recruit in 8 weeks, we will use the data from all participants.

Inclusion criteria

Standards of care and access to clinical trials vary dramatically across the globe.²⁸ Therefore, the relative effect of trial features in the UK on the decision to take part in trials may be very different to another country. To maximise the utility of this study, we are only opening the survey to those receiving care in the UK. Currently, we only have the resources to conduct the survey online and in English.

• Person with CF.

• Age 16+.

• Receives their clinical care within the UK.

• Able to read English.

• Willing to consent to participation.

Participants will be asked to self-confirm eligibility based on three screening questions; age, location of clinical care and whether they have CF. Those who do not meet the eligibility will be unable to proceed with the survey.

Withdrawal

Participants are free to stop the survey at any time without giving reasons. The opening participant information explains their right to do so, and a statement that this will not impact on their right to access clinical care or clinical trials. An email address for the team will be provided in case participants have questions. No identifiable data will be collected, so if patients choose to withdraw, no follow-up will be attempted and reasons for withdrawal cannot be sought.

Patient public involvement

Patient engagement is critical to ensure the success and relevance of this study. The key themes to explore, as well as the relevant attributes and levels were identified through a series of interviews, focus groups and a Delphi survey, thus more than 90 patients and 30 healthcare professionals working within the field of CF contributed to the study’s development. The draft survey was pretested and adjusted following feedback from six people with CF with a range of trial participation experience. We will disseminate the results in patient accessible forums to ensure the findings reach people with CF.

Study timelines

Survey development ran from November 2019 to June 2020. Survey administration and data collection will be conducted from October to December 2020. Analysis will be conducted from January to April 2021.

Analysis

General considerations

Descriptive statistics will be used to record respondent characteristics and questions not in the DCE. The cross-sectional DCE data will be explored using conditional multinomial logit model.²⁹ We will use mixed logit models, such as random-parameters logit (RPL) and latent class (LC) models,^{30 31} to analyse the choice-format data. Unobserved variation in preferences across the sample can bias estimates in conventional conditional-logit choice models. The RPL and the LC models avoid this bias by estimating a distribution of preferences around each model parameter that accounts for variations among individual preferences.^{30 32 33} In an attempt to explore the drivers of preference heterogeneity, RPL subgroup and LC analysis with the membership probability function modelled including respondents’ characteristics will be implemented to better explore preference variation across the sample by identifying heterogeneity.³⁴

Analyses will be performed in STATA 16.

Description of respondent characteristics

Descriptive analyses will be provided for each question not included in the DCE. For continuous and ordinal variables, number of available observations, mean, SD, median, minimum, IQR and maximum will be calculated. For categorical variables, number of available observations and frequency and percentage in each response category will be reported. Analyses will be performed on observed data only, and no imputation for missing values will be performed. The number of missing responses will be displayed for each question; denominators will not include missing values.

Preference analysis

Utility specification

The number of choice questions in the experimental design is expected to be sufficient to estimate main effects. Main effects include the preference weight for each attribute level independent of the other attributes and levels included in the study (table 1), using effects-coding for each categorical attribute level. A linear and continuous function will be tested and compared with a categorical representation for the stipend attribute, implementing the best to fit the data in the final model.

Determination of the appropriate model estimation approach

Each respondent will answer 12 questions from the experimental design, each of which offers the participant to select one of two trials, or to say that they would not participate in either trial. The DCE questions generate cross-sectional data that require analysis using advanced statistical techniques.³⁵ An RPL model will be used to analyse the DCE data collected and estimate the primary endpoints (ie, all attribute-level preference weights) assuming all random parameters are normally distributed and independent unless goodness-of-fit measures indicate that an RPL model assuming all random parameters are normally distributed and correlated better fits the data. An LC model will also be used to analyse the DCE data, comparing it with the final specification of the RPL model. Goodness of fit, as well as the optimal number of classes in the LC model, will be evaluated using the Bayesian information criterion^{36 37} and the Akaike information criterion.³⁸

Conditional relative attribute importance

To determine the relative importance of an attribute, conditional relative importance will be calculated as the difference between the attribute level with the highest preference weight and the level with the lowest preference weight. This difference represents the maximum change in conjoint utility achievable with any attribute, given the levels chosen for the attributes in the study. The conditional relative importance of an attribute also describes the relative importance of each attribute relative to all other attributes included in the study, conditional on the range of levels of the attribute.

Analysis of preference heterogeneity

Subgroup analysis will be used to determine whether average preferences vary among respondents in subgroups for selected attributes (eg, perceived disease severity and access to modulators through clinic). In addition to exploring preference heterogeneity using a set of predefined subgroups in the RPL analysis, the LC model can also be used to explore preference heterogeneity by including covariates in the membership probability function, based on the same subgroups of interest.^{31 34}

Incomplete data

Respondents may choose not to answer some questions in the DCE. All respondents who answer at least one DCE choice question will be included in the analysis and descriptive summary statistics. Respondents who do not answer any choice questions will be excluded from the analysis and descriptive summary statistics. Respondents who choose not to answer certain demographic or treatment history questions will be noted in the summary statistics, and their responses will be included in the DCE. If a question is missed, the survey will be programmed to give a reminder to participants to input a value.

Study validity

Discrete-choice experiments have been widely employed in recent decades to quantify preferences for new and existent treatments and devices in a variety of clinical applications and settings²² and is considered by the authors the best approach to address our research question among multiple stated preference approaches.²² The design was developed following good research practices as defined by International Society for Pharmacoeconomics and Outcomes Research guidelines and standard practice in preference studies.³⁹ The DCE has a number of strengths derived from the use of best practices.²³ The survey was built on extensive preliminary investigation, significant experience of the study team and close collaboration with invested stakeholders. It was pretested with a well-balanced and representative population. The trial-choice data will be analysed using RPL methods following good research practices³⁵ that avoid estimation bias from unobserved variation in preferences across the sample and within-sample correlation in the choice sequence for each respondent.

Internal validity

Measurement error/misclassification

Measurement error increases with task difficulty, which decreases the precision with which coefficients (ie, preference weights) can be estimated for any given sample size. Therefore, the number of attributes included in the final survey instrument has been limited to those that are necessary to answer the research questions.

The data collected are based on responses to hypothetical design or implementation features selected for a hypothetical clinical trial. These choices are intended to simulate possible enrolment decisions but do not have the same consequences of actual, real-world enrolment and may not have the same impact. Thus, differences can arise between stated and actual choices. We have attempted to limit potential hypothetical bias by constructing choice questions that mimic realistic design choice features as closely as possible and that map clearly into the current trials landscape.

Information bias

Information necessary to take the survey will be provided to all respondents. Participants may be unfamiliar with some of the concepts, therefore the explanations risk presenting information bias. The study team have collaborated with people with CF to refine the language and reduce biased presentation of potential attributes and levels.

Selection bias

Selection bias is a potential limitation of this study, and all voluntary online surveys. Research has shown that results from online stated-preference studies are, in general, not statistically different from those elicited through face-to-face interviews.^{40 41} We will use multiple social media forums to attempt to reach out to a maximally diverse cohort. Finally, the preferences of those who agree to participate in this study may be systematically different from preferences of those who do not agree to participate. The team will be unable to measure or control for this bias.

Residual confounding

Patient choice could potentially be confounded with unobserved variables, including prior experience or unobserved sociodemographic characteristics. In addition, because this study is limited to eliciting preferences for a subset of potential outcomes and features of trials, patients may make assumptions about outcomes and features that are not included when completing the DCE questions. We will try to mitigate the latter effect through survey pretesting; however, these remain potential study limitations.

External validity of study design

Although there are standards and good practices for stated-preference studies,²³ no definitive guidance exists to ensure a preference study is externally valid.

Ethics summary

The principal investigator (JCD) has obtained approval from the Joint Research Compliance Office at Imperial College London. The questionnaire opens with an embedded participant information page. Consent to enter the study will be sought from participants following this full explanation through a question asking patients to confirm their willingness to participate. The right of the participant to refuse to participate without giving reasons will be respected and stated explicitly in the consent information. Participants are free to withdraw at any time and this will be made clear in the opening statements. A participant will be considered to have withdrawn if they do not complete the questionnaire. If the respondent withdraws from the study no further evaluations will be performed, and no additional data will be collected. The study team may retain and continue to use any data collected before such withdrawal of consent. No identifying information will be collected. Therefore, confidentiality will be maintained throughout this study.

Dissemination policy

On study completion, the data will be analysed and tabulated and a final study report prepared. The report will form the basis of peer-reviewed publications. The findings will be presented at international conferences and patient accessible forums. A summary sheet will be generated for participants and circulated via the same channels used to advertise the survey.