Pilot and feasibility studies for pragmatic trials have unique considerations and areas of uncertainty

Abstract

Background and objective:

Feasibility studies are increasingly being used to support the development of, and investigate uncertainties around, future large-scale trials. The future trial can be designed with either a pragmatic or explanatory mindset. Whereas pragmatic trials aim to inform the choice between different care options and thus, are designed to resemble conditions outside of a clinical trial environment, explanatory trials examine the benefit of a treatment under more controlled conditions. There is existing guidance for designing feasibility studies, but none that explicitly considers the goals of pragmatic designs. We aimed to identify unique areas of uncertainty that are relevant to planning a pragmatic trial.

Results:

We identified ten relevant domains, partly based on the pragmatic-explanatory continuum indicator summary-2 (PRECIS-2) framework, and describe potential questions of uncertainty within each: intervention development, research ethics, participant identification and eligibility, recruitment of individuals, setting, organization, flexibility of delivery, flexibility of adherence, follow-up, and importance of primary outcome to patients and decision-makers. We present examples to illustrate how uncertainty in these domains might be addressed within a feasibility study.

Conclusion:

Researchers planning a feasibility study in advance of a pragmatic trial should consider feasibility objectives specifically relevant to areas of uncertainty for pragmatic trials.

1. Pilot and feasibility studies for pragmatic trials

Pilot and feasibility studies are often an essential step prior to embarking on a full-scale randomised controlled trial. Using the definitions set out in a recent framework [1], feasibility studies are studies that ask whether something can be done, whether we should proceed with it — and if so, how. Pilot studies are a subset of feasibility studies; they ask the same question but have a specific design feature (e.g., randomisation) which mirror something that is intended to happen in a future large trial. In other words, in a pilot study, a future study or part of it (e.g., an intervention arm only) is conducted on a smaller scale. Since feasibility studies are designed to support the development of a future study by investigating areas of uncertainty about that future study [2], they should include clearly formulated feasibility objectives together with pre-specified progression criteria to guide the decision of whether to progress to the next stage or not.

There has been a rising interest in pragmatic trials over the past two decades [3]. The term “pragmatic” was first used to describe approaches to trial design by Schwartz and Lellouch in 1967: they contrasted the explanatory approach, aimed at elucidating a mechanism of action, with the pragmatic approach, aimed at choosing between routine care options [4]. Roland and Torgerson distinguished between the former as measuring efficacy, i.e., the benefit of the treatment under ideal conditions, and the latter as measuring effectiveness, i.e., the benefit of the treatment in routine clinical practice [5]. Treweek and Zwarenstein argue that more trials should have a pragmatic attitude to trial design, urging trialists to think about design choices that maximise applicability as much as they think about internal validity [6].

There is existing guidance for designing pilot and feasibility studies [7, 8, 9, 10], but none that explicitly considers the goals of pragmatic trials. Because pragmatic trials emphasize external validity, a pilot pragmatic trial may need to include considerations of both internal and external validity. For example, an emphasis on piloting procedures to achieve high adherence to the protocol may not be useful to inform the design of a trial that aims to test effectiveness of an intervention in conditions of potentially imperfect adherence. In this paper, we consider potential areas of uncertainty that might be examined as part of a feasibility study in advance of a pragmatic trial. While some areas of uncertainty may be common across explanatory and pragmatic trials, (e.g., the need to estimate parameters required for sample size calculation), here we focus on considerations specific to pragmatic trials. First, we describe the framework that was used to identify potential areas of uncertainty relevant to pragmatic trials. Next, we describe these areas of uncertainty, identify potential feasibility objectives within each, and present examples of feasibility studies for pragmatic trials (published or unpublished) addressing such areas of uncertainty.

2. Identification of potential domains of uncertainty

We used the well-known pragmatic-explanatory continuum indicator summary (PRECIS-2) as a convenient initial framework to identify potential areas of uncertainty [11]. PRECIS-2 has been developed to assess the pragmatic or explanatory nature of trials, acknowledging that a trial may exist on a continuum between explanatory and pragmatic. It identifies nine domains in which a trialist can make explicit design choices according to its pragmatic or explanatory intention: eligibility, recruitment, setting, organization, flexibility in delivery, flexibility in adherence, follow-up, primary outcome, and primary analysis. Based on discussions amongst the study team and a literature review, we selected eight of the nine domains as potentially relevant. We omitted the primary analysis domain of PRECIS-2 since this domain is concerned with the extent to which all data are included in the analysis of the primary outcome which is a decision to be made rather than something that needs “testing” in a feasibility study. Moreover, analysis in feasibility studies tends to be based on descriptive statistics rather than formal statistical analyses of effectiveness. We also identified two additional domains relevant to feasibility studies for pragmatic trials: intervention development and research ethics. Pragmatic trials may be designed to inform a decision about whether to adopt a particular intervention in practice; thus, the feasibility study may be concerned not only with intervention implementation but also with refining components of the intervention and testing its acceptability in practice. Research ethics was added because pragmatic trials may raise unique ethical issues [12].

3. Domains of uncertainty for pragmatic trials

Table 1 describes the identified domains and explains how a highly pragmatic approach is distinct from a highly explanatory approach within that domain. Table 2 identifies potential feasibility questions or areas of uncertainty within each of the domains, as well as relevant examples for illustration. Appendix 1 provides summary boxes for each example showing the specific feasibility objectives, the study designs used to address the objectives, and the key findings.

Table 1.

Main features of explanatory versus pragmatic trials within ten domains of potential areas of uncertainty for pilot or feasibility studies

Domain	Highly pragmatic approach	Highly explanatory approach
Intervention development	Develop an intervention that, if shown to be effective, would be ready and acceptable for implementation in usual care	Develop an intervention that exerts its effects through a postulated causal pathway with less consideration to its complexity and acceptability in clinical practice
Research Ethics	Adopt waived or altered forms of consent to minimize additional burden over usual care procedures	Adopt traditional full informed consent procedures
Eligibility	Include participants in the trial that are similar to those who would receive the intervention if it were part of usual care	Include a subsample of the target population more likely to show a beneficial effect
Recruitment	Recruit participants with no more effort than would be used in usual care to engage with patients	Recruit participants using more intensive recruitment strategies set up for research purposes
Setting	Include a range of centers and settings similar to where the results are intended to apply	Perform the trial in a setting with conditions intended to maximize the potential of demonstrating efficacy
Organization	Use no more resources, provider expertise, or organizational structure than those available in usual practice	Employ specialized resources, such as trained professionals to deliver the intervention
Flexibility of delivery	Deliver the intervention with the same flexibility that is anticipated in usual care, often leaving the details of how to implement the intervention up to the providers	Ensure providers comply with a highly standardized protocol for delivery of the intervention
Flexibility of adherence	Allow participants to engage with the intervention with the same variability that is anticipated in usual care, monitoring and encouraging adherence no more than would take place in usual care	Put measures in place to ensure participants adhere to the intervention as much as possible
Follow-up	Data collection and follow-up guided by usual care practices	Follow participants intensively, through more frequent and longer visits
Primary outcome	Select a primary outcome that is directly relevant to participants	Select a primary outcome on which the intervention is expected to have a direct effect

Table 2.

Example of areas of uncertainty for pragmatic trials that might be addressed in pilot or feasibility studies

Design element	Feasibility questions for a pragmatic approach	Examples
Intervention development	Would the potential intervention, if shown to be effective, be acceptable to stakeholders and used in clinical practice?	ADNAT (Box 1)* Project ACE (Box 2)
Research ethics	Is the ethical approach acceptable to stakeholders and is the research ethics approval process feasible?	FLUID (Box 3)
Participant identification and eligibility	Does the proposed method of identifying participants correctly identify eligible participants?	STOP CRC (Box 4)
Recruitment of individuals	Can we successfully recruit participants that resemble the population that would be likely to receive the intervention if rolled out beyond a trial?	HOCKEY FIT (Box 5) DIAMOND (Box 6) oTTer (Box 7)
Setting (recruitment of sites)	Can we successfully recruit a variety of centres that resemble settings where the intervention would be used if implemented outside the trial? Is it feasible to recruit participants in such settings?	SHIFT (Box 8)
Organization	What feasibility challenges arise from implementing the trial using no more resources than those readily available?	QUEST (Box 9) Dodds (Box 10)
Flexibility of delivery	Are staff willing and able to deliver the intervention without additional training or support? Does being part of the trial result in staff delivering the intervention differently than the way they would deliver it as part of usual care?	STOP CRC (Box 4)
Flexibility of adherence	Is some minimum level of adherence possible such that the intervention can plausibly achieve a difference that would affect decision making?	FLUID (Box 3) TIME (Box 11)
Follow-up	Is it possible to obtain data for outcome assessment, without participant follow-up? Is it possible to collect data without imposing additional burden on participants?	FEMuR (Box 12) PREDOVE (Box 13)
Primary outcome	What is an appropriate outcome(s) that would be important to patients and decision-makers?	PROBE (Box 14)

^*Boxes are detailed in the appendix.

3.1. Domain 1: Intervention development

In the later stages of developing a complex intervention, it becomes important to establish whether it is feasible to implement it into routine clinical practice, whether it would be acceptable to stakeholders (e.g., patients and clinicians) and whether it is ready for uptake outside of the research setting after the main trial is completed [13]. The feasibility stage would be useful for identifying barriers to adoption, for example stakeholder-specific, resource, organizational, or cultural barriers. It would also be useful for identifying facilitators to effective adoption, and distinguishing elements that are required and the elements that are optional or may be administered flexibly (see domains 7 and 8) [14]. Identification of barriers and facilitators at an early stage allows for changes to be made before the main trial goes ahead. A variety of study designs may be used to assess barriers and facilitators and should ideally be guided by framework-based approaches such as the theoretical domains framework [15].

The ADNAT study [16] (Appendix 1: Box 1) and ACE [17] (Appendix 1: Box 2) are examples of mixed-methods feasibility studies that assessed acceptability of the intervention to stakeholders. Investigators conducted surveys, focus groups and interviews to assess feasibility outcomes such as resources needed to set up and sustain implementation, training needs, perceived value, acceptability, and potential barriers to adoption.

3.2. Domain 2: Research ethics

For many pragmatic trials, research ethics considerations are no different from those in explanatory trials. However, some pragmatic trials may involve waived or altered forms of consent [18] which are thought to facilitate recruitment. If the ethical approach, including notification or informed consent, is novel, or if there is otherwise uncertainty about whether patients and ethics committees would find the planned approach acceptable, a feasibility study can help refine the ethics-relevant procedures for the planned trial. This can take the form of focus groups or interview studies and/or feedback from research ethics committees in an initial pilot trial. Challenges during the review process of the large trial could delay trial implementation and lead to unwanted heterogeneity in procedures across participating sites. The pilot trial offers an opportunity to identify the likely range of concerns that might be raised by research ethics committees across jurisdictions to be included in the large trial and communicate with site investigators and research ethics committees about potential solutions. Identifying potential challenges in the research ethics review process is particularly important for common pragmatic trial designs such as stepped wedge and cluster cross-over trials in which the timing of the intervention delivery is varied according to a fixed schedule [19, 20].

The FLUID trial [21] (Appendix 1: Box 3) is a pilot cluster crossover trial in hospitalized patients comparing two commonly used resuscitation fluids. The intervention was conceptualized as a hospital policy to predominantly stock only one type of fluid for a period, and the trial was designed with a waiver of patient informed consent. The investigators recognized that different research ethics boards may have variable interpretation of the justification for waiver of consent which can result in delays to ethics approval and impede adherence to the scheduled timing of crossing over. The pilot trial aimed to measure the time to research ethics approval, with a successful time defined as taking no longer than 90 days from submission to approval.

3.3. Domain 3: Participant identification and eligibility

Pragmatic trials deliberately choose less restrictive eligibility criteria so that participants are similar to those who would receive the intervention if it were implemented outside a trial. While the eligibility criteria for a trial depend on the research question and objectives, a feasibility study might test whether the procedures or processes for implementing eligibility screening are adequate to select participants who resemble the target population. Investigators could compare their sample to patients in the target population to determine whether to loosen or tighten certain criteria. Overly restrictive eligibility criteria might also be detected during the pilot stage if recruitment is more challenging than anticipated (see domain 4).

In the STOP CRC study [22] (Appendix 1: Box 4), the investigators conducted a pilot study to determine whether they could use real-time electronic health record data to correctly identify eligible patients. They measured numbers of patients eligible and recruited and confirmed that their electronic health record embedded approach was able to identify eligible patients.

3.4. Domain 4: Recruitment of individuals

The most pragmatic approach to recruitment would be to simply include all eligible participants who present in settings where the intervention might eventually be used if shown to be successful. However, ethical design of randomized controlled trials usually requires participant consent, and outcomes may need to be collected from participants, which means that some form of recruitment is needed. Before embarking on a large-scale trial, investigators might want to ensure that their recruitment processes are adequate to ensure a sufficient number of participants resembling the target population can be recruited. It might be important to assess whether specific subgroups, such as vulnerable populations and populations traditionally excluded from clinical trials, can be successfully recruited, especially if the future planned trial aims to examine treatment effect heterogeneity across defined subgroups. The Trial Forge Include Ethnicity framework [23] provides a set of questions and accompanying worksheets to help trialists think more carefully about their target populations and how elements of their intervention and recruitment strategies can be designed to be more inclusive. Reflecting on these questions during the feasibility stage can help trialists implement changes to alleviate potential barriers to trial participation before embarking on the large trial. A theory-guided approach to designing pre-trial surveys for trialists seeking to optimize their trial recruitment strategies is also in development [24].

The HOCKEY FIT study [25] (Appendix 1: Box 5) was a pilot trial that aimed to recruit using methods that were easy to implement and with no added costs. To assess the feasibility of recruitment of hard to reach individuals, they examined the length of time needed to recruit, the number of individuals who expressed interest but were ineligible, and the number who were randomized but withdrew before follow-up sessions. The DIAMOND study [26] (Appendix 1: Box 6) and the ongoing oTTer project [27] (Appendix 1: Box 7) were pilot trials which aimed to identify any potential recruitment difficulties and determine how representative the trial participants were compared to the wider population receiving the intervention. The DIAMOND study identified several factors that negatively affected recruitment and concluded that alternative settings need to be considered for the future trial.

3.5. Domain 5: Setting (recruitment of sites)

A pragmatic trial can promote applicability by demonstrating effectiveness of an intervention across a range of settings, professionals who might be involved, and populations served by the sites. Thus, investigators might want to test and refine their processes for including a variety of relevant sites before embarking on a large trial. This might involve testing whether all types of sites (e.g., academic and community hospitals) can be recruited to participate or testing whether an adequate number of individuals can be identified at different types of sites. The feasibility study might determine willingness of sites to participate, level of commitment from staff, and possible challenges that might affect recruitment. Given their limited sample size, it may be difficult to demonstrate ability to recruit a “representative” sample of sites in a pilot trial, but a feasibility study may include a survey of available sites or providers to assess interest. If the future trial design is a cluster randomized trial, it may be useful to demonstrate within the feasibility study that sites would be willing to be randomized to a control arm which does not receive the novel intervention or, in the case of a wait-list control design, receives it at the end. Such results could inform the decision of whether to adopt a stepped wedge cluster randomized design (in which all sites gradually receive the intervention during the trial itself) or parallel arm design (in which potentially only half of the sites receive the intervention). As stepped wedge designs are vulnerable to increased risks of bias compared to parallel arm designs, a good rationale is required before adopting a stepped wedge [28]. A survey conducted as part of a feasibility study might provide convincing evidence that recruitment difficulties are likely unless all sites can be offered the intervention during the trial.

The SHIFT cluster randomized trial [29] (Appendix 1: Box 8) included embedded feasibility work in the form of semi-structured interviews during the main trial to investigate the feasibility of recruiting and retaining representative general practices in the trial. The feasibility work identified several factors that were important in recruiting and retaining practices.

3.6. Domain 6: Organization

A more pragmatic trial design would aim to use no more resources, provider expertise, or organizational structure than those readily available in usual practice. Explanatory trials often take place in research centers, whereas pragmatic trials often involve a broader range of centers, some of which may lack research expertise. Thus, it is important to investigate organizational challenges in advance of a pragmatic trial. A feasibility study may also be useful to determine what additional resources or training is needed for staff to participate in a trial (e.g., research ethics training, methods of handling and reporting adverse events).

The QUEST study [30] (Appendix 1: Box 9) included a pilot trial in which investigators recruited from three different sites to test feasibility and ensure they had experience with trying to set up the trial in sites with different characteristics. The Dodds study [31] (Appendix 1: Box 10) was a pilot study that encountered a major issue necessitating a change in the organization required to deliver the large trial.

3.7. Domain 7: Flexibility of intervention delivery

Pragmatic trials might allow the delivery to vary according to the needs of the different sites by not preventing or restricting access to other available treatments, and not closely monitoring adherence to the protocol. However, it is important to define the intervention clearly, with attention to the elements that are required and the elements that are optional or may be administered flexibly. If staff need training, then the training would be considered part of the intervention in a pragmatic trial. A feasibility study may assess whether the core part of the intervention can be delivered as intended and determine the degree of flexibility required to allow delivery without additional support, or to avoid disruption to usual care. The feasibility study may also aim to determine the extent to which being part of a trial may result in staff delivering the intervention differently than the way they would deliver it as part of usual care: if delivery in the trial deviates from how it would be done in usual care, the large trial may need to put additional procedures in place to ensure less research intrusion into care delivery.

In the STOP CRC pilot study [32] (Appendix 1: Box 4), existing clinic staff could choose which intervention components they would deliver. The feasibility study estimated the extent to which staff delivered different components of the intervention sufficiently well.

3.8. Domain 8: Flexibility of adherence

Fidelity violations are not necessarily a threat to the validity of a pragmatic trial in the same way that they are in an explanatory trial; we do not want to enforce adherence more than would be the case in usual care, but there still needs to be a certain level of adherence for the intervention to be evaluated and plausibly achieve a difference that would affect decision-making. Certain core components of the intervention may require higher degrees of implementation fidelity, as discussed in domains 1 and 7. A feasibility study may aim to determine whether this minimum level of adherence is possible in the large trial. The required minimum level of adherence needs to be set according to the specific question and context. The estimated level of adherence from the feasibility study can also usefully inform sample size calculation for the future trial as lower levels of adherence may lead to an attenuation of the detectable difference.

In the FLUID trial [21] (Appendix 1: Box 3) physicians could deviate from the allocated intervention for individual patients in their care. Investigators measured hospital adherence to the allocated study fluid, with a target of > 80% for the trial to be worthwhile. The TIME trial [33] (Appendix 1: Box 11) is an example of a pragmatic trial in which recruitment was terminated because of insufficient separation in dialysis session duration (the intervention) between the trial arms. This was because providers could deviate from the protocol. A pilot trial could have been useful to identify whether a minimum level of adherence was possible, before launching into the main trial.

3.9. Domain 9: Follow-up

In explanatory trials, participants are often followed up intensively, through more frequent and longer visits. In principle, the most pragmatic approach to follow-up would be to not obtain follow-up data directly from participants but to use, for example, electronic medical records instead. However, a defining feature of pragmatic trials is that the results should be useful to decision-makers. Outcome selection (choosing the right outcome) is therefore even more important than outcome source. Where an appropriate outcome to inform decision-making is not available routinely, pragmatic trials need to collect data directly from participants but in a way that does not interfere too much with routine clinical practice. A feasibility study can test the planned procedures for either routinely collected data or participant data collection in a non-intrusive manner. Mc-Cord et al. discuss the potential barriers and challenges that routinely collected data present for randomized trials [34]. During a feasibility study, a check of routinely collected data could be made for completeness and validity. Alternatively, participants could be asked to complete questionnaires and completeness of responses and length of time needed to complete the questionnaires can be assessed. Feasibility of less intrusive data collection methods such as text messaging can also be tested.

The FEMuR study [35] (Appendix 1: Box 12) aimed to test methods for obtaining routinely collected data on health service use, evaluate data quality, and compare the routinely collected data with patient-completed data. The PREDOVE study [36] (Appendix 1: Box 13) assessed the feasibility and acceptability of administering various repeat questionnaires to the sample of women; questionnaire completion rates were found to be low and a decision was made to obtain outcomes from routinely collected sources instead.

3.10. Domain 10: Primary outcome

In a pragmatic trial, the chosen outcome should directly inform decision-making and be measured as it would be measured in usual care. More than one candidate primary outcome may be of interest since clinical decisions are often made based on a variety of considerations including effectiveness, side-effects, and costs. Different stakeholders may have different perspectives about the outcome on which to base their decision. Furthermore, clinical and patient-reported outcomes may have different sample size requirements. Feasibility studies can be used to engage with stakeholders to inform the final choice of primary outcome(s) and to generate estimates that are useful to inform the final sample size. Stakeholder opinion on the choice of primary outcome will need to be balanced with the requirement that the outcome can be both accurately and feasibly collected (see domain 9).

The aim of the PROBE Project [37] (Appendix 1: Box 14) was to develop a questionnaire with outcomes that were important to patients living with haemophilia, and then perform a feasibility study of implementing the questionnaire. Patient representatives provided extensive input in the identification and measurement of key patient-reported outcomes.

4. Discussion

We identified ten domains relevant to pragmatic trials that researchers can consider in designing feasibility studies. As a first step, we recommend that trialists think about whether their future trial aims to answer a pragmatic or explanatory question. Unless a trial with pragmatic intention is deliberately designed to support applicability to usual care, it may not succeed in its goal of informing a clinical decision. Completing a PRECIS-2 wheel or table after a pilot trial can be useful in identifying domains in which pragmatic elements can be further improved in the future trial. We note that the example studies discussed here were selected specifically because they had feasibility objectives reflecting pragmatic goals in a future planned study. These studies may not necessarily be pragmatic in all possible respects; however, few trials are pragmatic in all domains. Even when a trial is not explicitly labelled as pragmatic, it may well have pragmatic elements and so the issues raised here for feasibility studies should be a consideration for many or most trialists.

We added the domain of research ethics as a potential area of uncertainty, recognizing that the ethics of pragmatic trials is an area in need of further development [38]. There are currently no explicit guidelines for the ethical conduct of pragmatic trials and different research ethics committees may vary in their assessments about the appropriateness of the planned ethical approach. Informed consent has implications for both internal and external validity of the trial and may be seen to be at odds with the pragmatic ideal. Although some believe that low risk pragmatic trials should be permitted with waivers of consent [39], others believe that alterations such as integrated consent are more appropriate [40]. Empirical reviews indicate that the majority of pragmatic trials do obtain participant informed consent although there is limited evidence about the use of alterations of consent [41, 42]. Differential identification and recruitment of participants is a particular challenge in cluster randomized trials in which participants may need to be recruited after randomization, and blinding is difficult or impossible [43]. Rather than adopt a waiver of informed consent, investigators may choose to implement alterations to streamline recruitment and engage with patients, ethicists, research ethics committees and other relevant stakeholders to refine procedures during the feasibility stages of their research.

Some of our proposed domains have similarities with the criteria proposed by the Readiness Assessment for Pragmatic Trials (RAPT) model which can be used to assess when a health systems intervention is ready for implementation in an embedded pragmatic trial [44]. The RAPT model identifies nine domains in which a non-pharmacological intervention of interest can be scored from low to high readiness. Interventions with high degrees of readiness are those that have previously been demonstrated to be efficacious and have well-documented protocols, have risks known to be minimal, can be implemented within existing health system resources, rely only on outcomes that are already routinely captured, are cost-effective, are acceptable to providers and staff, align with stakeholder priorities, and are likely to inform clinical care or policy. While some of the criteria in the RAPT model are associated with more pragmatic trials, they are not required features of pragmatic trials. Our focus here is on domains of potential uncertainty relevant to any planned trial with pragmatic intention or with one or more pragmatic elements.

This conceptual paper has provided domains of uncertainty relevant to pragmatic trials and examples of questions that researchers might ask. By considering specific areas of uncertainty due to the pragmatic elements in a future trial, as we have done here, researchers should be able to design feasibility studies that better inform their future trial. Finally, we believe that pre-registration of a pilot trial with clearly specified primary and secondary feasibility objectives and progression criteria can lower the risk of studies inappropriately moving on to a larger trial.

What is new?

Key findings

• Objectives of a feasibility study in preparation for a trial with pragmatic intention ought to differ from those of a study in preparation for a trial with explanatory intention.

What this adds to what is known?

• Examining the pragmatic-explanatory continuum indicator summary (PRECIS-2) we identified eight domains particularly relevant to feasibility studies for trials with pragmatic intention and added two new domains.

• Areas of uncertainty specific to pragmatic goals can be defined within each domain and should be considered when formulating feasibility objectives for a feasibility study.

What is the implication, what should change now?

• When undertaking feasibility studies, trialists should think about whether their proposed future trial has a pragmatic intention or has pragmatic elements.

• Feasibility studies for a trial with pragmatic intention should be deliberately designed to address pragmatic feasibility objectives.

APPENDIX 1. EXAMPLE STUDIES

Box 1. ADNAT Study [45].

Aim:

To evaluate the feasibility of integrating the Adolescent Diabetes Needs Assessment Tool (ADNAT) into paediatric diabetes care with respect to (i) Resources and processes that influence the clinical implementation of ADNAT; and (ii) Methodological issues in preparation for a large scale study.

Design:

Cohort mixed methods feasibility study.

Intervention:

ADNAT App – consists of reflective questioning to support adolescent decision-making in relation to type 1 diabetes self-care.

Pragmatic feasibility objective:

To determine the feasibility of integrating the ADNAT App into UK paediatric diabetes care, and determine how paediatric diabetes health care staff perceive use of ADNAT within the context of their clinical experiences.

Method to address this objective:

Survey and focus group interviews with health professional members of diabetes teams, to assess staff views on, for example, resources needed to set up and sustain use of ADNAT, training needs, perceived value of ADNAT, system and information quality, accessibility, social norms, and intention to use ADNAT in the future.

Findings:

The survey and qualitative findings indicated that ADNAT was acceptable to staff and the majority intended to use ADNAT in the future, but that lead clinician support would be essential, as well as a team approach and tailoring to fit each team. Authors also concluded that a cluster randomized design with sequential but random rollout of ADNAT over multiple time periods, that would fit the needs of a ‘hard to reach’ adolescent population, is necessary.

Box 2. Project ACE [46].

Aim:

To evaluate the impact of tailored self-management resources for 8–16-year-olds with type 1 diabetes on hemoglobin A1c and quality of life.

Design:

Randomized controlled trial, with embedded qualitative work with stakeholders that could be considered as feasibility work.

Intervention:

Group-based 75-minute self-management resource sessions.

Pragmatic feasibility objective:

To determine whether the study processes and intervention can be integrated into routine clinic workflows.

Method to address this objective:

Semi-structured interviews with clinicians, clinic leadership, clinic schedulers and medical assistants, and patients’ families, as well as stakeholder advisory boards consisting of parents and youth.

Findings:

Interviews raised potential challenges and solutions regarding study procedures and intervention implementation, and solutions were generated via the stakeholder advisory boards regarding challenges with recruitment, retention, and integration into the clinic.

Box 3. FLUID Trial [47].

Aim:

To examine the feasibility of conducting a larger pragmatic comparative effectiveness cluster crossover trial in hospitalised patients.

Design:

Cluster crossover pilot randomized controlled trial.

Intervention:

0.9% saline (control fluid) and Ringer’s lactate (treatment fluid).

Pragmatic feasibility objectives:

1. To determine whether having a waiver of consent results in delays to research ethics board approval at different sites.

2. To determine whether a minimum adherence of 80% to the study protocol is achievable.

Method to address these objectives:

1. Calculate time to research ethics board approval for each site.

2. Provide a small amount of the non-allocated study fluid available for use throughout the hospital if the treating physician chooses to opt out of using the study fluid for a given patient, and calculate adherence as the total use of the allocated study fluid divided by the total combined use of normal saline and Ringer’s lactate.

Findings:

Study in progress.

Box 4. STOP CRC Study [48].

Aim:

To assess the feasibility and potential effectiveness of a program to raise rates of colorectal-cancer screening.

Design:

Non-randomized pilot study.

Intervention:

Automated data-driven, Electronic Health Record (EHR)-embedded program for mailing Faecal Immunochemical Test (FIT) kits to patients due for colorectal-cancer screening. (Steps of the intervention: 1. Introductory letters mailed; 2. Kits mailed; 3. Reminder postcards mailed; 4. Completed call).

Pragmatic feasibility objectives:

1. To determine whether the screening registry function could be embedded directly into the EHR and use real-time data to identify patients eligible for each intervention step.

2. To determine whether clinic staff would successfully deliver each component of the intervention (considering the program was designed so that it could be incorporated into clinical practice and existing clinic staff were allowed to choose intervention components).

Method to address these objectives:

1. Numbers recruited and eligible.

2. Number of anticipated patients that the screening intervention was delivered to at each step of the intervention.

Findings:

Able to build an EHR-embedded program that used real-time data to identify eligible patients at each step in the intervention, but because the inclusion and exclusion criteria relied on EHR data they could not verify whether the intervention was delivered to patients who were ineligible, and further pilot work was warranted.Intervention was delivered to all anticipated patients at each step.

Box 5. HOCKEY FIT Study [49].

Aim:

To examine the feasibility of recruiting and retaining men in Hockey FIT and the acceptability of the research procedures.

Design:

Pilot randomized controlled trial.

Intervention:

Hockey FIT program, delivered over 12 weekly, 90-minute sessions by Hockey FIT coaches with 1) classroom-based teaching of behavior change techniques and simple information sharing on physical activity and healthy eating and 2) exercise sessions where men performed aerobic, strength, and flexibility exercises. Used a pragmatic approach for the program venue at each site, using a combination of the hockey teams arena and an affiliated health club facility.

Pragmatic feasibility objective:

To examine the feasibility of recruiting and retaining men in Hockey FIT and the acceptability of the research procedure.

Method to address this objective:

Length of time needed to recruit the desired sample size, the number of individuals who expressed interest but were not eligible, and of those who enrolled and were randomized, the number who withdrew before the follow-up measurement sessions.

Findings:

The most successful methods of recruitment included e-mail, social media and word of mouth. Recruitment of 80 eligible participants was completed in a short time frame using methods that were easy to implement and with no added costs.

Box 6. DIAMOND Study [50].

Aim:

To learn more about the challenges likely to be encountered in a fully powered phase 3 trial, specifically looking at recruitment, retention, testing the online randomization and data collection instruments, collecting data to inform the sample size calculation for the main trial, and understanding the reasons for any difficulties in recruitment and retention and with data collection or use of the intervention.

Design:

Pilot randomized controlled trial.

Intervention:

Online alcohol treatment programme for hazardous and harmful drinkers called Healthy Living for People who use Alcohol (HeLP-Alcohol).

Pragmatic feasibility objective:

To understand the reasons for any difficulties in recruitment and retention to the trial; recruitment was done by usual staff as per their usual practice.

Method to address this objective:

Recruitment as a percentage of eligible patients.

Findings:

Low numbers of participants recruited in the feasibility study suggested that community alcohol services were not suitable places to recruit due to the lack of equipoise of staff who would not randomize clients they believed would not manage the intervention.

Box 7. oTTer project [51].

Aim:

To establish the feasibility of conducting a definitive trial to evaluate the effectiveness of the Talking Together intervention, specifically looking at acceptability of the intervention outcome measures and identifying any challenges with the implementation and delivery of the Talking Together programme.

Design:

Pilot randomized controlled trial.

Intervention:

Talking Together – a 6-week home visiting programme to develop parents’ knowledge of the importance of a good language environment and help to improve parent-child interactions.

Pragmatic feasibility objective:

To identify how representative the trial participants are compared to the wider population receiving the intervention.

Method to address this objective:

Comparison of key demographic indicators.

Findings:

Study in progress.

Box 8. SHIFT Trial [52].

Aim:

To assess the effectiveness, feasibility, and acceptability of delivering antenatal sickle cell and thalassaemia screening in primary care.

Design:

Cluster randomized trial with an embedded feasibility study.

Intervention:

Antenatal sickle cell and thalassaemia screening offered to women, according to the randomisation group (offered in primary care, with partners offered testing at the same time; offered in primary care, with partners offered testing later and only if women are identified as carriers; offered in community-based secondary care, with partners offered testing later and only if women are identified as carriers).

Pragmatic feasibility objective:

To determine whether it is feasible to recruit and retain general practices (GPs) in clinical trials.

Method to address this objective:

Semi-structured interviews were completed with 20 GPs to explore their views on why their practice participated in the trial. Outcome measures were the number of practices recruited to, and completing, the trial.

Findings:

The findings indicated it is possible to recruit and retain representative primary care practices to a clinical trial.

Box 9. QUEST study [53].

Aim:

To assess the feasibility, sample size, and most appropriate outcomes for a large-scale trial to compare the clinical and cost-effectiveness of two models of mental health supported accommodation commonly used in England.

Design:

Pilot randomized controlled trial.

Intervention:

Mental health supported accommodation models (supported housing and floating outreach).

Pragmatic feasibility objective:

To test the feasibility of a randomized evaluation of two supported accommodation models.

Method to address this objective:

Recruit from three different sites to have experience of trying to implement the trial in sites with different characteristics and to identify any feasibility challenges.

Findings:

Staff interviews identified issues such as a lack of availability of independent tenancies and a perceived lack of staff resources. The investigators concluded that a large-scale trial was not feasible at this time.

Box 10. Dodds Study [54].

Aim:

To identify issues in a research trial with low-income, publically insured, high-risk adolescents who had long-term poorly controlled asthma.

Design:

Pilot randomized controlled trial.

Intervention:

Inhaled corticosteroid real-time medication monitoring (RTMM) devices and text messaging intervention. To preserve the pragmatic nature of the trial, RTMM devices were compatible with the medication most prescribed to the eligible population.

Pragmatic feasibility objective:

To identify potential issues in a research trial with low-income, publicly insured, minority adolescents using multiple technologies that, without proper execution, could reduce the effectiveness of the intervention and the accuracy of the method by which the intervention was measured.

Method to address this objective:

Monitoring to identify issues during the pilot trial.

Findings:

Eight months into the study, the medication for many of the patients changed and the RTMM devices were not compatible with monitoring this new medication. A solution might have been to provide the medication for the patient while they were in the study, but because this would be unfeasible in usual care the trial team did not want to compromise the pragmatic nature of the trial.

Box 11. TIME trial [55].

Aim:

To determine whether dialysis facility implementation of a minimum hemodialysis session duration for patients with end-stage renal disease has benefits on mortality, hospitalizations and health-related quality of life.

Design:

Cluster randomized trial.

Intervention:

Dialysis session of at least 4.25 hours (providers were allowed to deviate from the protocol).

Pragmatic feasibility objective:

None – but a pilot/feasibility study would have been useful to test whether some minimum level of adherence was possible.

Findings:

Recruitment terminated because of insufficient separation in dialysis session duration between the trial arms.

Box 12. FEMuR Study [56].

Aim:

Test methods for obtaining routinely captured health service activity data and compare with data collected using patient completed questionnaires.

Design:

Non-randomized feasibility study.

Intervention:

Enhanced rehabilitation intervention following hip fracture in the elderly.

Pragmatic feasibility objective:

Test methods for obtaining routinely collected data on health service use, evaluate the quality of the data acquired, and compare these data with data collected using the client service receipt inventory (CSRI) patient completed questionnaire over the same time period.

Method to address this objective:

Health service use data from the patient-completed CSRI were compared with those obtained from routinely collected data using the intraclass correlation coefficient (ICC) and descriptive statistics.

Findings:

It was possible to obtain the routinely collected data for comparison with that collected in the CSRI. Some areas showed high levels of comparability, but much of the routinely collected data was incomplete and there were lower levels of comparability for particular areas such as out-patient appointments and emergency department attendances.

Box 13. PREDOVE study [57].

Aim:

Assess the feasibility and acceptability of a general practice-based domestic violence intervention programme, and assess the feasibility of a randomized controlled trial design to evaluate outcomes of the intervention

Design:

Pilot cluster randomized trial.

Intervention:

Multi-faceted intervention including domestic violence education and training for participating health professionals, screening for abuse, an information campaign, provision of immediate support and follow-up, referral by the clinician or self-referral.

Pragmatic feasibility objective:

To assess the feasibility and acceptability of administering baseline questionnaires, administering repeat questionnaires over a 6-month period, and the acceptability of further, long term follow up.

Method to address this objective:

Numbers at follow up.

Findings:

Numbers providing outcome data were small and the authors therefore made the pragmatic decision that in the main trial they would obtain outcome data from routinely collected data at the practice level instead.

Box 14. PROBE project [58].

Aim:

To contribute patient reported outcomes relevant to haemophilia by direct patient involvement in the design, conduct, analysis and reporting of patient-centered outcome research in the field of haemophilia.

Design:

Non-randomized feasibility study.

Intervention:

N/A

Pragmatic feasibility objective:

To develop a standardized questionnaire to gather patient-reported outcomes and perform a feasibility study of implementing the questionnaire.

Method to address this objective:

Qualitative feedback from a pool of persons living with haemophilia on the questionnaire content and clarity.

Findings:

They identified metrics of importance to the haemophilia community, such as quality of life and family burden, and these metrics guided the initial version of the PROBE questionnaire. Patients were given the PROBE questionnaire in the feasibility study and they found that the response rate was inferior to that in other formal questionnaire development processes but noted this should be considered against the pragmatic design of their feasibility study. They concluded that it was feasible to engage diverse patient communities at all stages.