Acceptability, fidelity and trial experience of four intervention components to support medication adherence in women with breast cancer: A process evaluation protocol for a pilot fractional factorial trial

Sophie MC Green; Louise H Hall; Nikki Rousseau; David P French; Christopher D Graham; Michelle Collinson; Ellen Mason; Hollie Wilkes; Daniel Howdon; Robbie Foy; Rebecca Walwyn; Jane Clark; Catherine Parbutt; Erin Raine; Rachel Ellison; Jacqueline Buxton; Sally J L Moore; Galina Velikova; Amanda Farrin; Samuel G Smith

doi:10.3310/nihropenres.13337.2

. 2023 Apr 24;3:3. Originally published 2023 Jan 20. [Version 2] doi: 10.3310/nihropenres.13337.2

Acceptability, fidelity and trial experience of four intervention components to support medication adherence in women with breast cancer: A process evaluation protocol for a pilot fractional factorial trial

Sophie MC Green ^1,^a, Louise H Hall ¹, Nikki Rousseau ², David P French ³, Christopher D Graham ⁴, Michelle Collinson ², Ellen Mason ², Hollie Wilkes ², Daniel Howdon ¹, Robbie Foy ¹, Rebecca Walwyn ², Jane Clark ⁵, Catherine Parbutt ⁶, Erin Raine ¹, Rachel Ellison ², Jacqueline Buxton ⁷, Sally J L Moore ⁷, Galina Velikova ^8,⁹, Amanda Farrin ², Samuel G Smith ¹

¹Leeds Institute of Health Sciences, University of Leeds, Leeds, LS29NL, UK

²Leeds Institute of Clinical Trials Research, University of Leeds, Leeds, LS29NL, UK

³Manchester Centre for Health Psychology, University of Manchester, Manchester, M13 9PL, UK

⁴Department of Psychology, Queen's University Belfast, Belfast, BT9 5BN, Ireland

⁵Department of Clinical and Health Psychology, Leeds Teaching Hospitals NHS Trust, Leeds, LS9 7TF, UK

⁶Medicines Management and Pharmacy Services, Leeds Teaching Hospitals NHS Trust, Leeds, LS9 7TF, UK

⁷Independent Researcher, Leeds, UK

⁸Leeds Institute of Medical Research at St James’s, University of Leeds, Leeds, LS9 7TF, UK

⁹Leeds Cancer Centre, Leeds Teaching Hospitals NHS Trust, St James’s University Hospital, Leeds, LS9 7TF, UK

Email: s.m.c.green@leeds.ac.uk

No competing interests were disclosed.

Roles

Sophie MC Green: Conceptualization, Formal Analysis, Methodology, Project Administration, Resources, Writing – Original Draft Preparation, Writing – Review & Editing

Louise H Hall: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Nikki Rousseau: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

David P French: Conceptualization, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

Christopher D Graham: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Michelle Collinson: Formal Analysis, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Ellen Mason: Formal Analysis, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Hollie Wilkes: Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Daniel Howdon: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Robbie Foy: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Rebecca Walwyn: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Jane Clark: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Catherine Parbutt: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Erin Raine: Conceptualization, Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Rachel Ellison: Methodology, Project Administration, Writing – Original Draft Preparation, Writing – Review & Editing

Jacqueline Buxton: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Sally J L Moore: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Galina Velikova: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Amanda Farrin: Methodology, Writing – Original Draft Preparation, Writing – Review & Editing

Samuel G Smith: Conceptualization, Funding Acquisition, Methodology, Project Administration, Resources, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing

PMCID: PMC10593334 PMID: 37881449

Version Changes

Revised. Amendments from Version 1

We have revised the manuscript in line with the reviewer’s comments. The main changes to this new version include: - Additional detail in the introduction about existing interventions to support medication adherence in this population, and the limitations of these. - Additional detail in the introduction and discussion regarding the Multiphase Optimisation Strategy, where this fits in intervention development and evaluation, and what our future plans for this work are. - Inclusion of a figure of our conceptual model, detailing hypothesised mechanisms of actions for the intervention components. - Additional detail about the theoretical rationale for our intervention components, along with clearer signposting to our published intervention development paper. - Further detail about the predefined progression criteria we will use to decide whether to proceed to a full optimisation trial. - Description of the theoretical domains framework earlier on in the manuscript. - Additional paragraph in the data analysis section detailing how we will combine the findings from the process evaluation. - Clarification of the wording of the study aims and methods in the plain English summary, with clarification as to how these correspond with the aims in the main paper. - Additional description of Table 2 and 3 in the main text.

Abstract

Background

The Refining and Optimising a behavioural intervention to Support Endocrine Therapy Adherence (ROSETA) programme has developed four intervention components aiming to improve medication adherence in women with early-stage breast cancer. These are (a) text messages, (b) information leaflet, (c) Acceptance and Commitment Therapy-based guided self-help (ACT), (d) side-effect management website. Guided by the Multiphase Optimisation Strategy, our pilot trial will use a fractional factorial design to evaluate the feasibility of undertaking a larger optimisation trial. The pilot will include a process evaluation to maximise learning regarding the fidelity and acceptability of the intervention components before proceeding with a larger trial. The trial process evaluation has three aims: to assess the (1) fidelity and (2) acceptability of the intervention components; and (3) to understand participant’s trial experience, and barriers and facilitators to recruitment and retention.

Methods

The process evaluation will use multiple methods. Fidelity of the intervention components will be assessed using self-reported questionnaire data, trial data on intervention component adherence, and observations of the ACT sessions. Acceptability of the intervention components and trial experience will be explored using an acceptability questionnaire and interviews with patients and trial therapists. Trial experience will be assessed using a questionnaire and interviews with participants, while barriers and facilitators to recruitment and retention will be assessed using a questionnaire completed by research nurses and participant interviews. The pilot trial opened for recruitment on 20th May 2022 and was open at the time of submission.

Conclusions

This process evaluation will provide information regarding whether the intervention components can be delivered with fidelity within a national healthcare setting and are acceptable to participants. We will also better understand participant experience in a pilot trial with a fractional factorial design, and any barriers and facilitators to recruitment and retention.

Registration

ISRCTN registry ( ISRCTN10487576, 16/12/2021).

Keywords: process evaluation, intervention fidelity, acceptability, trial experience, fractional factorial

Plain English summary

Background: The majority of women with early-stage breast cancer are recommended adjuvant endocrine therapy (AET) to reduce the chances of their cancer coming back. Many women given this medication don’t take it every day or stop taking it earlier than they should. We have developed four different interventions to help women take AET. These are; text messages reminding women to take AET; an information leaflet explaining how AET works and its benefits and side-effects; a therapy programme to reduce distress, consisting of five support sessions and four module booklets; and a website with strategies to manage AET side-effects. We are now testing whether these interventions can be delivered within the NHS in different combinations, in a small trial.

Study methods:

We have three aims:

1. To find out if the interventions can be given and are received in the way they were supposed to (fidelity).
2. To find out if the support received as part of the trial was acceptable to women with breast cancer (acceptability).
3. To find out what women’s experience was of taking part in the trial overall (trial experience).

To do this we will:

1. Interview participants to ask them how acceptable they found the interventions, what they understood, whether they used the interventions, and how they found participating in the trial.
2. Interview therapists who delivered the therapy programme to see if they delivered it as they were supposed to, and how they found delivering the intervention.
3. Ask participants to complete questionnaires about how acceptable the interventions were, and whether they read and used them.
4. Ask the staff involved in finding participants for the trial about challenges and improvements.

We will use what we find to make improvements in a future trial where we will test whether the interventions help women to take AET.

Introduction

Breast cancer is the most common cancer in the UK ¹. Adjuvant endocrine therapy (AET) is prescribed to women with oestrogen receptor-positive (ER+) breast cancer for 5–10 years to reduce recurrence and mortality ^2,
3. However, up to three quarters of women do not adhere to AET, either taking medication inconsistently or stopping prematurely ^4–
6. Suboptimal adherence can lead to increased risk of recurrence and mortality, reduced health related quality of life, and reduced quality adjusted life years ^7–
9. Multiple factors affect non-adherence behaviours, and these are often described as intentional ( e.g., not believing the medication is necessary, and side-effects), and unintentional ( e.g., forgetting) ^10–
13.

There is a lack of evidence for effective interventions to support medication adherence to AET, with most interventions focusing solely on written educational components, and not targeting the range of barriers to adherence ^14–
16. Moreover, these interventions are typically evaluated using parallel group randomised controlled trials (RCTs). RCTs are able to evaluate whether an intervention package is more effective than a comparator, but they are unable to estimate the contributions of individual components, or the interactions between components. As such, intervention packages demonstrating a statistically significant effect in an RCT could contain ineffective or redundant components, reducing the efficiency of the overall intervention package ^17,
18.

The ROSETA programme

The Refining and Optimising a behavioural intervention to Support Endocrine Therapy Adherence (ROSETA) programme aims to develop and optimise an intervention package to support adherence to AET in women with early-stage breast cancer. The ROSETA programme is guided by the Multiphase Optimisation Strategy (MOST), which is a framework used to optimise multicomponent behavioural interventions ^17,
19,
20. MOST consists of three stages; (1) preparation, in which intervention components and a conceptual model detailing proposed mechanisms of action are developed, and any pilot testing is carried out, (2) optimisation, in which highly efficient experimental designs, such as factorial designs, are used to estimate the main effects and interaction effects of individual intervention components to build an optimal intervention package, and (3) evaluation, in which the optimised intervention package is compared with a comparator, typically using a parallel groups RCT ^17,
19,
20.

During the preparation phase of MOST, we combined the intervention mapping framework with MOST to develop four theory-based intervention components targeting distinct, unintentional and intentional barriers to AET adherence: (a) SMS messages to target forgetfulness; (b) information leaflet to target medication beliefs; (c) Acceptance and Commitment Therapy (ACT) guided self-help programme to increase psychological flexibility and reduce psychological distress; (d) side-effect self-management website to target AET side effects ²¹ ( Table 1). As part of this preparatory work, we developed a conceptual model for the intervention, detailing the mechanisms of action ( Figure 1). A full description of our approach to intervention development, along with detailed descriptions of the intervention components is available elsewhere ²¹.

Figure 1. — This figure has been reproduced with permission from Green *et al.*, (2023) ²¹.

Table 1. Summary of intervention components in the ROSETA pilot trial.

Component	Target	Description	Theoretical basis
SMS	Forgetfulness/habit formation	SMS messages will be sent over 4 months providing practical strategies to support regular medication taking each day. The messages will be sent daily for 2 weeks, twice weekly for 8 weeks and weekly for 6 weeks.	Habit theory
Information Leaflet	Medication beliefs	A written information leaflet containing five elements; an explanation of how AET works with diagrams to supplement, visual displays of the benefits of AET, accurate information about the side effects of AET, answers to common concerns about AET and quotes and pictures of breast cancer survivors.	Necessity Concerns Framework, Common Sense Model of Illness Representations
ACT	Psychological flexibility/psychological distress	A guided self-help intervention based on ACT principles involving four skills; mindfulness, unhooking, following values and living beyond labels. The modules consist of a participant booklet, home practice tasks and audio files. The modules are supported by five individual sessions with a psychologist; 1 x 15 minute opening session, 3 x 25 minute sessions following modules 1, 2 and 3, and 1 x 15 minute closing session following module 4.	ACT (based on relational frame theory)
Website	Side-effect self- management	A website containing strategies to self-manage common AET side effects including; arthralgia, fatigue, vulvovaginal symptoms, gastrointestinal symptoms, hot flushes, sleep difficulties. The website uses a rating system to summarise the strength of evidence for each strategy.	Informed by evidence of side-effect self- management strategies

Open in a new tab

Key: ROSETA= Refining and Optimising a behavioural intervention to Support Endocrine Therapy Adherence. SMS= Short Message Service. AET= adjuvant endocrine therapy. ACT= Acceptance and commitment therapy.

Experimental versions of the intervention components are currently being assessed for feasibility in an external, multi-site exploratory pilot trial, for which a detailed protocol has been published ²². This pilot trial is using a highly efficient 2 ^4-1 fractional factorial design. Participants are randomised to one of eight experimental conditions, in which they may receive usual care plus a combination of intervention components. Each intervention component has two levels; on or off. Across five UK sites, the pilot trial will recruit approximately 80 women over 18 with stage I-IIIa breast cancer who have been prescribed AET and have completed curative hospital-based treatment. Follow up will be at 2 and 4 months post randomisation. The primary objectives of the pilot trial are to establish eligibility, recruitment, retention and follow up rates, intervention component adherence and availability and feasibility of outcome and process data (ISRCTN: 10487576).

ROSETA pilot trial process evaluation

Process evaluations of complex interventions can maximise learning from trials, through investigating how an intervention was implemented, clarifying causal mechanisms and exploring contextual factors ²³. As highlighted in the UK Medical Research Council’s (MRC) guidance, process evaluations can have differing functions dependent on the trial stage ²³. In the context of a pilot trial, understanding the feasibility, fidelity and acceptability of an intervention is often the focus, falling within the implementation and context themes in MRC guidance ²³.

The fractional factorial design in the ROSETA pilot trial is uncommon in healthcare research ²⁴. When used with effect coding (-1, +1), factorial designs can efficiently estimate the main effects and interaction effects of multiple intervention components ²⁵. However, this design requires multiple experimental conditions, thereby reducing the number of participants randomised to each experimental condition (the condition corresponds to a unique combination of the levels of intervention components). While adequate recruitment and retention of participants is challenging in all healthcare trials ^26,
27, it is particularly important when using factorial designs, as empty cells may threaten the integrity of the trial ¹⁷, or increase complexity in analysis and interpretation. This trial design brings further uncertainties, such as whether multiple intervention components can be delivered with fidelity, participants are not unduly burdened with multiple intervention components, and whether the trial design is acceptable to, and understood by participants. Understanding participant trial experience and exploring barriers and facilitators to recruiting and retaining participants within a factorial design will provide important insights prior to proceeding with a larger optimisation trial. The Theoretical Domains Framework is a helpful tool to understand barriers and facilitators to recruitment and retention. The TDF synthesises 33 theories of behaviour change into factors that could influence behaviour, for example knowledge or available resources ²⁸.

Intervention fidelity refers to whether an intervention component was implemented as originally intended ²⁹. Fidelity assessments commonly focus solely on fidelity of delivery of an intervention, with less regard for the fidelity of training, receipt or engagement ^30,
31. The US National Institute of Health Behaviour Change Consortium (NIHBCC) framework suggests fidelity as a multi-faceted construct and recommends five domains of fidelity that should be assessed; design, training, delivery, receipt and enactment ³². Assessing fidelity at multiple levels can guide targeting of efforts to improve fidelity and hence improve the internal and external validity of a larger trial ^31,
33,
34.

Assessment of acceptability is also recommended in the pilot phase ^23,
35. However, acceptability is frequently assessed as one singular construct, and the definition varies considerably ³⁶. The Theoretical Framework of Acceptability (TFA) seeks to address this limitation by conceptualising acceptability as a multi-faceted construct consisting of seven components; affective attitude, burden, ethicality, intervention coherence, opportunity costs, perceived effectiveness and self-efficacy ³⁶. While preliminary work has confirmed the prospective acceptability of the ROSETA intervention components, the pilot trial offers the chance to evaluate acceptability of the intervention components as experienced by intervention recipients, with the potential to improve acceptability prior to a larger optimisation trial.

Protocol

Aims and objectives

This process evaluation has three aims: to assess (1) the fidelity of the intervention components; (2) the acceptability of the intervention components; (3) participant’s experience, and barriers and facilitators to recruitment and retention in a fractional factorial trial. These aims address the implementation and contextual factors themes outlined in the MRC framework ²³. The specific objectives for each aim are:

1.
Fidelity; Establish the fidelity of each intervention component across five domains of fidelity, guided by the NIHBCC fidelity framework ³²:
- a.
  Design. To establish to what extent the intervention components target the behaviour change techniques (BCTs) they are intended to, and to what extent the components are distinct from one another;
- b.
  Training. To evaluate the adequacy of therapist training for the delivery of the intervention;
- c.
  Delivery. To assess if each intervention was delivered as planned;
- d.
  Receipt. To evaluate if participants receive and understand the intervention components they were allocated to;
- e.
  Enactment. To understand the degree to which participants implement the intervention components in their daily life;

Identify barriers to the fidelity of training, delivery, receipt and enactment of the intervention components, including any influential contextual factors.

2.
Acceptability:
- a.
  Assess the acceptability of each intervention component
3.
Trial experience, recruitment and retention:
- a.
  Establish the barriers and facilitators to recruitment and retention of participants;
- b.
  Evaluate participant experience of trial participation.

Design

We will use quantitative and qualitative methods to address the three main aims, nested within the ROSETA pilot trial. The process evaluation will involve assessments with three participant groups; participants of the ROSETA pilot trial, therapists delivering the ACT intervention component, and research nurses (RNs) involved in recruitment.

Fidelity . Fidelity will be assessed in relation to the intended design and delivery, across five domains using a mix of methods. Table 2 describes the data collection methods for each objective and intervention component, along with the time point at which the data will be collected.

Table 2. Summary of fidelity assessments.

Objective	Intervention Component	Data	Data source	Method of completion	Time point
Design. To establish to what extent the intervention components target the behaviour change techniques they are intended to, and to what extent the components are distinct from one another.	All	BCTs coded as present in each intervention component, based on the BCTTv1	Independent coder	Independent Coder	Once the trial has commenced
Training. To evaluate the adequacy of therapist training for the delivery of the intervention component.	ACT	ACT-FM competency assessment	Questionnaire	ACT trainer	Following therapists first ACT session
	ACT	Number of booster training sessions required	Questionnaire	ACT Trainer	Throughout trial
	ACT	Therapist’s view on ACT training	Semi-structured interview	Therapist	End of trial
Delivery. To assess if each intervention component was delivered as planner.	SMS	Delivery of SMS messages	Online System	-	Throughout trial
	SMS	Number of participants opting out	Online System	-	Throughout trial
	Information Leaflet	Number of information leaflets sent	Site Recorded	Research Nurse	Throughout trial
	ACT	Procedural fidelity checklist	Questionnaire	Therapist	After every ACT session
	ACT	ACT-FM therapist stance score	Questionnaire	External rater	End of trial
	ACT	Qualitative data	Semi-structured interview	Therapist	End of trial
	Website	Number of log in details sent	Site Recorded	Research Nurse	Throughout trial
Receipt. To evaluate if participants receive and understand the intervention components they were allocated to.	All	Self-reported receipt of intervention component	Questionnaire	Participant	4 months
	All	Self-reported reading of intervention component	Questionnaire	Participant	4 months
	ACT	Session attendance, number of cancelled or missed	CRF	Therapist	After every ACT session
	ACT	Engagement with ACT module materials	CRF	Therapist	After every ACT session
	ACT	Engagement with ACT audio files and home practice tasks	Questionnaire	Participant	4 months
	ACT	Engagement with ACT module materials	Questionnaire	External rater (10% of ACT recordings)	End of trial
	ACT	Printing of ACT module booklet	Questionnaire	Participant	4 months
	Website	Website usage	Website Online System	-	2 and 4 months
	All	Understanding and receipt of the intervention component	Qualitative interview	Participant	End of trial
Enactment. To understand the degree to which participants implement the intervention components in their daily life.	All	Qualitative data	Qualitative interview	Participant	End of trial
Identify barriers to the fidelity of training, delivery, receipt and enactment of the intervention components, including any influential contextual factors.	All	Qualitative data	Semi-structured interview	Participants and therapists	End of trial

Open in a new tab

Key: BCT= Behaviour Change Technique. BCTTv1 = Behaviour Change Technique Taxonomy version 1. ACT = Acceptance and Commitment Therapy. ACT-FM: Acceptance and Commitment Therapy Fidelity Measure. SMS = Short Message Service. CRF = Case Report Form.

Fidelity of design. We will assess whether the intervention components show fidelity to the intended design, in terms of BCTs. BCTs are considered ‘active ingredients’ of behaviour change interventions (e.g. problem solving, action planning). They can be coded using the behaviour change taxonomy version 1 (BCTTv1), which contains 93 BCTs ^37,
38. The research team (SG, SS, LH and CG) have coded which BCTs are present for each intervention component ²¹. Two independent coders, external to the research team, will also code the intervention components. Coders will be provided with all intervention component materials and a coding manual and will be asked to identify which BCTs from the taxonomy are present in each intervention component. This process will be conducted once the trial has begun recruitment. The coders will have experience in using BCTs and will have completed BCTTv1 training. They will only code whether a BCT is present or absent, not the frequency of occurrence. Coding from the independent coders will be compared to the original coding ²¹, and an agreement coefficient will be calculated ³⁹. All discrepancies will be discussed between one member of the research team (SG) and the two independent coders, and a final code list will be produced.

Fidelity of training (competency of delivery). The only intervention component to be delivered by healthcare professionals that requires training is the ACT component. Therapists will receive two half days of bespoke training from a clinical psychologist with expertise in ACT (CDG). To establish the adequacy of the training for delivery of the ACT component, a clinical psychologist (CDG) will assess the therapists recording of their first ACT session using the ACT Fidelity Measure (ACT-FM), which assesses therapist fidelity to ACT principles ⁴⁰. The number of booster training sessions the therapists require will be monitored. Semi-structured interviews with the therapists will explore barriers to adequate training and scope for any improvements.

Fidelity of delivery. SMS receipt data will indicate whether a message has been successfully sent by the online system. The number of participants who opt-out of the SMS messages will be recorded. The number of information leaflets and website login details successfully sent to participants will be recorded by the site. For the ACT component, therapist scores on a procedural fidelity checklist will indicate which parts of the component were delivered ( e.g., reflection on the utility, relevance and barriers to home practice tasks and introducing the following module). An expert ACT practitioner independent to the trial team will review 10% of all ACT session recordings using the ACT-FM, which assesses therapist fidelity to ACT principles. Reviewed recordings will include those conducted early and late in the trial and from all therapists involved.

Fidelity of receipt. Participants will be asked a single self-reported item to assess which intervention components they received. Self-reported engagement with the intervention component(s) will be collected, along with reasons for non-engagement. For the website, tracking data will be collected, including number of logins, time spent on website, pages visited and videos watched. Each participant will only be asked about the components they were randomised to receive. Semi-structured interviews will investigate fidelity of receipt and elicit any barriers or enablers to the receipt of all intervention components.

ACT therapists will report the number of sessions attended, cancelled or missed by each participant, and the participant’s engagement with module materials (participant manual, associated audio files and home practice tasks). Participant’s engagement with the module materials will also be assessed by an external reviewer assessing 10% of the ACT session recordings.

Fidelity of enactment. Semi-structured interviews will be used to assess the extent to which the participant uses the intervention component(s) in their day-to-day life, and any barriers or facilitators to this.

Intervention component acceptability . The assessment of acceptability of the intervention components will be guided by the TFA ³⁶ ( Table 3). As described in Table 3, acceptability will be assessed using an acceptability questionnaire specific to each intervention component at the 4 month follow up ⁴¹, and semi-structured interviews with trial participants and therapists at the end of the trial.

Table 3. Summary of acceptability assessments.

Objective	Data	Data source	Method of completion	Time point
Assess the acceptability of each intervention component	Acceptability Questionnaire	Questionnaire	Participant	4 months
	Qualitative data	Semi-structured interview	Participant	End of trial
	Qualitative data	Semi-structured interview	Therapist	End of trial

Open in a new tab

Trial experience, recruitment and retention . A mix of methods will involve trial participants and RNs ( Table 4). Trial experience will be assessed in participants using a questionnaire assessing experience before, during and after the trial, a single item assessing overall trial acceptability, and during semi-structured interview with questions guided by the TFA ³⁶. The assessment of the barriers and facilitators to recruitment and retention to the factorial trial will be guided by the TDF ²⁸. Assessment will include one questionnaire for RNs exploring barriers and facilitators to recruitment, and interviews with participants with questions focusing on any barriers to participation and retention in the trial.

Table 4. Summary of trial experience, recruitment and retention assessments.

Objective	Data	Data source	Method of completion	Time point
Evaluate participant experience of trial participation	SPFQ	Questionnaire	Participant	Baseline, 2 months, 4 months
	Acceptability Questionnaire (single item)	Questionnaire	Participant	4 months
	Qualitative data	Semi-structured interview	Participant	End of trial
Establish the barriers and facilitators to recruitment and retention of participants	Recruitment Questionnaire	Questionnaire	Research Nurse	End of trial
	Qualitative data	Semi-structured interview	Participant	End of trial
	Qualitative data	Semi-structured interview	Research Nurse	End of trial

Open in a new tab

Key: SPFQ= Study Participant Feedback Questionnaire.

Sampling and recruitment

Trial participants . Women over 18 years old taking AET (tamoxifen, raloxifene, anastrozole, letrozole or exemestane) for early stage (I to IIIa) breast cancer, who have completed their last hospital treatment within the previous 12 months, are eligible to be part of the ROSETA pilot trial ( Table 5). All trial participants will be asked to complete the quantitative assessments of the process evaluation at baseline, 2 and 4-month follow-up alongside the pilot trial assessments.

Table 5. Eligibility criteria for participation in the ROSETA pilot trial.

Inclusion Criteria

Exclusion Criteria

1.  An informed consent form (signed and dated)
2.  Capacity to provide informed consent
3.  Women with early stage (1 to 3a) breast cancer according to the Tumour, Node, Metastasis
(TNM) / American Joint Committee on Cancer (AJCC) staging system.
      Note. Women being treated for a second primary breast cancer or a breast cancer local
recurrence are eligible for the study, providing the most recent cancer is being treated with
adjuvant endocrine therapy, and they meet all eligibility criteria. Women with bilateral breast
cancer are permitted, providing at least one breast is affected by hormone receptor-positive
disease.
4.  Aged ≥18 years old at time of screening for ROSETA’s pilot study
5.  Have sufficient proficiency in English to be able to adhere to all intervention components
and data collection required
6.  Treated with curative intent
7.  Completed their hospital-based treatment ( e.g., surgery, radiotherapy and/or
chemotherapy) for the current breast cancer within the last 12 months.
      Note. Women are still eligible for the study if they are being treated with monoclonal
antibody-based therapy such as trastuzumab, Kadcyla, pertuzumab, and Phesgo.
8.  Currently prescribed oral adjuvant Hormone Therapy (tamoxifen, raloxifene, anastrozole,
letrozole, exemestane)
9.  The participant is willing to complete the study questionnaires *
10.  The participant is willing to be audio recorded during the therapy sessions *
11.  The participant is willing and able to attend all ACT sessions either via video conference or
telephone *
12.  The participant is willing and able to complete home practice tasks *
13.  Access to a mobile phone to receive SMS messages *
14.  Willing to receive frequent SMS messages *
15.  Access to a computer or smart device that can access the internet *

1. Stopped taking adjuvant hormone therapy if it is clinically contraindicated
according to clinical recommendation
2.  Women with Metastatic breast cancer
3.  Currently or recently (last 6 months) involved in a similar research study
where medication adherence is a primary outcome *
4.  Currently attending psychotherapy/psycho-oncology/psychology/
counselling services, for any clinical reason *
5.  Need for treatment for a severe mental health disorder or crisis, which is
likely to interfere with participation ( e.g., active psychosis, bipolar disorder,
significant issues with addiction or self-harm or expressing active suicidal
ideation with active plans and intent *)
      Note, if concerned about the possible presence of risk of suicidal ideation
with active plans and intent, then this can be assessed with the following
questions, with patients ineligible if they answer ‘yes’ to 5c.
      Recently (in the last month):
         a.  Have you had any thoughts about ending your life?
         b.  (if yes) Have you thought about how you might go about it?
         c.  (if yes) Do you intend to carry out this plan?
6.  Patients with a scheduled date for breast reconstruction surgery that
is within their intervention delivery and follow-up period. Note: Women
planning to have a breast reconstruction but who have not scheduled a date
for surgery are permitted.
7.  Auditory problems that would prevent the patient from participating in a
telephone or video call, or hearing audio clips *

Open in a new tab

Items marked with * are collected via self-report. Key:ROSETA= Refining and Optimising a behavioural intervention to Support Endocrine Therapy Adherence. SMS= Short Message Service. ACT= Acceptance and commitment therapy. This table has been reproduced with permission from Smith et al., (2023) ²².

Participation in the end of trial interview is optional. When providing initial consent at baseline, participants will be asked if they are willing to be contacted for an interview at the end of the trial. If willing, participants will be contacted with further information approximately three-months post-randomisation. Consent may be written or via telephone. We aim to interview at least one participant from each experimental condition (corresponding to unique combinations of intervention components) in which they receive at least one component. Maximum variation purposive sampling will aim to achieve a mix of participants above and below 50 years old. Participants will be interviewed until we have gained sufficient information power for each study aim (fidelity, acceptability and trial experience) ⁴². Information power will be discussed in regular team meetings to inform further sampling. The specific aims of this study, high specificity of the sample and use of established theoretical frameworks to inform the interview indicates a smaller sample size may be appropriate. The interviewer (SG) has had experience interviewing women with breast cancer taking AET for similar purposes ⁴³ and therefore a good quality of dialogue is expected ⁴². However, this will require ongoing evaluation as this is dependent on the interviewer and participants. The analytic strategy will explore themes across cases, rather than individual in depth analysis, which is likely to require an increase in sample size. Thus, a small to moderate sample size is expected to be required ⁴².

Intervention deliverers (ACT therapists) . Therapists delivering the ACT component will be Health and Care Professions Council (HCPC) or United Kingdom Council for Psychotherapy (UKCP) registered band 7a or above practitioner psychologists, or psychotherapists. Once the site has opened to recruitment, the therapist will be sent an invitation letter, information sheet and consent form. All consenting therapists delivering the ACT intervention component will be asked to complete the quantitative aspect ( e.g., procedural fidelity checklists, fidelity assessments), and to participate in an end of trial interview.

One month before the end of the intervention delivery period, one therapist from each site will be randomly selected to be interviewed from those that have consented. Further therapists will be interviewed until the sample holds sufficient information power or there are no new consenting therapists left to interview. Information power will be discussed regularly in team meetings to inform further sampling ⁴². Based on the specific study aims, high specificity of the sample, use of theoretical frameworks, expected good quality dialogue and cross-case analysis, a small to moderate sample size is expected ⁴².

Research nurses . All RNs (or equivalent) involved in the recruitment of participants to the ROSETA pilot trial will be invited to participate in the process evaluation. Once the site has opened to recruitment, the RN will be sent an invitation letter, information sheet and consent form. If they consent, they will be asked to complete a questionnaire regarding the barriers and facilitators to recruitment at the end of the trial recruitment period.

Assessment measures

Participant

Adherence to intervention components. To assess fidelity of receipt of the intervention components, participants will be asked two self-report items for each intervention component they were allocated to receive. The questions ask whether they received each intervention component, and how much of the intervention component they read. In relation to the ACT component, participants will be asked two additional items asking how much of the home practice tasks they completed, and how many of the audio files they listened to. Responses are on a 3-point scale; “none”, “at least some” and “all of”.

Acceptability Questionnaire (AQ) ⁴¹. The acceptability questionnaire, based on the TFA, will assess the acceptability of the individual intervention components, modified for each component ^36,
41. Three items have been removed (ethicality, self-efficacy and opportunity costs) to reduce participant burden for those allocated to multiple intervention components. Five items remain assessing general acceptability, affective attitude, burden, perceived effectiveness and intervention coherence. Participants will answer on a 5-point Likert Scale.

For participants randomised to the ACT component, 15 extra items ask specifically about the acceptability of elements of ACT sessions and format. For participants randomised to the SMS component one additional item asks about the acceptability of the frequency of the SMS messages. One open question asks for any further comments about the acceptability of intervention components. The ‘general acceptability’ item from the AQ will assess the acceptability of participants’ experience in the trial as a whole. Participants will answer on a 5-point Likert Scale.

Study Participant Feedback Questionnaire (SPFQ) ⁴⁴. The SPFQ will assess participants’ experience throughout the trial. This has been modified for the current trial. At baseline, participants will be asked two items about information received prior to the trial. Participants will answer on a 5-point Likert Scale. At the 2-month follow up, participants will be asked to complete three yes/no items regarding their experience during the trial. At the 4-month follow up participants will complete two items about their overall satisfaction with their experience in the trial, answering on a 5-point Likert Scale.

Therapist

ACT Fidelity Measure (ACT-FM) ⁴⁰. ACT-FM is a measure of therapist fidelity to ACT principles when delivering treatment ( i.e., competency). The ACT-FM therapist stance subscale will be used to assess fidelity of training and delivery of the ACT component. Only the therapist stance subscale will be used, as this is the subscale most relevant to therapist delivery of the ACT component. Four items assess ACT-consistent behaviours, and three items assess ACT-inconsistent behaviours. Each item is scored on a 4-point scale (0–3). Scores can range from 0–9 on each subscale. A score of >4 on ACT consistent behaviours, and <5 on ACT inconsistent behaviours will be considered competent.

Procedural Fidelity Checklist. A procedural fidelity checklist for each individual session will assess fidelity of delivery of the ACT component. The checklist is designed specifically for the ACT component in ROSETA, for therapists to self-rate whether they undertook core intervention procedures, such as reflection on the skills exercises and home practice tasks. It includes eight items for session 1 and 2, seven items for session 3, six items for session 4, and four items for session 5. A percentage score is calculated for each session by dividing the score achieved by the maximum score achievable in that session and multiplying by 100. Additional items ask therapists to record to what extent a participant has engaged with the module materials (participant manual, home practice tasks and associated audio files). In each checklist, the therapists will also record the number of times the session has been cancelled.

Research Nurse

Barriers and facilitators to recruitment. RNs will complete a questionnaire to report their experiences of recruiting to a factorial trial. Questions are based on the TDF ²⁸ and cover knowledge of the trial design, the influence of resources, social pressures and emotional factors on recruitment, their motivation levels in recruiting to the trial, their decision processes when contacting a patient to schedule a recruitment appointment, and any procedural changes that would facilitate recruitment.

Qualitative interviews

A rapid qualitative approach will be used to ensure findings from the pilot trial process evaluation can inform the future optimisation trial in a timely manner ^45,
46. Semi-structured interviews with trial participants and therapists will last up to one hour, will be conducted via telephone or videoconferencing software ( e.g., Microsoft Teams), and will be recorded. During the interview, the interviewer will take notes. Immediately following each interview, the interviewer will complete an individual Rapid Research Evaluation and Appraisal Lab Rapid Assessment Procedure (RREAL RAP) sheet for the participant. RREAL RAP sheets are a two-column table summarising the information collected in an interview. Different RREAL RAP sheets will be used for therapists and participants.

Trial participants . Semi-structured interviews will assess fidelity of receipt and enactment, acceptability of the intervention components, and barriers and facilitators to recruitment and retention in the trial. Questions will be based on the NIH fidelity domains, the TFA, and the TDF. Interview guides have been reviewed and amended based on suggestions from our patient and public involvement group (participant interview questions can be found as Extended data ⁴⁷).

ACT therapists . Semi-structured interviews with therapists will assess fidelity of training and delivery and any barriers to these, relationship building with participants in the intervention component, and acceptability of the ACT component (therapist interview questions can be found as Extended data ⁴⁷). We will also explore any suggested improvements to the intervention component and establish whether it may be implementable in routine care in the NHS.

Data analysis plan

Quantitative analysis

Intervention fidelity. Descriptive statistics will summarise the quantitative intervention component fidelity assessments ( Table 2). For the ACT component, these will also be presented by site. Specific to the fidelity of design assessment, the first-order agreement coefficient statistic (AC1) will estimate the interrater reliability between the independent coders and the BCTs that the components intended to target ^21,
39. Strength of fidelity of design will be defined in terms of pre-established AC1 thresholds; <0.2=poor, 0.2 ≤ 0.4=fairly poor, >0.4 ≤0.6=moderate, >0.6 ≤0.8=good, >0.8 and ≤ 1=very good ⁴⁸. Discrepancies will be resolved through discussions between SG and the coders.

Intervention component acceptability. Descriptive statistics will be calculated for the individual items on the acceptability questionnaire, and overall, for each intervention component. Additional questions regarding the acceptability of specific aspects of the ACT and SMS components will be summarised. All summaries will be presented by component. Missing data will be summarised.

Barriers and facilitators to recruitment and retention of participants. Descriptive statistics will be presented for the RN barriers to recruitment and retention questionnaire, alongside qualitative data from the questionnaire. Summaries will be presented overall and by site. Missing data will be summarised.

Participant experience of trial participation. Descriptive statistics will be presented (1) for overall trial experience, and split by experience before starting the trial, during the trial and at the end of the trial; (2) for the one item of the AQ aimed at assessing general acceptability of the trial overall, by experimental condition and by site. Missing data will be summarised.

Qualitative analysis . Individual RREAL RAP sheets completed immediately after the interview will be collated into higher level RREAL RAP sheets; one per component for data relating to intervention component acceptability and fidelity, one overall for data relating to participant trial experience, and one overall for therapist interview data ^45,
46,
49. Regular team meetings will review emerging findings, changes to the interview schedule or RREAL RAP sheets, and data sampling. The interviews will be recorded and selectively transcribed verbatim. Specific quotes will be extracted from the transcripts and added to the RREAL RAP sheets to inform analysis. For each overall aim (fidelity, intervention component acceptability, and trial experience), the relevant quantitative and qualitative data will be combined narratively to determine whether the intervention components could be delivered with fidelity overall, whether each intervention component was acceptable to women with breast cancer, and whether the experience in the trial was acceptable to women with breast cancer, or if there were barriers to recruitment and retention within the pilot trial. Together these findings will be used to make improvements to the intervention components and trial processes, should these be necessary.

Ethics and dissemination

The study has been approved by Wales Research Authority Research Ethics Committee 3 (21/WA/0322) and is a registered clinical trial (ISRCTN registry, ISRCTN10487576, 16/12/2021). It is sponsored by the University of Leeds ( governance-ethics@leeds.ac.uk). Amendments to the protocol will be submitted to the ethics committee, and if approved, communicated to research sites. Study findings will be disseminated through peer-reviewed publications. At the end of the trial, all data held by the CTRU and all trial data will then be securely archived at the University of Leeds for a minimum of 5 years. This paper is a summary of protocol version 3.0 (18/08/2022).

Trial status

The study opened for recruitment on the 20th of May 2022 and was open at the time of submission.

Discussion

This paper describes the design and methods of a process evaluation embedded in the ROSETA pilot trial. This process evaluation aims to investigate whether four intervention components can be delivered with fidelity, whether the intervention components are acceptable to participants and trial therapists, and to explore participant’s experiences in the trial along with barriers and facilitators to recruitment and retention. Given there is little guidance available for the conduct of process evaluations within factorial trial designs, our protocol may guide others seeking to conduct similar process evaluations.

The results of this process evaluation will be used to adapt and improve the intervention components where necessary, should progression criteria to a larger optimisation trial be met. Our progression criteria to progress to a full optimisation trial are based on predefined criteria for consent rates, intervention component adherence, and availability of outcome data. These are described in detail elsewhere ²². The full optimisation trial would aim to identify the most effective combination of intervention components to support medication adherence in women with early stage breast cancer, without exceeding a pre-specified cost of £3397 per patient, based on health economic modelling ²¹.

This process evaluation has the potential to highlight key issues in the specific intervention components in terms of fidelity and acceptability, and regarding the overall trial design and its implementation within an NHS setting. As factorial designs are relatively uncommon in behavioural trials, we hope that this process evaluation can offer useful insights for others.

Acknowledgments

We would like to thank our patient and public involvement members; Rebecca Whittemore, Michaela Kneafsey and Hazel Abey. We also thank the Trial Steering Committee for their support in designing and delivering the trial.

Funding Statement

This project is funded by the National Institute for Health Research (NIHR) under its NIHR Fellowships Programme (Grant Reference Number NIHR300588; to Samuel Smith). Smith is also supported by a Yorkshire Cancer Research Fellowship (L417). Foy is supported by grant funding from the National Institute for Health and Care Research and Yorkshire Cancer Research. DF is supported by the NIHR Manchester Biomedical Research Centre (IS-BRC-1215-20007). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 2; peer review: 2 approved]

Data availability

Underlying data

No data are associated with this article.

Extended data

Open Science Framework: ROSETA Pilot Trial Process Evaluation. https://doi.org/10.17605/OSF.IO/8DWRN ⁴⁷.

The project contains the following extended data:

Participant Interview Guide
Therapist Interview Guide

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

References

1. Cancer Research UK: Breast Cancer Statistics. Reference Source [Google Scholar]
2. Early Breast Cancer Trialists Collaborative Group. C Davies, J Godwin: Relevance of breast cancer hormone receptors and other factors to the efficacy of adjuvant tamoxifen: patient-level meta-analysis of randomised trials. Lancet. 2011;378(9793):771–84. 10.1016/S0140-6736(11)60993-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
3. Early breast Cancer Trialists Collaborative Group: Aromatase inhibitors versus tamoxifen in early breast cancer: patient-level meta-analysis of the randomised trials. Lancet. 2015;386(10001):1341–52. 10.1016/S0140-6736(15)61074-1 [DOI] [PubMed] [Google Scholar]
4. Hershman DL, Kushi LH, Shao T, et al. : Early discontinuation and nonadherence to adjuvant hormonal therapy in a cohort of 8,769 early-stage breast cancer patients. J Clin Oncol. 2010;28(27):4120–8. 10.1200/JCO.2009.25.9655 [DOI] [PMC free article] [PubMed] [Google Scholar]
5. Murphy CC, Bartholomew LK, Carpentier MY, et al. : Adherence to adjuvant hormonal therapy among breast cancer survivors in clinical practice: a systematic review. Breast Cancer Res Treat. 2012;134(2):459–78. 10.1007/s10549-012-2114-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
6. Huiart L, Ferdynus C, Giorgi R: A meta-regression analysis of the available data on adherence to adjuvant hormonal therapy in breast cancer: summarizing the data for clinicians. Breast Cancer Res Treat. 2013;138(1):325–8. 10.1007/s10549-013-2422-4 [DOI] [PubMed] [Google Scholar]
7. Makubate B, Donnan PT, Dewar JA, et al. : Cohort study of adherence to adjuvant endocrine therapy, breast cancer recurrence and mortality. Br J Cancer. 2013;108(7):1515–24. 10.1038/bjc.2013.116 [DOI] [PMC free article] [PubMed] [Google Scholar]
8. McCowan C, Wang S, Thompson AM, et al. : The value of high adherence to tamoxifen in women with breast cancer: a community-based cohort study. Br J Cancer. 2013;109(5):1172–80. 10.1038/bjc.2013.464 [DOI] [PMC free article] [PubMed] [Google Scholar]
9. Inotai A, Ágh T, Maris R, et al. : Systematic review of real-world studies evaluating the impact of medication non-adherence to endocrine therapies on hard clinical endpoints in patients with non-metastatic breast cancer. Cancer Treat Rev. 2021;100:102264. 10.1016/j.ctrv.2021.102264 [DOI] [PubMed] [Google Scholar]
10. Moon Z, Moss-Morris R, Hunter MS, et al. : Understanding tamoxifen adherence in women with breast cancer: A qualitative study. Br J Health Psychol. 2017;22(4):978–97. 10.1111/bjhp.12266 [DOI] [PubMed] [Google Scholar]
11. Cahir C, Guinan E, Dombrowski SU, et al. : Identifying the determinants of adjuvant hormonal therapy medication taking behaviour in women with stages I–III breast cancer: A systematic review and meta-analysis. Patient Educ Couns. 2015;98(12):1524–39. 10.1016/j.pec.2015.05.013 [DOI] [PubMed] [Google Scholar]
12. Brett J, Fenlon D, Boulton M, et al. : Factors associated with intentional and unintentional non-adherence to adjuvant endocrine therapy following breast cancer. Eur J Cancer Care (Engl). 2018;27(1): e12601. 10.1111/ecc.12601 [DOI] [PubMed] [Google Scholar]
13. Toivonen K, Williamson T, Carlson L, et al. : Potentially Modifiable Factors Associated with Adherence to Adjuvant Endocrine Therapy among Breast Cancer Survivors: A Systematic Review. Cancers (Basel). 2020;13(1):107. 10.3390/cancers13010107 [DOI] [PMC free article] [PubMed] [Google Scholar]
14. Finitsis DJ, Vose BA, Mahalak JG, et al. : Interventions to promote adherence to endocrine therapy among breast cancer survivors: A meta-analysis. Psychooncology. 2019;28(2):255–63. 10.1002/pon.4959 [DOI] [PubMed] [Google Scholar]
15. Heiney SP, Parker PD, Felder TM, et al. : A systematic review of interventions to improve adherence to endocrine therapy. Breast Cancer Res Treat. 2019;173(3):499–510. 10.1007/s10549-018-5012-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
16. Hurtado-De-Mendoza A, Cabling ML, Lobo T, et al. : Behavioral Interventions to Enhance Adherence to Hormone Therapy in Breast Cancer Survivors: A Systematic Literature Review. Clin Breast Cancer. 2016;16(4):247–255.e3. 10.1016/j.clbc.2016.03.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
17. Collins LM: Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). Springer;2018. 10.1007/978-3-319-72206-1 [DOI] [Google Scholar]
18. Collins LM: Conceptual Introduction to the Multiphase Optimization Strategy (MOST). Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). Cham: Springer International Publishing;2018;1–34. 10.1007/978-3-319-72206-1_1 [DOI] [Google Scholar]
19. Collins LM, Kugler KC, Gwadz MV: Optimization of Multicomponent Behavioral and Biobehavioral Interventions for the Prevention and Treatment of HIV/AIDS. AIDS Behav. 2016;20 Suppl 1(0 1):197–214. 10.1007/s10461-015-1145-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
20. Collins LM, Strayhorn JC, Vanness DJ: One view of the next decade of research on behavioral and biobehavioral approaches to cancer prevention and control: intervention optimization. Transl Behav Med. 2021;11(11):1998–2008. 10.1093/tbm/ibab087 [DOI] [PMC free article] [PubMed] [Google Scholar]
21. Green SMC, French DP, Graham CD, et al. : Supporting adjuvant endocrine therapy adherence in women with breast cancer: the development of a complex behavioural intervention using Intervention Mapping guided by the Multiphase Optimisation Strategy. BMC Health Serv Res. 2022;22(1): 1081. 10.1186/s12913-022-08243-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
22. Smith SG, Green SMC, Ellison R, et al. : Refining and optimising a behavioural intervention to support endocrine therapy adherence (ROSETA) in UK women with breast cancer: protocol for a pilot fractional factorial trial. BMJ Open. 2023;13(2):e069971. 10.1136/bmjopen-2022-069971 [DOI] [PMC free article] [PubMed] [Google Scholar]
23. Moore GF, Audrey S, Barker M, et al. : Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. 10.1136/bmj.h1258 [DOI] [PMC free article] [PubMed] [Google Scholar]
24. Nair B: Clinical Trial Designs. Indian Dermatol Online J. 2019;10(2):193–201. 10.4103/idoj.IDOJ_475_18 [DOI] [PMC free article] [PubMed] [Google Scholar]
25. Collins LM, Dziak JJ, Kugler KC, et al. : Factorial experiments: efficient tools for evaluation of intervention components. Am J Prev Med. 2014;47(4):498–504. 10.1016/j.amepre.2014.06.021 [DOI] [PMC free article] [PubMed] [Google Scholar]
26. McDonald AM, Knight RC, Campbell MK, et al. : What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):9. 10.1186/1745-6215-7-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
27. Walters SJ, Bonacho Dos Anjos Henriques-Cadby I, Bortolami O, et al. : Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017;7(3):e015276. 10.1136/bmjopen-2016-015276 [DOI] [PMC free article] [PubMed] [Google Scholar]
28. Cane J, O’Connor D, Michie S: Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7(1):37. 10.1186/1748-5908-7-37 [DOI] [PMC free article] [PubMed] [Google Scholar]
29. Carroll C, Patterson M, Wood S, et al. : A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40. 10.1186/1748-5908-2-40 [DOI] [PMC free article] [PubMed] [Google Scholar]
30. Lorencatto F, Gould NJ, McIntyre SA, et al. : A multidimensional approach to assessing intervention fidelity in a process evaluation of audit and feedback interventions to reduce unnecessary blood transfusions: a study protocol. Implement Sci. 2016;11(1):163. 10.1186/s13012-016-0528-x [DOI] [PMC free article] [PubMed] [Google Scholar]
31. Toomey E, Hardeman W, Hankonen N, et al. : Focusing on fidelity: narrative review and recommendations for improving intervention fidelity within trials of health behaviour change interventions. Health Psychol Behav Med. 2020;8(1):132–51. 10.1080/21642850.2020.1738935 [DOI] [PMC free article] [PubMed] [Google Scholar]
32. Bellg AJ, Borrelli B, Resnick B, et al. : Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23(5):443–51. 10.1037/0278-6133.23.5.443 [DOI] [PubMed] [Google Scholar]
33. Resnick B, Bellg AJ, Borrelli B, et al. : Examples of implementation and evaluation of treatment fidelity in the BCC studies: where we are and where we need to go. Ann Behav Med. 2005;29 Suppl(2):46–54. 10.1207/s15324796abm2902s_8 [DOI] [PubMed] [Google Scholar]
34. Hawkes RE, Miles LM, Bower P, et al. : Assessing and ensuring fidelity of the nationally implemented English NHS diabetes prevention programme: lessons learned for the implementation of large-scale behaviour change programmes. Health Psychol Behav Med. 2022;10(1):498–513. 10.1080/21642850.2022.2077205 [DOI] [PMC free article] [PubMed] [Google Scholar]
35. Skivington K, Matthews L, Simpson SA, et al. : A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374:n2061. 10.1136/bmj.n2061 [DOI] [PMC free article] [PubMed] [Google Scholar]
36. Sekhon M, Cartwright M, Francis JJ: Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17(1):88. 10.1186/s12913-017-2031-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
37. Lambert JD, Elliott LR, Taylor AH, et al. : A novel method for assessing design fidelity in web-based behavioral interventions. Health Psychol. 2021;40(3):217–25. 10.1037/hea0001046 [DOI] [PubMed] [Google Scholar]
38. Michie S, Richardson M, Johnston M, et al. : The Behavior Change Technique Taxonomy (v1) of 93 Hierarchically Clustered Techniques: Building an International Consensus for the Reporting of Behavior Change Interventions. Ann Behav Med. 2013;46(1):81–95. 10.1007/s12160-013-9486-6 [DOI] [PubMed] [Google Scholar]
39. Gwet K: Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Statistical methods for inter-rater reliability assessment.2002;1(6):1–6. Reference Source [Google Scholar]
40. O'Neill L, Latchford G, McCracken LM, et al. : The development of the Acceptance and Commitment Therapy Fidelity Measure (ACT-FM): A delphi study and field test. J Contextual Behav Sci. 2019;14:111–8. 10.1016/j.jcbs.2019.08.008 [DOI] [Google Scholar]
41. Sekhon M, Cartwright M, Francis JJ: Development of a theory-informed questionnaire to assess the acceptability of healthcare interventions. BMC Health Serv Res. 2022;22(1): 279. 10.1186/s12913-022-07577-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
42. Malterud K, Siersma VD, Guassora AD: Sample Size in Qualitative Interview Studies: Guided by Information Power. Qual Health Res. 2016;26(13):1753–60. 10.1177/1049732315617444 [DOI] [PubMed] [Google Scholar]
43. Smith SG, Ellison R, Hall L, et al. : Acceptance and Commitment Therapy to support medication decision-making and quality of life in women with breast cancer: protocol for a pilot randomised controlled trial. Pilot Feasibility Stud. 2022;8(1):33. 10.1186/s40814-022-00985-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
44. Greene A, Elmer M, Ludlam S, et al. : Evaluation of the Content Validity and Cross-Cultural Validity of the Study Participant Feedback Questionnaire (SPFQ). Ther Innov Regul Sci. 2020;54(6):1522–33. 10.1007/s43441-020-00179-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
45. Vindrola-Padros C, Johnson GA: Rapid Techniques in Qualitative Research: A Critical Review of the Literature. Qual Health Res. 2020;30(10):1596–604. 10.1177/1049732320921835 [DOI] [PubMed] [Google Scholar]
46. Vindrola-Padros C, Chisnall G, Polanco N, et al. : Iterative cycles in qualitative research: Introducing the RREAL Sheet as an innovative process.2022. 10.31219/osf.io/9dp2w [DOI] [Google Scholar]
47. Green SMC, Smith S: ROSETA Pilot Trial Process Evaluation. OSF. [Dataset].2022. 10.17605/OSF.IO/8DWRN [DOI]
48. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. 10.2307/2529310 [DOI] [PubMed] [Google Scholar]
49. Vindrola-Padros C, Brage E, Johnson GA: Rapid, Responsive, and Relevant?: A Systematic Review of Rapid Evaluations in Health Care. Am J Eval. 2021;42(1):13–27. 10.1177/1098214019886914 [DOI] [Google Scholar]

NIHR Open Res. 2023 May 15. doi: 10.3310/nihropenres.14552.r29195

Reviewer response for version 2

Molly Byrne ¹

Many thanks to the authors for their thorough considerations of my recommendations and for the amendments. I am now happy to approve the paper.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

NIHR Open Res. 2023 May 9. doi: 10.3310/nihropenres.14552.r29196

Reviewer response for version 2

Thomas Mills ¹

This is a much-improved manuscript: inclusion of the conceptual model helps convey what the intervention targets.

I still don’t feel that the Plain English summary sets up the study as the authors are aiming to achieve more than “find out” stuff, and will be doing more than interviewing and asking. There’s a balance between making papers accessible and dumbing down and I don’t think the authors are striking it here. This is a matter of style however.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Partly

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

The qualitative component of process evaluations.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

NIHR Open Res. 2023 Mar 21. doi: 10.3310/nihropenres.14463.r29031

Reviewer response for version 1

Thomas Mills ¹

Technically speaking, this is well thought out and looks to be an excellent process evaluation. One thing I’d like the authors to consider is *why* they are doing this. The initial set of aims needs developing: surely an aim here is to facilitate research-informed improvements to the intervention components ahead of the trial? I also feel there’s a gap pertaining to this aim in the protocol whereby the researchers draw on the findings to inform intervention changes, as well as their use of analytical frameworks to guide this: the TDF framework is mentioned on occasions but only in passing. A final section/sub-section after, or at the end of the data analysis section to outline a proposed plan for pulling it all together would add the cherry onto the cake, so to speak.

Study methods

The aims need further development. I suggest having a set of aims and a set of objectives. Also, if the first set of aims is numbered, could the second set be letter-ed, so as to differentiate between them further?
Aims 2 and 3 are similar – exploring women’s acceptability and experiential outcomes. I wonder if there is another aim, which is alluded to elsewhere in the paper, which is to use the process findings to refine intervention components ahead of the full trial.
In the “To do this we will” section, 3. implies that intervention components are all readable, when therapy sessions are planned also. All of these points relate to data collection activities: I presume you’ll be analysing the data too, to fulfil the aims? Key analytical strategies could be included here.

Roseta process evaluation intro

No mention of analytical strategy/data analysis – e.g., TDF framework

Protocol – Aims and objectives

How do these, new aims relate to the aims stated in the study methods? These different sets of aims need to be linked.

Design

Fidelity – another introduction to the five domains – it feels like you’re going backward because the domains have been introduced already: focus this sentence on describing Table 2 in more detail.

Fidelity design

Example BCTs would help the reader - possibility of a logic model to show the causal assumptions?

Fidelity of enactment

Ditto – text could focus describing Table 3 in more detail.

Trial experience, retention and recruitment

First mention of TDF framework – this is important because it will help the researchers understand the data and highlight areas of improvement

Qualitative interviews

Why will the interviewer be taking notes during the interview? This risks obstructing the flow of the interview. The interview will be recorded. I suggest the interviewer write up the interview immediately after the interview, using the recording as a guide; then check the summary against the transcript, once the recording is transcribed.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Partly

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Yes

Reviewer Expertise:

The qualitative component of process evaluations.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

NIHR Open Res. 2023 Apr 13.

Sophie Green ¹

Technically speaking, this is well thought out and looks to be an excellent process evaluation. One thing I’d like the authors to consider is *why* they are doing this.

Thank you for taking the time to review our manuscript and for your constructive comments.

The initial set of aims needs developing: surely an aim here is to facilitate research-informed improvements to the intervention components ahead of the trial? I also feel there’s a gap pertaining to this aim in the protocol whereby the researchers draw on the findings to inform intervention changes, as well as their use of analytical frameworks to guide this: the TDF framework is mentioned on occasions but only in passing.

RESPONSE: In line with our decision to undertake an external pilot trial, as opposed to a feasibility trial, using the findings from the process evaluation to refine the intervention components ahead of the optimisation trial was not a main aim of the research. The aims were to determine the acceptability of the intervention components, determine if they could be delivered with fidelity and to assess women’s experience within the trial. As such, this process evaluation contributes towards the broader goal of determining if a larger trial to optimise these components is feasible to undertake. We will make some minor amendments to the intervention components if it will improve their acceptability, fidelity or women’s trial experience, but this was not the primary purpose of doing a pilot trial with an embedded process evaluation.

In terms of our used of analytical frameworks, we have now added an introduction to the TDF in the introduction. In the introduction we had already introduced the other two theoretical frameworks we are using; the theoretical framework of acceptability and the five domains of fidelity. In the ‘design’ section in the methods, we then explain which framework we are using for each aim of the process evaluation. The five domains of fidelity is being used to address the fidelity aim (aim 1), the theoretical framework of acceptability is being used to determine acceptability of the intervention components (aim 2), and the TDF is being used to assess participant trial experience and barriers and facilitators to recruitment and retention (aim 3).

A final section/sub-section after, or at the end of the data analysis section to outline a proposed plan for pulling it all together would add the cherry onto the cake, so to speak.

RESPONSE: We have added an additional paragraph into the data analysis section as suggested to outline our proposed plan for pulling the quantitative and qualitative data together for each aim.

Methods: “For each overall aim (fidelity, intervention component acceptability, and trial experience), the relevant quantitative and qualitative data will be combined narratively to determine whether the intervention components could be delivered with fidelity overall, whether each intervention component was acceptable to women with breast cancer, and whether the experience in the trial was acceptable to women with breast cancer, or if there were barriers to recruitment and retention within the pilot trial. Together, these findings will be used to make improvements to the intervention components and trial processes, should these be necessary.”

Study methods

The aims need further development. I suggest having a set of aims and a set of objectives. Also, if the first set of aims is numbered, could the second set be letter-ed, so as to differentiate between them further?

RESPONSE: This comment is in reference to the aims within the plain English summary/ abstract of the work, with a word limit of 300 words. We therefore request that we do not go into detail about the additional objectives within each aim (as we also have not done in the scientific abstract), as there is insufficient space to include these here. In the main body of the paper, we have included aims, and specific objectives for each aim, using numbering and lettering to differentiate between these.

Aims 2 and 3 are similar – exploring women’s acceptability and experiential outcomes. I wonder if there is another aim, which is alluded to elsewhere in the paper, which is to use the process findings to refine intervention components ahead of the full trial.

RESPONSE: Aim 2 refers to the acceptability specifically of each of the four intervention components, while aim 3 refers to women’s experience within the trial in relation to the novel trial design we are using, and general trial processes, such as the length of questionnaires. We have made some amendments to the wording of these aims in the plain English summary to differentiate them further.

Please see our earlier response with regard to including an additional aim.

“Plain English Summary Aims:

2. To find out if the support received as part of the trial was acceptable to women with breast cancer (acceptability).

3. To find out what women’s experience was of taking part in the trial overall (trial experience).”

In the “To do this we will” section, 3. implies that intervention components are all readable, when therapy sessions are planned also. All of these points relate to data collection activities: I presume you’ll be analysing the data too, to fulfil the aims? Key analytical strategies could be included here.

RESPONSE: As part of the therapy intervention component, there are module booklets, which are the readable element that would be assessed here; we have clarified this in the background section of the plain English summary where we describe the intervention components, and we have included this information in Table 1, where we provide a full description of the intervention components.

We have not included the analysis strategies here as this is the plain English summary with a small word limit, and we have also not included analytic strategies in the scientific abstract. We are using a range of different analysis approaches for the different aims and quantitative and qualitative data in this process evaluation, and therefore we feel that summarising these analytical methods within this plain English summary would add confusion more than it would provide clarity. We have described our planned analysis in detail in the main paper, under the heading “data analysis plan”.

Roseta process evaluation intro

No mention of analytical strategy/data analysis – e.g., TDF framework

RESPONSE: We have now added in a description of the TDF into the introduction in the relevant section. The other frameworks we are using (theoretical framework of acceptability, and fidelity domains framework) are already referred to in the introduction. We have not included specific details of data analysis as we feel this is more appropriate in the methods section.

Introduction: “The Theoretical Domains Framework is a helpful tool to understand barriers and facilitators to recruitment and retention. The TDF synthesises 33 theories of behaviour change into factors that could influence behaviour, for example knowledge or available resources (36).”

Protocol – Aims and objectives

How do these, new aims relate to the aims stated in the study methods? These different sets of aims need to be linked.

RESPONSE: These are not new aims; the aims in the ‘study methods’ the reviewer refers to, are the aims written in a plain English summary. In both sets of aims, (1) refers to investigating intervention fidelity, (2) refers to intervention acceptability, and (3) refers to trial experience, which are the 3 overall aims of this process evaluation. These have been listed in the same order each time to indicate that they are the same aims. In the plain English summary we have now added detail in brackets to make the link clearer to the aims in the main text.

Design

Fidelity – another introduction to the five domains – it feels like you’re going backward because the domains have been introduced already: focus this sentence on describing Table 2 in more detail.

RESPONSE: We have revised this section you mention, focusing on describing Table 2 rather than referring to the five domains.

Methods: “Fidelity will be assessed in relation to the intended design and delivery, across five domains using a mix of methods. Table 2 describes the data collection methods for each objective and intervention component, along with the time point at which the data will be collected.”

Fidelity design

Example BCTs would help the reader - possibility of a logic model to show the causal assumptions?

RESPONSE: We have now added in two example BCTs to the text, and have included our conceptual model as a figure within the paper (Figure 1), which details our proposed mechanisms of action for each intervention component. We have also referenced our intervention development paper in the introduction, which contains the full list of BCTs we have coded for each intervention component.

Introduction: “A full description of our approach to intervention development, along with detailed descriptions of the intervention components is available elsewhere (14). As part of this preparatory work, we developed a conceptual model for the intervention, detailing the mechanisms of action (Figure 1).”

Methods: “BCTs are considered ‘active ingredients’ of behaviour change interventions (e.g. problem solving, action planning).”

Fidelity of enactment

Ditto – text could focus describing Table 3 in more detail.

RESPONSE: Here, we assume the reviewer is referring to the “Intervention component acceptability” section which relates to Table 3 mentioned, rather than the fidelity of enactment section referred to above. We have kept the sentence referring to the TFA here, as it is the first time we explicitly state that we are using the TFA (we only introduce the framework in the introduction). However, we have added an additional sentence describing Table 3 as suggested by the reviewer.

Methods: “The assessment of acceptability of the intervention components will be guided by the TFA (30)(Table3). As described in Table 3, acceptability will be assessed using an acceptability questionnaire specific to each intervention component at the 4 month follow up (35), and semi-structured interviews with trial participants and therapists at the end of the trial.”

Trial experience, retention and recruitment

First mention of TDF framework – this is important because it will help the researchers understand the data and highlight areas of improvement

RESPONSE: We have now introduced the TDF earlier in the manuscript, in the introduction, in line with the other theoretical frameworks we are using in this work (theoretical framework of acceptability, and five domains of fidelity).

Qualitative interviews

Why will the interviewer be taking notes during the interview? This risks obstructing the flow of the interview. The interview will be recorded. I suggest the interviewer write up the interview immediately after the interview, using the recording as a guide; then check the summary against the transcript, once the recording is transcribed.

RESPONSE: The interviewer will be taking notes during the interviews as this is part of the recommended process in rapid approaches. We have previously used this method and did not find it disrupted the interviews. The note taking will not be extensive and we will make every effort to ensure it does not disrupt the flow of the interview. Using the recording as a guide to write up the interview would take a substantial period of time to re listen to the interview, and would not fit with the rapid analysis approach that we are taking.

NIHR Open Res. 2023 Feb 14. doi: 10.3310/nihropenres.14463.r28987

Reviewer response for version 1

Molly Byrne ¹

This is an excellent paper which I enjoyed reading. The team are commended for this ambitious and important programme of research. They are addressing a clinically important question. The paper is well-written and describes a novel trial design which has the potential to address important questions about active ingredients of complex interventions.

I have a few suggestions which I believe would improve the proposal:

Introduction: There is a lack of discussion and information about what we know already in relation to improving medication adherence. The research is presented in a bit of a vacuum, and would be strengthened by including a little more detail summarising what is already known, on which this research is building.
Introduction: The team are employing a novel trial design, using MOST, or a pilot trial with a fractional factorial design. This may be new to many readers. It would be interesting and helpful to readers if the authors could include some comment of where a MOST trial sits in the intervention development and evaluation process. What would the next steps of this study be? An full scale optimisation trial? What would the aims of this trial be? Where is this work going? Some discussion of these important questions would really strengthen this paper and help move along discussions about the place of novel trial designs like these in the development/evaluation/implementation pipeline.
One of the key steps in an optimisation trial is developing a conceptual model of the intervention, detailing the components and proposed mechanisms of action. Did the team develop a conceptual model for this intervention? It would be helpful to readers to see that.
Related to point 3., there is very little in the paper about the intervention components and their behaviour change techniques. Can the authors include any more detail about the intervention components? Is there any theoretical rationale for the intervention? This discussion is very limited in the introduction.
The proposed timeframe for follow up (4 months) seems short. Is there a clinical rationale for selecting this timeframe? Would this be the proposed timeframe in a future optimisation trial?
There is not mention of economic considerations or an economic evaluation of the interventions. Is this being included in the work? The intervention components appear very variable in terms of cost, which is an important consideration in the evaluation of the interventions. Are there plans to consider this aspect in furfure research?
Discussion: Can the authors clarify the ‘progression criteria’, which referred to in the Discussion section.

Is the study design appropriate for the research question?

Yes

Is the rationale for, and objectives of, the study clearly described?

Yes

Are sufficient details of the methods provided to allow replication by others?

Yes

Are the datasets clearly presented in a useable and accessible format?

Not applicable

Reviewer Expertise:

Behaviour science; Health behaviour change research

NIHR Open Res. 2023 Apr 13.

Sophie Green ¹

Thank you very much for taking the time to review our work and for your helpful comments to improve the manuscript.

Introduction: There is a lack of discussion and information about what we know already in relation to improving medication adherence. The research is presented in a bit of a vacuum, and would be strengthened by including a little more detail summarising what is already known, on which this research is building.

RESPONSE: We have added an additional paragraph in the introduction (2 ^nd paragraph) to explain that interventions within this field are largely ineffective. We have additionally explained the limitations of randomised controlled trials that are typically used to evaluate these interventions, as we are building upon this through using the multiphase optimisation strategy within our research.

Introduction: “There is a lack of evidence for effective interventions to support medication adherence to AET, with most interventions focusing solely on written educational components, and not targeting the range of barriers to adherence (46-48). Moreover, these interventions are typically evaluated using parallel groups randomised controlled trials (RCTs). RCTs are able to evaluate whether an intervention package is more effective than a comparator, but they are unable to estimate the contributions of individual components, or the interactions between components. As such, intervention packages demonstrating a statistically significant effect in an RCT could contain ineffective or redundant components, reducing the efficiency of the overall intervention package (16, 49).

Introduction: The team are employing a novel trial design, using MOST, or a pilot trial with a fractional factorial design. This may be new to many readers. It would be interesting and helpful to readers if the authors could include some comment of where a MOST trial sits in the intervention development and evaluation process. What would the next steps of this study be? An full scale optimisation trial? What would the aims of this trial be? Where is this work going? Some discussion of these important questions would really strengthen this paper and help move along discussions about the place of novel trial designs like these in the development/evaluation/implementation pipeline.

RESPONSE: We have added in additional explanation of the MOST framework, and the three main stages, to demonstrate where optimisation sits within the development and evaluation process. We have also made it clearer that this work falls within the preparation phase of MOST. We have now added additional detail regarding the next stages of this work in the discussion section, as that is where we felt it was most relevant. This includes an explanation of our progression criteria to move to the larger optimisation trial, and what the aim of the optimisation trial would be.

Introduction: “The Refining and Optimising a behavioural intervention to Support Endocrine Therapy Adherence (ROSETA) programme aims to develop and optimise an intervention package to support adherence to AET in women with early-stage breast cancer. The ROSETA programme is guided by the Multiphase Optimisation Strategy (MOST), which is a framework used to optimise multicomponent behavioural interventions (15-17). MOST consists of three stages; (1) preparation, in which intervention components and a conceptual model detailing proposed mechanisms of action are developed, and any pilot testing is carried out, (2) optimisation, in which highly efficient experimental designs, such as factorial designs, are used to estimate the main effects and interaction effects of individual intervention components to build an optimal intervention package, and (3) evaluation, in which the optimised intervention package is compared with a comparator, typically using a parallel groups RCT (15-17).

During the preparation phase of MOST, we combined the intervention mapping framework with MOST to develop four theory-based intervention components targeting distinct, unintentional and intentional barriers to AET adherence: (a) SMS messages to target forgetfulness; (b) information leaflet to target medication beliefs; (c) Acceptance and Commitment Therapy (ACT) guided self-help programme to increase psychological flexibility and reduce psychological distress; (d) side-effect self-management website to target AET side effects (14)(Table1). As part of this preparatory work, we developed a conceptual model for the intervention, detailing the mechanisms of action (Figure 1). A full description of our approach to intervention development, along with detailed descriptions of the intervention components is available elsewhere (14).”

Discussion: “The results of this process evaluation will be used to adapt and improve the intervention components where necessary, should progression criteria to a larger optimisation trial be met. Progression criteria to progress to a full optimisation trial are based on predefined criteria for consent rates, intervention component adherence, and availability of outcome data. These are described in detail elsewhere (45){Smith, 2023 #560}. The full optimisation trial would aim to identify the most effective combination of intervention components to support medication adherence in women with early stage breast cancer, without exceeding a pre-specified cost of £3397 per patient, based on health economic modelling (14).”

One of the key steps in an optimisation trial is developing a conceptual model of the intervention, detailing the components and proposed mechanisms of action. Did the team develop a conceptual model for this intervention? It would be helpful to readers to see that.

RESPONSE: We did develop a conceptual model for the intervention, which we have published elsewhere and have referenced in the current paper (Green et al., 2022, BMC Health Services Research). We have now included our conceptual model as a figure in the present paper (Figure 1), and have referred readers to this in the introduction.

Related to point 3., there is very little in the paper about the intervention components and their behaviour change techniques. Can the authors include any more detail about the intervention components? Is there any theoretical rationale for the intervention? This discussion is very limited in the introduction.

RESPONSE: We have now made it clear that the intervention components were developed based on psychological theory in the text, and we have included details of this in Table 1, alongside a description of the intervention components. We undertook an extensive intervention development process, which has been published elsewhere, and we have now made this more explicit to the reader here. Included in this previous publication is a full list of BCTs that we coded in each of our intervention components.

Introduction: “During the preparation phase of MOST, we combined the intervention mapping framework with MOST to develop four theory-based intervention components targeting distinct, unintentional and intentional barriers to AET adherence: (a) SMS messages to target forgetfulness; (b) information leaflet to target medication beliefs; (c) Acceptance and Commitment Therapy (ACT) guided self-help programme to increase psychological flexibility and reduce psychological distress; (d) side-effect self-management website to target AET side effects (14)(Table1). As part of this preparatory work, we developed a conceptual model for the intervention, detailing the mechanisms of action (Figure 1). A full description of our approach to intervention development, along with detailed descriptions of the intervention components is available elsewhere (14).”

The proposed timeframe for follow up (4 months) seems short. Is there a clinical rationale for selecting this timeframe? Would this be the proposed timeframe in a future optimisation trial?

RESPONSE: A short follow up was chosen in this pilot trial for a number of reasons. Firstly, retention was not included as part of the progression criteria to a full optimisation trial, and therefore we did not need to have a long follow up to assess this. The progression criteria (based on consent rates, intervention component adherence and availability of outcome data) are all based on short term outcomes, so a longer term follow up was not required. Finally, the funding structure for this work means that intervention development, a pilot trial and a full optimisation trial with a 12 month follow up are taking place within 6 years; therefore a shorter follow up period in the pilot trial felt appropriate to allow for this body of work to be completed within the timeframe.

There is not mention of economic considerations or an economic evaluation of the interventions. Is this being included in the work? The intervention components appear very variable in terms of cost, which is an important consideration in the evaluation of the interventions. Are there plans to consider this aspect in furfure research?

RESPONSE: We are planning an economic analysis in the larger optimisation trial, for which we will use self-reported data, alongside NHS reference costs and costs from the Personal Social Services Research Unit, who estimate unit costs for healthcare professionals and services in the NHS. The economic analysis will form part of the main trial paper, as it is not considered part of the process evaluation, and therefore we have not included it within this protocol. However, we have now included our optimisation objective for the full optimisation trial in the discussion, which highlights that we are considering cost in the optimisation of the intervention package.

Discussion: “The full optimisation trial would aim to identify the most effective combination of intervention components to support medication adherence in women with early stage breast cancer, without exceeding a pre-specified cost of £3397 per patient, based on health economic modelling (14).”

Discussion: Can the authors clarify the ‘progression criteria’, which referred to in the Discussion section.

RESPONSE: We have now added in a description of the progression criteria for this pilot trial, along with a reference to the trial protocol where progression criteria are described in greater detail.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Citations

Green SMC, Smith S: ROSETA Pilot Trial Process Evaluation. OSF. [Dataset].2022. 10.17605/OSF.IO/8DWRN [DOI]

Data Availability Statement

Underlying data

No data are associated with this article.

Extended data

Open Science Framework: ROSETA Pilot Trial Process Evaluation. https://doi.org/10.17605/OSF.IO/8DWRN ⁴⁷.

The project contains the following extended data:

Participant Interview Guide
Therapist Interview Guide

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

[ref-1] 1. Cancer Research UK: Breast Cancer Statistics. Reference Source [Google Scholar]

[ref-2] 2. Early Breast Cancer Trialists Collaborative Group. C Davies, J Godwin: Relevance of breast cancer hormone receptors and other factors to the efficacy of adjuvant tamoxifen: patient-level meta-analysis of randomised trials. Lancet. 2011;378(9793):771–84. 10.1016/S0140-6736(11)60993-8 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-3] 3. Early breast Cancer Trialists Collaborative Group: Aromatase inhibitors versus tamoxifen in early breast cancer: patient-level meta-analysis of the randomised trials. Lancet. 2015;386(10001):1341–52. 10.1016/S0140-6736(15)61074-1 [DOI] [PubMed] [Google Scholar]

[ref-4] 4. Hershman DL, Kushi LH, Shao T, et al. : Early discontinuation and nonadherence to adjuvant hormonal therapy in a cohort of 8,769 early-stage breast cancer patients. J Clin Oncol. 2010;28(27):4120–8. 10.1200/JCO.2009.25.9655 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-5] 5. Murphy CC, Bartholomew LK, Carpentier MY, et al. : Adherence to adjuvant hormonal therapy among breast cancer survivors in clinical practice: a systematic review. Breast Cancer Res Treat. 2012;134(2):459–78. 10.1007/s10549-012-2114-5 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-6] 6. Huiart L, Ferdynus C, Giorgi R: A meta-regression analysis of the available data on adherence to adjuvant hormonal therapy in breast cancer: summarizing the data for clinicians. Breast Cancer Res Treat. 2013;138(1):325–8. 10.1007/s10549-013-2422-4 [DOI] [PubMed] [Google Scholar]

[ref-7] 7. Makubate B, Donnan PT, Dewar JA, et al. : Cohort study of adherence to adjuvant endocrine therapy, breast cancer recurrence and mortality. Br J Cancer. 2013;108(7):1515–24. 10.1038/bjc.2013.116 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-8] 8. McCowan C, Wang S, Thompson AM, et al. : The value of high adherence to tamoxifen in women with breast cancer: a community-based cohort study. Br J Cancer. 2013;109(5):1172–80. 10.1038/bjc.2013.464 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-9] 9. Inotai A, Ágh T, Maris R, et al. : Systematic review of real-world studies evaluating the impact of medication non-adherence to endocrine therapies on hard clinical endpoints in patients with non-metastatic breast cancer. Cancer Treat Rev. 2021;100:102264. 10.1016/j.ctrv.2021.102264 [DOI] [PubMed] [Google Scholar]

[ref-10] 10. Moon Z, Moss-Morris R, Hunter MS, et al. : Understanding tamoxifen adherence in women with breast cancer: A qualitative study. Br J Health Psychol. 2017;22(4):978–97. 10.1111/bjhp.12266 [DOI] [PubMed] [Google Scholar]

[ref-11] 11. Cahir C, Guinan E, Dombrowski SU, et al. : Identifying the determinants of adjuvant hormonal therapy medication taking behaviour in women with stages I–III breast cancer: A systematic review and meta-analysis. Patient Educ Couns. 2015;98(12):1524–39. 10.1016/j.pec.2015.05.013 [DOI] [PubMed] [Google Scholar]

[ref-12] 12. Brett J, Fenlon D, Boulton M, et al. : Factors associated with intentional and unintentional non-adherence to adjuvant endocrine therapy following breast cancer. Eur J Cancer Care (Engl). 2018;27(1): e12601. 10.1111/ecc.12601 [DOI] [PubMed] [Google Scholar]

[ref-13] 13. Toivonen K, Williamson T, Carlson L, et al. : Potentially Modifiable Factors Associated with Adherence to Adjuvant Endocrine Therapy among Breast Cancer Survivors: A Systematic Review. Cancers (Basel). 2020;13(1):107. 10.3390/cancers13010107 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-14] 14. Finitsis DJ, Vose BA, Mahalak JG, et al. : Interventions to promote adherence to endocrine therapy among breast cancer survivors: A meta-analysis. Psychooncology. 2019;28(2):255–63. 10.1002/pon.4959 [DOI] [PubMed] [Google Scholar]

[ref-15] 15. Heiney SP, Parker PD, Felder TM, et al. : A systematic review of interventions to improve adherence to endocrine therapy. Breast Cancer Res Treat. 2019;173(3):499–510. 10.1007/s10549-018-5012-7 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-16] 16. Hurtado-De-Mendoza A, Cabling ML, Lobo T, et al. : Behavioral Interventions to Enhance Adherence to Hormone Therapy in Breast Cancer Survivors: A Systematic Literature Review. Clin Breast Cancer. 2016;16(4):247–255.e3. 10.1016/j.clbc.2016.03.006 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-17] 17. Collins LM: Optimization of behavioral, biobehavioral, and biomedical interventions: The multiphase optimization strategy (MOST). Springer;2018. 10.1007/978-3-319-72206-1 [DOI] [Google Scholar]

[ref-18] 18. Collins LM: Conceptual Introduction to the Multiphase Optimization Strategy (MOST). Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). Cham: Springer International Publishing;2018;1–34. 10.1007/978-3-319-72206-1_1 [DOI] [Google Scholar]

[ref-19] 19. Collins LM, Kugler KC, Gwadz MV: Optimization of Multicomponent Behavioral and Biobehavioral Interventions for the Prevention and Treatment of HIV/AIDS. AIDS Behav. 2016;20 Suppl 1(0 1):197–214. 10.1007/s10461-015-1145-4 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-20] 20. Collins LM, Strayhorn JC, Vanness DJ: One view of the next decade of research on behavioral and biobehavioral approaches to cancer prevention and control: intervention optimization. Transl Behav Med. 2021;11(11):1998–2008. 10.1093/tbm/ibab087 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-21] 21. Green SMC, French DP, Graham CD, et al. : Supporting adjuvant endocrine therapy adherence in women with breast cancer: the development of a complex behavioural intervention using Intervention Mapping guided by the Multiphase Optimisation Strategy. BMC Health Serv Res. 2022;22(1): 1081. 10.1186/s12913-022-08243-4 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-22] 22. Smith SG, Green SMC, Ellison R, et al. : Refining and optimising a behavioural intervention to support endocrine therapy adherence (ROSETA) in UK women with breast cancer: protocol for a pilot fractional factorial trial. BMJ Open. 2023;13(2):e069971. 10.1136/bmjopen-2022-069971 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-23] 23. Moore GF, Audrey S, Barker M, et al. : Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. 10.1136/bmj.h1258 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-24] 24. Nair B: Clinical Trial Designs. Indian Dermatol Online J. 2019;10(2):193–201. 10.4103/idoj.IDOJ_475_18 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-25] 25. Collins LM, Dziak JJ, Kugler KC, et al. : Factorial experiments: efficient tools for evaluation of intervention components. Am J Prev Med. 2014;47(4):498–504. 10.1016/j.amepre.2014.06.021 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-26] 26. McDonald AM, Knight RC, Campbell MK, et al. : What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7(1):9. 10.1186/1745-6215-7-9 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-27] 27. Walters SJ, Bonacho Dos Anjos Henriques-Cadby I, Bortolami O, et al. : Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open. 2017;7(3):e015276. 10.1136/bmjopen-2016-015276 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-28] 28. Cane J, O’Connor D, Michie S: Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7(1):37. 10.1186/1748-5908-7-37 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-29] 29. Carroll C, Patterson M, Wood S, et al. : A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40. 10.1186/1748-5908-2-40 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-30] 30. Lorencatto F, Gould NJ, McIntyre SA, et al. : A multidimensional approach to assessing intervention fidelity in a process evaluation of audit and feedback interventions to reduce unnecessary blood transfusions: a study protocol. Implement Sci. 2016;11(1):163. 10.1186/s13012-016-0528-x [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-31] 31. Toomey E, Hardeman W, Hankonen N, et al. : Focusing on fidelity: narrative review and recommendations for improving intervention fidelity within trials of health behaviour change interventions. Health Psychol Behav Med. 2020;8(1):132–51. 10.1080/21642850.2020.1738935 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-32] 32. Bellg AJ, Borrelli B, Resnick B, et al. : Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23(5):443–51. 10.1037/0278-6133.23.5.443 [DOI] [PubMed] [Google Scholar]

[ref-33] 33. Resnick B, Bellg AJ, Borrelli B, et al. : Examples of implementation and evaluation of treatment fidelity in the BCC studies: where we are and where we need to go. Ann Behav Med. 2005;29 Suppl(2):46–54. 10.1207/s15324796abm2902s_8 [DOI] [PubMed] [Google Scholar]

[ref-34] 34. Hawkes RE, Miles LM, Bower P, et al. : Assessing and ensuring fidelity of the nationally implemented English NHS diabetes prevention programme: lessons learned for the implementation of large-scale behaviour change programmes. Health Psychol Behav Med. 2022;10(1):498–513. 10.1080/21642850.2022.2077205 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-35] 35. Skivington K, Matthews L, Simpson SA, et al. : A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374:n2061. 10.1136/bmj.n2061 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-36] 36. Sekhon M, Cartwright M, Francis JJ: Acceptability of healthcare interventions: an overview of reviews and development of a theoretical framework. BMC Health Serv Res. 2017;17(1):88. 10.1186/s12913-017-2031-8 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-37] 37. Lambert JD, Elliott LR, Taylor AH, et al. : A novel method for assessing design fidelity in web-based behavioral interventions. Health Psychol. 2021;40(3):217–25. 10.1037/hea0001046 [DOI] [PubMed] [Google Scholar]

[ref-38] 38. Michie S, Richardson M, Johnston M, et al. : The Behavior Change Technique Taxonomy (v1) of 93 Hierarchically Clustered Techniques: Building an International Consensus for the Reporting of Behavior Change Interventions. Ann Behav Med. 2013;46(1):81–95. 10.1007/s12160-013-9486-6 [DOI] [PubMed] [Google Scholar]

[ref-39] 39. Gwet K: Kappa statistic is not satisfactory for assessing the extent of agreement between raters. Statistical methods for inter-rater reliability assessment.2002;1(6):1–6. Reference Source [Google Scholar]

[ref-40] 40. O'Neill L, Latchford G, McCracken LM, et al. : The development of the Acceptance and Commitment Therapy Fidelity Measure (ACT-FM): A delphi study and field test. J Contextual Behav Sci. 2019;14:111–8. 10.1016/j.jcbs.2019.08.008 [DOI] [Google Scholar]

[ref-41] 41. Sekhon M, Cartwright M, Francis JJ: Development of a theory-informed questionnaire to assess the acceptability of healthcare interventions. BMC Health Serv Res. 2022;22(1): 279. 10.1186/s12913-022-07577-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-42] 42. Malterud K, Siersma VD, Guassora AD: Sample Size in Qualitative Interview Studies: Guided by Information Power. Qual Health Res. 2016;26(13):1753–60. 10.1177/1049732315617444 [DOI] [PubMed] [Google Scholar]

[ref-43] 43. Smith SG, Ellison R, Hall L, et al. : Acceptance and Commitment Therapy to support medication decision-making and quality of life in women with breast cancer: protocol for a pilot randomised controlled trial. Pilot Feasibility Stud. 2022;8(1):33. 10.1186/s40814-022-00985-6 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-44] 44. Greene A, Elmer M, Ludlam S, et al. : Evaluation of the Content Validity and Cross-Cultural Validity of the Study Participant Feedback Questionnaire (SPFQ). Ther Innov Regul Sci. 2020;54(6):1522–33. 10.1007/s43441-020-00179-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-45] 45. Vindrola-Padros C, Johnson GA: Rapid Techniques in Qualitative Research: A Critical Review of the Literature. Qual Health Res. 2020;30(10):1596–604. 10.1177/1049732320921835 [DOI] [PubMed] [Google Scholar]

[ref-46] 46. Vindrola-Padros C, Chisnall G, Polanco N, et al. : Iterative cycles in qualitative research: Introducing the RREAL Sheet as an innovative process.2022. 10.31219/osf.io/9dp2w [DOI] [Google Scholar]

[ref-47] 47. Green SMC, Smith S: ROSETA Pilot Trial Process Evaluation. OSF. [Dataset].2022. 10.17605/OSF.IO/8DWRN [DOI]

[ref-48] 48. Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74. 10.2307/2529310 [DOI] [PubMed] [Google Scholar]

[ref-49] 49. Vindrola-Padros C, Brage E, Johnson GA: Rapid, Responsive, and Relevant?: A Systematic Review of Rapid Evaluations in Health Care. Am J Eval. 2021;42(1):13–27. 10.1177/1098214019886914 [DOI] [Google Scholar]

PERMALINK

Acceptability, fidelity and trial experience of four intervention components to support medication adherence in women with breast cancer: A process evaluation protocol for a pilot fractional factorial trial

Sophie MC Green

Louise H Hall

Nikki Rousseau

David P French

Christopher D Graham

Michelle Collinson

Ellen Mason

Hollie Wilkes

Daniel Howdon

Robbie Foy

Rebecca Walwyn

Jane Clark

Catherine Parbutt

Erin Raine

Rachel Ellison

Jacqueline Buxton

Sally J L Moore

Galina Velikova

Amanda Farrin

Samuel G Smith

Roles

Version Changes

Revised. Amendments from Version 1

Abstract

Background

Methods

Conclusions

Registration

Plain English summary

Introduction

The ROSETA programme

Figure 1. Conceptual model for the ROSETA intervention.

Table 1. Summary of intervention components in the ROSETA pilot trial.

ROSETA pilot trial process evaluation

Protocol

Aims and objectives

Design

Table 2. Summary of fidelity assessments.

Table 3. Summary of acceptability assessments.

Table 4. Summary of trial experience, recruitment and retention assessments.

Sampling and recruitment

Table 5. Eligibility criteria for participation in the ROSETA pilot trial.

Assessment measures

Qualitative interviews

Data analysis plan

Ethics and dissemination

Trial status

Discussion

Acknowledgments

Funding Statement

Data availability

Underlying data

Extended data

References

Reviewer response for version 2

Molly Byrne

Roles

Reviewer response for version 2

Thomas Mills

Roles

Reviewer response for version 1

Thomas Mills

Roles

Sophie Green

Reviewer response for version 1

Molly Byrne

Roles

Sophie Green

Associated Data

Data Citations

Data Availability Statement

Underlying data

Extended data

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles