Acceptability and feasibility of a need-supportive intervention to increase trial retention: a randomised feasibility study within a randomised controlled allergy trial in Denmark

Abstract

Introduction

Given that low retention rates are a prevalent challenge in clinical trials, which ultimately affects trial validity, it is recommended that interventions be developed and evaluated to increase trial retention. In the context of trial retention, incorporating behavioural science is endorsed, as it provides a theoretical foundation for considering human behaviour. We hypothesised that an intervention informed by self-determination theory could increase retention in a randomised allergy trial on intralymphatic immunotherapy, as the support of basic psychological needs for autonomy, competence and relatedness is anticipated to lead to more sustained engagement and better outcomes.

Objective

To assess the acceptability and feasibility of the intervention and evaluation design, following the complex intervention framework by the Medical Research Council, before proceeding to a randomised evaluation.

Design

A parallel two-arm randomised feasibility study was conducted within the randomised allergy trial.

Participants

All participants at one Danish site were eligible for recruitment.

Intervention

The intervention was a web app informed by self-determination theory to support the basic psychological needs through its thoughtfully designed features. Participants were allocated unblinded across treatment groups to complete daily online questionnaires over a 100-day period from May to August 2022. All participants received a daily text message with a link for the questionnaires. On completion, participants in the control group received a confirmation message, while participants in the intervention group had a browser with the menu of the web app opened for them. The features within the menu were voluntary to use.

Outcome measures

The prespecified assessments included evaluating the recruitment rate, retention rate (which reflected both sustained participation and the proportion of completed daily questionnaire entries), the suitability of outcome measures and the acceptability of the intervention and evaluation design to both participants and staff. Qualitative data were collected through a collaborative learning process with participants from the intervention group in November 2022.

Results

A total of 30 participants were invited, randomly assigned 1:1 and analysed, resulting in a recruitment rate of 100%. None were lost to follow-up as all remained in the study for the entire duration. The response rate was 84.5% in the intervention group and 79.1% in the control group, indicating satisfactory retention. Outcome measures were deemed appropriate. No unintended adverse events were identified. The collaborative learning meetings involved three participants in the first meeting and two in the second, comprising a total of five different individuals. Participants found the intervention acceptable. They used it differently but agreed that its components were useful. Technical issues needed fixing, and voluntary free text boxes and registration of medication dosage should be added.

Conclusions

The intervention and evaluation design were assessed as acceptable and feasible. Technical issues were fixed, and additional response options were added before a randomised evaluation.

Trial registration number

ILIT.NU: EudraCT 2020-001060-28. ClinicalTrials.gov NCT05191186.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• Involving patients in the research process provided deeper insights and led to valuable additions to the intervention, helping ensure alignment with the construct being measured.

• Using the Medical Research Council framework for complex interventions facilitated a systematic and staged approach to translating theory into a contextually relevant and operational intervention.

• Basing the intervention on self-determination theory supported an ethically sensitive approach to trial retention by balancing participants’ right to withdraw with efforts to promote continued engagement.

• Greater diversity among patient contributors might have enhanced the relevance and transferability of the intervention.

• Analysing which components of the intervention participants engaged with would have provided valuable insights into patterns of use and variation in engagement across participants.

Introduction

Retention refers to participants actively taking part in a trial after recruitment and random assignment, indicating whether they remain in the trial and provide outcome data.¹ Retention is compromised, and so is the validity of a trial, when participants fail to provide outcome data.² Indeed, bias may be introduced already at a 95% retention rate.³ Compromised validity may lead to research waste, including the loss of valuable patient time, misallocation of financial resources and an increased risk of drawing incorrect conclusions and recommendations.⁴ A systematic review by Walters and colleagues suggested that the median retention rate was 89% among 151 randomised controlled trials.⁵ This underscores the relevance of focusing on retention, which is further underpinned by its recognition as one of the top three priorities on the methodological research agenda in the UK.⁶

Retention was a research focus in a 3-year trial entitled ILIT.NU (EudraCT 2020-006020-28). The ILIT.NU trial evaluated the efficacy of intralymphatic immunotherapy (ILIT) among participants with grass-pollen-induced allergy. The condition is a seasonal public health issue affecting over 20% of the global population, and its prevalence continues to rise.^{7 8} The primary manifestations of this condition include nasal and ocular symptoms.⁹ Such symptoms can diminish quality of life, as they interfere with sleep, hinder cognitive performance and restrict involvement in recreational pursuits.^{10 11} As a result, individuals may find their ability to live as they desire severely constrained during times of allergen exposure.^{10 11} Adherence to conventional treatment for grass-pollen-induced allergy has been low, whereas ILIT appears more appealing due to its markedly shortened treatment duration and promising clinical efficacy.¹²

Participants in the ILIT.NU trial were treated three times over 8 weeks with either verum or placebo. Furthermore, they attended two annual consultations and completed daily questionnaires about allergy symptoms, their use of rescue medication and health-related quality of life throughout the grass pollen seasons of 2021–2023. Given the time required for daily outcome reporting and clinical attendance, participants were anticipated to exhibit a significant dedication to trial activities. Nonetheless, trials investigating allergen immunotherapy have faced challenges with participant commitment, leading to withdrawals and incomplete outcome data.13,17 This indicates that retention is a problem within the field of allergy as well. In response, a retention intervention was developed to increase retention in the ILIT.NU trial.

The retention intervention was based on self-determination theory, recognising that participation in trial activities entails specific behavioural patterns. Effectively supporting ongoing behaviours requires that factors influencing trial participation are considered, enabling accommodation and encouragement for the continuation of sustained trial engagement.¹⁸ Self-determination theory is promising as it views motivation on a continuum. When actions stem from autonomous motivation, supported by fulfilling the basic psychological needs for autonomy, competence and relatedness, sustained engagement is more likely. This enhances individual effort and positively impacts the quality of outcomes achieved.^{19 20}

The retention intervention was developed in collaboration with patient partners. The development process drew on the complex intervention framework by the Medical Research Council (MRC)^{21 22} and is described in detail elsewhere (manuscript under review). The retention intervention was intended to support basic psychological needs for autonomy, competence and relatedness through its features to enhance response rate and clinical attendance in the ILIT.NU trial.

Evaluating retention interventions is essential to increase the evidence base for designing and conducting randomised trials according to the Trial Forge Guidance.²³ Hence, the intervention was planned to be evaluated in a randomised study within the ILIT.NU trial. Before advancing to a large-scale evaluation, the complex interventions framework recommends conducting a comprehensive feasibility assessment.²¹ This involves assessing the feasibility and acceptability of the intervention and evaluation design, using qualitative and quantitative methods to identify key uncertainties.²¹ Conducting this preparatory feasibility work is important to avoid compromising the validity of a future definitive trial by intervention or evaluation design issues.²¹

Methods

This feasibility study aimed to assess the acceptability and feasibility of a need-supportive retention intervention and its evaluation design within the ILIT.NU trial to pave the way for the evaluation phase. In particular, the study aimed to determine the acceptability of the intervention and the feasibility of recruitment, data collection, outcome measure completion and randomisation procedures.

Study design

A feasibility study within the ILIT.NU trial following the MRC framework²¹ was conducted using an open two-arm parallel randomised design. To improve the quality and transparency of the reporting of this randomised feasibility study, the ‘Consolidated Standards of Reporting Trials (CONSORT) 2010 statement: extension to randomised pilot and feasibility trials’ were applied.²⁴ The completed guideline is available in a online supplemental file S1.

Participants

To address practical and time constraints, eligibility for inclusion was limited to participants enrolled in the ILIT.NU trial at one of the Danish sites. Specifically, all participants who attended the April 2022 third treatment consultation on four randomly selected days were considered eligible.

Recruitment

The target enrolment was set at 30 participants to ensure a sufficient sample size for assessing acceptability and feasibility while balancing ethical considerations to avoid enrolling too many participants unnecessarily. Since this was a feasibility study, no formal sample size calculation was needed.²⁴ Participants were initially approached by a study nurse; if interested, they were subsequently recruited by the first author. Recruitment ceased once 30 participants had been enrolled.

Study procedures

Participants who consented to participate were randomly assigned without any restrictions in a 1:1 ratio across treatment groups. An independent consultant set up the web-based platform REDCap for simple randomisation.²⁵ To ensure allocation concealment, no assessors had access to the allocation code. After the assignment, participants and health professionals were aware of the group assignment. To reduce the risk of biased results, assessors remained unaware of the allocation. No changes to methods were made after trial commencement.

Intervention

To enable replication of the intervention, item five in the CONSORT reporting guideline was unfolded using the ‘Template for intervention description and replication’.²⁶ The completed guideline is available in a online supplemental file S1. During the grass pollen season in 2022, both the intervention and the control groups received a daily text message at 20:00 hours with links to questionnaires that were open for 48 hours. The Combined Symptom Medication Score (CSMS)²⁷ should be completed daily, and the Rhinoconjunctivitis Quality of Life Questionnaire (RQLQ)²⁸ should be completed once a week. REDCap provided the questionnaires. After completing the questionnaires, participants in the control group received a confirmation message. In contrast, participants in the intervention group had a browser with the front page of the intervention opened for them. Using the intervention was optional, and the participants were not instructed on how to use it.

The intervention was a prototype of a novel need-supportive web app informed by self-determination theory.²⁹ The features were intended to support basic psychological needs for autonomy, competence and relatedness. The intervention was developed in collaboration with patient partners from the ILIT.NU trial and was inspired by findings from Peters et al.³⁰ The intervention had the following features: (a) the possibility to complete a shorter questionnaire on days without symptoms, (b) the option to answer up to 48 hours later and visualisation of missed questionnaires, (c) positive feedback on questionnaire completion, (d) pollen counts and forecasts, (e) personal and total response rate, (f) visual representation of entered data, (g) advice about grass pollen allergy, (h) updates on the ILIT.NU trial, (i) ability to change the time of the daily short message service, (j) guide on downloading an icon for the home screen and (k) contact information. Features supporting autonomy included offering choices, integrating trial activities into daily routines and minimising the time needed to complete these activities. Features supporting competence focused on ensuring the web app was easy to use and accessible, along with providing tools to improve knowledge and skills related to managing grass pollen-induced allergy. Features supporting relatedness highlighted establishing a connection to the ILIT.NU trial, reminding participants that their efforts in completing trial activities were valued to ensure the trial’s success. A mock-up of the front page of the web app is presented in figure 1.

Figure 1. Mock-up of the front page of the web app.

Outcomes and analysis

Table 1 illustrates the central aspects, the corresponding research questions and the methods used to address each research question regarding the feasibility and acceptability of the intervention and the evaluation design. The research questions address the key uncertainties identified during the development phase and in the literature. StataSE V.18 for Mac was used for the statistical analyses.³¹

Table 1. Central aspects, research questions and assessment methods regarding the feasibility and acceptability of the intervention and the evaluation design adapted from Holmberg and colleagues⁴¹.

Central aspects	Research questions	Methods
Intervention design
Acceptability	Is the intervention well-received by participants, clinicians and researchers? Does the intervention require refinement?	Assessed during collaborative learning meetings, where agreements were reached32
Reach	Does the intervention reach those with the greatest need for it—those with a low response rate?	Assessed using descriptive statistics
Mechanisms of change	Do the mechanisms of change appear to work as intended—increasing retention?	Assessed during collaborative learning meetings
Unintended effects	Are there any unintended effects? Does the intervention lead to flawed data for the ILIT.NU trial? for example, the shortened questionnaire on days without symptoms Other unintended effects?	Assessed during collaborative learning meetings and by using descriptive statistics
Participant responsiveness	Do the participants actively engage with the intervention and its features?	Assessed during collaborative learning meetings
Software bugs	Are there any software-related issues?	Assessed during collaborative learning meetings and by using descriptive statistics
Evaluation design
Efficacy signals	Do the initial descriptive analyses suggest that there are changes and differences between the intervention and control groups on the following outcomes: Response rate Combined Symptom Medication Score Rhinoconjunctivitis Quality of Life Questionnaire Loss to follow-up IMI scores What is the power of a future definitive trial to detect an expected difference in response rate?	Assessed using descriptive statistics; by applying the Wilcoxon rank sum test for non-parametric comparisons as the assumption of normality was not met; and by performing a power calculation.42 43 Outliers were included in the dataset. A significance level of 5% was employed for the analysis. In the future definitive trial, the sample size could be a maximum of 286, as this was the number of participants still retained in the ILIT.NU trial at that time. Data from the feasibility study was used to compute the effect estimate (difference in response rate) and the intra-class correlation. This is to determine the future sample’s power to detect the expected difference in the response rate.
Recruitment	Is the recruitment target met within the specified timeframe? Is it possible to recruit a diverse study population?	Assessed using descriptive statistics
Retention	Do the participants provide outcome data? Are the questionnaires completed? Are there any losses to follow-up?	Assessed using descriptive statistics
Methods for data collection	The Intrinsic Motivation Inventory: Do the participants complete it? Is it the right scales and items? Is the number of scales and items appropriate? Is there any floor or ceiling effects?	Assessed using descriptive statistics
Randomisation	Is the randomisation procedure feasible?

The prespecified stopping guidelines included provisions for discontinuation if the intervention led to flawed data collection for the ILIT.NU trial or if it influenced the outcomes of the ILIT.NU trial following guidance from the Trial Forge.²³ These stopping guidelines were assessed under the central aspect, ‘unintended effects.’

Collaborative learning meetings

Online supplemental file S2 provides the interview guide for the collaborative learning meetings.³² All participants from the intervention group were invited by telephone. The last author, an experienced qualitative researcher, conducted the meetings online using Zoom. The screen-sharing feature was used to demonstrate the features of the web app. During both sessions, participants and the last author discussed each research question together to reach a consensus.³² The collaborative learning approach fostered a shared understanding and revealed diverse perspectives, supporting a more indepth understanding of ways to support autonomous motivation for participation in the ILIT.NU trial. The agreements reached in the first meeting were revisited and further refined during the second meeting to ensure consensus among participants. As mutual understanding was established through this iterative process, the final agreements were not subjected to additional analysis.

Patient and public involvement in research

The short form of the ‘Guidance for reporting involvement of patients and the public, version 2’ (GRIPP2-SF)³³ was used to describe how patients were involved in the research process. The completed guideline is available in the online supplemental file S1.

Patients were involved in the research process as this could potentially improve trial design and delivery.³⁴ They participated at two levels. First, five participants from the ILIT.NU trial took part in the collaborative learning meetings. Second, five patients with grass pollen-induced allergy who were involved in developing the web app also participated in this feasibility study. These five patients individually discussed the results from the collaborative learning meetings with the first author, especially on how to ensure a satisfying user experience. While the researchers made the final decisions, they were heavily influenced by these patient partners’ preferences. Without involving patients in the research process, it would not have been possible to assess the acceptability of the intervention from the patient’s perspective or ensure that the changes made would continue to be acceptable. Therefore, patient involvement in this feasibility study played a crucial role in shaping the final intervention. There were no negative effects related to it.

Results

A total of 30 participants were approached, all of whom provided written consent and were randomised. The participant flow is illustrated in figure 2. No participants were lost to follow-up or excluded after randomisation during the grass pollen season. Participants were enrolled in April 2022. Data were collected during the 2022 Danish grass pollen season from May until August. Data from the collaborative learning meetings were collected in November 2022. All data were assessed during the winter of 2022/2023. The feasibility study ended at the scheduled time.

Figure 2. Consolidated Standards of Reporting Trials-inspired²⁴ flow diagram of the progress of enrolment, intervention allocation and data analysis.

Baseline demographics and clinical characteristics of the participants are presented in table 2.

Table 2. Baseline demographics and clinical characteristics.

Characteristics	Total, n=30 No. (%)	Intervention, n=15 No. (%)	Control, n=15 No. (%)	Meetings, n=5 No. (%)
Gender
Female	16 (53)	8 (53)	8 (53)	3 (60)
Age in years
18–25	12 (40)	4 (27)	8 (53)	1 (20)
26–34	10 (33)	7 (46)	3 (20)	1 (20)
35–60	8 (27)	4 (27)	4 (27)	3 (60)
Combined Symptom Medication Score
0.3–1.0	11 (37)	5 (33)	6 (40)	–
1.1–1.5	10 (33)	6 (40)	4 (27)	–
1.6–4.0	9 (30)	4 (27)	5 (33)	–

Out of the 15 invited participants, seven declined to participate in the collaborative learning meetings due to time constraints, as the ILIT.NU trial already involved numerous study-related activities. Of the eight who agreed to participate, five were scheduled for the first and three in the second. However, due to technical issues and forgetfulness, only three participated in the first, which lasted 46 min; two in the second, which lasted 42 min. Table 3 shows the results of the meetings and the remaining part of the feasibility study.

Table 3. Results regarding the acceptability and feasibility of the intervention and the evaluation design.

Central aspects	Research questions
Intervention design
Acceptability	The participants agreed that the intervention was easy to navigate and use and they found each feature useful.
	However, independently of each other, participants in both groups needed the option to elaborate on their daily responses in the Combined Symptom Medication Score (CSMS), believing it would yield more accurate data for the ILIT.NU trial. Some days, they felt that their responses were inaccurate because the influence of different factors on their responses was not considered. It concerned:
	Their behaviour and level of exposure, for example, whether they had spent much time outdoors or indoors or whether they had been travelling abroad.
	If they were unsure whether their symptoms were caused by grass pollen or other factors, such as dust mites or influenza.
	Additional symptoms caused by grass pollen that were important to them but were not included in the CSMS, such as itching of the mouth or throat.
	The number of puffs, drops and pills taken each day, as they thought it had an important impact on the severity of their symptoms.
	There were no discrepancies in the findings and agreements from the two meetings. The findings and agreements from the first were reaffirmed in the second, illustrated in the first quotation. Although not explicitly stated, the phrase ‘what the others said’ is understood to refer not only to participants in the current discussion, but also to those from the previous meeting, indicating alignment across both groups:
	“I can align myself with what the others said – it was easy to use (…). And just as the others said, I also sometimes need to be able to add some comments. I think there were some days where my symptoms, for good reasons, were non-existent (…). You know, you think that you’ve just had 3 days without symptoms – but that’s not true. I wish I could have reported that there was a reason my symptoms were extreme. That was something I missed (…). Another thing I didn't think was considered was, have you taken symptom-relieving medication? Yes, I have. How much have you actually taken? Well, there’s a big difference between just taking one pill and it goes away or having to take four.” Participant 4
	“It’s more that I feel there were some days where my symptoms, for good reason, were non-existent. Like, either it was one of those days where it was just really rainy, or I had been indoors, or I had… you know, one of those times where you think, “I've had 3 days without symptoms,” but that’s not really accurate. And I felt like there was no space to note that there were reasons why… for example, I live in the countryside. Often, that results in the pollen levels being way up… and that meant I was totally knocked out. I really wish I could have written down that there was a reason why my symptoms were extreme — that was missing. I actually called you to ask about it. I said, “I can’t really register anything correctly — well, I can log how I’m feeling — but there’s a reason for it.” I was also missing that field the others mentioned.” Participant 5
	“I found it very easy and straightforward to use. I did experience a few issues where not all parts of the system worked properly. But yes, overall it was the same – it was easy, very manageable, very simple. You quickly got used to assessing your day, like how it had been, and then it was actually quite quick to go in and go through the questions that were there. But I do think it could have gone into more detail in some places — for example, whether you had taken medication several times a day, and how many pills, because that was also something that could happen.” Participant 2
	Scoring their symptoms on a scale from 0 to 3 also posed challenges, as they found it difficult to assess what the research team meant by the different graduations. Therefore, they requested a guide to assist with this.
	Summing up, the intervention was generally well-received by both participants, researchers and clinicians, and only minor refinements were needed.
Reach	While the overall response rate on the CSMS was high in both groups, a few participants in each group had response rates below 60%. An exclusion threshold for the ILIT.NU trial. This raises uncertainty about whether the intervention reached those with the greatest need for support.
Mechanisms of change	Four out of five participants believed that using the intervention had a positive impact on their response rate; especially the ‘questionnaires to be answered’ feature, which allowed them to answer missed questionnaires up to 48 hours later. The last participant perceived themselves as very conscientious and, therefore, responded every day regardless of the intervention’s features.
Unintended effects	The intervention did not result in flawed data for the ILIT.NU trial or any other unintended effects. The entered data were correctly recorded in REDCap. When it comes to the option to only complete the entire questionnaire on days with symptoms, participants had no doubts about the questions: ‘Your use of medication in the last 24 hours—did you use symptom-relieving medication?’ and ‘Your allergy symptoms in the last 24 hours—did you have allergy symptoms?’. They argued that there was no uncertainty, especially when dealing with a chronic condition like grass pollen allergy. Additionally, they were already familiar with the questionnaire from the baseline season as they had answered it throughout the entire grass pollen season of 2021.
Participant responsiveness	The participants mainly used the ‘questionnaires to be answered’ feature, allowing them to answer missed questionnaires up to 48 hours later. They used the other features in varying ways, but they found every component useful.
Software bugs	Due to technical issues, the pollen count did not work on most days. It needed to be fixed as the participants wanted to use the feature. The remaining features functioned as intended.
Evaluation design
Efficacy signals	The initial analyses indicate notable differences between the intervention and control groups regarding the IMI scores and the response rates of the CSMS and the Rhinoconjunctivitis Quality of Life Questionnaire (RQLQ). Participants in the intervention group tended to have higher scores and response rates than those in the control group, as reflected by a greater proportion of completed daily questionnaire entries over the 100-day period.
	Response rate	Mean (min; max) (%)	Median (IQR) (%)	Intrinsic motivation	Median (IQR)
				Inventory Scale
	CSMS		p=0.587	Interest/enjoyment	p=0.008
	Intervention	84.7 (40;100)	93 (83–98)	Intervention	5.5 (4.5–5.5)
	Control	79.1 (16;100)	87 (65–96)	Control	4.0 (2.8–4.5)
	RQLQ		p=0.858	Perceived competence	p=0.047
	Intervention	76.5 (15;100)	85 (62–92)	Intervention	6.7 (6.3–7.0)
	Control	73.0 (8;100)	77 (62–92)	Control	6.0 (5.3–7.0)
				Pressure/tension	p=0.366
				Intervention	1.0 (1.0–1.0)
				Control	1.0 (1.0–1.7)
				Perceived choice	p=0.444
				Intervention	5.0 (4.3–5.7)
				Control	5.0 (4.0–5.7)
				Value/usefulness	p=0.067
				Intervention	7.0 (5.7–7.0)
				Control	5.7 (5.0–6.0)
	With an intraclass correlation of 0.2 and an effect estimate of 5.6 percentage points, the power of the future definitive trial with an expected difference in response rate of 5.6 percentage points would be 0.79, if all eligible participants consent to participate.
Recruitment	The recruitment target was met within the specified timeframe, as all 30 who were invited provided written consent and were randomised. Therefore, we considered the recruitment rate to be 100%. Both genders were represented, and the age distribution in the ILIT.NU trial was reflected.
Retention	All participants provided outcome data. However, as mentioned above, a few participants had a response rate lower than 60% on the CSMS. None were lost to follow-up during the grass pollen season, but three participants did not attend the collaborative learning meetings.
Methods for data collection	In the intervention group, nine participants (60%) completed the Intrinsic Motivation Inventory (IMI), while 11 participants (73%) in the control group completed it, which was considered acceptable. There was a tendency for a ceiling effect, especially on the pressure/tension scale items.
	Completion of the IMI was acceptable, with 60% in the intervention group and 73% in the control group. A ceiling effect was observed, particularly on the pressure/tension scale.
Randomisation	The randomisation procedure was feasible. Participants were willing to be randomised and there were no issues with using REDCap for randomisation. Researchers who were not authorised to access the randomisation key and the allocation concealment were also unable to do so.

Discussion

In this feasibility study, we assessed the acceptability and feasibility of a retention intervention before proceeding to a randomised evaluation.

Evaluation design

The randomisation procedure was successful, the recruitment rate was 100% and participants provided outcome data; the intervention group seemed to have higher IMI scores and response rates on the CSMS and the RQLQ than the control group. All these findings were positive, providing no evidence against the acceptability and feasibility of the evaluation design. However, before beginning the study, we were unaware of the need to establish predefined progression criteria, as recommended by the MRC.²¹ As a result, we lacked clear criteria to determine whether the outcomes justified moving to the evaluation phase or if additional development and feasibility work was necessary.²¹ This made it more challenging to decide whether the central aspect of ‘methods for data collection’ needed changes. The response rate for the IMI was 60% in the intervention group and 73% in the control group. The literature is not unanimous on an acceptable response rate, but most studies consider a response rate above 60% acceptable.³⁵ We also found a ceiling effect, especially on the ‘pressure/tension scale’, where many participants scored at the upper end of the scale. This could make it difficult to discriminate among the scores in the intervention and control groups.³⁶ We decided to keep the scale in the questionnaire, recognising it as the only measure negatively predicting internal motivation³⁷ despite the risk of measurement error and underreporting of true differences. Therefore, no changes were made to the evaluation design.

Intervention design

The collaborative learning meetings results indicated that the intervention was acceptable for the participants and improved their data reporting. Although each participant used the intervention in their own way, they all agreed that every feature was useful. Yet, they wanted to fix some technical issues with the pollen count and requested an option to elaborate on their answers on the CSMS. In response, we fixed the technical issues and incorporated additional voluntary ad hoc questions. This included registering medication dosages using checkboxes and two free text boxes to elaborate on their allergy symptoms and other comments. We decided to add the questions as ad hoc items at the end of the questionnaire as this may help prevent disturbing the validity of a questionnaire.³⁸ We had a particular focus on this because a study conducted within a trial should not influence the outcome of the host trial, thereby introducing bias.²³ Therefore, it was also impossible to comply with the participants’ suggestion of providing a guide on scoring their symptoms in the CSMS. Offering the guide exclusively to one group could introduce bias between the intervention and control groups. This concern led us to discuss in the collaborative learning meetings whether certain questions might affect the validity of the responses: ‘Your use of medication in the last 24 hours—did you use symptom-relieving medication?’ and ‘Your allergy symptoms in the last 24 hours—did you have allergy symptoms?’. These questions were not placed at the end of the questionnaire as recommended, and they auto-filled many of the items if the answer was no. However, the results from the collaborative learning meetings indicated that the participants were not in doubt about whether they experienced symptoms consistent with their allergy. Also, they were familiar with the questionnaire as they answered it daily for 4 months during the grass pollen season in 2021. Therefore, the ILIT.NU trial and the circumstances surrounding the CSMS are unique and not comparable to a standard research survey, where the participants typically complete a questionnaire one or two times. For this reason, we argue that adding these questions poses no risk of unintended effects on the outcome of the ILIT.NU trial outcome. In conclusion, we considered the intervention design acceptable and feasible after the technical issues were resolved and the ad hoc questions added.

Remaining uncertainty

While several aspects are relevant for investigation in a feasibility study, economic and time constraints require prioritising the most critical ones based on the literature and key uncertainties.²¹ In this study, ‘participant responsiveness’ was initially planned to be evaluated using descriptive statistics and qualitative methods. However, this aspect was only explored through the collaborative learning meetings due to time constraints. It would have been valuable to examine how participants used and interacted with the intervention, including the frequency of use for each feature. While collaborative learning findings indicated that participants did use the intervention, the extent of their engagement remains uncertain. Another uncertainty is ‘reach’, as we still do not know whether the intervention reaches those participants with the lowest response rate. Although participants in the intervention group tended to have higher response rates on the CSMS and RQLQ, some still had a response rate below 60%. Furthermore, this feasibility study does not provide insights into whether the intervention could enhance clinical attendance, which is a critical area of need, as most retention interventions primarily focus on patient-reported outcome data.²

Strengths and limitations

Involving patients in the research process had an important impact on the final intervention. Specifically, participants identified that the CSMS did not adequately capture symptoms they found most relevant in the context of grass pollen-induced allergy. Without the patient’s perspective, we might not have questioned the comprehensiveness of the CSMS, a measure that remains unvalidated,²⁷ and whether it fully covers the items most relevant for patients with grass pollen-induced allergy. This feedback led to refinements aimed at better aligning the intervention with participants’ lived experiences and priorities, which may ultimately enhance the relevance and quality of data collection in the ILIT.NU trial. However, only five participants attended the collaborative learning meetings, and three did not participate, despite having initially agreed to do so. Those who participated were highly engaged in the ILIT.NU trial and consistently had high response rates, which may have introduced a degree of selection bias. Insights from participants with lower response rates and engagement would have enhanced the depth and credibility of the findings. Therefore, it is essential to emphasise that the evaluation of participant responsiveness was based solely on input from five individuals. This represents an explicit limitation and should be considered when interpreting the recommendations for future work. Nevertheless, the development of the intervention was informed by a broader group, including 10 additional individuals with grass pollen-induced allergy, as well as researchers and health professionals. Furthermore, the intervention components were grounded in existing evidence on trial retention and behavioural theory.

Although the intervention group tended to have slightly higher response rates across outcomes, overall retention rates were similar in both groups. This may be explained by the already high baseline motivation and adherence among participants, who were part of a demanding 3-year allergy trial. Consequently, the feasibility study may not have included enough variation in engagement levels to detect a meaningful difference between groups. Furthermore, the study was not powered to assess efficacy but instead focused on acceptability and feasibility. Therefore, any observed patterns should be interpreted with caution.

Implications and perspectives

Although the intervention was designed explicitly for the ILIT.NU trial, many of the features aimed at supporting basic psychological needs to enhance retention can be adapted for future trials in various healthcare settings. The collaborative learning meetings revealed that allowing participants to respond up to 48 hours later was particularly valuable for improving response rates. An especially appreciated feature was that participants could see on the app’s front page if they still needed to complete a questionnaire. Although this method may enhance response rates, there are considerations regarding the accuracy of recall. However, evidence suggests that short recall periods, such as 48 hours, can yield valid data, particularly when reporting concrete symptoms or behaviours like medication use.³⁹ While future studies may face economic constraints in developing an app for daily data collection, allowing participants to access missed questionnaires after completion would be simple to implement. Additionally, incorporating free text boxes for ad hoc questions could be helpful in allowing participants to elaborate on their answers to secure a more complete understanding of the construct to be measured. The CSMS is recommended by the European Academy of Allergy and Clinical Immunology as the primary endpoint for trials investigating the efficacy of allergen immunotherapy.²⁷ Therefore, this finding is particularly relevant for trials utilising the CSMS as an endpoint within the field of allergy. Finally, the intervention did not seem to reach those with the lowest response rates. While the reasons remain unclear, one possibility is that the digital format created a barrier for some participants. A digital intervention is only one influence; another could lie in the interaction between trial staff and participants. Incorporating self-determination theory at the design stage of a trial may help ensure that both procedures and interpersonal aspects of a trial support participants’ psychological needs. Guidance on how to apply this in practice can be found in Teixeira et al.⁴⁰

Conclusions

Our feasibility study demonstrated that the retention intervention and its evaluation design were acceptable and feasible. After addressing technical issues and adding free text boxes to one of the questionnaires, participants found the intervention easy to use, useful and without unintended effects. Recruitment reached the target within the specified timeframe, randomisation was implemented without issues, and all participants provided outcome data. Although the intervention’s reach among participants with low response rates and its effect on clinical attendance remain uncertain, the findings overall support progression to a full randomised controlled trial embedded in the ILIT.NU trial.

Data availability statement

Data are available upon reasonable request.