Developing and evaluating complex interventions

In their manuscript ‘Methodological development of an exploratory randomised controlled trial of an early years' nutrition intervention: Cherry (Choosing Healthy Eating when Really Young)’, Watt et al. (2014) describe how they developed a nutrition intervention to be delivered in children's centres and then performed an assessment of its feasibility and acceptability using a cluster‐randomised controlled trial design. They use the principles outlined in the Medical Research Council (MRC) Framework for Developing and Evaluating Complex Interventions (Medical Research Council 2008) to underpin their intervention and the plans for a full‐scale evaluation. To develop the intervention, they used a mixed methods approach, encompassing interaction with local stakeholders, focus groups and surveys to elicit the views of parents and children's centre staff on barriers to and facilitators of dietary change and factors influencing the children's diet together with a systematic review of the literature. They used this research to create a logic model, which informed the content and approach to delivering the intervention and also how its component parts were believed to interact in order to improve the dietary nutritional intake of children aged under 5 and their families. This development stage was followed by a cluster‐randomised trial of 16 children's centres to pilot‐test the intervention, investigate its acceptability and assess the feasibility of performing a full‐scale evaluation trial.

The MRC first published guidance on developing and evaluating complex interventions in 2000 (MRC 2000), defining a complex intervention as one that is ‘built up from a number of components, which may act both independently and inter‐dependently’. This guidance document mapped the phases of development of complex intervention trials onto the phases of trials of pharmaceuticals: preclinical, phase I, phase II, phase III and phase IV. However, with the increase in knowledge and experience of developing and evaluating interventions using this framework, it became clear that greater flexibility was required. The guidance was therefore reviewed and a revised version was published in 2008 (MRC 2008). The new guidance stressed the iterative nature of development, feasibility/piloting, evaluating and implementing complex interventions. Moreover, while acknowledging that the individually randomised controlled trial will still usually provide the strongest evidence of effectiveness, it recognised that there are many situations for which such a design is impractical. Situations include those where the intervention is delivered to health professionals but evaluated on individuals under their care and those where an intervention is part of a policy change and will necessarily be introduced across the board, either concurrently or over a short period of time. Case studies are therefore provided, which illustrate the use of alternative randomised and, for situations where randomisation is impractical, non‐randomised designs. Alternative randomised designs include the cluster‐randomised design used by Watt et al. (2014), randomised stepped‐wedge designs and patient preference designs (MRC 2008): non‐randomised designs include natural experiments using, e.g., interrupted time series designs (MRC 2011).

Although some examples of methods appropriate for the development of a complex intervention are provided in the MRC guidance document, various approaches can and have been used. While many interventions are developed via a systematic review to identify the evidence on which to base the intervention, fewer may identify or develop appropriate theory to explain how the intervention might cause change and fewer still formally model process and outcomes, based on the proposed intervention (Corry et al. 2013). Modelling involves some testing of the intervention and key aspects of how it might be delivered, received and affect process, health or cost‐related outcomes; modelling can be performed either theoretically, following empirical data collection or using these approaches in combination. Watt et al. (2014) used qualitative and quantitative data collected from children's centres to form a logic model, which explained how the multiple components of the CHERRY intervention would interact and lead to specific positive process and nutritional outcomes.

Feasibility and pilot testing is also key to preparing for the evaluation of the effectiveness of a complex intervention. All too often, key assumptions are made regarding the delivery and uptake of an intervention, the number of eligible participants and the consent rate among those eligible. While such assumptions may be necessary, the MRC guidance indicates that these should be formally assessed via feasibility/pilot testing. McDonald et al. (2006) report on a review of a cohort of 114 trials and eight sub‐trials funded by the MRC or Health Technology Assessment programmes from 1994 onwards, which had planned to end recruitment by the end of 2002. They found that 38 (31%) recruited successfully, with 55 (45%) failing to achieve even 80% of their target recruitment; those with a dedicated trial manager were most likely to achieve target recruitment.

There are many examples of full evaluation trials, which, had the problems experienced been observed during a feasibility study, would not have been performed without substantive changes. In this journal, Labarère et al. (2011) reported the evaluation of an intervention centred around using a CD‐ROM providing information on how to initiate and maintain breastfeeding in France. As recommended by the MRC guidance (MRC 2008), considerable work was performed prior to the trial to develop the intervention. In the trial, the CD‐ROM was introduced to women during prenatal classes, was made available to women to play on a PC during their hospital stay and they were given a personal copy, together with a four‐page booklet to help with installation and use, at discharge. However, although sound development work was performed initially, key aspects underlying the evaluation of its effectiveness were not assessed for feasibility prior to full‐scale evaluation. Firstly, the study was powered on the assumption that there would be a 14% absolute increase in breastfeeding rates, from 70% in the control arm to 84% in the intervention arm; the 70% control arm rate was estimated from a previous study (Lelong et al. 2000). However, both control and intervention arm were found to have baseline breastfeeding rates of at least 86%, essentially meaning that the study was likely to be substantially underpowered. Secondly, over 50% of mothers never used the CD‐ROM. Appropriate feasibility testing could have identified these issues prior to embarking on a full‐scale evaluation of an intervention, which, given these limitations, was extremely unlikely to demonstrate effectiveness to the clinically important magnitude suggested.

Similar problems are in evidence in other settings and journals. Krauss‐Silva et al. (2011) report on a randomised trial performed in Brazil to evaluate the effectiveness of early administration of specially formulated probiotics to pregnant women to prevent spontaneous preterm delivery associated with bacterial vaginosis. They powered the study at 80% to detect a reduction from 6% to 3% in the percentage of deliveries before 34 weeks gestation. This resulted in a target total sample size of 1500 women, from an expected sequence of almost 3500. However, although 4204 women were screened, only 644 were recruited and only 3/320 (0.9%) and 2/324 (0.6%) had a spontaneous delivery before 34 weeks. Again, problems of under‐recruitment and non‐adherence were cited as the reasons for failure to detect a statistically significant effect, although preliminary effectiveness results showed promise with a relative risk of 0.69 (95% confidence interval 0.26–1.78) of spontaneous premature birth for the intervention relative to the placebo arm. Moreover, the spontaneous premature delivery rate was lower than expected. Appropriate feasibility work would have been likely to have enabled identification of these issues; it might then have been possible to introduce appropriate strategies make the trial feasible, or the trial may simply not have been commissioned.

For many interventions of interest to the readers of this journal, there are large numbers eligible, so identifying likely numbers of participants is often not a major issue. However, when trial populations involve subgroups or the intervention or trial procedures could lead to a lower consent rate, then it is important to investigate eligibility and consent rates in a pilot or feasibility trial prior to finalising the design of an effectiveness trial, particularly as researchers are often overly optimistic regarding numbers eligible for the trial (McDonald et al. 2006).

In summary, more researchers should follow the lead of Watt and colleagues, using a multi‐method development phase appropriate to the complex intervention, mechanism and processes by which the intervention is believed to work and the population to which it will be delivered. This development phase should be followed by appropriate feasibility assessment around key assumptions, concerns or unknowns to inform the execution of an effectiveness evaluation, which has the best possible chance of detecting a positive effect, if one indeed exists. Moreover, it is important that journals publish expositions of this early‐phase research to inform others of both the rationale for the alternative possible approaches and the importance of careful developmental work prior to a full‐scale evaluation of the effectiveness of the intervention.