Examining Intervention Design: Lessons from the Development of Eight Related Malaria Health Care Intervention Studies

Abstract

Abstract—Rigorous evidence of “what works” to improve health care is in demand, but methods for the development of interventions have not been scrutinized in the same ways as methods for evaluation. This article presents and examines intervention development processes of eight malaria health care interventions in East and West Africa. A case study approach was used to draw out experiences and insights from multidisciplinary teams who undertook to design and evaluate these studies. Four steps appeared necessary for intervention design: (1) definition of scope, with reference to evaluation possibilities; (2) research to inform design, including evidence and theory reviews and empirical formative research; (3) intervention design, including consideration and selection of approaches and development of activities and materials; and (4) refining and finalizing the intervention, incorporating piloting and pretesting. Alongside these steps, projects produced theories, explicitly or implicitly, about (1) intended pathways of change and (2) how their intervention would be implemented.The work required to design interventions that meet and contribute to current standards of evidence should not be underestimated. Furthermore, the process should be recognized not only as technical but as the result of micro and macro social, political, and economic contexts, which should be acknowledged and documented in order to infer generalizability. Reporting of interventions should go beyond descriptions of final intervention components or techniques to encompass the development process. The role that evaluation possibilities play in intervention design should be brought to the fore in debates over health care improvement.

INTRODUCTION

In recent decades, demand for evidence of “what works” in health care policy and practice has increased. This has led to intensified efforts to advance and standardize methodologies for the generation of evidence relating to health interventions.^1-4 Interventions that aim to improve health care have been classified as “complex,” reflecting their composition of multiple, often interacting, components within dynamic and multifaceted systems.^5,6 Significant investment has been made into methodological developments for evaluating such interventions.^3,4,7-10 However, less attention has been paid to how such interventions are designed and to reflecting on how this design process works in practice.

The ACT Consortium joined together 45 leading malaria researchers from 26 institutions around the world who were concerned about increasing access to new first-line artemisinin combination therapy (ACT) antimalarial drugs, targeting the use of ACTs to those with malaria infection, the safety of ACTs when used routinely, and the quality of ACTs accessed by malaria-affected communities (www.actconsortium.org). Research had shown that improvement in malaria case management was not amenable to simple interventions and that strategies would be required that worked with different components of the health system, including formal public health care facilities, private drug retailers, and community health workers.¹¹ In particular, the overdiagnosis of malaria was known to be a deeply embedded social practice that persisted despite World Health Organization policy supporting test-based treatment¹² and availability of rapid diagnostic tests (RDTs).¹³ Interventions would need to go beyond the behavior of individuals to shift social expectations and to change structures, attending to health care as a construct of dynamic social, political, and economic processes.

The Medical Research Council's (MRC) guidance on developing and evaluating complex interventions⁴ recommends building on existing evaluations of behavioral interventions to design complex health interventions. However, literature on existing interventions is difficult to learn from and apply to other settings. Even when interventions and their constituent components are clearly described, the processes undertaken and people, agendas, and disciplines that have influenced the design of complex health care improvement interventions are often not made explicit, either internally within the design process or to external audiences in reporting.¹⁴ This reduces the ability to interpret how findings of intervention effects could be inferred from one setting to another or to a scale-up scenario.

Tools and protocols have been developed in the health promotion field to aid intervention development.^15,16 However, the focus of these has been on individual health-related behaviors that may be amenable to psychological or environmental change factors. There is limited guidance on how to develop interventions that attend to the social nature of health care. This article aims to discuss the process and challenges faced by evaluation projects developing interventions to improve health care for malaria as an example of a systematic attempt to tackle multiple facets of a problem simultaneously.

We designed eight interventions to improve malaria care in five African countries. The interventions were to be rigorously evaluated according to current evidence-based standards,¹⁷ mostly through cluster-randomized trials (Table 1). All but one intervention (the Nigerian study) had a measurably positive impact in one or more outcome. Reflecting calls for multidisciplinary teams for such endeavors,¹⁸ project teams consisted of clinicians, public health practitioners, economists, anthropologists, epidemiologists, and statisticians. In this article, we summarize the intervention design methods and describe challenges faced and lessons learned about the process of health care improvement intervention design from across our projects. We have described lessons learned around evaluation methods elsewhere.¹⁹ The projects had similar aims and overall approaches to intervention design but varied in the detail of objectives, policy-related intentions, team knowledge bases and expertise, and budgets allocated to the design process. These similarities and differences enabled us to draw out lessons for intervention design across projects.

TABLE 1.

Summary of ACT Projects and Interventions Discussed in This Article

Study Titlea	Study Design	Setting and Dates of Implementation	Intervention	Publications on Intervention and Trial Design, Formative Research	Publications of Intervention Effect (Including Forthcoming)
The PRIME study	Two-arm cluster-randomized trial	Uganda Public health facilities 2011–2013	Enhanced health facility–based care for malaria and febrile illnesses in children	23,70-72	35,73-75
The REACT project, Cameroon (Research on the Economics of ACTs)	Three-arm cluster-randomized trial	Cameroon Public and mission health facilities 2010–2011	Basic and enhanced provider interventions to improve malaria diagnosis and appropriate use of ACTs in public and mission health facilities	76-79	80,81
The REACT project, Nigeria (Research on the Economics of ACTs)	Three-arm cluster-randomized trial	Nigeria Public primary health facilities and private medicine retailers 2010–2011	Provider and community interventions to improve malaria diagnosis using RDTs and appropriate use of ACTs in public health facilities and private-sector medicine retailers	82-84	85
The TACT trial (Targeting ACTs)	Three-arm cluster-randomized trial	Tanzania Public health facilities 2011–2012	Health worker and patient-oriented interventions to improve uptake of malaria RDTs and adherence to results in primary health facilities	86	87,88
ACT Pharmacovigilance project	Participatory research design	Uganda Health facilities and community drug distributors 2010–2012	Development of adverse event reporting forms for use by nonclinical workers to collect data on the effects of ACTs	65	89
RDTs for home management of malaria	Two-arm cluster-randomized trial	Uganda Community drug distributors 2010–2012	Introduction of RDTs for the home management of malaria at the community level		90
RDTs for drug shop management of malaria	Two-arm cluster-randomized trial	Uganda Drug shop vendors 2010–2012	Introduction of RDTs to drug shops to encourage the rational drug use for case management of malaria	91-94	95
Effect of test-based versus presumptive diagnosis in the management of fever in under-five children	Two-arm cluster-randomized trial	Ghana Public primary health facilities 2011–2012	Test-based diagnosis of malaria with RDT with restricting ACT to children who test positive	96	97,98

^aSee the ACT Consortium website (www.actconsortium.org) for more information on each of these studies.

MATERIALS AND METHODS

Intervention design steps and lessons learned were developed through a multiple case study approach,²⁰ with each of our eight intervention projects representing a case. This included all intervention studies undertaken by the ACT Consortium. Initially, an external researcher (LB) undertook reviews of project documents and phone or face-to-face interviews with investigators to learn their perspectives of the most important elements in the design of their interventions and lessons learned in the processes they had undertaken. All study investigators were invited to participate or to nominate relevant team members, and interviews followed a loose topic guide. Additional insights were brought together through a series of face-to-face and e-mail discussions with a team of core scientists who worked across the ACT Consortium projects to support intervention and evaluation designs. A first summary of steps and lessons learned that emerged in common across projects was produced for review by study teams. Each study team then provided further reflections and insights, forming reflexive accounts of their research processes that could enable learning for others.²¹ The experiences of our study teams were reviewed together with existing literature on intervention development to characterize a series of key steps and challenges experienced that resonated across studies and were poorly addressed in the literature. This formed an iterative analytical process that continued through the process of writing.²² Throughout, we provide empirical examples from our work in the text, boxes, and supplemental material.

RESULTS AND DISCUSSION

Each of our projects aimed to produce an evidence-based intervention package tailored for a particular context. Seven were to be delivered and evaluated through cluster-randomized trials and one through routine implementation (Table 1). The starting point for our interventions was technical—typically provision of commodities (ACTs and/or RDTs) and some form of training to support a change in practice. The intervention design processes were to yield details of these technical interventions and their delivery into local contexts. Four broad steps in this process were identified from across our projects (Figure 1):

1. Definition of scope

2. Research to inform intervention design

3. Design of the intervention

4. Refining and finalizing the intervention

FIGURE 1.

Phases in the Development of Complex Interventions

Alongside these activities, projects either explicitly or implicitly developed two sets of theory: program theory (or a logic model), which depicted the intended pathway for change from the intervention to study outcomes, and implementation theory, which depicted the intended vehicle for change, consisting of the “nuts and bolts” of the intervention itself. Each of these sets of theory attended specifically to the local social, political, and economic contexts where the interventions would be tested and potentially scaled up.

In all projects, the time and resources spent in designing interventions exceeded expectations. From the start of formative research to being ready for implementation, the overall process of intervention design took between six months and two years, requiring project durations to be extended by at least 20%. Project expenditure in this phase of work ranged between 10% and 25% of the overall budget. The variation in investment related to the starting point and goals of each project but also the relative investment compared with evaluations, which in some cases were large scale and costly. The knock-on effects of unexpected investment required in this phase of work included narrowing of scope of both the intervention and evaluation activities. The highest costs for projects were household surveys used in formative research and the professional development of intervention materials. It was not always possible to predict the length of time intervention design would take, particularly when responding to unexpected local priorities, which made interventions more relevant but less easy to budget time and resources for.

Step 1. Definition of Scope

As is often the case, the target problem to be tackled by the ACT Consortium projects had been established prior to the intervention design phase and particular components of the intervention, as well as the study design, had already been proposed as part of securing research funding. On initiation of funding, decisions were required to define the scale and potential scope of interventions. We found that three key areas needed consideration in defining the scope for interventions: the intended audience for results, the level of control required for the evaluation and intervention, and what is possible for evaluation designs.

Consideration of the Intended Audience for Results

ACT Consortium studies aimed to assist health policy makers at the global level and/or program managers at national and district levels to decide how to maximize health investments in relation to ACTs and RDTs. Keeping this aim and audience in mind was important when articulating the key criteria around which to design our interventions, which included feasibility, replicability, scalability, and cost-effectiveness. In each study, the intervention design process involved local and/or national stakeholders, who helped define intervention scope. We recognized that our interventions needed to be acceptable to different actors who had power to support the interventions in the future: from those in a position to fund and promote interventions, such as ministers of health, to those expected to take up the intervention in their daily practice, such as clinicians. The interventions therefore needed to fit with politically acceptable framings of the target problem and potential solutions.²³ See Table 2 for an example of how this was negotiated with stakeholders in Cameroon.

TABLE 2.

Examples from Case Studies of Lessons Learned for Intervention Design. Note. CMD = community medicine distributor, RDT = rapid diagnostic test.

Defining scope: consideration of intended audience for results	In the Cameroon REACT project, the initial focus of the intervention, defined in 2008, reflected concerns about appropriate use of first-line antimalarial drugs after recent policy changes to ACTs. In 2010, the project's focus was changed to appropriate diagnosis and treatment of malaria, incorporating the use of malaria RDTs. This responded to the upcoming roll-out of RDTs by the government and questions raised by them as stakeholders and the malaria community more broadly around how this could best be supported, given findings elsewhere that basic training was insufficient to support uptake of RDT results and adherence to test results. The trial therefore set out to answer specific concerns of Cameroonian policy makers by providing information about the cost-effectiveness of introducing RDTs alongside either basic training or an enhanced training intervention, compared with existing practice without RDTs. Furthermore, the initial inclusion of private sector providers was removed after feedback from the Ministry of Health that they preferred the tests first to be introduced at public and mission facilities.
Defining scope: consideration of level of control	For example, in the two Ugandan trials that introduced RDTs among community medicine distributors and drug shops, the objective was to learn the effect of the intervention if all providers allocated to the intervention received the full intervention. Training and follow-up supervision were delivered by members of the research team. The intention was not to produce an off-the-shelf intervention directly applicable for scale-up. By contrast, in the Nigerian trial, which introduced RDTs at public health facilities and private pharmacies and patent medicine dealers, the objective was to learn the effect of an intervention under routine conditions. Providers were invited to training sessions but were not followed up if they did not attend, and for a school-based intervention, school teachers and students were provided with intervention ideas and materials but were encouraged to undertake whatever activities they considered feasible. The intention was to produce interventions and results that would be directly applicable in practice. The latter study was closer to an effectiveness design than the former two.
Evidence review: scoping to identify potential intervention components in Uganda	The Ugandan PRIME project aimed to improve the quality of health care at health facilities in order to improve health outcomes and uptake of services. The target problem was identified as multifaceted, with several components of quality of care identified as targets for improvement in the project's formative research with health workers and community members. The targets were used as a focus for reviewing evidence of previous interventions:  Interventions to improve communication of health workers with patients  Interventions to improve working relationships among health workers  Interventions to improve facility-based supervision or coaching of health workers  Interventions to improve the way patients are received and offered services equitably  Interventions to improve the management of primary health facilities For each scoping review, which were conducted in parallel over a period of about three months, the team compiled a document to detail the search strategy, including search terms; inclusion and exclusion criteria; specific aspects of the intervention, including a taxonomy of potential intervention types that followed Abraham and Michie99; and how to assess outcomes of evaluations, whether qualitative or quantitative. For each paper identified, the team documented details of the intervention and evaluation as well as their perceptions of whether the intervention might be effective and feasible in the project's setting and whether any intervention materials already existed that could be drawn on. This process enabled the team to narrow down their search to interventions that were found to be effective at changing the target problem of interest and that were potentially transferable to the project's setting. This short list of evidence was then reviewed in conjunction with a review of behavior change theory, review of the findings of formative research, and discussion with local stakeholders.
Formative research: utility	Formative research prior to the Ugandan trial with CMDs involved 29 in-depth interviews with CMDs, health workers, and district health officials and 13 focus group discussions with mothers, fathers, and community leaders. The research aimed to understand existing CMDs' motivations, practices, and experiences and to explore the potential for introducing RDTs into the work and profile of these voluntary workers. The findings suggested that specific liaison personnel would be required to provide support to CMDs and that acknowledgment of their work through provision of commodities to support their roles would be required to sustain motivation.
Formative research: challenges with “barriers” approach	First, many of the barriers identified in our research were not amenable to change within the predefined scope of the intervention. For example, where wider policy dictated that certain providers were not allowed to sell or distribute certain drugs, such as antibiotics, we were unable to meet demand for training on treatment of nonmalarial febrile illnesses. Second, even when a barrier might be amenable to change, the research focus on barriers and problems provided little to inform positive action through intervention. For example, the finding in the Cameroon formative research that clinicians considered treatment with antimalarials to be a psychological treatment suggested a need for a change in expectations of consultation outcomes but did not in itself indicate what might be effective in achieving this. Third, the focus on barriers diverted attention from the motivation and agency of those enacting the problem behaviors; the practices desired by the intervention may not be in line with their priorities and motivations. For example, the Cameroonian clinicians' motivation for prescribing antimalarial drugs was to treat the whole patient, rather than the laboratory result or the malaria parasite. This represented a fundamental conflict between the focus of the malaria policy and of the study clinicians.79
Formative research: value of appreciative enquiry	In one of our studies in Uganda, identifying the aspirations of health workers for strengthening the quality of health care they provided gave us a framework for designing the PRIME intervention, based on their desires to strengthen technical, interpersonal, and management capacities.70

Consideration of Level of Control

Unlike in drug trials, which have established phases with different levels of control over the intervention in each, it is often not clear whether health care improvement evaluations are attempting to establish efficacy, effectiveness, or both.¹ Our involvement in both the intervention and evaluation designs of our projects meant active decision making around trade-offs between evaluating predefined interventions or interventions-in-action. Consciously defining the level of standardization for interventions and their constituents therefore emerged as important early on in the studies. Because of the different contexts of our studies and different gaps in evidence, our studies fell at different points along a spectrum from efficacy to effectiveness,²⁴ which affected intervention design and scope (see Table 2 for an example from two community-based studies in Uganda). On the whole, we required that interventions would be standardized to some degree but allowed for varying levels of flexibility in the content of interventions as delivered by implementers. This enabled some adaptation of content to different local contexts, such as different health facilities, provider types, or schools, as has been described elsewhere.^6,25 In most projects, rather than ensure that all members of the target population had the same experience of the intervention, we opted to encourage adaption and evaluate through process evaluations the fidelity, reach, dose delivered, and dose received of intervention components such as training.^26,27

Engagement with Evaluation Options

Unlike situations where evaluations are undertaken by researchers external to the intervention, which are intended to be more objective but can suffer from post hoc interpretations of intervention intentions and procedures,²⁸ our interventions were designed with evaluation options in mind. Our remit, driven by a desire for transferability and scalability, was to identify minimal essential interventions that could stimulate change and be scaled up in low resource settings. This essentialist agenda fitted well with the experimental paradigm in which randomized controlled trials (RCTs) are a gold standard. Seven of our studies had already planned RCTs to evaluate their interventions, with ancillary anthropological and economic studies. The use of RCTs affected the potential scope of their interventions. For example, although large-scale interventions have been hypothesized to be effective in changing behavior,^29,30 such interventions would be difficult to evaluate with RCTs due to logistical and budgetary constraints.⁵ Similarly, multifaceted interventions have been recommended to promote behavior change,^31,32 but because RCTs typically allow for a small number of comparison study arms, a trade-off emerged between potentially more effective multicomponent interventions and our ability to evaluate what worked with an RCT. Some trials compared existing practice with simple interventions such as RDTs plus instructions; others tested enhanced intervention packages, such as a series of peer group workshops. However, the more enhanced interventions that responded to the complex needs of local situations consisted of multiple and interacting components (material, human, theoretical, social, or procedural³³), requiring more complex evaluations to attempt to unpick their relative effects. Some projects employed cluster designs³⁴ due to predicted benefits of group-level intervention, and some employed additional evaluation activities to understand the process and mechanisms of change.²⁷ For interventions with a long-intended mechanism of effect, these additional evaluation activities became especially important in understanding intervention impact.³⁵

Step 2. Research to Inform Intervention Design

Once the overall goal and broad scope for each intervention were defined, teams were faced with numerous options for the detailed design of interventions. Three domains for research have been recommended for guiding the detail of intervention design,¹⁵ with different emphases from different disciplines: evidence review (most strongly recommended in medicine⁴), incorporation of theory (most strongly recommended in health psychology³⁶), and formative research (most strongly recommended in anthropology³⁷). All of our projects undertook research in each of these domains, often concurrently, although the lack of clear evidence or theory to guide our interventions, which required engagement with local social constructions of health care, meant that our greatest investment was in empirical formative research.

Evidence Review

Systematic reviews are advocated for use in complex intervention design, to bring together all evidence of the effectiveness of interventions for a particular outcome.⁴ We faced three major challenges in following this recommendation. First, there were few systematic reviews available related to interventions to improve antimalarial prescribing practice or use of malaria diagnostics. Second, undertaking such reviews ourselves would be methodologically challenging and time consuming given the number of potential interventions that could be considered to change health care practices, especially if attempting to account for heterogeneity between interventions lumped together and offering mixed results (such as supervision³⁸ or training³⁹) that could benefit from being split according to materials, people, theory, and procedures.⁴⁰ Third, the quality of reporting of intervention components was found to be poor, as well as the quality of evaluations, as others have described.^9,41 These challenges point to the importance of having readily available systematic reviews of complex interventions and their component parts in the evidence base. In lieu of this, some of our teams undertook scoping reviews “to map rapidly the key concepts underpinning a research area and the main sources and types of evidence available.”⁴² See Table 2 for an example from a health center improvement study in Uganda. These reviews were neither systematic nor comprehensive but were sufficient to achieve the following:

1. Recognition of the breadth of interventions that have been implemented and evaluated to change a particular practice.

2. A sense of the potential effectiveness of particular intervention types in given contexts.

3. Specific intervention ideas and components that could be successful in our proposed intervention contexts.

Incorporation of Theory

It has been argued that interventions with strong theoretical underpinnings can lead to stronger effects, more refined theories for understanding behavior change, more replicable interventions, and more generalizable results.^9,14,36 However, we could not identify clear guidance for incorporating theory into intervention design beyond the inclusion of individually oriented behavior change theoretical methods from health psychology.^15,43 On initiating our attempts to incorporate theory, we entered what appeared as a minefield of competing and conflicting ideas and definitions, whose presentation under the same term, “theory,” makes decision making for would-be designers challenging. Additional difficulty in navigating this field comes with the contradictions and debates between different political perspectives—for example, do cognitive theories and resulting interventions shift responsibility for healthy behavior to individuals, ignoring broader structural factors influencing behavior?⁴⁴ Some researchers have questioned whether theories used in complex health interventions to date have offered much beyond common sense.⁴⁵

Our endeavors to incorporate behavior change theories into our intervention designs involved attempting to uncover the theoretical basis of interventions identified as successful through empirical literature reviews (described above). This required some familiarity with different theoretical perspectives commonly used (a useful summary of the evolution of clinician behavior change approaches can be found in Mann⁴⁶), because most often the theory, model, or hypothesis for a program was not clearly reported. It also involved building on theoretical understandings and implications of formative research (described below). For example, we found communities of practice theory⁴⁷ to be a useful framing for several of our intervention designs, highlighting different ways that clinicians may learn and change their practice in groups, given our prescribing contexts where colleague relationships appeared important. We felt that our reviews of the theories behind interventions and incorporation of this into our intervention designs achieved the following:

1. Recognition of the breadth and strength of different approaches that have been applied to behavior change regarding target problems or behaviors of interest.

2. Identification of specific theories that may be successfully adopted to inform the approach to interventions within the proposed parameters.

3. Familiarity with the use of frameworks to conceptualize the way an intervention is proposed to achieve an effect, to inform project logic models that explicitly outline intended mechanisms of change (as described below).

Formative Research

Formative research has been advocated to enable optimal intervention design by understanding the scale of and reasons for the target problem in the particular context where the intervention will take place.^37,48 Most of our ACT Consortium projects undertook qualitative and quantitative research prior to intervention design, to understand the extent and nature of current practices, including perceptions and enactment of care and treatment seeking, as well as local histories of previous and existing interventions. We had anticipated formative research phases for our ACT Consortium projects to last between three and nine months, but this phase ended up requiring significantly more time and human resources for the fieldwork and analysis. This initial investment was considered valuable because the target behavior of antimalarial prescribing was known to be difficult to change.^49,50 However, in future studies we would hope that this period could be condensed. One reason our formative research took a long time from conception to informing intervention design may have been our focus on current behavior as the problem. This approach emphasizes identification of barriers to desired practices, such as physical, economic, cognitive, social, or policy factors, with the assumption that release of such barriers through an intervention would lead to the emergence of desired behavior.⁵¹ Important challenges arose for our intervention designs based on this approach (see Table 2).

Our experiences suggest areas of our formative research that were more informative for intervention design and that might be most productive in future studies. First, the areas of formative research that focused on eliciting stories of past success were particularly useful. The qualitative research in some of our projects borrowed from the perspective of appreciative inquiry, which proposes that solutions already exist in organizations, and analysis of these can allow interventions to amplify what works in that context (illustrated in Table 2).⁵² Second, understanding the landscape in which practices were embedded helped with understanding the motivations and priorities of the targets of interventions and to align intervention messages and modes of delivery with these.

Program Theory Development

The decisions that emerged from steps 1 and 2 above fed into explicit or implicit program theories of our interventions:⁵³ the way the intervention was intended to achieve particular outcomes. This has also been described as the intended mechanism of change⁵⁴ or change theory.⁵⁵ These descriptions of the intended journey on which the target of an intervention is hoped to travel can usefully be distinguished from the vehicle in which the journey is intended to be taken. The latter has been variously referred to as implementation theory,⁵³ an action model,⁵⁵ and process theory⁵⁶ and reflects the nuts and bolts of the intervention, discussed after step 3 (design of the intervention) below. Typically, these theories are developed post hoc, in relation to evaluation design.⁵⁷ We found it useful to articulate our assumptions and rationales for interventions during intervention design.

A useful method to depict program theory was logic modeling. Logic models describe the presumed causal linkages from project start to goal attainment.⁵⁸ Building on the work of others,^59,60 several of our Consortium projects developed logic models containing some or all of the components listed in Table 3. Supplemental Material File 1 provides an example of a logic model from one project.

TABLE 3.

Example Components of a Logic Model of an Intervention

Inputs (resources)	Human, financial, and material resources needed for the intervention.
Inputs (activities of the intervention)	Specific activities in which the target audience(s) participate, such as training activities, workshops, events, and requisition of supplies.
Conditions	Factors among recipients and in their environment that are expected to affect the mechanism of effect of an intervention; for example, the presence of supporting resources or leaders.
Outputs	Measurable proximal outputs of intervention activities; for example, knowledge or motivation of a direct or indirect target audience.
Outcomes	Changes that occur in the target audience(s), which can be either proximal—for example, drug use behavior, patient satisfaction—or distal—for example, community health indicators.

Our experiences of developing a logic model during intervention design suggest that key benefits are as follows:

1. To assist with the choice of intervention by articulating presumed hypotheses linking intervention options to intended outcomes.

2. To ensure that the intervention has internal consistency, that a mechanism of effect is predicted for each intervention component, that supporting components are accounted for in the model and therefore also in the evaluation activities, and that there are no important gaps or additional activities that are not justified within the model.

3. To act as a visual aid to communication, enabling the team and wider stakeholders to reach a common and consistent understanding of the components of the intervention.

4. To guide data collection for the intervention evaluation by showing where, when, and what information needs to be documented or collected.

Of note, our logic models rarely remained static. They became dynamic tools, being adapted and in turn adapting the design of both intervention and evaluation activities. Crucially, these logic models were developed, and assumptions articulated, with close attention to the specific contexts in which the interventions would take place, particularly the social, political, and economic contexts. Although we did not state the findings of formative research explicitly in our logic models, this was implicit in our processes of considering context and mechanisms of effect and could be included explicitly in future work.

Step 3. Design of the Intervention

Once research was completed to inform design options, we attempted to take an evidence-based approach to developing the detail of intervention components and materials. In most ACT Consortium projects, two activities were undertaken: workshops to review research undertaken so far and to select which specific interventions would be implemented and detailed development of intervention content, activities, and materials.

Intervention Selection

We found small-scale workshops to be a useful format to bring together findings from steps 1 and 2 with input from across the research team to consider potential interventions and their feasibility and potential effectiveness. Inviting stakeholders to the workshops or to individual follow-up meetings was useful in ensuring that the interventions fitted with priorities and other previous or current interventions and ensured that policy makers and those who would be responsible for scaling up interventions felt that the interventions were relevant to their concerns. We adapted recommendations of other researchers⁶¹ to structure these workshops, which had a similar format across the projects. Broadly, these included a discussion and agreement of criteria for the intervention design (e.g., effective, feasible, replicable, sustainable), informed by both the parameters considered in step 1 as well as the values and priorities of stakeholders attending the meeting. The workshops were an opportunity to present and discuss reviews and formative research undertaken by different members of the research team. Following this, a collaborative effort by the researchers, stakeholders, and field teams led to a long list of potential interventions that was refined to a short list that fitted the criteria set out at the start (see Supplemental Material File 2 for an example structure of our intervention design workshops).

A key challenge to note at this stage was the bringing together of disparate evidence from theory, literature, and empirical research. In some cases, we had to negotiate conflicts between these different sources or between members of study teams from different disciplinary backgrounds. For example, preferences over one or other model of behavior change could conflict and were not easy to resolve in a context where evidence was weak. In these cases, the resulting intervention component or mode of delivery represents a compromise across different disciplinary and individual preferences.

Development of Content, Activities, and Materials

ACT Consortium projects found that designing the detail of intervention materials took considerable time and resources. Activities and materials used in interventions included facilitated group learning, self-reflection tasks, participatory dramas, peer education, supervisory visits, tools for referral of patients or requisition of supplies, and distribution of posters and leaflets. Each required project teams to return to literature or to the field or to seek external expertise to identify evidence, best practice, and user perspectives on the implementation of activities. For several projects that used workshops to facilitate change, a six-step learning process was developed, based on literature of theory and best practice in adult learning (see Supplemental Material File 3).

Investment in the additional work in developing materials at this stage was considered valuable for the following reasons:

1. To ensure quality of intervention activities and materials and optimize the likelihood of effect.

2. To ensure consistency in intervention delivery in order that components are easily replicable.

3. To enable evaluation of the intended intervention through clear documentation of the activities, materials and procedures to be implemented.

Some ACT Consortium projects attempted a bottom-up approach to the design of some intervention materials, explicitly recognizing that target recipients are best placed to identify or refine content, messages, modes of delivery, and visual details that are likely to be effective and acceptable to end users.^62-64 Participatory research was found to be valuable for these projects because it enabled them to draft material quickly for further testing and revision in intensive rounds of development (see, for example, Davies et al.⁶⁵ for a description of the use of participatory research in the pharmacovigilance materials project in Uganda).

To assist with evaluation of interventions and our ability to draw conclusions from results, ACT Consortium projects recognized the need for consistency in the delivery of interventions, from the procedures followed and materials delivered to participants, to detailed manuals for workshop trainers. Such manuals required careful design of visuals and layout; for example, with the use of summary boxes and icons to assist the reader to follow activities during and after the workshop (see Supplemental Material File 4).

Implementation Theory

Step 3 gave rise to our theories and protocols for how the interventions should be delivered, which is sometimes known as “implementation theory.” This was most commonly articulated through process objectives that encompassed both content—for example, perceptions that specific workshop objectives were relevant and achieved—and procedures—for example, participant attendance at workshops or receipt of specific supplies at a particular time. These were depicted in manuals, protocols, and standard operating procedures for trainers, those delivering resources, supervisors, feedback messengers, and others engaged in the implementation process. This implementation theory was tested in process evaluations by assessing the fidelity, reach, dose delivered, dose received, effectiveness, and context of implementation.²⁶

Step 4. Refining and Finalizing the Intervention

Once intervention activities, materials, and protocols were drafted, most of our projects undertook a period of piloting and pretesting these components in order to evaluate comprehension, acceptability, and relevance and to refine final versions. Our project teams noted the importance of investment in this stage, when a gap was revealed between the materials and procedures developed so far and the reality of delivery to and understanding of these by the target audience in practice. From across the ACT Consortium projects, investment in this stage is reported to have led to the following consequences:

1. Optimization of materials and activities through pretesting to identify and adapt any components that failed to communicate intended messages, were misunderstood, or were not deemed relevant to the target audience⁶⁶ (see Supplemental Material File 5).

2. Ability to adapt procedures for ease and impact of delivery and receipt of the intervention during implementation; for example, decisions on grouping of participants to maximize peer interactions, timing, and transport for workshops to ensure timely participation with minimized disruption and feasibility of intervention intensity in practice.

3. Opportunity to train delivery staff during pilots, with two-way review of the intervention and delivery practices, which could feed into updated protocols for implementation procedures.

4. Opportunity to involve stakeholders in reviewing and revising the content and implementation of the intervention.

5. Opportunity for evaluation teams to pilot tools to document the implementation of the intervention.

Piloting and pretesting involved presentation of the draft intervention component, such as a training module or leaflet, with various methods to elicit feedback from the target audience, implementers, and/or observers. Methods included structured questionnaires, focus groups, and informal discussions. Several rounds of revisions to draft materials were often made, with each new draft tested and the feedback used to improve the subsequent draft, until the quality, suitability, and comprehension of the final product were deemed sufficient to implement and evaluate formally.

CONCLUSIONS

Intervention design is a crucial, yet often neglected and black-boxed, process in the field of health care research. We believe that it is time that more attention is paid to how it is done. This article shares methodological experiences from eight ACT Consortium projects, which designed and evaluated a variety of complex health interventions to improve malaria care in five malaria-endemic countries in Africa. This adds empirical examples to existing guidance on health intervention research, which thus far has been limited in terms of detail⁴ and mostly focused on individual health behavior change.^15,16 Our attention to the processes of intervention design shows that intervention characteristics are not merely technical but are produced in response to social, political, and economic priorities. As such, we alert others to the requirements and realities of such endeavors and encourage greater transparency in articulating these processes. The steps outlined here provide a framework with which to view processes of intervention design in research projects. Routine articulation of such processes would allow for improved assessments of transferability of interventions and inference of their potential effects to other scenarios.

The time taken to design interventions within an evidence-based paradigm was invariably longer than expected, required multiple rounds of protocols and ethics approvals, and, crucially, required a substantial proportion of overall project budgets. Funders and researchers both need to recognize that health care improvement interventions cannot be taken off the shelf; they require substantial investment to develop, and this should be planned for accordingly. Without this investment, funders and researchers risk further well-conducted evaluations that describe the lack of impact of poorly designed interventions. In situations with limited funding, those designing interventions would benefit greatly from learning about the rationales and processes of the design of other similar interventions, emphasizing a need for better reporting.

The dearth of methodological and empirical literature on the process of intervention design unnecessarily lengthened our efforts to design robust interventions. We argue strongly that the process of intervention development should be routinely reported, in the same way as trial protocols are now requested to be published. Criteria for reporting interventions have been proposed,⁶⁷ largely from the health psychology field, and through a lens of interventions as behavior change techniques.⁶⁸ Though such taxonomies are useful to understand what finally constituted an intervention, we propose that the process by which such interventions were arrived at is equally crucial for transferability of findings. Reporting of interventions should go beyond their final constituents, to describe the process of development including reflection of the social, political, and economic contexts that led to that particular intervention package. Such reporting could follow the framework of steps and theory outlined in this article. Specific sections of journals where intervention designs can be published would support and promote both publication and debate over methods. Until this happens, the publication of evaluations of interventions whose process of development has not been clearly articulated will continue, with a consequent risk of replicating mistakes and reinventing wheels that could have been avoided with greater and better quality reporting of the process of intervention design.

This article highlights a paradox in evidence-based intervention research: on the one hand, basing interventions on an international and local evidence base should strengthen their effectiveness, whereas on the other hand intervention options are constrained by the requirements of evaluation design that are in place to strengthen the evidence base. This paradox represents a challenge that goes beyond the design of malaria interventions. Operating within these demands can result in the production of high-quality, well-understood interventions that have smaller impacts than less well-controlled, more flexible interventions that may be more difficult to assess for attribution of effect. This reflects a tension between desires for experimental evidence, which promises generalizability of interventions over time and space, and more contingent ways of knowing what works.⁶⁹ The role that evaluation possibilities play in intervention design should be brought to the fore in debates over health care improvement. The development of best practice in designing interventions will need to go hand in hand with the development of best practice in evaluating complex health interventions in action.

DISCLOSURE OF POTENTIAL CONFLICTS OF INTEREST

The authors report no potential conflict of interest.

SUPPLEMENTAL MATERIAL

Supplemental data for this article can be accessed on the publisher's website.