Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care project (SHIP)

The development and evaluation of complex interventions within randomised controlled designs is a challenging area in health services research. The process usually entails a pilot phase to confirm the feasibility and potential effectiveness of the design before embarking on large and costly trials. However, the focus is often more on the study design and measures than on the theoretical base and extent to which the intervention can be appropriately applied. In this article, we use a case study to describe an approach to pilot work that addresses this gap.

Summary points

• Interventions are often defined pragmatically and lack any clear theoretical basis, which limits generalisability

• Implementation is rarely described, which limits understanding of why an intervention is or is not locally successful

• Integration of qualitative methods within pilot trials can help interpret the quantitative result by clarifying process and testing theory

• This approach defines three levels of understanding: the evidence and theory which inform the intervention, the tasks and processes involved in applying the theoretical principles, and people with whom, and context within which, the intervention is operationalised

• A case study shows how this novel method of programme development and evaluation can be applied

Background

Compared with drug trials or trials of surgical procedures, the design and development of a health service intervention is highly complex. In practice such interventions are often defined pragmatically, according to local circumstance, rather than building on any specific theoretical approach.¹ Even if an approach or technology can be clearly grounded in theory and evidence, it must still be operationalised and evaluated among specific practitioners and patients. There is thus a tension between evaluation of complex interventions and generalisability of results. Randomised trials alone can not tell us why an intervention was or was not successful, or whether the theory and evidence informing the intervention were appropriate or needed revision.

To clarify this, we propose three levels for defining a complex intervention: the evidence and theory which inform the intervention, the tasks and processes involved in applying the theoretical principles, and the people with whom, and context within which, the intervention is operationalised.

Case study

The Southampton heart integrated care programme brought together the principal researchers from the family heart and OXCHECK studies—two trials of primary care interventions for risk reduction in cardiovascular disease.²,³ Debate after publication of the trials highlighted the importance of focusing on the design and process of interventions as much as on the evaluation of outcomes, to understand the process of application of an intervention in a way that would allow success to be understood and replicated, and unsuccessful approaches to be abandoned.⁴

The comparatively disappointing results from primary preventive programmes for cardiovascular disease also moved the focus of research to secondary prevention programmes. The Southampton heart integrated care programme was such a programme. It was led by specialist nurses who coordinated and supported follow up care in general practice of patients who had a hospital diagnosis of myocardial infarction or angina.

Definition of levels

The table describes the three levels of intervention as applied to the Southampton heart integrated care programme. Level 1 summarises the theory and evidence underpinning, in this case, the choice of a target population, service provision, and management of behaviour change within the programme; it deals with the gap between evidence of efficacy and provision of treatments for people with established ischaemic heart disease.^4–7 It also deals with best practice in enabling behaviour change among practitioners and patients, using guidelines and psychological models.^19–22 Level 2 defines the essential tasks and processes required for operationalisation in these areas, at a generalisable level. Level 3 defines who would do what locally—elements which are specific to a local setting.

Quantitative approach

The pilot trial was designed to assess the impact of the programme on lifestyle and cardiovascular risk. The two arms of the trial compared the new approach with the usual care of patients with myocardial infarction or angina.⁸ Overall, 597 adult patients (from all 67 general practices in Southampton and south west Hampshire) with myocardial infarction or angina were randomised to intervention (33 practices) or control (34) groups. Follow up was 90% complete. The intervention increased follow up in general practice at 4 months and 1 year, and improved attendance for rehabilitation. No important difference was, however, observed between the intervention and control groups in any of the primary outcome measures of cardiovascular risk.⁸

Integration of a qualitative approach

The trial quantified the effect on contacts in primary care, but not the quality of those contacts. It did not allow interpretation of negative findings at the three levels of individual people, processes, or theory. To understand how the intervention was delivered, whether some of the elements seen as generalisable by the research team were particularly important or problematic, and the appropriateness of underlying theory, a qualitative approach is necessary.

The methods and results from the qualitative part of the programme have been reported in detail.⁹,¹⁰ Twenty five patients from the intervention group and, where appropriate, their partners were interviewed in depth. Participants were selected on the basis of maximum variety sampling.¹¹ Patients were interviewed twice: shortly after discharge from hospital and again around 3 months later. In addition, 22 practitioners (hospital and practice nurses) involved with the intervention were interviewed or participated in a focus group once. A semistructured schedule of topics was used to guide the interviews and focus groups. Interviews and focus groups were audiotaped, fully transcribed, and subject to thematic analysis to identify emergent themes. Analysis was based on the grounded theory approach, and preliminary analysis of early interviews was used to inform further data collection and analysis.¹²

Explanation of numerical results

Level 3: people and context

Early data from interviews were used to refine the programme. To optimise implementation of the approach, the findings were fed back to the development group 4 months into the 18 month intervention. Initial analyses concentrated on the experiences and understanding that patients, their partners, and providers had of the role of the liaison nurse, primary care team, and rehabilitation services. The analyses examined the way these individuals perceived the intervention as being implemented. This phase of the analysis showed that patients were confused about the nature of the rehabilitation programme, that progression through the system was often slower than practitioners led them to expect, and that conflicting messages about the need for follow up were sometimes provided by primary and secondary care.

These findings were used to optimise the intervention. One improvement involved each specialist liaison nurse recruiting patients from specific practices for which they were solely responsible, rather than from all practices as before. This change encouraged communication between liaison nurses and practice nurses and between practice nurses and patients.

Level 2: tasks and processes

The later in-depth phase of the qualitative analysis was concerned with the interviewees’ experiences, perceptions, and understandings over time after myocardial infarction or angina was diagnosed. The purpose of this analysis was to understand in more detail the tasks and processes needed to successfully operationalise the intervention. This analysis identified issues that may clarify the negative findings of the programme. For example, the focus groups for practice nurses suggested that in order for them to effectively follow up patients with established heart disease, greater sophistication in specification of essential tasks and processes was required than we had anticipated (box).

Suggestions from focus groups with practice nurses, about changes to their role in follow up of patients with ischaemic heart disease

1. Status within the primary healthcare team must be developed

2. Training must address knowledge and skills of cardiac assessment, and drug use and adherence, as well as facilitating behaviour change in relation to lifestyle

3. Opportunity must be given for nurses to give continuity of care

4. Improved integration at the primary-secondary care interface needs to take place, with secondary care staff clearly recognising the role of the practice nurse

Level 1: testing theory

The second purpose of the in-depth analysis was to confirm or question the evidence and theory on which the intervention was based. At this level, the qualitative inquiry also identified potentially important insights. For example, the findings suggest that understanding of heart attack as an acute but short term event will inhibit the adoption of subsequent long term lifestyle change, and that the intervention failed to address this possibility. There were several reasons. Initially patients described their astonishment at surviving a heart attack, which they had previously understood to be a fatal event, and therefore defined their own event as necessarily mild. In the period immediately after the heart attack, the information provided to patients by practitioners apparently encouraged this view of heart attack as a self limited episode from which complete recovery was probable, with little reference to the continuing underlying disease processes. At this stage lifestyle change seemed to be understood by patients as being linked to recovery in the short term rather than a long term preventive measure.

At a later stage patients’ understandings about heart attack were subject to change particularly in cases where experience of recovery did not reflect the information given. For example, information encouraged patients to believe that they would be able to have sex in 2-3 weeks, would be back to work in 6 weeks, and would be back to normal within 3 months, when this was often not the case.¹³ At this later stage, faith could be lost in “official” information from practitioners and evidence drawn instead from personal experience. Conflict between the two was associated with questioning the explanatory power of information from practitioners, and viewing the adoption of long term lifestyle change as action that would not guarantee protection from a further heart attack.

The programme thus found that patients’ understandings of heart attack are closely linked to their attitudes to the potential of lifestyle change to keep them well. The failure of the intervention to acknowledge that the occurrence of heart attack, the severity of heart attack, and the natural history of recovery from heart attack cannot be accounted for entirely by lifestyle seemed to be a central feature in patients’ understanding.⁹

Comment

It is increasingly recognised that using qualitative research methods can “reach the parts other methods cannot reach.”¹⁴ Qualitative methods can be used on their own as a preliminary to a quantitative study in order, for example, to establish meaningful wording for a questionnaire or to develop the elements of an intervention.¹⁵,¹⁶ They can be used to explain findings after quantitative research has been completed.¹⁷ Quantitative and qualitative methods are also used in parallel with substantive trials.¹⁸,²³,²⁴ In the pilot trial of the Southampton heart integrated care programme, we have moved a step further by explicitly integrating qualitative methods within a pilot trial design. We argue here that parallel application of qualitative methods in a pilot trial can contribute significantly and efficiently to both optimising and evaluating a new health services intervention.

The pilot randomised controlled trial tests a hypothesis formulated from existing theory and evidence before data collection, examines inputs (resource use) and outcome (effect size), and provides evidence of whether the approach is feasible within a specific locality and worth substantive evaluation. Complementary to this, the qualitative research is concerned with the perspectives of the people involved in delivering and receiving the intervention, and the context in which the data are produced. Analysis of these phenomena can provide information about the process of implementing an intervention and can lead to suggestions about logistical changes needed to optimise the completion of tasks and processes within a local context.

There is always a learning curve in applying new interventions, but it is usually hidden. Our approach makes it explicit and formalises it through systematic feedback, allowing both description and optimisation of the application of an intervention at local level. This begins to move intervention development in the direction of what industrialists call “evolutionary operations.”²⁵ This process involves performing rolling analyses over time—integrating both quantitative and qualitative findings to systematically optimise a production process.

At the deeper level of analysis for emergent themes, qualitative research can help understanding of the process whereby particular outcomes come about. It can thus enable a more sophisticated definition of what needs doing, as was shown with the practice nurses.⁹ It can also examine and test the theoretical basis of an intervention and question or affirm the principles on which the tasks and processes have been based. In this case the qualitative analysis raised questions about the information on natural history that is currently presented to patients by practitioners. It may be that in an effort to minimise fears and anxieties, practitioners are inadvertently providing an over- optimistic view of the natural history of convalescence for some patients and minimising the chronic nature of the underlying disease. These theoretical ideas are open to future hypothesis testing.

In summary, an integrated quantitative and qualitative approach to developing and evaluating complex interventions in health service research is both efficient and generalisable. Within one pilot study it formalises the usually hidden learning curve of implementation and optimisation. It allows better judgment of transferability of potentially effective programmes to other settings for confirmatory trials. It helps interpret quantitative findings, and questions underlying theory and assumptions to better inform future hypotheses and intervention designs.

Table.

Defining a complex intervention at three levels for health services research: case of secondary prevention of ischaemic heart disease

	Levels for defining intervention
Key issue	Level 1: Theory and evidence	Level 2: Essential tasks and processes*	Level 3: People and context
Target population	Focus on high risk patients2 3	Identification of high risk patients Recruitment of patients to intervention	Patients seen in secondary care with newly diagnosed angina or myocardial infarction
Service provision	Gaps in provision of secondary preventive care for patients with established ischaemic heart disease, in both primary and secondary care, in particular:
	Lack of primary-secondary care liaison	Effective linking of patient care from hospital through general practice	Two existing local cardiac rehabilitation sisters Three new cardiac liaison nurses working between two hospitals in Southampton 67 general practices
	Lack of appropriate prescribing	Delivery of appropriate prescribing for patients	General practitioner guidelines, monitoring by practice nurses
	Lack of cardiac rehabilitation4 6	Delivery of cardiac rehabilitation for patients	Limited local provision of cardiac rehabilitation services, supplemented by cardiac liaison nurses
Changing behaviour	Effective ways of enabling behaviour change:
	Guideline development19	Provision of prescribing guidelines for general practitioners and nurses, sent with patient	Local expert group to formulate, general practitioners to implement supported by practice nurses
	Behavioural techniques, e.g. patient prompts	Patient held booklet providing self monitoring and prompts to care	Practice nurses to implement supported by cardiac liaison nurses and general practitioners
	Theoretical models (applicable to patients and practitioners):
	Self efficacy20	Provision of individualised self management plans
	Stages of change21 Motivational interviewing22	Seminars for nurses Support groups for nurses	Practice nurses trained by local health promotion services, using the existing Helping People Change training package (often in their own time)

^*What needed doing—generalisable. 

^†Who was involved, and where—specific to local setting.