Integrating qualitative research with trials in systematic reviews

Short abstract

An example review from public health shows how integration is possible and some potential benefits

The value of including data from different types of studies in systematic reviews of health interventions is increasingly recognised. A recent editorial accepted that qualitative research should be included in systematic reviews, but pointed to a “daunting array of theoretical and practical problems.”¹ This article presents an approach to combining qualitative and quantitative research in a systematic review. We describe how we used this approach in a systematic review of interventions to promote healthy eating among children, full details of which are available.²

The review framework

The review question was: “What is known about the barriers to, and facilitators of, healthy eating among children aged 4-10 years?” The specific focus of the review was fruit and vegetable intake. We searched for two types of research: controlled trials (randomised or non-randomised) that examined interventions to promote healthy eating and studies that examined children's perspectives and understandings (views studies), often by using qualitative research methods—for example, in-depth interviews and focus groups.

Figure 1.

But will she eat her greens?

Credit: PAUL DARRAH/REX

We used conventional systematic review methods: sensitive searching, systematic screening, and independent quality assessment. These methods found 33 trials and eight qualitative studies that met our prespecified inclusion criteria.

We assessed studies for quality and reliability according to standards for their specific study types; they were then synthesised individually by using methods appropriate to the study. We conducted a meta-analysis with the data extracted from trials, used qualitative methods to synthesise the textual data extracted from the qualitative studies, and then integrated the findings from the qualitative synthesis with those from the meta-analysis. This gave us one review with three syntheses (fig 1).

Fig 1.

Stages of the review

Quality assessment

We maintained the key principles of avoiding bias and maximising transparency and accountability when conducting a systematic review. Both types of study went through a stage of quality assessment with two reviewers working independently and then meeting to discuss their findings. We used different tools for the different types of studies, building on recent developmental work and established consensus on quality assessment for both experimental studies^3-6 and qualitative research.^7-11 The studies were assessed in terms of reporting quality, internal validity or reliability, and, for qualitative studies, the extent to which the findings were rooted in children's perspectives (box).

We judged 21 of the 33 trials to be sufficiently reliable to enter the meta-analysis. Five of the eight qualitative studies met nine or more of the 12 quality criteria. The remaining three met six or fewer criteria. We conducted a sensitivity analysis and found that the results of these three studies did not contradict those from studies of a higher quality. The synthesis would have come to the same conclusions with or without their inclusion. In future, we have decided to exclude poorer quality studies from the synthesis and are conducting methodological work to assess the impact this has on the findings of the review.

Synthesis 1: meta-analysis of data from trials

In the first synthesis we carried out a traditional meta-analysis and pooled the effect sizes on six outcomes. We explored heterogeneity by carrying out subgroup analyses on a limited range of categories that we had specified in advance. Combining the results of the trials using a random effects model we found that, on average, the interventions described in the trials were able to increase children's fruit and vegetable consumption by about half a portion a day.

There was great variability between the studies. For example, one intervention was able to increase consumption by nearly two portions a day, while most of the others did not achieve one portion. Since all but two of the studies were evaluating different interventions, the summary statistic seems to conceal more than it reveals. We were unable to explain the statistical heterogeneity using prespecified categories covering study quality, study design, setting, and type of intervention.

Synthesis 2: synthesis of qualitative studies

The data for the second synthesis were in text form. In order to synthesise these, we copied the authors' findings verbatim into NVivo software¹² and then followed guidelines for the thematic analysis of textual data collected in primary research. The aim of the analysis was to infer barriers to, and facilitators of, healthy eating and ideas for effective interventions from children's views. We examined the findings of each study in turn and assigned codes to describe relevant sentences or paragraphs—for example, one code was “children prefer fruit to vegetables.” We then looked for similarities and differences between the codes to organise these into a hierarchical tree structure centred on children's understandings of healthy eating and the factors, in their views, that influence the food they eat.

In the next stage of the synthesis, three reviewers independently examined the descriptive themes and their associated data in the light of the review question to infer barriers, facilitators, and implied recommendations for developing interventions. The reviewers then met to discuss their findings and to develop a set of more abstract analytical themes.

As an example, one of the themes was that children do not see their personal health as their responsibility but that of their parents. Children do not regard purchasing fruit for health reasons as a legitimate use of their pocket money. Again, that is the job of parents. Children prioritise taste over health; as one child said, “Everything that is healthy tastes awful.” This theme suggested that future health promotion interventions should promote fruit and vegetables as tasty rather than healthy and any emphasis on health messages should be minimised. A second theme was that children distinguish between fruit and vegetables—they do not regard them as the same kind of food. The implication here is that fruit and vegetables should not be promoted in the same way within the same intervention. Although the five a day catchphrase might be appropriate for adults, it could be a disincentive for children.

Criteria for assessing quality

Quantitative studies (controlled trials)

Provision of data on outcomes before and after the intervention

Provision of data on all outcomes measured

Use of an equivalent control or comparison group

Qualitative studies

Quality of reporting (5 items)

Sufficiency of strategies for establishing reliability or validity (4 items)

Extent to which study findings were rooted in children's perspectives (3 items)

Synthesis 3: bringing the two sets of studies together

In the third synthesis we used the results of our qualitative synthesis to combine the findings from the controlled trials and qualitative studies. We then devised a matrix that juxtaposed the barriers, facilitators, and implied recommendations against the actual interventions that had been implemented and evaluated. Since we could not know the outcome of the qualitative synthesis in advance, we had to go back to the original interventions evaluated in the trials to identify those that built on the barriers and facilitators suggested by the children. This comparative analysis was guided by three questions:

• Which interventions match recommendations derived from children's views and experiences?

• Which recommendations have yet to be tried in soundly evaluated interventions?

• Do those interventions that match recommendations show bigger effect sizes or explain heterogeneity?

The table shows an example from the matrix. We used the good quality trials to assess whether the evidence of effectiveness supported or contradicted the children's views and to identify gaps in the evidence. The interventions that had not been evaluated well but were identified as building on a potential facilitator were recommended for more rigorous evaluation.

Table 1.

Example of the synthesis matrix

	Trials
Recommendation for intervention from children's views	Good quality	Other
Do not promote fruit and vegetables in the same way	0	0
Reduce health emphasis in messages to promote fruit and vegetables, particularly those which concern future health	5	6

For the recommendation that fruit and vegetables should not be promoted in the same way, we found no sound trials, so we identified a research gap. We found five good quality trials relevant to the second recommendation—reducing the emphasis on health messages. Two of these provided results on the same outcome, so we were able to conduct a statistical subgroup analysis dividing the studies into those that emphasised health messages and those that did not. (Since the original meta-analysis in synthesis 1 suggested that interventions targeting physical activity as well as healthy eating were qualitatively different from those that did not, the subgroup analysis excluded the interventions with a physical activity component.)

Summary points

Qualitative studies improve understanding of the views of the target group of an intervention

Uncertainty exists about how to include qualitative research within systematic reviews

A three stage method is described to integrate qualitative studies with controlled trials in one systematic review

Integration of the two types of studies can identify ways to improve interventions and their implementation

Figure 2 shows data from the trials in this subgroup analysis. The only two studies to increase vegetable consumption by more than 0.4 portion a day were the two that had little or no emphasis on health messages. We found that highly significant heterogeneity was explained by this subdivision.¹³ As with any exploration of heterogeneity, this was an exercise in hypothesis generation. Since we were dealing with small numbers of studies, our conclusions had to be cautious.

Fig 2.

Increase in consumption of fruit and vegetables in trials with data on health emphasis

Discussion

Integrating different types of study and data within the same review is one of the key challenges facing systematic reviewers today. We have discussed the issues raised by this at greater length elsewhere.^14-15 The main importance of this challenge is that the conclusions of reviews may be substantially altered by the inclusion of qualitative data, which are more likely to reflect the experiences of the target groups for intervention. This in turn could lead to the development of more appropriate and effective interventions.

However, the approach outlined here raises some methodological and conceptual challenges. Firstly, it challenges the notion that subgroup analyses should always be specified before the review. The inductive approach used in the thematic analysis of data from the qualitative studies meant that our categories for subgroup analysis could not be defined in advance. Secondly, the use of children's views to structure the final synthesis challenges traditional notions of who experts are and what constitutes expert opinion. Thirdly, the method is dependent on the judgment of reviewers when evaluating the extent to which an intervention meets a recommendation from the qualitative synthesis. Decisions also have to be made when the findings of the two syntheses conflict or when different parts of the matrix suggest contrary approaches.

The technique presented here breaks new ground in review methodology, offering an alternative to Bayesian methods for combining different types of studies in systematic reviews.¹⁶ Conceptually, the method allows the integration of quantitative estimates of benefit and harm with qualitative understanding from people's lives. The insights gained from the synthesis of qualitative studies allows exploration of statistical heterogeneity in ways that it would be difficult to imagine in advance. More work is needed to develop the method and test its relevance to different areas of health care and health promotion research.