Abstract
Objective To examine the use of qualitative approaches alongside randomised trials of complex healthcare interventions.
Design Review of randomised controlled trials of interventions to change professional practice or the organisation of care.
Data sources Systematic sample of 100 trials published in English from the register of the Cochrane Effective Practice and Organisation of Care Review Group.
Methods Published and unpublished qualitative studies linked to the randomised controlled trials were identified through database searches and contact with authors. Data were extracted from each study by two reviewers using a standard form. We extracted data describing the randomised controlled trials and qualitative studies, the quality of these studies, and how, if at all, the qualitative and quantitative findings were combined. A narrative synthesis of the findings was done.
Results 30 of the 100 trials had associated qualitative work and 19 of these were published studies. 14 qualitative studies were done before the trial, nine during the trial, and four after the trial. 13 studies reported an explicit theoretical basis and 11 specified their methodological approach. Approaches to sampling and data analysis were poorly described. For most cases (n=20) we found no indication of integration of qualitative and quantitative findings at the level of either analysis or interpretation. The quality of the qualitative studies was highly variable.
Conclusions Qualitative studies alongside randomised controlled trials remain uncommon, even where relatively complex interventions are being evaluated. Most of the qualitative studies were carried out before or during the trials with few studies used to explain trial results. The findings of the qualitative studies seemed to be poorly integrated with those of the trials and often had major methodological shortcomings.
Introduction
Randomised controlled trials are used widely for showing causal relations in health and social care because their study design is the only one that is able to control for unknown or unmeasured confounders. Randomised controlled trials are sometimes used to evaluate “complex” interventions—that is, those “made up of various interconnecting parts”1 that act both “independently and inter-dependently.”2 3 Qualitative approaches can contribute in several ways to the development and evaluation of both complex and other health interventions (box 1). Consequently, increasing numbers of randomised controlled trials of such interventions include qualitative components4 5 and interest in this approach is growing. The use of multiple, integrated approaches may be particularly useful in the evaluation of the effects of complex health and social care interventions as these involve social or behavioural processes that are difficult to explore or capture using quantitative methods alone.1
Box 1 Ways in which qualitative methods can be used alongside randomised controlled trials
Before a trial
To explore issues related to the healthcare question of interest or context of the research
To generate hypotheses for examination in the randomised controlled trial
To develop and refine the intervention
To develop or select appropriate outcome measures
During a trial
To examine whether the intervention was delivered as intended, including describing the intervention as delivered
To “unpack” processes of implementation and change
To explore deliverers’ and recipients’ responses to the intervention
After a trial
To explore reasons for the findings of the trial
To explain variations in effectiveness within the sample
To examine the appropriateness of the underlying theory
To generate further questions or hypotheses
The need for methodological research on the ways in which qualitative approaches should be used in randomised controlled trials has been discussed widely.6 7 8 9 10 11 However, to our knowledge no studies have attempted to examine systematically current practice on the use of qualitative approaches in randomised controlled trials of complex healthcare interventions and how they could be used to improve the usefulness and policy relevance of the findings of a trial. By complex interventions we mean those including at least some of the following characteristics: several elements that “may act both independently and inter-dependently”2; complex systems or mechanisms for delivery of the intervention; an intervention that is difficult to describe and replicate; complex explanatory pathways, either physiological or psychosocial; and a degree of uncertainty about the mechanism of action of the intervention or its “active ingredient.”2
We systematically examined the use of qualitative approaches alongside randomised controlled trials of complex healthcare interventions to provide an overview of current practice in this area and ways of identifying qualitative studies undertaken alongside randomised controlled trials. We also explored how trial teams could improve the quality of qualitative studies linked to randomised controlled trials and how synergies between qualitative approaches and randomised controlled trials can be maximised.
Methods
We obtained a list of all randomised controlled trials published in English during 2001-3 and included in the register of the Cochrane Effective Practice and Organisation of Care Review Group.12 From the list for each year (492 randomised controlled trials in total) we sampled every fifth study to obtain a sample of 100 trials (33 or 34 trials from each year). We chose this approach for several reasons. Firstly, we wanted a sample of recently published trials of more complex interventions as we assumed that the use of qualitative methods alongside such trials has increased in recent years. Secondly, the randomised controlled trials needed to have been published sufficiently long ago to allow associated qualitative studies also to have been published. Thirdly, we thought the Cochrane Effective Practice and Organisation of Care Review Group register, with its specific focus on interventions to change professional practice and the organisation of care, more likely to include a higher proportion of relatively complex randomised controlled trials than databases such as Medline or Embase. Details on studies included in this register and the search strategies used to locate them are available elsewhere.12
Where the sampled report was not the primary paper for the randomised controlled trial, the primary report was located. We attempted to identify all published and unpublished qualitative studies linked to these randomised controlled trials. We defined a qualitative study as any study that used qualitative methods for data collection or analysis. We initially checked the primary randomised controlled trial for citations of qualitative studies. We then located the primary randomised controlled trial in PubMed and searched for related studies and other studies published by the authors of the randomised controlled trial. We also located the randomised controlled trial in the Science and Social Science Citation Index and checked the list of studies citing the paper. Any potentially relevant titles and abstracts were examined and full papers obtained where necessary. Finally, we contacted the authors of the randomised controlled trials for information on any published or unpublished qualitative studies linked to their trials. We received responses for 76 of the 100 papers.
Two reviewers used a standard form to extract data from each “case”—that is, the randomised controlled trial and any qualitative studies. This included descriptions of the randomised controlled trials and qualitative studies; the quality of the randomised controlled trials and the qualitative studies; and information on the approaches (if any) used by the authors to combine the findings of the randomised controlled trials and qualitative studies. The quality of the trials was assessed using the quality checklist of the Cochrane Effective Practice and Organisation of Care Review Group.13 The quality of the qualitative studies was assessed using a modified checklist from the critical appraisal skills programme.14 These modifications included further details on whether the qualitative approach was justified and appropriate to the research question, whether the research context was described adequately, and items to differentiate adequate reporting of methods from the appropriateness of those methods, in relation to the research question. We summarised the findings of the study narratively.
Results
Thirty of the 100 included randomised controlled trials had qualitative work associated with them. Nineteen of these qualitative studies were published, either as stand alone papers or within another paper. In 23 (77%) of the 30 cases in which qualitative approaches were used, the researchers employed qualitative methods for both data collection and analysis. In the remaining seven cases some form of qualitative data collection (for example, group discussions or individual interviews) was used, but no formal analysis of these data using qualitative approaches was reported. Most of the qualitative studies (n=25) were carried out before or during the randomised controlled trial (figure). An explicit theoretical basis for the intervention was reported in 12 of the 30 cases.
Use of qualitative studies in sampled randomised controlled trials
Randomised controlled trials that included qualitative research
The 30 trials that included qualitative research were carried out in a variety of settings, from general practices to communities and consumers’ homes. Twenty four of the trials were carried out in primary care and the remaining six trials evaluated interventions in secondary care or across a mix of levels.
The trials dealt with a wide range of healthcare issues, the most common being mental health, the appropriate use of medicines, and sexual health. All the trials were carried out in high income countries. The methodological quality of the trials that included qualitative research was similar to those without such studies.
Qualitative studies
Objectives of the studies
The objectives of the qualitative studies varied widely (table 1). The 16 studies done either before the trial, or before and during the trial, had one or more of the following objectives: to explore the knowledge, attitudes, or practices of the target groups about the topic in question; to explore the illness experience of consumers; to develop the intervention; and to develop the instrument used to measure the effects of the intervention in the randomised controlled trial.
Table 1.
Characteristics of qualitative studies included in this review and their level of integration with accompanying randomised controlled trials (studies that used only qualitative methods of data collection are not described)
| Randomised controlled trial, country | Qualitative study | Health service issue tackled by intervention | Trial stage during which qualitative research undertaken | Methodological approach to qualitative work | Reasons for including qualitative methods in trial | Nature and degree of qualitative and trial data integration in analysis and interpretation |
|---|---|---|---|---|---|---|
| Burns 2002,w1 UK | Weaver 2003w2 | Community based management of people with psychosis | During trial | “Thematic interview survey” and “critical incident approach” | 6, 7, 8, 9, 11 | B |
| Davies 2002,w3 Canada | Davies 2002,w4 Graham 2004w5 | Uptake of obstetric guidelines | After trial | “Qualitative case study research methodology” | 7, 8, 9 | B |
| Faithfull 2001,w6 UK | Faithfull 1995w7 | Morbidity and satisfaction with care in men treated with radical radiotherapy for cancer of prostate and bladder | Before trial | Not stated | 1, 4 | A |
| Fretheim 2003,w8 Norway | Fretheim 2004w9 | Appropriate use of antihypertensive and cholesterol lowering drugs for prevention of cardiovascular disease | Before and during trial | Not stated | 3 | A |
| Jaatinen 2002,w10 Finland | Jaatinen 2003w11 | Managing doctor referrals to specialist | After trial | Action research | 12 | B (but hard to say as so little data) |
| Jibaja-Weiss 2003,w12 USA | Jibaja-Weiss 2003w12 | Breast and cervical cancer screening | Before trial | Not stated | 3 | F (No information) |
| Kaner 2003,w13 UK | Lock 2002w14 | Alcohol use | Before trial | Grounded theory | 1, 3 | A |
| Koniak-Griffen 2003,w15 USA | Lesser 2003w16 | HIV prevention in adolescent mothers | During trial | Ethnography | 1, 6 | E |
| McKinstry 2002,w17 UK | Unpublished | Telephone versus face to face consultations to manage requests for same day appointments | After trial | None specified | 7 | D |
| Miles 2002,w18 UK | Miles 2002w19 | Whether sexual health clinics for women led by specialist nurses are as effective as those led by senior house officers | Before and during trial | Focused ethnographic approach, grounded theory approach | 1, 5, 6, 7, 8 | B |
| Osganian 2002w20 (primary paper Luepker 2000), USA | Finnegan 2000,w21 Zapka 1999w22 | Delay in presentation to hospital for acute myocardial infarction | Before trial | Not specified, but social cognitive theory and self regulatory model of health and illness behaviour both informed interview guide | 1, 3 | B |
| Robling et al 2002,w23 UK | Robling 1998,w24 Hale 1999w25 | Use of direct access magnetic resonance imaging by general practitioners | Before trial | Critical incident technique in which each magnetic resonance imaging requested was regarded as a critical incident and was focus for interview | 1, 4 | A |
| Scheel 2002,w26 Norway | Scheel 2002w27 | Assisting people with lower back pain to return to work | Before trial | None specified | 3 | E |
| Stange 2003,w28 USA | Goodwin 2001w29 | Improving delivery of preventive services | During trial | Ethnography | 12 | D |
| Szmukler 2003,w30 UK | Stern 1999w31 | Support for carers of patients with psychosis | During trial | Narrative approach plus “relative influence questioning” for question guidance | 13 | D |
| Thapar 2002,w32 UK | Unpublished | Primary care of people with epilepsy | Before trial | Not stated | 3 | D |
| Tijhuis 2002,w33 Netherlands | Unpublished, Tijhuis 2003w34 | Care for people with rheumatoid arthritis | Before trial | Not stated | 4 | D |
| Von Koch 2001,w35 Sweden | Von Koch 1998,w36 Von Koch 2000w37 | Rehabilitation after acute stroke | During trial. | Study a: “qualitative case study”; study b: not stated | Study a: 5, 6, 7; study b: 5 | A |
| Wallace 2002,w38 UK | Unpublished, MacFarlane 2002w39 | Communication between general practitioners, specialists, and patients about specialist consultations | During trial | Study a: none specified (called “qualitative evaluation”; study b: grounded theory approach | Study a: 7; study b: 6, 7 | Study a: B; study b: F |
| Watson 2002,w40 UK | Watson 2000w41 | Evidence based prescribing by pharmacists | Before trial | Not stated, but used theory of planned behaviour to design interview guide | 1, 3 | A |
| Wheeler 2003,w42 USA | Clark 1994w43 | Management of heart disease in older women | Before trial | None specified. Implied to be loosely based on grounded theory | 1, 3, 5 | B |
| Young 2003,w44 Canada | Young 2004w45 | Care for patients after myocardial infarction | After trial | None specified (study notes that “qualitative design used” | 7 | B |
| Zermansky 2002,w46 UK | Petty 2001w47 | Appropriateness of repeat drugs prescribed to elderly patients | During trial | Not stated | 12 | A |
1=exploring issues related to healthcare question of interest or research context; 2=generating hypotheses for examination in randomised controlled trial; 3=developing and refining intervention; 4=development or selection of appropriate outcome measures; 5=examining fidelity or integrity of interventions delivered, including describing intervention as delivered; 6=“unpacking” implementation and change processes; 7=exploring deliverers’ and recipients’ responses to intervention; 8=exploring reasons for trial findings; 9=explaining variations in effectiveness within sample; 10=examining appropriateness of underlying theory; 11=generating further questions or hypotheses; 12=other; 13=unclear; A=no integration (analyses and interpretation done separately); B=analyses done separately, with degrees of integration in interpretation; C=integration during both analysis and interpretation; D=not reported; E=other; F=unclear.w48
The nine qualitative studies done during the trials had a wide range of objectives. These included describing the intervention as delivered and exploring issues influencing the effects of the intervention, the illness experience of consumers, participants’ experiences of the intervention (box 2), and reasons for refusal to participate in the trial (box 3).
Box 2 Example of a qualitative study exploring how trial participants experienced the intervention (adapted from Harrison et al 200615)
Background
Researchers carried out a randomised controlled trial in the United Kingdom to evaluate the effects of joint teleconsultations on hospital follow-up appointments. They concluded that patients’ overall satisfaction was higher for teleconsultations than for conventional outpatient appointments. A proportion of patients were, however, dissatisfied with their teleconsultation. It was not possible to determine from the trial findings the reasons for satisfaction or dissatisfaction or how satisfaction could be increased in other telemedicine programmes. The qualitative study aimed to answer these questions.
Qualitative methods used
Semistructured individual interviews were carried out with 24 patients within one month of their teleconsultations. The researchers used the framework approach to carry out a thematic analysis of these data.
Findings
Joint teleconsultations were, overall, highly acceptable to patients for several reasons. These included enhanced customer care, such as enhanced convenience, reduced cost, and increased punctuality. Most patients also appreciated the presence of the general practitioner in the consultation, feeling that this improved communication between the specialist and generalist and allowed the general practitioner to summarise and interpret the consultation for the patient. However, one patient stated that they had been excluded from the consultation. Other patients were dissatisfied with parts of the consultation because they would have preferred to be examined directly by the specialist. Some patients found that the technology interfered with their communication with the doctor—for instance, because of lack of synchronisation between sound and vision. The authors of the study discussed how some of these factors could be tackled in joint teleconsultations in the future.
Box 3 Example of a qualitative study exploring why people declined to participate in a randomised controlled trial (adapted from Petty et al 200116)
Background
Researchers carried out a randomised controlled trial in the UK to evaluate the effectiveness of pharmacist run clinical drug reviews in patients aged 65 years or more from general practices. Eligible participants were contacted by post, sent one reminder, and contacted by telephone if no response had been made. Twenty six per cent (n=68) of those contacted by telephone did not wish to participate in the study. The aim of the qualitative study was to identify reasons for non-participation.
Qualitative methods used
Researchers used unstructured questions to ask the 68 patients their reasons for not wishing to participate. The responses were recorded in writing. The researchers independently carried out a thematic analysis of these data, then agreed on a set of categories.
Results from the qualitative study
Ten broad categories of reasons for non-participation were identified. These included administrative categories such as difficulties in reading the invitation letter and not being available at the time suggested. Other categories were tied to behavioural factors such as confusion or lack of understanding of the trial, negative attitudes to health care, and mistrust of the objectives of the trial. The authors suggested that these factors needed to be addressed to increase the number of patients consenting to studies on drug review.
Of the four qualitative studies carried out after the trial, two explored participants’ experiences of the intervention, one explored factors influencing the effects of the intervention (box 4), and one analysed the process for development of the intervention.
Box 4 Example of a qualitative study exploring the reasons for the findings of the trial (adapted from Davies et al 200217 and Graham et al 200418)
Background
Researchers evaluated a strategy for implementing guidelines for nursing care during labour in hospital in Canada. The results of the trial were mixed. The researchers then carried out a qualitative study to explore why this guideline was introduced successfully in some settings but not in others.
Qualitative methods used
A case study approach was used, with individual interviews and group discussions done with nurses, nurse administrators, and nurse educators at the study sites. Interviews were audiorecorded and transcribed and a form of thematic analysis was applied.
Results from the qualitative study
A wide range of factors related to the study settings, the recipients of the guideline, and the characteristics of the guideline interacted to affect implementation of the guideline. Important factors included changes to the external environment of clinical practice; leadership and the availability of equipment in the study settings; concerns among the health professionals targeted; and strategies used to promote uptake of the guideline. The authors concluded that more attention was needed to identify organisational barriers to change and to address these using tailored implementation strategies.
Methodological approach, sampling, data collection, and data analysis
The methodological approaches of the included studies were heterogeneous. Whereas 19 of the qualitative studies did not refer to any specific methodological approach, 11 mentioned approaches such as grounded theory, ethnography, action research, and narrative approaches. Ten studies used several methods for data collection (most often combinations of individual interviews and group discussions), 10 utilised individual interviews only, five used focus groups only, and two used different forms of observation. The remaining three studies were unpublished and we were not able to obtain further information on data collection from the authors.
A number of studies inadequately reported several aspects of the methods: 13 did not describe their sampling approach and the remainder used a mix of purposive, convenience, and random sampling. In 14 studies we could find no information on the approach used to analyse data. Where such information was reported, thematic or content analysis or framework analysis was utilised (n=10) and, more rarely, a grounded theory approach (n=2). Four studies used other approaches.
Links between qualitative studies and randomised controlled trials
Where the findings of the trial and qualitative studies were reported in separate papers, the link between the two was not always clear from the papers themselves. Sixteen of the qualitative studies shared authors with the report of the randomised controlled trial. Only nine papers explicitly described some level of linkage between the study teams.
In two of the studies the researchers stated that they had used a “mixed method” approach. Our review of the studies indicated some integration in the interpretation of the results from the trials and the qualitative research in eight cases although the analysis of the data was carried out separately. In most cases (n=13), however, we found no evidence of integration at the level of interpretation.
Quality of the qualitative studies
Ten qualitative studies (including the seven with no formal analysis of the qualitative data) did not provide sufficient data to allow assessment of methodological quality. Quality assessment was therefore carried out on 20 studies (table 2). This showed high variability, with the most common weaknesses including lack of a clear justification for the qualitative approach used (no information in 16 studies); inadequate descriptions of context, sampling, data collection, and analysis methods; little reflection on the researcher’s role in the research process (no information in 17 studies); lack of clarity on how ethical issues had been taken into consideration (no information in 15 studies), and insufficient evidence to support the claims made in the paper.
Table 2.
Methodological quality of 20 of 30 included qualitative studies that had sufficient information to carry out a quality assessment
| Quality criterion | Met criterion | Did not meet criterion | Unclear |
|---|---|---|---|
| Is this study qualitative research? | 18 | 0 | 2 |
| Are research questions clearly stated? | 18 | 2 | 0 |
| Have ethical issues been taken into consideration? | 4 | 1 | 15 |
| Is qualitative approach clearly justified? | 4 | 10 | 6 |
| Is approach appropriate for research question? | 17 | 0 | 3 |
| Is study context clearly described? | 8 | 9 | 3 |
| Is role of researcher clearly described? | 2 | 17 | 1 |
| Is sampling method clearly described? | 11 | 6 | 3 |
| Is sampling strategy appropriate for research question? | 8 | 0 | 12 |
| Is method of data collection clearly described? | 12 | 4 | 4 |
| Is data collection method appropriate to research question? | 16* | 0 | 4 |
| Is method of analysis clearly described? | 13 | 5 | 2 |
| Is analysis appropriate for research question? | 13 | 0 | 7 |
| Are claims made supported by sufficient evidence? | 11 | 6 | 3 |
*For five studies for which the details of methods of data collection were not clearly described or were unclear (for example, where and when data were collected), the data collection method itself, such as individual interviews or focus group discussions, was assessed as appropriate to research question.
We did not identify any relation between interventions that reported using an explicit theoretical framework and the quality of qualitative studies carried out alongside trials of those interventions.
Discussion
Qualitative studies undertaken alongside randomised controlled trials of interventions to change organisation and practice remain uncommon. Less than one third of recently completed trials of relatively complex interventions in the Cochrane Effective Practice and Organisation of Care register included some form of qualitative research. Of these, only about two thirds were published studies. This is surprising given the nature of these interventions and the growing awareness of the role that qualitative research can play in the design and evaluation of interventions.5 8 19 20 Furthermore, contacts with authors suggested that many valued the findings of qualitative studies. Why then are qualitative approaches not used more extensively alongside trials? Constraints on resources and poor access to relevant expertise were mentioned by study authors in response to our requests for information on qualitative studies. It has also been suggested that linear models for evaluating interventions may impede the use of qualitative approaches. These models, it is suggested, view such evaluation as passing through a series of phases from the development of hypotheses to efficacy trials and then effectiveness trials. This may contribute to the view that earlier phases of research, such as efficacy trials, do not need to incorporate qualitative studies to explore the effects of contextual and other moderating factors. Such methods are seen as important only in the later phases of evaluation.21
Although much has been written on qualitative process evaluation alongside trials of complex interventions, the largest group of qualitative studies identified were those carried out before trials. Firstly, this suggests that reviewers who aim to understand better the effects of interventions through examining qualitative process evaluations may locate little data. Secondly, it indicates the need for more attention to this aspect of trial design.
The rigour of qualitative studies undertaken alongside randomised controlled trials, or at least the reporting of methods used, is an important concern. We identified major shortcomings in many of these studies, particularly issues of sampling, analysis, and critical analysis of the researchers’ roles. Interestingly an explicit theoretical basis for the intervention was reported in over a third of cases—a higher proportion than reported in recent reviews on the use of theory in implementation research.22 23 Twice as many of the randomised controlled trials that included qualitative work also had a clearly specified theoretical basis (40%) compared with randomised controlled trials without any qualitative work (20%). However, the use of theory is by no means the norm in studies in this specialty (only 27% of randomised controlled trials did so explicitly) and it remains unclear whether interventions based explicitly on a particular theoretical approach are more likely to be effective than those designed using pragmatic processes.24 25 26
In our sample we found little evidence of explicit integration of data from qualitative studies and randomised controlled trials and few cases discussed mixed methods approaches. Such data could be integrated in several ways. Discussion of the trial findings could draw on both the qualitative and quantitative data in, for example, exploring reasons for success or failure of the intervention or for variation in effects across sites or individuals. Description of the intervention could also make explicit how qualitative approaches contributed, for example, to identifying barriers to change and developing the intervention. The extent of collaboration within trial teams between researchers from different disciplines, such as social scientists and epidemiologists, is another important aspect. The reported data did not, however, allow us to explore this adequately, and further work on the basis of case studies of trial teams is needed.
Limitations of the study
This study has several possible limitations. Firstly, we may not have identified all qualitative studies linked to the index randomised controlled trials. However, we did receive a high response rate from the authors of randomised controlled trials, and other reviews have indicated that this approach identifies the largest number of additional studies.23 All methods of identifying studies were resource intensive—a potential barrier to examining qualitative work done alongside trials. Secondly, trials sampled from the Cochrane Effective Practice and Organisation of Care database may not be representative of all randomised controlled trials evaluating interventions to change professional practice and the organisation of care. The sampled trials are unlikely to be representative of randomised controlled trials more widely but are likely to be similar, in terms of their use of qualitative methods, to other randomised controlled trials of complex interventions. Finally, our analysis is based largely on study reports. These may not reflect the extent of integration of qualitative and quantitative findings.
Conclusions
Although well conducted qualitative studies can support trial design and improve our understanding of the effects of complex interventions and the mechanisms through which changes occur, qualitative studies remain relatively uncommon alongside trials of complex interventions. Most of the qualitative studies were carried out before the trial, had important methodological shortcomings, and the findings were poorly integrated with those of the trials. This study highlights ways in which the quality and usefulness of qualitative studies carried out alongside randomised controlled trials can be improved (box 5). Further work is needed to develop further methodological and practical guidance for trial teams who plan to utilise qualitative approaches.8 27
Box 5 Improving the quality and usefulness of qualitative studies carried out alongside randomised controlled trials
There is potential for far greater use of qualitative approaches alongside randomised controlled trials of complex interventions. In many randomised controlled trials, opportunities to understand better the effects of interventions and how they are experienced by recipients are not fully utilised. Funders, journals, and other stakeholders should encourage trial teams to use and report qualitative approaches alongside randomised controlled trials and to consider the implications of these qualitative findings for interpretation of the trial results
Further efforts are needed to improve the reporting of qualitative studies, including the context of the research, the role of researchers, and the methods used
Methods need to be developed for linking qualitative studies and randomised controlled trials in medical databases, including efforts to ensure that qualitative studies carried out alongside trials include a universal trial reference number (www.who.int/ictrp/utrn/en/index.html), so that the studies can be located more easily
The ways in which quantitative and qualitative researchers work together in developing and evaluating interventions needs further exploration so as to understand better how to maximise the potential synergies between these different approaches
What is already known on this topic
Complex healthcare interventions involve social processes that can be difficult to explore using quantitative methods alone
Qualitative research can support the design of interventions and improve understanding of the mechanisms and effects of complex healthcare interventions
Increasing numbers of randomised trials of complex interventions are now thought to include qualitative components
What this study adds
Qualitative studies remain relatively uncommon alongside trials of complex healthcare interventions
Most of the qualitative studies identified were carried out before the trial so opportunities to understand better the effects of interventions and how they are experienced by recipients are not being fully utilised
Most of the qualitative studies had important methodological shortcomings and their findings were often poorly integrated with those of the trial in which they were nested
We thank Xavier Bosch-Capblanch, Anna Gaze, Marit Johansen, Matthew Oxman, and Marcus Prescott for their assistance with various aspects of the study.
Contributors: SL and ADO conceived the idea for the paper. SL and CG retrieved the articles, extracted the data, and did the analysis. SL, CG, and ADO drafted the paper. SL is the guarantor.
Funding: This work was supported by a UK Medical Research Council health services research training fellowship to SL. The Norwegian Knowledge Centre for the Health Services provided support to CG and ADO. The funding sources played no part in the conception of the study, data collection, analysis, or writing the article.
Competing interests: ADO was an investigator for two of the included trials and CG worked in the unit in which these trials were coordinated.
Ethical approval: Not required.
Cite this as: BMJ 2009;339:b3496
References
- 1.Campbell M, Fitzpatrick R, Haines A, Kinmonth A-L, Sandercock P, Spiegelhalter D, et al. Framework for design and evaluation of complex interventions to improve health. BMJ 2000;321:694-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. London: MRC, 2000.
- 3.Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Hawe P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. J Epidemiol Community Health 2004;58:788-93. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Team RS. Process evaluation in randomised controlled trials of complex interventions. BMJ 2006;332:413-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess 1998;2(16). [PubMed]
- 7.Pearson A. Cochrane Qualitative Research Methods Group. About the Cochrane Collaboration (Methods Groups). Cochrane Library 2007;(2):CE000142.
- 8.Rousseau N, McColl E, Eccles M, Hall L. Qualitative methods in implementation research. In: Thorsen T, Makela M, eds. Changing professional practice. Theory and practice of clinical guidelines implementation. Copenhagen: Danish Institute for Health Services Research and Development, 1999.
- 9.Gibson G, Timlin A, Curran S, Wattis J. The scope for qualitative methods in research and clinical trials in dementia. Age Ageing 2004;33:422-6. [DOI] [PubMed] [Google Scholar]
- 10.Donovan J, Mills N, Smith M, Brindle L, Jacoby A, Peters T, et al. Quality improvement report: improving design and conduct of randomised trials by embedding them in qualitative research: ProtecT (prostate testing for cancer and treatment) study. Commentary: presenting unbiased information to patients can be difficult. BMJ 2002;325:766-70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Wainberg ML, Alfredo Gonzalez M, McKinnon K, Elkington KS, Pinto D, Gruber Mann C, et al. Targeted ethnography as a critical step to inform cultural adaptations of HIV prevention interventions for adults with severe mental illness. Soc Sci Med 2007;65:296-308. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Bero L, Eccles M, Grilli R, Grimshaw J, Gruen R, Mayhew A, et al. Cochrane Effective Practice and Organisation of Care Group. About the Cochrane Collaboration (Cochrane Review Groups (CRGs)). Cochrane Library 2007;(1).
- 13.Cochrane Effective Practice and Organisation of Care Review Group. The data collection checklist. Ottawa: Cochrane Effective Practice and Organisation of Care Review Group, 2007.
- 14.Critical Appraisal Skills Programme (CASP). Making sense of evidence: 10 questions to help you make sense of qualitative research. 2007. www.phru.nhs.uk/learning/casp_qualitative_tool.pdf.
- 15.Harrison R, MacFarlane A, Wallace P, Murray E. Patients’ perceptions of joint teleconsultations: qualitative evaluation. Health Expect 2006;9:81-90. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Petty DR, Zermansky AG, Raynor DK, Lowe CJ, Buttress AD, Vail A, et al. “No thank you”: why elderly patients declined to participate in a research study. Pharm World Sci 2001;23:22-7. [DOI] [PubMed] [Google Scholar]
- 17.Davies BL, Hodnett E. Labor support: nurses’ self-efficacy and views about factors influencing implementation. J Obstet Gynecol Neonatal Nurs 2002;31:48-56. [DOI] [PubMed] [Google Scholar]
- 18.Graham ID, Logan J, Davies B, Nimrod C. Changing the use of electronic fetal monitoring and labor support: a case study of barriers and facilitators. Birth 2004;31:293-301. [DOI] [PubMed] [Google Scholar]
- 19.Oakley A, Strange V, Stephenson J, Forrest S, Monteiro H. Evaluating processes. A case study of a randomized controlled trial of sex education. Evaluation 2004;10:440-62. [Google Scholar]
- 20.Riley T, Hawe P, Shiell A. Contested ground: how should qualitative evidence inform the conduct of a community intervention trial? J Health Serv Res Policy 2005;10:103-10. [DOI] [PubMed] [Google Scholar]
- 21.Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health 2003;93:1261-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Eccles M, Grimshaw J, Walker A, Johnston M, Pitts N. Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. J Clin Epidemiol 2005;58:107-12. [DOI] [PubMed] [Google Scholar]
- 23.Farmer A, Grimshaw J, Mayhew A, McGowan J, Graham I, Driedger M, et al. Systematic reviews of knowledge translation interventions: contributions of process evaluations and contact with authors. CCOHTA final performance report. Ottawa: University of Ottawa, 2005.
- 24.Oxman AD, Fretheim A, Flottorp S. The OFF theory of research utilization. J Clin Epidemiol 2005;58:113-6; discussion 117-20. [DOI] [PubMed] [Google Scholar]
- 25.Perkins M, Jensen P, Jaccard J, Gollwitzer P, Oettingen G, Pappadopulos E, et al. Applying theory-driven approaches to understanding and modifying clinicians’ behavior: what do we know? Psychiatr Serv 2007;58:342-8. [DOI] [PubMed] [Google Scholar]
- 26.West R. Time for a change: putting the transtheoretical (stages of change) model to rest. Addiction 2005;100:1036-9. [DOI] [PubMed] [Google Scholar]
- 27.Grol R, Wensing M, Eccles M. Improving patient care. The implementation of change in clinical practice. Oxford: Butterworth Heineman, 2004.