Qualitative research in systematic reviews

The recent publication by the NHS Centre for Reviews and Dissemination of the second edition of the guidance on undertaking systematic reviews of research on effectiveness deserves to be warmly welcomed, for many reasons.¹ Perhaps chief among these is the increased recognition given to the diverse types of evidence that can contribute to systematic reviews. This suggests that the rigid insistence on controlled trials as the sole source of evidence on effectiveness that characterised the beginnings of the evidence based healthcare movement is fading. Qualitative research is now given explicit consideration in the new guidance. This is consistent with other recent recommendations emphasising the contribution of qualitative evidence to healthcare evaluation.² The argument for giving a place to qualitative research in systematic reviews seems to have been won. There remain several issues, however, that need to be addressed in making the role of qualitative evidence in reviews more systematic.

The move to recognise the potential value of qualitative research could do much to address arguments that evidence based health care has tended to focus on those variables that can be easily measured and has lacked a critical perspective, particularly with respect to social and educational interventions. For example, it is intuitively obvious that a recent Cochrane review could have benefited from using qualitative research to assess how we can improve communication with children and adolescents about their cancer.³ The outcome of the decision to focus solely on controlled trials and before and after studies was that only six of 1500 identified studies were included in the review, and few firm conclusions could be reached. Clearly a more inclusive view of what constitutes evidence is necessary to answer such complex questions, as well as acknowledgement of the explanatory power of non-quantitative forms of evidence.

Qualitative research has, of course, already contributed to a number of published reviews, though rarely of the Cochrane type and often in relatively informal ways. Several issues still need to be dealt with to make the role of qualitative evidence in reviews more systematic. Firstly, the centre's guidance rightly emphasises the need for rigour in the identification of research. However, despite efforts by the centre and the University of London's Institute of Education, among others, searching for and identifying appropriate qualitative research remains frustrating and difficult. This is partly because there is no equivalent of the Cochrane controlled trials register or other clinical trials registers for qualitative research (qualitative research is catalogued on a wide range of databases or not at all) and partly because indexes and search filters require substantial improvement. Investment in these areas is needed, especially if reviewers are to demonstrate that their searches are systematic and exhaustive.

Secondly, the problem of how to appraise the quality of qualitative studies remains. Directly applying the models developed for quantitative evidence is inappropriate: constructing hierarchies of evidence for types of qualitative research studies is clearly problematic, as the guidance highlighted, and undesirable. More generally, there has been a failure to agree on suitable methods for assessing the quality of qualitative research. This has inhibited the development of a process similar to CONSORT, which aims to improve the reporting of clinical trials.⁴ Qualitative research is a priority for the National Health Service's research and development methodology programme (inherited from the health technology assessment programme). We need to reach the stage soon where the accepted criteria provide guidelines for judging a paper, for deciding whether it should be included in a review, and on how to weight it.

Thirdly, a formidable question is how to make qualitative evidence—which may be produced with widely varying theoretical perspectives and diverse analytical approaches—submit to the disciplines of secondary summary and synthesis. More progress must be made on methods for synthesising qualitative data from across studies and synthesising qualitative and quantitative data. A daunting array of theoretical and practical problems awaits reviewers who attempt the secondary manipulation of the concepts or themes that are the staple product of qualitative research. A natural tension exists between an approach that relies on interpretation and reflection (qualitative research) and an approach that seeks to expunge the potential for anarchy associated with such ungovernable processes (the systematic review). Bayesian approaches to meta-analysis offer hope of synthesising qualitative—or qualitative and quantitative—forms of data, by treating qualitative research as a resource for identifying variables for synthesis and attaching weights to the strength of evidence associated with those variables.⁵ However promising these approaches are, systematic means of more narrative based and other forms of synthesis for qualitative research are clearly needed.

A place for qualitative research in systematic reviews now seems established. A Cochrane qualitative methods network has existed since 1998. Like other groups that seek to move forward to a more inclusive view of evidence that nevertheless remains systematic, it has its work—valuable as it is—cut out.