Skip to main content
Epidemiology and Health logoLink to Epidemiology and Health
. 2014 Sep 11;36:e2014018. doi: 10.4178/epih/e2014018

Narrative reviews

Jong-Myon Bae 1,
PMCID: PMC4207097  PMID: 25223334

Abstract

Although qualitative researches (QR) are invaluable in understanding complex healthcare situations, the quantitative systematic reviews could not treat them. To improve quality of healthcare services, results of QR should be considered in healthcare decision-making processes. Several methods and theories for synthesizing evidences of QR have been developed. In order to activate the narrative reviews and mixed methods reviews in Korean healthcare academies, I arranged the related nomenclatures and suggested some issues to conduct them.

Keywords: Review literature as topic, Qualitative research, Qualitative evaluation, Meta-analysis as topic, Evidence-based practice

INTRODUCTION

In providing healthcare services, the evidence-based medicine (EBM) paradigm [1-3]—the idea that healthcare-related decisions should be made on the basis of the best evidence—is expanding to healthcare-related policy-making in nursing and health science as well as medicine [4-6]. Amidst this trend, quantitative systematic reviews is a new research approach. It is a term used to designate the approach to examining the effects of a treatment or an intervention adopting a statistical method called meta-analysis [7]. However, the results of qualitative research which cannot obtain quantitative outcomes, such as relative risk, odds ratio, and number needed to treat (NNT), tend to be ignored or excluded in the application of the systematic reviews process [8].

Not only the definition of qualitative research, but also the characteristic comparisons between qualitative and quantitative research are well documented in the papers by Murphy et al. [9] and Draper [10]. In short, qualitative research can be a useful guide for understanding complicated situations in the real world and can serve as groundwork for new hypotheses [11-15]. Consequently, efforts to use the synthesis of qualitative research results as the grounds for decision-making were already underway in 1990s when EBM emerged [16-23]. However, recently, in line with the trend toward improving medical treatment quality through patient-centered and evidence-based diagnostic and therapeutic services [24,25], efforts for complementary utilization of qualitative research results are experiencing a revival [13, 15,26-34]. In particular, the claim that both quantitative and qualitative research should be utilized for proper understanding of overall healthcare problems is gaining influence. [18,35,36].

Meanwhile, because the synthesis of qualitative research results is inevitably different from a quantitative systematic reviews, there have been many attempts to overcome this gap [8, 37-40]. For this reason, I would like to systemize the attempts to synthesize qualitative research results undertaken so far. This work is expected to show the framework of the discussions related to qualitative research and lay a cornerstone for the vitalization of systematic reviews on qualitative research in South Korea.

MAIN BODY

Terms related to the synthesis of qualitative research

Among the theories about qualitative research presented so far, works of Draper [10], Barnett-Page & Thomas [41], Dixon-Woods et al. [42], and Thorne et al. [43] may be representative. I arranged various terms related to qualitative research, grouping them according to the emphasis intended.Appendix 1 is the overview of the healthcare research applying this nomenclature. At a glance, we can see many terms are suggested with the emphasis placed on the meaning of the term “synthesis” of qualitative research results in contrast to the application of the meta-analytic statistical method of quantitative research results. In particular, we can see the term “mixed methods research” has often been applied recently in the attempt to include both quantitative and qualitative research [44].

Process of synthesis of qualitative research

As can be seen in Appendix 1, the high number of related terms implies that the establishment of the relevant research methodology is a difficult task [45]. Sinuff et al. [46] shows the difference between quantitative and qualitative research processes in a diagram. However, a look at the suggested processes in relation to qualitative research synthesis [34,45,47-54] reveals that they stick to the big frame of “Ask - Acquire - Appraise - Apply - Assess,” 5A of the evidence cycle, although they show a certain diversity [55].

Searches of qualitative research to collect a body of related literature are more difficult than quantitative research [56-58]. This is because of different database services and the need to search for gray literature that has been issued but not officially published and made available in market, such as reports published by institutions or academic narrative reports [8]. It is also attributable to the need to resort, in addition to securing lists through search formulae, to hand searching which involves bibliography browsing in search of related papers and snowballing searching which traces one paper after another in chronological order [14,59].

Because of the diversity of research methods and fields of application, qualitative research does not easily lend itself to standardizing the items of qualitative evaluation in the literature of interest [8,59-64]. Nevertheless, the following achievements harvested so far deserve to be listed: (1) Thomas et al. [65] suggested evaluation items which matched quantitative-qualitative research. (2) Clark [66] developed ‘RATS’ evaluation tool, which is an acronym for Relevance, Appropriateness, Transparency, and Soundness. (3) Daly et al. [67] suggested a stratified structure by the contents of qualitative research. (4) Rodgers et al. [34] applied the EPPI approach that evaluates the persuasive power of evidence. And (5) Dixon-Woods et al. [68] suggested a tool called CASP (Critical Appraisal Skills Programme tool) which is composed of 10 items.

For the synthesis of evaluation results, a best fit frameworks is established [65,69-73] or a simulations model is selected [74]. The commercial program called NVivo has been developed [52, 75,76]. Feasibility research examining the applicability of this program in Korean society needs to be accumulated.

Also, a reporting guideline named RAMESES (Realist And MEta-narrative Evidence Synthesis: Evolving Standards) has been developed for application when the research results of systematic reviews of qualitative research are to be reported in papers [77,78]. Making a flow chart is also suggested in cases where the mixed methods reviews approach is adopted which involves both quantitative and qualitative research [79,80].

CONCLUSION AND PROPOSAL

There are few systematic reviews on qualitative research in healthcare-related scholarship in South Korea. For the qualitative improvement of healthcare in this scarce situation, research in various healthcare fields should be conducted. Therefore, I suggest three things as below.

First, establishing the nomenclature of systematic reviews on qualitative research is an urgent task, because systematic and coherent use of well-established terms is important when a multitude of suggested terms are in use, as shown in Appendix 1. The synthesis of quantitative research through the meta-analytic statistical method can be termed ‘quantitative systematic reviews’ and that of qualitative research, ‘qualitative systematic reviews’ [81]. However, given the current situation that systematic reviews have been established as the major research methodology of the synthesis of the evidences of quantitative research and the meta-analysis applied to this is recognized as statistical methodology [81], systematic reviews in the narrow sense mean quantitative systematic reviews [82]. On the contrary, qualitative systematic reviews is also called narrative systematic reviews and recently in a more abbreviated form called “narrative reviews” [29,83-85]. However, the term ‘meta-narrative reviews’ [77,78] does not fit and its use should be avoided because the meta-analysis corresponding to this is a statistical method, not a research method [81]. Therefore, I suggest a terminological differentiation between (quantitative) systematic reviews and (qualitative) narrative reviews, whereby the term “mixed methods reviews” may be used when both quantitative and qualitative research are involved. It was in consideration of this point that I titled this study “Narrative Reviews.”

Second, the practical way to revitalize narrative reviews research when Korean researchers do not have much experience conducting narrative reviews is the critical reading of good research cases from various academic fields and their application to practical use. To facilitate this process, I organized useful research cases by academic field and presented them in Appendix 2.

Third, there is a need to establish a research-supporting organization and expand research human resources for qualitative research in the process of planning and conducting clinical studies [15]. The basic prior condition of proper narrative reviews is good results from qualitative research. First and foremost, given the fact that multidisciplinary cooperation is of vital importance for qualitative research, an organizational reshuffling appears necessary to facilitate efficient cooperation.

Acknowledgments

This study was conducted by a 2014 academic promotion research funding project grant of Jeju National University.

Appendix 1. Summary tables of nomenclatures about methodologies for synthesis of qualitative researches

To Nomenclatures [References]
Emphasize qualitative synthesis Narrative summary [A01, A02]
Thematic synthesis [A03]
Textual narrative synthesis [A04]
Critical interpretive synthesis [A05, A06]
Framework synthesis [A07]
Realist synthesis [A08, A09]
Contrast quantitative systematic review Qualitative systematic review [A10,A11]
Synthesis of qualitative research [A12]
Narrative systematic review [A13]
Qualitative comparative analysis [A14, A15]
Qualitative meta-synthesis [A16]
Qualitative meta-summary [A17]
Contrast meta-analysis Meta-synthesis [A18, A19]
Meta-narrative review [A20, A21]
Meta-ethnography [A22, A23]
Meta-study [A24]
Meta-interpretation [A25]
Include quantitative & qualitative researches Mixed methods research [A26-A29]
Ecological triangulation [A01]

References

  • A01.Abbott A. Conceptions of time and events in social science methods: causal and narrative approaches. Hist Methods. 1990;23:140–150. [Google Scholar]
  • A02.Fairbank L, O'Meara S, Renfrew MJ, Woolridge M, Sowden AJ, Lister-Sharp D. A systematic review to evaluate the effectiveness of interventions to promote the initiation of breastfeeding. Health Technol Assess. 2000;4:1–171. [PubMed] [Google Scholar]
  • A03.Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol. 2008;8:45. doi: 10.1186/1471-2288-8-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A04.Lucas PJ, Baird J, Arai L, Law C, Roberts HM. Worked examples of alternative methods for the synthesis of qualitative and quantitative research in systematic reviews. BMC Med Res Methodol. 2007;7:4. doi: 10.1186/1471-2288-7-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A05.Dixon-Woods M, Cavers D, Agarwal S, Annandale E, Arthur A, Harvey J, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol. 2006;6:35. doi: 10.1186/1471-2288-6-35. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A06.Flemming K. Synthesis of quantitative and qualitative research: an example using Critical Interpretive Synthesis. J Adv Nurs. 2010;66:201–217. doi: 10.1111/j.1365-2648.2009.05173.x. [DOI] [PubMed] [Google Scholar]
  • A07.Oliver SR, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008;11:72–84. doi: 10.1111/j.1369-7625.2007.00476.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A08.Pawson R. Evidence-based policy: the promise of ‘Realist Synthesis’. Evaluation. 2002;8:340–358. [Google Scholar]
  • A09.Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21–34. doi: 10.1258/1355819054308530. [DOI] [PubMed] [Google Scholar]
  • A10.Flemming K. Synthesis of qualitative research and evidence-based nursing. Br J Nurs. 2007;16:616–620. doi: 10.12968/bjon.2007.16.10.23510. [DOI] [PubMed] [Google Scholar]
  • A11.Timulak L. Meta-analysis of qualitative studies: a tool for reviewing qualitative research findings in psychotherapy. Psychother Res. 2009;19:591–600. doi: 10.1080/10503300802477989. [DOI] [PubMed] [Google Scholar]
  • A12.Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59. doi: 10.1186/1471-2288-9-59. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A13.Roundtree AK, Kallen MA, Lopez-Olivo MA, Kimmel B, Skidmore B, Ortiz Z, et al. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. J Clin Epidemiol. 2009;62:128–137. doi: 10.1016/j.jclinepi.2008.08.003. [DOI] [PubMed] [Google Scholar]
  • A14.Haggerty TR. Unravelling patterns of multiple conjunctural causation in comparative research: Ragin’s qualitative comparative method. J Comp Phys Educ Sports. 1992;14:19–27. [Google Scholar]
  • A15.Cress DM, Snow DA. The Outcomes of homeless mobilization: the influence of organization, disruption, political mediation, and framing. Am J Sociol. 2000;105:1063–1104. [Google Scholar]
  • A16.Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M. Qualitative metasynthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004;14:1342–1365. doi: 10.1177/1049732304269888. [DOI] [PubMed] [Google Scholar]
  • A17.Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59. doi: 10.1186/1471-2288-9-59. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A18.Sandelowski M, Docherty S, Emden C. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59. doi: 10.1186/1471-2288-9-59. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A19.Walsh D, Downe S. Meta-synthesis method for qualitative research: a literature review. J Adv Nurs. 2005;50:204–211. doi: 10.1111/j.1365-2648.2005.03380.x. [DOI] [PubMed] [Google Scholar]
  • A20.Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629. doi: 10.1111/j.0887-378X.2004.00325.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A21.Abu Dabrh AM, Firwana B, Cowl CT, Steinkraus LW, Prokop LJ, Murad MH. Health assessment of commercial drivers: a meta-narrative systematic review. BMJ Open. 2014;4:e2014018. doi: 10.1136/bmjopen-2013-003434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A22.Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, et al. Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Soc Sci Med. 2003;56:671–684. doi: 10.1016/s0277-9536(02)00064-3. [DOI] [PubMed] [Google Scholar]
  • A23.Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill R, et al. Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess. 2011;15:1–164. doi: 10.3310/hta15430. [DOI] [PubMed] [Google Scholar]
  • A24.Thorne S, Paterson B, Acorn S, Canam C, Joachim G, Jillings C. Chronic illness experience: insights from a metastudy. Qual Health Res. 2002;12:437–452. doi: 10.1177/104973202129120007. [DOI] [PubMed] [Google Scholar]
  • A25.Finfgeld DL. Courage as a process of pushing beyond the struggle. Qual Health Res. 1999;9:803–814. doi: 10.1177/104973299129122298. [DOI] [PubMed] [Google Scholar]
  • A26.Sandelowski M, Voils CI, Barroso J. Defining and designing mixed research synthesis studies. Res Sch. 2006;13(1):29. [PMC free article] [PubMed] [Google Scholar]
  • A27.Flemming K. The knowledge base for evidence-based nursing: a role for mixed methods research? ANS Adv Nurs Sci. 2007;30:41–51. doi: 10.1097/00012272-200701000-00005. [DOI] [PubMed] [Google Scholar]
  • A28.Sandelowski M, Voils CI, Leeman J, Crandell JL. Mapping the mixed methods-mixed research synthesis terrain. J Mix Methods Res. 2012;6:317–331. doi: 10.1177/1558689811427913. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • A29.Seymour J. Combined qualitative and quantitative research designs. Curr Opin Support Palliat Care. 2012;6:514–524. doi: 10.1097/SPC.0b013e328358214e. [DOI] [PubMed] [Google Scholar]

Appendix 2. Some articles related to qualitative researches about health care services

Domain [References]
Preventive medicine Pharmacoepidemiology [B01, B02]
Occupation [B03]
Hospital management [B04]
Clinical medicine Obstetrics [B05]
Hospice [B06]
Patient-doctor relationship [B07]
Nurse Diabetic care [B08, B09]
Insight into cancer [B10]
Health promotion Mental health [B11]
School health [B12-B15]
Health program [B16-B20]
Vaccine program [B21]
Health policy Policy decision [B22]
Control plan of tuberculosis [B23]

References

  • B01.Pound P, Britten N, Morgan M, Yardley L, Pope C, Daker-White G, et al. Resisting medicines: a synthesis of qualitative studies of medicine taking. Soc Sci Med. 2005;61:133–155. doi: 10.1016/j.socscimed.2004.11.063. [DOI] [PubMed] [Google Scholar]
  • B02.Teasdale E, Santer M, Geraghty AW, Little P, Yardley L. Public perceptions of non-pharmaceutical interventions for reducing transmission of respiratory infection: systematic review and synthesis of qualitative studies. BMC Public Health. 2014;14:589. doi: 10.1186/1471-2458-14-589. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B03.Ruggiero JS, Redeker NS. Effects of napping on sleepiness and sleep-related performance deficits in night-shift workers: a systematic review. Biol Res Nurs. 2014;16:134–142. doi: 10.1177/1099800413476571. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B04.Mousazadeh Y, Jannati A, Jabbari Beiramy H, Asgharijafarabadi M, Ebadi A. Advantages and disadvantages of different methods of hospitals’ downsizing: a narrative systematic review. Health Promot Perspect. 2013;3:276–287. doi: 10.5681/hpp.2013.032. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B05.Boath E, Bradley E, Henshaw C. The prevention of postnatal depression: a narrative systematic review. J Psychosom Obstet Gynaecol. 2005;26:185–192. doi: 10.1080/01674820400028431. [DOI] [PubMed] [Google Scholar]
  • B06.Flemming K. The use of morphine to treat cancer-related pain: a synthesis of quantitative and qualitative research. J Pain Symptom Manage. 2010;39:139–154. doi: 10.1016/j.jpainsymman.2009.05.014. [DOI] [PubMed] [Google Scholar]
  • B07.Santer M, Ring N, Yardley L, Geraghty AW, Wyke S. Treatment non-adherence in pediatric long-term medical conditions: systematic review and synthesis of qualitative studies of caregivers' views. BMC Pediatr. 2014;14:63. doi: 10.1186/1471-2431-14-63. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B08.Davidson MB. The effectiveness of nurse- and pharmacist-directed care in diabetes disease management: a narrative review. Curr Diabetes Rev. 2007;3:280–286. doi: 10.2174/157339907782330058. [DOI] [PubMed] [Google Scholar]
  • B09.Davidson MB. Detailed treatment algorithms for effective nurse- and pharmacist-directed diabetes care: a personal approach. Diabetes Educ. 2009;35:61–71. doi: 10.1177/0145721708327287. [DOI] [PubMed] [Google Scholar]
  • B10.Jayde V, White K, Blomfield P. Symptoms and diagnostic delay in ovarian cancer: a summary of the literature. Contemp Nurse. 2009;34:55–65. doi: 10.5172/conu.2009.34.1.055. [DOI] [PubMed] [Google Scholar]
  • B11.Leamy M, Bird V, Le Boutillier C, Williams J, Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. Br J Psychiatry. 2011;199:445–452. doi: 10.1192/bjp.bp.110.083733. [DOI] [PubMed] [Google Scholar]
  • B12.Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, et al. Integrating qualitative research with trials in systematic reviews. BMJ. 2004;328:1010–1012. doi: 10.1136/bmj.328.7446.1010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B13.Harden A, Brunton G, Fletcher A, Oakley A. Teenage pregnancy and social disadvantage: systematic review integrating controlled trials and qualitative studies. BMJ. 2009;339:b4254. doi: 10.1136/bmj.b4254. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B14.Bonell C, Harden A, Wells H, Jamal F, Fletcher A, Petticrew M, et al. Protocol for a systematic review of the effects of schools and school-environment interventions on health: evidence mapping and syntheses. BMC Public Health. 2011;11:453. doi: 10.1186/1471-2458-11-453. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B15.Jamal F, Fletcher A, Harden A, Wells H, Thomas J, Bonell C. The school environment and student health: a systematic review and meta-ethnography of qualitative research. BMC Public Health. 2013;13:798. doi: 10.1186/1471-2458-13-798. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B16.Liu J, Davidson E, Bhopal R, White M, Johnson M, Netto G, et al. Adapting health promotion interventions to meet the needs of ethnic minority groups: mixed-methods evidence synthesis. Health Technol Assess. 2012;16:1–469. doi: 10.3310/hta16440. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B17.Neville LM, O'Hara B, Milat A. Computer-tailored physical activity behavior change interventions targeting adults: a systematic review. Int J Behav Nutr Phys Act. 2009;6:30. doi: 10.1186/1479-5868-6-30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B18.Neville LM, Milat AJ, O'Hara B. Computer-tailored weight reduction interventions targeting adults: a narrative systematic review. Health Promot J Austr. 2009;20:48–57. doi: 10.1071/he09048. [DOI] [PubMed] [Google Scholar]
  • B19.Neville LM, O'Hara B, Milat AJ. Computer-tailored dietary behaviour change interventions: a systematic review. Health Educ Res. 2009;24:699–720. doi: 10.1093/her/cyp006. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B20.Thomas R, Perera R. School-based programmes for preventing smoking. Cochrane Database Syst Rev. 2006;(3):e2014018. doi: 10.1002/14651858.CD001293.pub2. [DOI] [PubMed] [Google Scholar]
  • B21.Mills E, Jadad AR, Ross C, Wilson K. Systematic review of qualitative studies exploring parental beliefs and attitudes toward childhood vaccination identifies common barriers to vaccination. J Clin Epidemiol. 2005;58:1081–1088. doi: 10.1016/j.jclinepi.2005.09.002. [DOI] [PubMed] [Google Scholar]
  • B22.Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q. 2010;88:444–483. doi: 10.1111/j.1468-0009.2010.00608.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • B23.Noyes J, Popay J. Directly observed therapy and tuberculosis: how can a systematic review of qualitative research contribute to improving services? A qualitative meta-synthesis. J Adv Nurs. 2007;57:227–243. doi: 10.1111/j.1365-2648.2006.04092.x. [DOI] [PubMed] [Google Scholar]

Footnotes

The author has no conflicts of interest to declare for this study

SUPPLEMENTARY MATERIAL

Supplementary material is available at http://www.e-epih.org/.

REFERENCES

  • 1.Miles A, Bentley P, Polychronis A, Grey J. Evidence-based medicine: why all the fuss? This is why. J Eval Clin Pract. 1997;3:83–86. doi: 10.1046/j.1365-2753.1997.00103.x. [DOI] [PubMed] [Google Scholar]
  • 2.Miles A, Bentley P, Polychronis A, Grey J, Price N. Recent progress in health services research: on the need for evidence-based debate. J Eval Clin Pract. 1998;4:257–265. doi: 10.1111/j.1365-2753.1998.tb00084.x. [DOI] [PubMed] [Google Scholar]
  • 3.Miles A, Bentley P, Polychronis A, Grey J, Price N. Advancing the evidence-based healthcare debate. J Eval Clin Pract. 1999;5:97–101. doi: 10.1046/j.1365-2753.1999.00205.x. [DOI] [PubMed] [Google Scholar]
  • 4.King S, Forbes K, Hanks GW, Ferro CJ, Chambers EJ. A systematic review of the use of opioid medication for those with moderate to severe cancer pain and renal impairment: a European Palliative Care Research Collaborative opioid guidelines project. Palliat Med. 2011;25:525–552. doi: 10.1177/0269216311406313. [DOI] [PubMed] [Google Scholar]
  • 5.Fox DM. Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions. Health Aff (Millwood) 2005;24:114–122. doi: 10.1377/hlthaff.24.1.114. [DOI] [PubMed] [Google Scholar]
  • 6.Greenhalgh T, Wong G, Westhorp G, Pawson R. Protocol--realist and meta-narrative evidence synthesis: evolving standards (RAMESES) BMC Med Res Methodol. 2011;11:115. doi: 10.1186/1471-2288-11-115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Bae JM. Methodological issues for determining intervals of subsequent cancer screening. Epidemiol Health. 2014;36:e2014018. doi: 10.4178/epih/e2014010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Jones K. Mission drift in qualitative research, or moving toward a systematic review of qualitative studies, moving back to a more systematic narrative review. Qual Rep. 2004;9:95–112. [Google Scholar]
  • 9.Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology assessment: a review of the literature. Health Technol Assess. 1998;2:iii-ix:1–274. [PubMed] [Google Scholar]
  • 10.Draper AK. The principles and application of qualitative research. Proc Nutr Soc. 2004;63:641–646. doi: 10.1079/pns2004397. [DOI] [PubMed] [Google Scholar]
  • 11.Jones R. Why do qualitative research? BMJ. 1995;311:2. doi: 10.1136/bmj.311.6996.2. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Murthy GV, Gupta SK. Qualitative research in ophthalmic sciences. Indian J Ophthalmol. 1999;47:257–264. [PubMed] [Google Scholar]
  • 13.Sofaer S. Qualitative methods: what are they and why use them? Health Serv Res. 1999;34:1101–1118. [PMC free article] [PubMed] [Google Scholar]
  • 14.Greenhalgh T, Heath I. Measuring quality in the therapeutic relationship--part 1: objective approaches. Qual Saf Health Care. 2010;19:475–478. doi: 10.1136/qshc.2010.043364. [DOI] [PubMed] [Google Scholar]
  • 15.Rapport F, Storey M, Porter A, Snooks H, Jones K, Peconi J, et al. Qualitative research within trials: developing a standard operating procedure for a clinical trials unit. Trials. 2013;14:54. doi: 10.1186/1745-6215-14-54. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Popay J, Rogers A, Williams G. Rationale and standards for the systematic review of qualitative literature in health services research. Qual Health Res. 1998;8:341–351. doi: 10.1177/104973239800800305. [DOI] [PubMed] [Google Scholar]
  • 17.Barbour RS. Mixing qualitative methods: quality assurance or qualitative quagmire? Qual Health Res. 1998;8:352–361. doi: 10.1177/104973239800800306. [DOI] [PubMed] [Google Scholar]
  • 18.McDowell I, MacLean L. Blending qualitative and quantitative study methods in health services research. Health Inform J. 1998;4:15–22. [Google Scholar]
  • 19.Green J, Britten N. Qualitative research and evidence based medicine. BMJ. 1998;316:1230–1232. doi: 10.1136/bmj.316.7139.1230. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Harding G, Gantley M. Qualitative methods: beyond the cookbook. Fam Pract. 1998;15:76–79. doi: 10.1093/fampra/15.1.76. [DOI] [PubMed] [Google Scholar]
  • 21.Popay J, Williams G. Qualitative research and evidence-based healthcare. J R Soc Med. 1998;91 Suppl 35:32–37. doi: 10.1177/014107689809135s08. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Dixon-Woods M, Fitzpatrick R. Qualitative research in systematic reviews. Has established a place for itself. BMJ. 2001;323:765–766. doi: 10.1136/bmj.323.7316.765. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Donovan J, Mills N, Smith M, Brindle L, Jacoby A, Peters T, et al. Quality improvement report: Improving design and conduct of randomised trials by embedding them in qualitative research: ProtecT (prostate testing for cancer and treatment) study. Commentary: presenting unbiased information to patients can be difficult. BMJ. 2002;325:766–770. doi: 10.1136/bmj.325.7367.766. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Bae JM. Global trends in the use of nationwide big data for solving healthcare problems. J Korean Med Assoc. 2014;57:386–390. (Korean) [Google Scholar]
  • 25.Mohammed K, Nolan MB, Rajjo T, Shah ND, Prokop LJ, Varkey P, et al. Creating a patient-centered health care delivery system: a systematic review of health care quality from the patient perspective. Am J Med Qual. 2014 doi: 10.1177/1062860614545124. (in press) [DOI] [PubMed] [Google Scholar]
  • 26.Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review--a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10 Suppl 1:21–34. doi: 10.1258/1355819054308530. [DOI] [PubMed] [Google Scholar]
  • 27.Barbour RS. The case for combining qualitative and quantitative approaches in health services research. J Health Serv Res Policy. 1999;4:39–43. doi: 10.1177/135581969900400110. [DOI] [PubMed] [Google Scholar]
  • 28.Morgan DL. Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res. 1998;8:362–376. doi: 10.1177/104973239800800307. [DOI] [PubMed] [Google Scholar]
  • 29.Collins JA, Fauser BC. Balancing the strengths of systematic and narrative reviews. Hum Reprod Update. 2005;11:103–104. doi: 10.1093/humupd/dmh058. [DOI] [PubMed] [Google Scholar]
  • 30.Moffatt S, White M, Mackintosh J, Howel D. Using quantitative and qualitative data in health services research - what happens when mixed method findings conflict? [ISRCTN61522618] BMC Health Serv Res. 2006;6:28. doi: 10.1186/1472-6963-6-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.Martin M. The need for an overall strategy. J Health Serv Res Policy . 2005;10 Suppl:49–50. doi: 10.1258/1355819054308567. [DOI] [PubMed] [Google Scholar]
  • 32.Flemming K. The synthesis of qualitative research and evidence-based nursing. Evid Based Nurs. 2007;10:68–71. doi: 10.1136/ebn.10.3.68. [DOI] [PubMed] [Google Scholar]
  • 33.Jack SM. Utility of qualitative research findings in evidence-based public health practice. Public Health Nurs. 2006;23:277–283. doi: 10.1111/j.1525-1446.2006.230311.x. [DOI] [PubMed] [Google Scholar]
  • 34.Rodgers M, Sowden A, Petticrew M, Arai L, Roberts H, Britten N, et al. Testing methodological guidance on the conduct of narrative synthesis in systematic reviews: effectiveness of interventions to promote smoke alarm ownership and function. Evaluation. 2009;15:47–71. [Google Scholar]
  • 35.Black N. Why we need qualitative research. J Epidemiol Community Health. 1994;48:425–426. doi: 10.1136/jech.48.5.425-a. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36.Berwick DM. The science of improvement. JAMA. 2008;299:1182–1184. doi: 10.1001/jama.299.10.1182. [DOI] [PubMed] [Google Scholar]
  • 37.Hoddinott P, Pill R. A review of recently published qualitative research in general practice More methodological questions than answers? Fam Pract. 1997;14:313–319. doi: 10.1093/fampra/14.4.313. [DOI] [PubMed] [Google Scholar]
  • 38.Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ. 2001;322:1115–1117. doi: 10.1136/bmj.322.7294.1115. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Dixon-Woods M, Fitzpatrick R, Roberts K. Including qualitative research in systematic reviews: opportunities and problems. J Eval Clin Pract. 2001;7:125–133. doi: 10.1046/j.1365-2753.2001.00257.x. [DOI] [PubMed] [Google Scholar]
  • 40.Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496. doi: 10.1136/bmj.b3496. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol. 2009;9:59. doi: 10.1186/1471-2288-9-59. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy. 2005;10:45–53. doi: 10.1177/135581960501000110. [DOI] [PubMed] [Google Scholar]
  • 43.Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M. Qualitative metasynthesis: reflections on methodological orientation and ideological agenda. Qual Health Res. 2004;14:1342–1365. doi: 10.1177/1049732304269888. [DOI] [PubMed] [Google Scholar]
  • 44.O’Cathain A, Murphy E, Nicholl J. Why, and how, mixed methods research is undertaken in health services research in England: a mixed methods study. BMC Health Serv Res. 2007;7:85. doi: 10.1186/1472-6963-7-85. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Forbes A, Griffiths P. Methodological strategies for the identification and synthesis of ‘evidence’ to support decision-making in relation to complex healthcare systems and practices. Nurs Inq. 2002;9:141–155. doi: 10.1046/j.1440-1800.2002.00146.x. [DOI] [PubMed] [Google Scholar]
  • 46.Sinuff T, Cook DJ, Giacomini M. How qualitative research can contribute to research in the intensive care unit. J Crit Care. 2007;22:104–111. doi: 10.1016/j.jcrc.2007.03.001. [DOI] [PubMed] [Google Scholar]
  • 47.Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629. doi: 10.1111/j.0887-378X.2004.00325.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 48.Abu Dabrh AM, Firwana B, Cowl CT, Steinkraus LW, Prokop LJ, Murad MH. Health assessment of commercial drivers: a meta-narrative systematic review. BMJ Open. 2014;4:e2014018. doi: 10.1136/bmjopen-2013-003434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy. 2005;10 Suppl 1:6–20. doi: 10.1258/1355819054308576. [DOI] [PubMed] [Google Scholar]
  • 50.Flemming K. Synthesis of quantitative and qualitative research: an example using Critical Interpretive Synthesis. J Adv Nurs. 2010;66:201–217. doi: 10.1111/j.1365-2648.2009.05173.x. [DOI] [PubMed] [Google Scholar]
  • 51.Walsh D, Downe S. Meta-synthesis method for qualitative research: a literature review. J Adv Nurs. 2005;50:204–211. doi: 10.1111/j.1365-2648.2005.03380.x. [DOI] [PubMed] [Google Scholar]
  • 52.Leamy M, Bird V, Le Boutillier C, Williams J, Slade M. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. Br J Psychiatry. 2011;199:445–452. doi: 10.1192/bjp.bp.110.083733. [DOI] [PubMed] [Google Scholar]
  • 53.Harden A, Garcia J, Oliver S, Rees R, Shepherd J, Brunton G, et al. Applying systematic review methods to studies of people’s views: an example from public health research. J Epidemiol Community Health. 2004;58:794–800. doi: 10.1136/jech.2003.014829. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54.Tricco AC, Tetzlaff J, Moher D. The art and science of knowledge synthesis. J Clin Epidemiol. 2011;64:11–20. doi: 10.1016/j.jclinepi.2009.11.007. [DOI] [PubMed] [Google Scholar]
  • 55.Bae JM, Park BJ, Ahn YO. Perspectives of clinical epidemiology in Korea. J Korean Med Assoc. 2013;56:718–723. [Google Scholar]
  • 56.Evans D. Database searches for qualitative research. J Med Libr Assoc. 2002;90:290–293. [PMC free article] [PubMed] [Google Scholar]
  • 57.Flemming K, Briggs M. Electronic searching to locate qualitative research: evaluation of three strategies. J Adv Nurs. 2007;57:95–100. doi: 10.1111/j.1365-2648.2006.04083.x. [DOI] [PubMed] [Google Scholar]
  • 58.Shaw RL, Booth A, Sutton AJ, Miller T, Smith JA, Young B, et al. Finding qualitative research: an evaluation of search strategies. BMC Med Res Methodol. 2004;4:5. doi: 10.1186/1471-2288-4-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 59.Methley AM, Chew-Graham C, Campbell S, Cheraghi-Sohi S. Experiences of UK health-care services for people with Multiple Sclerosis: a systematic narrative review. Health Expect. 2014 doi: 10.1111/hex.12228. doi: 10.1111/hex.12228. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Mays N, Pope C. Qualitative research in health care. Assessing quality in qualitative research. BMJ. 2000;320:50–52. doi: 10.1136/bmj.320.7226.50. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61.Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative research. Qual Saf Health Care. 2004;13:223–225. doi: 10.1136/qshc.2003.008714. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Reynolds J, Kizito J, Ezumah N, Mangesho P, Allen E, Chandler C. Quality assurance of qualitative research: a review of the discourse. Health Res Policy Syst. 2011;9:43. doi: 10.1186/1478-4505-9-43. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 63.Boeije HR, van Wesel F, Alisic E. Making a difference: towards a method for weighing the evidence in a qualitative synthesis. J Eval Clin Pract. 2011;17:657–663. doi: 10.1111/j.1365-2753.2011.01674.x. [DOI] [PubMed] [Google Scholar]
  • 64.Crossland L, Janamian T, Jackson CL. Key elements of high-quality practice organisation in primary health care: a systematic review. Med J Aust. 2014;201:S47–S51. doi: 10.5694/mja14.00305. [DOI] [PubMed] [Google Scholar]
  • 65.Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, et al. Integrating qualitative research with trials in systematic reviews. BMJ. 2004;328:1010–1012. doi: 10.1136/bmj.328.7446.1010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66.Clark JP. How to peer review a qualitative manuscript. In: Godlee F, Jefferson Tm, editors. Peer review in health sciences. 2nd ed. London: BMJ Books; 2003. pp. 219–235. [Google Scholar]
  • 67.Daly J, Willis K, Small R, Green J, Welch N, Kealy M, et al. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007;60:43–49. doi: 10.1016/j.jclinepi.2006.03.014. [DOI] [PubMed] [Google Scholar]
  • 68.Dixon-Woods M, Sutton A, Shaw R, Miller T, Smith J, Young B, et al. Appraising qualitative research for inclusion in systematic reviews: a quantitative and qualitative comparison of three methods. J Health Serv Res Policy. 2007;12:42–47. doi: 10.1258/135581907779497486. [DOI] [PubMed] [Google Scholar]
  • 69.Carroll C, Booth A, Leaviss J, Rick J. “Best fit” framework synthesis: refining the method. BMC Med Res Methodol. 2013;13:37. doi: 10.1186/1471-2288-13-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 70.Lemire M, Demers-Payette O, Jefferson-Falardeau J. Dissemination of performance information and continuous improvement: a narrative systematic review. J Health Organ Manag. 2013;27:449–478. doi: 10.1108/JHOM-08-2011-0082. [DOI] [PubMed] [Google Scholar]
  • 71.Jacobson N, Butterill D, Goering P. Development of a framework for knowledge translation: understanding user context. J Health Serv Res Policy. 2003;8:94–99. doi: 10.1258/135581903321466067. [DOI] [PubMed] [Google Scholar]
  • 72.Oliver SR, Rees RW, Clarke-Jones L, Milne R, Oakley AR, Gabbay J, et al. A multidimensional conceptual framework for analysing public involvement in health services research. Health Expect. 2008;11:72–84. doi: 10.1111/j.1369-7625.2007.00476.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 73.Eakin JM, Mykhalovskiy E. Reframing the evaluation of qualitative health research: reflections on a review of appraisal guidelines in the health sciences. J Eval Clin Pract. 2003;9:187–194. doi: 10.1046/j.1365-2753.2003.00392.x. [DOI] [PubMed] [Google Scholar]
  • 74.Fone D, Hollinghurst S, Temple M, Round A, Lester N, Weightman A, et al. Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health Med. 2003;25:325–335. doi: 10.1093/pubmed/fdg075. [DOI] [PubMed] [Google Scholar]
  • 75.QSR International NVivo 10 for windows. [cited 2014 Sep 3]. Available from: http://www.qsrinternational.com/products_nvivo.aspx.
  • 76.Noble H, Smith J. Qualitative data analysis: a practical example. Evid Based Nurs. 2014;17:2–3. doi: 10.1136/eb-2013-101603. [DOI] [PubMed] [Google Scholar]
  • 77.Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: meta-narrative reviews. BMC Med. 2013;11:20. doi: 10.1186/1741-7015-11-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 78.Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R. RAMESES publication standards: meta-narrative reviews. J Adv Nurs. 2013;69:987–1004. doi: 10.1111/jan.12092. [DOI] [PubMed] [Google Scholar]
  • 79.Gough D, Thomas J, Oliver S. Clarifying differences between review designs and methods. Syst Rev. 2012;1:28. doi: 10.1186/2046-4053-1-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 80.Sandelowski M, Voils CI, Leeman J, Crandell JL. Mapping the Mixed Methods-Mixed Research Synthesis Terrain. J Mix Methods Res. 2012;6:317–331. doi: 10.1177/1558689811427913. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 81.Crombie IK, Davies HT. What is meta-analysis? [cited 2014 Sep 3] Available from: http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/meta-an.pdf.
  • 82.Ahn HS, Kim HJ. An introduction to systematic review. J Korean Med Assoc. 2014;57:49–59. (Korean) [Google Scholar]
  • 83.Del Pozzo-Magaña BR, Rieder MJ, Lazo-Langner A. Quality of life in children with adverse drug reactions: a narrative and systematic review. Br J Clin Pharmacol. 2014 doi: 10.1111/bcp.12423. doi: 10.1111/bcp.12423. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 84.Cipriani A, Geddes J. Comparison of systematic and narrative reviews: the example of the atypical antipsychotics. Epidemiol Psichiatr Soc. 2003;12:146–153. doi: 10.1017/s1121189x00002918. [DOI] [PubMed] [Google Scholar]
  • 85.Icahn School of Medicine at Mount Sinai EBM skills on the wards. [cited 2014 Sep 3]. Available from: http://libguides.mssm.edu/content.php?pid=417116&sid=3882519.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials


Articles from Epidemiology and Health are provided here courtesy of Korean Society of Epidemiology

RESOURCES