Abstract
The use of mixed methods research (MMR) by health scientists has expanded in recent years. The growth of MMR reflects the complexity of health-related research questions including the need to understand stakeholder perspectives. MMR is further incentivized by the release of the National Institutes of Health’s (NIH) first version of “Best Practices for Mixed Methods Research in the Health Sciences” in 2011 and their revised version which was released in early 2018, and the increase in federal funding awarded to MMR projects. Realizing the need to train health scientists, several training initiatives are underway. In this commentary, we discuss: characteristics of MMR; the rise of MMR in the health sciences; the growing need and efforts to train health scientists in MMR approaches; and the graduate-level course in MMR that was launched at the University of Maryland School of Pharmacy.
Keywords: research methods education, mixed methods research, mixed methods action research
INTRODUCTION
Defining and Characterizing Mixed Methods Research
Although no universally accepted definition or characterization of MMR exists, there is widespread agreement that its hallmark feature is the integration or “mixing” of quantitative and qualitative research components.1-4 The MMR definition advanced by the National Institutes of Health’s (NIH) Office of Behavioral and Social Science Research (OBSSR) Best Practices for MMR in the Health Sciences is probably the one most readily adapted to a health research setting. This document offers the view that MMR is a methodology.5 According to the authors, MMR “focuses on research questions that call for real-life contextual understandings…; employs rigorous quantitative and qualitative research; involves multiple sources and types of data; systematically integrates and triangulates different types of data to maximize the strengths and counterbalance the weaknesses of each data type, and, develops and integrates conceptual and theoretical frameworks into the development of research questions.”5
Therefore, as defined, quantitative and qualitative research approaches may be integrated across methodological levels to include mixing of the: conceptual and theoretical underpinnings for the project; research questions; data collection; data analysis; and inferences, conclusions and theories gleaned from the research. The goal of MMR is to establish deeper, context-rich answers to health science questions with the thought that combining approaches will overcome the limitations of mono-method studies. Qualitative and quantitative data and analytical procedures are integrated in all phases of research in a fully integrated mixed methods research (FIMMR) design.6
Rise of MMR in the Health Sciences
Broadly, MMR is gaining popularity in the health sciences due to several contemporary trends. The publication of the NIH’s best practices document in 2011 and the updated document in early 2018 gave researchers guidance to create and appraise MMR.5 Since the release of the first edition of best practices, the report has received the most traffic on the OBSSR website; and this web traffic has been accompanied by an increase in federally funded MMR proposals.5,7 Beyond the NIH’s best practices document, other sources of methodologic support to conduct MMR have increased in recent years. Leading mixed methodologists have published an armamentarium of excellent MMR textbooks including broad scoping handbooks. There are two journals dedicated to MMR (Journal of Mixed Methods Research and International Journal of Multiple Research Approaches); and the Mixed Methods International Research Association (MMIRA) just celebrated its fifth year as an organization and held its third biennial global conference in Vienna, Austria in August, 2018.8
Although landmark guidance documents, textbooks and research forums support the use of MMR in the health sciences, the definition of MMR provides the greatest insight into its growing use. Determining the overall effectiveness of health interventions across communities, particularly among health disparity populations, a driving goal for health scientists and the Healthy People 2020 initiative by the Centers for Disease Control and Prevention, requires a deep understanding of contextual factors and structural inequality.9,10 Mertens, a transformative MMR expert, has referred to health science research and other complicated lines of social inquiry as research that addresses “wicked problems” and thereby necessitates the use of MMR; not just “traditional analysis of vast amounts of data or more sophisticated statistical analyses.”11
Yet, the contemporary trends and need to address complicated health and health services questions have not elevated the use of MMR in all health research areas. For example, Hadi and colleagues documented minimal application of MMR approaches to health services questions in pharmacy practice.12 The authors specifically highlight the potential benefit of applying MMR to evaluate pharmacy-led health care services, but also raise concerns about the rigor of the limited MMR reports that are published.12 Reflecting on Greene and her colleagues’ classic reasons for mixing methods: “triangulation, development, initiation, expansion, and complementarity,” it becomes clear that mixed methods is well suited to examining pharmacy-based health services.13 For example, if researchers want to understand the full range of benefits associated with pharmacy-led telemonitoring and education services, they may concurrently collect and triangulate interview and biometric data. Investigators may explore whether patients and other stakeholders who report (through semi-structured interviews) positive experiences with pharmacy-led services achieve proximal and long-term treatment goals like normotension and stroke prevention (abstracted from electronic health records).13 Interpreting this information together may help to characterize the types of patients who respond differently to care, the mechanisms that underpin their responses, and an approach to adapt the services delivered.
MMR Capacity-building Among Health Scientists
Rogers’ diffusion of innovation (DOI) model informs us that conditions have to be favorable for an innovation like MMR to be adopted by health scientists.14 For example, the innovation needs to be considered to have “relative advantage” over current approaches; be in concert with a researcher’s beliefs; there has to be communication channels to convey the new information effectively; researcher’s abilities and factors about their personalities matter and therefore, some groups will adopt earlier than others; and systems and peers or near peer experts have to champion the intervention.14 According to Oldenburg and Glanz, dissemination involves practices to ”maximize the reach and adoption” of innovations.15 NIH’s best practices document, MMR textbooks, increased funding opportunities and the development of an MMR community have enhanced uptake among health scientists by improving communication channels and increasing researcher skills and abilities. However, MMR remains underused in specific health research fields, indicating that building a researcher’s capacity to conduct MMR, providing expert support, and improving communication channels are still required to disseminate MMR practices.12
An array of training approaches that address elements highlighted by Rogers’ DOI model are being used to build MMR skills and provide peer experts to local research communities, and, thereby expand MMR adoption. A novel approach that tackles several barriers to MMR uptake is the MMR Training Program for the Health Sciences (MMRTP). This year-long, NIH-funded, multi-disciplinary mentoring program annually accepts 14 health science researchers and admitted its fourth cohort of scholars in June 2018.16 Since the program is grant funded and participation is limited, the goal is not to train an entire health scientist workforce, but to equip program scholars to disseminate MMR on their campuses and in their regions, thus increasing access to experts and communication channels.16
Workshops and other professional development experiences are also increasingly available to support skill development. The University of Michigan Mixed Methods Program and the Odum Institute at the University of North Carolina, Chapel Hill offer a variety of introductory and advanced mixed methods workshops on a regular basis. A web-based certificate program in MMR is available for applied researchers through the School of Social Work at the University of Michigan; and the MMIRA is currently developing a massive open online course (MOOC) which will offer both introductory and advanced MMR training modules.8
With respect to formal didactic training, in the early 2000s, founding mixed methods researchers including Tashakorri, Teddlie and Creswell, wrote about the lack of MMR courses and acknowledged they did not have the benefit of taking coursework themselves.17-19 Therefore, they and their peers, are the “first generation of faculties” in MMR.17 Frels and colleagues, Ivankova and Baran, are other early MMR instructors. They wrote about the challenges associated with teaching MMR across disciplinary and theoretical boundaries, and their experiences teaching in various instructional formats.19-21 Still, minimal empirical evidence is available to guide effective development and delivery of MMR courses, or to ensure optimal student learning of these methodologic concepts.22-24 Though we need to build an evidence base for MMR pedagogy, newer MMR teaching faculty like the author of this commentary, at least benefit from taking courses from founding faculty and being able to develop our courses informed by their syllabi, teaching formats, and our lived training experiences.19
Over the last several years, the availability of graduate-level coursework in MMR has increased.19 To our knowledge, there are no pharmacy schools offering MMR courses to students, pharmacy professionals or post-doctoral trainees. We hope to expand our MMR teaching into the pharmacy student curriculum as a second wave of dissemination of these concepts. As of fall 2018, we began offering a graduate-level course in Mixed Methods Action Research (MMAR) at the University of Maryland School of Pharmacy. Our MMAR class is adapted from a course offered by Ivankova at the University of Alabama, Birmingham. The course is interdisciplinary and offered by web-conferencing to any student in a graduate-level training program, professionals and post-doctoral trainees with the consent of the instructor.
MMR and Action Research Combined
Given the proliferation of general introductory MMR courses, workshops and textbooks in recent years, we chose to develop and offer a course that involves a complex application of MMR study conceptualization and design. According to the current iteration of Creswell and Plano Clark’s MMR design typology, there are three core MMR designs (explanatory sequential, exploratory sequential, convergent).25 Two primary characteristics delineate the core designs. The first characteristic is timing – with parallel quantitative and qualitative phases in the convergent design; and quantitative and qualitative components occurring in stages and informed by one another in the sequential designs. The second characteristic is the purpose for the design – to “explain, explore or converge quantitative and qualitative components.”25 Fixed complex mixed method designs, as described by Creswell and Plano Clark, include a core design and, for example, a framework and/or an additional methodology.25 Poth posits that fixed design typologies are not dynamic enough to be of service to research studies addressing complex issues.26 She argues for mixed methods researchers to consider contextual factors by using adaptable designs and a “complexity-sensitive approach to MMR.”26 Ivankova’s MMAR approach, reflected in our class, while advocating for a set of fixed designs at study conception, describes the need for an iterative or recursive research process throughout the conduct of the study.27 Also, action research requires us to consider stakeholder perspectives and the socio-ecologic domains that affect health and well-being.28,29
Action research, as used in our MMAR course, includes a host of related methodologies (eg, community-engaged research, community-based participatory research) that are informed by a social justice lens and have the goal of creating an intervention that redounds to the benefit of a community.27,28 Community, as we discuss it in the course, could be a community of practitioners like pharmacists, or a group that shares some other affiliation like geographic boundaries.27 We believe that offering a course that combines MMR and AR is appropriate to our setting because our research frequently involves populations with health disparities, their caregivers and associated health care professionals.
Our three-credit hour elective MMAR course has a minimum and maximum enrollment of two and 10 students, respectively, and it is taught through a web-conferencing platform. In fall 2018, the first offering of the course, three students enrolled (one graduate student from the pharmacy school who is also a PharmD; and two doctoral students from the medical school). Two additional students, a doctoral student from the school of social work and a pharmacy school post-doctoral fellow, informally audited the course. There are no examinations in the course; students learn MMAR methodology through a series of lectures and by applying the concepts to interrogate published literature and develop their own research projects. In one assignment, for example, students write and present a sampling scheme that reflects MMAR principles and is suitable for their proposed research. They use their assignments, lectures and readings to build an MMAR proposal and presentation for their final assessment at the end of the course.
Importantly, the research we critically evaluate from published examples and develop for student projects are complicated or “wicked” and require the features of MMAR, both in the democratization of knowledge with input from stakeholders and the delivery of interventions that the community finds acceptable, contextually relevant and that address structural inequalities.9,11,28
CONCLUSION
MMR is an approach that involves integration of quantitative and qualitative research components to gain deeper, context-rich answers to complicated questions. MMR applications in the health sciences has grown in recent years due to increased access to training opportunities, methods books, peer and organization support and NIH incentives like providing the best practices document and increased federal funding for MMR proposals. The growth of MMR has not been consistent across all health science areas. However, much like the conduct of MMR, the best approaches to teach, learn and disseminate MMR concepts is not a settled science. We believe it is important for pharmaceutical educators to join the MMR training effort and help shape teaching and learning principles in MMR.
REFERENCES
- 1.Teddlie C, Tashakkori A, editors. Foundations of Mixed Methods Research: Integrating Quantitative and Qualitative Approaches in the Social and Behavioral Sciences. 1st ed. Thousand Oaks, CA: Sage Publications, Inc; 2009. [Google Scholar]
- 2.Johnson RB, Onwuegbuzie AJ, Turner LA. Toward a definition of mixed methods research. J Mix Meth Res. 2007;1(2):112–133. [Google Scholar]
- 3.Teddlie C, Tashakkori A. Common “core” characteristics of mixed methods research: a review of critical issues and call for greater convergence. Am Behav Scientist. 2012;56(6):774–788. [Google Scholar]
- 4.Morse JM, Niehaus L. Mixed Methods Design: Principles and Procedures. 1st ed. Walnut Creek, CA: Left Coast Press; 2009. [Google Scholar]
- 5.Creswell JW, Klassen AC, Plano Clark VL, Smith KC, National Institutes of Health Office of Behavioral and Social Sciences Research Best practices for mixed methods research in the health sciences. Published August, 2011. Revised 2018; https://obssr.od.nih.gov/wp-content/uploads/2018/01/Best-Practices-for-Mixed-Methods-Research-in-the-Health-Sciences-2018-01-25.pdf. Accessed August 18, 2018.
- 6.Creamer EG. An Introduction to Fully Integrated Mixed Methods Research. Thousand Oaks, CA: Sage Publications, Inc; 2018. [Google Scholar]
- 7.Coyle CE, Schulman-Green D, Feder S, et al. Federal funding for mixed methods research in the health sciences in the united states: recent trends. J Mix Meth Res. 2018;12(3):305–324. [Google Scholar]
- 8.Mixed Methods International Research Association. http://mmira.wildapricot.org/page-1859605 . Accessed September 26, 2018.
- 9.Thomas SB, Quinn SC, Butler J, Fryer CS, Garza MA. Toward a fourth generation of disparities research to achieve health equity. Annu Rev Public Health. 2011;32:399–416. doi: 10.1146/annurev-publhealth-031210-101136. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Center for Disease Control and Prevention. National Center for Health Statistics. Healthy People. 2020 https://www.cdc.gov/nchs/healthy_people/hp2020.htm . Accessed September 28, 2018. [Google Scholar]
- 11.Mertens DM. Mixed methods and wicked problems. J Mix Meth Res. 2015;9(1):3–6. [Google Scholar]
- 12.Hadi MA, Alldred DP, Closs SJ, Briggs M. Mixed-methods research in pharmacy practice: recommendations for quality reporting (part 2) The international journal of pharmacy practice. 2014;2014(22):96–100. doi: 10.1111/ijpp.12015. [DOI] [PubMed] [Google Scholar]
- 13.Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis. 1989;11(3):255–274. [Google Scholar]
- 14.Rogers EM. Diffusion of Innovations. 5th ed. New York, NY: New York: Free Press; 2003. [Google Scholar]
- 15.Oldenburg B, Glanz K. Diffusion of innovations. In: Glanz K, Rimer BK, Viswanath K, editors. Health Behavior and Health Education Theory, Research, and Practice. San Francisco, CA: Jossey-Bass; 2008. pp. 313–333. [Google Scholar]
- 16.John Hopkins Bloomberg School of Public Health. Mixed Methods Research Training Program for the Health Sciences. “About the program.” https://www.jhsph.edu/academics/training-programs/mixed-methods-training-program-for-the-health-sciences/about-the-program/. Accessed September 26, 2018.
- 17.Tashakkori A, Teddlie C. Issues and dilemmas in teaching research methods courses in social and behavioral sciences: US perspective. Int J Soc Res Meth. 2003;6(1):61–77. [Google Scholar]
- 18.Creswell JW, Tashakkori A, Jensen KD, Shapley KL. Teaching mixed methods research: practices, dilemmas, and challenges. In: Tashakkori A, Teddlie C, editors. Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks, CA: Sage Publications, Inc.; 2003. pp. 619–637. [Google Scholar]
- 19.Frels RK, Onwuegbuzie AJ, Leech NL, Collins KMT. Challenges to teaching mixed research courses. J Eff Teach. 2012;12(2):23–44. [Google Scholar]
- 20.Ivankova NV. Teaching and learning mixed methods research in computer-mediated environment: challenges and rewards. Int J Mult Res Approaches. 2010;4:49–65. [Google Scholar]
- 21.Baran M. Teaching multi-methodology research courses to doctoral students. Int J Mult Res Approaches. 2010;4:19–27. [Google Scholar]
- 22.Greene JC. Beginning the conversation. Int J Mult Res Approaches. 2010;4(1):3–5. [Google Scholar]
- 23.Earley MA. A synthesis of the literature on research methods education. Teach Higher Education. 2014;19:242–253. [Google Scholar]
- 24.Poth C. What constitutes effective learning experiences in a mixed methods research course? An examination from the student perspective. Int J Mult Res Approaches. 2014;8(1):74–86. [Google Scholar]
- 25.Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 3rd ed. Thousand Oaks, CA: Sage Publications, Inc.; 2018. [Google Scholar]
- 26.Poth C. The curious case of complexity: implications for mixed methods research practices. Int J Mult Res Approaches. 2018;10(1):403–411. [Google Scholar]
- 27.Ivankova NV. Mixed Methods Applications in Action Research: From Methods to Community Action. 1st ed. Thousand Oaks, California: Sage Publications, Inc; 2015. [Google Scholar]
- 28.Bradbury H. The Sage Handbook of Action Research. 3rd ed. Thousand Oaks, CA: Sage Publications, Inc.; 2015. [Google Scholar]
- 29.Wallerstein N, Duran B, Oetzel J, Minkler M. Community-based Participatory Research for Health: Advancing Social and Health Equity. 3rd ed. California: Jossey-Bass; 2018. [Google Scholar]