Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Jan 1.
Published in final edited form as: Circ Cardiovasc Qual Outcomes. 2013 Jan 1;6(1):119–123. doi: 10.1161/CIRCOUTCOMES.112.967885

Mixed Methods in Biomedical and Health Services Research

Leslie A Curry *,‡, Harlan M Krumholz **,‡,†,§, Alicia O’Cathain #, Vicki L Plano Clark ***, Emily Cherlin *, Elizabeth H Bradley *,‡
PMCID: PMC3569711  NIHMSID: NIHMS432078  PMID: 23322807

Abstract

Mixed methods studies, in which qualitative and quantitative methods are combined in a single program of inquiry, can be valuable in biomedical and health services research, where the complementary strengths of each approach can yield greater insight into complex phenomena than either approach alone. Although interest in mixed methods is growing among science funders and investigators, written guidance on how to conduct and assess rigorous mixed methods studies is not readily accessible to the general readership of peer-reviewed biomedical and health services journals. Furthermore, existing guidelines for publishing mixed methods studies are not well known or applied by researchers and journal editors. Accordingly, this paper is intended to serve as a concise, practical resource for readers interested in core principles and practices of mixed methods research. We briefly describe mixed methods approaches and present illustrations from published biomedical and health services literature, including in cardiovascular care, summarize standards for the design and reporting of these studies, and highlight four central considerations for investigators interested in using these methods.

Keywords: Mixed Methods, Health Services Research

Introduction

Mixed methods studies, in which qualitative1 and quantitative methods are combined in a single program of inquiry,2 are increasingly common and can be valuable in biomedical and health services research, where the complementary strengths of each approach can characterize complex phenomena more fully than either approach alone.3, 4 To effectively address complex problems in health and health care delivery, including heterogeneous and dynamic systems of care, a multi-level approach is needed to capture the perspectives of patients, providers and organizations. Mixed methods offer enhanced capabilities to this end. Consequently, interest in mixed methods studies is growing among funders, as evidenced by recent calls for proposals using these methods from the National Institutes of Health (NIH),5 the Agency for Healthcare Research and Quality,6 independent research organizations (e.g., Patient Centered Outcomes Research Institute)7 and foundations (e.g., The Robert Wood Johnson Foundation).8 Training in mixed methods is also sponsored by NIH,9 the Agency for Healthcare Research and Quality,10 and professional associations.11 Nevertheless, written guidance on how to conduct rigorous mixed methods research is not readily available to the general readership of peer-reviewed biomedical and health services journals, a group which may be less familiar with this approach.

Accordingly, in this paper we describe applications of mixed methods in biomedical and health services research and provide a concise overview of key principles to facilitate best practices. First, we define mixed methods approaches and present illustrations from published literature, including in cardiovascular care. Second, we summarize standards for the design and conduct of rigorous mixed methods studies, and third, we highlight four central considerations for investigators interested in using these methods.

Mixed methods research in biomedical and health services research: approaches and illustrations

Mixed methods can be useful in pursuing a broad range of focal topics and study aims in the biomedical and health services research arenas, including but not limited to clinical or quality issues,12–14 health care organizational performance,15 behavioral interventions,16, 17 processes of implementation of innovations,18, 19 health care decision making,20 and measurement development for complex constructs.21, 22 Including a supplemental qualitative component within experimental or quasi-experimental studies of complex interventions is becoming increasingly common (see Lewin et al, BMJ17 for a review of qualitative methods within randomized clinical trials). In this approach, the qualitative component can examine whether the intervention was delivered as intended, describe implementation processes, generate understanding of why the intervention failed to work, or how its effectiveness was promoted or limited in the real world. Qualitative findings can help mitigate publication biases against studies lacking intervention effectiveness by both explaining negative results and informing subsequent research. With regard to health care organizational performance, we are often not only seeking to measure performance or change in performance but also to understand why organizations perform well or poorly and what diverse types of factors might influence performance. Of particular importance is the careful matching of the method to the research question of interest. Illustrations from a variety of focal topics in the published literature, together with the specific contributions of the qualitative and quantitative components toward the overall research aim, are summarized in Table 1.

Table 1.

Focal topics well suited for mixed methods and illustrative studies

Focal topic or aim Illustrative study Qualitative component Quantitative component
Clinical or quality issue Medication errors in computerized order entry systems14 Discover potential sources of error risk and characterize context in which errors occur Quantify frequency of error risks reported by house staff
Organizational performance Quality of AMI hospital care23, 24 Describe complex processes and organizational environment Identify factors associated with 30-day risk-standardized mortality rates
Complex intervention trials/RCTs Secondary preventive follow up care for patients with AMI or angina25 Clarify process and examine underlying theory to inform interpretation of quantitative results and future intervention designs Assess impact of intervention on lifestyle and cardiovascular risk
Implementation science Organizational readiness to adopt new protocol for acute stroke care 26 Elicit patient and staff perceptions of facilitators and barriers to adoption Assess organizational readiness using the Team Climate Inventory Questionnaire
Medical decision making Informed consent for abdominal aortic aneurysm repair20 Characterize patient perspectives on informed consent process Assess variation in surgeon reports and factors associated with variation
Develop quantitative measurement of a complex construct Patient-centered measures of outcomes of treatment for prostate cancer27 Identify core facets of the phenomenon from patients’ perspective Develop and validate items and scales through psychometric testing
Open in a new tab

Key factors in mixed methods study design are the relative timing of when each method is carried out (concurrently or sequentially) and the emphasis accorded to each component for addressing the study’s purpose (whether they are equally weighted or whether one is primary and the other secondary).28 Whether in a concurrent or sequential design, several features characterize the connections between components in a mixed methods study. These include: a) a priori intention to conduct the second component and integration of design elements to facilitate this linkage; b) use of a common sample (e.g., drawing a purposive sample for a qualitative study based on the survey results of the first quantitative component); c) a unifying aim and/or research question; d) the contingency of the questions/design of one study on the findings of the other; e) the degree to which findings feed iteratively into the design or conduct of the other; and f) the degree of integration of findings.29

There are four basic types of mixed methods designs (Figure 1).2 The first is the sequential explanatory strategy, in which the quantitative component is followed by a qualitative component and the qualitative results assist in explaining the findings of the quantitative study. For instance, in a quantitative study of patients with acute myocardial infarction (n=500) Spertus et al30 found 14% of patients receiving a drug-eluting stent discontinued clopidogrel, a life-sustaining antiplatelet drug, before the recommended duration despite potentially fatal consequences for early termination. To understand potential reasons for this patient behavior, Garavalia et al conducted a qualitative study31 of patients with AMI who had discontinued either clopidogrel (n=11) or cholesterol-lowering therapy (n=29); findings informed the development of a guide to support patient-clinician communication about heart medications.21 The second design is the sequential exploratory strategy, in which the qualitative component is followed by a quantitative component. For example, a study of hospital performance in care of patients with acute myocardial infarction comprised an initial qualitative component to characterize features of top performing hospitals and to generate hypotheses about factors related to performance that were then tested in a nationally representative sample of hospitals. Aspects of the organizational environment (e.g., creative problem solving) were identified in the qualitative component and statistically associated with lower risk standardized mortality rates in the quantitative findings.23, 24 The third design is the convergent parallel strategy, where the quantitative and qualitative data collection is concurrent, the components are given equal weight, and the two datasets are analyzed and compared. For example, Kerr et al32 sought to gain a more complete understanding of the effectiveness of a web-based intervention for heart disease self-management in decreasing inequalities in access to self-management support for patients with coronary heart disease (CHD). Patients with CHD (n=168) using a modified version of The Comprehensive Health Enhancement and Social Support (CHESS) tool were followed in a prospective cohort design with complementary quantitative and qualitative components. Quantitative data identified factors statistically associated with use of the tool; these findings were integrated with qualitative data from in depth interviews with a subset of participants (n=19) to understand how and why the identified factors influenced participants’ use of the tool. Finally, in the concurrent embedded strategy, quantitative and qualitative data collection occurs at the same time; however, one component is predominant. For instance, a randomized controlled trial of a computerized decision support tool for patients with atrial fibrillation being considered for anti-coagulation treatment included a qualitative process evaluation to provide insights into the process and progress of the trial, and to inform monitoring and auditing decisions.33 Non-participant observation and in depth interviews with participants (n=30) generated critical information that led to the discontinuation of an intervention arm of the trial.

Figure 1. The major mixed methods research designs.

Figure 1

Open in a new tab

This figure is based on Creswell and Plano Clark’s (2011) discussion of mixed methods designs.2 Figure adapted with permission of Sage Publications.

Standards for designing and conducting mixed methods research

Guidance for designing and conducting mixed methods research is available in multiple reference texts2, 34, 35 and journal articles, 3, 36–38 some of which are focused on health care.39 In addition, the US NIH Office of Behavioral and Social Science Research recently commissioned a report defining best practices in mixed methods research.29 This report is intended to assist investigators in preparing competitive mixed methods applications for support from NIH, to guide review panel members in evaluating proposals that use these methods, and to serve as a resource to NIH Institutes and Centers as they consider potential contributions of mixed methods, plan new initiatives, and set priority areas for their science.

Central considerations in conducting mixed methods studies

Despite this available guidance, the quality and rigor of mixed methods research in the published empirical literature is highly variable.4 We highlight four central considerations for investigators seeking to conduct rigorous mixed methods research: alignment of aims, methods and research team capacity; attention to methodological standards for each component; articulation and implementation of plans for deliberate integration of qualitative and quantitative components; and adherence to recommended guidelines for writing mixed methods papers.

Alignment of aims, methods and research team capacity

A mixed methods study is best suited to address a multifaceted research aim (e.g., one that seeks to generate evidence requiring distinct forms of measurement). For instance, a research goal to generate a model of health services utilization might use an exploratory sequential design with a qualitative component to identify core dimensions and generate a theory, and a quantitative component to test the theory. The motivation for a mixed methods design must be explicit and compelling. Reasons might include: pursuing a topic about which little is known and hence using a qualitative component to inform hypothesis generation; producing a comprehensive account of the nature and magnitude of a phenomenon; seeking to both understand context and to produce generalizable findings; aiming to describe both process and outcomes; and seeking increased confidence in findings by addressing threats to validity by either approach alone.3 One recent review applied extant frameworks for critical appraisal of published mixed methods health services studies,3 and found only one third of reports provided justification for this design;40 and only half did so in a review of mixed methods mental health literature. In addition, each specific aim in a mixed methods study should be substantive, rather than instrumental, in nature (e.g., ‘to explore reasons for patient non-adherence to an intervention protocol’ rather than ‘to conduct patient focus groups’).

Finally, the team composition and resources must be appropriate to achieve the study aims, including quantitative analysis, qualitative analysis, and integration strategies. Achieving optimal team composition is difficult because of the diverse areas of expertise required, and while quantitative expertise is typically present, mixed methods research teams can suffer from under representation of expertise in either qualitative or mixed methods or both. Even when the team composition is appropriate, the dynamics within such highly diverse teams may present challenges to effective collaboration, with the qualitative component undervalued or the mixed methods aspect not well understood.40, 41 The time-and resource-intensive nature of mixed methods designs is a notable challenge and must be explicitly recognized and planned for.

Adherence to standards for each component

A central consideration in mixed methods studies is the adherence to methodological standards for each component2 The quantitative component must be designed and carried out with deliberate attention to principles of internal validity, external validity42 and reliability.43 The qualitative research must be conceptualized and implemented in accordance with established principles for rigor (e.g., to ensure credibility, transferability, and dependability).44, 45 Because these principles are distinctly different in qualitative and quantitative methods, explicit attention must be directed to adhering to the respective standards for each component throughout the research process to ensure key aspects of the design are not invalidated or undermined.46 The risk of undermining these respective standards is heightened in mixed methods studies, in which experts in quantitative methods may argue for large representative sample sizes (although inconsistent with principles for sampling in qualitative studies), or qualitative experts may criticize standardized quantitative data collection instruments as introducing excessive researcher bias.

Integration of findings across components

Essential to a mixed methods approach is the deliberate ‘mixing’ or integration of the quantitative and qualitative findings of each component2 and techniques to accomplish integration have been described; however, a lack of such integration persists in published research. The overall aim is to ensure that study components are directly linked and the output is synergistic, such that the end product is more than two parts alone.47 Integration of findings from the qualitative and quantitative component can occur in all of the designs displayed in Figure 1 at the interpretation stages of a study with one component explaining, enhancing, confirming, challenging or quantifying findings from the other component. Alternatively, findings from one component may lead to further analysis within the other component which in turn may lead to new insights.

Adherence to recommendations for reporting mixed methods research

A substantial challenge for researchers seeking to publish findings from mixed methods studies in biomedical and health services journals is that space constraints often preclude full reporting of findings from both components in the same paper.48 A recent review of mixed methods articles in health services research journals4 found incomplete reporting of key methodological information. For instance, only 36% of mixed methods studies reported the sampling selection for the quantitative component and 17% for the qualitative component; 40% reported the data analysis for the quantitative component and 31% for the qualitative component. One proposed template is Good Reporting of Mixed Methods Studies, which identifies the aspects of a study that should be addressed for appropriately transparent reporting, including the rationale for a mixed method approach, and description of the design, the methods of each component, procedures for integration, limitations of each method, and insights gained from mixing methods.40 Potential ways researchers might convey this information are to make explicit linkages across papers if they are published as single reports, to request additional space if they are reporting both components in one paper, and to use web appendices to provide additional information.

Conclusion

Mixed methods approaches can be extraordinarily valuable to biomedical and health services research efforts. Studies using mixed methods can uncover novel causal factors, can open new areas of research, and can result in more flexible and holistic thinking about health and medicine. The methods are well established and guidelines for reporting of rigorous mixed methods research exist. Application of rigorous mixed methods research approaches can enhance our ability to understand and address the pressing issues of clinical care in an increasingly complex health care system.

Acknowledgments

We gratefully acknowledge John Creswell, PhD, University of Nebraska/Lincoln and Helen Meissner, PhD, NIH/OBSSR for their comments on an earlier version of this manuscript.

Funding Sources:

There are no funding sources or competing interests to declare for any authors.

Footnotes

Conflict of Interest Disclosures: Dr. Krumholz reports that he is supported by grant U01 HL105270-02 (Center for Cardiovascular Outcomes Research at Yale University) from the National Heart, Lung, and Blood Institute. He discloses that he is the recipient of a research grant from Medtronic, Inc. through Yale University and is chair of a cardiac scientific advisory board for United Health.

References

  • 1.Pope C, Mays N. Reaching the parts other methods cannot reach: an introduction to qualitative methods in health and health services research. BMJ. 1995;311:42–45. doi: 10.1136/bmj.311.6996.42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 2. Thousand Oaks, CA: Sage; 2011. [Google Scholar]
  • 3.O’Cathain A, Murphy E, Nicholl J. Why, and how, mixed methods research is undertaken in health services research in England: A mixed methods study. BMC Health Serv Res. 2007;7:85. doi: 10.1186/1472-6963-7-85. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Wisdom JP, Cavaleri MA, Onwuegbuzie AJ, Green CA. Methodological reporting in qualitative, quantitative, and mixed methods health services research articles. Health Serv Res. 2011 doi: 10.1111/j.1475-6773.2011.01344.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.National Institutes of Health. [Accessed on March 2, 2012];Funding Opportunity PA-11-063. http://grants.nih.gov/grants/guide/pa-files/PA-11-063.html.
  • 6.Agency for Healthcare Research and Quality. [Accessed on March 4, 2012];AHRQ Mentored Career Enhancement Award in Patient Centered Outcomes Research (PCOR) for Mid-Career and Senior Investigators (K18) http://grants.nih.gov/grants/guide/pa-files/PAR-12-115.html.
  • 7.Patient-Centered Outcomes Research Center (PCORI) [Accessed on February 6, 2012];PCORI funding announcement. www.pcori.org/assets/PCORI-Pilot-Projects-Funding-Announcement-Amendment-1-_v2_-09302011.pdf.
  • 8.Robert Wood Johnson Foundation (RWJ) [Accessed on February 6, 2012];The Robert Wood Johnson Foundation Approach to Evaluation. http://www.rwjf.org/pr/product.jsp?id=51508.
  • 9.National Institute on Aging. [Accessed on March 3, 2012];Using Mixed Methods to Optimize Dissemination and Implementation of Health Interventions. http://conferences.thehillgroup.com/obssr/MixedMethodsDI/agenda.html.
  • 10.Agency for Healthcare Research and Quality. [Accessed on February 9, 2012];Opportunities for Advancing Delivery System Research. http://www.ahrq.gov/qual/deliverysys/2011mtg/mtgsumm.htm.
  • 11.Academy Health. [Accessed on February 9, 2012];An Introduction to Mixed Methods. http://academyhealth.org/Training/ResourceDetail.cfm?itemnumber=7916.
  • 12.Dean JE, Hutchinson A, Escoto KH, Lawson R. Using a multi-method, user centred, prospective hazard analysis to assess care quality and patient safety in a care pathway. BMC Health Serv Res. 2007;7:89. doi: 10.1186/1472-6963-7-89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Ginsburg LR, Chuang YT, Norton PG, Berta W, Tregunno D, Ng P, Richardson J. Development of a measure of patient safety event learning responses. Health Serv Res. 2009;44:2123–2147. doi: 10.1111/j.1475-6773.2009.01021.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL. Role of computerized physician order entry systems in facilitating medication errors. JAMA. 2005;293:1197–1203. doi: 10.1001/jama.293.10.1197. [DOI] [PubMed] [Google Scholar]
  • 15.Groene O, Klazinga N, Wagner C, Arah OA, Thompson A, Bruneau C, Sunol R Deepening our Understanding of Quality Improvement in Europe Research P. Investigating organizational quality improvement systems, patient empowerment, organizational culture, professional involvement and the quality of care in European hospitals: the ‘Deepening our Understanding of Quality Improvement in Europe (DUQuE)’ project. BMC Health Serv Res. 2010;10:281. doi: 10.1186/1472-6963-10-281. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Clark AM, Mundy C, Catto S, MacIntyre PD. Participation in community-based exercise maintenance programs after completion of hospital-based cardiac rehabilitation: a mixed-method study. J Cardiopulm Rehabil Prev. 2011;31:42–46. doi: 10.1097/HCR.0b013e3181f68aa6. [DOI] [PubMed] [Google Scholar]
  • 17.Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496. doi: 10.1136/bmj.b3496. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Riley DL, Krepostman S, Stewart DE, Suskin N, Arthur HM, Grace SL. A mixed methods study of continuity of care from cardiac rehabilitation to primary care physicians. Can J Cardiol. 2009;25:e187–192. doi: 10.1016/s0828-282x(09)70096-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Robertson A, Cresswell K, Takian A, Petrakaki D, Crowe S, Cornford T, Barber N, Avery A, Fernando B, Jacklin A, Prescott R, Klecun E, Paton J, Lichtner V, Quinn C, Ali M, Morrison Z, Jani Y, Waring J, Marsden K, Sheikh A. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ. 2010;341:c4564. doi: 10.1136/bmj.c4564. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Berman L, Curry L, Gusberg R, Dardik A, Fraenkel L. Informed consent for abdominal aortic aneurysm repair: The patient’s perspective. J Vasc Surg. 2008;48:296–302. doi: 10.1016/j.jvs.2008.03.037. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Garavalia L, Garavalia B, Spertus JA, Decker C. Medication Discussion Questions (MedDQ): developing a guide to facilitate patient-clinician communication about heart medications. J Cardiovasc Nurs. 2011;26:E12–19. doi: 10.1097/JCN.0b013e3181efea94. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Krause N. A comprehensive strategy for developing closed-ended survey items for use in studies of older adults. J Gerontol B Psychol Sci Soc Sci. 2002;57:S263–274. doi: 10.1093/geronb/57.5.s263. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Curry LA, Spatz E, Cherlin E, Thompson JW, Berg D, Ting HH, Decker C, Krumholz HM, Bradley EH. What distinguishes top-performing hospitals in acute myocardial infarction mortality rates? A qualitative study. Ann Intern Med. 2011;154:384–390. doi: 10.7326/0003-4819-154-6-201103150-00003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Bradley EH, Curry L, Spatz ES, Herrin J, Cherlin EJ, Curtis J, Thompson JW, Ting HH, Wang Y, Krumholz HM. Hospital strategies for reducing risk-standardized mortality rates in acute myocardial infarction. Ann Intern Med. 2012;156:618–626. doi: 10.1059/0003-4819-156-9-201205010-00003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Bradley F, Wiles R, Kinmonth AL, Mant D, Gantley M. Development and evaluation of complex interventions in health services research: case study of the Southampton heart integrated care project (SHIP). The SHIP Collaborative Group. BMJ. 1999;318:711–715. doi: 10.1136/bmj.318.7185.711. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 26.Hamilton S, McLaren S, Mulhall A. Assessing organisational readiness for change: use of diagnostic analysis prior to the implementation of a multidisciplinary assessment for acute stroke care. Implement Sci. 2007;2:21. doi: 10.1186/1748-5908-2-21. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Clark JA, Bokhour BG, Inui TS, Silliman RA, Talcott JA. Measuring patients’ perceptions of the outcomes of treatment for early prostate cancer. Med Care. 2003;41:923–936. doi: 10.1097/00005650-200308000-00006. [DOI] [PubMed] [Google Scholar]
  • 28.Morgan DL. Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res. 1998;8:362–376. doi: 10.1177/104973239800800307. [DOI] [PubMed] [Google Scholar]
  • 29.Creswell JW, Klassen AC, Plano Clark VL, Smith KC for the Office of Behavioral Health and Social Sciences Research. Best Practices for Mixed Methods Research in Health Sciences. National Institutes of Health; 2011. [Google Scholar]
  • 30.Spertus JA, Kettelkamp R, Vance C, Decker C, Jones PG, Rumsfeld JS, Messenger JC, Khanal S, Peterson ED, Bach RG, Krumholz HM, Cohen DJ. Prevalence, predictors, and outcomes of premature discontinuation of thienopyridine therapy after drug-eluting stent placement: results from the PREMIER registry. Circulation. 2006;113:2803–2809. doi: 10.1161/CIRCULATIONAHA.106.618066. [DOI] [PubMed] [Google Scholar]
  • 31.Garavalia L, Garavalia B, Spertus JA, Decker C. Exploring patients’ reasons for discontinuance of heart medications. J Cardiovasc Nurs. 2009;24:371–379. doi: 10.1097/JCN.0b013e3181ae7b2a. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Kerr C, Murray E, Noble L, Morris R, Bottomley C, Stevenson F, Patterson D, Peacock R, Turner I, Jackson K, Nazareth I. The potential of Web-based interventions for heart disease self-management: a mixed methods investigation. J Med Internet Res. 2010;12:e56. doi: 10.2196/jmir.1438. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33.Murtagh MJ, Thomson RG, May CR, Rapley T, Heaven BR, Graham RH, Kaner EF, Stobbart L, Eccles MP. Qualitative methods in a randomised controlled trial: the role of an integrated qualitative process evaluation in providing evidence to discontinue the intervention in one arm of a trial of a decision support tool. Qual Saf Health Care. 2007;16:224–229. doi: 10.1136/qshc.2006.018499. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Patton MQ. Qualitative Research & Evaluation Methods. 3. Thousand Oaks, CA: Sage Publications; 2002. [Google Scholar]
  • 35.Tashakkori A, Teddlie C. Sage Handbook of Mixed Methods in Social and Behavioral Research. 2. Thousand Oaks, CA: Sage; 2010. [Google Scholar]
  • 36.Bryman A. Integrating quantitative and qualitative research: How is it done? Qual Res. 2006;6:97–113. [Google Scholar]
  • 37.Jick TD. Mixing qualitative and quantitative methods: Triangulation in action. Admin Sci Quart. 1979;24:602–611. [Google Scholar]
  • 38.Onwuegbuzie AJ, Bustamante RM, Nelson JA. Mixed research as a tool for developing quantitative instruments. J Mixed Meth Res. 2010;4:56–78. [Google Scholar]
  • 39.Curry LA, Nembhard IM, Bradley EH. Qualitative and mixed methods provide unique contributions to outcomes research. Circulation. 2009;119:1442–1452. doi: 10.1161/CIRCULATIONAHA.107.742775. [DOI] [PubMed] [Google Scholar]
  • 40.O’Cathain A, Murphy E, Nicholl J. The quality of mixed methods studies in health services research. J Health Serv Res Policy. 2008;13:92–98. doi: 10.1258/jhsrp.2007.007074. [DOI] [PubMed] [Google Scholar]
  • 41.Curry LA, O’Cathain A, Plano Clark VL, Aroni R, Fetters M, Berg D. The role of group dynamics in mixed methods in health sciences research teams. Journal of mixed methods research. 2012;6:5–20. [Google Scholar]
  • 42.Campbell DT. Factors relevant to the validity of experiments in social settings. Psychol Bull. 1957;54:297–312. doi: 10.1037/h0040950. [DOI] [PubMed] [Google Scholar]
  • 43.Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16:297–334. [Google Scholar]
  • 44.Lincoln Y, Guba E. Naturalistic Inquiry. Newbury Park, CA: Sage; 1985. [Google Scholar]
  • 45.Mays N, Pope C. Rigour and qualitative research. BMJ. 1995;311:109–112. doi: 10.1136/bmj.311.6997.109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46.Morse JM. Principles and procedures of maintaining validity for mixed-method design. In: Curry LRS, Welte T, editors. Improving Aging and Public Health Research: Qualitative and Mixed Methods. Washington, DC: American Public Health Association; 2006. [Google Scholar]
  • 47.O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods studies. BMJ. 2010;341:c4587. doi: 10.1136/bmj.c4587. [DOI] [PubMed] [Google Scholar]
  • 48.Stange KC, Crabtree BF, Miller WL. Publishing multimethod research. Ann Fam Med. 2006;4:292–294. doi: 10.1370/afm.615. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES