Combining qualitative and quantitative research within mixed method research designs: A methodological review

Abstract

Objectives

It has been argued that mixed methods research can be useful in nursing and health science because of the complexity of the phenomena studied. However, the integration of qualitative and quantitative approaches continues to be one of much debate and there is a need for a rigorous framework for designing and interpreting mixed methods research. This paper explores the analytical approaches (i.e. parallel, concurrent or sequential) used in mixed methods studies within healthcare and exemplifies the use of triangulation as a methodological metaphor for drawing inferences from qualitative and quantitative findings originating from such analyses.

Design

This review of the literature used systematic principles in searching CINAHL, Medline and PsycINFO for healthcare research studies which employed a mixed methods approach and were published in the English language between January 1999 and September 2009.

Results

In total, 168 studies were included in the results. Most studies originated in the United States of America (USA), the United Kingdom (UK) and Canada. The analytic approach most widely used was parallel data analysis. A number of studies used sequential data analysis; far fewer studies employed concurrent data analysis. Very few of these studies clearly articulated the purpose for using a mixed methods design. The use of the methodological metaphor of triangulation on convergent, complementary, and divergent results from mixed methods studies is exemplified and an example of developing theory from such data is provided.

Conclusion

A trend for conducting parallel data analysis on quantitative and qualitative data in mixed methods healthcare research has been identified in the studies included in this review. Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings, help researchers to clarify their theoretical propositions and the basis of their results. This can offer a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and develop new theory.

What is already known about the topic?

• •Mixed methods research, where quantitative and qualitative methods are combined, is increasingly recognized as valuable, because it can potentially capitalize on the respective strengths of quantitative and qualitative approaches.
• •There is a lack of pragmatic guidance in the research literature as how to combine qualitative and quantitative approaches and how to integrate qualitative and quantitative findings.
• •Analytical approaches used in mixed-methods studies differ on the basis of the sequence in which the components occur and the emphasis given to each, e.g. parallel, sequential or concurrent.

What this paper adds

• •A trend for conducting parallel analysis on quantitative and qualitative data in healthcare research is apparent within the literature.
• •Using triangulation as a methodological metaphor can facilitate the integration of qualitative and quantitative findings and help researchers to clearly present both their theoretical propositions and the basis of their results.
• •Using triangulation as a methodological metaphor may also support a better understanding of the links between theory and empirical findings, challenge theoretical assumptions and aid the development of new theory.

1. Introduction

Mixed methods research has been widely used within healthcare research for a variety of reasons. The integration of qualitative and quantitative approaches is an interesting issue and continues to be one of much debate (Bryman, 2004, Morgan, 2007, Onwuegbuzie and Leech, 2005). In particular, the different epistemological and ontological assumptions and paradigms associated with qualitative and quantitative research have had a major influence on discussions on whether the integration of the two is feasible, let alone desirable (Morgan, 2007, Sale et al., 2002). Proponents of mixed methods research suggest that the purist view, that quantitative and qualitative approaches cannot be merged, poses a threat to the advancement of science (Onwuegbuzie and Leech, 2005) and that while epistemological and ontological commitments may be associated with certain research methods, the connections are not necessary deterministic (Bryman, 2004). Mixed methods research can be viewed as an approach which draws upon the strengths and perspectives of each method, recognising the existence and importance of the physical, natural world as well as the importance of reality and influence of human experience (Johnson and Onquegbuzie, 2004). Rather than continue these debates in this paper, we aim to explore the approaches used to integrate qualitative and quantitative data within healthcare research to date. Accordingly, this paper focuses on the practical issues of conducting mixed methods studies and the need to develop a rigorous framework for designing and interpreting mixed methods studies to advance the field. In this paper, we will attempt to offer some guidance for those interested in mixed methods research on ways to combine qualitative and quantitative methods.

The concept of mixing methods was first introduced by Jick (1979), as a means for seeking convergence across qualitative and quantitative methods within social science research (Creswell, 2003). It has been argued that mixed methods research can be particularly useful in healthcare research as only a broader range of perspectives can do justice to the complexity of the phenomena studied (Clarke and Yaros, 1988, Foss and Ellefsen, 2002, Steckler et al., 1992). By combining qualitative and quantitative findings, an overall or negotiated account of the findings can be forged, not possible by using a singular approach (Bryman, 2007). Mixed methods can also help to highlight the similarities and differences between particular aspects of a phenomenon (Bernardi et al., 2007). Interest in, and expansion of, the use of mixed methods designs have most recently been fuelled by pragmatic issues: the increasing demand for cost effective research and the move away from theoretically driven research to research which meets policymakers’ and practitioners’ needs and the growing competition for research funding (Brannen, 2009, O’Cathain et al., 2007).

Tashakkori and Creswell (2007) broadly define mixed methods research as “research in which the investigator collects and analyses data, integrates the findings and draws inferences using both qualitative and quantitative approaches” (2007:3). In any mixed methods study, the purpose of mixing qualitative and quantitative methods should be clear in order to determine how the analytic techniques relate to one another and how, if at all, the findings should be integrated (O’Cathain et al., 2008, Onwuegbuzie and Teddlie, 2003). It has been argued that a characteristic of truly mixed methods studies are those which involve integration of the qualitative and quantitative findings at some stage of the research process, be that during data collection, analysis or at the interpretative stage of the research (Kroll and Neri, 2009). An example of this is found in mixed methods studies which use a concurrent data analysis approach, in which each data set is integrated during the analytic stage to provide a complete picture developed from both data sets after data has been qualitised or quantitised (i.e. where both forms of data have been converted into either qualitative or quantitative data so that it can be easily merged) (Onwuegbuzie and Teddlie, 2003). Other analytic approaches have been identified including; parallel data analysis, in which collection and analysis of both data sets is carried out separately and the findings are not compared or consolidated until the interpretation stage, and finally sequential data analysis, in which data are analysed in a particular sequence with the purpose of informing, rather than being integrated with, the use of, or findings from, the other method (Onwuegbuzie and Teddlie, 2003). An example of sequential data analysis might be where quantitative findings are intended to lead to theoretical sampling in an in depth qualitative investigation or where qualitative data is used to generate items for the development of quantitative measures.

When qualitative and quantitative methods are mixed in a single study, one method is usually given priority over the other. In such cases, the aim of the study, the rationale for employing mixed methods, and the weighting of each method determine whether, and how, the empirical findings will be integrated. This is less challenging in sequential mixed methods studies where one approach clearly informs the other, however, guidance on combining qualitative and quantitative data of equal weight, for example, in concurrent mixed methods studies, is rather less clear (Foss and Ellefsen, 2002). This is made all the more challenging by a common flaw which is to insufficiently and inexplicitly identify the relationships between the epistemological and methodological concepts in a particular study and the theoretical propositions about the nature of the phenomena under investigation (Kelle, 2001).

One approach to combining different data of equal weight and which facilitate clear identification of the links between the different levels of theory, epistemology, and methodology could be to frame triangulation as a ‘methodological metaphor’, as argued by Erzberger and Kelle (2003). This can help to; describe the logical relations between the qualitative and quantitative findings and the theoretical concepts in a study; demonstrate the way in which both qualitative and quantitative data can be combined to facilitate an improved understanding of particular phenomena; and, can also be used to help generate new theory (Erzberger and Kelle, 2003) (see Fig. 1 ). The points of the triangle represent theoretical propositions and empirical findings from qualitative and quantitative data while the sides of the triangle represent the logical relationships between these propositions and findings. The nature and use of the triangle depends upon the outcome from the analysis, whether that be convergent, where qualitative and quantitative findings lead to the same conclusion; complementary, where qualitative and quantitative results can be used to supplement each other or; divergent, where the combination of qualitative and quantitative results provides different (and at times contradictory) findings. Each of these outcomes requires a different way of using the triangulation metaphor to link theoretical propositions to empirical findings (Erzberger and Kelle, 2003).

Fig. 1.

Illustrating the triangulation triangle (Erzberger and Kelle, 2003)

1.1. Purpose of this paper

In the following paper, we identify the analytical approaches used in mixed methods healthcare research and exemplify the use of triangulation (Erzberger and Kelle, 2003) as a methodological metaphor for drawing inferences from qualitative and quantitative findings. Papers reporting on mixed methods studies within healthcare research were reviewed to (i) determine the type of analysis approach used, i.e. parallel, concurrent, or sequential data analysis and, (ii) identify studies which could be used to illustrate the use of the methodological metaphor of triangulation suggested by Erzberger and Kelle (2003). Four papers were selected to illustrate the application of the triangulation metaphor on complementary, convergent and divergent outcomes and to develop theory.

2. Methods

This literature review has used systematic principles (Cochrane, 2009, Khan, 2001) to search for mixed methods studies within healthcare research. The first search was conducted in September 2009 in the data bases CINAHL, Medline and PsycINFO on papers published in English language between 1999 and 2009. To identify mixed methods studies, the search terms (used as keywords and where possible as MeSH terms) were: “mixed methods”, “mixed research methods”, “mixed research”, “triangulation”, “complementary methods”, “concurrent mixed analysis” and “multi-strategy research.” These terms were searched individually and then combined (with OR). This resulted in 1896 hits in CINAHL, 1177 in Medline and 1943 in PsycINFO.

To focus on studies within, or relevant to, a healthcare context the following search terms were used (as keywords or as MeSH terms and combined with OR): “health care research”; “health services research”; and “health”. These limits applied to the initial search (terms combined with AND) resulted in 205 hits in Medline and 100 hits in PsycINFO. Since this combination in CINAHL only limited the search results to 1017; a similar search was conducted but without using the search term triangulation to capture mixed methods papers; resulting in 237 hits. In CINAHL the search result on 1017 papers was further limited by using “interventions” as a keyword resulting in 160 papers also selected to be reviewed. Moreover; in Medline the mixed methods data set was limited by the MeSH term “research” resulting in 218 hits and in PsycINFO with “intervention” as keyword or MeSH term resulting in 178 hits.

When duplicates were removed the total numbers of papers identified were 843. The abstracts were then reviewed by each author and those identified as relevant to the review were selected to be retrieved and reviewed in full text. Papers were selected based on the following inclusion criteria: empirical studies; published in peer review journals; healthcare research (for the purpose of this paper defined as any study focussing on participants in receipt, or involved in the delivery, of healthcare or a study conducted within a healthcare setting, e.g. different kinds of care, health economics, decision making, and professionals’ role development); and using mixed methods (defined as a study in which both qualitative and quantitative data were collected and analysed (Halcomb et al., 2009b). To maintain rigour, a random sample (10%) of the full text papers was reviewed jointly by two authors. Any disagreements or uncertainties that arose between the reviewers regarding their inclusion or in determining the type of analytic approach used were resolved through discussion between the authors.

In addition to the criteria outlined above, papers were excluded if the qualitative element constituted a few open-ended questions in a questionnaire, as we would agree with previous authors who have argued such studies do not strictly constitute a mixed methods design (Kroll and Neri, 2009). Papers were also excluded if they could not be retrieved in full text via the library services at the University of Edinburgh, Glasgow Caledonian University or the Karolinska Institutet, or did not adequately or clearly describe their analytic strategy, for example, failing to report how the qualitative and quantitative data sets were analysed individually and, where relevant, how these were integrated. See Table 1 for reasons for the exclusion of subsequent papers.

Table 1.

Reasons for exclusion.

Reason for exclusion	Number of papers
Not empirical study/not published in peer reviewed journal/full text unavailable	296
Not mixed method report/both data sets not presented or retrievable	287
Not healthcare research	88
Analysis/findings not clearly describeda	4

^aDue to lack of clarity in these papers method and/or result presentations it was not possible to determine what analysis technique had been used.

A second search was conducted within the databases of Medline, PsychInfo and Cinahl to identify studies which have specifically used Erzberger and Kelle's (2003) triangulation metaphor to frame the description and interpretation of their findings. The term ‘triangulation metaphor’ (as keywords) and author searches on ‘Christian Erzberger’ and ‘Udo Kelle’ were conducted. Three papers, published by Christian Erzberger and Udo Kelle, were identified in the PsychInfo databases but none of these were relevant to the purpose of this review. There were no other relevant papers identified in the other two databases.

168 Papers were included in the final review and reviewed to determine the type of mixed analysis approach used, i.e. parallel, concurrent, or sequential mixed analysis. Four of these papers (identified from the first search on mixed methods studies and healthcare research) were also used to exemplify the use of the methodological metaphor of triangulation (Erzberger and Kelle, 2003). Data was extracted from included papers accordingly in relation to these purposes.

3. Results

In total, 168 papers were included in our review. The majority of these studies originated in the USA (n  = 63), the UK (n  = 39) and Canada (n  = 19), perhaps reflecting the considerable interest and expertise in mixed methods research within these countries. The focus of the studies included in the review varied significantly and the populations studied included both patients and healthcare professionals.

3.1. Analytic approaches

Table 2 illustrates the types of analytic approaches adopted in each of the studies included in the review. The most widely used analytic approach (n  = 98) was parallel analysis (Creswell and Plano Clark, 2007). However, very few of the studies employing parallel analysis clearly articulate their purpose for mixing qualitative and quantitative data, the weighting (or priority) given to the qualitative and quantitative data or the expected outcomes from doing so, mirroring previous research findings (O’Cathain et al., 2008). The weighting, or priority, of the qualitative and quantitative data in a mixed methods study is dependent upon various factors including; the aims of the study and whether the purpose is, for example, to contextualise quantitative data using qualitative data or to use qualitative data to inform a larger quantitative approach such as a survey. Nonetheless, the omission of this statement makes it difficult to determine which data set the conclusions have been drawn from and the role of, or emphasis on, each approach. Therefore, is of importance for authors to clearly state this in their papers (Creswell and Plano Clark, 2007). A number of studies had also used sequential data analysis (n  = 46), where qualitative approaches were visibly used to inform the development of both clinical tools (e.g. Canales and Rakowski, 2006) and research measures and surveys (e.g. Beatty et al., 2004) or where quantitative surveys were supplemented by and issues further explored using qualitative approaches (e.g. Abadia and Oviedo, 2009, Cheng, 2004, Halcomb et al., 2008).

Table 2.

Included papers illustrating their analytical approach and country of origin.

Papers used a sequential analytic approach (n = 46) References (country of origin)

Abadia, C.E., Oviedo, D.G., 2009. Bureaucratic Itineraries in Colombia. A theoretical and methodological tool to assess managed-care health care systems. Social Science & Medicine 68 (6), 1153–1160. (South America)

Anderson, K.K., Sebaldt, R.J., Lohfeld, L., Karwalajtys, T., Ismaila, A.S., Goeree, R., Donald, F.C., Burgess, K., Kaczorowski, J., 2008. Patient views on reminder letters for influenza vaccinations in an older primary care patient population: a mixed methods study. Canadian Journal of Public Health 99 (2), 133–136. (Canada)

Aubel, J., Toure, I., Diagne, M., 2004. Senegalese grandmothers promote improved maternal and child nutrition practices: the guardians of tradition are not averse to change. Social Science & Medicine 59 (5), 945–959. (Italy/Sengal)

Bailey, A., Hutter, I., 2008. Qualitative to quantitative: linked trajectory of method triangulation in a study on HIV/AIDS in Goa, India. AIDS Care 20 (9), 1119–1124. (The Netherlands)

Beatty, P.W., Neri, M.T., Bell, K., DeJong, G., 2004. Use of outcomes information in acute inpatient rehabilitation. American Journal of Physical Medicine & Rehabilitation 83 (6), 468–478. (USA)

Bennet, I., Switzer, J., Aguirre, A., Evans, K., Barg, F., 2006. ‘Breaking it down’: Patient-clinician communication and prenatal care among African American women of low and higher literacy. Annals of Family Medicine. 4 (4), 334–340. (USA)

Brazier, A., Cooke, K., Moravan, V., 2008. Using mixed methods for evaluating an integrative approach to cancer care: a case study. Integrative Cancer Therapies 7 (1), 5–17. (2008) (Canada)

Canales, M.K., Rakowski, W., 2006. Development of a culturally specific instrument for mammography screening: an example with American Indian women in Vermont. Journal of Nursing Measurement 14 (2), 99–115. (USA)

Cheng, G.Y., 2004. A study of clinical questions posed by hospital clinicians. Journal of the Medical Library Association 92 (4), 445–458. (China)

Cole, M., 2009. Exploring the hand hygiene competence of student nurses: a case of flawed self assessment. Nurse Education Today 29, 380–388. (UK)

Coombes, L., Allen, D., Marsh, M., Foxcroft, D., 2009. The strengthening families programme (SFP) 10–14 and subscale misuse in Barnsley: the perspective of facilitators and families. Child Abuse Review 18 (1), 41–59. (UK)

Cox, P., McNair, R., 2009. Risk reduction as an accepted framework for safer-sex promotion among women who have sex with women. Sexual Health 6 (1), 15–18. (Australia)

Daley, E., Perrin, K., Vamos, C., Webb, C., Mueller, T., Packing-Ebuen, J., Rayko, H., McFarlane, M., and McDermott, R., 2008. HPV knowledge among HPV+ women. American Journal of Health Behaviour 32 (5), 477–487. (USA)

Davila, Y.R., 2006. Increasing nurses’ knowledge and skills for enhanced response to intimate partner violence. Journal of Continuing Education in Nursing 37 (4), 171–177. (USA)

Dibb, B., Yardley, L., 2006. Factors important for the measurement of social comparison in chronic illness: a mixed-methods study. Chronic Illness 2 (3), 219–230. (UK)

Goodridge, D., Duggleby, W., Gjevre, J., Rennie, D., 2009. Exploring the quality of dying of patients with chronic obstructive pulmonary disease in the intensive care unit: a mixed methods study. Nursing in Critical Care 14 (2), 51–60. (Canada)

Graham, R.J., Pemstein, D.M., Palfrey, J.S., 2008. Included but isolated: early intervention programmes provision for children and families with chronic respiratory support needs. Child: Care, Health and Development 34 (3), 373–379. (USA)

Halcomb, E.J., Davidson, P.M., Griffiths, R., Daly, J., 2008. Cardiovascular disease management: time to advance the practice nurse role? Australian Health Review 32 (1), 44–53. (Australia)

Harding, R., Molloy, T., 2008. Positive futures? The impact of HIV infection on achieving health, wealth and future planning. AIDS care 20 (5), 565–570. (UK)

Hornick, T.R., Higgins, P.A., Stollings, C., Wetzel, L., Barzilai, K., Wolpaw, D., 2006. Initial evaluation of a computer-based medication management tool in a geriatric clinic. The American Journal of Geriatric Pharmacotherapy 4 (1), 62–69. (USA)

Im, E.O., Meleis, A.I., 2001. Women's work and symptoms during midlife: Korean immigrant women. Women & Health 33 (1–2), 83–103. (USA)

Jinks, C., Ong, B.N., Richardson, J., 2007. A mixed methods study to investigate needs assessment for knee pain and disability: population and individual perspectives. BMC Musculoskeletal Disorders 8, 59. (UK)

Jones, N.R., Haynes, R., 2006. The association between young people's knowledge of sexually transmitted diseases and their behaviour: A mixed methods study. Health, Risk & Society 8 (3), 293–303. (UK)

Kaczorowski, J., Karwalajtys, T., Lohfeld, L., Laryea, S., Anderson, K., Roder, S., Sebaldt, R.J., 2009. Women's views on reminder letters for screening mammography. Canadian Family Physician 55 (6), 622-3.e1-4. (Canada)

Kaldjian, L.C., Jones, E.W., Rosenthal, G.E., Tripp-Reimer, T., Hillis, S.L., 2006. An empirically derived taxonomy of factors affecting physicians’ willingness to disclose medical errors. Journal of General Internal Medicine 21 (9), 942–948. (USA)

Kelly, K.M., Phillips, C.M., Jenkins, C., Norling, G., White, C., Jenkins, T., Armstrong, D., Petrik, J., Steinkuhl, A., Washington, R., Dignan, M., 2007. Physician and staff perceptions of barriers to colorectal cancer screening in Appalachian Kentucky. Cancer Control 14 (2), 167–175. (USA)

King, M., Jones, L., Richardson, A., Murad, S., Irving, A., Aslett, H., Ramsay, A., Coelho, H., Andreou, P., Tookman, A., Mason, C., Nazareth, I., 2008. The relationship between patients’ experiences of continuity of cancer care and health outcomes: a mixed methods study. British Journal of Cancer 98 (3), 529–536. (UK)

Kinter, E.T., Schmeding, A., Rudolph, I., dosReis, S., Bridges, J.F., 2009. Identifying patient-relevant endpoints among individuals with schizophrenia: an application of patient-centered health technology assessment. International Journal of Technology Assessment in Health Care 25 (1), 35–41. (Germany)

Krein, S.L., Kowalski, C.P., Damschroder, L., Forman, J., Kaufman, S.R., Saint, S., 2008. Preventing ventilator-associated pneumonia in the United States: a multicenter mixed-methods study. Infection Control and Hospital Epidemiology 29 (10), 933–940. (USA)

Krouse, R.S., Mohler, M.J., Wendel, C.S., Grant, M., Baldwin, C.M., Rawl, S.M., McCorkle, R., Rosenfeld, K.E., Ko, C.Y., Schmidt, C.M., Coons, S.J., 2006. The VA Ostomy Health-Related Quality of Life Study: objectives, methods, and patient sample. Current Medical Research and Opinion 22 (4), 781–791. (USA)

Lee, G.M., Santoli, J.M., Hannan, C., Messonnier, M.L., Sabin, J.E., Rusinak, D., Gay, C., Lett, S.M., Lieu, T.A., 2007. Gaps in vaccine financing for underinsured children in the United States. JAMA 298 (6), 638–643. (USA)

Leviton, L.C., Finnegan, J.R.J., Zapka, J.G., Meischke, H., Estabrook, B., Gilliland, J., Linares, A., Weitzman, E.R., Raczynski, J., Stone, E., 1999. Formative research methods to understand patient and provider responses to heart attack symptoms. Evaluation and Program Planning 22 (4), 385–397. (USA)

Lukkarinen, H., 2005. Methodological triangulation showed the poorest quality of life in the youngest people following treatment of coronary artery disease: a longitudinal study. International Journal of Nursing Studies 42 (6), 619–627. (Finland)

Nakkash, R., Afifi Soweid, R.A., Nehlawi, M.T., Shediac-Rizkallah, M.C., Hajjar, T.A., Khogali, M., 2003. The development of a feasible community-specific cardiovascular disease prevention program: triangulation of methods and sources. Health Education & Behavior 30 (6), 723–739. (Lebanon)

Norris, C., 2008. Evaluation of a back stability CPD course. Journal of Bodywork and Movement Therapies 12, 305–311. (UK)

Portillo, M.C., Corchon, S., Lopez-Dicastillo, O., Cowley, S., 2009. Evaluation of a nurse-led social rehabilitation programme for neurological patients and carers: an action research study. International Journal of Nursing Studies 46 (2), 204–219. (Spain)

Santos, S.R., Carroll, C.A., Cox, K.S., Teasley, S.L., Simon, S.D., Bainbridge, L., Cunningham, M., Ott, L., 2003. Baby boomer nurses bearing the burden of care. Journal of Nursing Administration 33 (4), 243–250. (USA)

Shih, S.N., Gau, M.L., Kao Lo, C.H., Shih, F.J., 2005. Health Needs Instrument for hospitalized single-living Taiwanese elders with heart disease: triangulation research design. Journal of Clinical Nursing 14 (10), 1210–1222. (Taiwan)

Shih, S.N., Gau, M.L., Tsai, J.C., Li, I.C., Liou, Y.M., Shih, F.J., 2008. A Health Need Satisfaction Instrument for Taiwan's single-living older people with chronic disease in the community. Journal of Clinical Nursing 17 (5A), 67–77. (Taiwan)

Shin, Y., Pender, N.J., Yun, S., 2003. Using methodological triangulation for cultural verification of commitment to a plan for exercise scale among Korean adults with chronic diseases. Research in Nursing & Health 26 (4), 312–321. (Korea)

Strolla, L.O., Gans, K.M., Risica, P.M., 2006. Using qualitative and quantitative formative research to develop tailored nutrition intervention materials for a diverse low-income audience. Health Education Research 21 (4), 465–476. (USA)

Sussman, A.L., Williams, R.L., Leverence, R., Gloyd, P.W., Jr., Crabtree, B.F., 2006. The art and complexity of primary care clinicians’ preventive counseling decisions: obesity as a case study. Annals of Family Medicine 4 (4), 327–333. (USA)

Thogersen-Ntoumani, C., Fox, K.R., 2005. Physical activity and mental well-being typologies in corporate employees: A Mixed methods approach. Work & Stress 19 (1), 50–67. (UK)

Weine, S., Knafl, K., Feetham, S., Kulauzovic, Y., Klebic, A., Sclove, S., Besic, S., Mujagic, A., Muzurovis, J., Spahovic, D., 2005. A mixed methods study on refugee families engaging in multiple-family groups. Family Relations 54 (4), 558–568. (USA)

Werrett, J.A., Helm, R.H., Carnwell, R., 2001. The primary and secondary care interface: the educational needs of nursing staff for the provision of seamless care. Journal of Advanced Nursing 34 (5), 629–638. (UK)

Wittink, M.N., Barg, F.K., Gallo, J.J., 2006. Unwritten rules of talking to doctors about depression: Integrating qualitative and quantitative methods. Annals Of Family Medicine 4 (4), 302–309. (USA)

Papers used a parallel analytic approach (n = 98) References (country of origin)

Andreatta, P.B., Hillard, M.L., Murphy, M.A., Gruppen, L.D., Mullan, P.B., 2009. Short-term outcomes and long-term impact of a programme in medical education for medical students. Medical Education 43 (3), 260–267. (USA)

Arbon, P., Bail, K., Eggert, M., Gardner, A., Hogan, S., Phillips, C., van Dieman, N., Waddington, G., 2009. Reporting a research project on the potential of aged care nurse practitioners in the Australian Capital Territory. Journal of Clinical Nursing 18 (2), 255–262. (Australia)

Aronson, B.S., Rebeschi, L.M., Killion, S.W., 2007. Enhancing evidence bases for interventions in a baccalaureate program. Nursing Education Perspectives 28 (5), 257–262. (USA)

Bachmann, M.O., O’Brien, M., Husbands, C., Shreeve, A., Jones, N., Watson, J., Reading, R., Thoburn, J., Mugford, M., 2009. Integrating children's services in England: national evaluation of children's trusts. Child: Care, Health and Development 35 (2), 257–265. (UK)

Barnes, D., Carpenter, J., Dickinson, C., 2006. The outcomes of partnerships with mental health service users in interprofessional education: a case study. Health & Social Care in the Community 14 (5), 426–435. (UK)

Beecher, B., 2009. Mental health practitioners’ views of the families of individuals with schizophrenia and barriers to collaboration: a mixed methods study. Journal of Family Social Work 12, 264–282. (USA)

Beck, S.L., Towsley, G.L., Caserta, M.S., Lindau, K., Dudley, W.N,. 2009. Symptom experiences and quality of life of rural and urban older adult survivors. Cancer Nursing 32 (5), 359–369. (USA)

Benoit, C., Westfall, R.D., Treloar, A.E.B., Phillips, R.S., Jansson, S.M., 2007. Social factors linked to postpartum depression: A Mixed-methods longitudinal study. Journal of Mental Health 16 (6), 719–730. (Canada)

Bidmead, C., Cowley, S., 2005. Evaluating family partnership training in health visitor practice. Community practitioner 78 (7), 239–245. (UK)

Bogner, H.R., Cahill, E., Frauenhoffer, C., Barg, F.K., 2009. Older primary care patient views regarding antidepressants: a mixed methods approach. Journal of Mental Health 18 (1), 57–64. (USA)

Cartwright, E., Schow, D., Herrera, S., Lora, Y., Mendez, M., Mitchell, D., Pedroza, E., Pedroza, L., Trejo, A., 2006. Using participatory research to build an effective type 2 diabetes intervention: the process of advocacy among female Hispanic farmworkers and their families in Southeast Idaho. Women & Health 43 (4), 89–109. (USA)

Chan, E.A., Chung, J.W., Wong, T.K., 2008. Learning from the severe acute respiratory syndrome (SARS) epidemic. Journal of Clinical Nursing 17 (8), 1023–1034. (Hong Kong)

Cheater, F.M., Hearnshaw, H., Baker, R., Keane, M., 2005. Can a facilitated programme promote effective multidisciplinary audit in secondary care teams? An exploratory trial. International Journal of Nursing Studies 42 (7), 779–791. (UK)

Comino, E.J., Zwar, N.A., Hermiz, O., 2007. The Macarthur GP after-hours service: a model of after-hours care for Australia. Australian Health Review 31 (2), 223–230. (Australia)

dela Cruz, F.A., Brehm, C., Harris, J., 2004. Transformation in family nurse practitioner students’ attitudes toward homeless individuals after participating in a homeless outreach clinic. Journal of the American Academy of Nurse Practitioners 16 (12), 547–554. (USA)

Davidson, P., Digiacomo, M., Zecchin, R., Clarke, M., Paul, G., Lamb, K., Hancock, K., Chang, E., Daly, J., 2008. A cardiac rehabilitation program to improve psychosocial outcomes of women with heart disease. Journal of Women's Health 17 (1), 123–134. (Australia)

Dodding, C.J., Nasel, D.D., Murphy, M., Howell C., 2008. All in for mental health: a pilot study of group therapy for people experiencing anxiety and/or depression and a significant other of their choice. Mental Health in Family Medicine 5, 41–49. (Australia)

Dongre, A.R., Deshmukh, P.R., Garg, B.S., 2008. Perceptions and health care seeking about newborn danger signs among mothers in rural Wardha. Indian Journal of Pediatrics 75 (4), 325–329. (India)

Donnelly, C., Macmillan, M.S., 2007. Devising and evaluating a model for teaching therapeutic handling and moving skills to complementary therapists. Complementary Therapies in clinical Practice 13, 201–209. (UK)

Ducharme, F., Lebel, P., Lachance, L., Trudeau, D., 2006. Implementation and effects of an individual stress management intervention for family caregivers of an elderly relative living at home: a mixed research design. Research in Nursing & Health 29 (5), 427–441. (Canada)

Duggleby, W., Wright, K., Williams, A., Degner, L., Cammer, A., Holtslander, L., 2007. Developing a living with hope program for caregivers of family members with advanced cancer. Journal of Palliative Care 23 (1), 24–31. (Canada)

Egilson, S.T., Traustadottir, R., 2009. Participation of students with physical disabilities in the school environment. The American Journal of Occupational Therapy 63 (3), 264–272. (Iceland)

Etowa, J., Keddy, B., Egbeyemi, J., Eghan, F., 2007. Depression: the ‘invisible grey fog’ influencing the midlife health of African Canadian women. International Journal of Mental Health Nursing 16 (3), 203–213. (Canada)

Farrell, J., Anderson, S., Hewitt, K., Livingston, M.H., Stewart, D., 2007. A survey of occupational therapists in Canada about their knowledge and use of the ICF. Canadian Journal of Occupational Therapy 74 Spec No., 221–232. (Canada)

Felton, G., Saunders, R.P., Ward, D.S., Dishman, R.K., Dowda, M., Pate, R.R., 2005. Promoting physical activity in girls: a case study of one school's success. The Journal of School Health 75 (2), 57–62. (USA)

Finn, L.D., Bishop, B., Sparrow, N.H., 2007. Mutual help groups: an important gateway to wellbeing and mental health. Australian Health Review 31 (2), 246–255. (Australia)

Finucane, A., Mercer, S.W., 2006. An exploratory mixed methods study of the acceptability and effectiveness of Mindfulness-Based Cognitive Therapy for patients with active depression and anxiety in primary care. BMC Psychiatry 6, 14. (UK)

Flocke, S.A., Gordon, L.E., Pomiecko, G.L., 2006. Evaluation of a community health promotion resource for primary care practices. American Journal of Preventive Medicine 30 (3), 243–251. (USA)

Galantino, M.L., Shepard, K., Krafft, L., Laperriere, A., Ducette, J., Sorbello, A., Barnish, M., Condoluci, D., Farrar, J.T., 2005. The effect of group aerobic exercise and t’ai chi on functional outcomes and quality of life for persons living with acquired immunodeficiency syndrome. Journal of Alternative and Complementary Medicine 11 (6), 1085–1092. (USA)

Glik, D.C., Parker, K., Muligande, G., Hategikamana, B., 2005. Integrating qualitative and qualitative survey techniques. 1986–87. International Quarterly of Community Health Education 25 (1–2), 115–133. (USA)

Godley, S.H., Passetti, L.L., White, M.K., 2006. Employment and adolescent alcohol and drug treatment and recovery: an exploratory study. The American Journal on Addictions 15 Suppl. 1, 137–143. (USA)

Graff, M.J., Vernooij-Dassen, M., Zajec, J., Olde-Rikkert, M., Hoefnagels, W., Dekker, J., 2006. How can occupational therapy improve the daily performance and communication of an older patient with dementia and his primary caregiver? A case study. Dementia 5, 503–532. (The Netherlands)

Gustafson, D.L., Goodyear, L., Keough, F., 2008. When the dragon's awake: a needs assessment of people injecting drugs in a small urban centre. The International Journal on Drug Policy 19 (3), 189–194. (Canada)

Happ, M.B., Roesch, T.K., Kagan, S.H., 2005. Patient communication following head and neck cancer surgery: a pilot study using electronic speech-generating devices. Oncolology Nursing Forum 32 (6), 1179–1187. (USA)

Hoffman-Goetz, L., Donelle, L., 2007. Chat room computer-mediated support on health issues for aboriginal women. Health Care for Women International 28 (4), 397–418. (Canada)

Hohenadel, J., Kaegi, E., Laidlaw, J., Kovacik, G., Cortinois, A., Kang, R., Jadad, A.R., 2007. Leveling the playing field: the personal coach program as an innovative approach to assess and address the supportive care needs of underserved cancer patients. The Journal of Supportive Oncology 5 (4), 185–193. (Canada)

Holtzmann, J., Timm, H., 2005. The experiences of and the nursing care for breast cancer patients undergoing immediate breast reconstruction. European Journal of Cancer Care 14 (4), 310–318. (Denmark)

Hoskins, R., Gow, A., McDowell, J., 2007. Corporate solutions to caseload management—an evaluation. Community Practitioner 80 (9), 20–24. (UK)

Huyck, M.H., Ayalon, L., Yoder, J., 2007. Using mixed methods to evaluate the use of a caregiver strain measure to assess outcomes of a caregiver support program for caregivers of older adults. International Journal of Geriatric Psychiatry 22 (2), 160–165. (USA)

Hyrkas, K., Paunonen, M., 2000. Patient satisfaction and research-related problems (Part 2). Is triangulation the answer? Journal of Nursing Management 8 (4), 237–245. part 2 of 2 part paper (Finland)

Hyrkas, K., Paunonen, M., Laippala, P., 2000. Patient satisfaction and research-related problems (Part 1). Problems while using a questionnaire and the possibility to solve them by using different methods of analysis. Journal of Nursing Management 8 (4), 227–236. part 1 of 2 part paper (Finland)

Jurgens, C.Y., Hoke, L., Byrnes, J., Riegel, B., 2009. Why do elders delay responding to heart failure symptoms? Nursing Research 58 (4), 274–282. (USA)

Kaushal, M., Aggarwal, R., Singal, A., Shukla, H., Kapoor, S.K., Paul, V.K., 2005. Breastfeeding practices and health-seeking behavior for neonatal sickness in a rural community. Journal of Tropical Pediatrics 51 (6), 366–376. (India)

Kemp, L.A., Harris, E., Comino, E.J., 2005. Changes in community nursing in Australia: 1995–2000. Journal of Advanced Nursing 49 (3), 307–314. (Australia)

Killaspy, H., Johnson, S., Pierce, B., Bebbington, P., Pilling, S., Nolan, F., King, M., 2009. Successful engagement: a mixed methods study of the approaches of assertive community treatment and community mental health teams in the REACT trial. Social Psychiatry and Psychiatric Epidemiology 44 (7), 532–540. (UK)

Knifton, L., Walker, A., Quinn, N., 2009. Workplace interventions can reduce stigma. Journal of Public Mental Health 7 (4), 40–50. (UK)

Kramer, M., Schmalenberg, C., Maguire, P., Brewer, B.B., Burke, R., Chmielewski, L., Cox, K., Kishner, J., Krugman, M., Meeks-Sjostrom, D., Waldo, M., 2008. Structures and practices enabling staff nurses to control their practice. Western Journal of Nursing Research 30 (5), 539–559. (USA)

Kubicek, K., Ramirez, M., Limbos, M.A., Iverson, E., 2008. Knowledge and behaviors of parents in planning for and dealing with emergencies. Journal of Community Health 33 (3), 158–168. (USA)

Lacey, E.A., Kalsi, G.S., Macintosh, M.J., 2004. Mixed method evaluation of an innovation to improve secondary prevention of coronary heart disease in primary care. Quality in Primary Care 12, 259–266. (UK)

Latham, J., Moffat, T., 2007. Determinants of variation in food cost and availability in two socioeconomically contrasting neighborhoods of Hamilton, Ontario, Canada. Health & Place 13, 273–287. (Canada)

Lee, S.Y., Hoerr, S.L., Weatherspoon, L., Schiffman, R.F., 2008. Nutrition students improve attitudes after a guided experiential assignment with older adults. Journal of Nutrition Education and Behavior 40 (5), 279–287. (USA)

Liang, W., Kasman, D., Wang, J.H., Yuan, E.H., Mandelblatt, J.S., 2006. Communication between older women and physicians: preliminary implications for satisfaction and intention to have mammography. Patient Education and Counseling 64 (1–3), 387–392. (USA)

Llewellyn, P., Northway, R., 2007. The views and experiences of learning disability nurses concerning their advocacy education. Nurse Education Today 27 (8), 955–963. (UK)

Macpherson, C.F., 2008. Peer-supported storytelling for grieving pediatric oncology nurses. Journal of Pediatric Oncology Nursing 25 (3), 148–163. (USA)

Mason, S., O’Keeffe, C., Coleman, P., Edlin, R., Nocholl, J., 2007. Effectiveness of emergency care practitioners working within existing emergency service model of care. Emergency Medical Journal 24, 239–243. (UK)

Mbonye, A.K., Bygbjerg, I.C., Magnussen, P., 2008. Prevention and treatment practices and implications for malaria control in Mukono District Uganda Journal of Biosocial Science 40 (2), 283–296. (Uganda)

McAyley, C., McCurry, N., Knapp, M., Beecham, J., Sleed, M., 2006. Young families under stress: assessing maternal and child well-being using a mixed-methods approach. Child and Family Social Work 11, 43–54. (UK)

Melancon, J., Oomen-Early, J., del Rincon, L.M., 2009. Using the PEN-3 model to assess knowledge, attitudes and beliefs about diabetes type 2 among Mexican American and Mexican Native men and women in North Texas. International Electronic Journal of Health Education 12, 203–221. (USA)

Midtgaard, J., Rorth, M., Stelter, R., Adamsen, L., 2006. The group matters: an explorative study of group cohesion and quality of life in cancer patients participating in physical exercise intervention during treatment. European Journal of Cancer Care 15 (1), 25–33. (Denmark)

Mills, M.E., Murray, L.J., Johnston, B.T., Donnelly, M., 2008. Feasibility of a standardised quality of life questionnaire in a weekly diary format for inoperable lung cancer patients. European Journal of Oncology Nursing 12 (5), 457–463. (UK)

Mimiaga, M.J., Reisner, S.L., Tinsley, J.P., Mayer, K.H., Safren, S.A., 2009. Street workers and internet escorts: contextual and psychosocial factors surrounding HIV risk behavior among men who engage in sex work with other men. Journal of Urban Health 86 (1), 54–66. (USA)

Minvielle, E., Sicotte, C., Champagne, F., Contandriopoulos, A.P., Jeantet, M., Preaubert, N., Bourdil, A., Richard, C., 2008. Hospital performance: competing or shared values? Health Policy 87 (1), 8–19. (France)

Mirza, M., Anandan, N., Madnick, F., Hammel, J., 2006. A participantory program evaluation of a sustem change program to improve access to infomration technology by people with disabilities. Disability and Rehabilitation 28 (19), 1185–1199. (USA)

Mirza, I., Tareen, A., Davidson, L.L., Rahman, A., 2009. Community management of intellectual disabilities in Pakistan: a mixed methods study. Journal of Intellectual Disability Research 53 (6), 559–570. (Pakistan)

Monasta, L., Andersson, N., Ledogar, R.J., Cockcroft, A., 2008. Minority health and small numbers epidemiology: a case study of living conditions and the health of children in 5 foreign Roma camps in Italy. American Journal of Public Health 98 (11), 2035–2041. (Italy)

Mortenson, W.B., Miller, W.C., Miller-Pogar, J., 2007. Measuring wheelchair intervention outcomes: development of the wheelchair outcome measure. Disability and rehabilitation. Assistive Technology 2 (5), 275–285. (Canada)

Newton, P.J., Halcomb, E.J., Davidson, P.M., Denniss, A.R., 2007. Barriers and facilitators to the implementation of the collaborative method: reflections from a single site. Quality & Safety in Health Care 16 (6), 409–414. (Australia)

Oh, H.S., Park, H.A., 2004. Decision tree model of the treatment-seeking behaviors among Korean cancer patients. Cancer Nursing 27 (4), 259–266. (Korea)

Oliver, D.P., Wittenberg-Lyles, E., Demiris, G., Washington, K., Porock, D., Day, M., 2008. Barriers to pain management: caregiver perceptions and pain talk by hospice interdisciplinary teams. Journal of Pain and Symptom Management 36 (4), 374–382. (UK)

Olson, T., Vera, B., Perez, O., 2006. Preliminary study of OCD and health disparities at the U.S.-boarder. Hispanic Health Care International 4 (2), 89–99. (USA)

Ostler, T., Haight, W., Black, J., Choi, G.Y., Kingery, L., Sheridan, K., 2007. Case series: mental health needs and perspectives of rural children reared by parents who abuse methamphetamine. Journal of the American Academy of Child and Adolescent Psychiatry 46 (4), 500–507. (USA)

Ralphs-Thibodeau, S., Knoefel, F., Benjamin, K., Leclrec, A., Pisterman, S., Sohmer, J., Scrim, C., 2006. Patient choice: an influencgin factor on policy-related research to decrease bedrail use as physical restrain. Worldviews on Evidence-Based Nursing 3 (1), 31–39. (Canada)

Rosenfeld, A.G., 2004. Treatment-seeking delay among women with acute myocardial infarction: decision trajectories and their predictors. Nursing Research 53 (4), 225–236. (USA)

Sanders, R., Roach, G., 2007. Closing the gap? The effectiveness of referred access family support services. Child & Family Social Work 12 (2), 161–171. (UK)

Shiell, A., Gold, L., 2003. If the price is right: vagueness and values clarification in contingent valuation. Health Economics 12 (11), 909–919. (UK)

Shipman, C., Burt, J., Ream, E., Beynon, T., Richardson, A., Addington-Hall, J., 2008. Improving district nurses’ confidence and knowledge in the principles and practice of palliative care. Journal of Advanced Nursing 63 (5), 494–505. (UK)

Sinha, S., Curtis, K., Jayakody, A., Viner, R., Roberts, H., 2007. ‘People make assumptions about our communities’: sexual health amongst teenagers from black and minority ethnic backgrounds in East london. Ethnicity & Health 12 (5), 423–441. (UK)

Skilbeck, J.K., Payne, S.A., Ingleton, M.C., Nolan, M., Carey, I., Hanson, A., 2005. An exploration of family carers’ experience of respite services in one specialist palliative care unit. Palliative Medicine 19 (8), 610–618. (UK)

Stacey, D., Graham, I.D., O’Connor, A.M., Pomey, M.P., 2005. Barriers and facilitators influencing call center nurses’ decision support for callers facing values-sensitive decisions: a mixed methods study. Worldviews on Evidence-Based Nursing 2 (4), 184–195. (Canada)

Stewart, M., Reutter, L., Letourneau, N., Makwarimba, E., 2009. A support intervention to promote health and coping among homeless youths. The Canadian Journal of Nursing Research 41 (2), 55–77. (Canada)

Suter, E., Hyman, M., Oelke, N., 2007. Measuring key integration outcomes: a case study of a large urban health center. Health Care Management Review 32 (3), 226–235. (Canada)

Sy, A., Glanz, K., 2008. Factors influencing teachers’ implementation of an innovative tobacco prevention curriculum for multiethnic youth: Project SPLASH. The Journal of School Health 78 (5), 264–273. (USA)

Symon, A.G., McStea, B., Murphy-Black, T., 2006. An exploratory mixed-methods study of Scottish midwives’ understandings and perceptions of clinical near misses in maternity care. Midwifery 22 (2), 125–136. (UK)

Thomason, S.S., Evitt, C.P., Harrow, J.J., Love, L., Moore, D.H., Mullins, M.A., Powell-Cope, G., Nelson, A.L., 2007. Providers’ perceptions of spinal cord injury pressure ulcer guidelines. The Journal of Spinal Cord Medicine 30 (2), 117–126. (USA)

Thorpe, K., Loo, R., 2003. Balancing professional and personal satisfaction of nurse managers: current and future perspectives in a changing health care system. Journal of Nursing Management 11 (5), 321–330. (Canada)

Thogersen-Ntoumani, C., Fox, K.R., 2005. Physical activity and mental well-being typologies in corporate employees: A Mixed methods approach. Work & Stress 19 (1), 50–67. (UK)

Tolmie, E.P., Lindsay, G.M., Kelly, T., Tolson, D., Baxter, S., Belcher, P.R., 2009. Are older patients’ cardiac rehabilitation needs being met? Journal of Clinical Nursing 18 (13), 1878–1888. (UK)

Tuicomepee, A., Romano, J.L., 2008. Thai adolescent survivors 1 year after the 2004 tsunami: A mixed methods study. Journal of Counseling Psychology 55 (3), 308–320. (Thailand)

Turnbull, B., Martinez-Andrade, G., Huerfano, N., Ryan, G.W., Martinez, H., 2009. A contrast between mothers’ assessments of child malnutrition and physical anthropometry in rural Mexico: a mixed methods community study. Journal of Nutrition Education and Behavior 41 (3), 201–206. (Mexico)

Valimaki, M., Anttila, M., Hetonen, H., Koivunen, M., Jakobsson, T., Pitkanen, A., Herrala, J., Kusmanen, L., 2008. Design and development process of patient-centered computer-based support system for patients with schizophrenia spectrum psychosis. Informatics for Health & Social Care 33 (2), 113–123. (Finland)

Waitzkin, H., Schillaci, M., Willging, C.E., 2008. Multimethod evaluation of health policy change: an application to Medicaid managed care in a rural state. Health Services Research 43 (4), 1325–1347. (USA)

Waldrop, D.P., 2007. Caregiver grief in terminal illness and bereavement: a mixed-methods study. Health % Social Work 32 (3), 197–206. (USA)

Walters, J.A., Hansen, E.C., Johns, D.P., Blizzard, E.L., Walters, E.H., Wood-Baker, R., 2008. A mixed methods study to compare models of spirometry delivery in primary care for patients at risk of COPD. Thorax 63 (5), 408–414. (Australia)

Ward, M., Cowman, S., 2007. Job satisfaction in psychiatric nursing. Journal of Psychiatric and Mental Health Nursing 14 (5), 454–461. (Ireland)

Williamson, K.M., 2007. Home health care nurses’ perceptions of empowerment. Journal of Community Health Nursing 24 (3), 133–153. (USA)

Williamson, G.R., Webb, C., Abelson-Mitchell, N., 2004. Developing lecturer practitioner roles using action research. Journal of Advanced Nursing 47 (2), 153–164. (UK)

Wilson, M.G., Goetzel, R.Z., Ozminkowski, R.J., DeJoy, D.M., Della, L., Roemer, E.C., Schneider, J., Tully, K.J., White, J.M., Baase, C.M., 2007. Using formative research to develop environmental and ecological interventions to address overweight and obesity. Obesity 15 Suppl. 1, 37S–47S. (USA)

Wye, L., Digby, K., 2008. Building research capacity amongst kinesiologists: results from a mixed methods study. Complementary Therapies in Clinical Practice 14 (1), 65–72. (UK)

Papers used a concurrent analytic approach (n = 20) References (country of origin)

Bussing, R., Koro-Ljungberg, M.E., Gary, F., Mason, D.M., Garvan, C.W., 2005. Exploring help-seeking for ADHD symptoms: a mixed-methods approach. Harvard Review of Psychiatry 13 (2), 85–101. (USA)

Dickson, V.V., McCauley, L.A., Riegel, B., 2008. Work-heart balance: the influence of biobehavioral variables on self-care among employees with heart failure. AAOHN Journal 56 (2), 63–73. (USA)

Girot, E., Rickaby, C., 2008. Education for new role development: the Community Matron in England. Journal of Advanced Nursing 64 (1), 38–48. (UK)

Harvey, W.J., Reid, G., Bloom, G.A., Staples, K., Grizenko, N., Mbekou, V., Ter-Stepanian, M., Joober, R., 2009. Physical activity experiences of boys with and without ADHD. Adapted Physical Activity Quarterly 26, 131–150. (USA)

Hatonen, H., Kuosmanen, L., Malkavaara, H., Valimaki, M., 2008. Mental health: patients’ experiences of patient education during inpatient care. Journal of Clinical Nursing 17 (6), 752–762. (Finland)

Hildon, Z., Smith, G., Netuveli, G., Blane, D., 2008. Understanding adversity and resilience at older ages. Sociology of health & illness 30 (5), 726–740. (UK)

Hodges, P., 2009. Factors impacting readmissions of older patients with heart failure. Critical Care Nursing Quarterly 32 (1), 33–43. (USA)

Holt, C.L., Schulz, E., Wynn, T.A., 2009. Perceptions of the religion—health connection among African Americans in the southeastern United States: sex, age, and urban/rural differences. Health Education & Behavior 36 (1), 62–80. (USA)

Kartalova-O’Doherty, Y., Doherty, D.T., 2008. Coping strategies and styles of family carers of persons with enduring mental illness: a mixed methods analysis. Scandinavian Journal of Caring Sciences 22 (1), 19–28. (Ireland)

Kartalova-O’Doherty, Y., Tedstone Doherty, D., 2009. Satisfied carers of persons with enduring mental illness: who and why? The International Journal of Social Psychiatry 55 (3), 257–271. (Ireland)

Knox, A.B., Underbaake, G., McBride, P.E., Mejicano, G.C., 2001. Organization development strategies for continuing medical education. The Journal of Continuing Education in the Health Professions 21 (1), 15–23. (USA)

Lehna, C., McNeil, J., 2008. Mixed-methods exploration of parents’ health information understanding. Clinical Nursing Research 17 (2), 133–144. (USA)

Lemon, S.C., Zapka, J.G., Estabrook, B., Benjamin, E., 2006. Challenges to research in urban community health centers. American Journal of Public Health 96 (4), 626–628. (USA)

Luck, L., Jackson, D., Usher, K., 2008. Innocent or culpable? Meanings that emergency department nurses ascribe to individual acts of violence. Journal of Clinical Nursing 17, 1071–1078. (Australia)

Marsiglia, F.F., Miles, B.W., Dustman, P., Sills, S., 2002. Ties That Protect: An Ecological Perspective on Latino/a Urban Pre-Adolescent Drug Use. Journal of Ethnic & Cultural Diversity in Social Work 11 (3–4), 191–220. (USA)

Powers, B.A., Watson, N.M., 2008. Meaning and practice of palliative care for nursing home residents with dementia at end of life. American Journal of Alzheimer's Disease and Other Dementias 23 (4), 319–325. (USA)

Proctor, E.K., Hasche, L., Morrow-Howell, N., Shumway, M., Snell, G., 2008. Perceptions about competing psychological problems and treatment priorities among older adults with depression. Psychiatric Services 59 (6), 670–675. (USA)

Riegel, B., Vaughan Dickson, V., Goldberg L.R., Deatrick, J.A., 2007. Factors associated with the development of expertice in heart failure self-care. Nursing Research 56 (4), 235–243. (USA)

Sands, N., 2004. Mental health triage nursing: an Australian perspective. Journal of Psychiatric and Mental Health Nursing 11 (2), 150–155. (Australia)

Yang, S., 2008. A mixed methods study on the needs of Korean families in the intensive care unit. Australian Journal of Advanced Nursing 25 (4), 79–86. (South Korea)

Papers used a combination of analytic approaches (n = 4) References (country of origin)

Juvonen-Posti, P., Piirainen, K., Kallanranta, T., Keinanen-Kiukaanniemi, S., 2004. The reality of returning to work and training: experiences from a long-term unemployment project. International Journal of Rehabilitation Research 27 (3), 215–227. (Finland)a

Merkouris, A., Papathanassoglou, E.D., Lemonidou, C., 2004. Evaluation of patient satisfaction with nursing care: quantitative or qualitative approach? International Journal of Nursing Studies 41 (4), 355–367. (Greece)b

Tanna, N.K., Pitkin, J., Anderson, C., 2005. Development of the specialist menopause pharmacist (SMP) role within a research framework. Pharmacy World & Science 27 (1), 61–67. (UK)c

Thompson, C., McCaughan, D., Cullum, N., Sheldon, T., Raynor, P., 2005. Barriers to evidence-based practice in primary care nursing—why viewing decision-making as context is helpful. Journal of Advanced Nursing 52 (4), 432–444. (UK)c

^aSequential and concurrent.

^bParallel and concurrent.

^cSequential and parallel.

Most notably, with only 20 included studies using a concurrent approach to data analysis, this was the least common design employed. Compared to the studies using a parallel or sequential approach, the authors of concurrent studies more commonly provided an explanation for their purpose of using a mixed methods design in their study, e.g. how it addressed a gap or would facilitate and advance the state of knowledge (e.g. Bussing et al., 2005, Kartalova-O’Doherty and Tedstone Doherty, 2009). Despite this, there remained a lack of clarity within these studies about the weighting given to, and priority of, each method. Consequently, the importance and relevance of the findings produced by each approach and how these have informed their conclusions and interpretation is lacking. In four of the included papers a combination of approaches to data analysis (i.e. sequential and concurrent, parallel and concurrent, or sequential and parallel) were used. In the next section, we have selected papers to illustrate the methodological metaphor of triangulation (Erzberger and Kelle, 2003).

3.2. Using the methodological metaphor of triangulation

We have selected four papers from our review (Lukkarinen, 2005, Midtgaard et al., 2006, Shipman et al., 2008, Skilbeck et al., 2005) to illustrate how the methodological metaphor of triangulation (Erzberger and Kelle, 2003) can be applied to mixed methods studies. Each of these studies has been used to illustrate how the metaphor of triangulation can be applied to studies producing: (i) complementary findings, (ii) convergent findings, and (iii) divergent findings. In the following section, we demonstrate how the application of the metaphor can be used as a framework both to develop theory and to facilitate the interpretation of the findings from mixed methods studies and their conclusions in each of these scenarios, and how using the metaphor can help to promote greater clarity of the study's purpose, its theoretical propositions, and the links between data sets.

3.2.1. Triangulating complementary results

To exemplify the use of the methodological metaphor of triangulation (Erzberger and Kelle, 2003) for drawing inferences from complementary results, we have drawn on the results of a UK based study by Shipman et al. (2008) (Fig. 2 ). In the UK, members of district nursing teams (DNs) provide most nursing care to people at home in the last year of life. Following concerns that inadequate education might limit the confidence of some DNs to support patients and their carers’ at home, and that low home death rates may in part be related to this, the Department of Health (DH) identified good examples of palliative care educational initiatives for DNs and invested in a 3-year national education and support programme in the principles and practice of palliative care. Shipman et al.’s study evaluates whether the programme had measurable effects on DN knowledge and confidence in competency in the principles and practice of palliative care. The study had two parts, a summative (concerned with outcomes) quantitative component which included ‘before and after’ postal questionnaires which measured effects on DNs’ (n  = 1280) knowledge, confidence and perceived competence in the principles and practice of palliative care and a formative (concerned with process) qualitative component which included semi-structured focus groups and interviews with a sub-sample of DNs (n  = 39).

Fig. 2.

Illustrating the use of triangulation (Erzberger and Kelle, 2003) on complementary results in the study by Shipman et al. (2008).

While their theoretical proposition may not be explicitly stated by the authors, there is clearly an implicit theoretical proposition that the educational intervention would improve DNs knowledge and confidence (theoretical proposition 1, Fig. 2). This was supported by the quantitative findings which showed significant improvement in the district nurses confidence in their professional competence post intervention. Qualitative results supported and complemented the quantitative findings as the district nurses described several benefits from the program including greater confidence in tackling complex problems and better communication with patient and carers’ because of greater understanding of the reasons for symptoms. Thus, a complementary theoretical proposition (theoretical proposition 2, Fig. 2) can be deduced from the qualitative findings: the DN's better understanding of factors contributing to complex problems and underlying reasons for symptoms led to improved confidence in competence raised from district nurses increased understanding.

Fig. 2 illustrates the theoretical propositions, the empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 is supported by the quantitative findings. From qualitative findings, a complementary theoretical proposition (theoretical proposition 2) can be stated explaining the process that led to the DNs improved confidence in competence.

3.2.2. Triangulating convergent results

To illustrate how the methodological metaphor of triangulation can be used to draw inferences from convergent findings, we have drawn on the example of a Danish study by Midtgaard et al. (2006) (Fig. 3 ). This study was conducted to explore experiences of group cohesion and changes in quality of life (QoL) among people (n  = 55) who participated in a weekly physical exercise intervention (for six weeks) during treatment for cancer. The study, conducted in a Danish hospital, involved the use of structured QoL questionnaires, administered at baseline and post intervention (at six weeks) to determine changes in QoL and health status, and qualitative focus groups, conducted post intervention (at six weeks), to explore aspects of cohesion within the group. With regards to the theoretical proposition of the study (Fig. 3), group cohesion was seen as essential to understand the processes within the group that facilitated the achievement of desired outcomes and the satisfaction of affective needs as well as promoting a sense of belonging to the group itself.

Fig. 3.

Illustrating the use of triangulation (Erzberger and Kelle, 2003) on convergent results in the study by Midtgaard et al. (2006).

This proposition was deductively tested in an intervention where patients exercised in mixed gender groups of seven to nine members during a nine hour weekly session over a six week period and was supported by both the empirical quantitative and qualitative findings. The quantitative data showed significant improvements in peoples’ emotional functioning, social functioning and mental health. The qualitative data showed how the group setting motivated the individuals to pursue personal endeavors beyond physical limitations, that patients used each others as role models during ‘down periods’ and how exercising in a group made individuals feel a sense of obligation to train and to do their best. This subsequently helped to improve their social functioning which in turn satisfied their affective needs, improving their improved emotional functioning and mental health.

Fig. 3 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Both the quantitative and qualitative findings, demonstrating improvements in participants’ emotional and social functioning and their mental health, can be attributed to the nature of group cohesion within the programme as expected.

3.2.3. Triangulating divergent results

Qualitative and quantitative results that seem to contradict each other are often explained as resulting from methodological error. However, instead of a methodological flaw, a divergent result could be a consequence of the inadequacy of the theoretical concepts used. This may indicate the need for changing or developing the theoretical concepts involved (Erzberger and Kelle, 2003). The following example of using the methodological metaphor of triangulation (Erzberger and Kelle, 2003) for drawing inferences from divergent results is intended as an example rather than an attempt to change the theoretical concept involved. In a study by Skilbeck et al. (2005) (Fig. 4 ), some results were found to be divergent which was explained as resulting from the use of inadequate questionnaires. We do not wish to critique their conclusion; rather we intend to simply offer an alternative interpretation for their findings.

Fig. 4.

Illustrating the use of triangulation (Erzberger and Kelle, 2003) on divergent results using the study by Skilbeck et al. (2005).

The study aimed to explore family carers’ expectations and experiences of respite services provided by one independent hospice in North England. This hospice provides inpatient respite beds specifically for planned respite admission for a two-week period. Referrals were predominated from general practitioners and patients and their carers were offered respite care twice a year, during the study this was reduced to once a year for each patient. Data was collected prior to respite admission and post respite care by semi-structured interviews and using the Relative Stress Scale inventory (RSSI), a validated scale to measure relative distress in relation to caring. Twenty-five carers were included but pre- and post-data were completed by 12 carers. Qualitative data was analysed by using a process of constant comparison and quantitative data by descriptive and comparative statistical analysis.

No clear theoretical proposition was stated by the authors, but from the definition of respite care it is possible to deduce that ‘respite care is expected to provide relief from care-giving to the primary care provider’ (theoretical proposition 1, Fig. 4). This proposition was tested quantitatively by pre- and post-test using the RSSI showing that the majority of carers experienced either a negative or no change in scores following the respite stay (no test of significance was stated). Accordingly, the theoretical proposition was not supported by the quantitative empirical data. The qualitative empirical results, however, were supportive in showing that most of the carers considered respite care to be important as it enabled them to have a break and a rest from ongoing care-responsibilities. From this divergent empirical data it could be suggested changing or developing the original theoretical proposition. It seems that respite care gave the carers relief from their care-responsibilities but not from the distress carers experienced in relation to caring (measured by the used scale). We would therefore suggest that in order to lessen distress related to caring, other types of support is also needed which would change the theoretical proposition as suggested (theoretical proposition 2).

Fig. 4 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the logical relationships between these. Theoretical proposition 1 was not supported by the quantitative findings (indicated in Fig. 4 by the broken arrow), but the qualitative findings supported this proposition. From these divergent empirical findings, the theoretical proposition could accordingly be changed and developed. Respite care seemed to provide relief from carers’ on-going care-responsibilities, but other types of support need to be added to provide relief from distress experienced (theoretical proposition 2).

3.2.4. Triangulation to produce theoretical propositions

Methodological triangulation has also been applied to illustrate how theoretical propositions can be produced by drawing on the findings from a Finnish study by Lukkarinen (2005) (Fig. 5 ). The purpose of this longitudinal study was to describe, explain and understand the subjective health related quality of life (QoL) and life course of people with coronary artery disease (CAD). A longitudinal quantitative study was undertaken during the year post treatment and 19 individuals also attended thematic interviews one year after treatment. This study is one of the few studies that clearly defines theoretical underpinnings for both the selected methods and their purpose, namely “to obtain quantitatively abundant average information about the QoL of CAD patients and the changes in it as well as the patients’ individual, unique experiences of their respective life situations” (Lukkarinen, 2005:622).

Fig. 5.

Illustrating the use of triangulation (Erzberger and Kelle, 2003) to develop theory from the study by Lukkarinen (2005).

The results of the quantitative analysis showed that the male and female CAD patients in the youngest age group had the poorest QoL. While patients’ QoL improved in the dimensions of pain, energy and mobility it deteriorated on dimensions of social isolation, sleep and emotional reactions. From the viewpoint of methodological triangulation used in the study the aim of the quantitative approach was to observe changes in QoL at the group level and also explore correlations of background factors to QoL. The qualitative approach generated information concerning both QoL in the individuals’ life situation and life course and the individuals’ rehabilitation. Both the quantitative and the qualitative analysis showed the youngest CAD patients to have the poorest psychosocial QoL. The results obtained using qualitative methods explained the quantitative findings and offered new insight into the factors related to poor psychosocial QoL, which could be used to help develop theoretical propositions around these. Patients at risk of poorer QoL were those with an acute onset of illness at a young age that led to an unexpected termination of career, resulting in financial problems, and worries about family. This group also experienced lack of emotional support (especially the females with CAD) and were concerned for the illness that was not alleviated by treatment. The interviews and the method of phenomenological psychology therefore helped to gain insight into the participants’ situational experience of QoL and life course, not detectable by the use of a questionnaire.

Fig. 5 illustrates the theoretical propositions, empirical findings from qualitative and quantitative data and the relationships between these. The use of the mixed methods approach enabled a clearer understanding to emerge in relation to the lived experience of CAD patients and the factors that were related to poor QoL. This understanding allows new theoretical propositions about these issues to be developed and further explored, as depicted at the theoretical level.

4. Discussion

As the need for, and use of, mixed methods research continues to grow, the issue of quality within mixed methods studies is becoming increasingly important (O’Cathain et al., 2008, O’Cathain et al., 2007). Similarly, the need for guidance on the analysis and integration of qualitative and quantitative data is a prominant issue (Bazeley, 2009). This paper firstly intended to review the types of analytic approaches (parallel, concurrent or sequential data analysis) that have been used in mixed methods studies within healthcare research. As identified in previous research (O’Cathain et al., 2008), we found that the majority of studies included in our review employed parallel data analysis in which the different analyses are not compared or consolidated until the full analysis of both data sets have been completed. A trend to conduct separate analysis on quantitative and qualitative data is apparent in mixed methods healthcare studies, despite the fact that if the data were correlated, a more complete picture of a particular phenomenon may be produced (Onwuegbuzie and Teddlie, 2003). If qualitative and quantitative data are not integrated during data collection or analysis, the findings may be integrated at the stage of interpretation and conclusion.

Although little pragmatic guidance exists within the wider literature, Erzberger and Kelle (2003) have published some practical advice, on the integration of mixed methods findings. For mixed methodologists, the ‘triangulation metaphor’ offers a framework to facilitate a description of the relationships between data sets and theoretical concepts and can also assist in the integration of qualitative and quantitative data (Erzberger and Kelle, 2003). Yet despite the fact that the framework was published in 2003 within Tashakkori and Teddlie's (2003) seminal work, the Handbook for Mixed Methods in Social and Behavioural Research, our search revealed that it has received little application within the published body of work around mixed methods studies since its publication. This is surprising since mixed methodologists are acutely aware of the lack of guidance with regards to the pragmatics and practicalities of conducting mixed methods research (Bryman, 2006, Leech et al., 2010). Furthermore, there have been frequent calls to move the field of mixed methods away from the “should we or shouldn’t we” debate towards the practical application, analysis and integration of mixed methods and its’ findings and what we can learn from each other's work and advice. Consequently, we have a state of ambiguity and instability in the field of mixed methods in which mixed methodologists find themselves lacking appropriate sources or evidence to draw upon with which to facilitate the future design, conduct and interpretation of mixed methods studies. It is for these reasons that we, in this paper, also intended to identify and select studies that could be used as examples for the application of Erzberger and Kelle's (2003) triangulation metaphor.

When reviewing the studies it was clear that the majority of theoretical assumptions were implicit, rather than explicitly stated by authors. Wu and Volker (2009) previously acknowledged that while studies undoubtedly have a theoretical basis in their literature reviews and the nature of their research questions, they often fail to clearly articulate a particular theoretical framework. This is unfortunate as theory can help researchers to clarify their ideas and also help data collection, analysis and to improve the study's rigour (Wu and Volker, 2009). When using triangulation as a methodological metaphor (Erzberger and Kelle, 2003), researchers are encouraged to articulate their theoretical propositions and to validate their conclusions in relation to the chosen theories. Theory can also guide researchers when defining outcome measures. Should the findings not support the chosen theory, as shown in our examples on complementary and divergent results, researchers can modify or expand their theory accordingly and new theory may be developed (Wu and Volker, 2009). It is therefore our belief that using triangulation as a methodological metaphor in mixed methods research can also benefit the design of mixed method studies.

Like other researchers (O’Cathain et al., 2008), we have also found that most of the papers reviewed lacked clarity in whether the reported results primarily stemmed from qualitative or quantitative findings. Many of the papers were even less clear when discussing their results and the basis of their conclusions. The reporting of mixed methods studies is notoriously challenging, but clarity and transparency are, at the very least, crucial in such reports (O’Cathain, 2009). Using triangulation as a methodological metaphor (Erzberger and Kelle, 2003) may be one way of addressing this lack of clarity by explicitly showing the types of data that researchers have based their interpretations on. It may even help address some of the issues raised in the debate on the feasibility of integrating research methods and results stemming from different epistemological and ontological assumptions and paradigms (Morgan, 2007, Sale et al., 2002). In order to carry out methodological triangulation researchers also need to identify and observe the consistency and adequacy of the two methods, positivistic and phenomenological regarding the research questions, data collection, methods of analysis and conclusions.

While we used systematic principles in our search for mixed methods studies in healthcare research, we cannot claim to have included all such studies. In many cases, reports of mixed methods studies are subjected to ‘salami slicing’ by researchers and hence the conduct of, and findings from, individual approaches are addressed in separate papers. Since these papers are often not indexed as a ‘mixed method’ study, they are undoubtedly more difficult to identify. Furthermore, different terminologies are used to describe and index mixed methods studies within the electronic databases (Halcomb and Andrew, 2009a), making it challenging to be certain that all relevant studies were captured in this review. However, the studies included in this review should give a sufficient overview of the use of mixed analysis in healthcare research and most importantly, they enable us to make suggestions for the future design, conduct, interpretation and reporting of mixed methods studies. It is also important to emphasise that we have based our triangulation examples on the data published but have no further knowledge of the analysis and findings undertaken by the authors. The examples should thus be taken as examples and not alternative explanations or interpretations.

Mixed methods research within healthcare remains an emerging field and its use is subject to much debate. It is therefore particularly important that researchers clearly describe their use of the approach and the conclusions made to improve transparency and quality within mixed methods research. The use of triangulation as a methodological metaphor (Erzberger and Kelle, 2003) can help researchers not only to present their theoretical propositions but also the origin of their results in an explicit way and to understand the links between theory, epistemology and methodology in relation to their topic area. Furthermore it has the potential to make valid inferences, challenge existing theoretical assumptions and to develop or create new ones.

Conflict of interest

None declared.

Funding

None.

Ethical approval

Not required.