Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2010 Jan 22.
Published in final edited form as: Res Sch. 2006 Spring;13(1):29.

Defining and Designing Mixed Research Synthesis Studies

Margarete Sandelowski 1, Corrine I Voils 2, Julie Barroso 3
PMCID: PMC2809982  NIHMSID: NIHMS151622  PMID: 20098638

Abstract

Mixed research synthesis is the latest addition to the repertoires of mixed methods research and systematic review. Mixed research synthesis requires that the problems generated by the methodological diversity within and between qualitative and quantitative studies be resolved. Three basic research designs accommodate this diversity, including the segregated, integrated, and contingent designs. Much work remains to be done before mixed research synthesis can secure its place in the repertoires of mixed methods research and systematic review, but the effort is well worth it as it has the potential to enhance both the significance and utility for practice of the many qualitative and quantitative studies constituting shared domains of research.

Mixed research synthesis is the latest addition to the repertoires of mixed methods research and systematic review. Mixed research synthesis is our name for the type of systematic review aimed at the integration of results from both qualitative and quantitative studies in a shared domain of empirical research. In contrast to mixed methods research in which the data set subject to analysis and interpretation is composed of the qualitative and quantitative data (e.g., from interviews, observations, questionnaires, physiologic measures, and the like) obtained directly from research participants within a single study or program of research, the data in mixed research synthesis studies are the findings of primary qualitative and quantitative studies in a designated body of empirical research. The focus of mixed research synthesis studies is on researchers integrations of their data, or the results they report; the products of mixed research synthesis studies are other researchers’ (i.e., reviewers of research) integrations of those results to “sum up” what is known about a target phenomenon and, thereby, to direct both practice and future research.

In this article, we offer an overview of the impetus for mixed research synthesis and the challenges it presents, and propose three basic research designs to conduct mixed research synthesis studies. We have drawn from the large body of literature in the general areas of systematic review, research synthesis, and mixed methods research; the small body of literature on mixed research synthesis; and from our own ongoing study aimed at developing methods to synthesize qualitative and quantitative research findings.1

The Impetus for Mixed Research Synthesis

The new interest in mixed research synthesis (Dixon-Woods, Agarwal, Young, Jones, & Sutton, 2004; Forbes & Griffiths, 2002; Harden & Thomas, 2005; Hawker, Payne, Kerr, Hardey, & Powell, 2002; Lemmer, Grellier, & Steven, 1999; Mays, Pope, & Popay, 2005; Popay & Roen, 2003) is the result of the convergence of two “growth industries” (Estabrooks, 1999, p. 274) discussed in the following sections, namely, evidence-based practice and qualitative research.

The Turn to Evidence-Based Practice

Over the last two decades, scholars in the practice disciplines have increasingly turned to evidence-based practice to facilitate better use of research findings and to close the research-practice gap. Appearing in various guises across the disciplines as evidence-based health care, medicine, nursing, education, social work, and librarianship (Trinder & Reynolds, 2000), evidence-based practice is generally defined as the conscientious, explicit, and judicious use of information to serve as the foundation for practice (Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996; Sackett, Straus, Richardson, Rosenberg, & Haynes, 2000). At the heart of evidence-based practice are evidence syntheses, or integrations of research findings derived from systematic reviews of empirical research in targeted research areas to answer specific research questions addressing specific practice problems. Such evidence syntheses are viewed as having the potential to increase the utility of research and the effectiveness of practice as they enable answers to such questions as which treatments of a disease produce the best health outcomes, or which teaching strategies produce the best learning outcomes.2

The Rise of Qualitative Research

Concurrent with the turn to evidence-based practice has been the growth of qualitative research. Over the last 30 years, the number of qualitative studies and of instructional texts on qualitative methods has increased exponentially across the behavioral, social science, and practice disciplines. The dramatic proliferation of qualitative studies and rising concern about their under-utilization, occurring against the backdrop of renewed interest in enhancing the utility of research through systematic reviews of research, sparked the interest in conducting syntheses of qualitative research (Sandelowski, 2004), Most commonly referred to as qualitative metasynthesis or meta-ethnography, qualitative research synthesis studies were promoted to fulfill the promise of qualitative research findings to effect desired changes in health, education, and social welfare. A spate of articles and books has appeared since the late 1980s addressing qualitative research synthesis methods (e.g., Noblit & Hare, 1988; Paterson, Thorne, Canam, & Jillings, 2001; Sandelowski & Barroso, in press), reporting the results of qualitative research synthesis studies (e.g., Campbell et al., 2003; Pound et al., 2005; Sandelowski & Barroso, 2003, 2005), and calling for the inclusion of qualitative research into evidence-based practice (Barbour, 2000; Dixon-Woods, Fitzpatrick, & Roberts, 2001; Giacomini, 2001; Green & Britten, 1998; Greenhalgh, 2002; Leys, 2003; Popay & Williams, 1998).

The Challenges of Mixed Research Synthesis

Advances in qualitative and quantitative research synthesis and the increasing prominence of mixed methods research as the “third research paradigm” (Johnson & Onwuegbuzie, 2004, p. 14) for the practice disciplines have contributed to the current interest in mixed research synthesis. But for mixed research synthesis to advance, researchers must solve the problems generated by the methodological diversity within and between qualitative and quantitative studies. Difference has recurrently been identified as the most important factor complicating both the qualitative and quantitative research synthesis enterprises (Cooper, 1998; Mulrow, Langhorne, Grimshaw, 1997; Sandelowski, Docherty, & Emden, 1997; Sandelowski, Voils, & Barroso, 2006). Even studies of ostensibly the same variables or target events, experiences, or phenomena in ostensibly similar groups of people employing ostensibly the same methodological approaches will have differences sufficient to require finding the means to enable their findings to be compared and combined.

Qualitative Research Synthesis

In the qualitative research synthesis literature, the difference problem is most often addressed in relation to the philosophical differences among research traditions and the singularity of every research participant and research encounter (Sandelowski et al., 1997). The qualitative research methods literature is characterized, in part, by efforts to differentiate among ontological positions (e.g., realist, idealist, and relativist); epistemological positions (e.g., objectivist, constructionist, subjectivist); paradigms of inquiry (e.g., neo-positivism, interpretivism, critical theory, postmodernism); foundational theories and philosophies (e.g., symbolic interactionism, Heideggerian phenomenology, Foucaultian geneology); and methodologies (e.g., grounded theory, phenomenology, ethnography, narrative/discourse study; Crotty; 1998; Guba & Lincoln, 2005).3 These differences can make the synthesis of qualitative findings alone a daunting enterprise. In addition, the qualitative research emphasis on delineating the complexities and contradictions of “N=1 experiences” (Eisner, 1991, p. 197) seems at odds with the conventional research synthesis emphasis on simplification and summary. Moreover, the diversity in the implementation and reporting of qualitative research complicates both the identification of the methods actually used in a study and the findings produced from those methods (Sandelowski & Barroso, in press).

Accordingly, qualitative researchers have urged the development of synthesis methods distinctive to qualitative inquiry and warned against reliance on quantitative research synthesis as the model for qualitative research synthesis (Barbour & Barbour, 2003; Jones, 2004; Sandelowski et al., 1997). Although much progress has been made in the development of these methods, debates continue over such issues as: (a) terminology (e.g., qualitative meta-analysis [or metaanalysis], meta-synthesis [or metasynthesis], meta-data-analysis); (b) whether to retrieve all of the research reports in a targeted domain or only a purposeful sample of them; (c) whether and how to use quality criteria for evaluating qualitative studies; (d) what the goals of qualitative research synthesis should be (e.g., topical review, aggregation, integration, interpretive comparison, critique); and (d) whether to advance one or multiple interpretations of studies (Booth, 2001; Doyle, 2003; Evans & Pearson, 2001; Jones, 2004; Thorne, Jensen, Kearney, Noblit, & Sandelowski, 2004). Because of the “multidisciplinary pedigree” (Barbour & Barbour, 2003, p. 183) of qualitative research, the resolution of these debates will vary with individual research practitioners’ understanding of the imperatives of qualitative research as applied to their own disciplines.

Quantitative Research Synthesis

In the quantitative research literature, the problem of difference is referred to as heterogeneity and is addressed as the methodological diversity within and between observational and experimental studies, and in terms of the contrast between the “real” differences in the target phenomena and “artifactual” differences owing to the way these phenomena were studied (Glasziou & Sanders, 2002). The constellation of statistical techniques known as meta-analysis, which is most identified with quantitative research synthesis and intended to accommodate these differences, continues to engender debate. One continuing criticism is that, because no two studies can ever be perfectly identical, meta-analysis are comparing apples and oranges (Glass, 2000; Hunt, 1997), thereby, calling into question the validity and generalizability of meta-analyses (Matt, 2003). Another criticism is that meta-analysis involves testing hypotheses about parameters in populations of studies, yet the criteria for inferential statistics (e.g., random sampling) are rarely met. Yet other concerns include the management of variable primary study quality (Conn & Rantz, 2003) and of results from diverse research designs (e.g., combining findings from single group pre-post designs and randomized controlled trials, from cross-sectional and longitudinal studies, or from experimental and observational studies). The Cochrane Collaboration and Library, an icon of the evidence-based practice movement in health care, includes only randomized controlled trials in their research syntheses, but most studies conducted that could serve as the basis for clinical practice are not randomized controlled trials.

Although many strategies have been proposed to address these issues, no consensus exists on any one set of strategies. For example, a variety of methods exist to address the heterogeneity among studies (Higgins & Green, 2005). In recent years, meta-analysis has moved away from approaches directed toward common estimates and fixed-effects models and toward approaches estimating the extent and sources of heterogeneity among studies and random-effects models (Stangl & Berry, 2000). In addition, a variety of strategies exist to test for bias (e.g., publication bias; Schulze, Holling, & Bohning, 2003; Stangl & Berry, 2000). In the end, the desire for greater objectivity in reviewing and drawing conclusions from studies that remains the impetus for quantitative research synthesis has been curbed by the need to make highly subjective and idiosyncratic judgments that fit the nature of the primary study findings to be integrated. Despite the drive for greater objectivity evident in the growing popularity of meta-analysis, objectivity continues to reside in the defense and documentation of largely subjective judgments.

Mixed Research Synthesis

Mixed research synthesis complicates the difference problem as qualitative and quantitative research are themselves viewed as exemplifying difference. Mixed research synthesis entails the “mixing” of the differences characterizing efforts to integrate qualitative research findings with the differences characterizing efforts to integrate quantitative research findings (Sandelowski, Voils, & Barroso, 2006). Scholars have debated whether these differences preclude mixed research synthesis.

For “purists” (Johnson & Onwuegbuzie, 2004, p. 14) who view qualitative and quantitative research as two wholly different species of inquiry, the chasm between qualitative and quantitative modes of inquiry is deep enough to make it difficult or even impossible to cross it without endangering the imperatives and integrity of one or both domains of inquiry. Hammersley (2001, p. 544) cautioned against using a “positivist model” of systematic review, appropriate when using quantitative meta-analysis, but not appropriate for synthesizing qualitative findings. Drawing from Freese’s (1980) work on cumulative knowledge, she suggested that meta-analysis derives from an “additive” orientation to knowledge development, while qualitative research synthesis may require a largely “multiplicative” one (Hammersley, 2001, p. 548). Only recently have qualitative research results been considered worthy enough even to be considered for inclusion in evidence-based practice as both good research and evidence have been defined in ways that favor highly controlled quantitative studies and, thereby, automatically exclude qualitative studies (Hampton, 2002; Madjar & Walton, 2001; McKenna, Cutliffe, & McKenna, 1999; Miller & Fredericks, 2003; Mitchell, 1999; Ray & Mayan, 2001; Upshur, 2001), In the hierarchy of evidence in which the randomized clinical trial is the gold standard against which all studies are evaluated as good or bad research and as yielding strong or weak evidence, qualitative research will inevitably be ranked at the bottom (Evans, 2003; Petticrew & Roberts, 2003).

Proponents of qualitative research have expressed concern about the cooptation of qualitative research that occurs when its distinctive imperatives appear to be celebrated when they are actually only tolerated (Sandelowski & Barroso, in press). Rolfe (2002) described the veneer of acceptance of diversity that often masks efforts toward conformity, or the erasure of difference. Evidence of cooptation may be seen in the prevailing solutions proposed in the mixed research synthesis literature to the problems of “mixing” qualitative and quantitative research results. These solutions include primarily the one-way assimilation of qualitative data into quantitative data, or the use of qualitative data largely as an adjunct to quantitative research synthesis. Few solutions involving the assimilation of quantitative data into qualitative data have been proposed probably because any “narrative” or “qualitative” solution is viewed as reverting to the type of subjective review that the more objective quantitative meta-analysis was supposed to replace. Noteworthy here is the recurring association of the words narrative and qualitative with reviews considered unscientific and unworthy in the hierarchy of evidence in which randomized clinical trials and quantitative meta-analyses are placed on the top rungs. Conceived as useful for such purposes as clarifying the objectives of largely quantitative research reviews, delineating criteria for inclusion of studies, identifying key variables for analysis, and explaining and appraising the practical significance of reviews (Dixon-Woods, Fitzpatrick, & Roberts, 2001), qualitative research is rarely viewed as having any utility in the research synthesis enterprise outside an accessory role.

The mere toleration (as opposed to real acceptance) of qualitative research as an equal partner with quantitative research in systematic review is also apparent in the argument that because all clinical or practice problems have not been, or cannot be, addressed with the gold standard randomized clinical trial, room has to be made (albeit reluctantly) for research that is less controlled and, therefore, more biased. If this were not true, there would be no need to include these “weaker” forms of evidence. Although tolerators of qualitative research may concede that appropriately conducted study designs other than the randomized controlled trial may be stronger sources of evidence than clinical trials inappropriately conducted (i.e., that they offer the “best evidence” available [Slavin, 1995]), the randomized controlled trial remains for many the gold standard against which all modes of inquiry are judged. Accordingly, for tolerators, qualitative research enters evidence-based practice by default, not by design.

Yet for proponents of qualitative research, it enters evidence-based practice, not only by design, but also out of necessity. Proponents of qualitative research view it as essential to achieving the goals of evidence-based practice because of its distinctive capacity for reaching facets of human experience unreachable with quantitative methods, and because of its central role in the development and testing of culturally-sensitive instruments and participant-centered interventions, and in enhancing the practical significance, and even salvaging, of quantitative research findings (e.g., Onwuegbuzie & Leech, 2004; Pope & Mays, 1995; Sandelowski, 2004; Weinholtz, Kacer, & Rocklin, 1995).

Designing Mixed Research Synthesis Studies

Researchers’ views of the nature and impact of the differences between qualitative and quantitative research will influence how they design mixed research synthesis studies. Table 1 shows three basic designs for conducting mixed research synthesis studies that are our adaptations of designs used in primary mixed methods research (Creswell, Plano Clark, Gutmann, & Hanson, 2003; Tashakkori & Teddlie, 2003). These three designs—segregated, integrated, and contingent—accommodate different views of the relationship between qualitative and quantitative research findings and different definitions of mixed research synthesis.

Table 1.

Designs for Mixed Research Synthesis Studies

Segregated Design
“Qualitative” research question in domain A “Quantitative” research question in domain A
↓ ↓
Retrieval of qualitative studies Retrieval of quantitative studies
↓ ↓
Qualitative analysis of findings Quantitative analysis of findings
↓ ↓
Qualitative synthesis of findings Quantitative synthesis of findings
↓
Mixed research synthesis (configuration)
Integrated Design
Research question
↓
Retrieval of empirical qualitative, quantitative, or primary mixed methods studies
↓
Mixed methods analysis of findings
↓
Mixed research synthesis (assimilation)
Contingent Design
Qualitative, quantitative, or generic research question 1
↓
Group A studies (only qualitative, only quantitative, or any empirical studies)
↓
Mono-/mixed-methods analysis
↓
Mono-/mixed-methods synthesis → Qualitative, quantitative, or generic research question 2
↓
Group B studies (only qualitative, only quantitative, or any empirical studies)
↓
Mono-/mixed-methods analysis
↓
Mono-/mixed-methods synthesis → Qualitative, quantitative, or generic research question 3
↓
Group C studies (only qualitative, only quantitative, or any empirical studies)
↓
Mono-/mixed-methods analysis
↓
Mono-/mixed-methods synthesis → Mixed research synthesis
Open in a new tab

Segregated Design

The segregated design shown in Table 1 maintains the conventional binary distinction between qualitative and quantitative research. This design is based on the assumptions that: (a) qualitative and quantitative studies are wholly different entities and, therefore, ought to be treated separately; (b) qualitative and quantitative studies can readily be distinguished from each other; (c) the differences between qualitative and quantitative studies warrant separate analyses and syntheses of their findings; (d) syntheses of qualitative findings require methods developed just for synthesizing qualitative findings; and (e) syntheses of quantitative findings require methods developed just for synthesizing quantitative findings. The synthesis of qualitative findings produced from such methods as qualitative metasummary, constant targeted comparison, and reciprocal translation of concepts (Sandelowski & Barroso, in press), are combined with the synthesis of quantitative findings produced from meta-analysis techniques to configure a mixed research synthesis. Only after each set of qualitative and quantitative findings in a common domain of research has been separately synthesized with methods distinctive to it can the separate synthesis products themselves be synthesized (e.g., into a set of conclusions, a theoretical framework, or path analysis). The segregated design is most appropriate when: (a) qualitative and quantitative findings in a designated body of research are viewed as complementing (as opposed to either confirming or refuting) each other and when (b) mixed research synthesis is defined as the configuration (as opposed to the assimilation) of research findings.

Complementarity vs. confirmation/refutation

Confirmation and refutation are processes that rest on the assumption that qualitative and quantitative studies can address the same research purposes or answer the same research questions and, thereby, yield findings about the same aspect of a target phenomenon. Confirmation and refutation are exercises in seeking to establish convergent validation (or triangulation) both within the qualitative and quantitative studies, respectively, and between qualitative and quantitative studies in a shared domain of research. Confirmation occurs when the same finding (e.g., that depression does or does not contribute to antiretroviral non-adherence) is repeated within and across both qualitative and quantitative studies. Refutation occurs when a designated relationship yields divergent findings, or findings in direct opposition. For example, one set of qualitative and/or quantitative studies indicates that depression contributes to non-adherence, while another set of studies addressing the same relationship indicates the opposite conclusion (e.g., that depression has no influence on adherence).4

Whereas confirmation and refutation rest on the assumption that qualitative and quantitative research can address the same research questions and, thereby, yield findings about the same aspects of phenomena, complementarity rests on the assumption that qualitative and quantitative research differ, in part, because they do not address the same questions. Barbour and Barbour (2003, p. 180) observed that qualitative research answered questions different from quantitative research as they engaged “a different sort of curiosity.” (Complicating the difference problem in mixed research synthesis and, therefore, the selection of design is that whereas the research question in quantitative studies is always fixed prior to beginning them, in qualitative studies, the research question that will ultimately be answered is often the product of analysis, or arrived at only after entering the field of study). Because they address different aspects or dimensions of a target phenomenon, qualitative and quantitative research findings can neither confirm nor refute, but rather only complement, each other. Complementarity here rests on the conception of findings as related to each other—in that they are in the same domain of research (e.g., antiretroviral adherence)—but not as addressing the same aspects in that domain.

An example of complementarity is when a set of qualitative studies indicates that caring for children influences antiretroviral adherence in women and a set of quantitative studies indicates that sex influences adherence. The one finding neither confirms nor refutes the other as they are not referring to the same phenomenon and, therefore, are not subject to convergent validation. But the quantitatively-produced finding that sex predicted adherence may be clarified or explained by the qualitatively-produced finding that caring for children influenced adherence because childrearing is a gender-marked obligation, or a responsibility culturally prescribed for women as opposed to men. Similarly, the quantitatively-produced finding that race predicted non-adherence may be clarified or explained by the qualitatively-produced finding that HIV-positive African Americans’ knowledge of the Tuskegee syphilis study was a recurring reason given for their not taking antiretroviral drugs, as they saw these drugs as reprising the experimental and genocidal imperatives of the Tuskegee study.

These examples suggest that qualitative and quantitative research findings are complementary in linking causal explanations to causal observations (Maxwell 2004a, 2004b). Quantitative findings indicate that-knowledge (e.g., that being female and being African American led to lower levels of adherence in comparison to being male or being white), while qualitative findings indicate why-knowledge, or the gender or race performances or relations that might explain these observations.

Configuration vs. assimilation of findings

The segregated design is also the design of choice when mixed research synthesis is conceived as the configuration, as opposed to assimilation, of qualitative and quantitative research findings. The configuration of findings is the arrangement of complementary findings into a line of argument (Noblit & Hare, 1988), a theory that posits relationships among concepts, or a narrative that posits a temporal ordering of events (e.g., Greenhalgh et al., 2005; Pound et al., 2005). Because qualitative and quantitative findings are viewed as addressing different aspects of a target phenomenon (i.e., as in a complementary relationship), they are also viewed as resistant to direct assimilation into each other. Unlike findings across studies seen to address the same relationship or aspect of a phenomenon, findings conceived as complementary cannot be reduced. Instead, they can only be organized into a coherent whole. For example, qualitative findings may be positioned as antecedent, mediating, or moderating variables explaining or delineating the conditions for the occurrence of events depicted in quantitative findings. Alternatively, quantitative findings can be used to make more explicit comparisons between groups only implied in qualitative findings.

Integrated Design

In the integrated design shown in Table 1, the methodological differences between qualitative and quantitative studies are minimized as both kinds of studies are viewed as producing findings that can readily be transformed into each other. This design is based on the assumptions that: (a) any differences between qualitative and quantitative studies that do exist do not warrant separate analyses and syntheses of their findings; (b) studies designated as qualitative or quantitative are not necessarily distinguishable from each other; (c) both qualitative and quantitative studies in a common research domain can address the same research purposes and questions; and (d) syntheses of both qualitative and quantitative findings can be produced from methods developed for qualitative and quantitative findings. The integrated design is most appropriate when: (a) qualitative and quantitative findings in a designated body of research are viewed as able to confirm, extend, or refute each other and when (b) mixed research synthesis is defined as the assimilation (as opposed to configuration) of research findings.

In integrated designs, the studies in a targeted domain are grouped for synthesis not by methods (i.e., qualitative and quantitative), but rather by findings viewed as answering the same research questions, or addressing the same aspects of a target phenomenon. Here findings addressing the same aspects may extend each other, which can be seen as a form of confirmation. An example of extension is when one set of findings indicates that having to take a large number of pills is a reason for not adhering to antiretroviral therapy, while another set of findings specifies the number of pills below which HIV-positive persons generally adhere and above which few persons adhere.

Mixed research synthesis is accomplished by mixed methods analysis. The analytic emphasis is on transforming findings to enable them to be combined. Transformation includes qualitizing, or converting quantitative findings into qualitative form so that they can be combined with other qualitative data and subjected to qualitative analysis, and quantitizing, or converting qualitative findings into quantitative form so that they can be combined with other quantitative data and subjected to quantitative analysis (Onwuegbuzie & Teddlie, 2003). Because few mixed research synthesis efforts exist, and because few reported qualitative or quantitative research syntheses have derived from the use of transformation techniques, instances of qualitizing and quantitizing have appeared largely in reports of primary mixed methods research. Accordingly, examples of quantitizing with which we are experimenting in our on-going methods study that hold promise for mixed research synthesis include the calculation of effect sizes of qualitative findings and the translation of qualitatively-produced themes into predictor variables (Onwuegbuzie, 2003; Sandelowski & Barroso, in press). Examples of qualitizing that we are experimenting with include the conversion of quantitatively-produced correlations to themes, typologies, or case profiles (Onwuegbuzie & Teddlie, 2003; Tashakkori & Teddlie, 1998). Methods proposed to bridge the case-intensive world of qualitative research with the variable-extensive world of quantitative research include Ragin’s (1987, 2000) qualitative comparative case method and Bayesian approaches to meta-analysis (Howard, Maxwell, & Fleming, 2000; Roberts, Dixon-Woods, Fitzpatrick, Abrams, Jones, 2002).

Contingent Design

In the contingent design shown in Table 1, the results of synthesizing the findings in a designated group of studies to answer one research question determine the next group of studies that will be retrieved and analyzed to answer a second research question the results of which, in turn, may lead to the analysis of a third group of studies retrieved to answer yet another research question. The cycle of systematic review continues until a comprehensive research synthesis can be presented that addresses researchers’ objectives. For example, an initial focus on all reports of findings of studies testing the effectiveness of interventions to promote antiretroviral adherence in women might lead to subsequent searches for reports of findings concerning gender differences in antiretroviral adherence, the experience of HIV infection in women, or concerning HIV-related stigma to explain how and/or why these interventions succeeded or failed.

Contingent designs may or may not depend on hard lines drawn between qualitative and quantitative studies and between qualitative and quantitative methods of research synthesis. Contingent designs may be more like segregated designs in posing a series of research questions conceived to be amenable only to qualitative or quantitative studies, each set of which are analyzed with qualitative or quantitative methods, respectively, to produce the qualitative and quantitative research syntheses that will ultimately be configured into a theoretical or narrative rendering of findings. Alternatively, contingent designs may be more like integrated designs in posing a series of research questions deemed answerable by both qualitative and quantitative studies in a targeted domain of research. The findings of these studies can then be assimilated. In short, the defining feature of contingent designs is the cycle of research synthesis studies conducted to answer questions raised by previous syntheses, not the grouping of studies or methods as qualitative and quantitative.

The Future of Mixed Research Synthesis

The viability of the mixed research synthesis enterprise rests on finding ways to make the seemingly incomparable comparable in order to make the seemingly uncombinable combinable (i.e., assimilable, arrangeable, or some other process of putting or using qualitative and quantitative findings together). Given the complexity of these goals, it is not surprising that mixed research synthesis methods have yet to be developed that satisfactorily accommodate the singularity, descriptive precision, and intricacy of qualitative research findings and the generality, numerical precision, and single-dimensionality of quantitative research findings (Buchanan, 1992; Sivesind, 1999). Although much work remains before mixed research synthesis can secure its place in the repertoires of mixed methods research and systematic review, the effort is well worth it as it has the potential to enhance both the significance and utility for practice of the many qualitative and quantitative research studies constituting shared domains of research.

Footnotes

1

This study, entitled Integrating qualitative & quantitative research findings, is funded by the National Institute of Nursing Research, National Institutes of Health, 5 R01 NR004907, June 3, 2005–March 31, 2010. Support was also provided by Career Development Award MRP 04-216-1 awarded to Corrine I. Voils from the Health Services Research and Development Service of the Department of Veterans Affairs. The views expressed in this manuscript are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

2

Both systematic review and evidence-based practice have generated strong and even scathing criticism. See, for example, Clarke, 1999; Estabrooks, 1999; Gupta, 2003; Hampton, 2002; MacLure, 2005; Mykhalovskiy & Weir, 2004; Pellegrino, 2002; Rolfe, 2002; Timmermans & Berg, 2003; Traynor, 2002; & Trinder, 2000.

3

Differences even exist in categorizing these differences; for example, what we refer to here as paradigms others refer to as theories.

4

Having ascertained what appears to be a refutation, or a contradictory view of the same relationship, researchers will likely want to ascertain conditions related to both the target phenomenon and to the nature of the research itself that might explain why depression influenced adherence in one group of studies and had no influence in a second group of studies. That is, they may want to ascertain whether an apparent refutation is an actual refutation. Similarly, apparent confirmations may not be actual confirmations when reviewers probe findings that seem to replicate each other. Confirmation and refutation are processes not so simple as we depict them here and as they are typically presented in the research synthesis literature. We plan to address the complexities of these processes in a future paper.

The lead editors for this article were R. Burke Johnson and Anthony J. Onwuegbuzie.

Contributor Information

Margarete Sandelowski, University of North Carolina at Chapel Hill School of Nursing.

Corrine I. Voils, Durham Veterans Affairs Medical Center, Duke University Medical Center Department of Medicine

Julie Barroso, Duke University School of Nursing.

References

  1. Barbour RS. The role of qualitative research in broadening the “evidence base” for clinical practice. Journal of Evaluation in Clinical Practice. 2000;6:155–163. doi: 10.1046/j.1365-2753.2000.00213.x. [DOI] [PubMed] [Google Scholar]
  2. Barbour RS, Barbour M. Evaluating and synthesizing qualitative research: The need to develop a distinctive approach. Journal of Evaluation in Clinical Practice. 2003;9:179–186. doi: 10.1046/j.1365-2753.2003.00371.x. [DOI] [PubMed] [Google Scholar]
  3. Booth A. Cochrane or cock-eyed? How should we conduct systematic reviews of qualitative research?. Paper presented at the Qualitative Evidence-Based Practice Conference: Taking a critical stance; Coventry University; 2001. May, Retrieved December 24, 2005, from Education-Line at http://www.leeds.ac.uk/educol/documents/00001724.doc. [Google Scholar]
  4. Buchanan DR. An uneasy alliance: Combining qualitative and quantitative research methods. Health Education Quarterly. 1992;19:117–135. doi: 10.1177/109019819201900108. [DOI] [PubMed] [Google Scholar]
  5. Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, et al. Evaluating meta-ethnography: A synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science & Medicine. 2003;56:671–684. doi: 10.1016/s0277-9536(02)00064-3. [DOI] [PubMed] [Google Scholar]
  6. Clarke JB. Evidence-based practice: A retrograde step? The importance of pluralism in evidence generation for the practice of health care. Journal of Clinical Nursing. 1999;8:89–94. doi: 10.1046/j.1365-2702.1999.00222.x. [DOI] [PubMed] [Google Scholar]
  7. Conn VS, Rantz MJ. Research methods: Managing primary study quality in meta-analyses. Research in Nursing & Health. 2003;26:322–333. doi: 10.1002/nur.10092. [DOI] [PubMed] [Google Scholar]
  8. Cooper H. Synthesizing research: A guide for literature reviews. 3. Thousand Oaks, CA: Sage; 1998. [Google Scholar]
  9. Creswell JW, Plano Clark VL, Gutmann ML, Hanson WE. Advanced mixed methods research designs. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage; 2003. pp. 209–240. [Google Scholar]
  10. Crotty M. The foundations of social research: Meaning and perspective in the research process. London: Sage; 1998. [Google Scholar]
  11. Dixon-Woods M, Agarwal S, Young B, Jones D, Sutton A. Integrative approaches to qualitative and quantitative evidence. London: Health Development Agency; 2004. Retrieved November 13, 2005, from http://www.med.umich.edu/csp/Course%20materials/Summer%202005/Goold_Integrative_Approaches.pdf. [Google Scholar]
  12. Dixon-Woods M, Fitzpatrick R, Roberts K. Including qualitative research in systematic reviews: Opportunities and problems. Journal of Evaluation in Clinical Practice. 2001;7:125–133. doi: 10.1046/j.1365-2753.2001.00257.x. [DOI] [PubMed] [Google Scholar]
  13. Doyle LH. Synthesis through meta-ethnography: Paradoxes, enhancements, and possibilities. Qualitative Research. 2003;3:321–345. [Google Scholar]
  14. Eisner EW. The enlightened eye: Qualitative inquiry and the enhancement of educational practice. New York: Macmillan; 1991. [Google Scholar]
  15. Estabrooks CA. Will evidence-based nursing practice make practice perfect? Canadian Journal of Nursing Research. 1999;30:273–294. [PubMed] [Google Scholar]
  16. Evans D. Hierarchy of evidence: A framework for ranking evidence evaluating health care interventions. Journal of Clinical Nursing. 2003;12:77–84. doi: 10.1046/j.1365-2702.2003.00662.x. [DOI] [PubMed] [Google Scholar]
  17. Evans D, Pearson A. Systematic reviews of qualitative research. Clinical Effectiveness in Nursing. 2001;5:111–119. (with commentaries and author response) [Google Scholar]
  18. Forbes A, Griffiths P. Methodological strategies for the identification and synthesis of “evidence” to support decision-making in relation to complex healthcare systems and practices. Nursing Inquiry. 2002;9:141–155. doi: 10.1046/j.1440-1800.2002.00146.x. [DOI] [PubMed] [Google Scholar]
  19. Freese L. The problem of cumulative knowledge. In: Freese L, editor. Theoretical methods in sociology. Pittsburgh, PA: University of Pittsburgh Press; 1980. pp. 13–69. [Google Scholar]
  20. Giacomini MK. The rocky road: Qualitative research as evidence. Evidence-Based Medicine. 2001;6:4–6. [PubMed] [Google Scholar]
  21. Glass GV. Meta-analysis at 25. 2000 Retrieved December 24, 2005, from http://glass.ed.asu.edu/gene/papers/meta25.html.
  22. Glasziou PP, Sanders SL. Investigating causes of heterogeneity in systematic reviews. Statistics in Medicine. 2002;21:1503–1511. doi: 10.1002/sim.1183. [DOI] [PubMed] [Google Scholar]
  23. Green J, Britten N. Qualitative research and evidence based medicine. British Medical Journal. 1998;316:1230–1232. doi: 10.1136/bmj.316.7139.1230. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Greenhalgh T. Integrating qualitative research into evidence based practice. Endocrinology & Metabolism Clinics of North America. 2002;31:583–601. doi: 10.1016/s0889-8529(02)00009-9. [DOI] [PubMed] [Google Scholar]
  25. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R. Storylines of research in diffusion of innovation: A meta-narrative approach to systematic review. Social Science & Medicine. 2005;61:417–430. doi: 10.1016/j.socscimed.2004.12.001. [DOI] [PubMed] [Google Scholar]
  26. Guba EG, Lincoln YS. Paradigmatic controversies, contradictions, and emerging confluences. In: Denzin NK, Lincoln YS, editors. The Sage handbook of qualitative research. 3. Thousand Oaks, CA: Sage; 2005. pp. 191–215. [Google Scholar]
  27. Gupta M. A critical appraisal of evidence-based medicine: Some ethical considerations. Journal of Evaluation in Clinical Practice. 2003;9:111–121. doi: 10.1046/j.1365-2753.2003.00382.x. [DOI] [PubMed] [Google Scholar]
  28. Hammersley M. On “systematic” reviews of research literatures: A “narrative” response to Evans & Benefield. British Educational Research Journal. 2001;27:543–554. [Google Scholar]
  29. Hampton JR. Evidence-based medicine, opinion-based medicine, and real-world medicine. Perspectives in Biology and Medicine. 2002;45:549–568. doi: 10.1353/pbm.2002.0070. [DOI] [PubMed] [Google Scholar]
  30. Harden A, Thomas J. Methodological issues in combining diverse study types in systematic reviews. Internal Journal of Social Research Methodology. 2005;8:257–271. [Google Scholar]
  31. Hawker S, Payne S, Kerr C, Hardey M, Powell J. Appraising the evidence: Reviewing the disparate data systematically. Qualitative Health Research. 2002;12:1284–1299. doi: 10.1177/1049732302238251. [DOI] [PubMed] [Google Scholar]
  32. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions 4.2.5. 2005 May; Retrieved December 24, 2005, from http://www.cochrane.org/resources/handbook/hbook.htm.
  33. Howard GS, Maxwell SE, Fleming KJ. The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis. Psychological Methods. 2000;5:315–332. doi: 10.1037/1082-989x.5.3.315. [DOI] [PubMed] [Google Scholar]
  34. Hunt M. How science takes stock. New York: Russell Sage Foundation; 1997. [Google Scholar]
  35. Johnson RB, Onwuegbuzie AJ. Mixed methods research: A research paradigm whose time has come. Educational Researcher. 2004;33:14–26. [Google Scholar]
  36. Jones K. Mission drift in qualitative research, or moving toward a systematic review of qualitative studies, moving back to a more systematic narrative review. The Qualitative Report. 2004;9(1):95–112. On line journal, retrieved August 29, 2004, from www.nova.edu/ssss/QR/QR9-1/jones.pdf.
  37. Lemmer B, Grellier R, Steven J. Systematic review of nonrandom and qualitative research literature: Exploring and uncovering an evidence base for health visiting and decision making. Qualitative Health Research. 1999;9:315–328. [Google Scholar]
  38. Leys M. Health care policy: Qualitative evidence and health technology assessment. Health Policy. 2003;65:217–226. doi: 10.1016/s0168-8510(02)00209-9. [DOI] [PubMed] [Google Scholar]
  39. MacLure M. “Clarity bordering on stupidity”: Where’s the quality in systematic review? Journal of Education Policy. 2005;20:393–416. [Google Scholar]
  40. Madjar I, Walton JA. What is problematic about evidence? In: Morse JM, Swanson JM, Kuzel AJ, editors. The nature of qualitative evidence. Thousand Oaks, CA: Sage; 2001. pp. 28–45. [Google Scholar]
  41. Matt GE. Will it work in Munster? Meta-analysis and the empirical generalization of causal relationships. In: Schulze R, Holling H, Bohning D, editors. Meta-analysis: New developments and applications in medical and social sciences. Cambridge, MA: Hogrefe & Huber; 2003. pp. 113–139. [Google Scholar]
  42. Maxwell JA. Causal explanation, qualitative research, and scientific inquiry in education. Educational Researcher. 2004a;33:3–11. [Google Scholar]
  43. Maxwell JA. Using qualitative methods for causal explanation. Field Methods. 2004b;16:243–264. [Google Scholar]
  44. Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. Journal of Health Services Research and Policy. 2005;10(Suppl 1):6–20. doi: 10.1258/1355819054308576. [DOI] [PubMed] [Google Scholar]
  45. McKenna H, Cutliffe J, McKenna P. Evidence-based practice: Demolishing some myths. Nursing Standard. 1999;14:39–42. doi: 10.7748/ns2000.01.14.16.39.c2738. [DOI] [PubMed] [Google Scholar]
  46. Miller S, Fredericks M. The nature of “evidence” in qualitative research methods. International Journal of Qualitative Methods. 2003;2(1) Article 4. Retrieved September 22, 2003, from http://www.ualbcrta.ca/~iiqm/backissues/2_1/html/miller.html.
  47. Mitchell GJ. Evidence-based practice: Critique and alternative view. Nursing Science Quarterly. 1999;12:30–35. doi: 10.1177/08943189922106387. [DOI] [PubMed] [Google Scholar]
  48. Mulrow C, Langhorne P, Grimshaw J. Integrating heterogeneous pieces of evidence in systematic reviews. Annals of Internal Medicine. 1997;127:989–995. doi: 10.7326/0003-4819-127-11-199712010-00008. [DOI] [PubMed] [Google Scholar]
  49. Mykhalovskiy E, Weir L. The problem of evidence-based medicine: Directions for social science. Social Science & Medicine. 2004;59:1059–1069. doi: 10.1016/j.socscimed.2003.12.002. [DOI] [PubMed] [Google Scholar]
  50. Noblit GW, Hare RD. Meta-ethnography: Synthesizing qualitative studies. Newbury Park, CA: Sage; 1988. [Google Scholar]
  51. Onwuegbuzie AJ. Effect sizes in qualitative research: A prolegomenon. Quality & Quantity. 2003;37:393–409. [Google Scholar]
  52. Onwuegbuzie AJ, Leech NL. Enhancing the interpretation of “significant” findings: The role of mixed methods research. The Qualitative Report. 2004;9(4):770–792. Retrieved December 1, 2005, from http://www.nova.edu/ssss/QR/QR9-4/onwuegbuzie.pdf.
  53. Onwuegbuzie AJ, Teddlie C. A framework for analyzing data in mixed methods research. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in social and behavioral research. Thousand Oaks, CA: Sage; 2003. pp. 351–383. [Google Scholar]
  54. Paterson BL, Thorne SE, Canam C, Jillings C. Meta-study of qualitative health research: A practical guide to meta-analysis and meta-synthesis. Thousand Oaks, CA: Sage; 2001. [Google Scholar]
  55. Pellegrino ED. Medical evidence and virtue ethics: A commentary on Zarkovich and Upshur. Theoretical Medicine. 2002;23:397–402. doi: 10.1023/a:1021241024613. [DOI] [PubMed] [Google Scholar]
  56. Petticrew M, Roberts H. Evidence, hierarchies, and typologies: Horses for courses. Journal of Epidemiology & Community Health. 2003;57:527–533. doi: 10.1136/jech.57.7.527. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Popay J, Roen K. Synthesis of evidence from research using diverse study designs: A preliminary review of methodological work. Lancaster University; UK: 2003. May 23, Unpublished report for the Social Care Institute of Excellence. [Google Scholar]
  58. Popay J, Williams G. Qualitative research and evidence-based health care. Journal of the Royal Society of Medicine. 1998;91(Suppl 35):32–37. doi: 10.1177/014107689809135s08. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Pope C, Mays N. Reaching the parts other methods cannot reach: An introduction to qualitative methods in health and health services research. British Medical Journal. 1995;311:42–45. doi: 10.1136/bmj.311.6996.42. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Pound P, Britten N, Morgan M, Yardley L, Pope C, Daker-White G, et al. Resisting medicines: A synthesis of qualitative studies of medicine taking. Social Science & Medicine. 2005;6:133–155. doi: 10.1016/j.socscimed.2004.11.063. [DOI] [PubMed] [Google Scholar]
  61. Ragin C. The comparative method: Moving beyond qualitative and quantitative strategies. Berkeley: University of California Press; 1987. [Google Scholar]
  62. Ragin C. Fuzzy-set social science. Chicago: University of Chicago Press; 2000. [Google Scholar]
  63. Ray LD, Mayan M. Who decides what counts as evidence? In: Morse JM, Swanson JM, Kuzel AJ, editors. The nature of qualitative evidence. Thousand Oaks, CA: Sage; 2001. pp. 50–73. [Google Scholar]
  64. Roberts KA, Dixon-Woods M, Fitzpatrick R, Abrams KR, Jones DR. Factors affecting uptake of childhood immunization: A Bayesian synthesis of qualitative and quantitative evidence. The Lancet. 2002;360:1596–1599. doi: 10.1016/S0140-6736(02)11560-1. [DOI] [PubMed] [Google Scholar]
  65. Rolfe G. Faking a difference: Evidence-based nursing and the illusion of diversity. Nurse Education Today. 2002;22:3–12. doi: 10.1054/nedt.2001.0736. [DOI] [PubMed] [Google Scholar]
  66. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: What it is and what it isn’t. British Medical Journal. 1996;312(7023):71–72. doi: 10.1136/bmj.312.7023.71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: How to practice and leach EBM. Edinburgh: Churchill Livingstone; 2000. [Google Scholar]
  68. Sandelowski M. Using qualitative research. Qualitative Health Research. 2004;14:1366–1386. doi: 10.1177/1049732304269672. [DOI] [PubMed] [Google Scholar]
  69. Sandelowski M, Barroso J. Motherhood in the context of maternal HIV infection. Research in Nursing & Health. 2003;26:470–482. doi: 10.1002/nur.10109. [DOI] [PubMed] [Google Scholar]
  70. Sandelowski M, Barroso J. The travesty of choosing after positive prenatal diagnosis. JOGNN: Journal of Obstetric, Gynecologic, and Neonatal Nursing. 2005;34:307–318. doi: 10.1177/0884217505276291. [DOI] [PubMed] [Google Scholar]
  71. Sandelowski M, Barroso J. Handbook for synthesizing qualitative research. New York: Springer; In press. [Google Scholar]
  72. Sandelowski M, Docherty S, Emden C. Qualitative metasynthesis: Issues and techniques. Research in Nursing & Health. 1997;20:365–371. doi: 10.1002/(sici)1098-240x(199708)20:4<365::aid-nur9>3.0.co;2-e. [DOI] [PubMed] [Google Scholar]
  73. Sandelowski M, Voils CI, Barroso J. Comparability work and the management of difference in mixed research synthesis. 2006. Manuscript submitted for publication. [DOI] [PMC free article] [PubMed] [Google Scholar]
  74. Schulze R, Holling H, Bohning D. Meta-analysis: New developments and applications in medical and social sciences. Cambridge, MA: Hogrefe & Huber; 2003. [Google Scholar]
  75. Sivesind KH. Structured, qualitative comparison: Between singularity and single-dimensionality. Quality & Quantity. 1999;33:361–380. [Google Scholar]
  76. Slavin RE. Best evidence synthesis: An intelligent alternative to meta-analysis. Journal of Clinical Epidemiology. 1995;48:9–18. doi: 10.1016/0895-4356(94)00097-a. [DOI] [PubMed] [Google Scholar]
  77. Stangl DK, Berry DA. Meta-analysis in medicine and health policy. New York: Marcel Dekker; 2000. [Google Scholar]
  78. Tashakkori A, Teddlie C. Mixed methodology: Combining qualitative and quantitative approaches. Thousand Oaks, CA: Sage; 1998. [Google Scholar]
  79. Tashakkori A, Teddlie C. The past and future of mixed methods research: From data triangulation to mixed model designs. In: Tashakkori A, Teddlie C, editors. Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage; 2003. pp. 671–701. [Google Scholar]
  80. Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M. Qualitative meta-synthesis: Reflections on methodological orientation and ideological agenda. Qualitative Health Research. 2004;14:1342–1365. doi: 10.1177/1049732304269888. [DOI] [PubMed] [Google Scholar]
  81. Timmermans S, Berg M. The gold standard: The challenge of evidence-based medicine and standardization in health care. Philadelphia, PA: Temple University Press; 2003. [Google Scholar]
  82. Traynor M. The oil crisis, risk and evidence-based practice. Nursing Inquiry. 2002;9:162–169. doi: 10.1046/j.1440-1800.2002.00153.x. [DOI] [PubMed] [Google Scholar]
  83. Trinder L. A critical appraisal of evidence-based practice. In: Trinder L, Reynolds S, editors. Evidence-based practice: A critical appraisal. Oxford, UK: Blackwell Science; 2000. pp. 212–241. [Google Scholar]
  84. Trinder L, Reynolds S. Evidence-based practice; A critical appraisal. Oxford, UK: Blackwell Science; 2000. [Google Scholar]
  85. Upshur RE. The status of qualitative research as evidence. In: Morse JM, Swanson JM, Kuzel AJ, editors. The nature of qualitative evidence. Thousand Oaks, CA: Sage; 2001. pp. 5–26. [Google Scholar]
  86. Weinholtz D, Kacer B, Rocklin T. Salvaging quantitative research with qualitative data. Qualitative Health Research. 1995;5:388–397. [Google Scholar]

RESOURCES