A Strategy to Identify Critical Appraisal Criteria for Primary Mixed-Method Studies

Abstract

The practice of mixed-methods research has increased considerably over the last 10 years. While these studies have been criticized for violating quantitative and qualitative paradigmatic assumptions, the methodological quality of mixed-method studies has not been addressed. The purpose of this paper is to identify criteria to critically appraise the quality of mixed-method studies in the health literature. Criteria for critically appraising quantitative and qualitative studies were generated from a review of the literature. These criteria were organized according to a cross-paradigm framework. We recommend that these criteria be applied to a sample of mixed-method studies which are judged to be exemplary. With the consultation of critical appraisal experts and experienced qualitative, quantitative, and mixed-method researchers, further efforts are required to revise and prioritize the criteria according to importance.

1. Introduction

The practice of mixed-methods research has increased considerably over the last 10 years, as seen in numerous articles, chapters, and books published^{1,2,3,4,5,6,7,8,9}. Journal series have been devoted to this topic as well (see Volume 19 of Health Education Quarterly¹⁰; Number 74 of New Directions for Evaluation¹¹; Volume 34 of Health Services Research¹²). Despite being criticized for violating quantitative and qualitative paradigmatic assumptions, the methodological quality of mixed-method studies has not been examined. While one could argue that the quality of mixed-method studies cannot be assessed until the quantitative-qualitative philosophical debate is resolved, it will be demonstrated that under the assumption that methods are linked to paradigms, it is possible to identify criteria to critically appraise mixed-method studies.

There are difficulties with developing criteria to critically appraise mixed-method studies. The definitions for qualitative and quantitative methods as well as the paradigms linked with these methods vary considerably. The critical appraisal criteria proposed for the methods often overlap depending on the dominant paradigm view taken. Finally, the language of criteria is often vague, requiring the inexperienced reader to make judgement calls¹³. This paper will address the above difficulties. It will attempt to clarify the language applied to critical appraisal and the assumptions of the authors will be presented.

The purpose of this paper is to identify criteria to critically appraise the methods of primary studies in the health literature which employ mixed-methods. A mixed-methods study is defined as one in which quantitative and qualitative methods are combined in a single study. A primary study is defined as one that contains original data on a topic¹⁴. The methods refer to the procedures of the study. Before conducting the literature review, the authors will first outline their assumptions concerning methods linked to paradigms, criteria linked to paradigms, and the quantitative and qualitative method. Second, the concept of critical appraisal as it applies to quantitative and qualitative methods will be reviewed. Third, the theoretical framework upon which the authors rely will be described.

1.1 Assumptions

1.1.1 Methods linked to paradigms

A paradigm is a patterned set of assumptions concerning reality (ontology), knowledge of that reality (epistemology), and the particular ways of knowing that reality (methodology)¹⁵. The assumption of linking methods to a paradigm or paradigms is not a novel idea. It has been argued that different paradigms should be applied to different research questions within the same study and that the paradigms should remain clearly identified and separated from each other¹⁶. In qualitative studies, the methods are most often linked with the paradigms of interpretivism ^16,17,18 and constructivism³². However, other paradigms for qualitative methods have been proposed such as positivism¹⁹, post-positivism ^13,19, postmodernism²⁰, and critical theory²⁰. Historically, quantitative methods have been linked with the paradigm of positivism. In recent years, quantitative methods have been viewed as evolving into the practice of post-positivism^21,22. In relation to critical appraisal, we view positivism and post-positivism as compatible. For the remainder of the text, the term positivism will apply to both positivism and post-positivism.

This paper assumes that quantitative methods are linked with the paradigm of positivism and that qualitative methods are linked with the paradigms of constructivism and interpretivism. This is an initial proposal for paradigms. We realize that critical appraisal criteria may change based on other paradigmatic assumptions.

1.1.2 Criteria linked to paradigms

Under the assumption that methods are linked to paradigms, it follows that the criteria to critically appraise these methods should also be linked to paradigms. The application of separate criteria for each method and paradigm is supported by many researchers^{23,24,25,26,27,28,29,30}. Method-specific criteria provides an alternative to blending methodological assumptions²⁷. Method-specific criteria also implies that each method is valued equally. While it is assumed that criteria are linked to paradigms, it is acknowledged that not all researchers ^17,31 maintain this perspective.

1.1.3 The quantitative method

Under the assumptions that the quantitative methods are based on the paradigm of positivism, the underlying view is that all phenomenon can be reduced to empirical indicators which represent the truth. The ontological position of this paradigm is that there is only one truth, an objective reality that exists independent of human perception. Epistemologically, the investigator and investigated are independent entities. Therefore, an investigator is capable of studying a phenomenon without influencing it or being influenced by it; “inquiry takes place as through a one way mirror” ³².

1.1.4 The qualitative method

Under the assumption that qualitative methods are based on the paradigm of interpretivism and constructivism, multiple realities or multiple truths exist based on one’s construction of reality. Reality is socially constructed³³ and so is constantly changing. On an epistemological level, there is no access to reality independent of our minds, no external referent by which to compare claims of truth³⁴. The investigator and the object of study are interactively linked so that findings are mutually created within the context of the situation which shapes the inquiry^35,36. This suggests that reality has no existence prior to the activity of investigation, and reality ceases to exist when we lose interest in it³⁴. The emphasis of qualitative research is on process and meanings³⁶.

1.2 The Concept of Critical Appraisal for Quantitative and Qualitative Methods

Guidelines for critically appraising quantitative methods exist in many peer-reviewed journals. For example, BMJ has published criteria for the critical appraisal of manuscripts which are submitted to it. More recently, qualitative journals have been publishing their own set of criteria. For example, the Canadian Journal of Public Health now has published guidelines for qualitative research papers. However, it has been argued that the idea of static criteria may violate the assumptions of the paradigms to which qualitative methods belong^37,38,39. In this paper, we posit that there are static elements of qualitative methods which should be subject to criticism.

1.3 Theoretical Framework – Trustworthiness and Rigor proposed by Lincoln and Guba^23,24

Our paper relies on Lincoln and Guba’s^23,24 framework of trustworthiness and rigor. This framework which was selected due to its cross-paradigm appeal, guided the organization of the criteria for critically appraising mixed-method studies generated by the literature review. Although Lincoln and Guba intended the framework to reflect conventional (positivistic) versus naturalistic paradigms, we feel the framework encompasses constructivism and interpretivism under that which is “naturalistic”. (Guba and Lincoln themselves have used the term constructivism to refer to naturalistic inquiry⁴⁰.) The term “trustworthiness”, referred to as “goodness criteria”³⁵, parallels the term “rigor” used in quantitative methods²⁴. Trustworthiness and rigor encompass four goals which apply to the respective methods and paradigms:

1. Truth value – internal validity for quantitative methods versus credibility for qualitative methods;

2. Applicability – external validity for quantitative methods versus transferability or fittingness for qualitative methods;

3. Consistency – reliability for quantitative methods versus dependability for qualitative methods;

4. Neutrality – objectivity for quantitative methods versus confirmability for qualitative methods.

Consistent with these goals, we propose to extend Lincoln and Guba’s theoretical framework to include parallel quantitative and qualitative critical appraisal criteria to which mixed-method studies can be applied.

2. Method

The purpose of the literature review was to generate criteria which have been proposed for critically appraising the methods of quantitative, qualitative, and mixed-method studies. Although the purpose of this review was not to develop a measure of critical appraisal, the initial search for criteria was based on the principles of item generation⁴¹ used in measurement theory. Following the principles of item generation, inclusion and exclusion rules were applied to the criteria considered.

2.1 Search Strategy

Articles judged to be exemplary by the authors were extracted from personal files and the files of colleagues, and were reviewed for MESH headings and keywords. The Subject Headings Indices for Medline, Cinahl, and PsychINFO were searched for terms referring to quantitative methods, qualitative methods, mixed-methods research, and critical appraisal. Textwords (words that appear in the titles or abstracts) were specified where there were no subject headings for quantitative, qualitative, and mixed-method studies. Keywords/textwords were assigned to a particular block; blocks were constructed using the Boolean ‘OR’ operator. Critical appraisal terms were then combined with the blocks for qualitative, quantitative, or mixed-methods using the Boolean ‘AND’ operator. Table 1 outlines the search strategy which included all available years of the selected databases.

Table 1.

Search strategy for critically appraising mixed-method studies

Medline, HealthStar (limit to nonmedline) (MESH)	CINAHL	WebSPIRS (includes PsycINFO, Sociological Abstracts, Social Sciences Index)	ERIC	Current Contents
* Nursing Research * Research Design/st [Standards] * Health Services Research/mt [Methods] * Program Development/mt [Methods] * Program Evaluation/mt [Methods] * Research/mt [Methods] * Research Design/mt [Methods]	* Quality Assessment /mt [Methods] * Quality Assessment/st [Standards] * Research/ev [Evaluation] * Research/st [Standards] * Study Design	critical appraisal standards
AND
quantitative.tw qualitative.tw quantitative qualitative.tw mixed method.tw mixed methods.tw mixed methodologies.tw multimethod.tw. multiple methods.tw	* Qualitative studies quantitative.tw * Multimethod Studies mixed method.tw	quantitative qualitative quantitative qualitative mixed method mixed methods mixed methodologies multimethod multiple methods	quantitative [&] qualitative mixed [&] method multimethod multiple methods mixed methodologies quantitative/qualitative qualitative/quantitative quantitative- qualitative qualitative- quantitative	quantitative methods qualitative methods mixed method multimethod multiple methods

^*limit to focus

The databases Medline (1966–2000), HealthStar (1975–1999), CINAHL (1982–1999), PsycINFO (1967–2000/2001), Sociological Abstracts (1963–2000), Social Sciences Index (1983–1999), ERIC (1966–1999/09), and Current Contents (Jan 3, 1999–Feb 14, 2000) were searched. A “words anywhere” search was conducted in WebSPIRS as there are limited subject headings in the included databases. The terms for the critical appraisal block were limited in WebSPIRS as other terms such as “guidelines”, “criteria”, “assessment”, and “quality” produced a high volume of hits which were not relevant. Articles were limited to those in the English language. To complement the literature search strategy, reference lists of articles were also perused for relevant articles and the journal Qualitative Health Research for the years 1997–2001 was hand-searched. A total of four articles could not be located and therefore were not retrieved.

2.2 Inclusion and Exclusion Criteria

The articles retrieved were reviewed and the criteria for critical appraisal were subjected to inclusion and exclusion criteria. Criteria appearing in a list or scale format were extracted from this format and screened independently for inclusion and exclusion. The criteria had to refer to the methods or methods sections of the research. Finally, the criteria had to be operational in that a reader would be able to review an article and determine whether criteria were met or not. (The authors operationalized the language typically applied in the critical appraisal literature; this document is available upon request). Criteria which required judgement calls on the part of the reader such as “is the purpose of the study one of discovery and description, conceptualization, illustration, or sensitization?”⁴² and “was the design of the study sensible?” ⁴³ were excluded. Other exclusion criteria are noted in Table 2. Where appropriate, examples of the exclusions are given.

Table 2.

Exclusion Criteria

criteria which do not fit the respective paradigm assumptions as stated – in cases where assumptions were not stated, judgement calls were made based on the criteria themselves. For example, Morse44 suggests selecting a large enough random sample in qualitative research to compensate for the exclusion of those who are not good informants. Under the assumption of constructivism and interpretivism, random sampling is not employed. criteria which are disease-specific criteria specific to design or tradition within each method and not applicable to other designs or traditions within that method – for example, criteria specific to randomized controlled trials (RCTs), case studies, or ethnographies criteria specific to review articles criteria specific to statistical tests criteria concerning the credibility of the researcher criteria specific to the quality and reporting of the literature review criteria for the elements of a proposal criteria specific to the following topics in medicine: etiology, improving clinical performance, diagnosis, prognosis, therapy, causation, economics of health care programs or interventions, practice guidelines criteria specific to the importance, relevance, or significance of the research criteria specific to measurement scales

3. Results

As we suspected, no criteria for critically appraising mixed-method studies were found. Table 3 is a review of the criteria for critically appraising quantitative and qualitative methods separately. Because this paper assumes that methods and criteria are linked to paradigms, the criteria appear separately. As we have proposed, the organization of the table is based on the framework of trustworthiness and rigor proposed by Lincoln and Guba^23,24. The criteria have been grouped accordingly but it is acknowledged that there may be some overlap of criteria across the four goals.

Table 3.

Review of Critical Appraisal Criteria for Quantitative and Qualitative Methods

Goals of Criteria	Qualitative Methods	Quantitative Methods
Truth value (Credibility vs. Internal Validity)	triangulation of sources 23,45,46,47 triangulation of methods23,45,47,48 triangulation of investigators23,45,47 triangulation of theory/perspective45,47 peer debriefing 23,49 negative case analysis or searching for disconfirming evidence 13,23,45,47,50 member checks23,44, 48,49,50 use of quotations 18 informed consent stated 24,25,48 ethical review or human subject review undertaken44,48 statement that confidentiality protected26,48 consent procedures described 26,44	extraneous51 or confounding variables52 identified extraneous or confounding variable(s) or baseline differences controlled for in the analysis43,51,52,53,54 statement about comparability of control group to intervention group at baseline55,56 statement that comparison group treated equally to aside from intervention 55 informed consent stated 48,57 ethical review undertaken48,57 statement that confidentiality protected 48,57
Applicability (Transferability/Fittingness vs. External Validity/Generalizability)	statement of purpose,25,42,48 statement of research question(s) 13,26,28,42,44,49,50,66 phenomenon of study stated25,58 rationale for the use of qualitative methods 13 rationale for the tradition within qualitative methods 26 description of study context 26,50 or setting 23 statement of how setting was selected 66 sampling procedure described 25,59,66 justification or rationale for sampling strategy 49,50 description of participants or informants25,26,48 data gathering procedures described 23,26,48 audiotaping procedures described 50 transcription procedures described 50 field note procedures described 50 data analysis described23,26 coding techniques described 23,25,28,50 data collection to saturation specified 28 statement that reflexive journals or logbooks kept49,60 description of raw data 23	statement of purpose 48 objective of study explicitly stated or described 52,61 description of intervention if appropriate 51 outcome measure(s) defined 43,53,54 assessment of outcome blinded 43,53 description of setting or conditions under which data collected 51,57,67 design stated explicitly i.e. case study, cross-sectional study, cohort study, RCT 51,52 subject recruitment or sampling selection described43,54,67 sample randomly selected62,63 inclusion and exclusion criteria for subject selection stated explicitly43,51,52,64 study population defined or described 43,51,54,64 source of subjects stated i.e. sampling frame identified28,61 source of controls stated 52 selection of controls described65 control or comparison group described 52 statement about nonrespondents 52 or dropouts or deaths 43,52 missing data addressed 52 power calculation to assess adequacy of sample size52,61 or sample size calculated for adequate power 43 statistical procedures referenced or described 61 p values stated66 confidence intervals given for main results 54,55,56,61,66 data gathering procedures described 48 data collection instruments or source of data described43,51 at least one hypothesis stated 64 both statistical and clinical significance acknowledged67
Consistency (Dependability vs. Reliability)	external audit of process 23	standardization of observers described 64
Neutrality (Confirmability vs. Objectivity)	external audit of data and reconstructions of the data 23 bracketing 18,25,47 statement of researcher’s assumptions 13,48 or statement of researcher’s perspective 66

4. Discussion

This paper has attempted to propose criteria for the purpose of critically appraising primary mixed-method studies. It is acknowledged that this is not a simple task; rather than a final product, this paper is an initial step in the process of identifying such criteria. It is necessary that the “final product” is manageable and realistic to achieve in the length of a peer-reviewed journal article. We therefore envision the final product to be a reduced version of Table 3 combined with criteria specific to mixed-methods such as acknowledgement of paradigm assumptions/differences and identification of the mixed-methods design.

An alternative to this recommendation might be the criteria shared by both methods such as the description of participants/study population and the description of the study context or setting. However, it is the authors’ opinion that the methodological evaluation of mixed-method studies should not be limited to those criteria which are shared. Paradigmatic differences between quantitative and qualitative methods imply that criteria shared by both methods may not be equally important to each method.

It is interesting to note that most of the criteria generated by the literature review fit into the first 2 categories of Lincoln and Guba’s framework. That is, most of these criteria address the goals of truth value (credibility vs. internal validity) and applicability (transferability vs. external validity). It is possible that the remaining two goals are not as relevant for critical appraisal or that it is more difficult to operationalize criteria which should be included under these goals. It is also possible that the criteria overlap and that those which are suitable for these goals are more suitable under one of the first 2 goals.

While this paper has attempted to identify criteria which are generic to mixed-method studies, future guidelines for criteria might be more specific depending upon the mixed-methods design. As mixed-methods research becomes more sophisticated, there may be varying combinations of criteria for the specific designs. For example, the criteria to critically appraise an ethnographic-randomized controlled trial design may differ from those for a phenomenological-case-control design.

The overall goal of proposing criteria for critically appraising mixed-method studies is to promote standards for guiding and assessing the methodological quality of studies of this nature. It is anticipated that, as with quantitative and qualitative research, there will eventually be guidelines for reporting and writing about mixed-method studies. It is hoped that this paper will stimulate future dialogue on the critical appraisal of such studies.

5. Future Considerations in the Pursuit of Critical Appraisal for Mixed-method Studies

The application of criteria to mixed-method studies will rely upon reported criteria. It is possible that criteria will have been met but not reported due to editorial revisions or journal guidelines concerning the length of articles. However, one purpose of this paper is to improve the reporting of methods in mixed-method studies; it can be argued, therefore, that is appropriate to make recommendations based on criteria which have been reported.

This paper is limited by the judgements made by the authors concerning exclusion and inclusion of criteria. We believed that the proposed criteria for each method should be generic and applicable to other studies within that method. We also believed that criteria had to be operationally defined. Having said this, “description of study population” does not specify what elements of a study population should be described. For the purpose of this paper, any description of the sample allowed this criteria to be met. All exclusions have been identified in Table 1; it is possible that these exclusions may be challenged during the revision of criteria. The authors welcome such debate. In the spirit of furthering this debate, we recommend the following considerations:

The criteria in Table 3 should be further refined and then applied to a sample of mixed-method studies in the health literature. It is also recommended that experts in mixed-methods research be asked to identify mixed-method studies which they judge to be exemplary so that the criteria met by these studies can be assessed as well.

It is noted that there are few subject headings in the selected databases which reflect the key concepts of this paper. An attempt was made to devise a comprehensive search strategy which often relied on the use of textwords. The inclusiveness of this strategy is unknown. The search strategy also focused on the peer-reviewed literature; relevant books, conference proceedings, and unpublished manuscripts may have been missed.

This paper assumed that all criteria generated by the literature were equally important. However, it is possible that the criteria met by mixed-method studies might be those which are easiest to meet rather than those which are important. The revision of criteria should involve consultation with critical appraisal experts and experienced qualitative, quantitative, and mixed-method researchers. It is proposed that this panel of experts and experienced researchers rate the criteria according to importance and that this rating be taken into account during the revision process.