Risk of bias in chiropractic mixed methods research: a secondary analysis of a meta-epidemiological review

Abstract

Objective

To examine the risk of bias in chiropractic mixed methods research.

Methods

We performed a secondary analysis of a meta-epidemiological review of chiropractic mixed methods studies. We assessed risk of bias with the Mixed Methods Appraisal Tool (MMAT) and used generalized estimating equations to explore factors associated with risk of bias.

Results

Among 55 eligible studies, a mean of 62% (6.8 [2.3]/11) of MMAT items were fulfilled. In our adjusted analysis, studies published since 2010 versus pre-2010 (adjusted odds ratio [aOR] = 2.26; 95% confidence interval [CI], 1.39 to 3.68) and those published in journals with an impact factor versus no impact factor (aOR = 2.21; 95% CI, 1.33 to 3.68) were associated with lower risk of bias.

Conclusion

Our findings suggest opportunities for improvement in the quality of conduct among published chiropractic mixed methods studies. Author compliance with the MMAT criteria may reduce methodological bias in future mixed methods research.

Introduction

Mixed methods research involves combining quantitative and qualitative approaches in a single study. With multiple methods of data collection and analysis, research questions are answered with a greater breadth and depth of understanding than what could be achieved with only a quantitative or qualitative approach.^1–4 As such, the use of mixed methods designs in research involving the chiropractic⁵ and allied health care professions^1,2 has increased in recent years. For instance, in the PubMed database alone, there has been an exponential rise in the number of “mixed methods” articles published since 2001 (Figure 1). However, despite the added value of mixed methods approaches, these studies can become complex investigations requiring additional time and resources and a research team with expertise in quantitative, qualitative, and mixed methodologies.^1,2

Figure 1.

Frequency of “mixed methods” articles published over the last 20 years in PubMed.

The explicit mixing or linking of qualitative and quantitative approaches within a mixed methods study is particularly useful for assessing multilevel programs and interventions^3,6–8, and is therefore a methodology well-suited to address research problems in health professions including chiropractic. For example, Maiers et al.⁹ used a multistage, experimental mixed methods design², where interviews (qualitative) were conducted to help explain differences in outcomes (quantitative) in a randomized controlled trial of elderly patients with chronic neck pain. The four main types of study designs used in mixed methods research, namely convergent, explanatory sequential, exploratory sequential, and complex / multistage, are described in Table 1.

Table 1.

Types of mixed methods study designs.¹,²

Study design	Descriptiona
Convergent	A mixed methods design in which the researcher collects and analyzes two separate databases – quantitative and qualitative – and then merges the two for the purpose of comparing the results or adding transformed qualitative data as numeric variables into the quantitative database.
Explanatory sequential	A two-phase mixed methods design in which the researcher starts with the collection and analysis of quantitative data, which is then followed by the collection and analysis of qualitative data to help explain the initial quantitative results.
Exploratory sequential	A three-phase mixed methods design in which the researcher starts with the collection and analysis of qualitative data, which is then followed by a design phase of translating the qualitative findings into an approach or tool that can be tested quantitatively. Then, in the third phase, this approach or tool is tested quantitatively. This means that the approach or tool will be grounded in the views of participants.
Complex / multistage
1. Experimental (or intervention)	A complex mixed methods approach in which the researcher combines the collection and analysis of both quantitative and qualitative data and integrates the information within an experimental quantitative research design.
2. Case study	A type of complex mixed methods study in which both quantitative and qualitative data collection and their results are used to develop a case or multiple cases for further analysis and comparisons.
3. Participatory-social justice	A type of complex mixed methods design in which the researcher adds a core design (i.e., convergent, explanatory sequential, or exploratory sequential) to a theoretical framework.
4. Evaluation	A type of complex mixed methods design in which one or more core designs (i.e., convergent, explanatory sequential, exploratory sequential) are added into the steps of an evaluation procedure.

^aSource: adapted from Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 3rd ed. Thousand Oaks, CA: Sage 2018.

Previous methodological reviews have examined the mixed methods literature in healthcare fields outside chiropractic^10,11 and have highlighted areas for improvement. One review¹⁰ of complementary and alternative medicine literature (excluding studies on chiropractic) found that most mixed methods studies did not contain adequate details on qualitative analysis, or quantitative and qualitative sampling and recruitment procedures. To date, no reviews have investigated the extent of methodological bias among published mixed methods studies involving chiropractic research. To address this knowledge gap, we undertook a secondary analysis of a meta-epidemiological review of reporting quality in chiropractic mixed methods research⁵ to examine the risk of bias among chiropractic mixed methods studies. Methodological bias is a serious threat to the internal validity of studies and limits the strength of inferences generated from primary research. As such, our findings will inform areas for improvement regarding the methodological quality of chiropractic research employing mixed methods designs.

Methods

Reporting

Our review is reported in accordance with an adapted version of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for meta-epidemiological research.¹²

Information sources

In line with our published protocol⁵, we searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Index to Chiropractic Literature (ICL) to identify all published chiropractic mixed methods studies from database inception to December 31, 2020. An academic librarian (RJC) assisted with the development of our search strategy (Online Supplementary File 1).⁵ We also hand-searched the reference lists of eligible articles and contacted two mixed methods experts to identify any additional citations. The eligibility criteria for our review are listed in Table 2.

Table 2.

Article inclusion and exclusion criteria.

Inclusion criteria	Exclusion criteria
1. Published in English in a peer-reviewed journal;	1. Study protocols, letters, editorials, or commentaries;
2. Authored by one or more chiropractic researchers;	2. Case reports or series;
3. Was an empirical study reporting primary data collection;	3. Books and book chapters;
4. Involved any type of chiropractic research (e.g., therapeutic, educational, policy, or scope of practice); and	4. Grey literature (e.g., conference proceedings, abstracts, lectures, dissertations or unpublished manuscripts); and
5. Reported the use of both qualitative and quantitative methods, or mixed qualitative methods, in the same research study.a	5. Secondary sources of evidence, including clinical practice guidelines, systematic, scoping or narrative reviews.

^a‘Mixed’ surveys (i.e., those with both closed- and open-ended questions) were only included if the use of “mixed methods” was explicitly stated in the title or abstract.

Study selection

Two independent reviewers (PCE, CC) screened titles and abstracts of identified citations, and full texts of potentially eligible studies. Disagreements were resolved by discussion or, when needed, with the help of an adjudicator (KJS). We used online systematic review software (DistillerSR, Evidence Partners, Ottawa, Canada; https://www.evidencepartners.com) to facilitate literature screening.

Data extraction and assessment of risk of bias

Pairs of reviewers (PCE, KJS, PSN, JVN, CAB) independently extracted data and assessed risk of bias of included articles using standardized, pilot-tested data extraction forms.⁵ Discrepancies were resolved by discussion to achieve consensus or, if needed, adjudication by a third reviewer with expertise in mixed methods (PCE, KJS, LM, or MO). We extracted the following information from all eligible studies: (1) first author, (2) number of authors, (3) journal name, (4) year of publication, (5) country where the study was conducted (or country of residence of the corresponding author when the country of conduct was unavailable or when the study was international), (6) type of mixed methods design, and (7) inclusion of a methodologist among the authors (rated as “yes,” “no,” or “unclear/not reported”).

We defined a methodologist as a contributing author with training in qualitative and/or mixed methods research, public health, epidemiology, health technology assessment, health services research, knowledge translation/implementation science, or biostatistics. The involvement of a methodologist was determined by examining each article for authors’ qualifications or affiliations and information reported in the methodology section. In instances where authorship reporting of methodological expertise was “unclear or not reported,” we used a conservative approach and combined these counts with the “no” responses. When available, we also obtained the impact factor at the time of publication for each journal in which an eligible study was published, either directly from the journal’s website or from the Journal Citation Report (https://jcr.clarivate.com/).

We assessed risk of bias of included articles using the Mixed Methods Appraisal Tool (MMAT).^13,14 The MMAT (version 2011) has been validated as a quality appraisal tool for systematic reviews of mixed studies (i.e., qualitative, quantitative, and mixed methods studies).¹⁴ This tool is comprised of two screening items for mixed methods research, followed by 11 appraisal items in three sections, including: (1) four items on the qualitative component, (2) four items on the quantitative component (i.e., randomized controlled, non-randomized, or descriptive), and (3) three items on mixed methods.

Reviewers independently evaluated the risk of bias of all selected articles with the MMAT, on an item-by-item basis, rating each item with a “yes” (if the item was addressed), “can’t tell/partial” (if the item was partially addressed), or “no” (if the item was not addressed).^14,15 Before assessing the risk of bias of articles, reviewers completed the online tutorial by Pluye et al.¹⁵ to rate MMAT items. We assigned a score for each of the eleven items as follows: 1 = “yes”; 0.5 = “can’t tell/partial”; 0 = “no”, for a total score ranging from 0 to 11.

Synthesis of results

Agreement on full-text screening was assessed using the adjusted kappa (κ) statistic.¹⁶ Values of 0 to 0.20 represented slight agreement, 0.21 to 0.40 represented fair agreement, 0.41 to 0.60 represented moderate agreement, 0.61 to 0.80 represented substantial agreement, and greater than 0.80 represented almost perfect agreement. For the purpose of analysis, studies reporting quantitative and qualitative results in separate papers were combined and considered as a single article. We summarized article characteristics and MMAT score data across included studies using mean and standard deviation (SD) for continuous variables that were normally distributed, and median and inter-quartile range (IQR) for continuous variables that were not. All distributions were analyzed for normality by examining the data histograms, probability and quantile-quantile plots, and the Kolmogorov-Smirnov test.

We built a generalized estimating equation (GEE) to explore the association between risk of bias and article characteristics including publication date, authorship, and journal impact factor. Based on previous literature^17,18, we hypothesized that studies published since 2010 (i.e., ≥ 1 year after the first publication of the MMAT criteria¹³), studies published in journals with an impact factor, those with a greater number of authors, and those that included a methodologist would be associated with lower risk of bias.

We modelled our dependent variable as the number of MMAT items addressed in each article (maximum value of eleven) divided by the total number of MMAT items (eleven), and used the ‘events/trials’ function in SPSS to generate a binary outcome. We regressed the dependent variable on the year of article publication (2010 and later versus pre-2010), availability of a journal impact factor (yes versus no), number of authors (higher versus lower), and inclusion of a methodologist (yes versus no). These factors have previously been shown to be associated with reported methodological quality.^17,18 We dichotomized author number at the median value (four) calculated across included studies. In our original protocol⁵, we planned to explore inclusion of a mixed methodologist as an independent variable; however, we modified our approach because most studies did not clearly report mixed methodological expertise.

For our GEE, we employed a binomial distribution and logit link function to generate a crude and adjusted odds ratio (OR), and a 95% confidence interval (CI) and corresponding p-value, for each independent variable. We assessed goodness-of-fit by comparing our model’s deviance to its degrees of freedom and by examining the associated residual plot. We addressed over- or under-dispersion by re-running our model with a scale parameter calculated by dividing the deviance by its degrees of freedom. To account for potential clustering or similarity of articles published in the same journal, we assumed an exchangeable working correlation matrix and specified the journal name as a grouping factor.

A minimum sample of 40 chiropractic mixed methods articles was required to guard against over-fitting of our regression model (i.e., minimum of 10 observations per independent variable).¹⁹ We also explored variance inflation factors (VIFs) to assess for multicollinearity among independent variables, and considered a VIF ≥ 10 as problematic.²⁰ The two-sided statistical significance level (α) was 5%, and all data and comparative analyses were performed using SPSS v26.0 (IBM SPSS Statistics©).

Results

We identified 1,040 citations, and 65 articles met our eligibility criteria for review. Ten studies reported quantitative and qualitative results in separate articles. As such, 55 unique mixed methods studies were analyzed (Figure 2). There was substantial agreement at the full-text screening stage between reviewers (κ = 0.70).

Figure 2.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram.

Study characteristics

Of the 55 eligible studies, most (80%) were conducted in, or had corresponding authors from, three countries – the United States, Canada, or Australia; over half (53%) had four or fewer authors and three-quarters (75%) were published after 2010 (Table 3). Two-thirds of studies employed a complex/multistage (34%) or convergent (33%) mixed methods design, and the remainder used sequential explanatory (20%) or exploratory (13%) designs. Over half of eligible studies (29 of 55; 53%) were published in journals that had an impact factor (median impact factor at the time of publication = 1.9 [IQR: 1.2 to 2.6]) and just under half (25 of 55; 45%) included a methodologist among their authors.

Table 3.

Characteristics of the 55 included studies.

Study characteristic	Category	n (%)
Year of publication	Pre-2010	14 (25.5)
	Post-2010	41 (74.5)
Number of authors a,b	≤ 4	29 (52.7)
	> 4	26 (47.3)
Country	USA	28 (50.9)
	Canada	10 (18.2)
	Australia	6 (10.9)
	Other c	11 (20.0)
Mixed methods design	Complex / multistage	19 (34.6)
	Convergent	18 (32.7)
	Explanatory sequential	11 (20.0)
	Exploratory sequential	7 (12.7)
Methodologist	Yes	25 (45.5)
	No/unclear d	30 (54.5)
Journal impact factor	Yes	29 (52.7)
	No	26 (47.3)

USA = United States of America.

^aAverage values were used when studies reported quantitative and qualitative results in separate articles.

^bThe cut-off point for author number was derived from the median value measured across eligible studies.

^cIncluded studies from Denmark (n = 3), United Kingdom (n =3), Switzerland (n = 2), Germany (n = 1), South Africa (n = 1), and Sweden (n = 1).

^dInclusion of a methodologist was “unclear” in 15 (27.3%) of the 55 included studies.

Risk of bias of included studies

Referring to the eleven MMAT criteria, items pertaining to qualitative data (i.e., archives, documents, informants, observations) (89%); the qualitative analysis (86%); the mixed methods design, in terms of its relevance to addressing the research questions (86%); and the quantitative randomization, recruitment, or sampling procedures (for randomized, non-randomized, or descriptive study components, respectively) (80%) were commonly addressed. Authors’ descriptions of the integration of qualitative and quantitative data (57%); how qualitative findings related to the context (e.g., the setting, in which the data were collected) (36%) or to the researchers’ influence (e.g., through their interactions with participants) (26%); and specific limitations arising from the integration of qualitative and quantitative components (16%) were the most poorly addressed items (Table 4). Only two (4%) of the 55 studies met all eleven MMAT criteria for methodological quality in mixed methods research. Six studies (11%) met ten criteria, 11 studies (20%) met at least eight criteria, and most studies (36 of 55; 65%) met seven criteria or less.

Table 4.

Risk of bias of the 55 eligible studies according to the Mixed Methods Appraisal Tool (MMAT), version 2011.¹⁵

MMAT Item	Description	Risk of Bias Score and Percentage of Studies (n = 55) fulfilling each MMAT Item
		Score (0–55)a	Percentage
1. Qualitative	Are sources of qualitative data (archives, documents, informants, observations) relevant to address the research question (objective)?	49.0	89.1%
2. Qualitative	Is the process for analyzing qualitative data relevant to address the research question (objective)?	47.0	85.5%
3. Qualitative	Is appropriate consideration given to how findings relate to the context, e.g., the setting, in which the data were collected?	19.0	34.5%
4. Qualitative	Is appropriate consideration given to how findings relate to researchers’ influence, e.g., through their interactions with participants?	13.5	24.5%
5. Quantitative	Randomized controlled (trials): Is there a clear description of the randomization (or an appropriate sequence generation)? Non-randomized: Are participants (organizations) recruited in a way that minimizes selection bias? Descriptive: Is the sampling strategy relevant to address the quantitative research question (quantitative aspect of the mixed methods question)?	44.0	80.0%
6. Quantitative	Randomized controlled (trials): Is there a clear description of the allocation concealment (or blinding when applicable)? Non-randomized: Are measurements appropriate (clear origin, or validity known, or standard instrument; and absence of contamination between groups when appropriate) regarding the exposure/intervention and outcomes? Descriptive: Is the sample representative of the population under study?	36.5	66.4%
7. Quantitative	Randomized controlled (trials): Are there complete outcome data (80% or above)? Non-randomized: In the groups being compared (exposed vs. non-exposed; with intervention vs. without; cases vs. controls), are the participants comparable, or do researchers take into account (control for) the difference between these groups? Descriptive: Are measurements appropriate (clear origin, or validity known, or standard instrument)?	40.0	72.7%
8. Quantitative	Randomized controlled (trials): Is there low withdrawal/drop-out (below 20%)? Non-randomized: Are there complete outcome data (80% or above), and, when applicable, an acceptable response rate (60% or above), or an acceptable follow-up rate for cohort studies (depending on the duration of follow-up)? Descriptive: Is there an acceptable response rate (60% or above)?	39.0	70.9%
9. Mixed Methods	Is the mixed methods research design relevant to address the qualitative and quantitative research questions (or objectives), or the qualitative and quantitative aspects of the mixed methods question (or objective)?	47.5	86.4%
10. Mixed Methods	Is the integration of qualitative and quantitative data (or results) relevant to address the research question (objective)?	31.5	57.3%
11. Mixed Methods	Is appropriate consideration given to the limitations associated with this integration, e.g., the divergence of qualitative and quantitative data (or results) in a triangulation design?	8.5	15.5%
		Risk of Bias Score and Percentage of Studies (n = 55) fulfilling all 11 MMAT Items
		2.0	3.6%

MMAT = Mixed Methods Appraisal Tool.

^aCount scores are summed as 1 = “yes”; 0.5 = “can’t tell/partial”; and 0 = “no”.

The mean (SD) number of the eleven MMAT items fulfilled across studies was 6.8 (2.3). The Kolmogorov-Smirnov test was not significant for the frequency of item scores on the MMAT instrument (p = 0.173), confirming the approximation to a normal distribution. See Figure 3 and Appendix 1 for summaries of the risk of bias scores for the 55 included studies.

Figure 3.

Summary of risk of bias assessments of the 55 eligible studies according to the Mixed Methods Appraisal Tool (MMAT), version 2011.¹⁵ Overall judgements are based on methods by Pluye et al.¹⁵ (Risk-of-bias plot was created using: McGuinness LA, Higgins JPT. Risk-of-bias VISualization (robvis): An R package and Shiny web app for visualizing risk-of-bias assessments. Res Syn Meth. 2020; 1–7.)

Appendix 1.

Article characteristics and Mixed Methods Appraisal Tool (MMAT) item scores for the individual and combined 55 chiropractic mixed methods studies.

First author	Year of publication	Journal	IF at year of publication	MMAT score n = 0–11 (%)
Jamison	1996	Chiropr Tech	NA	1.5 (13.6)
Peterson	1996	J Manipulative Physiol Ther	NA	4 (36.4)
Jamison	1998	Chiropr J Aust	NA	3.5 (31.8)
Perle	1999	J Chiropr Educ	NA	5.5 (50.0)
Waalen	2000	J Chiropr Educ	NA	2 (18.2)
Ammendolia	2002	J Manipulative Physiol Ther	1.041	4.5 (40.9)
Evans/Bronfort	2003/2004	J Manipulative Physiol Ther	0.950/0.457	10 (90.9)
Russell/Page	2004/2006	Vaccine/J Manipulative Physiol Ther	2.824/0.918	8 (72.7)
Pincus	2006	Eur J Pain	3.333	5.5 (50.0)
Evans	2007	J Am Chiropr Assoc	NA	6 (54.6)
Spegman	2007	J Chiropr Educ	NA	6 (54.6)
Garner	2008	Explore (NY)	0.712	5 (45.5)
Rowell	2008	J Manipulative Physiol Ther	1.102	4 (36.4)
Talmage	2009	J Chiropr Med	NA	2 (18.2)
Jones-Harris	2010	Chiropr Man Therap	NA	10 (90.9)
Bronfort/Haanstra	2011/2013	Spine J/Eur Spine J	3.290/2/437	10 (90.9)
Smith	2012	J Manipulative Physiol Ther	1.647	6 (54.6)
Evans	2012/2014	Spine/Eur Spine J	2.159/2.066	11 (100)
Khorsan	2013	Evid Based Complement Alternat Med	2.175	5.5 (50.0)
Palmgren	2013	J Chiropr Educ	NA	8.5 (77.3)
Pohlman	2013	J Chiropr Educ	NA	4.5 (40.9)
Wong	2013	J Can Chiropr Assoc	NA	5 (45.5)
Maiers	2014/2014	Spine J/J Rehabil Med	2.426/1.683	11 (100)
Wong	2014	J Chiropr Educ	NA	6.5 (59.1)
Maiers	2014/2015	Spine J/Man Ther	2.426/1.869	8 (72.7)
Myburgh	2014/2016	J Interprof Care/Chiropr Man Therap	1.399/NA	8 (72.7)
Evans	2015	Glob Adv Health Med	NA	7 (63.6)
Gudavalli	2015	Trials	1.859	8 (72.7)
Bronfort/Maiers	2014/2016	Ann Intern Med/Man Ther	17.810/2.158	10 (90.9)
Testern	2015	Chiropr Man Therap	NA	4.5 (40.9)
Lyons/Goertz	2013/2017	BMC Complement Altern Med/BMC Geriatr	1.877/2.866	8.5 (77.3)
Amorin-Woods	2016	Chiropr Man Therap	NA	3.5 (31.8)
Miller	2016	J Clin Chiropr Pediatr	NA	8.5 (77.3)
Amorin-Woods	2017	Chiropr J Aust	NA	7 (63.6)
Hawk	2017	J Chiropr Educ	NA	4.5 (40.9)
Goertz/Salisbury	2017/2018	BMC Geriatr/Gerontologist	2.866/NA	9 (81.8)
Eilayyan	2018	BMC Musculoskelet Disord	2.002	7.5 (68.2)
Langenfeld	2018	Spine	2.903	5.5 (50.0)
Stuber	2018	Complement Ther Med	1.979	7 (63.6)
Goertz/Wells	2017/2020	BMC Geriatr/J Patient Exp	3.077/NA	10 (90.9)
Stochkendahl	2018/2019	Chiropr Man Therap	NA/1.512	7.5 (68.2)
Amorin-Woods	2019	J Chiropr Educ	NA	7.5 (68.2)
Hestbaek	2019	Chiropr Man Therap	1.512	10 (90.9)
Peterson	2019	Chiropr Man Therap	1.512	7 (63.6)
Whitley	2019	J Manipulative Physiol Ther	1.230	7.5 (68.2)
Cockrell	2020	Gerontol Geriatr Med	NA	5.5 (50.0)
Connell	2020	J Can Chiropr Assoc	NA	9 (81.8)
Emary	2020	Chiropr Man Therap	1.512	7.5 (68.2)
Kim	2020	CMAJ Open	NA	3.5 (31.8)
Major	2020	J Chiropr Educ	NA	7 (63.6)
Pohlman	2020	Chiropr Man Therap	1.512	8.5 (77.3)
Pohlman	2020	J Manipulative Physiol Ther	1.230	5.5 (50.0)
Rae	2020	J Chiropr Med	NA	7 (63.6)
Rist/Connor	2020/2021	Cephalgia/J Manipulative Physiol Ther	4.868/1.230	8.5 (77.3)
Peterson	2021	J Chiropr Educ	NA	7.5 (68.2)

BMC = BioMed Central, IF = Impact Factor, MMAT = Mixed Methods Appraisal Tool, NA = Not Applicable, NY = New York

BMC = BioMed Central, IF = Impact Factor, MMAT = Mixed Methods Appraisal Tool, NA = Not Applicable, NY = New York

Factors associated with risk of bias

In our univariate GEE analyses, studies published since 2010 (OR = 2.64; 95% CI, 1.60 to 4.34), studies published in journals with an impact factor (OR = 2.23; 95% CI, 1.45 to 3.44), those that included more than four authors (OR = 2.01; 95% CI, 1.29 to 3.14), and those that had a methodologist among the study team (OR = 1.64; 95% CI, 1.03 to 2.60) were associated with lower risk of bias (Table 5). In our multivariable GEE analysis, we found that studies published since 2010 (adjusted OR = 2.26; 95% CI, 1.39 to 3.68) and those published in journals with an impact factor (adjusted OR = 2.21; 95% CI, 1.33 to 3.68) remained associated with lower risk of bias (Table 5). As a sensitivity analysis (not reported), we ran the same model but with author number included as a continuous variable rather than a dichotomous variable and this did not change the results. All VIFs were less than 1.9, suggesting no important multicollinearity among the independent variables.

Table 5.

Unadjusted and adjusted odds ratios for the proportion of Mixed Methods Appraisal Tool (MMAT) items fulfilled among the 55 eligible studies.

Factor		Unadjusted OR (95% CI)	P-value	Adjusted OR (95% CI)	P-value
Year of publication
1.	Post-2010	2.64 (1.60–4.34)	< 0.001	2.26 (1.39–3.68)	0.001
2.	Pre-2010	Reference		Reference
Journal impact factor
1.	Yes	2.23 (1.45–3.44)	< 0.001	2.21 (1.33–3.68)	0.002
2.	No	Reference		Reference
Number of authors a
1.	> 4	2.01 (1.29–3.14)	0.002	1.20 (0.76–1.91)	0.441
2.	≤ 4	Reference		Reference
Inclusion of methodologist
1.	Yes	1.64 (1.03–2.60)	0.036	0.79 (0.48–1.31)	0.355
2.	No/unclear	Reference		Reference

CI = confidence interval, OR = odds ratio.

^aThis factor was dichotomized at the median value (i.e., 4), calculated across eligible studies.

Discussion

Summary of main findings

The methodological quality among chiropractic mixed methods studies published in the biomedical and allied health literature is suboptimal. According to the MMAT^14,15, an average of only 60% of the quality criteria in mixed methods research were addressed across the 55 eligible studies. Considerations of reflexivity (i.e., the impact of research setting, or of the researchers themselves, on the qualitative methods and/or findings), as well as the limitations of combining qualitative and quantitative methods, were poorly addressed in approximately 75% of articles. Forty percent of studies also failed to either provide adequate details about allocation concealment, instrument validation, or assessment of selection bias (for studies that employed randomized, non-randomized, or descriptive quantitative components, respectively), or describe the mixing or integration of quantitative and qualitative methods. In addition, follow-up or response rates were inadequate in one-third of studies (see Table 4 for follow-up/response rate thresholds) and a similar number that employed non-randomized or descriptive study components used non-standardized outcome measures. Of the 55 eligible studies, only 4% addressed all eleven MMAT criteria.

Comparison with relevant literature

Our findings are consistent with the results of methodological reviews of mixed methods research in other healthcare fields, including complementary and alternative medicine¹⁰ and nursing¹¹. Similar to our findings, the methodological rigour of many of the mixed methods studies in these reviews was found to be unsatisfactory across various MMAT domains. As most journals have yet to adopt mixed methods quality appraisal guidelines^5,15, authors of chiropractic mixed methods studies have not been required to comply with published methodological standards^13–15. However, with the growing number of mixed methods studies being published within the chiropractic profession in recent years, the findings of our review indicate a need to improve the methodological quality in chiropractic mixed methods research.

In our adjusted analysis, we found that publications in journals with an impact factor and those published in 2010 or later were more than twice as likely to have reported the incorporation of stronger methodological rigour in their mixed methods compared to publications in journals without an impact factor or those published before 2010. Associations between study quality and journal impact factor or year of publication have also been reported in previous reviews.^17,18 It is possible that authors of chiropractic mixed methods studies, particularly those who submit articles to higher impact journals, are increasingly using available risk of bias tools and methodological guidelines to assist in the conduct and reporting of their research. In contrast with previous research^17,18, we did not find a statistically significant association between lower risk of bias and a higher number of authors or inclusion of methodologists. However, methodological scope was broadened in our review to include additional areas of methodological expertise such as qualitative research, epidemiology, or statistics because most studies did not clearly report the inclusion of a mixed methodologist. As mixed methods research requires specialized skills in qualitative and quantitative data integration and analysis^1–4, reporting of mixed methodological expertise would ideally be examined in future studies. Despite our lack of finding a significant association, chiropractors conducting mixed methods studies may wish to undertake training in mixed methods research or collaborate with researchers possessing expertise in mixed methods. Details on the inclusion of mixed methodologists should also be made explicit in future publications.

Strengths and limitations

Our review methodology has several strengths.⁵ First, we conducted a comprehensive search to identify all eligible studies involving chiropractic mixed methods research. Second, we specified the anticipated direction of association for each independent variable in our regression models a priori to give reassurances that significant associations were unlikely to be spurious. Third, we controlled for between-group differences when exploring associations and used GEE modelling to account for hierarchical clustering of articles within journals. Fourth, we performed article screening, data extraction and quality appraisals in duplicate, and all reviewers underwent training in the assessment of MMAT items.

A limitation of our review is we may not have accounted for all important variables (e.g., country of authorship), or interactions between variables¹⁸, relevant to the methodological quality of chiropractic mixed methods research. In addition, the ‘methodologist’ variable as we defined it does not guarantee training in mixed methods. We originally intended to explore the inclusion of a mixed methodologist as an independent factor, but only one of the 55 included studies in our review provided this level of detail. The risk of bias assessments in our review were also limited by the reporting quality of included studies. For instance, some methodologic safeguards may have been implemented by authors but unreported²¹, possibly due to journal restrictions of mixed methods manuscript word counts^18,22, and some methodologic safeguards that were reported may not have been implemented²³. Another limitation of our review is the exclusion of non-English publications, which may have led to selection bias.

Implications for authors and chiropractic journals

To reduce the risk of bias in chiropractic mixed methods research, authors of such studies should be required by journal editors to comply with the MMAT criteria.¹⁵ Compliance with critical appraisal guidelines has been associated with reduced methodological bias in research studies^17,18, particularly if authors are required to meet these standards as a condition of submission. Therefore, editorial review boards of journals within the chiropractic profession could play an important role in improving the quality of conduct in chiropractic mixed methods studies by incorporating mixed methods appraisal tools, such as the MMAT checklist¹⁵, into the peer review process. For example, the MMAT could be a supplementary review checklist, completed by peer reviewers, for each mixed methods journal submission (see MMAT version 2011 checklist in Online Supplementary File 2).¹⁵ Editors of chiropractic journals could highlight the MMAT in their online submission instructions to peer reviewers and prospective authors, and cite well-conducted mixed methods studies involving chiropractic research (e.g., Maiers et al.⁹, Evans et al.²⁴) to serve as exemplars of good methodological quality. Chiropractic journals should also ensure they have at least one mixed methodologist on their editorial board.

Conclusion

Despite a reduction in the risk of bias among chiropractic mixed methods studies in recent years, our findings suggest there is room for improvement. Adoption and utilization of the MMAT criteria by chiropractic journals is one strategy that may reduce methodological bias in future mixed methods studies.