Language bias in orthodontic systematic reviews: A meta-epidemiological study

Samer Mheissen; Loukia M Spineli; Baraa Daraqel; Ahmad Saleem Alsafadi

doi:10.1371/journal.pone.0300881

. 2024 Apr 1;19(4):e0300881. doi: 10.1371/journal.pone.0300881

Language bias in orthodontic systematic reviews: A meta-epidemiological study

Samer Mheissen ^1,^*, Loukia M Spineli ², Baraa Daraqel ^3,⁴, Ahmad Saleem Alsafadi ⁵

Editor: Luiz Alexandre Chisini⁶

PMCID: PMC10984547 PMID: 38557691

Abstract

Background

Orthodontic systematic reviews (SRs) include studies published mostly in English than non-English languages. Including only English studies in SRs may result in a language bias. This meta-epidemiological study aimed to evaluate the language bias impact on orthodontic SRs.

Data source

SRs published in high-impact orthodontic journals between 2017 and 2021 were retrieved through an electronic search of PubMed in June 2022. Additionally, Cochrane oral health group was searched for orthodontic systematic reviews published in the same period.

Data collection and analysis

Study selection and data extraction were performed by two authors. Multivariable logistic regression was implemented to explore the association of including non-English studies with the SRs characteristics. For the meta-epidemiological analysis, one meta-analysis from each SRs with at least three trials, including one non-English trial was extracted. The average difference in SMD was obtained using a random-effects meta-analysis.

Results

174 SRs were included in this study. Almost one-quarter (n = 45/174, 26%) of these SRs included at least one non-English study. The association between SRs characteristics and including non-English studies was not statistically significant except for the restriction on language: the odds of including non-English studies reduced by 89% in SRs with a language restriction (OR: 0.11, 95%CI: 0.01 0.55, P< 0.01). Out of the sample, only fourteen meta-analyses were included in the meta-epidemiological analysis. The meta-epidemiological analysis revealed that non-English studies tended to overestimate the summary SMD by approximately 0.30, but this was not statistically significant when random-effects model was employed due to substantial statistical heterogeneity (ΔSMD = -0.29, 95%CI: -0.63 to 0.05, P = 0.37). As such, the overestimation of meta-analysis results by including non-English studies was statistically non-significant.

Conclusion

Language bias has non-negligible impact on the results of orthodontic SRs. Orthodontic systematic reviews should abstain from language restrictions and use sensitivity analysis to assess the impact of language on the conclusions, as non-English studies may have a lower quality.

Introduction

Systematic reviews (SRs) of randomised controlled trials provide the best summary of evidence to fill the gap by answering a research question, thus, drawing evidence-based practice. Systematic reviews are useful when they have a high methodological quality and include studies with a low risk of bias in the conduct, analysis and reporting [1]. Several shortcomings of SRs were reported in the orthodontic literature. For instance, the lack of grey literature searches [2], the language restriction in the search [2], lack of prediction interval reporting [3], high statistical heterogeneity in the meta-analysis [4], meta-analysis results distorted by small-study effects and publication bias [5], and flaws in the reporting and interpretation of their abstracts [6]. The limitations mentioned above may lead to low-quality evidence [7], as SRs should use standardised and transparent methods to reduce bias and introduce reliable evidence [8].

A well-conducted SR should identify all relevant studies to the question at hand. For this purpose, the Cochrane handbook recommends extensive search in multiple bibliographic databases enhanced by manual and grey literature searches without restriction on the language or the search date [9]. The sensitivity of the search is considered high when the search identifies approximately all relevant reports. Increasing the sensitivity of the search will decrease its precision, as the search will retrieve a large number of studies to include mostly the relevant studies as much as possible. The search precision is defined as the ratio of relevant reports to all identified reports [9]. For instance, if the number of included studies was ten and the search resulted in 1000 records, the precision would be 0.01 (10⁄1000). A high-standard search strategy guarantees the balance between the search’s sensitivity and precision [9].

The English language is considered the dominant language in research [10] including dentistry. While publications from languages other than English is usually considered as of secondary importance. A previous study [11] found that German authors are more likely to publish trials in English when results are statistically significant, increasing the risk of language bias. Likewise, authors from less developed countries tend to publish more positive results than negative results [12]. Language bias may result from publishing significant findings in English language more than other languages [13]. Subsequently, results from only English language studies could provide a biased assessment of the topic.

Systematic review authors usually include studies published in English or languages spoken by the investigators’ team. The search restriction to studies written in the English language was found in approximately 70% of orthodontic SRs [2], which may lead to language bias [11]. A meta-epidemiological study [14] indicated that including non-English language studies might not change the results of some SRs. However, not posing language restrictions to the retrieved studies may enable the researchers to inspect possible geographical bias in reporting positive results [15] and improve the quality of the conclusions drawn from the SRs [2].

A recent review [15] included methodological studies investigating the impact of the English language restriction on the effect estimate and found that the search restriction to English studies only had little impact on the effect estimate. To our knowledge, it is unknown if language bias may impact the meta-analysis results in orthodontics. Therefore, this study aimed to explore whether including non-English language studies in orthodontic SRs may affect the meta-analysis results and to estimate the extent of language bias in orthodontic SRs.

Materials and methods

Protocol and registration

The reporting of this study followed the proposed items to be used for reporting meta-epidemiological methodology research [16]. There was no registration for the protocol.

Eligibility criteria

This study included orthodontic SRs published between 1 January 2017 and 31 December 2021 in five orthodontic journals with the highest impact factor (2021): American Journal of Orthodontic and Dentofacial Orthopedics (3.6), European Journal of Orthodontics (4.3), Progress in Orthodontics (3.7), Angle Orthodontist (3.4), and Orthodontics & Craniofacial Research (2.8). Also, Cochrane orthodontic reviews in the same period were included. Methodological studies, scoping reviews, literature reviews, and systematic reviews with fewer than two included studies were excluded.

Search and study selection

An electronic search was undertaken in Medline via PubMed and Cochrane library for systematic reviews on 20 June 2022. One author (SM) performed the search of PubMed using text words and medical subject headings to retrieve systematic reviews published in the leading orthodontic journals indexed in PubMed (Table 1). All relevant Cochrane orthodontic reviews within the same period were also retrieved by another author (BD) through the Cochrane Oral Health Group. Initial screening for titles and abstracts was performed independently and duplicated by two authors (ASA and BD). Furthermore, two authors (ASA and BD) scrutinised the full text of the potential articles for eligibility. In the presence of disagreement, a consensus was reached after a discussion with a third author (SM).

Table 1. The search strategy in PubMed via medline.

Search Number	Query	Filters	Results
1	"Malocclusion"[Mesh] OR "Malocclusion, Angle Class III"[Mesh] OR "Malocclusion, Angle Class II"[Mesh] OR "Malocclusion, Angle Class I"[Mesh] OR "Malocclusion and Short Stature" [Supplementary Concept]		34,803
2	orthodontic OR orthodonti*		90,239
3	("Orthodontics"[Mesh]) OR ("Orthodontics, Preventive"[Mesh] OR "Orthodontics, Interceptive"[Mesh] OR "Orthodontics, Corrective"[Mesh])		54,829
4	#1 OR #2 OR #3		102,848
5	#4	Systematic Review	1,878
6	#5	Systematic Review, from 2017–2021	1,156

Open in a new tab

Data collection process

A pilot assessment of 30 SRs was undertaken between two authors (BD and ASA) to ensure consistency in the data extraction. After reaching 100% agreement, the same two authors extracted the data and a third author (SM) cross-checked the collected data. All data were entered in a pre-pilot Microsoft Excel^® (Microsoft, Redmond, Washington, USA). The following characteristics for each SR were extracted: the number of authors, continent of the first author, publication year, review type (Cochrane and non-Cochrane), protocol registration, the number of included studies, including non-English studies (yes, no), whether studies were excluded based on the language (yes, no), the number of non-English languages, language restriction in the inclusion/ exclusion criteria (yes, no), involvement of librarian in search (yes, no), type of SR (interventional, epidemiological, or diagnostic), and type of included studies (human, animal, or in vitro). We calculated the precision of the search for each SR by dividing the number of included studies by the number of search results after removing duplicates. If the SR included a non-English language, additional information regarding meta-analysis (MA) was extracted: non-English language included in MA (yes, no), and the statistical significance of summary effect estimate (yes, no). One outcome was selected from each SR that included at least one non-English study to investigate the impact of the non-English language on the meta-analysis results. Specifically, we considered the following ‘algorithm’: if the meta-analysis of primary outcome included at least three studies with at least one being non-English, this meta-analysis was included in our collection. If more meta-analyses were eligible, we opted for the first one addressing the primary outcome. If the meta-analysis of the primary outcome was not eligible, the meta-analysis for secondary outcomes was checked. Therefore, the corresponding forest plot should include at least one non-English study to extract the data. We opted for arm-level data (information reported in each arm of every trial); otherwise, we extracted the data in the contrast-level format as reported in the forest plot.

Statistical analysis and data synthesis

Multivariable binary logistic regression

Proper descriptive statistics were undertaken to summarise the collected characteristics. For categorical characteristics, the absolute and relative frequencies were reported. The median, interquartile range, and minimum and maximum values were provided for metric characteristics. A multivariable binary logistic regression was implemented to explore the association of including a non-English language (yes, no) with each characteristic. All characteristics were included in the model simultaneously, and no variable selection approach (e.g., stepwise selection) was performed. Due to convergence issues stemming from separation in some characteristics, Firth’s bias reduction approach was considered to improve estimation [17]. The results were presented in odds ratio (OR), 95% confidence intervals (CI), and p-values based on the profile penalised likelihood. We concluded a statistically significant association when the 95% CI did not include value of no association (OR = 1), which coincides with a p-value less than 5%; otherwise, the association was statistically non-significant, which coincides with a p-value at least 5%. These CIs are preferred to the Wald CIs in the presence of separation [18]. Precision is measured as percentages and was transformed into the logit scale to be included in the model. The publication year was centred on its mean value to improve the interpretation and convergence of the regression coefficient.

Meta-epidemiological analysis and sensitivity analysis

A two-stage approach was performed to analyse the meta-epidemiological data and estimate the average bias attributed to non-English studies (S1 File). The meta-epidemiological data comprised one meta-analysis from each eligible SR according to the aforementioned algorithm. The standardised mean difference (SMD) was the effect measure since all selected meta-analyses referred to a continuous primary outcome. Initially, a random-effects meta-regression was conducted to estimate the difference in standardised mean difference (ΔSMD) between non-English and English studies in each meta-analysis. Then a random-effects meta-analysis was performed to combine the ΔSMDs across the meta-analyses. We also pooled ΔSMDs using a fixed-effect meta-analysis as a sensitivity analysis to the model assumptions. A negative ΔSMD would indicate that non-English studies overestimate the SMD. The supplementary material provides further details on the meta-epidemiological analysis.

All analyses were conducted in the statistical software R (version 4.2.0) [19]. The summary statistics table was created using the R package gtsummary [20]. We used the R package logistf to apply logistic regression with Firth’s bias reduction [18], and the R package metafor to conduct the random-effects meta-regression using the function rma.mv [21]. The bar plots and forest plots were created using the R-package ggplot2 [22].

Results

Systematic review selection

A total of 1168 SRs were initially identified (Fig 1). Of those, 983 SRs were removed for referring to non-relevant journals. After full text reading of one hundred eighty-five SRs, 174 SRs were included in the present study. Of the 11 excluded SRs, four were methodological studies, two were scoping reviews, two were withdrawn, one did not find any eligible studies, one was a clinical guideline, and one was an erratum (S1 Table).

Characteristics of the systematic reviews

The non-English studies were included in around one quarter (n = 45/174, 26%) of the SRs, representing 3.78% of the total included studies in orthodontic SRs (98/2568). Most SRs included four to six authors (n = 111/174, 64%) affiliated with an institution in Europe (n = 77/174, 45%), comprised non-Cochrane reviews (n = 165/174, 95%) published in the last two years (n = 85/174, 49%) and were registered in PROSPERO (n = 125/174, 72%) (Table 2). These SRs included a median of 11 studies (interquartile range (IQR): 7 to 17) and had a low precision of searches (median: 2%, IQR: 1 to 4%). There were no explicit language restrictions (n = 135/174, 80%) nor explicit exclusion of non-English studies (n = 170/174, 98%) in most SRs. Only 37 (21%) SRs involved a librarian or/and search specialist during the conduct. More than half of the SRs performed a meta-analysis (n = 97/174, 56%). The systematic reviews were mostly interventional (n = 146/174, 84%) and pertained to human participants (n = 156/174, 91%). Overall, the characteristics were similarly distributed between SRs with and without non-English studies. However, SRs with non-English studies had slightly more studies (median: 13, IQR: 9 to 22) than SRs without such studies (median: 10, IQR: 7 to 17).

Table 2. Characteristics of systematic reviews with and without studies on non-English languages.

Characteristic	Non-English language included			OR (95% CI)^*	p-value
Characteristic	Overall (n = 174)	No (n = 129)	Yes (n = 45)	OR (95% CI)^*	p-value
Authors Number
1–3	43 (25%)	30 (23%)	13 (29%)	reference
4–6	111 (64%)	84 (65%)	27 (60%)	0.71 (0.27, 1.93)	0.50
7–9	20 (11%)	15 (12%)	5 (11%)	0.64 (0.14, 2.64)	0.54
Continent
America	46 (26%)	33 (26%)	13 (29%)	reference
Asia & others	51 (29%)	40 (31%)	11 (24%)	0.67 (0.21, 2.04)	0.48
Europe	77 (45%)	56 (43%)	21 (47%)	0.91 (0.35, 2.37)	0.85
Review type
Cochrane	9 (5%)	6 (5%)	3 (7%)	reference
non-Cochrane	165 (95%)	123 (95%)	42 (93%)	3.66 (0.60, 25.81)	0.16
Publication Year
2017	28 (16%)	17 (13%)	11 (24%)	0.81 (0.60, 1.08)	0.15
2018	32 (18%)	26 (20%)	6 (13%)
2019	29 (17%)	22 (17%)	7 (16%)
2020	37 (21%)	25 (19%)	12 (27%)
2021	48 (28%)	39 (30%)	9 (20%)
PROSPERO registration
No	49 (28%)	42 (33%)	7 (16%)	reference
Yes	125 (72%)	87 (67%)	38 (84%)	1.64 (0.58, 5.00)	0.36
Number of included studies
Median (IQR) [range]	11.00 (7.00, 17.00) [2.00, 94.00]	10.00 (7.00, 16.50) [2.00, 94.00]	13.00 (9.00, 22.00) [4.00, 63.00]	1.02 (0.99, 1.05)	0.19
The precision of the search¹
Median (IQR) [range]	0.02 (0.01, 0.04) [0.001, 0.27]	0.02 (0.01, 0.04) [0.001, 0.27]	0.02 (0.01, 0.07) [0.001, 0.26]	1.04 (0.77, 1.40)	0.82
Number of included non-English languages¹
Median (IQR) [range]	1 (1, 3) [1, 19]	NA	1 (1, 3) [1, 19]	NA	NA
Restriction on language¹
No	135 (80%)	91 (74%)	44 (98%)	reference
Yes	33 (20%)	32 (26%)	1 (2%)	0.11 (0.01, 0.55)	0.004
Excluded studies on non-English language
No	170 (98%)	125 (97%)	45 (100%)	reference
Yes	4 (2%)	4 (3%)	0 (0%)	0.03 (0.01, 1.47)	0.08
Librarian/search specialist
No	137 (79%)	103 (80%)	34 (76%)	reference
Yes	37 (21%)	26 (20%)	11 (24%)	2.98 (0.97, 9.46)	0.06
Meta-analysis performed
No	77 (44%)	63 (49%)	14 (31%)	reference
Yes	97 (56%)	66 (51%)	31 (69%)	1.92 (0.79, 4.91)	0.15
Type of systematic review
Diagnostic	9 (5%)	8 (6%)	1 (2%)	reference
Epidemiological	19 (11%)	18 (14%)	1 (2%)	0.48 (0.03, 7.50)	0.58
Interventional	146 (84%)	103 (80%)	43 (96%)	2.32 (0.38, 25.68)	0.38
Type of included studies
Animal	12 (7%)	7 (5%)	5 (11%)	reference
Human	159 (91%)	119 (92%)	40 (89%)	0.29 (0.05, 1.45)	0.13
In vitro	3 (2%)	3 (3%)	0 (0%)	0.08 (0.01, 1.74)	0.11

Open in a new tab

CI, confidence interval; IQR, interquartile range; NA, not applicable; OR, odds ratio

*Strong evidence when the 95% confidence interval excludes 1 (p-value < 5%); otherwise, weak evidence for the corresponding association.

¹ ‘Precision of the search’ and ‘Number of included non-English languages’ were not reported in one systematic review. ‘Restriction on language’ was not reported in six systematic reviews.

Association of including non-English studies with several characteristics

The multivariable binary logistic regression revealed an association of substantial magnitude between the inclusion of non-English studies and various characteristics of the SRs (Table 2). However, the association was not statistically significant for all characteristics apart from the restriction on language: the odds of including a non-English study were lower by 89% (OR: 0.11, 95% CI: 0.01 to 0.55, P = 0.004) in SRs posing a language restriction. Though statistically non-significant, SRs including fewer than four authors, originating from America, being non-Cochrane, having a protocol registration, published in earlier years, involving a librarian, conducting a meta-analysis and specialising in interventional studies were associated with higher odds of including non-English studies (OR range: 1.13 to 3.66) than SRs with more authors, originating from other continents, published recently, being Cochrane SRs, without a protocol registration, a librarian, or a meta-analysis and specialising in other study types.

Frequency of non-English languages

Fig 2 presents the frequency of non-English studies based on non-English language and SR type. The median number of non-English studies was one with an IQR of 1 to 3. The most commonly used language was Chinese, followed by Portuguese and German. Non-Cochrane SRs included more non-English studies than Cochrane SRs. One Cochrane SR included several non-English languages (Farsi, French, Italian, and Portuguese). Five Cochrane SRs considered two non-English languages (Chinese and Portuguese, Chinese and Turkish, German and Portuguese, or Dutch and German), and one Cochrane SR included three non-English studies (Chinese, Portuguese, and Turkish).

Statistical significance and inclusion of non-English studies

Of the 29 SRs that conducted a meta-analysis and reported the language of the synthesised studies, 69% (n = 20) included at least one non-English study; of those meta-analyses, 80% (16/20) provided a statistically significant effect. (Table 3) The univariate binary logistic regression indicated that the odds of statistical significance in the summary effect estimate was 3.20 times larger in systematic reviews with non-English studies than in systematic reviews with only English studies. However, the association was statistically non-significant (OR:3.20, 95%CI: 0.57, 18.92, P = 0.18).

Table 3. The association of statistical significance with the inclusion of non-English studies.

Non-English language included	Statistical significance		Total	OR, (95%CI)	P value
Non-English language included	Yes	No	Total	OR, (95%CI)	P value
No	5	4	9	reference
Yes	16	4	20	3.20 (0.57, 18.92)	0.18

Open in a new tab

Examining the influence of non-English studies on summary results

Out of the sample, only fourteen meta-analyses comprised the dataset of the meta-epidemiological study. The supplementary material provides descriptive statistics on the number of studies and sample size of the included meta-analyses. The meta-epidemiological analysis revealed that non-English studies tended to overestimate the SMD by approximately 0.30 on average compared to English studies (Fig 3). Specifically, the difference of SMD was not statistically significant (ΔSMD = -0.29, 95%CI: -0.63 to 0.05, P = 0.37) using a random-effects model due to substantial statistical heterogeneity, but it was spuriously statistically significant with a fixed-effect model (ΔSMD = -0.31, 95%CI: -0.58 to -0.05, P = 0.03) (Fig 3). The ΔSMD was positive in four meta-analyses, implying a larger SMD from English studies. Overall, including non-English studies improved the precision of the summary SMD, whilst including only English studies decreased SMD substantially in some meta-analyses (S1 Fig). Most meta-analyses were associated with fairly high or extreme statistical heterogeneity, regardless of language restriction (S2 Fig). Restricting inclusion to English studies increased statistical heterogeneity in almost half meta-analyses compared to including all studies regardless of language (S2 Fig).

Fig 3 — *Negative bias favours the English studies.

Discussion

Evidence summary

In general, 26% of orthodontic SRs included non-English language studies in this assessment, which may lead to language bias. There was evidence of language bias also in other medical specialities [23], though the proportion of included non-English studies was lower compared to the orthodontics field (7.4% versus 26%) [24]. This could be related to the fact that less than half of the non-English medical studies are indexed in Medline, and 68% of orthodontic SRs searched Medline [2]. The most frequently included non-English language in orthodontic SRs was the Chinese language, which is in disagreement with the findings of Cochrane reviews as the Chinese studies were the least included in two medical domains [23]. However, the Chinese language comprised 47% of the medical literature published in languages other than English, followed by German and Spanish studies [24].

The language restriction in the search decreased the likelihood of including non-English studies in SRs by 89%, which is a reasonable finding. However, the majority of SRs performing no language restriction did not include a non-English study (67%; 91/135) and four SRs excluded non-English studies after the search process. That could be attributed to limiting the eligibility criteria to English studies without limiting the search strategy, rendering the conduct of SR unnecessarily time-consuming and requiring additional human and financial resources. As such, if only English studies were included in the SR, restricting the search to only English studies would be a valid option and saving factor for time, human and financial recourses. In this approach, non-English records would be removed from the search results, thus, reducing the number of retrieved reports and, subsequently, the initial screening time and efforts. In this regard, manual search, particularly in google scholar may result in non-English studies even if the search was restricted in the other bibliographies. This was evident in one included SR [25] with additional manual search, although the search was restricted in the searching bibliographies.

Furthermore, language bias in SRs may not result only from language restrictions in search strategy or eligibility criteria. It could be due to the information sources used in the search. For instance, MEDLINE, Embase, and CINAHL are international English bibliographic databases which comprise only limited non-English literature. In contrast, special databases can be used for non-English studies [26].

The present study found a statistically non-significant association between including non-English studies in orthodontic SRs and SRs with small co-authorship or SRs led by an author affiliated in American institute. International collaborations including authors speaking several languages may be a possible explanation for this finding. Text screening, risk of bias assessment, and data extraction require an expert researcher who understands the language of the retrieved articles. In this respect, Cochrane can hire translators who are not necessarily directly involved in the reviews (https://www.cochrane.org/join-cochrane/translate).

It is well established that SRs with a registered protocol have a higher quality than non-registered SRs [27]. Our study revealed that SRs with a protocol registration were more likely to include non-English studies. The inclusion of non-English studies also coincided with a higher quality of SRs when performing the unrestricted comprehensive search as provided by the AMSTAR2 tool [28].

We found that meta-analyses with non-English studies were more likely to yield statistical significance than meta-analyses that included only English studies. However, this association was not statistically significant due to the small analysed sample, which may have led to insufficient power to detect the difference. Relevant empirical studies on several medical fields uncovered that excluding non-English studies from SRs impacted the meta-analysis results, with the change in the effect estimate varying from negligible to large [15, 23]. Moreover, the statistical heterogeneity was reduced after removing non-English studies from the meta-analyses [29, 30]. Our meta-epidemiological analysis demonstrated a larger summary effect size from including non-English studies, though this was not statistically significant due to material statistical heterogeneity. Study characteristics such as risk of bias and sample size may have contributed to the heterogeneity in ΔSMDs observed across the meta-analyses. Investigating sources of heterogeneity that may explain the difference in treatment effects between non-English and English studies was out of this study scope. The analysis also found that the statistical heterogeneity was reduced in almost half of meta-analyses when both non-English and English studies were included (S2 Fig).

Moreover, different studies [24, 31–33] assessed the quality of the English and non-English studies, and found a higher risk of bias in non-English studies due to suboptimal randomisation, insufficient reporting of the blinding, and incomplete data. As such, non-English studies may be removed from the meta-analysis in the context of a sensitivity analysis to inspect the robustness of the meta-analysis results.

Though including non-English studies in SR may be laborious in terms of time and resources [15], it is still an imperative option in the case of trials’ paucity. For instance, one SRs [34] in our sample included only non-English studies. Likewise, including non-English studies has a crucial impact when the disease of interest has a different prevalence between ethnicities. For example, class III malocclusion is more prevalent in Asian people than in other ethnicities [35], rendering studies from these regions important in providing valuable information. On a positive note, authors’ collaboration may facilitate including these studies in the SRs if at least one expert author in the language is involved.

Limitations and strength

The search in this study was restricted to two databases and SRs published in the last five years, possibly missing some relevant studies. A wider search may have some impact on the importance of including non-English studies. For instance, it may turn statistically non-significant results into statistically significant, particularly when more meta-analyses with non-English language are included in the meta-epidemiological study. Besides, this study attempted to map the problem and merely explore the implications of language bias on the meta-analysis results. The bias associated with including non-English studies may be confounded by the quality of the conduct, analysis and reporting of these studies. We did not investigate the quality of the SRs as a function of language bias since it was out of the scope. Three experienced investigators implemented the study selection and the data acquisition, and the analysis was conducted by an experienced biostatistician using state-of-the-art methods, ensuring a high-quality meta-epidemiological study. The authors were concerned about the unaddressed correlation that may arise from including more than one meta-analysis from the same SR in the meta epidemiological analysis. As such, only one meta-analysis with non-English study was involved. The prior protocol was not registered due to the constraints imposed by the COVID period, which may introduce some bias. However, the authors diligently adhered to their protocol and rigorous guidelines.

Conclusion

Language restriction seems to have no statistically significant impact on the results of orthodontic SRs. Including non-English studies in orthodontic SRs aligns with the scope of systematic review to retrieve all relevant evidence and may also increase the statistical power due to the increase in the sample of studies. Furthermore, language inclusiveness may aid in gauging the quality of the evidence base and uncovering knowledge gaps to prioritise future research. For instance, research questions addressed exclusively by non-English studies of questionable quality render the evidence at hand restrictive and inappropriate for guideline recommendations. Then, a research agenda may be set up to answer the research question anew using a living systematic review that is constantly updated with new studies and poses no language restriction. However, authors should bear in mind that non-English studies may have a higher risk of bias and should assess the appropriateness of non-English studies individually according to the specific topic of the review.

Supporting information

S1 Table. Studies excluded with the reason.

(DOCX)

pone.0300881.s001.docx^{(15.1KB, docx)}

S1 File. Technical notes.

(DOCX)

pone.0300881.s002.docx^{(35.6KB, docx)}

S1 Fig. Forest plots on the summary standardised mean difference and 95% confidence interval in 14 meta-analyses.

(DOCX)

pone.0300881.s003.docx^{(212.5KB, docx)}

S2 Fig. Line plot on the between-study standard deviation (x-axis) in 14 meta-analyses (y-axis).

(DOCX)

pone.0300881.s004.docx^{(200.2KB, docx)}

Data Availability

Data Availability: The data underlying this paper were shared on open repository: https://doi.org/10.5281/zenodo.10201400.

Funding Statement

The author(s) received no specific funding for this work.

References

1.Hooper EJ, Pandis N, Cobourne MT, Seehra J. Methodological quality and risk of bias in orthodontic systematic reviews using AMSTAR and ROBIS. Eur J Orthod. 2021;43(5):544–50. doi: 10.1093/ejo/cjaa074 . [DOI] [PubMed] [Google Scholar]
2.AlMubarak D, Pandis N, Cobourne MT, Seehra J. Reporting of the methodological quality of search strategies in orthodontic quantitative systematic reviews. Eur J Orthod. 2021;43(5):551–6. doi: 10.1093/ejo/cjaa078 . [DOI] [PubMed] [Google Scholar]
3.Seehra J, Stonehouse-Smith D, Pandis N. Prediction intervals reporting in orthodontic meta-analyses. Eur J Orthod. 2021;43(5):596–600. doi: 10.1093/ejo/cjab037 . [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Koletsi D, Fleming PS, Michelaki I, Pandis N. Heterogeneity in Cochrane and non-Cochrane meta-analyses in orthodontics. Journal of dentistry. 2018;74:90–4. Epub 20180505. doi: 10.1016/j.jdent.2018.05.003 . [DOI] [PubMed] [Google Scholar]
5.Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Assessing small study effects and publication bias in orthodontic meta-analyses: a meta-epidemiological study. Clin Oral Investig. 2014;18(4):1031–44. Epub 2014/02/15. doi: 10.1007/s00784-014-1196-3 . [DOI] [PubMed] [Google Scholar]
6.Makou O, Eliades T, Koletsi D. Reporting, interpretation, and extrapolation issues (SPIN) in abstracts of orthodontic meta-analyses published from 2000 to 2020. European Journal of Orthodontics. 2021;43(5):567–75. doi: 10.1093/ejo/cjab009 [DOI] [PubMed] [Google Scholar]
7.Koletsi D, Fleming PS, Eliades T, Pandis N. The evidence from systematic reviews and meta-analyses published in orthodontic literature. Where do we stand? Eur J Orthod. 2015;37(6):603–9. Epub 2015/02/11. doi: 10.1093/ejo/cju087 . [DOI] [PubMed] [Google Scholar]
8.Higgins JPT TJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Second edition ed.
9.Lefebvre C G J, Briscoe S, Featherstone R, Littlewood A, Marshall C, Metzendorf M-I, et al. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T. Searching for and selecting studies. In: Page MJ W V, editor. Cochrane Handbook for Systematic Reviews of Interventions version 63 (updated February 2022). UK: Cochrane; 2022. [Google Scholar]
10.FERGUSON G, PÉREZ-LLANTADA C, PLO R. English as an international language of scientific publication: a study of attitudes. World Englishes. 2011;30(1):41–59. doi: 10.1111/j.1467-971X.2010.01656.x [DOI] [Google Scholar]
11.Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. The Lancet. 1997;350(9074):326–9. doi: 10.1016/S0140-6736(97)02419-7 [DOI] [PubMed] [Google Scholar]
12.Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JP. Comparative effect sizes in randomised trials from less developed and more developed countries: meta-epidemiological assessment. BMJ. 2013;346:f707. Epub 20130212. doi: 10.1136/bmj.f707 . [DOI] [PMC free article] [PubMed] [Google Scholar]
13.J B, EA S, C H. Langauge bias. In: Catalogue Of Bias 2017. https://www.catalogueofbias.org/biases/language-bias.
14.Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, et al. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int J Technol Assess Health Care. 2012;28(2):138–44. doi: 10.1017/S0266462312000086 . [DOI] [PubMed] [Google Scholar]
15.Dobrescu AI, Nussbaumer-Streit B, Klerings I, Wagner G, Persad E, Sommer I, et al. Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review. J Clin Epidemiol. 2021;137:209–17. Epub 20210430. doi: 10.1016/j.jclinepi.2021.04.012 . [DOI] [PubMed] [Google Scholar]
16.Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42. Epub 20170712. doi: 10.1136/ebmed-2017-110713 . [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Firth D. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80(1):27–38. doi: 10.1093/biomet/80.1.27 [DOI] [Google Scholar]
18.Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Stat Med. 2002;21(16):2409–19. doi: 10.1002/sim.1047 . [DOI] [PubMed] [Google Scholar]
19.R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2022. https://www.R-project.org/.
20.DD S, Whiting K C M, Lavery JA, Larmarange J. Reproducible summary tables with the gtsummary package. The R Journal. 2021;13:570–80. [Google Scholar]
21.Viechtbauer W. Conducting Meta-Analyses in R with the metafor Package. Journal of Statistical Software. 2010;36(3):1–48. doi: 10.18637/jss.v036.i03 [DOI] [Google Scholar]
22.Wickham H. ggplot2: Elegant Graphics for Data Analysis. New York: Springer-Verlag; 2016. [Google Scholar]
23.Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64. Epub 20170419. doi: 10.1186/s12874-017-0347-z . [DOI] [PMC free article] [PubMed] [Google Scholar]
24.Dechartres A, Atal I, Riveros C, Meerpohl J, Ravaud P. Association Between Publication Characteristics and Treatment Effect Estimates: A Meta-epidemiologic Study. Ann Intern Med. 2018;169(6):385–93. Epub 20180821. doi: 10.7326/M18-1517 . [DOI] [PubMed] [Google Scholar]
25.Elsten E, Caron C, Dunaway DJ, Padwa BL, Forrest C, Koudstaal MJ. Dental anomalies in craniofacial microsomia: A systematic review. Orthod Craniofac Res. 2020;23(1):16–26. Epub 20191028. doi: 10.1111/ocr.12351 . [DOI] [PMC free article] [PubMed] [Google Scholar]
26.Xia J, Wright J, Adams CE. Five large Chinese biomedical bibliographic databases: accessibility and coverage. Health Info Libr J. 2008;25(1):55–61. doi: 10.1111/j.1471-1842.2007.00734.x . [DOI] [PubMed] [Google Scholar]
27.Sideri S, Papageorgiou SN, Eliades T. Registration in the international prospective register of systematic reviews (PROSPERO) of systematic review protocols was associated with increased review quality. J Clin Epidemiol. 2018;100:103–10. Epub 2018/01/18. doi: 10.1016/j.jclinepi.2018.01.003 . [DOI] [PubMed] [Google Scholar]
28.Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. Epub 2017/09/25. doi: 10.1136/bmj.j4008 at http://www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work. [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76. doi: 10.1016/j.jclinepi.2004.08.021 . [DOI] [PubMed] [Google Scholar]
30.Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90. doi: 10.3310/hta7410 . [DOI] [PubMed] [Google Scholar]
31.Hugues A, Di Marco J, Bonan I, Rode G, Cucherat M, Gueyffier F. Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials. PloS one. 2020;15(3):e0229822. Epub 20200309. doi: 10.1371/journal.pone.0229822 . [DOI] [PMC free article] [PubMed] [Google Scholar]
32.Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol. 2002;31(1):115–23. doi: 10.1093/ije/31.1.115 . [DOI] [PubMed] [Google Scholar]
33.Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76. . [PubMed] [Google Scholar]
34.Guo R, Zhang L, Hu M, Huang Y, Li W. Alveolar bone changes in maxillary and mandibular anterior teeth during orthodontic treatment: A systematic review and meta-analysis. Orthod Craniofac Res. 2021;24(2):165–79. Epub 20200916. doi: 10.1111/ocr.12421 . [DOI] [PubMed] [Google Scholar]
35.Alhammadi MS, Halboub E, Fayed MS, Labib A, El-Saaidi C. Global distribution of malocclusion traits: A systematic review. Dental Press J Orthod. 2018;23(6):40.e1–.e10. doi: 10.1590/2177-6709.23.6.40.e1-10.onl . [DOI] [PMC free article] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0300881.r001

Decision Letter 0

Luiz Alexandre Chisini

21 Nov 2023

PONE-D-23-28166Language Bias in Orthodontic Systematic Reviews: A Meta-epidemiological StudyPLOS ONE

Dear Dr. Mheissen,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Jan 05 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Luiz Alexandre Chisini, Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf 2. We note that you have referenced (Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey) which has currently not yet been accepted for publication. Please remove this from your References and amend this to state in the body of your manuscript: (Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey [Unpublished]”) as detailed online in our guide for authorshttp://journals.plos.org/plosone/s/submission-guidelines#loc-reference-style. 3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Additional Editor Comments:

The article's proposal is interesting and innovative; however, the authors need to clarify some points of the methods and need to revise the interpretations, that are not accurate.

Abstract:

- The sentence should be revised: “The association between SRs characteristics and including non-English studies was overall statistically inconclusive apart from the restriction on language: the odds of including a non-English study were reduced by 89% in SRs with a language”. Was the result associated or not? please present the OR and 95% CI.

- In the sentence: “The evidence was inconclusive due to substantial statistical heterogeneity using a random-effects model, but it was spuriously conclusive with a fixed-effect model”. Inconclusive is not the best way to interpretate the p-value. Please revise all the paper and the interpretations of this result. Moreover, fixed-model is only a sensitivity analysis and should be only discussed in the discussion.

- Authors report that “174 SRs were eligible for inclusion in this study.”. However, this can lead the readers to wrong interpretations since the authors had two different analyses: i) SR and ii) meta-analysis. So, only 14 studies were included in the meta-analysis. This should be clear in the abstract.

Methods:

- Why did de authors not register a protocol? This introduces an important bias in the study

- Authors report “[…] in orthodontic journals with the highest impact factor (2021):”. Was there a cut-off? The authors need to clarify the criteria for each covariable used in the study.

- Why only two databases were searched?

- Considering that the statistical tests used by the authors are based on the assumption of independence of observations, did the authors investigate whether the studies included in each meta-analysis are from unique studies? Authors need to ensure that the independence of observations is not violated. So, if one meta-analysis study was included in more than one study, the results lose independence.

- Authors reported that: “One primary outcome was selected from each SR that included at least one non-English study to investigate the impact of the non-English language on the meta-analysis results.”; what were the criteria for standardizing this selection?

- Authors mention that did a meta-regression ”Initially, a random-effects meta-regression was

conducted to estimate the difference in standardised”, However, I didn’t find this in the paper. Please, revise it.

Results:

- It is recommended to cite the references of excluded studies in the sentence: “Of the 11 excluded SRs, four comprised methodological studies, two were scoping reviews, two were withdrawn, one did not find any eligible studies, one was a clinical guideline, and one was an erratum.”

- Please, highlight that only 14 studies were included as meta-analyses to avoid misinterpretation by the readers.

- Revise the paper and don’t use “statistically inconclusive“ as in the sentence: “the association was statistically inconclusive (p-value > 0.05)”. It is not inconclusive. In fact, there is no association.

- Is from which analysis the results “the odds of including a non-English study were reduced by 89% (95% CI: 45% to 99%)”? Because I didn’t find these results in the tables.

- In the sentence “80% provided a conclusive summary effect estimate (p-value < 0.05)”, please, change “conclusive” to “association”

- Did the authors include 14 or 20 studies in the meta-analysis evaluation? Because sometimes the authors' reporting is 20 and sometimes 14 In the text.

- In the sentence “Most meta-analyses were associated with fairly high or extreme statistical heterogeneity” how did the authors estimate the heterogeneity of the studies? This is not mentioned in the results. Please, consider that I2 cannot estimate the heterogeneity, but I2 represents the percentage of observed variability that is attributed to the variability of true effects.

Discussion

- Almost all results and discussions are misinterpreted. For example: The sentence “The present study found positive association between including non-English studies in orthodontic SRs and SRs with small co-authorship or SRs led by an author affiliated in American institute, but the evidence was inconclusive.” is not accurate. There was no association. Please, revise all the paper and correct it. Also here: “We found that meta-analyses with non-English studies were more likely to yield statistical significance than meta-analyses that included only English studies. However, the evidence was inconclusive due to the small analysed sample”. There was no association. Moreover, for meta-analysis results the authors should present power calculation and discuss it. Probably the results were not associated due to a lack of power and this is related to the limitations that authors impose on papers: 2017 to 2021. So, this results in a limited number of meta-analyses with at least one non-English study. Considering that the authors had two aims, they could increase the period for studies to be included in the meta-analysis to provide an “n” of studies adequate.

- The limitations of the study must be deeply discussed.

Figure 3 has a low quality. Please add a high-quality size in Figure 3.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: In general terms, I found the article's proposal interesting, innovative, and well-executed by the authors. Language Bias should be a source of concern within evidence synthesis, given that other studies have already shown how geographical differences can impact submission and publication processes mainly in high-impact journals. However, I believe that some further clarification is needed.

Introduction:

Paragraphs 3 and 4 (page 3) present studies that have addressed Language Bias in some way but the definition of Language Bias is unclear in the text. I suggest including a brief definition of Language Bias and the context of English as the dominant scientific spoken/written language in Dentistry.

Methods:

In the Data Collection Process subsection (page 4): Were there agreement measurements between two reviewers in the pilot assessment of SR?

Just another small note on the description of the variables collected (page 4 - Data Collection Process subsection): How was the continent variable collected and categorized? Did you consider the affiliation of the first author?

In the Statistical Analysis and Data Synthesis subsection (page 5): In the multivariable binary logistic regression, were all the variables of interest included in the final multivariable regression model? Was there any specific way of selecting variables for the final model? Also, the level of significance (p-value >0.05) could be added here instead in the results section.

Results:

If possible, present the list of the included systematic reviews as supplementary material to accomplish with the data availability statement.

Discussion:

In the sentence: “The present study found positive association between including non-English studies in orthodontic SRs and SRs with small co-authorship or SRs led by an author affiliated with an American institute, but the evidence was inconclusive.” (Page 10, 4th paragraph): I believe that an improvement could be made to the wording, considering that "positive association" could be interpreted as a p<0.05, which was not the case in the study. I suggest changing the wording to "a relationship between ..." or some similar term at the authors' discretion.

Regarding limitations and strength subsection: In the metanalysis, could the effect measures found be affected by the heterogeneity of outcomes considered as the main outcome? Could this be considered a limitation of this study?

Reviewer #2: Dear Editor Luiz Alexandre Chisini,

Thank you for the opportunity to review this article.

I hope you find my feedback valuable.

Major Review

This article aims to assess language bias in systematic reviews published in orthodontics;

however, it may incorporate other biases at various levels:

Searches were conducted only in two databases. While I understand that the majority

of relevant literature might be present in these databases, this choice needs a more

robust justification. A sample calculation can be used to justify the number of articles that need to be included, potentially reducing the need for searches in additional databases.

There is also a significant date restriction for the search that needs clarification

The most critical point is that there has been no prior publication of the study protocol.

Minnor Review

Other minor points that also require attention:

It would be beneficial to make it clearer what the primary outcome of this study is.

From what I understand, the analysis considered studies that included both non-English language articles and those that did not. In the results section, the authors report that there were no explicit language restrictions ( = 135, 80%) or explicit exclusion of non-English studies ( = 170, 98%) in the majority of systematic reviews. However, there is a possibility that some systematic reviews may include studies in other languages, even if these studies do not exist and, consequently, are not found. How would they be considered in the analysis? Perhaps, this potential scenario could be addressed in the discussion.

In the results section, when indicating the p-value, please provide the exact number rather than using p-value > 0.05. If the p-value is very small, describe it as p-value < 0.0001.

I also suggest including the "numerical" values in the results presented in the abstract.

Please define what would constitute an inconclusive association. Was there no

association? What is the power of the test used to prevent Type 1 error?

In the discussion, the publication bias can be explored. As the authors themselves point

out, studies in other languages may exhibit lower methodological quality.

Additionally, it is essential to provide the search strategy used for the Cochrane Library

database.

The theme of this study is very interesting. With some corrections to strengthen the methodological robustness, there is a good potential for publication.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Jaqueline Barbieri Machado

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2024 Apr 1;19(4):e0300881. doi: 10.1371/journal.pone.0300881.r002

Author response to Decision Letter 0

5 Dec 2023

Dear editor Luiz Alexandre Chisini,

We would like to thank you and the reviewers for the comments and suggestions that we think will improve our manuscript.

Please find below responses and actions taken. In the revised manuscript we highlight amended sections.

PONE-D-23-28166

Language Bias in Orthodontic Systematic Reviews: A Meta-epidemiological Study

PLOS ONE

We look forward to receiving your revised manuscript.

Kind regards,

Luiz Alexandre Chisini, Ph.D

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

Authors’ response

We have amended the manuscript accordingly.

2. We note that you have referenced (Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey) which has currently not yet been accepted for publication. Please remove this from your References and amend this to state in the body of your manuscript: (Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey [Unpublished]”) as detailed online in our guide for authors

http://journals.plos.org/plosone/s/submission-guidelines#loc-reference-style.

Authors’ response

We have checked the reference and it indicates a published study in BMC Medical research Methodology Journal in 2017 (https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-017-0347-z). if the editor think that we should refer it as unpublished we will do that.

3. Please include captions for your Supporting Information files at the end of your manuscript, and update any in-text citations to match accordingly. Please see our Supporting Information guidelines for more information: http://journals.plos.org/plosone/s/supporting-information.

Authors’ response

We have amended the manuscript accordingly.

Additional Editor Comments:

The article's proposal is interesting and innovative; however, the authors need to clarify some points of the methods and need to revise the interpretations, that are not accurate.

Abstract:

Authors’ response

There was no statistically significant association except for the language restriction. The abstract has been amended accordingly:

‘The association between SRs characteristics and including non-English studies was not statistically significant except for the restriction on language: the odds of including non-English studies reduced by 89% in SRs with a language restriction (OR: 0.11, 95%CI: 0.01 0.55, P<0.004)’

- In the sentence: “The evidence was inconclusive due to substantial statistical heterogeneity using a random-effects model, but it was spuriously conclusive with a fixed-effect model”. Inconclusive is not the best way to interpretate the p-value. be only discussed in the discussion. Please revise all the paper and the interpretations of this result. Moreover, fixed-model is only a sensitivity analysis and should be only discussed in the discussion.

Authors’ response

The fixed effect model was removed from abstract, and the abstract has been amended accordingly:

‘As such, the evidence of larger meta-analysis results with non-English studies is inconclusive.’

Authors’ response

We have added the following clarity to the results:

‘Out of the sample, only fourteen meta-analyses were included in the meta-epidemiological analysis’

Methods:

- Why did de authors not register a protocol? This introduces an important bias in the study

Authors’ response

There was a prior protocol to ensure a well-conducted study. Unfortunately, the authors did not register it on PROSPERO due to the constraints imposed by the COVID period and the simultaneous need for prompt initiation of the study. We have included this point in the study limitations:

‘The prior protocol was not registered due to the constraints imposed by the COVID period, which may introduce some bias. However, the authors diligently adhered to their protocol and rigorous guidelines.’

- Authors report “[…] in orthodontic journals with the highest impact factor (2021):”. Was there a cut-off? The authors need to clarify the criteria for each covariable used in the study.

Authors’ response

The cut-off was the five leading orthodontic journals. We have added the impact factor of these journals to the methods:

‘American Journal of Orthodontic and Dentofacial Orthopedics (3.6), European Journal of Orthodontics (4.3), Progress in Orthodontics (3.7), Angle Orthodontist (3.4), and Orthodontics & Craniofacial Research (2.8).’

- Why only two databases were searched?

Authors’ response

This was discussed between the authors before conducting the search due to its importance. Our understanding was to include the largest number of published systematic reviews in these journals for the last five years. The search in PubMed was undertaken to retrieve systematic reviews published in the leading orthodontic journals which is indexed in PubMed. While Oral health group was searched to retrieve the relevant Cochrane reviews.

We added the last two sentences of the above justification to the Methods.

Authors’ response

The authors ensured the independence of the data between the included meta-analyses by including unique data in each meta-analysis. If there was a violation of data structure by including correlated data in the same meta-analysis, the authors calculated the correlation and adjusted the standard error according to the Cochrane handbook; Chapter 23. Details were provided in the supplementary materials.

Authors’ response

This paragraph was amended according to the editor's comment as follows:

‘One outcome was selected from each SR that included at least one non-English study to investigate the impact of the non-English language on the meta-analysis results. Specifically, we considered the following ‘algorithm’: if the meta-analysis of primary outcome included at least three studies with at least one being non-English, this meta-analysis was included in our collection. If more meta-analyses were eligible, we opted for the first one addressing the primary outcome. If the meta-analysis of the primary outcome was not eligible, the meta-analysis for secondary outcomes was checked.’

- Authors mention that did a meta-regression ”Initially, a random-effects meta-regression was

conducted to estimate the difference in standardised”, However, I didn’t find this in the paper. Please, revise it.

Authors’ response

This was explained in the supplementary materials due to the limited word counts as follows:

‘A two-stage approach was performed to analyse the meta-epidemiological data and estimate the average bias attributed to non-English studies. The meta-epidemiological data comprised meta-analyses on the primary outcome (one from each eligible SR) that included at least three studies, with at least one being non-English. The standardised mean difference (SMD) was the effect measure since all selected meta-analyses referred to a continuous primary outcome. Initially, a random-effects meta-regression with language type (non-English versus English) as the covariate was conducted in each meta-analysis. The intercept referred to the summary SMD in English studies, and the slope measured the summary difference in SMD (ΔSMD) between non-English and English studies. This model allowed the between-study variance to be estimated separately for each subgroup with at least two studies using the restricted maximum likelihood estimator for heterogeneity. Then, a random-effects meta-analysis was performed to combine the ΔSMDs across the meta-analyses. A negative ΔSMD would indicate that non-English studies overestimated SMD.’

Results:

Authors’ response

We have found that citing erratum and withdrawn articles is difficult, so we have added a supplementary table with the details of these articles.

- Please, highlight that only 14 studies were included as meta-analyses to avoid misinterpretation by the readers.

Authors’ response

The text was amended according to the editor’s comment:

‘Out of the sample, only fourteen meta-analyses comprised the dataset of the meta-epidemiological study.’

Authors’ response

We have revised these statements throughout the paper and all variables related to statistical significance were amended. We kept only related words to the discussion. In summary, when p-value > 0.05; we concluded the association to be statistically non-significant; otherwise, the association was statistically significance.

- Is from which analysis the results “the odds of including a non-English study were reduced by 89% (95% CI: 45% to 99%)”? Because I didn’t find these results in the tables.

Authors’ response

We are sorry for this confusion. The value 89% came from subtracting an OR of 0.11 from 1, and the same approach was used to obtain the percentages in the bounds of the 95% CI of the OR. We have added these values between brackets in the proper place:

‘the odds of including a non-English study were lower by 89% (OR: 0.11, 95% CI: 0.01 0.55, P<0.004) in SRs posing a language restriction.’

- In the sentence “80% provided a conclusive summary effect estimate (p-value < 0.05)”, please, change “conclusive” to “association”

Authors’ response

This was amended accordingly.

- Did the authors include 14 or 20 studies in the meta-analysis evaluation? Because sometimes the authors' reporting is 20 and sometimes 14 In the text.

Authors’ response

20 systematic reviews conducted meta-analysis and included at least one non-English study. However, only 14 SRs were included in the meta-epidemiological analysis.

we have added table3 to clarify the association between the statistical significance and the inclusion of non-English studies. Also, we have added the following text under ‘Examining the influence of non-English studies on summary results’:

‘Out of the sample, only fourteen meta-analyses comprised the dataset of the meta-epidemiological study’

Authors’ response

We used the restricted maximum likelihood estimator to the between-study heterogeneity parameter, τ2, to estimate the absolute statistical heterogeneity. We also followed the Spiegelhalter et al. to classify the estimated absolute statistical heterogeneity as low, reasonable, fairly high and fairly extreme and this information was mentioned in details in the supplementary materials (Figure S2 in Supplementary Material).

Reference

Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian approaches to clinical trials and health-care evaluation. John Wiley and Sons, Chichester, 2004.

Discussion

Authors’ response

We have not revised the first indicated sentence as follows: ‘The present study found a positive, though, statistically non-significant association between including non-English studies in orthodontic SRs and SRs with small co-authorship or SRs led by an author affiliated in American institute.’

We have also amended the second indicated sentence as follows: ‘We found that meta-analyses with non-English studies were more likely to yield statistical significance than meta-analyses that included only English studies. However, this association was not statistically significant due to the small analyzed sample, which may have led to insufficient power to detect the difference.’

We concur with the Editor about the power issues, associated with the small meta-analyses, which may have led to statistically non-significant associations. However, any post-hoc calculations of power comprise a ‘self-fulfilling prophecy’ because ‘‘post-hoc power is misinterpreted as inadequate power for trials with no statistically significant results, and it does not provide any extra information in the analysis. [...] post-hoc power is a self-fulfilling prophecy that falsely justifies any negative result as a product of a small sample size. Power is defined a priori to determine the sample size needed to estimate a certain effect with a certain type I error. [...] We urge researchers to [...] resort to the vast amount of literature and regulatory guidelines explaining the reasons for avoiding such practice’’; quoted from the article of Christogiannis et al.. As such, we were afraid of justifying false results.

Christogiannis C, Nikolakopoulos S, Pandis N, Mavridis D. The self-fulfilling prophecy of post-hoc power calculations. Am J Orthod Dentofacial Orthop. 2022 Feb;161(2):315-317.

Regarding the number of meta-analyses with at least one non-English study, only 20 studies from the systematic reviews (out of 45) with non-English language included results from these non-English studies in the meta-analysis. Therefore, even if we extend the period by one or two years, it will only marginally increase the number of these meta-analyses, which is unlikely to significantly impact the findings. The primary objective of this study was to illuminate this issue in orthodontic systematic reviews rather than providing an absolute assessment.

- The limitations of the study must be deeply discussed.

Authors’ response

We have expanded the limitation section to discuss the small included sample and the protocol registration as follows:

‘A wider search is unlikely to change our conclusions on the importance of including non-English studies. However, it may turn statistically non-significant results into statistically significant, particularly when more meta-analyses with non-English language are included in the meta-epidemiological study.’

‘The authors were concerned about the unaddressed correlation that may arise from including more than one meta-analysis from the same SR in the meta epidemiological analysis. As such, only one meta-analysis with non-English study was involved. The prior protocol was not registered due to the constraints imposed by the COVID period, which may introduce some bias. However, the authors diligently adhered to their protocol and rigorous guidelines.’

Figure 3 has a low quality. Please add a high-quality size in Figure 3.

Authors’ response

This was added accordingly.

Review Comments to the Author

Authors response: We would like to thank the reviewer for the valuable comments that we think will improve our manuscript. In the revised version, we are making efforts to incorporate additional clarification.

Introduction:

Authors response: More clarification regarding the language bias detention was added to the introduction according to the reviewer’s comment:

‘The English language is considered the dominant language in research[1] including dentistry. While publications from languages other than English is usually considered as of secondary importance. A previous study [2] found that German authors are more likely to publish trials in English when results are statistically significant, increasing the risk of language bias. Likewise, authors from less developed countries tend to publish more positive results than negative results[3]. Language bias may result from publishing significant findings in English language more than other languages[4]. Subsequently, results from only English language studies could provide a biased assessment of the topic.’

Methods:

In the Data Collection Process subsection (page 4): Were there agreement measurements between two reviewers in the pilot assessment of SR?

Authors response: Yes, the pilot data extraction was performed and the conflicts were resolved through a discussion with a third authors until the authors reached 100% agreement. We have added a line to clarify this process:

‘After reaching 100% agreement, the same two authors extracted the data and a third author (SM) cross-checked the collected data.’

Authors response: Yes, the affiliation of the first author was considered. This was added to the data extraction section:

‘The following characteristics for each SR were extracted: the number of authors, continent of the first author,’

Authors response: We did not perform any variable selection; instead, we considered all variables in the multivariable regression model, which is considered optimal statistical practice [1], since all included variables were prespecified (before seeing the data) for being relevant and important to the aims of the study.

We added the following information in the Methods under ‘Multivariable binary logistic regression’:

‘All characteristics were included in the model simultaneously, and no variable selection approach (e.g., stepwise selection) was performed.’

About the significance level, since we report a 95% confidence interval, the significance level is at 5%. We added the following information in that section:

• For the OR results: ‘We concluded a statistically significant association when the 95% CI did not include an OR of 1 (value of no association), which coincides with a p-value below 5% (the significance level); otherwise, the association was statistically non-significant, which coincides with a p-value at least 5%.’

• For the ΔSMD results: ‘We concluded a statistically significant ΔSMD when the 95% CI did not include the value 0, which coincides with a p-value below 5% (the significance level); otherwise, the result was statistically non-significant, which coincides with a p-value at least 5% ‘

[1] Harrell, Frank E., Jr. 2016. Regression Modeling Strategies. Springer Series in Statistics. Cham, Switzerland: Springer International Publishing.

Results:

If possible, present the list of the included systematic reviews as supplementary material to accomplish with the data availability statement.

Authors response: The dataset was shared on open repository:

https://doi.org/10.5281/zenodo.10201400

Discussion:

Authors response: this sentence was amended according to the editors’ and reviewer’s comment, as follows:

‘The present study found a positive, though, statistically non-significant association between including non-English studies in orthodontic SRs and SRs with small co-authorship or SRs led by an author affiliated in American institute.’

We have included the term ‘statistically non-significant association’ to avoid a misinterpretation like ‘positive association" could be interpreted as a p<0.05’. Whether an association (measured using an OR) is positive or negative does *not* dependent on whether a p-value is below or above a significance threshold; it dependents on whether the OR is above or below 1 (value of no association), respectively. Whether an association (which is another term for relationship) is statistically significant (conclusive evidence) or statistically non-significant (inconclusive evidence) dependents on whether a p-value is below or above a significance threshold, respectively.

Authors response: The limitations was amended. According to our understanding, the reviewer is asking for the effect of heterogeneity in the included meta-analysis? We have used the standardized mean difference due to the variation in our sample to minimize these effects. However, the statical heterogeneity was in the individual meta-analyses and not in our meta epidemiological analysis.

Also, we have reported the following: Most meta-analyses were associated with fairly high or extreme statistical heterogeneity, regardless of language restriction (S2 Figure). Restricting inclusion to English studies increased statistical heterogeneity in almost half meta-analyses compared to including all studies regardless of language (S2 Figure).

If the reviewer can provide more details, we are happy to follow his/her guidance.

Reviewer #2: Dear Editor Luiz Alexandre Chisini,

Thank you for the opportunity to review this article.

I hope you find my feedback valuable.

Major Review

This article aims to assess language bias in systematic reviews published in orthodontics;

however, it may incorporate other biases at various levels:

Authors response: We would like to thank the reviewer for the valuable comments that we think will improve our manuscript.

Searches were conducted only in two databases. While I understand that the majority

of relevant literature might be present in these databases, this choice needs a more

robust justification. A sample calculation can be used to justify the number of articles that need to be included, potentially reducing the need for searches in additional databases.

There is also a significant date restriction for the search that needs clarification

Authors response: We agree with the reviewer regarding these concerns. This was discussed between the authors before conducting the search due to its importance. Our understanding was to include the largest number of published systematic reviews in the five leading orthodontic journals for the last five years. The search in PubMed was undertaken to retrieve systematic reviews published in these journals which is indexed in PubMed. While Oral health group was searched to retrieve the relevant Cochrane reviews.

Meta epidemiological study aim to investigate whether the characteristic of interest influences the treatment effect[5]. It a common practice in meta-epidemiological research to focus on specific journals in a specific period, and this is always incorporated in the aim of the study. For instance, these meta epidemiological studies[6-8] were published in a prestigious orthodontic journal, and this study[9] was published in BMJ with one year search. Likewise, published meta-epidemiological studies [10, 11] in journal of clinical epidemiology have a restricted search.

We have amended the following information in the Methods under Search and study selection: One author (SM) performed the search of PubMed using text words and medical subject headings to retrieve systematic reviews published in the leading orthodontic journals indexed in PubMed (Table1). All relevant Cochrane orthodontic reviews within the same period were also retrieved by another author (BD) through the Cochrane Oral Health Group.

The most critical point is that there has been no prior publication of the study protocol.

Authors’ response

We have added the following information in the limitations:

Minnor Review

Other minor points that also require attention:

It would be beneficial to make it clearer what the primary outcome of this study is.

Authors response A clarification was added to the methods according to the reviewer’s comment. “Specifically, we considered the following ‘algorithm’: if the meta-analysis of primary outcome included at least three studies with at least one being non-English, this meta-analysis was included in our collection. If more meta-analyses were eligible, we opted for the first one addressing the primary outcome. If the meta-analysis of the primary outcome was not eligible, the meta-analysis for secondary outcomes was checked.”

Authors response: in the rare case, the authors may restrict their search to English studies to minimize the results of the search that may facilitate the screening of the potential records. However, some authors perform a manual search to handle the gray literature and unregistered studies as well as studies through google scholar search. This may result in a non-English study which was may happen in one[12] of our included sample. This was added to the discussion:

‘In this regard, manual search, particularly in google scholar may result in non-English studies even if the search was restricted in the other bibliographies. This was evident in one included SR [25] with additional manual search, although the search was restricted in the searching bibliographies. ’

In the results section, when indicating the p-value, please provide the exact number rather than using p-value > 0.05. If the p-value is very small, describe it as p-value < 0.0001.

Authors response:

A p-value < 0.05 coincides with a 95% confidence interval that does not include the value of no association (i.e., 1 for OR, and 0 for ΔSMD), and a p-value at least 5% coincides with a 95% confidence interval that includes the value of no association. Therefore, in the Results, we emphasize on the 95% confidence intervals for conveying more information (about the range of values and the statistical significance) for interpretation than a p-value (it only informs about statistical significance).

We have also included the following information in the Methods:

I also suggest including the "numerical" values in the results presented in the abstract.

Authors response: this has been amended accordingly.

Please define what would constitute an inconclusive association. Was there no

association? What is the power of the test used to prevent Type 1 error?

Authors response: We have replaced ‘conclusive evidence’ with ‘statistically significant association’ and ‘weak or inconclusive evidence’ with ‘statistically non-significant association’.

We understand that the reviewer is interested in the power calculation, which is a post-hoc calculation in this case. However, any post-hoc calculations of power comprise a ‘self-fulfilling prophecy’ because ‘‘post-hoc power is misinterpreted as inadequate power for trials with nonstatistically significant results, and it does not provide any extra information in the analysis. [...] Post-hoc power is a self-fulfilling prophecy that falsely justifies any negative result as a product of a small sample size. Power is defined a priori to determine the sample size needed to estimate a certain effect with a certain type I error. [...] We urge researchers to [...] resort to the vast amount of literature and regulatory guidelines explaining the reasons for avoiding such practice’’; quoted from the article of Christogiannis et al. and we concur with the authors of the article.

Christogiannis C, Nikolakopoulos S, Pandis N, Mavridis D. The self-fulfilling prophecy of post-hoc power calculations. Am J Orthod Dentofacial Orthop. 2022 Feb;161(2):315-317.

In the discussion, the publication bias can be explored. As the authors themselves point

out, studies in other languages may exhibit lower methodological quality.

Authors response: Investigating publication bias in orthodontic is challenging due to the low number of included studies (usually <10 studies)[13], and it will not differentiate between the included English and non-English studies. As this make the interpretation of publication bias very difficult, we have recommended a sensitivity analysis to do so. The following information can be found in the Discussion

‘Moreover, different studies [24, 31-33] assessed the quality of the English and non-English studies, and found a higher risk of bias in non-English studies due to suboptimal randomization, insufficient reporting of the blinding, and incomplete data. As such, non-English studies may be removed from the meta-analysis in the context of a sensitivity analysis to inspect the robustness of the meta-analysis results.’

Additionally, it is essential to provide the search strategy used for the Cochrane Library

database.

Authors response: The search was undertaken manually through the Cochrane Oral Health Group.

The theme of this study is very interesting. With some corrections to strengthen the methodological robustness, there is a good potential for publication.

Authors response: We would like to thank the reviewer again for his valuable comments that improved our understanding for the topic and our study limitations.

References

1. FERGUSON G, PÉREZ-LLANTADA C, PLO R. English as an international language of scientific publication: a study of attitudes. World Englishes. 2011;30(1):41-59. doi: https://doi.org/10.1111/j.1467-971X.2010.01656.x.

2. Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. The Lancet. 1997;350(9074):326-9. doi: 10.1016/s0140-6736(97)02419-7.

3. Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JP. Comparative effect sizes in randomised trials from less developed and more developed countries: meta-epidemiological assessment. BMJ. 2013;346:f707. Epub 20130212. doi: 10.1136/bmj.f707. PubMed PMID: 23403829; PubMed Central PMCID: PMCPMC3570069.

4. J B, EA S, C H. Langauge bias. In: Catalogue Of Bias 2017. Available from: https://www.catalogueofbias.org/biases/language-bias.

5. Kataoka Y, Banno M, Tsujimoto Y, Furukawa TA. "Meta-epidemiological study" is a study in which the unit of analysis is a study, not a patient; response to Puljak et al. J Clin Epidemiol. 2023;154:219-20. Epub 20221209. doi: 10.1016/j.jclinepi.2022.12.002. PubMed PMID: 36503003.

6. Tatas Z, Koutsiouroumpa O, Seehra J, Mavridis D, Pandis N. Do pooled estimates from orthodontic meta-analyses change depending on the meta-analysis approach? A meta-epidemiological study. Eur J Orthod. 2023. Epub 20230712. doi: 10.1093/ejo/cjad031. PubMed PMID: 37435902.

7. Mheissen S, Khan H, Seehra J, Pandis N. Are longitudinal randomised controlled oral health trials properly analysed? A meta-epidemiological study. Journal of dentistry. 2022;124:104182. Epub 20220609. doi: 10.1016/j.jdent.2022.104182. PubMed PMID: 35691454.

8. Mheissen S, Khan H, Almuzian M, Alzoubi EE, Pandis N. Do longitudinal orthodontic trials use appropriate statistical analyses? A meta-epidemiological study. Eur J Orthod. 2021. Epub 2021/09/26. doi: 10.1093/ejo/cjab069. PubMed PMID: 34561710.

9. Moustgaard H, Clayton GL, Jones HE, Boutron I, Jorgensen L, Laursen DRT, et al. Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study. BMJ. 2020;368:l6802. Epub 2020/01/23. doi: 10.1136/bmj.l6802. PubMed PMID: 31964641; PubMed Central PMCID: PMCPMC7190062 at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

10. Tsujimoto Y, Tsujimoto H, Kataoka Y, Kimachi M, Shimizu S, Ikenoue T, et al. Majority of systematic reviews published in high-impact journals neglected to register the protocols: a meta-epidemiological study. J Clin Epidemiol. 2017;84:54-60. Epub 20170227. doi: 10.1016/j.jclinepi.2017.02.008. PubMed PMID: 28242481.

11. Smail-Faugeron V, Tan A, Caille A, Yordanov Y, Hajage D, Tubach F, et al. Meta-analyses frequently include old trials that are associated with a larger intervention effect: a meta-epidemiological study. Journal of Clinical Epidemiology. 2022;145:144-53. doi: https://doi.org/10.1016/j.jclinepi.2022.01.023.

12. Elsten E, Caron C, Dunaway DJ, Padwa BL, Forrest C, Koudstaal MJ. Dental anomalies in craniofacial microsomia: A systematic review. Orthod Craniofac Res. 2020;23(1):16-26. Epub 20191028. doi: 10.1111/ocr.12351. PubMed PMID: 31608577; PubMed Central PMCID: PMCPMC7003932.

13. Koletsi D, Fleming PS, Eliades T, Pandis N. The evidence from systematic reviews and meta-analyses published in orthodontic literature. Where do we stand? Eur J Orthod. 2015;37(6):603-9. Epub 2015/02/11. doi: 10.1093/ejo/cju087. PubMed PMID: 25667037.

Attachment

Submitted filename: PLOS nonenglish language rebuttal letter_LS.docx

pone.0300881.s005.docx^{(62.3KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0300881.r003

Decision Letter 1

Luiz Alexandre Chisini

7 Feb 2024

PONE-D-23-28166R1Language Bias in Orthodontic Systematic Reviews: A Meta-epidemiological StudyPLOS ONE

Dear Dr. Mheissen,

Please submit your revised manuscript by Mar 23 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Luiz Alexandre Chisini, Ph.D

Academic Editor

PLOS ONE

Additional Editor Comments:

I have still concerns about the paper. The paper's objective is very interesting. However, the interpretation of the results is still incorrect. The authors remain doing affirmations that cannot be maintained by the statistics. So, the paper spin is very high and can lead the authors to misinterpretation.

- When the authors mention “174 SRs were eligible for inclusion in this study” it is not possible to undertint if all 174 were included or not for the revision. They are eligible for inclusion, but could not be included. So, it is better to readers understand the number of “included” instead of “eligible”. This is confusing in the abstract and in the results. Please, Make clear the number of articles included in the review and meta-analysis at the beginning of the results sections (in the abstract and results)

- Please use the exact number of p-value.

- I have also concerns about the sentence: “As such, the evidence of the overestimation of meta-analysis results with non-English studies is inconclusive”. The meta-analysis with random models (which is the right model for studies with methodological variation) shows no difference. So, could be “inconclusive” only isn’t power enough, but then there are problems with the sample size. On the other hand, I believe that the conclusion is there was no evidence of significant overestimation.

- I dindnt found the result for this sentence: “The univariate binary logistic regression indicated that the odds of statistical significance in the summary effect estimate increased by 220% (OR: 3.20) in systematic reviews with non-English studies than in systematic reviews with only English studies. However, the evidence was weak because the 95% CI (0.57, 18.92) was substantially wide and included the null value”. Also, the authors cannot affirm that increase, because the 95% CI doesn’t allow this. Please, revise affirmations that cannot be supported by statistics.

- Here, another misinformation: “The present study found a positive, though, statistically non-significant association between including non-English studies in orthodontic SRs and SRs with small coauthorship or SRs led by an author affiliated in American institute”. The authors cannot affirm that the is a “positive” but non-significant association. Or there is an association or there is not. Please, revise.

- In the sentence “Our meta-epidemiological analysis demonstrated a larger summary effect size from including non-English studies, though the evidence was inconclusive due to material statistical heterogeneity”. Here again, they are not “inconclusive”.

- This sentence “A wider search is unlikely to change our conclusions on the importance of including non-English studies.” Is a supposition and cannot be maintained with data. So, I recommend the authors to avoid this type of affirmation.

- Both in the conclusion as the abstract conclusion, the authors cannot support this affirmation “Language restriction seems to have a non-negligible impact on the results of orthodontic SRs.” There was no change in the results according to the results. This interpretation leads to a spin on the study.

In Table 2, please, standardize the reference category as the first row

In table 3, please, provide the p-value

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: I Don't Know

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Reviewer #1: I would like to thank you for the opportunity to review this article again, as well as for the authors' thoughtful response. The authors have made considerable improvements to the paper that have enhanced the way the findings are communicated. Below are some minor corrections that can be made to ensure the best version of this manuscript.

Abstract:

In the "Eligibility Criteria" section, it is clearly stated that the two main criteria for selecting SRs for inclusion were (1) publication between 2017 and 2021 and (2) being published in one of the five highest-impact journals indexed in PubMed, in addition to Cochrane reviews from the same period. In this way, I believe that in the "Data Source" section of the Abstract, the wording could be improved by inserting separately that the SRs were retrieved from five high-impact journals indexed in PubMed. After this, could be inserted that searches were carried out in the Cochrane database, also considering the publication period (2017-2021).

Methods:

In the response letter (page 8), the authors stated that in the Statistical Analysis and Data Synthesis subsection, two sentences had been included to address the issue of statistical significance interpretation in different analyses carried out. However, the second of these, copied below, was not present in the main text. Please revise the insertion of the information.

“•For the ΔSMD results: ‘We concluded a statistically significant ΔSMD when the 95% CI did not include the value 0, which coincides with a p-value below 5% (the significance level); otherwise, the result was statistically non-significant, which coincides with a p-value at least 5%”

Results:

Please verify Table S1: Although the Systematic review selection subsection mentions 11 excluded studies with reasons, only 10 titles are shown in Table S1.

Reviewer #2: Most of my suggestions were accepted. The ones that couldn't be altered were justified accordingly. I appreciate the opportunity to review this study and recommend it for publication. Additionally, I would like to note that the authors have introduced an interesting and underexplored topic in the scientific literature.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Letícia Regina Morello Sartori

Reviewer #2: No

**********

PLoS One. 2024 Apr 1;19(4):e0300881. doi: 10.1371/journal.pone.0300881.r004

Author response to Decision Letter 1

8 Feb 2024

Dear editor Luiz Alexandre Chisini,

We would like to thank you and the reviewers for the comments and suggestions that we think will improve our manuscript.

Please find below responses and actions taken. In the revised manuscript we highlight amended sections.

Additional Editor Comments:

Authors’ response: the results and the abstract were amended according to the Editors’ comments, as follows:

Abstract (Results): ‘174 SRs were included in this study.’

Results (Systematic review selection): ‘After full text reading of one hundred eighty-five SRs, 174 SRs were included in the present study.’

- Please use the exact number of p-value.

Authors’ response: We have now added the p-value for the random-effects and fixed-effect model in the Results subsection ‘Examining the influence of non-English studies on summary results’.

Authors’ response: We amended the indicated sentence as follows: ‘As such, the overestimation of meta-analysis results by including non-English studies was statistically non-significant’.

Authors’ response: To avoid confusing the readers, we amended the indicated sentence as follows:

‘The univariate binary logistic regression indicated that the odds of statistical significance in the summary effect estimate was 3.20 times larger in systematic reviews with non-English studies than in systematic reviews with only English studies. However, the association was statistically non-significant (OR:3.20, 95%CI: 0.57, 18.92, P=0.18).’

Authors’ response: We removed ‘positive, though,’ from the indicated sentence.

Authors’ response: We replaced ‘the evidence was inconclusive’ with ‘this was not statistically significant’.

Authors’ response: We amended the indicated sentence as follows:

‘A wider search may have some impact on the importance of including non-English studies.’

Authors’ response: Professors Doug Altman and Martin Bland wrote a seminar commentary to raise awareness on how results from statistical tests should be interpreted. The article’s title is ‘Absence of evidence is not evidence of absence’ (1); namely, finding a p-value above the selected level of significance (e.g., 5%), and thus, failing to reject the null hypothesis does not mean that there is no difference in the compared groups. It just means that the difference is statistically non-significant, and hence, we cannot draw any firm conclusion in favour or against any of the compared groups due to lack of sufficient information (e.g., small sample size) or substantial variation in the measurements, or a combination of both. Taken from the abstract of Altman and Bland: “When statistical analysis of the study data finds a P value greater than 5%, it is convention to deem the assessed difference nonsignificant. Just because convention dictates that such study findings be termed nonsignificant, or negative, however, it does not necessarily follow that the study found nothing of clinical importance.” (1). And lastly, from the last paragraph of the article: “When we are told that “there is no evidence that A causes B” we should first ask whether absence of evidence means simply that there is no information at all. If there are data we should look for quantification of the association rather than just a P value. Where risks are small P values may well mislead: confidence intervals are likely to be wide, indicating considerable uncertainty. While we can never prove the absence of a relation, when necessary we should seek evidence against the link between A and B—for example, from case-control studies.” (1)

We amended the indicated sentence as follows: ‘Language restriction seems to have no statistically significant impact on the results of orthodontic SRs’

Reference

(1) Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995 Aug 19;311(7003):485. doi: 10.1136/bmj.311.7003.485.

In Table 2, please, standardize the reference category as the first row

In table 3, please, provide the p-value

Authors’ response: Tables were amended accordingly.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Authors response: We would like to thank the reviewer for the valuable comments that have improved our manuscript. In the revised version, we are making efforts to incorporate additional clarification.

Abstract:

Authors’ response: We have amended the Data source section in Abstract as follows:

‘SRs published in high-impact orthodontic journals between 2017 and 2021 were retrieved through an electronic search of PubMed. Additionally, Cochrane oral health group was searched in June 2022 for orthodontic systematic reviews published in the same period.’

Methods:

Authors’ response: Thank you for this observation. We have now added all P values to the text for better clarity. However, if the reviewer thinks that we should add this sentence, we will be happy to follow his/her guidance.

Results:

Please verify Table S1: Although the Systematic review selection subsection mentions 11 excluded studies with reasons, only 10 titles are shown in Table S1.

Authors’ response: Sorry for this mistake. This excluded study was added to the table.

Authors response: We would like to thank the reviewer for the valuable comments that have improved our manuscript.

Attachment

Submitted filename: Revision letter R2.docx

pone.0300881.s006.docx^{(19.8KB, docx)}

PLoS One. doi: 10.1371/journal.pone.0300881.r005

Decision Letter 2

Luiz Alexandre Chisini

7 Mar 2024

Language Bias in Orthodontic Systematic Reviews: A Meta-epidemiological Study

PONE-D-23-28166R2

Dear Dr. Samer Mheisse

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at http://www.editorialmanager.com/pone/ and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Luiz Alexandre Chisini, Ph.D

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #1: Yes

**********

6. Review Comments to the Author

Reviewer #1: Dear authors, my concerns about the article have been successfully solved.

Thank you very much for your careful review.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Letícia Sartori

**********

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Table. Studies excluded with the reason.

(DOCX)

pone.0300881.s001.docx^{(15.1KB, docx)}

S1 File. Technical notes.

(DOCX)

pone.0300881.s002.docx^{(35.6KB, docx)}

S1 Fig. Forest plots on the summary standardised mean difference and 95% confidence interval in 14 meta-analyses.

(DOCX)

pone.0300881.s003.docx^{(212.5KB, docx)}

S2 Fig. Line plot on the between-study standard deviation (x-axis) in 14 meta-analyses (y-axis).

(DOCX)

pone.0300881.s004.docx^{(200.2KB, docx)}

Attachment

Submitted filename: PLOS nonenglish language rebuttal letter_LS.docx

pone.0300881.s005.docx^{(62.3KB, docx)}

Attachment

Submitted filename: Revision letter R2.docx

pone.0300881.s006.docx^{(19.8KB, docx)}

Data Availability Statement

Data Availability: The data underlying this paper were shared on open repository: https://doi.org/10.5281/zenodo.10201400.

[pone.0300881.ref001] 1.Hooper EJ, Pandis N, Cobourne MT, Seehra J. Methodological quality and risk of bias in orthodontic systematic reviews using AMSTAR and ROBIS. Eur J Orthod. 2021;43(5):544–50. doi: 10.1093/ejo/cjaa074 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref002] 2.AlMubarak D, Pandis N, Cobourne MT, Seehra J. Reporting of the methodological quality of search strategies in orthodontic quantitative systematic reviews. Eur J Orthod. 2021;43(5):551–6. doi: 10.1093/ejo/cjaa078 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref003] 3.Seehra J, Stonehouse-Smith D, Pandis N. Prediction intervals reporting in orthodontic meta-analyses. Eur J Orthod. 2021;43(5):596–600. doi: 10.1093/ejo/cjab037 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0300881.ref004] 4.Koletsi D, Fleming PS, Michelaki I, Pandis N. Heterogeneity in Cochrane and non-Cochrane meta-analyses in orthodontics. Journal of dentistry. 2018;74:90–4. Epub 20180505. doi: 10.1016/j.jdent.2018.05.003 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref005] 5.Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Assessing small study effects and publication bias in orthodontic meta-analyses: a meta-epidemiological study. Clin Oral Investig. 2014;18(4):1031–44. Epub 2014/02/15. doi: 10.1007/s00784-014-1196-3 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref006] 6.Makou O, Eliades T, Koletsi D. Reporting, interpretation, and extrapolation issues (SPIN) in abstracts of orthodontic meta-analyses published from 2000 to 2020. European Journal of Orthodontics. 2021;43(5):567–75. doi: 10.1093/ejo/cjab009 [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref007] 7.Koletsi D, Fleming PS, Eliades T, Pandis N. The evidence from systematic reviews and meta-analyses published in orthodontic literature. Where do we stand? Eur J Orthod. 2015;37(6):603–9. Epub 2015/02/11. doi: 10.1093/ejo/cju087 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref008] 8.Higgins JPT TJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Second edition ed.

[pone.0300881.ref009] 9.Lefebvre C G J, Briscoe S, Featherstone R, Littlewood A, Marshall C, Metzendorf M-I, et al. Chapter 4: Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T. Searching for and selecting studies. In: Page MJ W V, editor. Cochrane Handbook for Systematic Reviews of Interventions version 63 (updated February 2022). UK: Cochrane; 2022. [Google Scholar]

[pone.0300881.ref010] 10.FERGUSON G, PÉREZ-LLANTADA C, PLO R. English as an international language of scientific publication: a study of attitudes. World Englishes. 2011;30(1):41–59. doi: 10.1111/j.1467-971X.2010.01656.x [DOI] [Google Scholar]

[pone.0300881.ref011] 11.Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. The Lancet. 1997;350(9074):326–9. doi: 10.1016/S0140-6736(97)02419-7 [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref012] 12.Panagiotou OA, Contopoulos-Ioannidis DG, Ioannidis JP. Comparative effect sizes in randomised trials from less developed and more developed countries: meta-epidemiological assessment. BMJ. 2013;346:f707. Epub 20130212. doi: 10.1136/bmj.f707 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0300881.ref013] 13.J B, EA S, C H. Langauge bias. In: Catalogue Of Bias 2017. https://www.catalogueofbias.org/biases/language-bias.

[pone.0300881.ref014] 14.Morrison A, Polisena J, Husereau D, Moulton K, Clark M, Fiander M, et al. The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int J Technol Assess Health Care. 2012;28(2):138–44. doi: 10.1017/S0266462312000086 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref015] 15.Dobrescu AI, Nussbaumer-Streit B, Klerings I, Wagner G, Persad E, Sommer I, et al. Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review. J Clin Epidemiol. 2021;137:209–17. Epub 20210430. doi: 10.1016/j.jclinepi.2021.04.012 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref016] 16.Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017;22(4):139–42. Epub 20170712. doi: 10.1136/ebmed-2017-110713 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0300881.ref017] 17.Firth D. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80(1):27–38. doi: 10.1093/biomet/80.1.27 [DOI] [Google Scholar]

[pone.0300881.ref018] 18.Heinze G, Schemper M. A solution to the problem of separation in logistic regression. Stat Med. 2002;21(16):2409–19. doi: 10.1002/sim.1047 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref019] 19.R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2022. https://www.R-project.org/.

[pone.0300881.ref020] 20.DD S, Whiting K C M, Lavery JA, Larmarange J. Reproducible summary tables with the gtsummary package. The R Journal. 2021;13:570–80. [Google Scholar]

[pone.0300881.ref021] 21.Viechtbauer W. Conducting Meta-Analyses in R with the metafor Package. Journal of Statistical Software. 2010;36(3):1–48. doi: 10.18637/jss.v036.i03 [DOI] [Google Scholar]

[pone.0300881.ref022] 22.Wickham H. ggplot2: Elegant Graphics for Data Analysis. New York: Springer-Verlag; 2016. [Google Scholar]

[pone.0300881.ref023] 23.Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64. Epub 20170419. doi: 10.1186/s12874-017-0347-z . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0300881.ref024] 24.Dechartres A, Atal I, Riveros C, Meerpohl J, Ravaud P. Association Between Publication Characteristics and Treatment Effect Estimates: A Meta-epidemiologic Study. Ann Intern Med. 2018;169(6):385–93. Epub 20180821. doi: 10.7326/M18-1517 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref025] 25.Elsten E, Caron C, Dunaway DJ, Padwa BL, Forrest C, Koudstaal MJ. Dental anomalies in craniofacial microsomia: A systematic review. Orthod Craniofac Res. 2020;23(1):16–26. Epub 20191028. doi: 10.1111/ocr.12351 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0300881.ref026] 26.Xia J, Wright J, Adams CE. Five large Chinese biomedical bibliographic databases: accessibility and coverage. Health Info Libr J. 2008;25(1):55–61. doi: 10.1111/j.1471-1842.2007.00734.x . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref027] 27.Sideri S, Papageorgiou SN, Eliades T. Registration in the international prospective register of systematic reviews (PROSPERO) of systematic review protocols was associated with increased review quality. J Clin Epidemiol. 2018;100:103–10. Epub 2018/01/18. doi: 10.1016/j.jclinepi.2018.01.003 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref028] 28.Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008. Epub 2017/09/25. doi: 10.1136/bmj.j4008 at http://www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0300881.ref029] 29.Pham B, Klassen TP, Lawson ML, Moher D. Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol. 2005;58(8):769–76. doi: 10.1016/j.jclinepi.2004.08.021 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref030] 30.Moher D, Pham B, Lawson ML, Klassen TP. The inclusion of reports of randomised trials published in languages other than English in systematic reviews. Health technology assessment (Winchester, England). 2003;7(41):1–90. doi: 10.3310/hta7410 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref031] 31.Hugues A, Di Marco J, Bonan I, Rode G, Cucherat M, Gueyffier F. Publication language and the estimate of treatment effects of physical therapy on balance and postural control after stroke in meta-analyses of randomised controlled trials. PloS one. 2020;15(3):e0229822. Epub 20200309. doi: 10.1371/journal.pone.0229822 . [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0300881.ref032] 32.Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol. 2002;31(1):115–23. doi: 10.1093/ije/31.1.115 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref033] 33.Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health technology assessment (Winchester, England). 2003;7(1):1–76. . [PubMed] [Google Scholar]

[pone.0300881.ref034] 34.Guo R, Zhang L, Hu M, Huang Y, Li W. Alveolar bone changes in maxillary and mandibular anterior teeth during orthodontic treatment: A systematic review and meta-analysis. Orthod Craniofac Res. 2021;24(2):165–79. Epub 20200916. doi: 10.1111/ocr.12421 . [DOI] [PubMed] [Google Scholar]

[pone.0300881.ref035] 35.Alhammadi MS, Halboub E, Fayed MS, Labib A, El-Saaidi C. Global distribution of malocclusion traits: A systematic review. Dental Press J Orthod. 2018;23(6):40.e1–.e10. doi: 10.1590/2177-6709.23.6.40.e1-10.onl . [DOI] [PMC free article] [PubMed] [Google Scholar]

PERMALINK

Language bias in orthodontic systematic reviews: A meta-epidemiological study

Samer Mheissen

Loukia M Spineli

Baraa Daraqel

Ahmad Saleem Alsafadi

Roles

Abstract

Background

Data source

Data collection and analysis

Results

Conclusion

Introduction

Materials and methods

Protocol and registration

Eligibility criteria

Search and study selection

Table 1. The search strategy in PubMed via medline.

Data collection process

Statistical analysis and data synthesis

Multivariable binary logistic regression

Meta-epidemiological analysis and sensitivity analysis

Results

Systematic review selection

Fig 1. PRISMA flow diagram for the included systematic reviews.

Characteristics of the systematic reviews

Table 2. Characteristics of systematic reviews with and without studies on non-English languages.

Association of including non-English studies with several characteristics

Frequency of non-English languages

Fig 2. Bar plots with the number of non-English studies for each non-English language and systematic review type.

Statistical significance and inclusion of non-English studies

Table 3. The association of statistical significance with the inclusion of non-English studies.

Examining the influence of non-English studies on summary results

Fig 3. Forest plot showing the difference in standardised mean difference (ΔSMD) between non-English and English studies using a set of meta-analysis.

Discussion

Evidence summary

Limitations and strength

Conclusion

Supporting information

Data Availability

Funding Statement

References

Decision Letter 0

Luiz Alexandre Chisini

Roles

Author response to Decision Letter 0

Decision Letter 1

Luiz Alexandre Chisini

Roles

Author response to Decision Letter 1

Decision Letter 2

Luiz Alexandre Chisini

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases