Skip to main content
NCBI home page
The BMJ logoLink to The BMJ
. 2025 Apr 23;389:e082068. doi: 10.1136/bmj-2024-082068

Investigating the impact of trial retractions on the healthcare evidence ecosystem (VITALITY Study I): retrospective cohort study

Chang Xu 1, Shiqi Fan 1, Yuan Tian 1, Fuchen Liu 2,✉, Luis Furuya-Kanamori 3, Justin Clark 4, Chao Zhang 5, Sheng Li 6, Lifeng Lin 7, Haitao Chu 8 ,9, Sheyu Li 10, Su Golder 11, Yoon Loke 12, Sunita Vohra 13, Paul Glasziou 4, Suhail A Doi 14, Hui Liu 1 ,2,✉, on behalf of the VITALITY Collaborative Research Network
PMCID: PMC12015725  PMID: 40268307

Abstract

Objective

To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem.

Design

Retrospective cohort study based on forward citation searching.

Data sources

Retraction Watch up to 5 November 2024.

Study selection

Randomised controlled trials in humans that were retracted for any reason.

Methods

Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching.

Results

The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents.

Conclusion

Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed.

Study registration

Open Science Framework (https://osf.io/7eazq/).

Introduction

In evidence based medicine, effective decision making relies on the establishment of a trustworthy evidence ecosystem.1 Evidence generation and evidence synthesis are key elements for the establishment of such an ecosystem, and they are key drivers to affect implementation of evidence based healthcare policy and practice.2 Clinicians and policy makers use evidence or synthesise evidence from existing research to create clinical practice guidelines or conduct health technology assessments, which further support healthcare practice. The degree to which evidence is valid and robust determines its value in promoting evidence based clinical guidelines and healthcare.

Randomised controlled trials serve as the main source of high quality data for evidence synthesis.3 4 The design of randomised controlled trials enables the minimisation of potential biases arising from methodological weaknesses such as confounding by indication.5 On this basis, synthesised evidence from randomised controlled trials is considered the highest level in the evidence pyramid, and such evidence has been widely adopted to inform clinical practice guidelines and health technology assessments.6 For this to hold, randomised controlled trials must report information transparently and honestly, without bias or distortion. However, this is not always the case, as an increasing number of retracted clinical trials have been documented owing to the questionable nature of their data and practices.7 8

The retraction of randomised controlled trials poses a severe threat to evidence based medicine, and when retracted trials are included as part of evidence synthesis, a question arises as to whether the evidence ecosystem is still trustworthy. The incorporation of “contaminated” evidence into clinical practice guidelines could lead to incorrect conclusions, mislead healthcare practice, increase the epistemic cost, and potentially cause harm to patients.9 10 To this end, investigating the impact of retracted trials and quantifying the extent of the potential contamination cascade on clinical practice guidelines is an urgent priority in the science of evidence synthesis. In this study, we aimed to answer the above question via a large scale investigation based on forward citation searching.

Methods

Design and settings

This study was based on forward citation searching. To start with, we identified a cohort of retracted randomised controlled trials from the Retraction Watch database (see below). We then used forward citation searching to identify systematic reviews and meta-analyses that had included these retracted trials. To quantify the influence of the retracted trials on the results of the systematic reviews and meta-analyses, we reanalysed the pooled data after excluding the retracted trials. Finally, we used a comprehensive literature search to identify relevant clinical practice guidelines that might be affected by evidence from contaminated systematic reviews. The conduct and analysis of this study were pre-specified by a pre-defined protocol on 13 April 2023, which can be accessed in the supplementary file or online via the Open Science Framework website (https://osf.io/7eazq/).11

Data sources and inclusion criteria

The database created for this study was constructed from three data sources: retracted trials, evidence synthesis research that included retracted trials, and clinical practice guidelines that used evidence from contaminated evidence synthesis research.

Retracted trials

We searched the Retraction Watch database for retracted publications up to 26 April 2023 under the agreements of the dataset owners and updated the search on 5 November 2024. We used two separate searches: retrieved records labelled as “Clinical Study” and retrieved records labelled as “Research Article” which limited the subjects of “(HSC) Health Sciences”. The search strategy was developed by two information specialists (SG and JC) after discussions with the lead author (CX), and the literature search was run by the lead author. Only articles with notification of “retraction” were considered. Two researchers (CX and YT) reviewed the titles and abstracts of the retracted publications via the Rayyan online application (https://www.rayyan.ai/) and then reviewed the full texts independently, identifying and including those featured as published randomised controlled trials in humans (not including preprints) and presented in English. We broadly selected studies for which the full text reported the design to be a randomised trial; we did not delve into the exact methods of sequence generation or allocation concealment. For secondary analyses that used data from already published randomised controlled trials, we considered only those in which the data were analysed as per the originally randomised groups. Any disagreements were resolved through discussion with another senior methodologist (LFK). The search strategy is presented in supplementary table S1.

Evidence synthesis research that included retracted trials

To identify evidence synthesis research that may have been affected by data from retracted trials, we used forward citation searching of each retracted trial via Google Scholar on 19 July 2023, with the last update from 21 November to 2 December 2024. In addition, we used forward citation searching via Scopus on 11 June 2024, updating it on 21 November 2024. All citations of retracted trials from Google Scholar and Scopus were reviewed by title and abstract, and all types of reviews (for example, narrative reviews, systematic reviews, rapid reviews, and umbrella reviews) were selected at this stage. The screening of the citations from Google Scholar was done by 13 well trained postgraduate or doctoral student team members (RZ, YZ, XY, SQF, YT, LJT, MYZ, LKKX, YT, RLZ, HRZ, RXW, and QNW). Screening of the citations from Scopus was done by using a filter of 21 keywords (supplementary file, page 14), according to a previous study.12 A test of the filter based on 2846 citations (limited to “article” and “review”) of the PREDIMED study13 showed that its sensitivity for detecting systematic reviews was 98.8% (supplementary table S2).

Further full text review of records obtained from Google Scholar in the first stage was done by two students (SQF and YT) from the same group to verify whether quantitative evidence synthesis was conducted in these reviews; systematic reviews (including preprints) with at least one pair-wise meta-analyses were identified for inclusion. Full text review of records obtained from Scopus was independently done by two senior methodologists (CX and CZ). To avoid duplication, we did not include scoping reviews, umbrella reviews, reanalysis of meta-analyses, or other types of reviews on reviews or meta-analyses. We did not consider conference abstracts as they did not contain sufficient information to allow for replication. We did not consider systematic reviews that had been retracted. We set restrictions by language for English only.

Clinical practice guidelines that used contaminated evidence

To identify potentially affected clinical practice guidelines, we collected the topics of systematic reviews and meta-analyses with conclusions that were substantially affected by retracted trials (see definition in the outcome section). The lead author (CX) searched Scopus and Google Scholar for titles of related systematic reviews up to 1 January 2025, with a manual check of the Guidelines International Network (GIN) (https://g-i-n.net/) and TRIP (https://www.tripdatabase.com/) databases based on the systematic review topics. We took the definition of clinical practice guidelines from Kataoka as “statements that include recommendations intended to optimize patient care. They are informed by systematic reviews of evidence and an assessment of the benefits and harms of alternative care options.”14 We did not make a distinction between evidence based and consensus based guidelines as suggested by Benjamin and colleagues.15 Instead, we considered any guideline documents (including position statements) that summarised existing evidence and formed clear recommendations to be clinical practice guidelines. We restricted clinical practice guidelines to those written in English and identified them by using the following keywords: “clinical practice guideline”, “clinical guideline”, “guideline”, “consensus”, “statements”, and “recommendations”. Initial screening of the guidelines by one researcher (CX) via the Rayyan application was based on title and abstract; this was followed by full text assessment by two researchers independently (CX and LFK), with any disagreements resolved through discussion with an expert in clinical practice guidelines (SYL).

Data collection

We extracted the following data from the retracted trials: Digital Object Identifier (DOI) number, total number of citations, title, journal of publication, date of publication, date of retraction, reasons for retraction (according to Retraction Watch), number of authors, geographical region of the corresponding author, trial registration information, data sharing statement, and source of funding.

We extracted the following information from systematic reviews and meta-analyses that included retracted trials in their evidence synthesis: DOI number, title, year of publication, synthesis approaches (quantitative versus qualitative), type of meta-analysis (for example, pair-wise, network, incidence), metadata of each meta-analysis and subgroups, methods for the data synthesis (for example, fixed effect model), effect estimate type (for example, odds ratio), software used for the synthesis, location of the meta-analyses with retracted trials, number of trials within each meta-analysis, number of retracted trials within each meta-analysis, and type of outcomes (benefit, harm) based on the categorisation of the review authors.

All data extraction was done by two groups of researchers independently. For the metadata of each contaminated meta-analysis, we recruited student volunteers to extract data on the basis of pre-designed data extraction forms in Excel. All student volunteers were trained for two rounds, and only those who achieved an accuracy of 90% and above in both rounds of training participated in data extraction. After training, 60 student volunteers met this criterion and extracted the metadata independently. Any disagreements were checked and resolved by one author (SQF).

Replication

Three evidence synthesis methodologists (CX, SQF, and YT) replicated the meta-analyses that incorporated retracted trials. This was done in two steps. We first used the same data from the original meta-analyses with the same methods and effect estimates to replicate the results. After obtaining the same results as the original meta-analysis, we excluded retracted trials and reanalysed the remaining data under the same settings (for example, same synthesis methods, effect estimates). Minor differences would occur during the replication owing to the rounding settings. We set a strict tolerable absolute difference as 0.01; we treated differences in the effect, the low boundary, and the upper boundary not exceeding 0.01 (or not exceeding 0.03 together) as fully replicated.

We replicated only pair-wise meta-analyses that compared the health outcomes of two interventions (including placebo or usual care) as aforementioned. In some cases, meta-analyses within a systematic review had identical effects and confidence intervals because the review authors used different settings of stratification for subgroups within the same group of datasets. To avoid double counting, we removed the duplicates and treated each subgroup analysis with retracted trials as a separate meta-analysis. For Cochrane reviews, different versions of the same review were common; we replicated only the latest version that incorporated retracted trials to avoid duplication and exaggeration of effects. We did not replicate network meta-analyses as very few of them provided sufficient data for replication.16 We also did not replicate meta-analyses of incidence as this type of meta-analysis uses data from only one arm of the trial, which therefore cannot be considered randomised trial data and does not meet our inclusion criteria.17

Outcomes

Our primary outcomes were the impact of retracted trials on the magnitude and direction of the effect sizes, as well as the significance of the P value, based on replication of the metadata. Our secondary outcome was an assessment of evidence syntheses that were substantially affected by retracted trials, which could potentially contaminate clinical practice guidelines. We defined “substantially affected” as those that changed both the direction of the effect size and the significance of the P value or those that changed the direction of the effect size while the P value remained significant, which would largely change the implied conclusion of such meta-analyses.

Statistical analysis

We summarised the baseline characteristics of retracted trials as frequency and proportion for binary or categorical data or as median and interquartile range for continuous data. For the main outcomes, we summarised the following metrics of the impact of retracted trials on the meta-analyses results: the proportion of meta-analyses that changed the direction of the effect size (for example, shifted from odds ratio >1 to odds ratio <1 or vice versa) after exclusion of the retracted trials; the proportion of meta-analyses that changed the significance of the P value (that is, shifted from <0.05 to >0.05 or vice versa) after exclusion of the retracted trials; the proportion of meta-analyses that changed both the direction of the effect size and the significance of the P value after exclusion of the retracted trials; and the proportion of meta-analyses whose effect size magnitude changed by more than 50% after exclusion of the retracted trials.16 We calculated point estimates and confidence intervals for these proportions on the basis of a generalised linear mixed model with potential clustering of meta-analyses by retracted trial and systematic review accounted for by using random effects.17 The random effects are not nested, but instead crossed, meaning that the effect due to retracted trials is the same regardless of the systematic review of origin.18 We defined a special case for changes in direction and significance when all the studies in a meta-analysis were retracted. In such cases, the results shifted from providing evidence for the outcome to providing no evidence at all.

We also fitted the same generalised linear mixed model to investigate the association between the number of studies within a meta-analysis and the likelihood of changes in the effect size direction or magnitude or the P value significance.19 The regression removed the special case in which all studies in a meta-analysis were retracted. To account for the anticipated non-linear relation between the number of studies and the outcomes, we applied a restricted cubic spline with three knots, following Harrell's recommended percentiles.20 21 As the proportion of retracted trials within a meta-analysis probably influences the likelihood of changes, we did stratified regression analyses by categorising meta-analyses into another four groups on the basis of the proportion of retracted trials: <25%, 25% to <50%, 50% to <75%, and 75% to <100%. We used the odds ratio with 95% confidence interval as the effect measure because it is not variation dependent and is thus portable across baseline risk.22

We used subgroup analyses to explore the impact of retracted trials under different settings, stratified by outcome data type (binary versus continuous) and nature (benefits versus harms).We used using the Stata SE/16 program for all analyses, with a two sided P value of 0.05 as the threshold for rejection of the null hypothesis of no effect. Visualisations were created using Excel 2016. The complete analysis codes are available in the supplementary file.

Patient and public involvement

This study investigated a methodological question related to the evidence ecosystem. For this reason, no patients or members of the public were involved in designing the research question or the outcome measures, and nor were they involved in design, implementation, interpretation, or writing up of the results.

Results

Our literature search on the Retraction Watch database resulted in 12 542 records. After reviewing the titles and abstracts, we excluded 10 034 of these; full text review of 2508 publications identified 1330 retracted trials that met the inclusion criteria (supplementary file, pages 39-114). Figure 1 shows the trial screening process. Supplementary table S3 shows the list of excluded publications with reasons for exclusion.

Fig 1.

Fig 1

Open in a new tab

Flowchart of selection of retracted trials. RCT=randomised controlled trial; SR=systematic review

In terms of forward citation searching, the 1330 retracted trials had a total of 82 398 citations from Google Scholar and Scopus. We further identified 847 systematic reviews with meta-analyses that incorporated retracted trials into synthesised evidence (see supplementary tables S4 and S5 for exclusion and inclusion lists). The agreement rate of the metadata extraction between the two volunteer groups was 95.6%.

Retracted randomised trials

Table 1 shows the baseline characteristics of the retracted trials; their median time to retraction was 21.3 (interquartile range 12.1-89.8) months from trial publication. Overall, 249 (18.7%) trials were registered and 34 (2.6%) were funded by industry. With regards to setting, 36.4% (n=484) of the retracted trials were single centre trials, 11.0% (n=146) were multicentre trials, and 52.6% (n=700) did not report such information. Only 27.8% (n=370) of the retracted trials had an explicit statement about data availability in the data sharing statements. The number of retracted trials increased rapidly in 2011 and 2023 (supplementary figure S1). Most of the retractions were initiated by journals (85.0%; n=1131), and only 9.2% (n=123) were initiated by the trial authors. The reported reasons for retraction involved a data related problem in 70.8% (n=942; see definition in table 1), and 12.9% (n=172) were marked as being from paper mills.

Table 1.

Baseline characteristics of retracted trials. Values are numbers (percentages)

Characteristics Summary of retracted trials
2010 and before (n=427) 2011 and after (n=903)
Retraction procedure
Author initiated* 39 (9.1) 84 (9.3)
Journal initiated 366 (85.7) 765 (84.7)
Author and journal initiated 7 (1.6) 19 (2.1)
Unclear 15 (3.5) 35 (3.9)
Reasons for retraction †
Data related retractions 284 (66.5) 658 (72.9)
Retractions for other reasons 143 (33.5) 245 (27.1)
Centre information
Single centre 127 (29.7) 357 (39.5)
Multiple centres 38 (8.9) 108 (12.0)
Missing 262 (61.4) 438 (48.5)
Trial registration
Yes 17 (4.0) 232 (25.7)
No 410 (96.0) 671 (74.3)
Source of funding
Industry supported 10 (2.3) 24 (2.7)
Non-industry supported 45 (10.5) 267 (29.6)
Not funded 18 (4.2) 88 (9.7)
Not reported 354 (82.9) 524 (58.0)
Raw data sharing statement
Data shared 0 (0.0) 320 (35.4)
Data not provided 0 (0.0) 50 (5.5)
Not reported 427 (100.0) 533 (59.0)
From paper mills
Yes 0 (0.0) 172 (19.0)
No 427 (100.0) 731 (81.0)
Open in a new tab

Retracted trials increased rapidly from 2011; characteristics are summarised separately cut by year of 2010 (also year when Retraction Watch was founded).

*

Refers to entities involved in retraction procedure including author of publication.

†

Based on records from Retraction Watch; data related retractions are defined as those trials labelled with data problem, with or without other problems.

Impact of retracted trials on evidence synthesis

Overall analyses

Of the 847 evidence synthesis publications that we identified, 324 (38.3%) were published after the earliest retraction within the review, and 4095 pair-wise meta-analyses that synthesised evidence from retracted trials were eligible for replication. With regards to the type of outcomes, 2372 (57.9%) were binary outcomes, and 1723 (42.1%) were continuous outcomes; 3689 (90.1%) investigated the benefits and 406 (9.9%) the harms of interventions. The number of retracted trials per meta-analysis ranged from one to 37, with a median number of 1 (interquartile range 1-1), and a minority of the meta-analyses (14.7%; 600/4095) had two or more retracted trials. The proportion of retracted trials per meta-analysis ranged from 0.7% to 100%, with a median proportion of 16.7% (interquartile range 9.1-33.3%); all included studies were retracted in 5.0% (205/4095) of the meta-analyses. We found that one fifth (687; 16.8%) of the meta-analyses potentially affected by retracted trials were from Cochrane reviews. We were able to fully replicate the original findings of 95.3% (3902/4095) meta-analyses from the reported data. However, despite our rigorous efforts, 193 (4.7%) could not be identically reconstructed to yield pooled estimates that exactly matched the original version. This was often because of incomplete and inconsistent reporting of the raw input data and the exact parameters of the meta-analytic model. The final analysis was based on the 3902 meta-analyses from 807 systematic reviews that we could fully replicate.

Figure 2 shows the impact of retracted trials on the results of meta-analyses (n=3902). We found that the exclusion of the retracted trials from the meta-analyses led to a change in the direction of the pooled effect in 8.4% (95% confidence interval (CI) 6.8% to 10.1%; n=392) of the meta-analyses, and a change in the significance of the P value occurred in 16.0% (14.2% to 17.9%; n=692). The proportion of meta-analyses in which both the direction of the effects and the significance of the P value were altered was 3.9% (95% CI 2.5% to 5.2%; n=218), and the proportion of meta-analyses in which either the direction of the effects or the significance of the P value was altered was 20.6% (18.5% to 22.8%; n=866). In 15.7% (95% CI 13.5% to 17.9%; n=659) meta-analyses, the magnitude of the effects changed by more than 50% when retracted trials were excluded.

Fig 2.

Fig 2

Open in a new tab

Impact of retracted trials on evidence synthesis research

Subgroup analysis

When we stratified the outcomes by type of data, we found that 9.5% (95% CI 7.2% to 11.7%) of the meta-analyses had a change in the direction of effects for binary outcomes and 7.0% (5.1% to 8.9%) for continuous outcomes, after exclusion of retracted trials (P for interaction=0.74). For the significance of the P value, 16.0% (95% CI 13.5% to 18.5%) changed in meta-analyses of binary outcomes and 15.8% (13.3% to 18.2%) changed in meta-analyses of continuous outcomes (P for interaction=0.78) after exclusion of retracted trials. For the magnitude of the effects, 11.0% (95% CI 8.4% to 13.5%) had a change by more than 50% for binary outcomes, and 22.5% (19.2% to 25.8%) had such a change for continuous outcomes (P for interaction<0.01) (fig 2).

When we stratified the outcomes by benefits and harms, we found that 8.0% (95% CI 6.4% to 9.7%) of the meta-analyses had a change in the direction of effect for benefit outcomes and 11.6% (7.0% to 16.2%) had a change for harms outcomes (P for interaction=0.001). For the significance of the P value, 16.0% (95% CI 14.1% to 18.0%) had a change in meta-analyses of benefit outcomes and 17.2% (12.2% to 22.2%) had a change for harms outcomes (P for interaction=0.31), after exclusion of retracted trials. For the magnitude of the effects, 15.7% (95% CI 13.5% to 18.0%) had a change by more than 50% for benefit outcomes and 17.6% (11.6% to 23.7%) for harms outcomes (P for interaction=0.19) (fig 2).

We did post hoc subgroup analysis by stratifying the meta-analyses on the basis of the reasons for retractions. For meta-analyses with retracted trials affected by data related problems (n=2885; 73.9%), 8.5% (95% CI 6.4% to 10.7%) had a change in the direction of the effects, 15.7% (13.5% to 17.8%) had a change in the significance of the P value, and 16.7% (14.0% to 19.5%) had a change in the magnitude of the effects by more than 50%. For meta-analyses with retracted trials without data problems (n=1017), 8.0% (95% CI 6.4% to 9.7%) had a change in the direction of the effects, 16.0% (14.1% to 18.0%) had a change in the significance of the P value, and 15.7% (13.5% to 18.0%) had a change in the magnitude of the effects by more than 50%. We observed no obvious between subgroup interactions (P for interaction=0.01, 0.30, and <0.001, respectively).

Number of studies and extent of impact on results

We observed an obvious non-linear association between the number of studies and the extent of impact of the inclusion of retracted trials on the results. Meta-analyses with a lower number of studies had a higher impact of retracted trials, regardless of the outcome of interest (for example, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). See supplementary figures S2-S7.

Contaminated evidence in clinical practice guidelines

We investigated the extent of contamination in clinical practice guidelines via a comprehensive literature search of the titles and topics for those meta-analyses that were substantially impacted after exclusion retracted trials. We found 218 such meta-analyses from 68 systematic reviews,23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 consisting of 19 (28%) Cochrane reviews and 49 (72%) non-Cochrane reviews.

The database search yielded 17 731 records, with 157 clinical practice guidelines identified that summarised evidence from these contaminated reviews (supplementary tables S6 and S7; supplementary figure S8). When stratified by categories, 89 (57%) were clinical practice guidelines, 42 (27%) were consensus statements, 12 (8%) were position statements, nine (6%) were practice bulletins, and five (3%) were committee opinions. The year of the release of these guideline documents ranged from 2009 to 2025, and 89 (57%) were released after 2018. Most (135; 86%) were developed by global, regional, or national academic associations (for example, World Health Organization,91 National Comprehensive Cancer Network 92). The full lists of the guidelines are in supplementary table S8.

Evidence contamination and evidence contamination chain

To show how retracted trials affect the evidence ecosystem, we established a simplified evidence contamination chain, shown in figure 3. Of the four potential contamination pathways, the one from retracted trials to evidence synthesis and then to clinical guideline documents may be the most important one. In terms of our quantitative analysis via forward citation searching, for the 1330 retracted trials, 312 (23.5%) caused contamination of 4095 meta-analyses from 847 systematic reviews, and the evidence from 218 meta-analyses (in 68 systematic reviews) that were substantially affected were further used in 157 English language clinical practice guidelines. This means that, on average, a single retracted trial would contaminate up to 13 meta-analyses from three systematic reviews, and each systematic review would further contaminate at least three clinical practice guidelines. Several examples are shown in table 2 and supplementary file, pages 391-393.

Fig 3.

Fig 3

Open in a new tab

Contamination chain of retracted trials on evidence ecosystem

Table 2.

Examples of retracted trials for clinical guideline documents via contaminating evidence synthesis

Systematic reviews Outcomes substantially affected Guidelines referring to contaminated evidence Recommendations using evidence from review
Carlisle et al, 200693 Comparisons: droperidol v granisetron; outcome (efficacy): nausea; with retracted trials (n=19): RR=1.36 (95% CI 1.05 to 1.77); without retracted trials (n=3): RR=0.94 (0.67 to 1.33) Society for Ambulatory Anaesthesia guideline (2014): Management of Postoperative Nausea and Vomiting 94 Result of outcome not used in guideline
Carlisle et al, 200693 Comparisons: placebo v droperidol; outcome (side effects): dizziness; with retracted trials (n=8): RR=1.01 (95% CI 0.40 to 2.54); without retracted trials (n=0): no evidence Society for Ambulatory Anaesthesia guideline (2014): Management of Postoperative Nausea and Vomiting 94 Guideline 7. Ensure PONV prevention and treatment is implemented in the clinical setting. “The safety of antiemetics is well established considering the huge amount of clinical data available and their summary in valid meta-analyses”
Liu et al, 201495 Comparisons: letrozole v clomiphene; outcome: No of dominant follicles; with retracted trials (n=4): MD=−0.40 (95% CI −1.68 to 0.89); without retracted trials (n=3): MD=0.20 (0.17 to 0.23) CFAS Guideline (2019): The Management of Unexplained Infertility 96 Recommendation on: ovarian stimulation with oral agents alone. “Aromatase inhibitors alone do not offer any benefit in comparison to clomiphene citrate alone and should not be offered to couples with UEI (Level 1A).”
Barrons et al, 201697 Comparisons: ACE inhibitors v placebo; outcome: walking distance; with retracted trials (n=4): MD=126 (95% CI −95 to 346); without retracted trials (n=2): MD=−47 (−71 to −24) No guideline was identified via forward citation searching NA
Open in a new tab

CI=confidence interval; MD=mean difference; NA=not applicable; RR=risk ratio.

Discussion

Principal findings

In this study, a large scale empirical investigation based on forward citation searching was conducted to evaluate the potential contamination cascade from retracted trials to the evidence ecosystem. The results suggest that, on average, a single retracted trial potentially contaminates approximately three systematic reviews, and each systematic review with meta-analyses contaminated by retracted trials would further contaminate almost three English language clinical practice guidelines. In addition, for meta-analyses in which retracted trials were present, the subsequent exclusion of the retracted trials led to a change in the direction of effects for 8.4% of the meta-analyses, a change in the magnitude by more than 50% in 15.7% of the meta-analyses, and a change in the significance of the P value in 16.0% of the pooled effect estimates. Our findings also indicate that this could lead to completely different conclusions (for example, change in both the direction of effects and the statistical significance) in at least 3.9% of meta-analyses, with subsequent contamination of 157 guideline documents.

The retracted trials (89.0%) were not usually multicentre studies, typically received academic funding or failed to report funding sources (89.5%), and were more likely to be unregistered (81.3%); these findings are in accordance with previous studies.14 98 99 In addition, we found that meta-analyses of harm outcomes were more susceptible to the impact of retracted trial data than were those of benefit outcomes; this could be expected as harm outcomes occur less frequently than benefits, involving larger random error and thus more fragile estimates.100 In addition, the numbers of studies reporting harm outcomes were often lower than for benefits (median: 6 v 7 in our dataset) owing to the less frequent nature and potential selective reporting problems.101 As meta-analyses with fewer studies were more likely to be affected, the impact of retracted trials on the results was more serious. The findings suggest that the nature of the data and the number of studies within a meta-analysis might be important factors in modulating the extent of the “contamination” of retracted trials on the results of the evidence synthesis.

Our forward citation searching showed that retracted trials continued to be cited after retraction, and the systematic reviews that synthesised these retractions also continued to be cited. Most the systematic reviews that synthesised retracted trials did not make a correction, and almost 40% of the reviews included retracted trials after the retraction, suggesting that these authors do not routinely assess the status of included studies and therefore remained unaware of the potential impact of retracted trials on their evidence synthesis. Although this problem has been identified previously,102 103 104 it has not yet been corrected. This problem becomes complex as we observed that even though some Cochrane reviews of the updated version removed the “old” retracted trials, new retractions occurred later (for example, Bordewijk et al105). Notably, we noticed that 172 of the retracted trials were from paper mills, which poses a big challenge to evidence synthesis research. The contamination cascade poses a serious risk, given the importance of data integrity to researchers, clinicians, and policy makers. More rigorous checks on the publication status of included trials should be required for authors of systematic reviews, whereas the burden of checking for retractions after publication of the systematic reviews would have to fall on the users of such meta-analyses or during updates of any systematic review.

Implications for future evidence synthesis practice

Our findings have major implications for the field of evidence synthesis. One of the most challenging aspects of evidence synthesis practice is the identification of retracted trials.106 107 Even where “signals” of concerns were observed, the investigation of problematic trials still took a long time. The median time from the submission of concern to the first journal correction is estimated to be 22.1 months.108 Existing databases keep current records of retracted trials, but almost inevitably many retractions will take place only after the completion and publication of associated systematic reviews. As such, we recommend the following approach for future evidence synthesis research. Firstly, authors of systematic review should identify already retracted trials before completion via multiple databases (for example, Retraction Watch, Scopus, Web of Science, PubMed) and pre-specify how such trials will be detected and handled in the review process. Secondly, many retractions occur well after publication of the evidence synthesis in question, so a priori evaluation of the integrity and trustworthiness of included trials may be useful for users of systematic reviews. Several important instruments have been released, such as the REAPPRIAISE checklist for checking publication integrity by Grey and colleagues,109 the Cochrane Pregnancy and Childbirth Trustworthiness Screening Tool (CPC‐TST) for assessment of trustworthiness,110 the TRACT checklist by Mol and colleagues for assessment of trustworthiness,111 and the ongoing project of INSPECT-SR for identifying problematic trials.112 Thirdly, for any evidence synthesis research, reviewers may wish to consider prioritising checking for retracted trials. Fourthly, authors of systematic reviews should work together with journal editors to publish updated systematic reviews and meta-analyses if any of the included trials have been retracted, although a method for flagging this is needed.

Further policy efforts should also make sure to keep up with the pace. For example, academic journals are expected to impose stricter policies on trial registration and data sharing. Although some journals (such as The BMJ and PLoS Medicine) already have such a policy,113 many journals still lack this. Some efforts have been made to track the potential retractions via forwards or backwards citation searching (for example, a systematic review accelerator called SpiderCite114) and to inform researchers when papers they have cited are retracted (for example, https://www.retracted.net/), and this may help the effort to correct the scientific record.

Strengths and limitations of study

To the best of our knowledge, this is the first attempt to establish the “contamination chain” of retracted trials in the evidence ecosystem. We used established methods of comprehensive literature searches and forward citation searching to collect randomised controlled trials, subsequent systematic reviews that quantitatively synthesised evidence from these trials, and clinical practice guidelines that used the contaminated evidence from these systematic reviews, even if retractions occurred afterwards. We believe that our comprehensive search to obtain a broadly representative sample and the painstaking replication of each single meta-analysis are major strengths that lend greater credibility to our findings. We also used strict data collection and data analytic procedures that helped to ensure the reliability of our results.

The study has some limitations that need to be discussed. Firstly, our study did not include patients or members of the public. This may have some potential downsides on subsequent dissemination, which we plan to overcome through extensive use of lay friendly material for social media, both in Chinese and in English. Secondly, we were unable to track all clinical practice guidelines on the basis of a literature search of four databases, because some clinical practice guidelines may not have been indexed in bibliographic databases,115 116 some were published in languages that were not included in our analysis, and some used evidence from randomised controlled trials directly, which would result in an underestimation of the number of affected guidelines. Thirdly, we focused on pair-wise meta-analyses only, so the impact of retracted trials on the results of network meta-analyses and any subsequent influence on relevant clinical practice guidelines needs to be further investigated. Fourthly, despite the use of Retraction Watch, we cannot be absolutely certain that we have identified all retracted trials included in systematic reviews. We also cannot be absolutely certain that we identified all systematic reviews that potentially synthesised evidence from retracted trials on the basis of forward citation searching, as some of the systematic reviews listed included studies in supplementary files (for example, Shi et al117). Moreover, the study focused only on retracted randomised trials, whereas some non-randomised studies of interventions would be retracted, which could also contaminate the evidence bodies. We may, therefore, have underestimated the impact of retracted trials on the results of systematic reviews and meta-analyses, as well as on clinical practice guidelines, and the real world contamination could possibly be even more serious.

Conclusions

On the basis of empirical evidence, retracted trials had a substantial impact on the evidence ecosystem. The problematic influence of retracted trial data on the pooled effect estimates could distort clinical practice guidelines and mislead decision making by healthcare practitioners and policy makers. Evidence generators, evidence synthesisers, evidence users, and policy makers should pay serious attention to the “contamination” by retracted trials of the evidence ecosystem, and reasonable approaches that assist with easier identification and correction of such contamination are urgently needed.

What is already known on this topic

  • In evidence based medicine, reliable decision making relies on the establishment of a trustworthy evidence ecosystem

  • Evidence generation and evidence synthesis are key elements for the establishment of such an ecosystem

  • The number of problematic or retracted studies has increased in the past decade, raising serious concern about the reliability of the data and conclusions of scientific research

What this study adds

  • This study of 1330 retracted trials showed an “evidence contamination chain” from retracted trial to evidence synthesis research and clinical guideline documents

  • A quarter of the retracted trials were quantitatively synthesised in subsequent systematic reviews, and each trial could potentially “contaminate” three systematic reviews

  • Subsequently, each systematic review could further be used in three English language guideline documents

Acknowledgments

VITALITY Collaborative Research Network: Yuqing Yang, research assistant (Shanghai Eastern Hepatobiliary Surgery Hospital, Naval Medical University, Shanghai, China); Hong Cao, research assistant (Shanghai Eastern Hepatobiliary Surgery Hospital, Naval Medical University, Shanghai, China); Zhen Peng, PhD student (Anhui Medical University, Hefei China); Ruixuan Tang, research assistant (West China Hospital, Sichuan University, Chengdu, China); Limin Li, Master student (Anhui Medical University, Hefei, China); Zhuanlan Sun, lecture (Nanjing University of Posts and Telecommunications, Nanjin, China); Jiahui Yin, lecture (Library, Tongji University, Shanghai, China); Rui Zhang, Postgraduate Student (Anhui Medical University, Hefei, China); Yi Zhu, Junior investigator (Shanghai Health Promotion Center, Shanghai, China); Xi Yang, Junior Public Health Professional (Shanghai Minhang District maternal and Child Health Hospital, Shanghai, China); Lijun Tang, Postgraduate Student (Anhui Medical University, Hefei, China); Mengya Zhao, Postgraduate Student (Anhui Medical University, Hefei, China); Manyu Zhang, Postgraduate Student (Anhui Medical University, Hefei, China); Ruilin Zhang, Postgraduate Student (Nanjing University of Posts and Telecommunications, Nanjin, China); Keke XuLiu, Postgraduate Student (Nanjing University of Posts and Telecommunications, Nanjin, China); Ying Tang, Postgraduate Student (Nanjing University of Posts and Telecommunications, Nanjin, China); Huoran Zhou, Postgraduate Student (Anhui Medical University, Hefei, China); Ruoxi Wang, Postgraduate Student (Anhui Medical University, Hefei, China); Qiannan Wang, Postgraduate Student (Anhui Medical University, Hefei, China); Limin Li, Postgraduate Student (Anhui Medical University, Hefei, China); Tolgor Bau, Postgraduate Student (Inner Mongolia Electronic Information Vocational Technical College, Inner Mongolia, China); Zhihui Jia, Postgraduate Student (Inner Mongolia Medical University, Inner Mongolia, China); Kexin Gao, Postgraduate Student (Inner Mongolia Medical University, Inner Mongolia, China); Yafang Hao, Postgraduate Student (Inner Mongolia Medical University, Inner Mongolia, China); Lijie Zhang, Postgraduate Student (Inner Mongolia Medical University, Inner Mongolia, China); Liying Zhang, Postgraduate Student (Inner Mongolia Medical University, Inner Mongolia, China); Kexin Fu, Postgraduate Student (Inner Mongolia Medical University, Inner Mongolia, China); Shuting Xing, Postgraduate Student (Inner Mongolia Medical University, Inner Mongolia, China); Ming Cheng, Postgraduate Student (Shihezi University, Xinjiang, China); Xiaojie Li, Postgraduate Student (The Second People's Hospital of Jiaozuo City, Henan, China); Kai Liu, Postgraduate Student (Wannan Medical College, Wuhu, China); Na He, Postgraduate Student (Anhui Medical University, Hefei, China); Mingyang Ma, Postgraduate Student (Anhui Medical University, Hefei, China); Debei Hao, Postgraduate Student (Beijing University of Posts and Telecommunications, Beijing, China); Shuya Sun, Postgraduate Student (Anhui Medical University, Hefei, China); Ruixian Tang, Research assistant (West China Hospital, Sichuan University, Chengdu, China); Qianqian Ji, Postgraduate Student (Anhui Medical University, Hefei, China); Chenglin Hu, Postgraduate Student (Wuchang Institute of Technology, Wuhan, China); Jiazhen Wang, Postgraduate Student (Anhui Engineering University, Hefei, China); Keer Jiang, Postgraduate Student (Wenzhou Medical University, Wenzhou, China); Weina Li, Postgraduate Student (Xi'an Jiaotong University, Xi’an, China); Yuxuan Guo, Postgraduate Student (Shanxi Medical University, Taiyuan, China); Guifang Jin, Postgraduate Student (Anhui Medical University, Hefei, China); Feifei Yan, Postgraduate Student (Jilin University, Jilin, China); Lizi Hu, Postgraduate Student (Chengdu University of TCM, Chengdu, China); Man Zhao, Postgraduate Student (Naval Medical University, Shanghai, China); Tingkai Zhang, Postgraduate Student (Naval Medical University, Shanghai, China); Feiyang Ye, PhD Student (Naval Medical University, Shanghai, China); Rui Zhou, Postgraduate Student (Naval Medical University, Shanghai, China); Gaowei Yang, Postgraduate Student (Naval Medical University, Shanghai, China); Xiaoqing Wu, Postgraduate Student (Naval Medical University, Shanghai, China); Jin Qian, Postgraduate Student (Naval Medical University, Shanghai, China); Jie Sheng, Postgraduate Student (Naval Medical University, Shanghai, China); Minghao Ruan, Postgraduate Student (Naval Medical University, Shanghai, China); Qingwen Ni, Postgraduate Student (Naval Medical University, Shanghai, China); Hongbo Wang, Postgraduate Student (Naval Medical University, Shanghai, China); Xiaocheng Feng, PhD student (Naval Medical University, Shanghai, China); Shuxun Wei, Postgraduate Student (Naval Medical University, Shanghai, China); Zhichao Wei, Postgraduate Student (Naval Medical University, Shanghai, China); Jingjing Xu, Postgraduate Student (Naval Medical University, Shanghai, China); Junhao Fan, Postgraduate Student (Naval Medical University, Shanghai, China); Liuyu Zhou, Postgraduate Student (Naval Medical University, Shanghai, China); Siwen Li, Postgraduate Student (Naval Medical University, Shanghai, China); Hucheng Liu, Postgraduate Student (Naval Medical University, Shanghai, China); Yuefan Wang, PhD student (Naval Medical University, Shanghai, China); Jin Du, Postgraduate Student (Naval Medical University, Shanghai, China); Wuyu Chen, Postgraduate Student (Naval Medical University, Shanghai, China); Wei Dong, attending doctor (Shanghai Eastern Hepatobiliary Surgery Hospital, Naval Medical University, Shanghai, China).

Web extra.

Extra material supplied by authors

Web appendix: Supplementary file

xuch082068.ww1.pdf (5.6MB, pdf)

Contributors: CX, HL, SD, YL, and SV conceived and designed the study. HL, FC, and CX contributed to funding acquisition and project management. CX contributed to literature screening, data collection, program coding, data analysis, visualisation, and manuscript drafting. SD contributed to statistical guidance, data checking, program coding, data analysis, and results interpretation. YT contributed to literature screening, data collection, data replication, data analysis, and results interpretation. SQF contributed to project management, literature screening, data collection, quality control, data replication, and data analysis. SQF and CX contributed to data verification. CX, SQF, and YT replicated the data. LFK, SYL, and CZ contributed to literature screening. SL, HL, SQF, and CX contributed to data checking. LL and HC provided statistical advice. SG and JC contributed to the search strategy. CX, SD, HL, PG, FCL, SL, SYL, SV, and YL contributed to results interpretation. HL, SV, LFK, SL, SYL, FCL, LL, HC, SD, PG, JC, SL, SG, and YL provided critical project advice and methodological comments. All authors carefully reviewed the manuscript and approved the final version to be published. CX and HL are the guarantors. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: This study was supported by four National Natural Science Foundation of China grants (72204003, 81970453, 82270634, 82302906) and institutional funding by Shanghai Eastern Hepatobiliary Surgery Hospital of Naval Medical University (TengFei Project, TF2024YZRH03). LFK was supported by the University of Queensland’s Amplify Initiative. SD was supported by programme grant #NPRP-BSRA01-0406-210030 from the Qatar National Research Fund. The funding bodies had no role in any process of the study (that is, study design, analysis, interpretation of data, writing of the report, and decision to submit the article for publication). The findings herein reflect the work and are solely the responsibility of the authors.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from four National Natural Science Foundation of China grants and institutional funding by Shanghai Eastern Hepatobiliary Surgery Hospital of Navy Medical University; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Transparency: The lead authors (the manuscript’s guarantors) affirm that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as originally planned (and, if relevant, registered) have been explained.

Dissemination to participants and related patient and public communities: We plan to present our findings at national and international scientific meetings. We also plan to use social media outlets to disseminate findings.

Provenance and peer review: Not commissioned; externally peer reviewed.

Contributor Information

Collaborators: VITALITY Collaborative Research Network, Yuqing Yang, Hong Cao, Zhen Peng, Ruixuan Tang, Limin Li, Zhuanlan Sun, Jiahui Yin, Rui Zhang, Yi Zhu, Xi Yang, Lijun Tang, Mengya Zhao, Manyu Zhang, Ruilin Zhang, Keke XuLiu, Ying Tang, Huoran Zhou, Ruoxi Wang, Qiannan Wang, Limin Li, Tolgor Bau, Zhihui Jia, Kexin Gao, Yafang Hao, Lijie Zhang, Liying Zhang, Kexin Fu, Shuting Xing, Ming Cheng, Xiaojie Li, Kai Liu, Na He, Mingyang Ma, Debei Hao, Shuya Sun, Ruixian Tang, Qianqian Ji, Chenglin Hu, Jiazhen Wang, Keer Jiang, Weina Li, Yuxuan Guo, Guifang Jin, Feifei Yan, Lizi Hu, Man Zhao, Tingkai Zhang, Feiyang Ye, Rui Zhou, Gaowei Yang, Xiaoqing Wu, Jin Qian, Jie Sheng, Minghao Ruan, Qingwen Ni, Hongbo Wang, Xiaocheng Feng, Shuxun Wei, Zhichao Wei, Jingjing Xu, Junhao Fan, Liuyu Zhou, Siwen Li, Hucheng Liu, Yuefan Wang, Jin Du, Wuyu Chen, and Wei Dong

Ethics statements

Ethical approval

Not required.

Data availability statement

The core datasets can be found at https://osf.io/ukg3j. The whole datasets could be obtained from the first author (xuchang2016@runbox.com) on request.

References

  • 1. Vandvik PO, Brandt L. Future of Evidence Ecosystem Series: Evidence ecosystems and learning health systems: why bother? J Clin Epidemiol 2020;123:166-70. [DOI] [PubMed] [Google Scholar]
  • 2. Murad MH, Montori VM. Synthesizing evidence: shifting the focus from individual studies to the body of evidence. JAMA 2013;309:2217-8. [DOI] [PubMed] [Google Scholar]
  • 3. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med 2000;342:1887-92. 10.1056/NEJM200006223422507  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Sibbald B, Roland M. Understanding controlled trials. Why are randomised controlled trials important? BMJ 1998;316:201. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Higgins JPT, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.5 (updated August 2024). 2024. www.training.cochrane.org/handbook.
  • 6. Murad MH, Asi N, Alsawas M, Alahdab F. New evidence pyramid. Evid Based Med 2016;21:125-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Wiedermann CJ, Joannidis M. The Boldt scandal still in need of action: the example of colloids 10 years after initial suspicion of fraud. Intensive Care Med 2018;44:1735-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Wadhwa RR, Rasendran C, Popovic ZB, Nissen SE, Desai MY. Temporal Trends, Characteristics, and Citations of Retracted Articles in Cardiovascular Medicine. JAMA Netw Open 2021;4:e2118263. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Avenell A, Bolland MJ, Gamble GD, et al. A randomized trial alerting authors, with or without coauthors or editors, that research they cited in systematic reviews and guidelines has been retracted. Account Res 2022;31:14-37. [Google Scholar]
  • 10. Fanelli D, Wong J, Moher D. What difference might retractions make? An estimate of the potential epistemic cost of retractions on meta-analyses. Account Res 2022;29:442-59. [DOI] [PubMed] [Google Scholar]
  • 11. Xu C. Impact of retracted RCTs for Evidence Synthesis. 2025. https://osf.io/7eazq/ .
  • 12. Salvador-Oliván JA, Marco-Cuenca G, Arquero-Avilés R. Development of an efficient search filter to retrieve systematic reviews from PubMed. J Med Libr Assoc 2021;109:561-74. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13. Estruch R, Ros E, Salas-Salvadó J, et al. PREDIMED Study Investigators . Primary prevention of cardiovascular disease with a Mediterranean diet. N Engl J Med 2013;368:1279-90. [DOI] [PubMed] [Google Scholar]
  • 14. Kataoka Y, Banno M, Tsujimoto Y, et al. Retracted randomized controlled trials were cited and not corrected in systematic reviews and clinical practice guidelines. J Clin Epidemiol 2022;150:90-7. [DOI] [PubMed] [Google Scholar]
  • 15. Djulbegovic B, Guyatt G. Evidence vs Consensus in Clinical Practice Guidelines. JAMA 2019;322:725-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. Xu C, Yu T, Furuya-Kanamori L, et al. Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study. BMJ 2022;377:e069155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Lin L, Chu H. Meta-analysis of Proportions Using Generalized Linear Mixed Models. Epidemiology 2020;31:713-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Guo Y, Dhaliwal J, Rights JD. Disaggregating level-specific effects in cross-classified multilevel models. Behav Res Methods 2024;56:3023-57. [DOI] [PubMed] [Google Scholar]
  • 19. Xu C, Furuya-Kanamori L, Lin L. Synthesis of evidence from zero-events studies: A comparison of one-stage framework methods. Res Synth Methods 2022;13:176-89. [DOI] [PubMed] [Google Scholar]
  • 20. Xu C, Doi SAR. The robust error meta-regression method for dose-response meta-analysis. Int J Evid Based Healthc 2018;16:138-44. [DOI] [PubMed] [Google Scholar]
  • 21. Harrell FE, Jr. Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer, 2001. 10.1007/978-1-4757-3462-1. [DOI] [Google Scholar]
  • 22. Doi SA, Furuya-Kanamori L, Xu C, Lin L, Chivese T, Thalib L. Controversy and Debate: Questionable utility of the relative risk in clinical research: Paper 1: A call for change to practice. J Clin Epidemiol 2022;142:271-9. 10.1016/j.jclinepi.2020.08.019  [DOI] [PubMed] [Google Scholar]
  • 23. Lin LJ, Liu J, Xu LZ, et al. The impact of endometrial mechanical stimulation in women with normal hysteroscopic findings undergoing IVF/ICSI: a meta-analysis. Clin Exp Obstet Gynecol 2022;49:27 10.31083/j.ceog4901027. [DOI] [Google Scholar]
  • 24. Jia D, Tan H, Faramand A, Fang F. One Anastomosis Gastric Bypass Versus Roux-en-Y Gastric Bypass for Obesity: a Systematic Review and Meta-Analysis of Randomized Clinical Trials. Obes Surg 2020;30:1211-8. 10.1007/s11695-019-04288-3  [DOI] [PubMed] [Google Scholar]
  • 25. Shi XY, Zou Z, He XY, Xu HT, Yuan HB, Liu H. Hydroxyethyl starch for cardiovascular surgery: a systematic review of randomized controlled trials. Eur J Clin Pharmacol 2011;67:767-82. 10.1007/s00228-011-1008-5  [DOI] [PubMed] [Google Scholar]
  • 26. Tassi A, Parisi N, Londero AP. Misoprostol administration prior to intrauterine contraceptive device insertion: a systematic review and meta-analysis of randomised controlled trials. Eur J Contracept Reprod Health Care 2020;25:76-86. 10.1080/13625187.2019.1706079  [DOI] [PubMed] [Google Scholar]
  • 27. Liu A, Zheng C, Lang J, Chen W. Letrozole versus clomiphene citrate for unexplained infertility: a systematic review and meta-analysis. J Obstet Gynaecol Res 2014;40:1205-16. 10.1111/jog.12393  [DOI] [PubMed] [Google Scholar]
  • 28. de Jong PG, Kaandorp S, Di Nisio M, Goddijn M, Middeldorp S. Aspirin and/or heparin for women with unexplained recurrent miscarriage with or without inherited thrombophilia. Cochrane Database Syst Rev 2014;2014:CD004734. 10.1002/14651858.CD004734.pub4  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29. Jacob M, Chappell D, Conzen P, Wilkes MM, Becker BF, Rehm M. Small-volume resuscitation with hyperoncotic albumin: a systematic review of randomized clinical trials. Crit Care 2008;12:R34. 10.1186/cc6812  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30. Bjelakovic G, Gluud LL, Nikolova D, et al. Vitamin D supplementation for prevention of mortality in adults. Cochrane Database Syst Rev 2014;2014:CD007470. 10.1002/14651858.CD007470.pub3  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Zhang L, Zhu J, Xu L, et al. Efficacy and safety of flurbiprofen axetil in the prevention of pain on propofol injection: a systematic review and meta-analysis. Med Sci Monit 2014;20:995-1002. 10.12659/MSM.890102  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. Mohamed HHB, Serag Eldien AM, Zahran A. Augmentation versus No Augmentation for Immediate Postextraction Implants. Int J Dent 2018;2018:5209108. 10.1155/2018/5209108  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 33. Gillespie LD, Robertson MC, Gillespie WJ, et al. Interventions for preventing falls in older people living in the community. Cochrane Database Syst Rev 2009;(2):CD007146. [DOI] [PubMed] [Google Scholar]
  • 34. Plantinga NL, de Smet AMGA, Oostdijk EAN, et al. Selective digestive and oropharyngeal decontamination in medical and surgical ICU patients: individual patient data meta-analysis. Clin Microbiol Infect 2018;24:505-13. 10.1016/j.cmi.2017.08.019  [DOI] [PubMed] [Google Scholar]
  • 35. Hernandez AV, Phan MT, Rocco J, et al. Efficacy and safety of hydroxychloroquine for hospitalized COVID-19 patients: a systematic review and meta-analysis. J Clin Med 2021;10:2503. 10.3390/jcm10112503  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 36. Juul S, Nielsen EE, Feinberg J, et al. Interventions for treatment of COVID-19: Second edition of a living systematic review with meta-analyses and trial sequential analyses (The LIVING Project). PLoS One 2021;16:e0248132. 10.1371/journal.pone.0248132  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 37. Lepine S, Jo J, Metwally M, Cheong YC. Ovarian surgery for symptom relief in women with polycystic ovary syndrome. Cochrane Database Syst Rev 2017;11:CD009526. 10.1002/14651858.CD009526.pub2  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Franik S, Eltrop SM, Kremer JA, Kiesel L, Farquhar C. Aromatase inhibitors (letrozole) for subfertile women with polycystic ovary syndrome. Cochrane Database Syst Rev 2018;5:CD010287. 10.1002/14651858.CD010287.pub3  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39. Li H, Cao Y, Ma P, et al. Novel visualization methods assisted transurethral resection for bladder cancer: an updated survival-based systematic review and meta-analysis. Front Oncol 2021;11:644341. 10.3389/fonc.2021.644341  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Shahin Y, Barnes R, Barakat H, Chetter IC. Meta-analysis of angiotensin converting enzyme inhibitors effect on walking ability and ankle brachial pressure index in patients with intermittent claudication. Atherosclerosis 2013;231:283-90. 10.1016/j.atherosclerosis.2013.09.037  [DOI] [PubMed] [Google Scholar]
  • 41. Barrons RW, Woods JA. The roles of ACE inhibitors in lower extremity peripheral artery disease. Am J Ther 2016;23:e7-15. 10.1097/MJT.0000000000000011  [DOI] [PubMed] [Google Scholar]
  • 42. O’Meara S, Al-Kurdi D, Ologun Y, Ovington LG, Martyn-St James M, Richardson R. Antibiotics and antiseptics for venous leg ulcers. Cochrane Database Syst Rev 2014;2014:CD003557. 10.1002/14651858.CD003557.pub5  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43. Turrini G, Purgato M, Acarturk C, et al. Efficacy and acceptability of psychosocial interventions in asylum seekers and refugees: systematic review and meta-analysis. Epidemiol Psychiatr Sci 2019;28:376-88. 10.1017/S2045796019000027  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44. Bala MM, Paszek E, Lesniak W, Wloch-Kopec D, Jasinska K, Undas A. Antiplatelet and anticoagulant agents for primary prevention of thrombosis in individuals with antiphospholipid antibodies. Cochrane Database Syst Rev 2018;7:CD012534. 10.1002/14651858.CD012534.pub2  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Carlisle JB, Stevenson CA. Drugs for preventing postoperative nausea and vomiting. Cochrane Database Syst Rev 2006;2006:CD004125. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Izcovich A, Peiris S, Ragusa M, et al. Bias as a source of inconsistency in ivermectin trials for COVID-19: A systematic review. Ivermectin’s suggested benefits are mainly based on potentially biased results. J Clin Epidemiol 2022;144:43-55. 10.1016/j.jclinepi.2021.12.018  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47. Yu C, Hequn C, Jinbo C, Feng Z, Xiongbing Z, Jian D. Gemcitabine/cisplatin versus methotrexate/vinblastine/doxorubicin/cisplatin for muscle-invasive bladder cancer: A systematic review and meta-analysis. J Cancer Res Ther 2018;14:1260-5. 10.4103/0973-1482.188434  [DOI] [PubMed] [Google Scholar]
  • 48. Sathianathen NJ, Hwang EC, Mian R, et al. Selective serotonin re-uptake inhibitors for premature ejaculation in adult men. Cochrane Database Syst Rev 2021;3:CD012799. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49. Cooper K, Martyn-St James M, Kaltenthaler E, Dickinson K, Cantrell A. Interventions to treat premature ejaculation: a systematic review short report [v-vi.]. Health Technol Assess 2015;19:1-180, v-vi. 10.3310/hta19210  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Wu T, Yue X, Duan X, et al. Efficacy and safety of tramadol for premature ejaculation: a systematic review and meta-analysis. Urology 2012;80:618-24. 10.1016/j.urology.2012.05.035  [DOI] [PubMed] [Google Scholar]
  • 51. Zhao LQ, Yu DY, Cheng JW. Intravenous glucocorticoids therapy in the treatment of Graves’ ophthalmopathy: a systematic review and Meta-analysis. Int J Ophthalmol 2019;12:1177-86. 10.18240/ijo.2019.07.20  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52. Shi C, Dumville JC, Cullum N, Connaughton E, Norman G. Compression bandages or stockings versus no compression for treating venous leg ulcers. Cochrane Database Syst Rev 2021;7:CD013397. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 53. Kongnyuy EJ, Wiysonge CS. Interventions to reduce haemorrhage during myomectomy for fibroids. Cochrane Database Syst Rev 2014;2014:CD005355. 10.1002/14651858.CD005355.pub5  [DOI] [PubMed] [Google Scholar]
  • 54. Howe TE, Shea B, Dawson LJ, et al. Exercise for preventing and treating osteoporosis in postmenopausal women. Cochrane Database Syst Rev 2011;(7):CD000333. 10.1002/14651858.CD000333.pub2  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 55. de Vaan MD, Ten Eikelder ML, Jozwiak M, et al. Mechanical methods for induction of labour. Cochrane Database Syst Rev 2019;10:CD001233. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 56. Deane KHO, Jimoh OF, Biswas P, et al. Omega-3 and polyunsaturated fat for prevention of depression and anxiety symptoms: systematic review and meta-analysis of randomised trials. Br J Psychiatry 2021;218:135-42. 10.1192/bjp.2019.234  [DOI] [PubMed] [Google Scholar]
  • 57. Haas DM, Hathaway TJ, Ramsey PS. Progestogen for preventing miscarriage in women with recurrent miscarriage of unclear etiology. Cochrane Database Syst Rev 2019;2019:CD003511. 10.1002/14651858.CD003511.pub5  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 58. Cheuk DK, Yeung WF, Chung KF, Wong V. Acupuncture for insomnia. Cochrane Database Syst Rev 2007;(3):CD005472. [DOI] [PubMed] [Google Scholar]
  • 59. Godoi A, Reis Marques I, Padrão EMH, et al. Glucose control and psychosocial outcomes with use of automated insulin delivery for 12 to 96 weeks in type 1 diabetes: a meta-analysis of randomised controlled trials. Diabetol Metab Syndr 2023;15:190. 10.1186/s13098-023-01144-4  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60. Dodd JM, Grivell RM, OBrien CM, Dowswell T, Deussen AR. Prenatal administration of progestogens for preventing spontaneous preterm birth in women with a multiple pregnancy. Cochrane Database Syst Rev 2017;10:CD012024. 10.1002/14651858.CD012024.pub2  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61. Sharpe A, Morley LC, Tang T, Norman RJ, Balen AH. Metformin for ovulation induction (excluding gonadotrophins) in women with polycystic ovary syndrome. Cochrane Database Syst Rev 2019;12:CD013505. 10.1002/14651858.CD013505  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62. Moll E, van der Veen F, van Wely M. The role of metformin in polycystic ovary syndrome: a systematic review. Hum Reprod Update 2007;13:527-37. 10.1093/humupd/dmm026  [DOI] [PubMed] [Google Scholar]
  • 63. Gao L, Yang L, Lv X, et al. A systematic review and meta-analysis of comparative studies on the efficacy of extended pelvic lymph node dissection in patients with clinically localized prostatic carcinoma. J Cancer Res Clin Oncol 2014;140:243-56. 10.1007/s00432-013-1574-2  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 64. Palacios C, Kostiuk LK, Peña-Rosas JP. Vitamin D supplementation for women during pregnancy. Cochrane Database Syst Rev 2019;7:CD008873. 10.1002/14651858.CD008873.pub4  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65. Li D, Cai Z, Pan Z, Yang Y, Zhang J. The effects of vitamin and mineral supplementation on women with gestational diabetes mellitus. BMC Endocr Disord 2021;21:106. 10.1186/s12902-021-00712-x  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 66. Zhou S, Zheng F, Zhan CG. Clinical data mining reveals analgesic effects of lapatinib in cancer patients. Sci Rep 2021;11:3528. 10.1038/s41598-021-82318-w  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 67. Showell MG, Mackenzie-Proctor R, Jordan V, Hart RJ. Antioxidants for female subfertility. Cochrane Database Syst Rev 2020;8:CD007807. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 68. Baradwan S, Abuzaid M, Sabban H, et al. Transvaginal needle versus laparoscopic ovarian drilling in hormonal profile and pregnancy outcomes of polycystic ovary syndrome: A systematic review and meta-analysis. J Gynecol Obstet Hum Reprod 2023;52:102606. 10.1016/j.jogoh.2023.102606  [DOI] [PubMed] [Google Scholar]
  • 69. Ding N, Chang J, Jian Q, Liang X, Liang Z, Wang F. Luteal phase clomiphene citrate for ovulation induction in women with polycystic ovary syndrome: a systematic review and meta-analysis. Gynecol Endocrinol 2016;32:866-71. 10.1080/09513590.2016.1197196  [DOI] [PubMed] [Google Scholar]
  • 70. Papanikolaou E, Peitsidis N, Tsakiridis I, et al. Endometrial scratching during hysteroscopy in women undergoing in vitro fertilization: a systematic review and meta-analysis. Front Surg 2023;10:1225111. 10.3389/fsurg.2023.1225111  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 71. Vitagliano A, Di Spiezio Sardo A, Saccone G, et al. Endometrial scratch injury for women with one or more previous failed embryo transfers: a systematic review and meta-analysis of randomized controlled trials. Fertil Steril 2018;110:687-702.e2. 10.1016/j.fertnstert.2018.04.040  [DOI] [PubMed] [Google Scholar]
  • 72. Pundir J, Psaroudakis D, Savnur P, et al. Inositol treatment of anovulation in women with polycystic ovary syndrome: a meta-analysis of randomised trials. BJOG 2018;125:299-308. 10.1111/1471-0528.14754  [DOI] [PubMed] [Google Scholar]
  • 73. Xiao Y, Tang L, Chen N. The Effects of Dexmedetomidine on Postoperative Sleep in Elderly Patients: A Systematic Review and Meta-Analysis. J Anesth Transl Med 2023;2:11-20 10.58888/2957-3912-2023-03-02. [DOI] [Google Scholar]
  • 74. Zeng Y, Zhang X, Luo W, Sheng Y. Effect of exercise intervention on clinical parameters in patients with non-alcoholic fatty liver disease and type 2 diabetes mellitus: a meta-analysis of randomized controlled trials. Eur J Gastroenterol Hepatol 2024;36:1-12. 10.1097/MEG.0000000000002662  [DOI] [PubMed] [Google Scholar]
  • 75. Lee HS, Yoon HY, Jin HJ, Hwang SH. Can Dexmedetomidine Influence Recovery Profiles from General Anesthesia in Nasal Surgery? Otolaryngol Head Neck Surg 2018;158:43-53. 10.1177/0194599817733735  [DOI] [PubMed] [Google Scholar]
  • 76. Lopez LM, Bernholc A, Zeng Y, et al. Interventions for pain with intrauterine device insertion. Cochrane Database Syst Rev 2015;2015:CD007373. 10.1002/14651858.CD007373.pub3  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 77. Christelle K, Norhayati MN, Jaafar SH. Interventions to prevent or treat heavy menstrual bleeding or pain associated with intrauterine-device use. Cochrane Database Syst Rev 2022;8:CD006034. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 78. Kamper SJ, Apeldoorn AT, Chiarotto A, et al. Multidisciplinary biopsychosocial rehabilitation for chronic low back pain. Cochrane Database Syst Rev 2014;2014:CD000963. 10.1002/14651858.CD000963.pub3  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 79. Pels A, Ganzevoort W, Kenny LC, et al. Interventions affecting the nitric oxide pathway versus placebo or no therapy for fetal growth restriction in pregnancy. Cochrane Database Syst Rev 2023;7:CD014498. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 80. Heshmati J, Farsi F, Yosaee S, et al. The Effects of Probiotics or Synbiotics Supplementation in Women with Polycystic Ovarian Syndrome: a Systematic Review and Meta-Analysis of Randomized Clinical Trials. Probiotics Antimicrob Proteins 2019;11:1236-47. 10.1007/s12602-018-9493-9  [DOI] [PubMed] [Google Scholar]
  • 81. Fairhall NJ, Dyer SM, Mak JC, Diong J, Kwok WS, Sherrington C. Interventions for improving mobility after hip fracture surgery in adults. Cochrane Database Syst Rev 2022;9:CD001704. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 82. Mu F, Yang X, Wang M, et al. Effect of Granulocyte Colony-Stimulating Factor on Clinical Pregnancy Outcomes of Recurrent Miscarriage and Recurrent Implantation Failure Cases: A Meta-Analysis. J Biol Regul Homeost Agents 2023;37:1933-47. [Google Scholar]
  • 83. Su Q, Pan Z, Yin R, Li X. The value of G-CSF in women experienced at least one implantation failure: a systematic review and meta-analysis. Front Endocrinol (Lausanne) 2024;15:1370114. 10.3389/fendo.2024.1370114  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 84. Geng J, Qian J, Cheng H, Ji F, Liu H. The Influence of Perioperative Dexmedetomidine on Patients Undergoing Cardiac Surgery: A Meta-Analysis. PLoS One 2016;11:e0152829. 10.1371/journal.pone.0152829  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 85. Abu Hashim H, Foda O, Ghayaty E. Combined metformin-clomiphene in clomiphene-resistant polycystic ovary syndrome: a systematic review and meta-analysis of randomized controlled trials. Acta Obstet Gynecol Scand 2015;94:921-30. 10.1111/aogs.12673  [DOI] [PubMed] [Google Scholar]
  • 86. Zhong JZ, Wei D, Pan HF, et al. Colloid solutions for fluid resuscitation in patients with sepsis: systematic review of randomized controlled trials. J Emerg Med 2013;45:485-95. 10.1016/j.jemermed.2013.05.018  [DOI] [PubMed] [Google Scholar]
  • 87. Zarychanski R, Abou-Setta AM, Turgeon AF, et al. Association of hydroxyethyl starch administration with mortality and acute kidney injury in critically ill patients requiring volume resuscitation: a systematic review and meta-analysis. JAMA 2013;309:678-88. [DOI] [PubMed] [Google Scholar]
  • 88. Carless PA, Rubens FD, Anthony DM, O’Connell D, Henry DA. Platelet-rich-plasmapheresis for minimising peri-operative allogeneic blood transfusion. Cochrane Database Syst Rev 2011;(3):CD004172. [DOI] [PubMed] [Google Scholar]
  • 89. Ruan J, Gong X, Kong J, Wang H, Zheng X, Chen T. Effect of B vitamin (folate, B6, and B12) supplementation on osteoporotic fracture and bone turnover markers: a meta-analysis. Med Sci Monit 2015;21:875-81. 10.12659/MSM.893310  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 90. Brown J, Farquhar C. Clomiphene and other antioestrogens for ovulation induction in polycystic ovarian syndrome. Cochrane Database Syst Rev 2016;12:CD002249. 10.1002/14651858.CD002249.pub5  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 91. World Health Organization . WHO antenatal care recommendations for a positive pregnancy experience: Nutritional interventions update: Multiple micronutrient supplements during pregnancy. World Health Organization, 2020. [PubMed] [Google Scholar]
  • 92. Denlinger CS, Sanft T, Baker KS, et al. Survivorship, Version 2.2017, NCCN Clinical Practice Guidelines in Oncology. J Natl Compr Canc Netw 2017;15:1140-63. 10.6004/jnccn.2017.0146  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 93. Carlisle JB, Stevenson CA. Drugs for preventing postoperative nausea and vomiting. Cochrane Database Syst Rev 2006;2006:CD004125. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 94. Gan TJ, Diemunsch P, Habib AS, et al. Society for Ambulatory Anesthesia . Consensus guidelines for the management of postoperative nausea and vomiting. Anesth Analg 2014;118:85-113. 10.1213/ANE.0000000000000002  [DOI] [PubMed] [Google Scholar]
  • 95. Liu A, Zheng C, Lang J, Chen W. Letrozole versus clomiphene citrate for unexplained infertility: a systematic review and meta-analysis. J Obstet Gynaecol Res 2014;40:1205-16. 10.1111/jog.12393  [DOI] [PubMed] [Google Scholar]
  • 96. Buckett W, Sierra S. The management of unexplained infertility: an evidence-based guideline from the Canadian Fertility and Andrology Society. Reprod Biomed Online 2019;39:633-40. 10.1016/j.rbmo.2019.05.023  [DOI] [PubMed] [Google Scholar]
  • 97. Barrons RW, Woods JA. The Roles of ACE Inhibitors in Lower Extremity Peripheral Artery Disease. Am J Ther 2016;23:e7-15. 10.1097/MJT.0000000000000011  [DOI] [PubMed] [Google Scholar]
  • 98. Avenell A, Stewart F, Grey A, Gamble G, Bolland M. An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature. BMJ Open 2019;9:e031909. 10.1136/bmjopen-2019-031909  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 99. Steen RG, Hamer RM. A case-control comparison of retracted and non-retracted clinical trials: Can retraction be predicted? Publications 2014;2:27-37 10.3390/publications2010027. [DOI] [Google Scholar]
  • 100. Turner RM, Bird SM, Higgins JP. The impact of study size on meta-analyses: examination of underpowered studies in Cochrane reviews. PLoS One 2013;8:e59202. 10.1371/journal.pone.0059202  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 101. Golder S, Loke YK, Wright K, Norman G. Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review. PLoS Med 2016;13:e1002127. 10.1371/journal.pmed.1002127  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 102. Candal-Pedreira C, Ruano-Ravina A, Fernández E, Ramos J, Campos-Varela I, Pérez-Ríos M. Does retraction after misconduct have an impact on citations? A pre-post study. BMJ Glob Health 2020;5:e003719. 10.1136/bmjgh-2020-003719  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 103. Wang Z, Shi Q, Zhou Q, et al. PRESS working group . Retracted systematic reviews continued to be frequently cited: a citation analysis. J Clin Epidemiol 2022;149:137-45. 10.1016/j.jclinepi.2022.05.013  [DOI] [PubMed] [Google Scholar]
  • 104. Mott A, Fairhurst C, Torgerson D. Assessing the impact of retraction on the citation of randomized controlled trial reports: an interrupted time-series analysis. J Health Serv Res Policy 2019;24:44-51. 10.1177/1355819618797965  [DOI] [PubMed] [Google Scholar]
  • 105. Bordewijk EM, Ng KYB, Rakic L, et al. Laparoscopic ovarian drilling for ovulation induction in women with anovulatory polycystic ovary syndrome. Cochrane Database Syst Rev 2020;2:CD001122. 10.1002/14651858.CD001122.pub5  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 106. Bakker C, Boughton S, Faggion CM, Jr, Fanelli D, Kaiser K, Schneider J. Reducing the residue of retractions in evidence synthesis: ways to minimise inappropriate citation and use of retracted data. BMJ Evid Based Med 2024;29:121-6. 10.1136/bmjebm-2022-111921  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 107. Boudry C, Howard K, Mouriaux F. Poor visibility of retracted articles: a problem that should no longer be ignored. BMJ 2023;381:e072929. 10.1136/bmj-2022-072929  [DOI] [PubMed] [Google Scholar]
  • 108. Grey A, Avenell A, Bolland M. Timeliness and content of retraction notices for publications by a single research group. Account Res 2022;29:347-78. 10.1080/08989621.2021.1920409  [DOI] [PubMed] [Google Scholar]
  • 109. Grey A, Bolland MJ, Avenell A, Klein AA, Gunsalus CK. Check for publication integrity before misconduct. Nature 2020;577:167-9. 10.1038/d41586-019-03959-6  [DOI] [PubMed] [Google Scholar]
  • 110. Weeks J, Cuthbert A, Alfirevic Z. Trustworthiness assessment as an inclusion criterion for systematic reviews—What is the impact on results? Cochrane Ev Synth 2023;1:e12037 10.1002/cesm.12037. [DOI] [Google Scholar]
  • 111. Mol BW, Lai S, Rahim A, et al. Checklist to assess Trustworthiness in RAndomised Controlled Trials (TRACT checklist): concept proposal and pilot. Res Integr Peer Rev 2023;8:6. 10.1186/s41073-023-00130-8  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 112. Wilkinson J, Heal C, Antoniou GA, et al. Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions. BMJ Open 2024;14:e084164. 10.1136/bmjopen-2024-084164  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 113. Naudet F, Sakarovitch C, Janiaud P, et al. Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine . BMJ 2018;360:k400. 10.1136/bmj.k400  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 114. Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM. A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol 2020;121:81-90. 10.1016/j.jclinepi.2020.01.008  [DOI] [PubMed] [Google Scholar]
  • 115. Haase A, Follmann M, Skipka G, Kirchner H. Developing search strategies for clinical practice guidelines in SUMSearch and Google Scholar and assessing their retrieval performance. BMC Med Res Methodol 2007;7:28. 10.1186/1471-2288-7-28  [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 116. Lefebvre C, Glanville J, Briscoe S, et al. Searching for and selecting studies. In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated October 2023). Cochrane, 2023. [Google Scholar]
  • 117. Shi Q, Wang Y, Hao Q, et al. Pharmacotherapy for adults with overweight and obesity: a systematic review and network meta-analysis of randomised controlled trials. Lancet 2022;399:259-69. 10.1016/S0140-6736(21)01640-8  [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Web appendix: Supplementary file

xuch082068.ww1.pdf (5.6MB, pdf)

Data Availability Statement

The core datasets can be found at https://osf.io/ukg3j. The whole datasets could be obtained from the first author (xuchang2016@runbox.com) on request.

Articles from The BMJ are provided here courtesy of BMJ Publishing Group

RESOURCES