Abstract
Introduction
It is essential to choose a realistic anticipated intervention effect when calculating a sample size for a randomised clinical trial. Unfortunately, anticipated intervention effects are often inflated, when compared with the ‘true’ intervention effects. This is documented for mortality in critical care trials. A similar pattern might exist across different medical specialties. This study aims to estimate the range of observed intervention effects for all-cause mortality in trials included in Cochrane Reviews, within each Cochrane Review Group.
Methods and analysis
We will include randomised clinical trials assessing all-cause mortality as an outcome. Trials will be identified from Cochrane Reviews published in the Cochrane Database of Systematic Reviews. Cochrane Reviews will be clustered according to the registered Cochrane Review Group (eg, Anaesthesia, Emergency and Critical Care) and the statistical analyses will be conducted for each Cochrane Review Group and overall. The median relative risk and IQR for all-cause mortality and the proportion of trials with a relative all-cause mortality risk within seven different ranges will be reported (relative risk below 0.70, 0.70–0.79, 0.80–0.89, 0.90–1.09, 1.10–1.19, 1.20–1.30 and above 1.30). Subgroup analyses will explore the effects of original design, sample size, risk of bias, disease, intervention type, follow-up length, participating centres, funding type, information size and outcome hierarchy.
Ethics and dissemination
Since we will use summary data from trials already approved by relevant ethical committees, this study does not require ethical approval. Regardless of our findings, the results will be published in an international peer-reviewed journal.
Keywords: ANAESTHETICS, INTENSIVE & CRITICAL CARE, ACCIDENT & EMERGENCY MEDICINE
Strengths and limitations of this study.
The design of this methodological study will rely on the principles of the published tutorial for the development of the MethodologIcal STudy reportIng Checklist.
A detailed description of the independent literature search and data extraction is presented in this protocol.
A detailed predefined statistical analysis plan including relevant subgroup analyses is described.
As trials reporting beneficial effects of interventions are more likely to be published, mortality estimates from our analyses will likely be exaggerated.
As data are extracted directly from the Cochrane Reviews instead of the trial publication, we will rely on the correctness of the Cochrane review authors’ data extraction.
Introduction
Sample size and power estimations are pivotal elements when planning and designing randomised clinical trials.1 2 To calculate a sample size, it is necessary to quantify an anticipated intervention effect. Large, anticipated intervention effects lead to relatively small sample sizes and small anticipated intervention effects lead to relatively large sample sizes. Different methods to estimate anticipated intervention effects are available, as described in the Difference ELicitation in TriAls reviews.3 4 Anticipated intervention effects may be estimated based on what a minimal important difference would be, but this approach is problematic when assessing binary patient important outcomes such as mortality, where every event seems clinically important. A commonly used alternative approach is to estimate the anticipated intervention effect based on previous evidence, such as previous trial results or pilot trials, that is, estimate anticipated realistic intervention effects.3 5 6 However, quantifications of intervention effects within most medical specialties have not previously been systematically assessed. It is problematic if the anticipated intervention effects are unrealistically large: (1) the sample sizes will be insufficient to detect smaller but more realistic intervention effects, even if still clinically relevant. This could potentially lead to false negative conclusions regarding the interventions’ potential benefits7; (2) there is an increased risk of showing misleading spurious intervention effects if too large intervention effects are anticipated8–10; and (3) relatively small randomised trials may not secure proper randomisation with good balance between the intervention groups of all important prognostic factors.11 On the other hand, unrealistically small, anticipated intervention effects will lead to unnecessary large sample sizes and thereby excessive randomisations, which is unethical and a misuse of participants and resources.12 Moreover, such ‘overpowered’ trials run the risks to demonstrate ‘significant’ findings of small irrelevant intervention effects.13 14 If at all possible, quantifications of anticipated intervention effects that are far from the ‘true’ effect should be avoided.
Mortality is a frequently used and patient-relevant outcome in randomised clinical trials within all medical specialties.15–17 Inflated, anticipated intervention effects for mortality are already well documented in critical care trials.18 19 It is likely that a similar trend exists across different medical specialties. Knowledge on observed intervention effects for mortality within different medical specialties could aid trialists in choosing realistic anticipated intervention effects in the trial design phase. Such knowledge could also be used by funding agencies to determine whether an anticipated intervention effect for a proposed trial is realistic.
Methods and analysis
Aim
The aim of this methodological study is to estimate the range of observed intervention effects for all-cause mortality in randomised clinical trials included in Cochrane Reviews within each Cochrane Review Group and overall.20 Additionally, we will conduct subgroup analyses to explore the effects of various methodological factors on the observed intervention effects for all-cause mortality.
Study design
We will conduct a descriptive, cross-sectional, methodological study. According to our knowledge, no sufficient reporting guidelines are available yet for methodological studies or adjacent protocols. Authors of the protocol for the development of the MethodologIcal STudy reportIng Checklist, a work-in-progress reporting checklist for methodological studies,21 have published a tutorial that provides an overview of the key aspects of methodological studies.22 The design and conduct of this methodological study rely on the principles from this tutorial.22
Eligibility criteria
Cochrane Reviews and randomised clinical trials will be selected according to the criteria below.
Types of studies
We will include randomised clinical trials, included in Cochrane Reviews from inception and until the date of the literature search.20 Cochrane Reviews will be identified by using the search string described below under ‘search strategy’. A Cochrane Review is considered eligible, if it includes at least one randomised clinical trial reporting on all-cause mortality. Eligible trials in the Cochrane Reviews must either assess all-cause mortality as a separate outcome or as part of a composite outcome where the number of participants and mortality events in both intervention groups are reported separately. Trials of factorial design will be included as well, but each separate comparison will be considered as one trial in the data analyses. Cluster randomised trials will not be included, as the unit of intervention and analysis method differs from ordinary randomised trials.23 Cross-over randomised trials will not be included, as the design is inappropriate when measuring longer-term outcomes such as mortality.23 Adaptive randomised trials will not be included, as the number of intervention groups or the sample size is modifiable during the trial conduct.24 Quasirandomised studies will not be included due to their lack of random group assignment.25
Types of participants
We will include participants of all ages, irrespectively of comorbidities and sex.
Types of interventions
We will include randomised clinical trials evaluating all types of interventions, including complex interventions.26 The type of intervention will be noted for all included trials and categorised as below:
Drug.
Medical device.
Surgical procedure.
Complex intervention.
Other interventions.
Outcomes
The primary and sole outcome in this methodological study is all-cause mortality at longest follow-up, as reported in the Cochrane Review. Thus, if more than one mortality outcome is reported for a meta-analysis within the Cochrane Review, we will extract data based on the mortality outcome with the longest follow-up (eg, death up until 30 days from discharge will be prioritised over death before discharge).
Search strategy
Within each Cochrane Review Group, we will identify potentially eligible Cochrane Reviews by conducting a systematic search in the Cochrane Database of Systematic Reviews (https://www.cochranelibrary.com/advanced-search). The search string will be as follows: “Title Abstract Keyword: ‘mortality’ or ‘death’ or ‘survival’” and will include ‘search word variations’. The following search limits will be imposed: ‘Cochrane reviews’, ‘all dates’, ‘all years’.
Data collection and analysis
Selection of studies
Within each Cochrane Review Group, all identified Cochrane Reviews will be uploaded to EndNote (Clarivate, Philadelphia, Pennsylvania, USA) and duplicates will be removed. Two authors will independently screen titles and abstracts of identified reviews. If necessary, the full Cochrane Review reports will be assessed for eligibility. If disagreement between the two authors regarding eligibility of a Cochrane Review, a third author will be presented to the disagreement and make a final decision regarding inclusion or exclusion of the Cochrane Review. Once the eligible Cochrane Reviews have been identified, trials published in the Cochrane Reviews will be assessed for eligibility according to the eligibility criteria defined in this protocol.
For each Cochrane Review Group, a flowchart of the literature search and study selection, including reasons for ineligibility, will be completed and included with the publication of the results. The flowchart will be based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram.27
Data extraction and management
For each eligible trial, the following data will be extracted directly from the Cochrane Reviews:
Trial characteristics: Cochrane Review Group, Cochrane Review title, trial authors and year of publication, original design (superiority, non-inferiority or equivalence trial design), industry funding or conflicts of interest (yes or no) as evaluated and reported by the Cochrane authors, number of participating centres (total number and categorised as 1 centre; 2–4 centres; less than 5 centres; 5–10 centres; or more than 10 centres), sample size (total number of participants and categorised as trials with less than 50 participants; 50–200 participants; or more than 200 participants), intervention type (specific intervention and categorised as drug; medical device; surgical procedure; complex intervention; or other interventions), disease area (specific disease area), factorial design (yes or no), method for risk-of-bias assessment (RoB or RoB V.2), RoB (high; unclear or of some concerns; or low) as evaluated and reported by the Cochrane authors, follow-up time on all-cause mortality (number of days and categorised as 14 days of follow-up; from 14 to 60 days of follow-up; from 61 to 90 days of follow-up; or more than 90 days follow-up), all-cause mortality as the primary outcome (yes or no), method ascertaining and validating death (eg, registry data, clinical records, direct contacts with relatives).
Results: observed intervention effect for mortality (in relative risk, odds ratio, hazard ratio etc with 95% CIs), number of participants in the experimental group in the mortality analysis (n), number of mortality events in the experimental group in the mortality analysis (n), number of participants in the control group in the mortality analysis (n), number of mortality events in the control group in the chosen mortality analysis (n).
The data extraction will be conducted independently by two authors and entered into a specific data extraction form designed for this study. If any disagreements in data extractions between the two authors, a third author will make the final decision.
Assessment of RoB for all-cause mortality in the included studies
The RoB assessment conducted by the Cochrane authors and reported in the Cochrane Review will be used to determine the RoB for each trial.
Original RoB tool
If the Cochrane authors used the Cochrane Collaboration’s original RoB tool28 in the Cochrane Review, we will consider a trial to be at low RoB, if the authors have judged all domains (random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting, other bias) to be at low RoB. We will consider a trial to be at unclear RoB, if one or more domains have been judged as unclear RoB, but no domains judged as high RoB. We will consider a trial to be at high RoB if one or more domains have been judged as high RoB.
RoB V.2 tool
If the Cochrane authors used the Cochrane Collaboration’s RoB V.2 tool, we will rely on the overall RoB judgement for the mortality outcome (low RoB, of some concerns or high RoB).29
Publication bias
To assess the risk of publication bias, we will create a funnel plot for all-cause mortality at longest follow-up including all eligible trials. The funnel plot will be visually inspected for asymmetry and the asymmetry will be evaluated by the Harbord test.30
Statistical analyses
The statistical analyses below will be conducted within each Cochrane Review Group. Thus, the analyses will be based on eligible trials from Cochrane Reviews within each Cochrane Review Group.
Primary analysis
In the primary analysis, we will estimate the median relative risk and IQR for all-cause mortality at longest follow-up. Additionally, we will present the median follow-up time and IQR in days for all-cause mortality at longest follow-up. Within each trial, the relative risk for all-cause mortality will be estimated based on the reported number of participants and number of mortality events in each arm (table 1).
Table 1.
Median relative risks and IQRs for all-cause mortality in the primary analysis and in subgroup analyses excluding disease area subgroups.
| n (%) of total included trials | Median relative risk and IQR | |
| Primary analysis of all-cause mortality at the longest follow-up | ||
| All trials | ||
| Follow-up time | * | |
| Subgroup analyses of all-cause mortality | ||
| Original design | ||
| Superiority | ||
| Non-inferiority | ||
| Equivalence | ||
| Sample size of participants | ||
| Less than 50 | ||
| 50–200 | ||
| More than 200 | ||
| Risk of bias | ||
| Low | ||
| Unclear or of some concerns | ||
| High | ||
| Intervention type | ||
| Drug | ||
| Medical device | ||
| Surgical procedure | ||
| Complex intervention | ||
| Other interventions | ||
| Length of follow-up for all-cause mortality | ||
| Less than 14 days | ||
| 14–60 days | ||
| 61–90 days | ||
| More than 90 days | ||
| Number of participating centres | ||
| 1 | ||
| 2–4 | ||
| 5–10 | ||
| More than 10 | ||
| Funding type | ||
| Industry funded or conflicts of interest | ||
| No industry funding and no conflicts of interest | ||
| Optimal information size | ||
| Less than 50% of the optimal information size | ||
| 50%–99% of the optimal information size | ||
| More than 99% of the optimal information size | ||
| Outcome hierarchy for all-cause mortality | ||
| Defined as primary outcome | ||
| Defined as any other outcome |
*Median days and IQR.
n, number of trials included in the analysis.
Optimal information size
For each included trial, we will also calculate the ‘optimal information size’. The calculation of the optimal information size will be based on the observed proportion of all-cause mortality events in the control group in the relevant Cochrane Review, a relative risk reduction of 25%, a 5% risk of type I error and a 20% risk of type 2 error.31
Secondary analyses
As secondary analyses, we will estimate the proportion of trials with a relative risk for all-cause mortality in the following ranges (table 2):
Table 2.
Number and percentage of trials with a relative risk within seven different ranges.
| Relative risk | Number and percentage of trials, n (%) |
| Below 0.70 | |
| 0.70–0.79 | |
| 0.80–0.89 | |
| 0.90–1.09 | |
| 1.10–1.19 | |
| 1.20–1.30 | |
| Above 1.30 |
Relative risk below 0.70.
Relative risk between 0.70 and 0.79.
Relative risk between 0.80 and 0.89.
Relative risk between 0.90 and 1.09
Relative risk between 1.10 and 1.19.
Relative risk between 1.20 and 1.30.
Relative risk above 1.30.
Subgroup analyses
Additionally, we will estimate the median relative risk with IQR for all-cause mortality in the following subgroups (tables 1 and 3):
Table 3.
Median relative risks and interquartile ranges for all-cause mortality in subgroup analysis of trials within specific disease areas (examples on disease areas for the manuscript on the Cochrane Review Group ‘Anaesthesia’).
| Disease area | n (%) of total included trials | Median relative risk and IQR |
| Abdominal surgery | ||
| Cardiac surgery | ||
| Abdominal aortic surgery | ||
| Thoracic non-cardiac surgery | ||
| Brain surgery | ||
| Spinal surgery | ||
| Orthopaedic surgery | ||
| Gynaecological surgery |
Trials of superiority design, non-inferiority design or equivalence design.
Trials with less than 50 randomised participants, 50–200 randomised participants or more than 200 randomised participants.
Trials at low RoB, unclear RoB or of some concern or high RoB, as reported in the Cochrane Review.
Trials within specific disease areas (eg, abdominal surgery, cardiac surgery, orthopaedic surgery in Anaesthesia trials).
Trials evaluating a drug intervention, a medical device intervention, a surgical procedure, a complex intervention or other interventions.
Trials with less than 14 days of follow-up, from 14–60 days of follow-up, from 61–90 days of follow-up or more than 90 days follow-up on all-cause mortality.
Trials with only 1 centre, 2–4 centres, 5–10 centres or more than 10 centres.
Trials with less than 50% of the optimal information size, 50%–99% of the optimal information size or more than 99% of the optimal information size.
Trials where all-cause mortality is defined as the primary outcome or defined as any other outcome.
Trials with and without industry support or conflicts of interest, as reported in the Cochrane Review.
Trials published per decade.
Patient and public involvement statement
No patients involved.
Discussion
This methodological study will provide important and clinically relevant information on the range of observed intervention effects for all-cause mortality in randomised clinical trials across different medical specialties. We believe this information can aid trialists in choosing a realistic, anticipated intervention effect when planning a randomised clinical trial. This will decrease the risk of initiating trials that are unable to detect smaller, but realistic and still clinically relevant, intervention effects.
The most important weakness of this methodological study is the risk of publication bias affecting our mortality estimates.32 It is likely that trials reporting a ‘positive result’ (a beneficial effect) of the intervention are more likely to be published than trials reporting a ‘neutral’ or ‘negative result’ (no benefit).33–35 Such publication bias skews the accrued literature towards benefit of interventions. In accumulated analyses of trial results, this may lead to exaggerated intervention effects. Therefore, it is plausible that the mortality estimates from our analyses are larger than the ‘true’ intervention effects. To address this potential issue, we will create funnel plots and evaluate any asymmetry by visual inspection and the Harbord test.30 In general, bias due to poor methodology can lead to an exaggeration of the intervention effect in single randomised clinical trials, which can also affect accumulated analyses.15 36 To evaluate the effect of bias on our observed intervention effects for mortality, we will conduct subgroup analyses stratified per the RoB level of the trials. Extracting trial data directly from the Cochrane Reviews instead of the trial publication could also be identified as a methodological weakness, since we will rely on the correctness of the Cochrane Review authors’ data extraction. However, Cochrane Reviews are based on solid methodology and data extraction is done by multiple persons to minimise random errors.37 Therefore, we believe that the available data in the Cochrane Reviews can be trusted. Additionally, extracting data directly from Cochrane Reviews allows us to identify and extract data from a larger number of trials, than if each eligible trial ought to be identified in a systemic search from various databases (eg, PubMed, EMBASE), followed by data extraction from the trial publication. Thus, we can reach a larger sample size and greater statistical power in our mortality effect estimates in a shorter period. Cochrane reviews also include randomised clinical trials regardless of quality, sample size and ‘news value’. This will give our observed mortality effect estimates a higher external validity and generalisability, than if the estimates were based on trial data from, for example, the top-five high impact journals. An alternative method to derive mortality data from randomised clinical trials is to extract data directly from the databases of regulating authorities, such as the US Food and Drug Administration Resources on Drugs and Devices and the European Union Drug Regulating Authorities Clinical Trials Database. This has been done for multiple disease entities in a previously published methodological study.38 However, these databases do not include RoB assessments as opposed to the Cochrane Reviews and therefore, this methodology is not feasible for the purpose of our present study.
Even though this methodological study has weaknesses, we still believe that it will provide valuable information for trialists. With limited available funding for investigator-initiated trials,39 40 it is important that trials succeeding in obtaining the necessary funding are based on sound methodology, including a sample size that is large enough to evaluate a realistic intervention effect with sufficient statistical power. In the future, it would be relevant to determine the range of observed intervention effects for various, frequently used trial outcomes.
Ethics and dissemination
Ethics
As this methodological study does not include any data from individual persons, ethical approval is not required.
Dissemination
The results will be reported separately for each Cochrane Review group. Thus, we intend to write one manuscript for each Cochrane Review Group, starting with one manuscript for the Cochrane Review group ‘Anaesthesia’ followed by one manuscript for the Cochrane Review Group ‘Emergency and Critical Care’. Once the results have been reported for each Cochrane Review groups, we intend to write a manuscript based on the results from the overall analysis including trials across each Cochrane Review group. Regardless of the findings, the results will be published in an international peer-reviewed journal. We plan to conduct the search and initiate data collection for the first study on the Cochrane Review Group ‘Anaesthesia’ by 1 September 2023. Expected date for submission of the adjacent manuscript is 1 December 2023.
Supplementary Material
Footnotes
Contributors: All authors contributed to the concept and design of this protocol. MLH drafted the first version of this manuscript. All additional authors revised the manuscript critically for important intellectual content and gave final approval of the published version.
Funding: This study was partly funded by the Copenhagen Trial Unit, Centre for Clinical Intervention Research, The Capital Region, Copenhagen University Hospital—Rigshospitalet, Copenhagen, Denmark.
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review: Not commissioned; externally peer reviewed.
Ethics statements
Patient consent for publication
Not applicable.
References
- 1.Feinstein AR. The other side of 'statistical significance': alpha, beta. Delta, and the calculation of sample size. Clin Pharmacol Ther 1975;18:491–505. 10.1002/cpt1975184491 Available: http://doi.wiley.com/10.1002/cpt.1975.18.issue-4 [DOI] [PubMed] [Google Scholar]
- 2.Jakobsen JC, Ovesen C, Winkel P, et al. Power Estimations for non-primary outcomes in randomised clinical trials. BMJ Open 2019;9:e027092. 10.1136/bmjopen-2018-027092 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hislop J, Adewuyi TE, Vale LD, et al. Methods for specifying the target difference in a randomised controlled trial: the difference Elicitation in trials (DELTA) systematic review. PLoS Med 2014;11:e1001645. 10.1371/journal.pmed.1001645 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Cook JA, Julious SA, Sones W, et al. DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial. BMJ 2018;363:k3750. 10.1136/bmj.k3750 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Rothwell JC, Julious SA, Cooper CL. A study of target effect sizes in randomised controlled trials published in the health technology assessment Journal. Trials 2018;19:544. 10.1186/s13063-018-2886-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Vanderhout S, Fergusson DA, Cook JA, et al. Patient-reported outcomes and target effect sizes in pragmatic randomized trials in Clinicaltrials.Gov: A cross-sectional analysis. PLoS Med 2022;19:e1003896. 10.1371/journal.pmed.1003896 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Sidebotham D. Are most randomised trials in anaesthesia and critical care wrong? an analysis using Bayes’ theorem. Anaesthesia 2020;75:1386–93. 10.1111/anae.15029 [DOI] [PubMed] [Google Scholar]
- 8.Ioannidis JPA. Why most discovered true associations are inflated. Epidemiology 2008;19:640–8. 10.1097/EDE.0b013e31818131e7 [DOI] [PubMed] [Google Scholar]
- 9.Dechartres A, Trinquart L, Boutron I, et al. Influence of trial sample size on treatment effect estimates: meta-epidemiological study. BMJ 2013;346:f2304. 10.1136/bmj.f2304 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Pereira TV, Horwitz RI, Ioannidis JPA. Empirical evaluation of very large treatment effects of medical interventions. JAMA 2012;308:1676–84. 10.1001/jama.2012.13444 [DOI] [PubMed] [Google Scholar]
- 11.Nguyen TL, Collins GS, Lamy A, et al. Simple randomization did not protect against bias in smaller trials. J Clin Epidemiol 2017;84:105–13. 10.1016/j.jclinepi.2017.02.010 [DOI] [PubMed] [Google Scholar]
- 12.World Medical Association . WMA Declaration of Helsinki ‐ ethical principles for medical research involving human subjects risks. Burdens and Benefits 2014:21–2. [Google Scholar]
- 13.Ioannidis JPA. Why most published research findings are false. PLoS Med 2005;2:e124. 10.1371/journal.pmed.0020124 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Faber J, Fonseca LM. How sample size influences research outcomes. Dental Press J Orthod 2014;19:27–9. 10.1590/2176-9451.19.4.027-029.ebo [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Savovic J, Turner RM, Mawdsley D, et al. Association between risk-of-bias assessments and results of randomized trials in Cochrane reviews: the ROBES meta-epidemiologic study. Am J Epidemiol 2018;187:1113–22. 10.1093/aje/kwx344 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Higgins JPT, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions 6.1. Wiley, 10.1002/9781119536604 [DOI] [Google Scholar]
- 17.Garattini S, Jakobsen JC, Wetterslev J, et al. Evidence-based clinical practice: overview of threats to the validity of evidence and how to minimise them. Eur J Intern Med 2016;32:13–21. 10.1016/j.ejim.2016.03.020 [DOI] [PubMed] [Google Scholar]
- 18.Aberegg SK, Richards DR, O’Brien JM. Delta inflation: a bias in the design of randomized controlled trials in critical care medicine. Crit Care 2010;14:R77. 10.1186/cc8990 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Sidebotham D, Popovich I, Lumley T. A Bayesian analysis of mortality outcomes in Multicentre clinical trials in critical care. Br J Anaesth 2021;127:487–94. 10.1016/j.bja.2021.06.026 [DOI] [PubMed] [Google Scholar]
- 20.Cochrane database of systematic Reviews(ISSN 1469-493X). 2022. Available: https://www.cochranelibrary.com/cdsr/about-cdsr
- 21.Lawson DO, Puljak L, Pieper D, et al. Reporting of methodological studies in health research: A protocol for the development of the methodological study reportIng checklist (MISTIC). BMJ Open 2020;10:e040478. 10.1136/bmjopen-2020-040478 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Mbuagbaw L, Lawson DO, Puljak L, et al. A Tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 2020;20:226. 10.1186/s12874-020-01107-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Higgins J, Eldrigde S, Li T, eds. Including variants of randomized trials. In: Cochrane Handbook for Systematic Reviews of Interventions. 2022. Available: www.training.cochrane.org/handbook [Google Scholar]
- 24.Pallmann P, Bedding AW, Choodari Oskooei B, et al. Adaptive designs in clinical trials: why use them, and how to run and report them. BMC Med 2018;16:29. 10.1186/s12916-018-1017-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Andrade C. The limitations of quasi-experimental studies, and methods for data analysis when a quasi-experimental research design is unavoidable. Indian J Psychol Med 2021;43:451–2. 10.1177/02537176211034707 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions to improve health. BMJ 2000;321:694–6. 10.1136/bmj.321.7262.694 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:71. 10.1136/bmj.n71 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Higgins JPT, Altman DG, Gøtzsche PC, et al. The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. 10.1136/bmj.d5928 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sterne JAC, Savović J, Page MJ, et al. Rob 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898. 10.1136/bmj.l4898 [DOI] [PubMed] [Google Scholar]
- 30.Harbord RM, Egger M, Sterne JAC. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med 2006;25:3443–57. 10.1002/sim.2380 [DOI] [PubMed] [Google Scholar]
- 31.Guyatt GH, Oxman AD, Kunz R, et al. GRADE guidelines 6. rating the quality of evidence—Imprecision. J Clin Epidemiol 2011;64:1283–93. 10.1016/j.jclinepi.2011.01.012 [DOI] [PubMed] [Google Scholar]
- 32.Guyatt GH, Oxman AD, Montori V, et al. GRADE guidelines: 5. rating the quality of evidence—publication bias. J Clin Epidemiol 2011;64:1277–82. 10.1016/j.jclinepi.2011.01.011 [DOI] [PubMed] [Google Scholar]
- 33.Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev 2007:MR000005. 10.1002/14651858.MR000005.pub3 [DOI] [PubMed] [Google Scholar]
- 34.Blackwell SC, Thompson L, Refuerzo J. Full publication of clinical trials presented at a national maternal-fetal medicine meeting: is there a publication bias? Am J Perinatol 2009;26:679–82. 10.1055/s-0029-1220786 [DOI] [PubMed] [Google Scholar]
- 35.Liebeskind DS, Kidwell CS, Sayre JW, et al. Evidence of publication bias in reporting acute stroke clinical trials. Neurology 2006;67:973–9. 10.1212/01.wnl.0000237331.16541.ac [DOI] [PubMed] [Google Scholar]
- 36.Savović J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012;157:429–38. 10.7326/0003-4819-157-6-201209180-00537 [DOI] [PubMed] [Google Scholar]
- 37.Higgins J, Thomas J, Chandler J, et al. Collecting data. In: Li T, Higgins J, Deeks J, eds. Cochrane Handbook for Systematic Reviews of Interventions version. Cochrane, 2022: 3. Available: www.training.cochrane.org/handbook [Google Scholar]
- 38.Khan A, Fahl Mar K, Gokul S, et al. Mortality during US FDA clinical trials in patients with diabetes, hypertension, depression and schizophrenia. World J Biol Psychiatry 2020;21:64–71. 10.1080/15622975.2018.1514465 [DOI] [PubMed] [Google Scholar]
- 39.Duffett M, Choong K, Foster J, et al. High-quality randomized controlled trials in pediatric critical care. Pediatric Critical Care Medicine 2017;18:405–13. 10.1097/PCC.0000000000001144 [DOI] [PubMed] [Google Scholar]
- 40.Leddy L, Sukumar P, O’Sullivan L, et al. An investigation into the factors affecting investigator-initiated trial start-up in Ireland. Trials 2020;21:962. 10.1186/s13063-020-04893-z [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.