Reporting of drug trial funding sources and author financial conflicts of interest in Cochrane and non-Cochrane meta-analyses: a cross-sectional study

Abstract

Objective

To (1) investigate the extent to which recently published meta-analyses report trial funding, author–industry financial ties and author–industry employment from included randomised controlled trials (RCTs), comparing Cochrane and non-Cochrane meta-analyses; (2) examine characteristics of meta-analyses independently associated with reporting funding sources of included RCTs; and (3) compare reporting among recently published Cochrane meta-analyses to Cochrane reviews published in 2010.

Design

Review of consecutive sample of recently published meta-analyses.

Data sources

MEDLINE database via PubMed searched on 19 October 2018.

Eligibility criteria for selecting articles

We selected the 250 most recent meta-analyses listed in PubMed that included a documented search of at least one database, statistically combined results from ≥2 RCTs and evaluated the effects of a drug or class of drugs.

Results

90 of 107 (84%) Cochrane meta-analyses reported funding sources for some or all included trials compared with 21 of 143 (15%) non-Cochrane meta-analyses, a difference of 69% (95% CI 59% to 77%). Percent reporting was also higher for Cochrane meta-analyses compared with non-Cochrane meta-analyses for trial author–industry financial ties (44% versus 1%; 95% CI for difference 33% to 52%) and employment (17% versus 1%; 95% CI for difference 9% to 24%). In multivariable analysis, compared with Cochrane meta-analyses, the odds ratio (OR) for reporting trial funding was ≤0.11 for all other journal category and impact factor combinations. Compared with Cochrane reviews from 2010, reporting of funding sources of included RCTs among recently published Cochrane meta-analyses improved by 54% (95% CI 42% to 63%), and reporting of trial author–industry financial ties and employment improved by 37% (95% CI 26% to 47%) and 10% (95% CI 2% to 19%).

Conclusions

Reporting of trial funding sources, trial author–industry financial ties and trial author–industry employment in Cochrane meta-analyses has improved since 2010 and is higher than in non-Cochrane meta-analyses.

Strengths and limitations of this study.

• The meta-analyses selected for inclusion in our study was a consecutive sample of meta-analyses of drug interventions published in 2016–2018.

• We compared reporting of funding and financial conflicts of interest among trials included in recent Cochrane and non-Cochrane meta-analyses.

• We compared reporting of funding and financial conflicts of interest among trials included in recent Cochrane meta-analyses with Cochrane systematic reviews from 2010.

• We were unable to examine whether meta-analyses published in different types of journals or journals with different impact factors are more or less likely to report on financial conflicts of interest from included trials because most meta-analyses of drug trials are published as Cochrane reviews or in relatively low-impact specialty medicine journals.

• Our study examined only disclosed financial conflicts of interest and did not attempt to identify non-disclosed conflicts.

Introduction

Financial conflicts of interest (FCOIs) can introduce bias in drug trials by influencing how a trial is designed, inclusion and exclusion criteria, choice of drug dosages and comparators, selection of trial outcomes, how analyses are conducted, interpretation of findings, which outcomes are reported and whether trial results are published.^1–10 Drug trials funded by industry are approximately 30% more likely to report favourable efficacy findings than non-industry trials,⁸ and drug trials with principal investigators with FCOIs have higher odds of reporting favourable outcomes than those led by principal investigators without FCOIs, even after controlling for trial funding sources.⁷

Previous studies that have examined meta-analyses of drug trials published in high-impact journals and Cochrane systematic reviews of drug trials have found that funding sources and author FCOIs of included randomised controlled trials (RCTs) were rarely reported.^{11 12} A 2011 study found that only 2 of a sample of 29 (7%) meta-analyses on the effects of drug interventions published in high-impact journals in 2009 reported the funding sources of included drug trials and that none reported trial author–industry financial ties or author–industry employment.¹¹ A second study, published in 2012, examined Cochrane systematic reviews of drug trials and found that only 46 of 151 (30%) eligible reviews published in 2010 reported information on the funding source of some or all included trials, 11 (7%) provided any information on author–industry financial ties and 10 (7%) provided any information on author–industry employment from included trials.¹²

In 2012, the Cochrane Collaboration began to require that Cochrane reviews report trial funding sources and FCOIs of the primary researchers of all included trials in the characteristics of included studies table (Methodolgical Expectations of Cochrane Intervention Reviews (MECIR), standards R69 and R70).^{13 14} The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement, however, has not been updated since its publication in 2009^{15 16} and does not address the reporting of trial funding or author FCOIs of trials included in systematic reviews and meta-analyses.

We do not know of any studies that have compared reporting among Cochrane meta-analyses with meta-analyses published in other journals or examined whether reporting in Cochrane reviews has improved since Cochrane implemented its reporting policy. The objectives of the present study were to: (1) investigate the extent to which Cochrane and non-Cochrane meta-analyses of drug trials report trial funding sources, author–industry financial ties and author–industry employment; (2) examine characteristics of meta-analyses that are independently associated with reporting funding sources of included RCTs; and (3) compare reporting among recently published Cochrane meta-analyses to reporting from Cochrane systematic reviews published in 2010,¹¹ prior to implementation of Cochrane’s reporting policy.

Methods

The methods for the present study were based on our previous study of reporting of funding sources, author–industry financial ties and author–industry employment from trials included in Cochrane systematic reviews published in 2010; however in the present study, we included only Cochrane reviews that contained a meta-analysis, whereas in the previous study, all Cochrane reviews that included results from at least one RCT were eligible.¹² Because of this difference, in our comparison, in addition to main analyses, we conducted sensitivity analyses that only included systematic reviews with meta-analyses from the previous study. A study protocol was developed prior to initiating the present study and was posted on the Open Science Framework (https://osf.io/njk5w/).

Selection of meta-analyses

Meta-analyses in any language were eligible if they: (1) included a documented search for eligible RCTs using at least one database, (2) statistically combined results from ≥2 RCTs and (3) evaluated the efficacy/effectiveness or harm of a drug or class of drugs against an alternative treatment (eg, placebo, alternative drug, non-pharmacological treatment) or no treatment. Meta-analyses that only assessed different methods of administration, dosages or dosage schedules of the same drug were excluded. Drugs were defined broadly to include biologicals and vaccines but not nutritional supplements or medical devices without a drug component. Meta-analyses that investigated a combination of pharmacological and non-pharmacological interventions or interventions that may or may not involve a drug (eg, amnioinfusion) were included if a study group was exclusively given a drug intervention or if the meta-analysis assessed the addition of a drug to a treatment received by both intervention and control groups. Interventions were classified as having a drug component if any form of the active ingredient (eg, dosage, route, strength, compound) was listed as an approved or discontinued brand name, generic drug or therapeutic biological product by the US Food and Drug Administration (FDA) as listed in the Drugs@FDA database at the time of review.¹⁷ If an agent was not listed in the Drugs@FDA database and was classified by the FDA as a non-drug (eg, food additive, supplement), then it was not considered a drug. If an agent was not regulated as a drug and was not listed as a non-drug by the FDA, drug status was determined based on consensus among investigators using publicly available sources that provided information on the agent.

We searched the MEDLINE database via PubMed on 19 October 2018 using a search developed by a medical librarian (see online supplementary eMethods 1 for strategy). Citations were uploaded into the systematic review software DistillerSR (Evidence Partners, Ottawa, Canada), which was used to code and track results. Two investigators independently evaluated titles and abstracts for potential eligibility. Full texts of titles and abstracts deemed potentially eligible by either investigator were then reviewed by two investigators independently. Disagreements at the full-text level were resolved through consensus with a third investigator consulted as necessary. Because we sought to include the most recently published meta-analyses that met eligibility criteria, prior to reviewing, citations were organised by PubMed reference identification numbers with the most recent first. Title and abstract and full-text reviews were conducted sequentially until we obtained our desired number of included meta-analyses based on our power analysis.

Data extraction

For each eligible meta-analysis, one reviewer initially extracted all data into a predefined form in DistillerSR, and a second reviewer validated all extracted data using the DistillerSR Quality Control function. Discrepancies were resolved by consensus and consultation with a third investigator, if needed. For each included meta-analysis, reviewers extracted first author last name; year of publication; journal name; Clarivate Analytics 2017 journal impact factor; journal specialty area based on Clarivate Analytics classification; whether it was a Cochrane meta-analysis published in the Cochrane Database of Systematic Reviews or elsewhere; funding source for the meta-analysis and author–industry financial ties and employment; reporting in the meta-analysis of trial funding sources, trial author–industry financial ties and trial author–industry employment; and whether the meta-analysis referenced a published protocol or contained a PROSPERO registration number. If a registration number was not provided, we searched the PROSPERO website (https://www.crd.york.ac.uk/PROSPERO/) using key terms from the published article, then attempted to match the principal investigator, funding source, intervention, non-intervention comparator group and design from the article to registrations obtained in the search.

To extract information on meta-analysis funding source, meta-analysis author–industry financial ties and meta-analysis author–industry employment and to determine whether trial funding sources, trial author–industry financial ties and trial author–industry employment were reported in the meta-analysis, for each included meta-analysis, reviewers examined all text, tables, figures, appendices, disclosure statements, acknowledgements and any online supplemental material, published with the manuscript or linked to the manuscript. Funding sources for meta-analyses were classified as: (1) non-industry (eg, public granting agency and private not-for-profit granting agency), (2) pharmaceutical industry, (3) combined pharmaceutical industry and non-industry, (4) no funding or (5) not reported. Financial ties of meta-analysis authors to industry were defined per the International Committee of Medical Journal Editors Uniform Disclosure Form for Potential Conflicts of Interest¹⁸ and included current or former board membership, current or former consultancy, current or former industry employment, expert testimony, industry grants (issued or pending), payment for lectures including service on speakers bureaus, payment for manuscript preparation, patents (planned, pending or issued), royalties, payment for development of educational presentations, stock or stock options, travel reimbursement or other relationships with industry, as disclosed in the review. Of these, we specifically coded if industry employees were part of the author group. If a meta-analysis did not contain a disclosure statement, meta-analysis author–industry financial ties were coded as not reported.

For reporting of: (1) trial funding sources, (2) trial author–industry financial ties and (3) trial author–industry employment, meta-analyses were coded as: (1) reporting for all included trials; (2) reporting for some, but not all, included trials (partial reporting); or (3) not reporting. Meta-analyses that included data from a pharmaceutical industry database or noted that trial drugs were supplied by the manufacturers for certain trials, but that did not make any explicit statement of trial funding sources, were coded as not reporting. For meta-analyses that reported information on funding sources or author FCOIs from included trials, either fully or partially, we recorded where in the meta-analysis the information was reported. Specifically, we recorded whether the information was reported in the abstract, lay summary, risk of bias material (text, figure or table, both), main text other than risk of bias, elsewhere in the main document (eg, characteristics of included studies table, other table, footnote of a table), or in an online appendix (see online supplementary eMethods 2.

Power analysis

To determine the number of meta-analyses to target, we first calculated the number of included meta-analyses that would be needed for 80% power to find a statistically significant difference if there were a 20% difference in reporting trial funding sources based on meta-analysis characteristics, with α=0.05. We varied the rates of reporting from 10% versus 30% to 70% versus 90% and considered scenarios where the proportion of reporting meta-analyses with each characteristic (eg, high-impact journals vs low-impact journals) was 50% vs 50% and 30% versus 70%. For a two-tailed binomial test with α=0.05, the maximum number of included meta-analyses needed in any scenario was 239. Because the consequence of overpowering the study was additional labour and not risk to human participants, we rounded this number up to 250 meta-analyses (see online supplementary eMethods 3).

Statistical analyses

We presented characteristics of included meta-analyses descriptively, including funding sources and FCOIs. We determined the proportion of meta-analyses that reported trial funding source, author–industry financial ties and author–industry employment of included trials for: (1) all included trials, (2) some, but not all, included trials and (3) no included trials, along with 95% CIs. We compared the difference between the proportion of recently published Cochrane meta-analyses that reported study funding, author–industry financial ties and author–industry employment from included RCTs with recently published non-Cochrane meta-analyses and with Cochrane systematic reviews published in 2010. Because the present study included meta-analyses only, but the previous study of Cochrane reviews included systematic reviews with or without meta-analyses,¹² we conducted a sensitivity analysis in which we excluded Cochrane systematic reviews from 2010 that did not include a meta-analysis and would not have been eligible for inclusion in the present study. We calculated 95% CIs for all differences.¹⁹

To assess the relationship between meta-analysis characteristics and reporting of funding sources for some or all included trials, versus not reporting, we fit unadjusted (bivariate) and adjusted (multivariate) logistic regression models with all predictors using the glm function in R (R version 3.2.3; RStudio Version 1.0.136).^{20 21} The predictor variables that were considered in bivariate and adjusted analyses were: (1) combined category (Cochrane, specialty medicine, general medicine and multidisciplinary) and impact factor of the journal in which the meta-analysis was published; and (2) whether there was industry funding for the meta-analysis or any FCOI disclosed by meta-analysis authors. We combined journal category and impact factor because of the small number of journals in some categories and the small number of journals with an impact factor greater than that of Cochrane. Thus, meta-analyses were categorised as: (1) low-impact (≤3.0) specialty medicine journals, (2) low-impact (≤3.0) general medicine or multidisciplinary journals, (3) medium-impact (3.1–6.7) specialty medicine journals, (4) high-impact (>6.8) specialty medicine or general medicine journals and (5) Cochrane meta-analyses (impact factor=6.8; reference category). Because 28 of 33 meta-analyses in general medicine journals were from a single journal (Medicine) and not necessarily representative of general medicine as a category, and because 9 of the 10 meta-analyses published in multidisciplinary science journals were published in a single journal (PLOS One), we combined general medicine and multidisciplinary journals.

Our initial protocol indicated that, if possible, we would include in the logistic regression model the year of publication of the meta-analysis and whether there was meta-analysis funding by industry, meta-analysis author–industry financial ties and meta-analysis author–industry employment, separately. However, 246 of 250 included meta-analyses were published in 2017–2018, and only three meta-analyses had industry funding; thus, we did not include year of publication, and we grouped meta-analysis funding source and author FCOIs into a single variable (no FCOIs including funding source vs any FCOI). Additionally, we only conducted a multivariable analysis for the reporting of funding sources of included RCTs and not for reporting of author–industry financial ties and author–industry employment, because there were not enough examples of meta-analyses that reported author–industry financial ties and author–industry employment.

Patient and public involvement

Patients and members of the public were not involved in the design, conduct, reporting, or plan for dissemination of our research.

Results

Selection of eligible meta-analyses

Our initial search of PubMed without date restrictions retrieved 9725 unique citations. To select 250 eligible meta-analyses, working backwards from the most recent, a total of 401 citations were screened for eligibility; 64 were excluded at the title and abstract level and 76 at the full-text level (see figure 1).

Figure 1.

Flow diagram of selection of eligible meta-analyses.

As shown in table 1, of the 250 included meta-analyses, 107 (43%) were Cochrane reviews, all of which were published in the Cochrane Database of Systematic Reviews. Among the 143 non-Cochrane meta-analyses, 33 (23%) were published in general medicine journals (including 28 in the journal Medicine), 100 (70%) in specialty medicine journals and 10 (7%) in multidisciplinary journals (including nine in PLOS One). The mean number of included RCTs for both Cochrane and non-Cochrane meta-analyses was approximately 20. Among the 143 non-Cochrane meta-analyses, 25 (17%) referenced a published protocol or were registered in PROSPERO, and 106 (74%) were published in a journal with impact factor ≤3.

Table 1.

Characteristics of included meta-analyses

	Cochrane meta-analyses (n=107)	Non-Cochrane meta-analyses (n=143)
Year of publication, N (%)
2016	0	4 (3)
2017	22 (21)	31 (22)
2018	85 (79)	108 (76)
Number of Included RCTs, mean±SD	21.4±24.4	19.6±46.4
Registered in PROSPERO or published protocol, N (%)	107 (100)	25 (17)*
Impact factor, mean±SD	6.8±0	3.6±5.4
≤3	0	106 (74%)
3.1–6.7	0	27 (19%)
6.8	107 (100%)	0
>6.8	0	10 (7.0%)
Meta-analysis funding sources, N (%)
Not reported	4 (4)†	39 (27)
Industry	0	3 (2)
Non-industry	93 (87)	55 (38)
No funding	10 (9)	46 (32)
Meta-analysis author financial ties to industry (including employment), N (%)‡
Not reported	1 (1)	11 (8)
No authors with reported financial ties	70 (65)	117 (81)
≥1 author with reported financial ties	36 (34)	15 (10)
Proportion of authors with financial ties, mean±SD§	11%±17%	4%±15%
Journal category, N (%)
Cochrane review	107 (100)	0
Specialty medicine	0	100 (70)
General medicine (non-Cochrane)¶	0	33 (23)
Multidisciplinary**	0	10 (7)

*One meta-analysis reported that they registered in PROSPERO but did not provide a registration number and one could not be found. We contacted the authors, and they did not provide us with further information; thus, this was coded as not registered.

†Only three included meta-analyses reported author–industry employment, and these were grouped with author–industry financial ties for this table.

‡Cochrane reviews typically have a ‘Sources of Support’ section with funding information. These reviews did not include that section.

§Proportion of authors with financial ties or employment of those that reported.

¶Of the 33 included general medicine journals, 28 were published in the journal ‘Medicine’.

**Of the 10 journals classified as multidisciplinary, nine were published in the journal ‘PLOS One’.

††Classifications for specialty medicine journals (note that some journals had more than one classification): anaesthesiology: n=3; biochemistry and molecular biology: n=1; biotechnology and applied microbiology: n=2; cardiac and cardiovascular systems: n=7; cell biology: n=1; chemistry, medicinal: n=4; chemistry, multidisciplinary: n=2; clinical neurology: n=6; critical care medicine: n=2; dermatology: n=3; emergency medicine: n=2; endocrinology and metabolism: n=2; gastroenterology and hepatology: n=6; genetics and heredity: n=1; haematology: n=2; immunology: n=6; infectious diseases: n=3; integrative and complementary medicine: n=1; medicine, research and experimental: n=3; microbiology: n=2; neurosciences: n=3; no classification: n=2; obstetrics and gynaecology: n=4; oncology: n=11; ophthalmology: n=3; orthopaedics: n=6; parasitology: n=1; peripheral vascular disease: n=5; pharmacology and pharmacy: n=13; physiology: n=1; psychiatry: n=4; psychology: n=1; reproductive biology: n=1; respiratory system: n=6; rheumatology: n=3; sport sciences: n=1; surgery: n=11; toxicology: n=2; tropical medicine: n=1; urology and nephrology: n=1.

Of the 250 meta-analyses, 3 (1%) reported being funded by industry, 148 (59%) reported funding from non-industry sources, 56 (22%) reported no funding and 43 (17%) did not report funding source; 3 (1%) had at least one author who reported current industry employment, 51 (20%) had at least one author that reported other financial ties with industry, 187 (75%) reported that there were no authors with FCOIs and 12 (5%) did not report any information about author FCOIs. Characteristics of each of the 250 included meta-analyses are shown in online supplementary eTable 1.

Reporting in meta-analyses of funding sources and author FCOIs from included drug trials

As shown in table 2, 111 of the 250 (44%) included meta-analyses reported the funding sources for some or all included trials, 49 (20%) reported author–industry financial ties for some or all included trials and 19 (8%) reported author–industry employment for some or all included trials. Of the 107 Cochrane meta-analyses, 90 (84%) reported funding sources for some or all included trials compared with 21 of 143 (15%) non-Cochrane meta-analyses, a difference of 69% (95% CI 59% to 77%); 47 (44%) Cochrane meta-analyses reported author–industry financial ties for some or all included trials compared with 2 (1%) non-Cochrane meta-analyses, a difference of 43% (95% CI 33% to 52%); 18 (17%) Cochrane meta-analyses reported, fully or partially (for some but not all trials), author–industry employment compared with 1 (1%) non-Cochrane meta-analysis, a difference of 16% (95% CI 9% to 24%).

Table 2.

Summary of reporting patterns of disclosed funding source and author-industry FCOI from included RCTs

	Number of meta-analyses reporting funding sources of included RCTs			Number of meta-analyses reporting author financial ties of included RCTs			Number of meta-analyses reporting author–industry employment of included RCTs
	Full	Partial	Full or partial	Full	Partial	Full or partial	Full	Partial	Full or partial
Recently published meta-analyses:
Cochrane (n=107), N (%)	70 (65)	20 (19)	90 (84)	24 (22)	23 (21)	47 (44)	1 (1)	17 (16)	18 (17)
Non-Cochrane (n=143), N (%)	14 (10)	7 (5)	21 (15)	1 (1)	1 (1)	2 (1)	0	1 (1)	1 (1)
Difference in reporting between Cochrane and non-Cochrane meta-analyses, % (95% CI)	56 (44 to 65)	14 (6 to 23)	69 (59 to 77)	22 (14 to 31)	21 (13 to 30)	43 (33 to 52)	1 (−2 to 5)	15 (9 to 23)	16 (9 to 24)
2010:
All Cochrane systematic reviews (n=151), N (%)*	30 (20)	16 (11)	46 (30)	2 (1)	9 (6)	11 (7)	0	10 (7)	10 (7)
Difference in reporting between recently published Cochrane meta-analyses versus Cochrane systematic reviews published in 2010, % (95% CI)	46 (34 to 56)	8 (−1 to 18)	54 (42 to 63)	21 (13 to 30)	16 (7 to 25)	37 (26 to 47)	1 (−2 to 5)	9 (2 to 18)	10 (2 to 19)
2010:
Cochrane meta-analyses (n=119), N (%)	21 (19)	15 (13)	36 (30)	0 (0)	7 (6)	7 (6)	0 (0)	7 (6)	7 (6)
Difference in reporting between recently published Cochrane meta-analyses versus Cochrane meta-analyses published in 2010, % (95% CI)	48 (36 to 58)	6 (−3 to 16)	54 (42 to 63)	22 (15 to 31)	16 (7 to 25)	38 (27 to 48)	1 (−2 to 5)	10 (2 to 19)	11 (3 to 20)

*Results from Roseman et al.¹²

FCOI, financial conflicts of interest; RCT, randomised controlled trial.

Among the 90 Cochrane meta-analyses that reported funding sources for some or all included trials, 77 (86%) provided this information in the characteristics of included studies table, including 23 (26%) that also included it in the assessment of risk of bias of included trials; 7 (8%) included it in the risk of bias assessment and at least one other place, but not the characteristics of included studies table, and 6 (7%) reported only as part of the risk of bias assessment. In total, 36 (40%) reported in the context of the risk of bias assessment (see online supplementary eTable 2 for reporting for all 250 included meta-analyses).

Factors associated with reporting FCOIs from included trials in multivariable analysis

As shown in table 3, the OR for reporting funding sources for some or all included RCTs among non-Cochrane meta-analyses was ≤0.11 compared with Cochrane meta-analyses for all journal category and impact factor combinations. Meta-analyses with any declared FCOI (OR 1.29, 95% CI 0.53 to 3.19) and meta-analyses for which the presence of FCOIs was not reported (OR 1.18, 95% CI 0.40 to 3.44) did not differ significantly in reporting compared with those with no declared FCOIs.

Table 3.

Factors associated with reporting funding sources of included RCTs

	Proportion that reported some or all declared funding sources from included RCTs	Unadjusted OR (95% CI)	Adjusted OR (95% CI)
FCOI of meta-analysis (including meta-analysis funding)
Reference=no FCOI	67/151 (44%)
Any disclosed FCOI	35/51 (69%)	2.74 (1.42 to 5.49)	1.29 (0.53 to 3.19)
Not reported	9/48 (19%)	0.29 (0.12 to 0.62)	1.18 (0.40 to 3.44)
Impact factor and journal type
Reference=Cochrane	90/107 (84%)
Specialty impact factor ≤3*	4/65 (6%)	0.01 (<0.01 to 0.03)	0.01 (<0.01 to 0.04)
General (n=31) or multidisciplinary (n=10) impact factor ≤3	4/41 (10%)	0.02 (<0.01 to 0.06)	0.02 (<0.01 to 0.06)
Specialty impact factor 3.1–6.7†	10/27 (37%)	0.11 (0.04 to 0.28)	0.11 (0.04 to 0.28)
Specialty (n=8) or general (n=2) impact factor >6.8	3/10 (30%)	0.08 (0.02 to 0.32)	0.08 (0.02 to 0.32)

*Two meta-analyses were from journals that did not have an impact factor, and these were coded as having an impact factor of 0.5 for our analyses.

†There were no multidisciplinary or general medicine journals with an impact factor of 3.1–6.7.

‡Not reported included meta-analyses for which the presence of FCOI could not be determined because either meta-anlaysis funding, meta-analysis author FCOI or both were not reported.

FCOI, financial conflicts of interest; RCT, randomised controlled trial.

Comparison of recent Cochrane meta-analyses versus Cochrane reviews published in 2010

Reporting of funding sources for some or all included trials improved from 30% in Cochrane reviews of drug trials published in 2010 to 84% in recently published Cochrane meta-analyses, an improvement of 54% (95% CI 42% to 63%). Reporting of author–industry financial ties for some or all included trials improved from 7% in 2010 to 44% in recent meta-analyses, a 37% change (95% CI 26% to 47%). Reporting of author–industry employment for some or all included trials improved from 7% in 2010 to 17% in recent meta-analyses (10%; 95% CI 2% to 19%). Results did not change when the comparison was restricted to Cochrane reviews published in 2010 that included a meta-analysis (see table 2). Figure 2 summarises reporting among recently published Cochrane and non-Cochrane meta-analyses and Cochrane reviews from 2010.

Figure 2.

Percentage of recently published Cochrane and non-Cochrane meta-analyses and 2010 Cochrane systematic reviews that reported included trial funding source, author–industry financial ties and author–industry employment for some or all included trials.

Discussion

Principal findings

We reviewed the 250 most recent meta-analyses of drug treatments listed in PubMed at the time of our search. Of these, 107 (43%) were Cochrane reviews, 100 (40%) were published in specialty medicine journals and 43 (17%) were published in general medicine or multidisciplinary journals, including 28 in Medicine and 9 in PLOS One. Of the 143 non-Cochrane meta-analyses, 106 (74%) were published in journals with impact factor ≤3.

Among Cochrane meta-analyses, 84% reported funding sources for some or all included RCTs compared with 15% of non-Cochrane meta-analyses. Cochrane meta-analyses were also more likely than non-Cochrane meta-analyses to report author–industry financial ties (44% vs 1%) and author–industry employment (17% vs 1%).

In 2010, only 30% of 151 Cochrane systematic reviews of drug treatments reported trial funding sources. ¹² This improved to 84% among recent Cochrane meta-analyses. Cochrane reviews also improved reporting of author–industry financial ties and author–industry employment of included RCTs from 7% to 44% and from 7% to 17%. It is possible that the reason that few meta-analyses reported author–industry employment is because some may have assumed that author–industry employment would be considered a type of author–industry financial tie and did not report employment separately, whereas we considered author–industry financial ties and employment separately.

Among the 90 Cochrane meta-analyses that reported funding sources of included trials in the present study, 86% included the information in the characteristics of included studies table, as required by Cochrane, and 40% included the information in the risk of bias assessment.

Findings in context

In 2012, soon after our previous results showed that few Cochrane systematic reviews of drug trials reported funding sources and author FCOIs of included trials,¹² the Cochrane Collaboration began to require that trial funding sources and FCOIs be reported for every included RCT in the characteristics of included studies table.^{13, 14} Reporting of trial funding sources among recent Cochrane meta-analyses has not reached 100%, and work is needed to improve the reporting of other types of author FCOIs, which was under 50% despite being required by Cochrane. Nonetheless, the improvements documented in the present study are substantial, both compared with previous Cochrane reviews and with contemporary non-Cochrane meta-analyses. Cochrane is a global organisation consisting of a large number of different review and methods groups that span numerous fields of health research. This diversity suggests that changes that have occurred likely resulted from change in the mandatory reporting requirements for Cochrane reviews and widespread adoption by the organisation. We did not examine whether performance differed by review groups or whether updated reviews based on initial protocols that pre-dated Cochrane’s reporting policy may have been less likely to fully report. It is possible that reporting in Cochrane reviews could be improved even further by ensuring that all review groups are fully compliant and that even reviews with older initial protocols report per Cochrane’s current MECIR standards, as required by Cochrane.¹⁴

The improved performance in reporting in Cochrane reviews suggests the possibility that other journals could improve the transparency of reporting of trial funding and trial author FCOI in evidence syntheses by adopting similar reporting requirements. Most journals that specify reporting requirements stipulate that authors follow reporting standards for meta-analyses articulated in the PRISMA statement. The current version of the PRISMA statement does not address reporting of trial funding sources and FCOIs of trial authors by investigators who publish systematic reviews and meta-analyses.^{16, 17} The forthcoming updated PRISMA statement will likely encourage that trial funding and trial author FCOIs, be reported; however, it will not be required and will not be included as a checklist item (D Moher, personal communication, 2020). Including encouragement to report trial FCOIs in the updated PRISMA reporting standards could result in authors being better informed about the need for reporting trial funding sources and FCOIs and therefore improve reporting, but the lack of a requirement and checklist item will undoubtedly limit this effect.

Members of our research team have previously recommended that risk of bias from trial funding and trial author FCOIs be included in the Cochrane Risk of Bias Tool based on evidence that links trial sponsorship and trial author FCOIs to outcomes.¹¹ This recommendation was debated at a Cochrane Methods Symposium in 2013, but consensus was not reached for inclusion.^{13, 22} The present study found that 40% of Cochrane meta-analyses that reported on FCOIs from included trials included this as part of a risk of bias assessment, even though this has not been recommended by Cochrane. Currently, a new tool, the Tool for Addressing Conflicts of Interest in Trials (TACIT),²³ which specifically addresses risk of bias from industry sponsorship of trials and author–industry financial ties and employment, is being developed for inclusion in Cochrane reviews. Once the TACIT tool is completed, risk of bias from trial funding and trial author FCOIs will be explicitly considered in Cochrane reviews and, potentially, in non-Cochrane reviews, as well. Meanwhile, authors should, at a minimum, describe FCOIs and discuss the degree to which they may influence confidence in findings.

Strengths and limitations

A strength of the present study is that we assessed reporting in a large number of recently published meta-analyses, including 107 Cochrane meta-analyses, which allowed us to compare reporting practices among Cochrane and non-Cochrane meta-analyses and recent Cochrane meta-analyses with Cochrane systematic reviews from 2010. However, there are limitations that should be considered. First, we used impact factor as a rough proxy of the quality of the meta-analyses included, but journal impact factor is very much an imperfect proxy; it does not necessarily reflect the quality of the methods of the included meta-analyses. Rating meta-analysis quality in all included meta-analyses was beyond the scope of our study, given the resources that would have been required. Second, since most meta-analyses of drug trials are published as Cochrane reviews or in relatively low-impact specialty medicine journals, we were not able to conduct robust assessments of whether meta-analyses published in different types of journals or journals with different impact factors are more or less likely to report on trial funding and trial author FCOIs for included drug trials. The vast majority of meta-analyses published in general medicine journals were from a single journal (Medicine), which further limited our ability to examine this factor. However, the meta-analyses included in our study constituted a consecutive sample of the most recent meta-analyses listed in PubMed and, thus, represented all meta-analyses of drug interventions listed in PubMed during the study period. Third, our study examined only disclosed FCOIs. A surprising finding was that a higher proportion of Cochrane meta-analysis authors indicated that they had FCOIs compared with non-Cochrane authors; it is not known if this reflects greater industry involvement among Cochrane authors or a higher propensity to report transparently and completely among this group of authors. Fourth, information about FCOIs from included RCTs was not extracted from the RCT publications. Fifth, our previous study of Cochrane reviews from 2010 included all systematic reviews, whereas the present study was restricted to reviews with meta-analyses. However, a sensitivity analysis showed that results did not change when we compared recent results to those from 2010 that were restricted to reviews with a meta-analysis.

Conclusions and policy implications

In summary, the percentage of recent Cochrane meta-analyses on the effects of drug interventions that transparently reported funding sources and trial author–industry financial ties and employment for included trials far exceeds reporting in other journals. It also far exceeds reporting in Cochrane systematic reviews published in 2010, before the implementation by Cochrane of its policy requiring the reporting of trial funding sources and author–industry FCOIs. These results suggest that it is possible to achieve more transparent reporting of FCOIs from trials included in meta-analyses. We encourage the uptake of reporting recommendations in the forthcoming updated PRISMA statement.²⁴ We also encourage the adoption of Cochrane’s new TACIT tool²³ by journals and authors in order to assess trial funding sources and author FCOIs as risks of bias. Continued non-disclosure of FCOIs when evidence is synthesised in meta-analyses misleads readers of medical journals into believing that there is not risk of bias from FCOIs to be considered, even though an increasingly robust evidence base tells us that this is often not the case.^{7, 8}