Reporting of financial and non-financial conflicts of interest by authors of systematic reviews: a methodological survey

Abstract

Background

Conflicts of interest may bias the findings of systematic reviews. The objective of this methodological survey was to assess the frequency and different types of conflicts of interest that authors of Cochrane and non-Cochrane systematic reviews report.

Methods

We searched for systematic reviews using the Cochrane Database of Systematic Reviews and Ovid MEDLINE (limited to the 119 Core Clinical Journals and the year 2015). We defined a conflict of interest disclosure as the reporting of whether a conflict of interest exists or not, and used a framework to classify conflicts of interest into individual (financial, professional and intellectual) and institutional (financial and advocatory) conflicts of interest. We conducted descriptive and regression analyses.

Results

Of the 200 systematic reviews, 194 (97%) reported authors' conflicts of interest disclosures, typically in the main document, and in a few cases either online (2%) or on request (5%). Of the 194 Cochrane and non-Cochrane reviews, 49% and 33%, respectively, had at least one author reporting any type of conflict of interest (p=0.023). Institutional conflicts of interest were less frequently reported than individual conflicts of interest, and Cochrane reviews were more likely to report individual intellectual conflicts of interest compared with non-Cochrane reviews (19% and 5%, respectively, p=0.004). Regression analyses showed a positive association between reporting of conflicts of interest (at least one type of conflict of interest, individual financial conflict of interest, institutional financial conflict of interest) and journal impact factor and between reporting individual financial conflicts of interest and pharmacological versus non-pharmacological intervention.

Conclusions

Although close to half of the published systematic reviews report that authors (typically many) have conflicts of interest, more than half report that they do not. Authors reported individual conflicts of interest more frequently than institutional and non-financial conflicts of interest.

Strengths and limitations of this study.

• First methodological survey of a large and representative sample of systematic reviews evaluating the frequency and different types of conflicts of interest that authors of systematic reviews report.

• Use of a comprehensive conflicts of interest framework that allowed assessment of the different types of conflicts of interest, including non-financial conflict of interest.

• Use of systematic and transparent methods, for example, duplicate and independent processes in screening and data collection.

• Includes systematic reviews limited to the clinical field, that is, our results may not apply to other fields such as health policy.

Background

According to the Institute of Medicine, a conflict of interest (COI) is ‘a financial or intellectual relationship that may impact an individual's ability to approach a scientific question with an open mind’.¹ The field of healthcare research has recognised, studied and considered financial relationships when setting COI disclosures and management policies.^2–4 Non-financial COIs, such as intellectual, professional and institutional, are increasingly gaining attention.^5–7

In an attempt to reduce bias associated with COI, institutions, journals and guideline development groups are increasingly requiring authors to disclose COI.^8–12 The World Association of Medical Editors policy statement on COI invites journals to provide authors with COI definitions and with instructions on what and how the different types of COI must be disclosed.¹³

Generally, authors conduct systematic reviews to ensure that clinical decision-making is based on the best available empirical evidence.^14–16 Such reviews also help in defining research gaps and directing funding.¹⁷ However, a number of studies have found that systematic review findings may be biased by COI.^18–21 This in turn may lead to suboptimal decisions made by patients, clinicians and policymakers.²² Given that COIs have the potential to bias the findings of systematic reviews,²³ understanding their nature and frequency among authors of systematic reviews is important. Therefore, we assessed the frequency and different types of conflicts of interest that authors of Cochrane and non-Cochrane systematic reviews report.

Methods

Design overview and definitions

This study consists of a methodological survey using systematic methods for study selection and data abstraction. The study involved no human participants and required no ethical approval.

We defined Cochrane systematic reviews as reviews published in the Cochrane Database of Systematic Reviews. We defined non-Cochrane systematic reviews as systematic reviews published in the Core Clinical Journals.

We defined a COI disclosure as the reporting of whether a COI exists or not (ie, includes a statement of the absence of COI). We developed a framework of the different types of COI based on our review of the literature and of the International Committee of Medical Journal Editors (ICMJE) COI disclosure form²⁴ (figure 1 and online supplementary appendix 1 present more details). We developed clear and specific definitions and instructions. The framework included the following types of COI:

1. Individual financial COI,

2. Individual professional COI,

3. Individual intellectual COI,

4. Institutional financial COI,

5. Institutional advocatory COI,

6. ‘Other types’ of COI.

Figure 1.

Conflict of interest framework.

We coined the word ‘loogly’ to label any additional statement in the COI disclosure that attempts to downplay a disclosed relationship by suggesting that it is unrelated to COI, for example, ‘this relationship did not influence his prescription of the drug’.

Eligibility criteria

We included articles meeting the following criteria:

• Described as a systematic review or a meta-analysis;

• Included a search strategy of at least one database;

• Planned to include randomised clinical trials or controlled clinical trials comparing at least two alternative therapeutic interventions in humans;

• Was published in English in 2015.

We excluded methodological systematic reviews, systematic reviews on causal associations or on diagnostic accuracy, conference abstracts and research letters.

Search strategy

We searched for Cochrane systematic reviews using the Cochrane Database of Systematic Reviews in June 2015. We searched for non-Cochrane reviews using Ovid MEDLINE (In-Process and Other Non-Indexed Citations and Ovid MEDLINE). We limited our search to the 119 Core Clinical Journals (Abridged Index Medicus (AIM)) and to the year 2015. Also, we applied the systematic review filter designed by the Health Information Research Unit of McMaster University, Hamilton, Ontario, Canada. Online supplementary appendix 2 presents the search strategy for each database.

Selection process

We drew a random sample from the set of citations captured by the literature search to undergo the selection process. We performed the random selection of citations by using an online sequence generator (http://www.random.org/sequences). Two review authors screened in duplicate and independently the titles and abstracts of citations against the inclusion criteria. We acquired the full text of the citations judged as potentially eligible by at least one of the two reviewers.

The same team of review authors screened the full texts in duplicate and independently. The authors resolved disagreements by discussion, involving a third review author when required. We recorded reasons for exclusion, and summarised the search and the results of the selection process using a PRISMA flow diagram.²⁵ The review authors working on study selection completed calibration exercises.

Data extraction process

We collected and managed study data using Research Electronic Data Capture (REDCap) tools hosted at the American University of Beirut.²⁶ REDCap is a secure, web-based application designed to support data capture for research studies.

We developed and pilot-tested a standardised data extraction form with detailed instructions. Seven data extractors completed calibration exercises. They were organised into six teams of two reviewers who extracted data in duplicate and independently. They compared their results and resolved disagreements through discussion, involving a third review author when required.

Data extracted

We extracted information about the following characteristics of the systematic review:

• Type of review (Cochrane vs non-Cochrane);

• Number of systematic review authors;

• Number of included trials;

• Number of participants per arm reported by the included trials;

• Type of intervention: pharmacological, surgical/invasive procedure, lifestyle intervention, screening/diagnostic intervention, psychotherapeutic intervention, rehabilitation, other;

• Type of control: active control (similar to above types of interventions) versus non-active control (no treatment, placebo, sham procedure, waiting list or standard of care);

• Risk of bias assessment tool: Cochrane risk of bias tool, a revised Cochrane risk of bias tool, Jadad's scale, other tool, no tool;

• Quality of evidence assessment tool: GRADE, other system or no system.

In addition, we extracted information about the characteristics of the reported funding of the systematic review:

• Whether authors reported funding, reported no funding or did not report any information on funding;

• Reported source(s) of funding (see online supplementary appendix 3 for more details on categories and definitions of the sources of funding);

• Whether the role of the funder was reported for: protocol/design of the study, data collection, data analysis and interpretation, preparation/review/approval of the manuscript, decision to submit the manuscript, management, team assembly, other, not involved.

Finally, we extracted information regarding the characteristics of the reported COI disclosures (per categories defined above and in online supplementary appendix 1) of the systematic review authors:

• Form in which COI disclosures were provided (eg, a narrative statement, an online document);

• Number of authors per paper who report any type of COI;

• Number of authors per paper who report each specific type of COI and, when applicable, the different subtypes of COI;

• Whether the paper reports relevant characteristics of the COI (eg, source, monetary value, duration);

• Number of authors who have the same disclosures or discrepant disclosures reported in the online documents and in the main documents;

• Number of authors who have the same disclosures or discrepant disclosures reported in the provided documents and in the main documents;

• COI disclosures described as available on request;

• Number of authors per paper providing a ‘loogly’ statement.

Data analysis

We assessed agreement between reviewers for inclusion of systematic reviews at the full-text screening stage using chance-corrected agreement (κ statistic).

For the included systematic reviews, we conducted descriptive analyses of the general characteristics of the systematic reviews, the characteristics of the reported funding of the systematic review, and the characteristics of the reported COI disclosures. We analysed the data combined as well as stratified by type of review (Cochrane vs non-Cochrane).

We tested the continuous variables for normality of distribution using the Kolmogorov-Smirnov test. We presented data of non-normally distributed variables as medians and quartiles. The distributions of COI-related variables were highly skewed, showing many papers with no authors reporting a COI and some papers with many authors reporting a COI. Therefore, we opted to present them for each type of COI in a tabular format as follows:

• The percentages of papers with at least one author reporting COI (denominator excluding papers that did not provide a COI statement);

• For papers with at least one author reporting the type of COI (denominator excluding papers that did not provide a COI statement and papers that reported absence of COI for all authors): the median and IQR of the percentages of authors per paper reporting COI.

We tested whether the differences between the Cochrane and non-Cochrane reviews for the percentage of papers with at least one author reporting COI were statistically significant.

We present results for categorical variables as frequencies and percentages, and analysed them using the χ² test or Fisher's exact test if the expected event number was <5. We considered a p<0.05 as statistically significant. We performed all calculations using SPSS, V.21.0 for Windows (SPSS, Chicago, Illinois, USA).

For each type of COI, we conducted a multiple logistic regression analysis using data from papers that reported authors' COI disclosures, ‘having at least one author reporting the COI type’ as the dependent variable. The independent variables were the characteristics of the systematic review and variables related to Journal COI policy.²⁷ The latter included the journal impact factor, the journal requirement for COI disclosure form and the journal requirement for disclosure of at least one non-financial COI. We tested the association of each candidate independent variable with the dependent variable and applied a univariable cut-off p value of 0.2 to include the variables in the model.

Results

Out of 1438 citations identified by the search strategy, we included a total of 200 systematic reviews: 100 Cochrane and 100 non-Cochrane systematic reviews (figure 2). The agreement at the full-text screening stage was moderate (κ=0.69).

Figure 2.

Study flow diagram. RCT, randomised controlled trial.

General characteristics of the systematic review

Table 1 presents the characteristics of the included systematic review papers, stratified by type of review (Cochrane vs non-Cochrane), with a p value for the test of difference between the two types. Cochrane reviews included fewer review authors and trial participants, and were more likely to use Cochrane risk of bias tool and the GRADE system to evaluate the quality of evidence by outcome.

Table 1.

General characteristics of the included systematic reviews (N=200)

	Overall (N=200)	Cochrane (N=100)	Non-Cochrane (N=100)	p Value*
Number of systematic review authors; median (IQR)	5 (4–7)	5 (4–6)	6 (3–8)	0.018
Number of included trials; median (IQR)	7 (2–14)	7 (2–13)	7 (2–15)	0.220
Number of randomised participants reported by the included trials; median (IQR)	832 (222–2848)	800 (191–2294)	932 (261–4257)	0.158
Type of intervention
Pharmacological	98 (49%)	49 (49%)	49 (49%)	0.250
Surgical/invasive procedure	55 (28%)	23 (23%)	32 (32%)
Lifestyle intervention	12 (6%)	5 (5%)	7 (7%)
Rehabilitation	7 (4%)	3 (3%)	4 (4%)
Screening/diagnostic intervention	5 (3%)	4 (4%)	1 (1%)
Psychotherapeutic intervention	13 (7%)	10 (10%)	3 (3%)
Other	10 (5%)	6 (6%)	4 (4%)
Type of control
Active control (as opposed to non-active control)	85 (43%)	31 (31%)	54 (54%)	0.001
Tools used to evaluate risk of bias
Cochrane risk of bias tool	148 (74%)	98 (98%)	50 (50%)	<0.0001
Revised Cochrane risk of bias tool	0 (0%)	0 (0%)	0 (0%)
Jadad's scale	6 (3%)	0 (0%)	6 (6%)
Other tool	17 (9%)	1 (1%)	16 (16%)
Not done	29 (15%)	1 (1%)	28 (28%)
Systems used to evaluate the quality of evidence by outcome
GRADE	79 (40%)	71 (71%)	8 (8%)	<0.0001
Other system	7 (4%)	0 (0%)	7 (7%)
Not done	114 (57%)	29 (29%)	85 (85%)

*p Value for the difference between Cochrane and non-Cochrane systematic reviews.

Characteristics of the reported systematic review funding

Table 2 presents the characteristics of the reported review funding stratified by type of review. Twenty-two per cent of reviews did not report their funding source, with non-Cochrane reviews reporting it less frequently. When reported, the top funding source was governmental. Cochrane reviews more frequently reported internal sources of funding and were less likely to report the role of the funder. Only non-Cochrane reviews reported private for profit sources of funding.

Table 2.

Characteristics of the reported review funding in Cochrane (N=100) and non-Cochrane (N=100) systematic reviews

	Overall N (%)	Cochrane n (%)	Non-Cochrane n (%)	p Value*
Funding
Not reported	44 (22%)	10 (10%)	34 (34%)	<0.0001
Reported as not funded	25 (13%)	4 (4%)	21 (21%)
Reported as funded	131 (66%)	86 (86%)	45 (45%)
Source of funding (when reported as funded)†
Internally funded	68 (52%)	58 (67%)	10 (22%)	<0.0001
Externally funded by
Government	88 (67%)	54 (63%)	34 (76%)	0.140
Private for profit	3 (2%)	0 (0%)	3 (7%)	0.039
Private not for profit with evidence of support by private for profit that is a drug/device industry	3 (2%)	1 (1%)	2 (4%)	0.271
Private not for profit with evidence of support by private for profit that is not a drug/device industry	1 (1%)	1 (1%)	0 (0%)	1.000
Private not for profit with no evidence of support by private for profit	17 (13%)	12 (14%)	5 (11%)	0.646
Paper reported on the role of the funder†	33 (25%)	11 (13%)	22 (49%)	<0.0001
Paper reported that the funder had no role in any part of the systematic review‡	27 (82%)	10 (91%)	17 (77%)	0.637

*p Value for the difference between Cochrane and non-Cochrane systematic reviews.

†Calculated using N=131 as the denominator in the overall group (86, 45 in the two respective subgroups), in reference to funded papers.

‡Calculated using N=33 as the denominator in the overall group (11, 22 in the two respective subgroups), in reference to funded papers that provided a statement reporting the role of the funder.

Characteristics of the reported COI disclosures

Table 3 presents the characteristics of the reported COI disclosures stratified by the type of systematic review. Overall, 100% and 94% of Cochrane and non-Cochrane reviews, respectively, reported authors' COI disclosures (difference statistically significant). Nine non-Cochrane reviews stated that COI disclosures are available on request; at our request, the corresponding authors for three of those reviews made those disclosures available, one in the form of a narrative statement and two as completed ICMJE forms.

Table 3.

Characteristics of the reported COI disclosures in Cochrane (N=100) and non-Cochrane (N=100) systematic reviews

	Overall	Cochrane	Non-Cochrane
	N (%)	n (%)	n (%)	p Value *
Papers that report authors’ COI disclosures	194 (97%)	100 (100%)	94 (94%)	0.029
Form of COI disclosures†
Provided as a narrative statement in the main document	193 (99%)	100 (100%)	93 (99%)	0.014
Provided in an online document that was accessible	4 (2%)‡	0 (0%)	4 (4%)	0.121
Provided in an online document that was not accessible	0 (0%)	0 (0%)	0 (0%)	NA
Available on request	9 (5%)	0 (0%)	9 (10%)	0.003

*p Value for the difference between Cochrane and non-Cochrane systematic reviews.

†Calculated using N=194 as the denominator in the overall group (100 and 94 in the two respective subgroups).

‡All four consisted of ICMJE uniform disclosure forms.

COI, conflict of interest; ICMJE, International Committee of Medical Journal Editors; NA, not applicable.

Three non-Cochrane reviews made their COI disclosures available both as narrative statements in the main document and in accessible online ICMJE forms. Two of those reviews had more information on COI in the online document. Out of the three reviews whose disclosures were made available at our request, one review had more information on COI in the provided disclosures compared with disclosures published in the main document.

Different types of COI: Figure 3 displays the distributions of the percentages of authors per paper reporting the different types of COI. Table 4 presents the same distributions in the tabular format described in the Methods section. Of the 194 systematic reviews that provided COI disclosures, 41% had at least one author reporting any type of COI. Institutional COIs were generally less frequently reported than individual COIs. The least reported type of COI was individual professional (3%). The statistically significant differences for the stratified analysis showed that there were more Cochrane reviews, relative to non-Cochrane reviews, with at least one author reporting the following COIs: at least one type, individual intellectual and ‘other types’, in addition to providing with their COI disclosures an additional ‘loogly’ statement, claiming that a relationship is unrelated or non-influential.

Figure 3.

The distributions of the percentages of authors per paper reporting the different types of COI. COI, conflict of interest.

Table 4.

Reporting of systematic review papers and systematic review authors of different types of COI, stratified by Cochrane (N=100), non-Cochrane (N=94) and overall (N=194)

	Percentages of papers with at least one author reporting the type of COI; N (%)			Distributions of the percentages of authors per paper reporting COI*; median (IQR)
	Overall	Cochrane	Non-Cochrane	Overall	Cochrane	Non-Cochrane
Types of COI
At least one type†	80 (41%)	49 (49%)	31 (33%)	35 (24–62)	33 (20–50)	50 (32–67)
Individual financial	60 (31%)	32 (32%)	28 (30%)	35 (20–59)	29 (20–50)	42 (27–62)
Individual professional	5 (3%)	5 (5%)	0 (0%)	33 (25–75)	33 (25–75)	NA
Individual intellectual†	24 (12%)	19 (19%)	5 (5%)	25 (17–50)	25 (20–50)	14 (9–47)
Institutional financial	19 (10%)	7 (7%)	12 (13%)	25 (11–29)	25 (20–25)	23 (10–29)
Institutional advocatory	7 (4%)	5 (5%)	2 (2%)	20 (9–25)	22 (18–25)	‡
‘Other types’†	16 (8%)	12 (12%)	4 (4%)	20 (14–25)	20 (15–25)	19 (7–25)
Provided a ‘loogly statement’†	29 (15%)	24 (24%)	5 (5%)	33 (23–63)	42 (21–69)	29 (19–67)

*Calculated for systematic review papers with at least one author reporting the type of COI (ie, papers counted in the preceding column).

†p Value for the difference between Cochrane and non-Cochrane systematic reviews is statistically significant.

‡Authors of only two non-Cochrane reviews reported institutional advocatory COI, with the percentages being 5% and 9%.

COI, conflict of interest; NA, not applicable.

Individual financial COI: Figure 4 displays the distributions of the percentages of authors per paper reporting the different subtypes of individual financial COI. Table 5 presents the same distributions in the tabular format described in the Methods section. The most frequently reported subtype was ‘personal fees’ (18%). There were no statistically significant differences for the stratified analysis (Cochrane reviews vs non-Cochrane reviews). Table 6 presents the characteristics of the individual financial COI reported in 60 systematic review papers. Source of COI was the most commonly reported characteristic (87%), particularly by non-Cochrane reviews. Monetary value of the COI was the least commonly reported characteristic (8%), particularly by Cochrane reviews.

Figure 4.

The distributions of the percentages of authors per paper reporting the different subtypes of individual financial conflict of interest.

Table 5.

Reporting of systematic review papers and systematic review authors of different subtypes of individual financial COI, stratified by Cochrane (N=100), non-Cochrane (N=94) and overall (N=194)

	Percentages of papers with at least one author reporting the type of COI; N (%)			Distributions of the percentages of authors per paper reporting COI*; median (IQR)
	Overall	Cochrane	Non-Cochrane	Overall	Cochrane	Non-Cochrane
Subtypes of individual financial COI	60 (31%)	32 (32%)	28 (30%)	35 (20–59)	29 (20–50)	42 (27–62)
Grant from source(s) same as funding source(s)	12 (6%)	7 (7%)	5 (5%)	35 (21–54)	25 (20–50)	50 (35–61)
Grant from source(s) different from funding source(s)	26 (13%)	10 (10%)	16 (17%)	26 (13–39)	25 (19–81)	16 (10–36)
Employment	8 (4%)	6 (6%)	2 (2%)	37 (25–50)	42 (25–56)	NA
Personal fees (other than employment)	34 (18%)	14 (14%)	20 (21%)	32 (17–50)	23 (17–38)	35 (21–54)
Non-monetary support	12 (6%)	4 (4%)	8 (9%)	5 (7–24)	23 (18–25)	9 (6–16)
Drug/equipment supplies	3 (2%)	3 (3%)	0 (0%)	NA	NA	NA
Patent(s)	1 (1%)	0 (0%)	1 (1%)	NA	NA	NA
Stocks, bonds, stock options, other securities (eg, equity)	2 (1%)	0 (0%)	2 (2%)	NA	NA	NA
‘Other subtypes’	1 (1%)	0 (0%)	1 (1%)	NA	NA	NA

When not applicable, this is due to small numbers or due to cells with a value of zero.

*Calculated for systematic review papers with at least one author reporting the type of individual financial COI (ie, papers counted in the preceding column).

COI, conflict of interest; NA, not applicable.

Table 6.

Characteristics of the reported individual financial conflict of interest (COI) of systematic review papers stratified by Cochrane (N=32), non-Cochrane (N=28) and overall (N=60)

	Overall	Cochrane	Non-Cochrane
	N (%)	n (%)	n (%)	p Value*
Percentage of disclosures that specify the following characteristics of the individual financial COI
Source	52 (87%)	24 (75%)	28 (100%)	0.005
Specify that a source produces one of the products subject of the SR†	4 (8%)	4 (17%)	0 (0%)	0.039
Specify that a source produces products not subject of the SR but under the same therapeutic area†	2 (4%)	1 (4%)	1 (4%)	1.000
Monetary value	5 (8%)	0 (0%)	5 (18%)	0.018
Duration	11 (18%)	4 (13%)	7 (25%)	0.212
During conduct of the study‡	8 (72%)	1 (25%)	7 (100%)	0.024
1 year‡	0 (0%)	0 (0%)	0 (0%)
2 years‡	1 (9%)	1 (25%)	0 (0%)
3 years‡	0 (0%)	0 (0%)	0 (0%)
4 years‡	1 (9%)	1 (25%)	0 (0%)
5 years‡	0 (0%)	0 (0%)	0 (0%)
More than 5 years‡	1 (9%)	1 (25%)	0 (0%)

*p Value for the difference between Cochrane and non-Cochrane systematic reviews.

†Calculated using N=52 as the denominator in the overall group (24 and 28 in the two respective subgroups).

‡Calculated using N=11 as the denominator in the overall group (4 and 7 in the two respective subgroups).

SR, systematic review.

Individual intellectual COI: Figure 5 displays the distributions of the percentages of authors per paper reporting the different subtypes of individual intellectual COI. Table 7 presents the same distributions in the tabular format described in the Methods section. The most frequently reported subtype was ‘authorship of primary studies not included in the systematic review’ (10%), particularly by Cochrane reviews. Table 8 presents the characteristics of the individual intellectual COI reported in 24 systematic review papers. Only Cochrane reviews reported whether any type related to one of the products/interventions subject of the systematic review (63%), with a statistically significant difference compared with non-Cochrane reviews.

Figure 5.

The distributions of the percentages of authors per paper reporting the different subtypes of individual intellectual conflict of interest.

Table 7.

Reporting of systematic review papers and systematic review authors of different subtypes of individual intellectual COI, stratified by Cochrane (N=100), non-Cochrane (N=94) and overall (N=194)

	Percentages of papers with at least one author reporting the type of COI; N (%)			Distributions of the percentages of authors per paper reporting COI*; median (IQR)
	Overall	Cochrane	Non-Cochrane	Overall	Cochrane	Non-Cochrane
Subtypes of individual intellectual COI	24 (12%)	19 (19%)	5 (5%)	25 (17–50)	25 (20–50)	14 (9–47)
Authorship of primary studies included in the SR†	9 (5%)	9 (9%)	0 (0%)	25 (17–63)	25 (17–63)	NA
Authorship of primary studies not included in the SR†	19 (10%)	15 (15%)	4 (4%)	20 (17–50)	20 (17–50)	13 (7–64)
Participation in a previous guideline panel	1 (1%)	0 (0%)	1 (1%)	NA	NA	NA
Writing an editorial	0 (0%)	0 (0%)	0 (0%)	NA	NA	NA
Peer-reviewed grant funding	0 (0%)	0 (0%)	0 (0%)	NA	NA	NA

*Calculated for systematic review papers with at least one author reporting the type of individual intellectual COI (ie, papers counted in the preceding column).

†p Value for the difference between Cochrane and non-Cochrane systematic reviews is statistically significant.

COI, conflict of interest; NA, not applicable; SR, systematic review.

Table 8.

Characteristics of the reported individual intellectual conflict of interest (COI) of systematic review papers stratified by Cochrane (N=19), non-Cochrane (N=5) and overall (N=24)

	Overall	Cochrane SR	Non-Cochrane SR
	N (%)	n (%)	n (%)	p Value*
Percentage of disclosures that specify the following characteristics of the individual intellectual COI
If any type relates to one of the products/interventions subject of the SR	12 (50%)	12 (63%)	0 (0%)	0.037
If any type relates to products/interventions not subject of the SR but under the same therapeutic area	7 (29%)	4 (21%)	3 (60%)	0.126

*p Value for the difference between Cochrane and non-Cochrane systematic reviews.

SR, systematic review.

Institutional financial COI: Table 9 presents the characteristics of the institutional financial COI reported in 19 systematic reviews. Source of COI was the most frequently reported characteristic (90%). All non-Cochrane reviews tend to always report the source of any financial COI (100%). None of the Cochrane reviews specified the monetary value or duration for this type of COI.

Table 9.

Characteristics of the reported institutional financial COI of systematic review papers stratified by Cochrane (N=7), non-Cochrane (N=12) and overall (N=19)

	Overall	Cochrane	Non-Cochrane
	N (%)	n (%)	n (%)	p Value*
Percentage of disclosures that specify the following characteristics of the institutional financial COI
Source	17 (90%)	5 (71%)	12 (100%)	0.123
Specify that a source produces one of the products subject of the SR†	1 (6%)	1 (20%)	0 (0%)	0.294
Specify that a source produces products not subject of the SR but under the same therapeutic area†	0 (0%)	0 (0%)	0 (0%)	NA
Monetary value	1 (5%)	0 (0%)	1 (8%)	1.000
Duration	3 (16%)	0 (0%)	3 (25%)	0.263
During conduct of the study‡	3 (100%)	NA	3 (100%)	NA
1 year‡	0 (0%)		0 (0%)
2 years‡	0 (0%)		0 (0%)
3 years‡	0 (0%)		0 (0%)
4 years‡	0 (0%)		0 (0%)
5 years‡	0 (0%)		0 (0%)
More than 5 years‡	0 (0%)		0 (0%)

*p Value for the difference between Cochrane and non-Cochrane systematic reviews.

†Calculated using N=17 as the denominator in the overall group (5 and 12 in the two respective subgroups).

‡Calculated using N=3 as the denominator in the overall group (0 and 3 in the two respective subgroups).

COI, conflict of interest; NA, not applicable; SR, systematic review.

Results of logistic regression analyses

Online supplementary appendix 4 presents the details of the multiple logistic regression analyses. These analyses were based on data from papers that reported authors' COI disclosures (n=194). The statistically significant associations for the respective models (ie, respective types of COI) were as follows:

• Having at least one author reporting at least one type of COI: journal impact factor (OR=1.16; 95% CI 1.04 to 1.28).

• Having at least one author reporting individual financial COI: journal impact factor (OR=1.14; 95% CI 1.04 to 1.25); and type of intervention being pharmacological (reference category being non-pharmacological; OR=2.02; 95% CI 1.04 to 3.91).

• Having at least one author reporting individual intellectual COI: no statistically significant associations.

• Having at least one author reporting institutional financial COI: journal impact factor (OR=1.09; 95% CI 1.00 to 1.18).

We did not conduct regression analyses for the variables ‘individual professional COI’ and ‘institutional advocatory COI’ due to the low number of papers that reported these COIs (n=5 and 7, respectively).

Discussion

Summary of findings

The vast majority of systematic reviews report authors' COI disclosures, typically in the main document, and in a few cases either online or on request. Forty-one per cent of reviews had at least one author reporting any type of COI (with a higher percentage for Cochrane reviews). Institutional COIs were generally less frequently reported than individual COIs; non-financial COIs were generally less frequently reported than financial COIs; and Cochrane reviews were more likely to report individual intellectual COI compared with non-Cochrane reviews. We found variability in the reporting of the different types and characteristics of COI (eg, source, monetary value, duration).

Comparison to similar studies

The chart in table 10 compares the current study to five other methodological surveys of COI of authors of systematic reviews. In brief, our review was not restricted to a specific topic, included the highest number of papers and both Cochrane and non-Cochrane reviews, making it the most representative. In addition, it is the only review that included non-financial and institutional COI. Beyari et al²⁸ found that 39% of systematic reviews provided a COI disclosure, whereas we found that 94% of non-Cochrane systematic reviews reported authors' COI disclosures. Four other surveys assessed financial but not other types of COI. Forbes,²⁹ Dunn et al,¹⁹ Bes-Rastrollo et al¹⁸ and Ebrahim et al²⁰ found that 17%, 27%, 35% and 65%, respectively, of systematic reviews reported individual financial COI. In our study, we found that 31% of systematic reviews reported individual financial COI. Although it is hard to discern any time trends, the variability in results can be attributed to differences in the areas and topics assessed in the different surveys.

Table 10.

Comparative chart including five related methodological surveys of conflicts of interest (COI) of authors of systematic reviews

Survey	Eligibility criteria	Number of included systematic reviews	Year of systematic review publication	Types of COI	Main findings
Beyari et al28	Papers on dental therapy (excluded Cochrane reviews)	129	2000–2012	Potential COI (does not specify types)	39% reported presence or absence of COI
Forbes29	Vascular surgery journals on three specific surgical procedures	66	2008–2009	Financial COI	17% reported a financial COI
Bes-Rastrollo et al18	Papers on sugar-sweetened beverages as a risk for weight gain or obesity	17	2006–2013	Financial COI	35% reported COI with industry; 65% reported no COI with industry or did not report COI
Dunn et al19	Papers on neuraminidase inhibitors	26	2006–2014	Financial COI	27% reported a financial COI
Ebrahim et al20	Papers on antidepressants	185	2007–2014	Employment, or any support from industry	65% reported authors’ COI with a for-profit organisation
Current survey	Was not restricted to a specific topic; included both Cochrane and non-Cochrane reviews	200	2015	Individual financial, professional and intellectual COI; institutional, financial and advocatory COI	97% reported presence or absence of any type of COI; 31% reported individual financial COI

A 2014 study found ‘substantial variability’ in the reporting of COI in meta-analyses published in dentistry journals.²⁸ We found similar evidence during our thorough evaluation of the reported COI in the included systematic reviews and meta-analyses published in medical journals. This supports the fact that more stringent regulations on the reporting of COI ought to be adopted by all health-related journals. Such regulations in turn help set standardised systems for reporting and managing COI, two processes that at present seem to lack rigour and remain as sources of confusion to societies and academic investigators.^{8 30}

The variability in reporting of COIs in medical journals may hinder the ability of readers to adequately judge those COIs and their implications. Indeed, as shown in previous experiments, a standard consistent section would aid readers in their perceptions of authors' COI disclosures and the reliability of articles.^{31 32} A paper by Maharaj³³ discusses a method for scoring financial COI; such scoring scales could aid in adding more integrity to medical research, as well as minimising influence of possibly biased results on decision-making by patients, clinicians and policymakers.

This study assessed disclosed COI but did not verify the accuracy of these disclosures. The US Sunshine Act Open Payments database and the Danish Health and Medicines Authority public disclosure database could be very helpful in undertaking such verification.^{34 35} Unfortunately, such databases are not common, and the task of verification remains challenging and lacks valid tools and methodologies.

Reporting of COI

We found a positive association between reporting COI and journal impact factor. We did not identify any previously published study showing a similar association. One explanation of this association is that journals with higher impact factors are more likely to have disclosure policies. Indeed, two studies found that journals with high impact factors are more likely to have published policies for reporting COI.^{10 36} Another explanation of association is that studies that get published in higher impact journals tend to have more COI. The finding of a positive association between reporting individual financial COI and pharmacological intervention might reflect higher efforts by the pharmaceutical industry to partner with systematic reviewers.

Reporting of the review funding

We found that Cochrane reviews were more likely to report being funded compared with non-Cochrane reviews (86% vs 45%). However, this might be an overestimation because the source of funding in Cochrane reviews need to be inferred from the section on ‘source of support’, which might report on non-funding type of support. Indeed, Cochrane reviews were more likely to report receiving internal funding. It is quite likely that Cochrane reviewers include under ‘internal source of support’ relationships that might not be typically considered as a source of funding. On the other hand, Cochrane reviews were less likely to report on the role of the funder. For these reasons, the Cochrane Collaboration needs to consider a more explicit and specific system for reporting of funding sources.

Strengths and limitations

This is the first methodological survey of a large and representative sample of systematic reviews evaluating how frequently they report their authors' COI, and the types of those COIs. We used a comprehensive framework of COI that allowed us to assess different types of COIs, including non-financial COI. An additional strength is the use of systematic and transparent methods, for example, duplicate and independent processes in screening and data collection.

Although we included a widely representative sample of systematic reviews, it was limited to the clinical field. Thus, our results may not apply to other fields such as health policy. Also, we only describe the COI reported by authors of the systematic reviews. Indeed, authors may not accurately report their COI, or the journal requirements for COI disclosure may be limited.^{37 38}

Implications for practice

Given the variability in the reporting of COI in systematic reviews, particularly non-Cochrane reviews, medical journals need to ensure the presence and implementation of COI disclosure policies. In addition, those policies need to better address institutional COIs and non-financial COIs such as professional and intellectual COIs. COI disclosures should also better report COI characteristics such as source, monetary value, duration and how they relate to the product subject of the paper.

Implications for future research

Assessment of the reporting of COI by authors of other types of publications (eg, randomised controlled trials) and in other fields (eg, policymaking) remains to be undertaken. Assessments in different areas may help to refine and validate our proposed COI framework. In addition, it would be important to explore whether the few proposed types of non-financial relationships actually introduce COI, that is, bias judgements. Additionally, there is a need for development of methods to allow checking and verifying the accuracy of COI disclosures.