Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality

Abstract

Background

Financial conflicts of interest in systematic reviews (e.g. funding by drug or device companies or authors' collaboration with such companies) may impact on how the reviews are conducted and reported.

Objectives

To investigate the degree to which financial conflicts of interest related to drug and device companies are associated with results, conclusions, and methodological quality of systematic reviews.

Search methods

We searched PubMed, Embase, and the Cochrane Methodology Register for studies published up to November 2016. We also read reference lists of included studies, searched grey literature sources, and Web of Science for studies citing the included studies.

Selection criteria

Eligible studies were studies that compared systematic reviews with and without financial conflicts of interest in order to investigate differences in results (estimated treatment effect and frequency of statistically favourable results), frequency of favourable conclusions, or measures of methodological quality of the review (e.g. as evaluated on the Oxman and Guyatt index).

Data collection and analysis

Two review authors independently determined the eligibility of studies, extracted data, and assessed risk of bias. We synthesised the results of each study relevant to each of our outcomes. For meta‐analyses, we used Mantel‐Haenszel random‐effects models to estimate risk ratios (RR) with 95% confidence intervals (CIs), with RR > 1 indicating that systematic reviews with financial conflicts of interest more frequently had statistically favourable results or favourable conclusions, and had lower methodological quality. When a quantitative synthesis was considered not meaningful, results from individual studies were summarised qualitatively.

Main results

Ten studies with a total of 995 systematic reviews of drug studies and 15 systematic reviews of device studies were included. We assessed two studies as low risk of bias and eight as high risk, primarily because of risk of confounding. The estimated treatment effect was not statistically significantly different for systematic reviews with and without financial conflicts of interest (Z‐score: 0.46, P value: 0.64; based on one study of 14 systematic reviews which had a matched design, comparing otherwise similar systematic reviews). We found no statistically significant difference in frequency of statistically favourable results for systematic reviews with and without financial conflicts of interest (RR: 0.84, 95% CI: 0.62 to 1.14; based on one study of 124 systematic reviews). An analysis adjusting for confounding due to methodological quality (i.e. score on the Oxman and Guyatt index) provided a similar result. Systematic reviews with financial conflicts of interest more often had favourable conclusions compared with systematic reviews without (RR: 1.98, 95% CI: 1.26 to 3.11; based on seven studies of 411 systematic reviews). Similar results were found in two studies with a matched design, which therefore had a reduced risk of confounding. Systematic reviews with financial conflicts of interest tended to have lower methodological quality compared with systematic reviews without financial conflicts of interest (RR for 11 dimensions of methodological quality spanned from 1.00 to 1.83). Similar results were found in analyses based on two studies with matched designs.

Authors' conclusions

Systematic reviews with financial conflicts of interest more often have favourable conclusions and tend to have lower methodological quality than systematic reviews without financial conflicts of interest. However, it is uncertain whether financial conflicts of interest are associated with the results of systematic reviews. We suggest that patients, clinicians, developers of clinical guidelines, and planners of further research could primarily use systematic reviews without financial conflicts of interest. If only systematic reviews with financial conflicts of interest are available, we suggest that users read the review conclusions with skepticism, critically appraise the methods applied, and interpret the review results with caution.

Plain language summary

Financial conflicts of interests and results, conclusions, and quality of systematic reviews

Patient treatment practices are often based on clinical research. Systematic reviews are a core type of such clinical research. When several similar studies (i.e. studies investigating the same research questions using similar methods) have been conducted, these can be identified and analysed in a systematic review. Systematic reviews thereby summarise existing studies and provide an overview of a specific research field. Thus, systematic reviews may have a major influence on decisions about patient care and it is essential that such reviews are trustworthy.

Sometimes, systematic reviews are funded by companies with a financial interest in the review's results and conclusions, for example because they produce a drug or device investigated in the review. At other times, systematic reviews are carried out by researchers with a personal financial interest in a specific result, for example when the researcher acts as a consultant for the company producing an intervention that is assessed in the review. These financial conflicts of interest may impact on how systematic reviews are conducted and reported. Our Cochrane Methodology Review focuses on financial conflicts of interest related to drug or device companies in systematic reviews. Our primary aim was to investigate the degree to which systematic reviews with financial conflicts of interest present review results and make conclusions that are more favourable than systematic reviews without such financial conflicts of interest. Our secondary aim was to investigate the degree to which systematic reviews with financial conflicts of interest differ in methodological quality from systematic reviews without such financial conflicts of interest.

We found 10 studies comparing systematic reviews with and without financial conflicts of interest. Based on two of these studies, we found no evidence of a difference in review results between systematic reviews with and without financial conflicts of interest. Based on seven studies, we found that systematic reviews with financial conflicts of interest more often had conclusions favourable towards the experimental intervention (risk ratio (RR): 1.98, 95% confidence interval (CI): 1.26 to 3.11). Also, based on four studies, systematic reviews with financial conflicts of interest tended to have lower methodological quality (RR for 11 dimensions of methodological quality spanned from 1.00 to 1.83).

Our analyses suggest that when systematic reviews have financial conflicts of interest related to drug or device companies, they are of lower methodological quality, and have more favourable conclusions. However, it is not clear whether this derives from actual differences in the review's results or the over‐interpretation of those results. Based on our findings, we suggest that people who use systematic reviews, including patients, clinicians, developers of clinical guidelines, and planners of future research, could primarily use systematic reviews without financial conflicts of interest. If such reviews are not available, we suggest that users are especially cautious when they read and interpret systematic reviews with financial conflicts of interest.

Summary of findings

for the main comparison.

Systematic reviews with financial conflicts of interest compared with systematic reviews without
Sample: systematic reviews Intervention: systematic reviews with financial conflicts of interest Comparison: systematic reviews without financial conflicts of interest
Outcomes	Absolute effect* (95% CI)		Relative effect	Number of studies	Certainty of the evidence (GRADE)	Comments**
	Assumed risk	Corresponding risk
	Number of industry reviews with the outcome	Number of non‐industry reviews with the outcome
Estimated treatment effect measured as Z‐scores*** after adjustment for the number of patients			Pooled Z‐score: 0.46 (P value: 0.64)	1 study including 7 pairs of industry and non‐industry systematic reviews	⊕⊝⊝⊝ very low	Downgraded due to imprecision (only one study of 14 systematic reviews)
Frequency of statistically favourable results	549 (405 to 745) reviews with statistically favourable results per 1000 industry reviews	653 reviews with statistically favourable results per 1000 non‐industry reviews	RR: 0.84 (0.62 to 1.14)	1 study including 124 systematic reviews	⊕⊝⊝⊝ very low	Downgraded due to limitations in design (only one study with high risk of bias) and imprecision (wide confidence intervals)
Frequency of favourable conclusions	895 (569 to 1000****) reviews with favourable conclusions per 1000 industry reviews	452 reviews with favourable conclusions per 1000 non‐industry reviews	RR: 1.98 (1.26 to 3.11)	7 studies including 411 systematic reviews	⊕⊕⊝⊝ low	Fairly large effect estimate, which was substantially higher in the one study with low risk of bias.
GRADE Working Group grades of evidence High quality: Further research is very unlikely to change our confidence in the estimate of effect. Moderate quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate. Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate. Very low quality: We are very uncertain about the estimate.

CI: confidence interval; RR: risk ratio

*The assumed risk of the control group (i.e. non‐industry systematic review group) was calculated as the mean risk (i.e. number of systematic reviews with favourable conclusions divided by total number of systematic reviews). The corresponding risk (and its 95% confidence interval) is based on the assumed risk in the control group and the relative effect of the intervention (and its 95% CI).

**See 'Assessment of certainty of the evidence' for a detailed description.

***A Z‐score expresses the number of standard deviations by which a value differs from the mean. If a Z‐score is interpreted as a standardised mean difference, a Z‐score of 0.46 would indicate a moderate effect (Kirkwood 2003). A Z‐score of 0.46 indicates that effect estimates are larger in industry reviews compared with non‐industry reviews (Jorgensen 2006).

****Upper limit of event rate truncated at 1000.

Background

Description of the problem or issue

Systematic reviews provide a rational basis for developing clinical guidelines, for therapeutic decision making, and for planning clinical trials. They have a major impact on which interventions are offered to patients (Guyatt 2008). It is therefore essential that such reviews are trustworthy and unbiased.

One area of concern is the degree to which financial conflicts of interest impact on the conduct and reporting of systematic reviews (Institute of Medicine 2011). The pharmaceutical and clinical device industries frequently fund clinical trials (Atal 2015; Chan 2005), and to a lesser extent also systematic reviews. For example, in a random sample of 300 systematic reviews, Page and colleagues found that eight (3%) were industry‐funded (Page 2016). On the other hand, another review found that a quarter of 185 meta‐analyses of trials of antidepressants were industry‐funded (Ebrahim 2016). Furthermore, systematic reviews are often produced by authors with financial conflicts of interest; a random sample of 194 systematic reviews found that 60 (31%) had at least one author with financial conflicts of interest (Hakoum 2016).

Numerous studies have investigated the relation between financial conflicts of interest and outcomes of individual research studies, mainly clinical trials. A recent update of a Cochrane Review reported clear associations between funding source and statistically significant trial efficacy results (based on 25 empirical studies) and trial conclusions (based on 29 empirical studies) (Lundh 2017). In contrast, fewer studies have investigated how financial conflicts of interest at the level of the systematic review impact on their results and conclusions.

Why it is important to do this review

The retrospective nature of a systematic review and the subjective element in selecting inclusion criteria and outcomes is likely to make such research more susceptible to influence from financial conflicts of interest than prospective clinical trials.

This concern is supported by a review of pairs of Cochrane Reviews and paper‐based reviews of the same drugs used for the same disease that reported that industry‐funded reviews had more favourable conclusions (Jorgensen 2006). However, other studies have reported a less clear association with wide confidence intervals (Yank 2007). To our knowledge, this Cochrane Methodology Review is the first systematic review of methodology to identify, analyse, and summarise such studies.

Objectives

Our primary objectives were to investigate the degree to which financial conflicts of interest related to drug or device companies in systematic reviews are associated with the following.

1. Results

   1. Estimated treatment effect

   2. Results statistically favourable to the experimental intervention

2. Conclusions favourable to the experimental intervention

Our secondary objective was to investigate the degree to which financial conflicts of interest related to drug or device companies in systematic reviews are associated with the following.

1. Methodological quality of the reviews

Terminology

We use the following definitions.

1. Financial conflicts of interest: any funding of the systematic reviews by drug or device companies or any review author with financial conflicts of interest in relation to such companies.

2. Industry funding: any funding of the systematic review by industry, authorship by full‐time industry employees, assistance by industry (e.g. statistical analysis by company statistician, or writing assistance by a medical writer funded by the company).

3. Author financial conflicts of interest: any financial relationship of authors, apart from full‐time employment, with a drug or device company (e.g. receiving grants, owning stocks, being on an advisory board, or consultancy work).

4. Industry reviews: reviews that are consistent with one or more of the above definitions.

5. Non‐industry reviews: reviews that fulfil none of the above definitions.

6. Drugs: medications that require approval from regulatory agencies.

7. Devices: instruments used in diagnosis, treatment, or prevention of a disease. This definition follows the definition of the Food and Drug Administration (FDA) (FDA 2017) and includes imaging technologies.

Methods

Criteria for considering studies for this review

Types of studies

We included studies that investigated samples of systematic reviews with and without financial conflicts of interest. We defined systematic reviews according to the definitions used by the authors of the included studies.

Eligible studies had to investigate at least one of our primary or secondary outcomes. If a study contained a mixture of systematic reviews and research of other designs (e.g. randomised trials), we included the study, but only included separate data for the systematic reviews. If this distinction was not reported in a study with a variety of research designs, we requested the data for the systematic reviews from the authors unless it contained fewer than five reviews and, therefore, was too small to be informative.

We excluded studies that investigated financial conflicts of interest related to non‐pharmaceutical or non‐device industries (e.g. tobacco and food industries). Studies of mixed domains (e.g. pharmaceuticals and nutritional supplements) were included in the review and in our analyses if separate data for the systematic reviews with drug or device industry financial conflicts of interest were obtainable. If this distinction was not reported in the study, we requested the data from the authors unless the number of systematic reviews was too small to be informative (i.e. less than five reviews).

Studies were eligible regardless of the language in which they had been reported.

Types of data

We included data on estimated treatment effect (e.g. Z‐scores and P values), frequency of statistically favourable results, and frequency of favourable conclusions (e.g. number of events and odds ratios). For methodological quality, we included both continuous and binary data for industry and non‐industry reviews (e.g. overall methodological quality score and number of events in each item of a tool such as the Oxman and Guyatt index) (Oxman 1991).

Types of methods

We included studies that investigated financial conflicts of interest related to drug and device companies. We included studies regardless of type of investigated financial conflicts of interest.

Types of outcome measures

Primary outcomes

We included the following primary outcomes.

1. Results

   1. Estimated treatment effect (e.g. relative risks)

   2. Frequency of statistically favourable results (e.g. occurrence of results statistically in favour of the experimental intervention)

2. Frequency of favourable conclusions (e.g. recommendation of the experimental intervention without reservations)

Secondary outcomes

We included one secondary outcome.

1. Methodological quality of the systematic reviews. This included, for example, assessment using the Oxman and Guyatt index (Oxman 1991) (i.e. how many industry and non‐industry reviews fulfilled each item that was assessed for quality)

Search methods for identification of studies

Electronic searches

We searched PubMed, Embase, and the Cochrane Methodology Register (searches performed 30 November 2016) for studies. We searched Web of Science (search performed 17 January 2017) for studies citing any of the included studies. We used the strategy shown in Appendix 1 for PubMed and adapted it for the other databases. These searches were done in advance of our protocol (Hansen 2017) being published in the Cochrane Library.

Searching other resources

Grey literature

We searched proceedings from Peer Review Congresses (American Medical Association 2017) and Cochrane Colloquia (Cochrane Community 2017) for conference abstracts published up to November 2016, bearing in mind the evidence on the high proportion of research studies that are presented at conferences but not published in full (Chapman 2012; Scherer 2018). We also searched PROSPERO (search performed 01 March 2017) for registered systematic reviews and the ProQuest database (search performed 01 March 2017) for dissertations and theses. Finally, we searched Google Scholar (search performed 16 March 2017) for unpublished studies. For all searches, we adapted the search strategy shown in Appendix 1.

Reference lists

We checked the reference lists of the included studies for additional potentially eligible studies (Horsley 2011).

Data collection and analysis

Selection of studies

One review author (CH) screened titles and abstracts of all retrieved records for obvious exclusions. Two review authors (CH and KR) independently assessed potentially eligible studies based on full text. Disagreements were resolved by discussion, and arbitration by another review author was not needed.

Data extraction and management

Two review authors (CH and AL) independently extracted data from the included studies. Any difference in data extraction was resolved by discussion or with arbitration by another review author (AH). We extracted data on basic characteristics and financial conflicts of interest of the included studies. For continuous outcome data, we extracted information on difference in estimated treatment effect between industry and non‐industry reviews (reported as pooled Z‐scores and P values). For binary outcome data, we extracted the number of industry and non‐industry reviews with statistically favourable results and favourable conclusions. When reported, we also extracted risk ratios (RR) or odds ratios (OR). We extracted data for industry and non‐industry reviews based on the definitions used by the authors of the included studies, but also for reviews with industry funding only and with author financial conflicts of interest only, based on our definitions. We ensured that all numbers and effect sizes had the same directionality, e.g. a RR >1 indicated that industry reviews more often had favourable conclusions than non‐industry reviews. The full plan for data extraction is shown in Appendix 2.

Assessment of risk of bias in included studies

There are no validated criteria for assessing risk of bias in these types of studies, so we developed a set of criteria similar to criteria developed for a previous Cochrane Review by one of the authors of this review (Lundh 2017), which were influenced by items from the AMSTAR tool (Shea 2007). Two review authors (CH and AL) independently assessed risk of bias. Any disagreements were resolved by discussion, with arbitration provided by a third review author (AH) when needed. We categorised each component as high risk of bias, low risk of bias, or unclear. We used the following criteria.

1. Whether there was a risk of bias in the study inclusion process (low risk of bias could, for example, include two or more assessors independently selecting studies)

2. Whether there was a risk of bias in the coding of financial conflicts of interest and outcomes (low risk of bias could, for example, include an extraction done independently by two or more assessors)

3. Whether there was a risk of bias in the comparability of systematic reviews (low risk of bias could, for example, imply industry and non‐industry reviews of the same intervention used for the same disease)

Our aim was primarily to differentiate between studies with higher and lower risk of bias. We coded a study as low risk of bias if all three criteria were assessed as low risk of bias. Otherwise, we coded it as high risk of bias.

Dealing with missing data

We contacted authors of the included studies in an attempt to obtain unpublished data, to clarify issues on our 'Risk of bias' assessments, or to receive unpublished protocols (Appendix 3) (Young 2011).

Assessment of heterogeneity

We assessed statistical heterogeneity using the I² statistic. We defined substantial heterogeneity as I² > 50%.

Data synthesis

Due to heterogeneity between the included studies, we used Mantel‐Haenszel random‐effects models to estimate RR with 95% confidence intervals (CIs) (RR > 1 indicated that industry reviews more often had statistically favourable results or favourable conclusions). We used a qualitative synthesis for our analysis of estimated treatment effect, and a quantitative synthesis for our analyses of frequency of statistically favourable results, frequency of favourable conclusions, and methodological quality. We also calculated prediction intervals (IntHout 2016; Riley 2011) (Appendix 4).

For our primary analyses, we complied with the definitions of financial conflicts of interest used by the authors of the included studies and analysed industry funding and author financial conflicts of interest together.

For our analysis of methodological quality, we pooled similar items across the different quality assessment tools used in the included studies. For example, we pooled items on appropriateness of search methods in the systematic reviews (Appendix 5). We did not pool items related to reporting quality (e.g. whether the inclusion criteria were reported) or statistical methods (e.g. whether a Bayesian framework was used). We used Mantel‐Haenszel random‐effects models to estimate RR with 95% CI (RR > 1 indicated that industry reviews had lower methodological quality, i.e. more often did not fulfil the item or did not provide information on the item).

Subgroup analysis and investigation of heterogeneity

We planned to conduct the following pre‐specified subgroup analyses for our primary outcomes.

1. High risk of bias studies versus low risk of bias studies

2. Cochrane Reviews versus non‐Cochrane systematic reviews

3. Systematic reviews of drug studies versus systematic reviews of device studies

4. Systematic reviews with major financial conflicts of interest versus systematic reviews with moderate financial conflicts of interest versus systematic reviews with minor financial conflicts of interest (where "major", "moderate" and "minor" were as defined by the authors of the included studies)

We planned to conduct the following post‐hoc subgroup analyses for our primary outcomes (see Differences between protocol and review).

1. Studies defining favourable conclusions as conclusions in favour of the intervention versus studies defining favourable conclusions as conclusions recommending the intervention without reservations

Sensitivity analysis

We planned to conduct the following pre‐specified sensitivity analyses for our primary outcomes.

1. Re‐categorising industry reviews into systematic reviews with industry funding only (i.e. excluding systematic reviews with author financial conflicts of interest from the industry group) and comparing with non‐industry reviews (i.e. systematic reviews without industry funding or author financial conflicts of interest)

2. Re‐categorising industry reviews into systematic reviews with author financial conflicts of interest only (i.e. excluding systematic reviews with industry funding from the industry group) and comparing with non‐industry reviews (i.e. systematic reviews without author financial conflicts of interest or industry funding)

3. Excluding included studies with conflicts of interest

4. Re‐analysing our data using fixed‐effect models

We planned to conduct the following post‐hoc sensitivity analyses for our primary outcomes (see Differences between protocol and review).

1. Excluding systematic reviews with unclear or undeclared financial conflicts of interest from the non‐industry group

2. Excluding one atypical study (Yank 2007) from our pooled analyses because it compared industry reviews (financial conflicts of interest with a single drug company) with a group of both industry and non‐industry reviews (multiple drug companies, no statement, both drug and non‐profit companies, and non‐profit companies)

3. Restricting our analyses to studies assessed as low risk of bias in the comparability criteria or studies performing adjusted regression analyses

4. Re‐categorising industry reviews into reviews with financial conflicts of interest related to any for‐profit organisation or to the manufacturer of the investigated intervention in two separate analyses

In addition, we planned to conduct sensitivity analyses for our secondary outcomes (i.e. methodological quality) to address the issue of confounding.

1. Restricting our analyses to studies assessed as low risk of bias in the comparability criteria

Assessment of certainty of the evidence

We graded the certainty of the evidence for each of our primary outcomes as high, moderate, low, or very low. In the standard GRADE approach for studies of treatment effect (Goldet 2013), observational studies are initially graded as low certainty and randomised trials as high certainty (Guyatt 2011; Schünemann 2017). We followed these principles, and initially graded the included observational studies as providing low certainty.

We assessed the following criteria for downgrading: limitations in the study design, indirectness of evidence, inconsistency of results, imprecision of results, and publication bias. We assessed the following criteria for upgrading the certainty: large magnitude of effect, dose‐response gradient, and all plausible confounding would decrease the effect (Guyatt 2011).

Results

Description of studies

See: Characteristics of included studies; Characteristics of excluded studies.

Results of the search

See: Figure 1.

1.

Study flow diagram.

We identified a total of 5227 records through our electronic database searches. After removing duplicates, we screened 3494 records. After assessing 17 full‐text article, we included 10 studies. Nine of these were identified through the database search and one study was identified through other sources (Figure 1). We did not identify any unpublished studies or protocols for planned studies.

Included studies

See: Characteristics of included studies.

The 10 included studies were published between 2006 and 2017 and investigated a total of 1010 systematic reviews. The median number of included systematic reviews per study was 48 (range 11 to 318). Three studies investigated systematic reviews of randomised trials, six investigated systematic reviews of both randomised trials and non‐randomised studies, and one investigated network meta‐analyses. One study included systematic reviews of both drug and device interventions, whereas nine studies included solely systematic reviews of drug interventions. Five studies included systematic reviews related to specific drug classes (e.g. antidepressants), one related to a specific disease (skin psoriasis), and four included various drug comparisons. One study investigated estimated treatment effects, one investigated frequency of statistically favourable results, seven investigated frequency of favourable conclusions, and six investigated methodological quality. According to the declarations in their reports, three studies had conflicts of interest in the form of using industry facilities during the study and author employment at pharmaceutical companies (Lane 2013), or receiving honoraria for research, lecturing, and consultancy from pharmaceutical companies (Gomez‐Garcia 2017; Hartog 2012).

We received unpublished data from five studies (Gomez‐Garcia 2017; Hartog 2012; Jorgensen 2008; Wang 2010; Yank 2007) and obtained published individual review data from three studies (Dunn 2014; Ebrahim 2016; Jorgensen 2006).

Risk of bias in included studies

See Figure 2 and Figure 3.

2.

'Risk of bias' graph: review authors' judgements about each risk of bias item presented as percentages across all included studies.

3.

'Risk of bias' summary: review authors' judgements about each risk of bias item for each included study.

Overall, we assessed two studies as low risk of bias (Jorgensen 2006; Lane 2013), and the other eight included studies as high risk of bias. The majority of the studies were assessed as low risk of bias in the study inclusion process and extraction methods, whereas they were assessed as high risk of bias in the comparability of investigated systematic reviews.

We were able to obtain one publicly available protocol (through PROSPERO) (Gomez‐Garcia 2017) and four unpublished protocols (Ebrahim 2016; Jorgensen 2006; Jorgensen 2008; Lane 2013) for the included studies. We compared outcomes between these protocols and the corresponding publications of the included studies and found that the reported outcomes were prespecified in the protocols in all cases.

Effect of methods

See: Table 1

Differences in results

Estimated treatment effect

One study comparing seven pairs of Cochrane Reviews and industry‐funded reviews of the same drug used for the same disease, investigated the estimated treatment effect. The authors calculated Z‐scores for the difference in effect estimates between each pair and found that the estimated treatment effect was not statistically significantly different between systematic reviews with and without financial conflicts of interest (pooled Z‐score: 0.46, P value: 0.64) (Jorgensen 2006).

Frequency of statistically favourable results

One study, including 124 meta‐analyses, investigated frequency of statistically favourable results (Yank 2007). Based on this study, we found no statistically significant difference in frequency of statistically favourable results (risk ratio (RR): 0.84, 95% confidence interval (CI): 0.62 to 1.14, Analysis 1.1).

1.1. Analysis.

Comparison 1 Frequency of statistically favourable results: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 1 Frequency of statistically favourable results.

Differences in frequency of favourable conclusions

Seven studies, including a total of 411 systematic reviews, investigated the frequency of favourable conclusions. Industry reviews more frequently had favourable conclusions compared with non‐industry reviews (RR: 1.98, 95% CI: 1.26 to 3.11, I²: 69%, Analysis 2.1). The analysis was mainly driven by two studies that contributed 33.2% (Yank 2007) and 30.2% (Ebrahim 2016) of the weight of the analysis.

2.1. Analysis.

Comparison 2 Frequency of favourable conclusions: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 1 Frequency of favourable conclusions.

The prediction interval for the RR including all studies was 0.62 to 6.31 (Appendix 4). Thus, one can expect industry reviews to more often have favourable conclusions compared with non‐industry reviews, although this relationship is reversed in some cases.

Differences in methodological quality

Five studies investigated methodological quality using either the Oxman and Guyatt index (Jorgensen 2006; Jorgensen 2008; Yank 2007), the AMSTAR tool (Gomez‐Garcia 2017), or a tool the authors developed for their study (Lane 2013). Similar items across these tools could be pooled from four studies only because the fifth (Yank 2007) reported an overall summary score without results of individual items. Additionally, a sixth study measured methodological quality of network meta‐analyses using the ISPOR guidance (Chambers 2015), but we analysed this study separately because these criteria for methodological quality differed from the quality criteria related to conventional systematic reviews.

In total, we analysed 11 different dimensions of methodological quality. The overall trend showed that all point estimates were above 1, and the methodological quality tended to be lowest among industry reviews (i.e. they tended to fulfil the dimensions of the quality less often) (Analysis 3.1 to Analysis 3.11). For three dimensions, the difference in methodological quality was statistically significant (Analysis 3.1; Analysis 3.3; Analysis 3.8).

3.1. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 1 Low methodological quality related to search methods.

3.11. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 11 Low methodological quality related to having conclusions supported by the data.

3.3. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 3 Low methodological quality related to assessing risk of bias.

3.8. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 8 Low methodological quality related to interpretation of results in light of risk of bias.

In two of these 11 analyses (Analysis 3.2; Analysis 3.3), we found substantial statistical heterogeneity (I²of 60% and 57%, respectively). One study was somewhat different from the other studies in these analyses, because the authors had used the AMSTAR tool to assess methodological quality and included both drug and device systematic reviews (Gomez‐Garcia 2017).

3.2. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 2 Low methodological quality related to selecting studies.

The study on methodological quality in the network meta‐analyses had similar results (Chambers 2015, Appendix 5).

Subgroup analyses

Due to lack of data, it was only meaningful to carry out three of five planned subgroup analyses (Appendix 6).

For frequency of favourable conclusions, we found no statistically significant difference between studies with high risk of bias compared with those with low risk of bias (RR: 1.81, 95% CI: 1.21 to 2.69, based on six studies of 397 systematic reviews; versus RR: 15.00, 95% CI: 1.02 to 220.92, based on one study of 14 systematic reviews, test for interaction P value: 0.13; Analysis 4.1).

4.1. Analysis.

Comparison 4 Subgroup analyses for primary outcomes, Outcome 1 Frequency of favourable conclusions: studies with high risk of bias versus studies with low risk of bias.

Our subgroup analysis comparing frequency of favourable conclusions between Cochrane Reviews and non‐Cochrane systematic reviews found no statistically significant difference between non‐Cochrane systematic reviews (RR: 1.32, 95% CI: 1.15 to 1.52, based on six studies of 351 non‐Cochrane systematic reviews) and Cochrane Reviews (RR: 2.17, 95% CI: 0.63 to 7.44, based on three studies of 38 Cochrane Reviews, test for interaction P value: 0.43, Analysis 4.2). Seven of the observational studies we included in this methodology review provided data from 411 systematic reviews (for our primary analysis of frequency of favourable conclusions). Out of these 411 systematic reviews, 53 were Cochrane Reviews (of which 38 were included in our subgroup analysis comparing Cochrane Reviews and non‐Cochrane Reviews). The authors of the included observational studies classified 15 of the 53 Cochrane Reviews as having authors with financial conflicts of interest or as having received industry funding. One of the 15 Cochrane Reviews was partly funded by the industry (published in 2004), and two had lead authors with financial conflicts of interest (published in 2009 and 2011). In ten of the Cochrane Reviews, less than half of the non‐lead authors had financial conflicts of interest (published between 2009 and 2013), and in the remaining two Cochrane Reviews, the reason for the classification was unclear.

4.2. Analysis.

Comparison 4 Subgroup analyses for primary outcomes, Outcome 2 Frequency of favourable conclusions: Cochrane Reviews versus non‐Cochrane systematic reviews.

For differences in the definition of favourable conclusions, we found no statistically significant difference between conclusions in favour of the interventions (RR: 1.94, 95% CI: 0.93 to 4.07, based on four studies of 173 systematic reviews) and conclusions recommending the intervention without reservations (RR: 2.11, 95% CI: 1.18 to 3.79, based on three studies of 238 systematic reviews) (test for interaction P value: 0.86; Analysis 4.3).

4.3. Analysis.

Comparison 4 Subgroup analyses for primary outcomes, Outcome 3 Frequency of favourable conclusions: recoding favourable conclusions.

Sensitivity analyses

We were able to carry out all of the four pre‐planned sensitivity analyses and the four post hoc sensitivity analyses for at least one of our primary outcomes (Appendix 7). In general, all sensitivity analyses showed similar findings for frequency of statistically favourable results and frequency of favourable conclusion as our primary analyses.

The statistical heterogeneity in our primary analysis of frequency of favourable conclusions was substantial (Analysis 2.1), and the RR spanned from 1.25 to 15.00. One of the studies (Yank 2007) compared systematic reviews with financial conflicts of interest to a single drug company with a group of industry and non‐industry systematic reviews. Excluding this atypical study, the statistical heterogeneity decreased substantially (I² reduced from 69% to 0%), but the effect estimate remained approximately the same (RR: 2.03, 95% CI: 1.56 to 2.64, Analysis 5.9).

5.9. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 9 Excluding one atypical study (Yank 2007): frequency of favourable conclusions.

We also carried out sensitivity analyses for 10 dimensions of methodological quality. These analyses showed similar findings as our primary analyses.

Assessment of certainty of the evidence

See Table 1.

The certainty of the evidence on estimated treatment effect was assessed as low. Only one study (Jorgensen 2006) investigated estimated treatment effect based on 14 systematic reviews (certainty downgraded for imprecision). The study may therefore be underpowered, and we find it likely that further research will have an impact on the estimate and our confidence in the estimate.

The certainty of the evidence on frequency of statistically favourable results was assessed as very low. Only one study (Yank 2007) investigated frequency of statistically favourable results, and this study was assessed as high risk of bias (certainty downgraded for limitation in design) and the confidence interval was quite wide (95% CI: 0.62 to 1.14, Analysis 1.1) (certainty downgraded for imprecision).

The certainty of the evidence on frequency of favourable conclusions was assessed as moderate. In our assessment, we did not up‐ or downgrade the certainty. The effect estimate was fairly large and based on seven studies (RR: 1.98, Analysis 2.1). Among the one study assessed as low risk of bias, the effect estimate was substantially higher, though with a wide confidence interval (RR: 15.00, 95% CI: 1.02 to 220.92, Analysis 4.1).

Discussion

Summary of main results

We included 10 studies that investigated a total of 1010 systematic reviews. All studies included systematic reviews of drug interventions (995 systematic reviews), and only one study also included systematic reviews of device interventions (15 systematic reviews). We found no statistically significant difference in results (estimated treatment effect and frequency of statistically favourable results) between systematic reviews with and without financial conflicts of interest. Systematic reviews with financial conflicts of interest more frequently had favourable conclusions compared with systematic reviews without. Systematic reviews with financial conflicts of interest tended to have lower methodological quality compared with systematic reviews without these financial conflicts of interest. Our findings were robust in a number of sensitivity analyses, and analyses based on matched studies or adjusted regression analyses (which had thereby reduced the risk of confounding) had similar results. Only two of the 10 included studies were assessed as low risk of bias.

Quality of the evidence

The majority of the included studies were assessed as having high risk of bias, because the investigated systematic reviews differed in aspects other than financial conflicts of interest (e.g. investigated different interventions used for different diseases). Thus, the included studies had a risk of confounding. Two of the included studies used a matched design and thereby reduced the risk of confounding markedly, but did not eliminate it (Jorgensen 2006; Lane 2013). We also noted a less pronounced reduction of risk of confounding in a third study adjusting for confounders (Yank 2007). Our analyses of results (estimated treatment effect and statistically favourable results) had some risk of confounding and were imprecise, and our analysis of conclusions and methodological quality had some risk of confounding.

We assessed the certainty of the evidence using the GRADE approach. All included studies were of observational design, and certainties of evidence ranged from very low to low. Despite being possible in theory, randomisation is not feasible in these types of studies, and the assessment may be too conservative and should be considered tentative.

Potential biases in the review process

Our review has several strengths. We obtained unpublished data from five studies, which enabled us to conduct subgroup and sensitivity analyses. We found no unpublished studies despite doing a comprehensive search for conference abstracts and unpublished literature. We addressed selective outcome reporting by comparing the published studies with their corresponding protocols for five studies (Ebrahim 2016; Gomez‐Garcia 2017; Jorgensen 2006; Jorgensen 2008; Lane 2013) and found no signs of selective outcome reporting.

We only found one study investigating estimated treatment effect. Investigating effect sizes involves complicated statistics in measuring differences between effect estimates expressed in varying units. Surprisingly, we only found one study investigating statistically favourable results, which seems contradictory to the fact that multiple studies investigated favourable conclusions. However, based on the five protocols we had access to, we have no reason to believe that this is due to selective reporting.

Nevertheless, our review has some limitations. First, our ability to conduct subgroup and sensitivity analyses was limited by inclusion of only 10 studies. Second, we only identified one study investigating estimated treatment effect and only one study investigating frequency of statistically favourable results. Third, for the association between financial conflicts of interest and frequency of favourable conclusions we found substantial statistical heterogeneity (I²: 69%). One possible reason for this was the inclusion of one study that compared systematic reviews with financial conflicts of interest to one drug company to a comparison group comprised of both industry and non‐industry reviews (Yank 2007). When we excluded this study from the analysis, the statistical heterogeneity decreased to an I² of 0%.

Agreements and disagreements with other studies or reviews

A Cochrane Methodology Review by Lundh and colleagues investigated the impact of industry sponsorship on outcomes in individual research studies, mainly clinical trials. It found that industry sponsored studies more often had statistically favourable results (RR: 1.27, 95% CI: 1.17 to 1.37) and favourable conclusions (RR: 1.34, 95% CI: 1.19 to 1.51) compared with non‐industry sponsored studies (Lundh 2017). Contrary to that study, we did not find an association between financial conflicts of interest and statistically favourable results in systematic reviews. This may be due, at least in part, to systematic reviews on the same topic often including the same randomised trials and thereby obtaining similar results in their meta‐analyses. However, our findings on estimated treatment effect and statistically favourable results are based on only one study each and are fairly imprecise with wide confidence intervals. Furthermore, the authors of the other Cochrane Methodology Review found no overall difference in risk of bias between industry and non‐industry funded trials (Lundh 2017). This is in contrast to the tendency that industry reviews had lower methodological quality than non‐industry reviews that we found. One potential explanation for the higher quality of non‐industry reviews could be that such reviews may more often be authored by methodologists (Gotzsche 2012).

The association between financial conflicts of interest and conclusions has also been investigated in relation to narrative reviews, editorials, and letters to the editor for drug interventions in a number of studies. Authors with financial conflicts of interest seem to recommend a drug more often compared with authors without such conflicts of interest (Stelfox 1998; Wang 2010), but we are not aware of any systematic review of this topic.

Finally, financial conflicts of interest have also been investigated in relation to non‐drug and non‐device industries. Chartres and colleagues undertook a systematic review of the association between financial conflicts of interest in relation to the food industry and study outcomes. They found a tendency that primary research studies and reviews with financial conflicts of interest more often had favourable conclusions compared with studies without (RR: 1.31, 95% CI: 0.99 to 1.72) (Chartres 2016). Similarly, Barnes and colleagues found that authors of review articles with financial conflicts of interest related to the tobacco industry more often concluded that passive smoking is not harmful compared with reviews without such conflicts of interest (odds ratio (OR): 88.4; 95% CI: 16.4 to 476.5) (Barnes 1998).

Meaning of our review

The association between financial conflicts of interest and conclusions of systematic reviews may be explained by underlying differences in results (i.e. estimated treatment effect and frequency of statistically significant results). However, we found no statistically significant difference in review results between industry and non‐industry reviews. Even though our analyses of results had some risk of confounding and were imprecise, this could suggest that results are interpreted differently in industry and non‐industry reviews, which might be associated with the lower methodological quality of industry reviews. For example, industry reviews less often interpret results in the light of risk of bias of included studies. Another reason could be a more frequent use of spin in industry review conclusions (Yavchitz 2016). Nonetheless, our finding on the association between financial conflicts of interest and review results remains uncertain, because it is based on only one study. Second, any association may be affected by confounding (e.g. if industry and non‐industry reviews investigate different types of interventions used for different diseases). However, it is unlikely that the differences in frequency of favourable conclusions is an issue of confounding, because the association was also found in one study that used a matched design (Jorgensen 2006).

Authors' conclusions

Implication for methodological research.

We found only single studies on the association between financial conflicts of interest and estimated treatment effect and frequency of statistically favourable results, respectively, and our findings for these outcomes are imprecise. Future research could focus on establishing the degree to which financial conflicts of interest are associated with the results of systematic reviews. Furthermore, none of the included studies investigated the association between financial conflicts of interest and results about the potential harms of interventions, and future studies could investigate this.

According to our 'Risk of bias' assessment, only two studies investigated comparable samples of systematic reviews. Future research should focus on comparing industry and non‐industry systematic reviews that are similar in important aspects other than financial conflicts of interest. Furthermore, the included studies defined financial conflict of interest differently and it remains unclear whether some types of financial conflicts of interest have a greater impact. Future studies should investigate various types of financial conflicts of interest, including non‐financial conflicts of interest.

None of the included studies provided an explanatory mechanism for the association between financial conflicts of interest and conclusions of systematic reviews. Future empirical studies should try to address such explanatory mechanisms, for example by addressing the use of spin in systematic review conclusions.

Appendices

Appendix 1. PubMed search strategy

Block 1: drug and device industry

1. Drug Industry (MeSH)

2. (Drug [Title/Abstract] OR drugs[Title/Abstract] OR pharmaceutical[Title/Abstract] OR pharmaceutic [Title/Abstract] OR pharmacological[Title/Abstract] OR pharma*[Title/Abstract] OR biotech*[Title/Abstract] OR biopharma*[Title/Abstract] OR biomed*[Title/Abstract] OR device[Title/Abstract] OR devices[Title/Abstract] OR imaging[Title/Abstract] OR for‐profit[Title/Abstract] OR private[Title/Abstract]) AND (industry[Title/Abstract] OR industries[Title/Abstract] OR company[Title/Abstract] OR companies[Title/Abstract] OR manufacturer[Title/Abstract] OR manufacturers[Title/Abstract] OR organisation[Title/Abstract] OR organisations[Title/Abstract] OR organization[Title/Abstract] OR organizations[Title/Abstract] OR agency[Title/Abstract] OR agencies[Title/Abstract] OR sector[Title/Abstract] OR sectors[Title/Abstract])

3. Health[Title/Abstract] AND (industry[Title/Abstract] OR industries[Title/Abstract])

4. 1 OR 2 OR 3

Block 2: conflicts of interest and industry funding

5. Conflict of interest (MeSH)

6. Financial support (MeSH)

7. Research support as topic (MeSH)

8. (Conflict[Title/Abstract] OR conflicts[Title/Abstract] OR conflicting[Title/Abstract]) AND (interest[Title/Abstract] OR interests[Title/Abstract])

9. (Competing[Title/Abstract] OR vested[Title/Abstract]) AND (interest[Title/Abstract] OR interests[Title/Abstract])

10. (Industry[Title/Abstract] OR industries[Title/Abstract] OR company[Title/Abstract] OR companies[Title/Abstract] OR manufacturer[Title/Abstract] OR manufacturers[Title/Abstract] OR finance[Title/Abstract] OR financial[Title/Abstract]) AND (funded[Title/Abstract] OR funding[Title/Abstract] OR sponsor[Title/Abstract] OR sponsors[Title/Abstract] OR sponsorship[Title/Abstract] OR sponsoring[Title/Abstract] OR support[Title/Abstract] OR supported[Title/Abstract] OR finance[Title/Abstract] OR financial[Title/Abstract] OR involvement[Title/Abstract] OR involving[Title/Abstract] OR payment[Title/Abstract] OR payments[Title/Abstract] OR relationship[Title/Abstract] OR relationships[Title/Abstract] OR relation[Title/Abstract] OR relations[Title/Abstract] OR tie[Title/Abstract] OR ties[Title/Abstract])

11. Industry‐funded[Title/Abstract] OR industry‐funding[Title/Abstract] OR industry‐sponsor*[Title/Abstract] OR company‐funded[Title/Abstract] OR company‐funding[Title/Abstract] OR company‐sponsor*[Title/Abstract] OR industry‐support[Title/Abstract] OR industry‐supported[Title/Abstract] OR company‐support[Title/Abstract] OR company‐supported[Title/Abstract]

12. (Commercial‐academic[Title/Abstract] OR academic‐industry[Title/Abstract] OR commercial‐industry[Title/Abstract] OR physician‐industry[Title/Abstract]) AND (interface[Title/Abstract] OR interfaces[Title/Abstract] OR interaction[Title/Abstract] OR interactions[Title/Abstract] OR relationship[Title/Abstract] OR relationships[Title/Abstract] OR relation[Title/Abstract] OR relations[Title/Abstract])

13. 5 OR 6 OR 7 OR 8 OR 9 OR 10 OR 11 OR 12

Block 3: systematic reviews or meta‐analyses

14. Review Literature as Topic (MeSH)

15. Meta‐Analysis as Topic (MeSH)

16. Meta‐analy*[Title/Abstract] OR metaanal*[Title/Abstract]

17. Meta[Title/Abstract] AND analy*[Title/Abstract]

18. (Systematic[Title/Abstract] OR systematically[Title/Abstract] OR systematical[Title/Abstract] OR Cochrane[Title/Abstract] OR literature[Title/Abstract] OR literatures[Title/Abstract]) AND (review[Title/Abstract] OR reviews[Title/Abstract] OR overview[Title/Abstract] OR overviews[Title/Abstract])

19. 14 OR 15 OR 16 OR 17 OR 18

Combined searches

20. 4 AND 13 AND 19

Appendix 2. Data extraction

Two review authors independently extracted data on the following.

Basic characteristics

1. Title

2. Year published

3. Name of first author

4. Name of journal

5. Primary aim of the study

6. Study design used in the study (cohort, cross‐sectional, systematic review or meta‐analysis, other)

7. Study domain (i.e. topic of interest) of systematic reviews. Verbatim extraction

8. Study domain coded (specific disease, specific therapy, mixed domain)

9. Sample strategy used to locate systematic reviews or meta‐analyses (e.g. search of PubMed and time period covered). Verbatim extraction

10. Types of publications (published/unpublished) included in the systematic reviews or meta‐analyses. Verbatim extraction

11. Types of publications included in the systematic reviews or meta‐analyses (published only, published and unpublished, not described)

12. Definition of systematic reviews or meta‐analyses used in the study. Verbatim extraction

13. Number of systematic reviews or meta‐analyses included in the study

14. Types of studies included in systematic review or meta‐analysis (e.g. clinical trials or cohort studies)

Outcome data

1. Definition of industry funding used in the study. Verbatim extraction

2. Definition of author conflicts of interest used in the study. Verbatim extraction

3. Definition of effect size estimates used in the study. Verbatim extraction

4. Definition of statistically favourable results (e.g. based on statistical significance) used in the study. Verbatim extraction

5. Definition of favourable conclusions used in the study. Verbatim extraction

6. Definition of methodological quality used in the study. Verbatim extraction

7. Definition of primary analysis used in the study. Verbatim extraction

8. Data on estimates of the association between financial conflicts of interest and estimated treatment effect

9. Data on estimates of the association between financial conflicts of interest and statistically favourable results

10. Data on estimates of the association between financial conflicts of interest and favourable conclusions

Data for informing subgroup analyses or reflection on heterogeneity

1. Types of interventions included in systematic reviews or meta‐analyses. Verbatim extraction

2. Types of interventions included in systematic reviews or meta‐analyses (drug, device, mixed)

3. Data on estimates of the association between industry funding and estimated treatment effect

4. Data on estimates of the association between author financial conflicts of interest and estimated treatment effect

5. Data on estimates of the association between industry funding and statistically favourable results

6. Data on estimates of the association between author financial conflicts of interest and statistically favourable results

7. Data on estimates of the association between industry funding and favourable conclusions

8. Data on estimates of the association between author financial conflicts of interest and favourable conclusions

9. Data on estimates of the association between industry funding and methodological quality

10. Data on estimates of the association between author financial conflicts of interest and methodological quality

11. Data on estimates of the association between different degrees of financial conflicts of interest (e.g. mild, moderate, and severe) and estimated treatment effect

12. Data on estimates of the association between different degrees of financial conflicts of interest (e.g. mild, moderate, and severe) and statistically favourable results

13. Data on estimates of the association between different degrees of financial conflicts of interest (e.g. mild, moderate, and severe) and favourable conclusions

Additional data

1. Declaration of funding source and other conflicts of interest in the study. Verbatim extraction

2. Additional relevant data

Appendix 3. Dealing with missing data

If the included studies investigated both industry funding and author financial conflicts of interest, but reported them together, we contacted the authors to obtain separate data. If the included studies contained a mixture of systematic reviews and other study designs or mixed domains (e.g. pharmaceutical and nutritional interventions), we contacted the authors to obtain separate data for systematic reviews on drug or device studies. If included studies investigated methodological quality, but reported the overall quality score only, we contacted the authors to obtain data on each item in the measurement tool used. In total, we contacted authors of six studies (Ebrahim 2016; Gomez‐Garcia 2017; Hartog 2012; Jorgensen 2008; Wang 2010; Yank 2007) and received data for five of these (Gomez‐Garcia 2017; Hartog 2012; Jorgensen 2008; Wang 2010; Yank 2007).

If the included studies did not report their methods in a way that enabled us to conduct our 'Risk of bias' assessment, we contacted the authors in an attempt to clarify these issues. In total, we contacted authors of nine studies and received clarifications for all nine of these (Chambers 2015; Dunn 2014; Ebrahim 2016; Gomez‐Garcia 2017; Hartog 2012; Jorgensen 2006; Jorgensen 2008; Wang 2010; Yank 2007).

We contacted authors of included studies in an attempt to obtain published or unpublished protocols for the studies. In total, we contacted authors of all 10 studies. All authors responded, five author teams supplied us with their protocols (Ebrahim 2016; Gomez‐Garcia 2017; Jorgensen 2006; Jorgensen 2008; Lane 2013).

Appendix 4. Prediction interval

Formula for prediction interval

To calculate prediction intervals, we used the formula presented in an article by Riley et al.:

û ‐ t_k‐2· √(Ƭ²+SE(û)²), û+t_k‐2· √(Ƭ²+SE(û)²)

Where û was the estimate of the average effect measure across studies, SE(û) was the standard error of û, Ƭ was the estimate of between study standard deviation, and t_k‐2 was the 100(1‐(α/2)) percentile of the t‐distribution with k‐2 degrees of freedom, where k was the number of observational studies in the meta‐analysis and α was 0.05 to give a 95% prediction interval. To meet the assumption on normal distribution, the prediction interval was derived on the natural log scale (Riley 2011). As Ƭ² is already a measure for the heterogeneity for ln(RR), this was used directly in the calculation (IntHout 2016).

Calculation of prediction interval

As our analyses on estimated treatment effect and frequency of statistically favourable results were based on one study each, calculation of prediction interval was only possible for one of our primary outcomes: frequency of favourable conclusions.

From Analysis 2.1, û was given as 1.98 with a 95% CI from 1.26 to 3.11, Ƭ² and was given as 0.15. The analysis on frequency of favourable conclusions included seven studies, which provided five degrees of freedom according to the formula. The 0.975 percentile of the t distribution with four degrees of freedom was 2.571 (Rosner 2006).

To calculate SE(û), we used the formula for transforming confidence intervals to the natural log scale from the Cochrane Handbook (Higgins 2011):

lower limit = ln(lower confidence limit given for RR)

upper limit = ln(upper confidence limit given for RR)

Thus, our transformed confidence interval was:

lower limit = ln(1.26) → lower limit = 0.231112

upper limit = ln(3.11) → upper limit = 1.134623

We used the formula for calculating standard errors from the Cochrane Handbook (Higgins 2011):

SE=(upper limit ‐ lower limit) / 3.92

Thus, we calculated the standard error as:

SE=(1.134623‐0.231112) / 3.92 → SE=0.230488

Finally, the prediction interval on the natural logarithm scale was calculated:

Prediction interval: ln(1.98) ‐ 2.571· √(0.15+0.230488²), ln(1.98) + 2.571· √(0.15+0.230488²) →

Prediction interval: ‐0.47564‐1.84183

Thus, the prediction interval for the risk ratio of frequency of favourable conclusions in industry reviews compared with non‐industry reviews was calculated as: 0.62 to 6.31.

Appendix 5. Methodological quality of the systematic reviews

Table A7.1: Methodological quality: Oxman and Guyatt index Number of systematic reviews assessed as not fulfilled or unclear in each item
	Jorgensen 2006		Jorgensen 2008		Yank 2007
	Industry N(%)	Non‐industry N(%)	Industry N(%)	Non‐industry N(%)	Industry N(%)	Non‐industry N(%)
1) Were the search methods used to find evidence on the primary question stated?	5 (63)	0 (0)	4 (40)	5 (17)
2) Was the search for evidence reasonable comprehensive?*	5 (63)	0 (0)	6 (60)	6 (21)
3) Were the criteria used for deciding which studies to include reported?	3 (38)	0 (0)	5 (50)	2 (7)
4) Was bias in the selection of studies avoided?*	6 (75)	0 (0)	9 (90)	14 (48)
5) Were the criteria used for assessing the validity of the included studies reported?	6 (75)	0 (0)	7 (70)	9 (31)
6) Was the validity of all studies referred to in the text assessed using appropriate criteria?*	8 (100)	0 (0)	9 (90)	18 (62)
7) Were the methods used to combine the findings of the relevant studies reported?	0 (0)	0 (0)	1 (10)	2 (7)
8) Were the findings of the relevant studies combined appropriately?*	4 (50)	0 (0)	3 (30)	5 (17)
9) Were the conclusions made by the author(s) supported by the data reported?*	3 (38)	0 (0)	1 (10)	5 (17)
Median quality score	2**	7**	2.5**	5**	4***	9***

* Items address methodological quality and were included in pooled analysis

**Score range from 0 to 7

***Score range from 0 to 18, as the authors assigned two points for fulfilling each criteria, one point for partially fulfilling it, and zero points for not fulfilling it (Yank 2007)

Table A7.2: Methodological quality: Lane 2013 (own tool) Number of systematic reviews assessed as not fulfilled or unclear in each item
	Lane 2013
	Industry N(%)	Non‐industry N(%)
A: Were the review methods adequate such that biases in location and assessment of studies were minimized or able to be identified?*	30 (48)	18 (29)
B: Were the individual studies analysed appropriately and without avoidable bias?*	42 (67)	32 (51)
C: Were the basic meta‐analysis methods appropriate?*	36 (57)	33 (52)
D: Are the conclusions justified and the interpretation sound?*	37 (59)	30 (48)
Q8: Was the search for evidence reasonable comprehensive?*	31 (49)	20 (32)
Q11: Was risk of bias (or quality) assessed for each included study?*	45 (71)	20 (32)
Q15 Are adequate methods used to address missing outcome data?*	55 (87)	50 (79)
Q26 Was a sensible strategy used to address statistical heterogeneity in meta‐analysis?	42 (67)	32 (51)
Q39: Were results appropriately interpreted in the light of risk of bias in included studies?*	47 (75)	35 (56)
Q40 Were results appropriately interpreted in the light of risk of reporting bias?*	40 (63)	37 (59)
Q41 Were results appropriately interpreted in the light of any multiplicity?*	28 (44)	22 (35)

* Items address methodological quality and were included in pooled analysis

Table A7.3: Methodological quality: AMSTAR Number of systematic reviews assessed as not fulfilled or unclear in each item
	Gomez‐Garcia 2017
	Industry N(%)	Non‐industry N(%)
1) Was an 'a priori' design provided?	31 (48)	39 (68)
2) Was there duplicate study selection and data extraction?*	31 (48)	27 (47)
3) Was a comprehensive literature search performed?*	9 (14)	7 (12)
4) Was the status of publication (i.e. grey literature) used as an inclusion criterion?	33 (52)	31 (54)
5) Was a list of studies (included and excluded) provided?	53 (83)	50 (88)
6) Were the characteristics of the included studies provided?	5 (8)	8 (14)
7) Was the scientific quality of the included studies assessed and documented?*	27 (42)	15 (26)
8) Was the scientific quality of the included studies used appropriately in formulating conclusions?*	31 (48)	16 (28)
9) Were the methods used to combine the findings of studies appropriate?*	27 (42)	27 (47)
10) Was the likelihood of publication bias assessed?*	53 (83)	47 (82)
11) Was the conflict of interest included?	12 (19)	24 (42)

* Items address methodological quality and were included in pooled analysis

Table A7.4: Methodological quality: ISPOR guidance Number of systematic reviews assessed as not fulfilled or unclear in each item
	Chambers 2015
	Industry N(%)	Non‐industry N(%)
1) Was a Bayesian or a frequentist framework used?	23 (23)	78 (36)
2) Was the risk of bias of included clinical trials assessed? (e.g., using the Cochrane Collaboration's tool for assessing risk of bias or the Jadad scale?)*	45 (46)	47 (22)
3) Did the analysis include adjustments for model covariates?	61 (62)	162 (75)
4) Was a fixed or random‐effects model used? Or, were the findings of both fixed and random‐effects models presented?	31 (32)	62 (29)
5) Was an assessment of model fit reported?	52 (53)	136 (63)
6) Was a sensitivity analysis performed? (e.g. varied the included clinical studies to evaluate the robustness of the findings)*	41 (42)	95 (44)
7) For studies with at least one closed loop, was the consistency of direct evidence and indirect evidence evaluated? (i.e., presented and compared the findings from the traditional meta‐analysis and the network meta‐analysis)*	82 (84)	117 (54)
8) Were the search terms reported?	37 (38)	24 (11)
9) Was a network diagram of included treatments presented?	36 (37)	85 (39)
10) Was data from the included clinical studies necessary to reproduce the network meta‐analysis presented?	40 (41)	68 (31)
11) Was a table of key clinical study characteristics presented?	9 (9)	20 (9)
12) Was the model code presented or source cited? (reported for studies performed using a Bayesian framework only)	90 (92)	190 (88)
13) Were pairwise comparisons of all included treatments presented?	54 (55)	58 (27)
14) Was the probability of each treatment being best reported? (reported for studies performed using a Bayesian framework only)	73 (74)	155 (71)
15) Was a ranking of treatments in terms of effectiveness reported? (reported for studies performed using a Bayesian framework only)	87 (89)	161 (74)

*Item address methodological quality

Appendix 6. Subgroup analyses

High risk of bias studies versus low risk of bias studies

We planned to compare studies with high and low risk of bias for our primary outcomes. Only one study investigated estimated treatment effect, and only one study investigated statistically favourable results. Thus, our data enabled us to carry out our subgroup analysis for one of our primary outcomes: frequency of favourable conclusions.

We found lower RR for studies assessed as high risk of bias (RR: 1.81, 95% CI: 1.21 to 2.69), than for studies assessed as low risk of bias (RR: 15.00, 95% CI: 1.02 to 220.92). The difference was not statistically significant (P value: 0.13, Analysis 4.1).

The statistical heterogeneity among studies assessed as high risk of bias was substantial (I²: 64%, Analysis 4.1). However, the effect estimates showed the same directionality and did not differ substantially (RR from 1.25 at the lowest to 3.26 at the highest). Thus, the statistical heterogeneity may be driven by the relatively high number of systematic reviews included in the analysis (n = 411). Only one study assessed as low risk of bias was included in this subgroup analysis.

Cochrane Reviews versus non‐Cochrane systematic reviews

The one study investigating estimated treatment effect (Jorgensen 2006) compared Cochrane Reviews with industry reviews, and re‐analysing the sample of studies for estimated treatment effect was not meaningful. Furthermore, the one study investigating statistically favourable results (Yank 2007) included only one Cochrane Review and recoding the sample was not considered appropriate for frequency of statistically favourable results. Thus, our data enabled us to carry out this subgroup analysis on one of our primary outcomes: frequency of favourable conclusions.

In total, six studies could be pooled in the subgroup with non‐Cochrane Reviews only and three studies could be pooled in a subgroup with Cochrane Reviews only (Jorgensen 2006, Yank 2007 and Dunn 2014 did not include any Cochrane Reviews classified as having authors with financial conflicts of interest or as having received industry funding). Our analysis included a total of 38 Cochrane Reviews, on which 15 Cochrane Reviews were classified as having authors with financial conflicts of interest or as having received industry funding by the authors of the included studies. We found lower RR for non‐Cochrane Reviews (RR: 1.32, 95% CI: 1.15 to 1.52) than for Cochrane Reviews (RR: 2.17, 95% CI: 0.63 to 7.44). The difference was not statistically significant (P value: 0.43, Analysis 4.2).

Systematic reviews of drug studies versus systematic reviews of device studies

Only one study (Gomez‐Garcia 2017) included systematic reviews of device studies and this study investigated methodological quality only. Our data thereby did not enable us to carry out a subgroup analysis comparing systematic reviews of drug studies with systematic reviews of device studies.

Systematic reviews with major financial conflicts of interest versus systematic reviews with moderate financial conflicts of interest versus systematic reviews with minor financial conflicts of interest according to the definitions used by the authors of the included studies

We planned to compare different degrees of financial conflicts of interest (i.e. severe, moderate, mild). However, none of the included studies graded the degree of financial conflicts of interest therefore, we were not able to carry out this subgroup analysis.

Studies defining favourable conclusions as conclusions in favour of the intervention versus studies defining favourable conclusions as conclusions recommending the intervention without reservations

Our subgroup analysis comparing conclusions in favour of the intervention and conclusions recommending the intervention without reservations showed minor differences between the two groups of favourable conclusions.

We found lower RR for conclusions in favour of the intervention (RR: 1.94, 95% CI: 0.93 to 4.07) than for interventions recommending the intervention without reservations (RR: 2.11, 95% CI: 1.18 to 3.79). The difference was not statistically significant (P value: 0.86). The statistical heterogeneity remained approximately the same (I² from 69% to 55%) for conclusions in favour of the intervention, whereas it decreased substantially (I² from 69% to 25%) for conclusions recommending the intervention without reservations (Analysis 4.3).

Appendix 7. Sensitivity analyses

Re‐categorising industry reviews into systematic reviews with industry funding only (i.e. excluding systematic reviews with author financial conflicts of interest from the industry group) and comparing with non‐industry reviews (i.e. systematic reviews without industry funding and author financial conflicts of interest)

We planned to re‐categorise industry reviews to systematic reviews with industry funding only. However, as only one study investigated estimated treatment effect and this study restricted the industry group to industry funding only, our data only enabled us to carry out this sensitivity analysis on two of our primary outcomes: frequency of statistically favourable results and frequency of favourable conclusions.

Our re‐categorisation showed similar results as our primary analyses, however with wider confidence intervals due to less data. For frequency of statistically favourable results, the effect estimate remained statistically insignificant (RR: 0.82, 95% CI: 0.47 to 1.42, Analysis 5.1). For frequency of favourable conclusions, the effect estimate decreased slightly, but remained statistically significant (RR: 1.60, 95% CI: 1.03 to 2.48, Analysis 5.2).

5.1. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 1 Re‐categorising industry reviews to systematic reviews with industry funding only: frequency of statistically favourable results.

5.2. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 2 Re‐categorising industry reviews to systematic reviews with industry funding only: frequency of favourable conclusions.

Re‐categorising industry reviews into systematic reviews with author financial conflicts of interest only (i.e. excluding systematic reviews with industry funding from the industry group) and comparing with non‐industry reviews (i.e. systematic reviews without author financial conflicts of interest and industry funding)

We planned to re‐categorise industry reviews into systematic reviews with author financial conflicts of interest only. However, as only one study investigated estimated treatment effect and this study restricted the industry group to industry funding only, our data only enabled us to carry out this sensitivity analysis on two of our primary outcomes: frequency of statistically favourable results and frequency of favourable conclusions.

Our re‐categorisation showed similar results as our primary analyses, however with wider confidence intervals due to less data. For frequency of statistically favourable results, the effect estimate increased, but remained statistically insignificant (RR: 1.12, 95% CI: 0.78 to 1.63, Analysis 5.3). For frequency of favourable conclusions, the effect estimate decreased, but remained statistically significant (RR: 1.27, 95% CI: 1.08 to 1.49, Analysis 5.4).

5.3. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 3 Re‐categorising industry reviews to systematic reviews with author financial conflicts of interest only: frequency of statistically favourable results.

5.4. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 4 Re‐categorising industry reviews to systematic reviews with author financial conflicts of interest only: frequency of favourable conclusions.

Excluding included studies with financial conflicts of interest

We planned to re‐analyse our primary outcomes excluding studies with financial conflicts of interest according to the declarations. Three studies declared any financial conflicts of interest related to for‐profit organisations. Lane 2013 declared that GSK teleconference facilities were used for meetings related to the study and that several authors were employees at pharmaceutical companies at the time the study was conducted. Gomez‐Garcia 2017 declared that no funding was received for the study by any pharmaceutical companies, but several authors had received honoraria for research and lecturing from different pharmaceutical companies. Hartog 2012 declared that one author had received research grants, and speaker/consultancy fees from a pharmaceutical company.

Lane 2013 and Gomez‐Garcia 2017 solely investigated our secondary outcome, and Hartog and colleagues investigated frequency of favourable conclusions. Therefore, we were able to carry out this sensitivity analysis for one of our outcomes: frequency of favourable conclusions.

Our sensitivity analyses showed similar results as our primary analysis. The effect estimate remained approximately the same and remained statistically significant (RR: 1.96, 95% CI: 1.23 to 3.13, Analysis 5.5).

5.5. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 5 Excluding studies with financial conflicts of interest: frequency of favourable conclusions.

Re‐analysing our data using fixed‐effect models

As the analyses for estimated treatment effect and frequency of statistically favourable results included only one study each, our re‐analysis of our primary outcomes using fixed‐effect models was only carried out for one of our primary outcomes: frequency of favourable conclusions.

Our sensitivity analysis showed similar results as our primary analysis for frequency of favourable conclusions. The effect estimate decreased slightly and remained statistically significant (RR: 1.68, 95% CI: 1.43 to 1.98, Analysis 5.6).

5.6. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 6 Re‐analysing using fixed‐effect models: frequency of favourable conclusions.

Excluding systematic reviews with unclear or undeclared financial conflicts of interest from the non‐industry group

The only study investigating estimated treatment effect included systematic reviews with undeclared financial conflicts of interest in a separate category and these reviews were thereby not included in our primary analysis. Our data, thus, enabled us to exclude systematic reviews with unclear or undeclared financial conflicts of interest for two of our primary outcomes: frequency of statistically favourable results and frequency of favourable conclusions.

Our sensitivity analysis showed similar results as our primary analyses for statistical favourable results and favourable conclusions however with wider confidence intervals due to less data. The effect estimate for frequency of statistically favourable results decreased slightly, but remained statistically insignificant (RR: 0.79, 95% CI: 0.58 to 1.07, Analysis 5.7). The effect estimate for frequency of favourable conclusions decreased slightly, but remained borderline statistically significant (RR: 1.57, 95% CI: 1.02 to 2.41). The statistical heterogeneity deceased slightly compared to our primary analysis (I² from 69% to 53%, Analysis 5.8).

5.7. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 7 Excluding undeclared conflicts of interest: frequency of statistically favourable results.

5.8. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 8 Excluding undeclared conflicts of interest: frequency of favourable conclusions.

Excluding one atypical study (Yank 2007) from our pooled analyses, because it compared industry reviews (financial conflicts of interest with a single drug company) with a group of both industry and non‐industry reviews (multiple drug companies, no statement, both drug and non‐profit companies, and non‐profit companies)

Our primary analyses on estimated treatment effect and frequency of statistically favourable results were based on one study. Thus, we excluded the atypical study from our pooled analysis on frequency of favourable conclusions.

Our sensitivity analysis showed similar results as our primary analysis. When excluding Yank 2007 from the pooled analysis, the effect estimate increased slightly and the statistical heterogeneity disappeared (RR: 2.03, 95% CI: 1.56 to 2.64, I²: 0%, Analysis 5.9).

Restricting our analyses to studies assessed as low risk of bias in the comparability criteria or studies performing adjusted regression analyses

The only study investigating estimated treatment effect was assessed as low risk of bias in the comparability of systematic reviews, and we did therefore not perform this sensitivity analysis for estimated treatment effect.

For frequency of statistically favourable results, the only study investigating the outcome performed regression analyses adjusted for methodological quality. The authors assessed methodological quality by using the summary scores from the Oxman and Guyatt index. Our analysis based on this study showed similar results as our primary analysis. The effect estimate indicated no difference in frequency of statistically favourable results between industry and non‐industry reviews (OR: 0.99, 95% CI: 0.44 to 2.23, Analysis 5.10). As with our primary analysis, the association was not statistically significant.

5.10. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 10 Restricting to studies with low risk of bias in the comparability criteria or studies performing adjusted regression analyses: frequency of statistically favourable results.

For frequency of favourable conclusions, one study was assessed as low risk of bias in the comparability of systematic reviews (Jorgensen 2006) and one study performed regression analyses adjusted for methodological quality (Yank 2007). An analysis based on these studies showed similar results as our primary analysis. The effect estimate indicated that industry reviews more often had favourable conclusions compared to non‐industry reviews (OR: 20.28, 95% CI: 0.57 to 719.88, Analysis 5.11). Similar to our main analysis the heterogeneity was substantial (I²: 68%).

5.11. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 11 Restricting to studies with low risk of bias in the comparability criteria or studies performing adjusted regression analyses: frequency of favourable conclusions.

Re‐categorising industry reviews into reviews with financial conflicts of interest related to any for‐profit organisation or to the manufacturer of the investigated intervention in two separate analyses

We planned to distinguish between financial conflicts of interest related to any for‐profit organisation and the manufacturer of the investigated intervention. The two studies that investigated estimated treatment effect and frequency of statistically favourable results did not assess whether financial conflicts of interest were related to the manufacturer or any for‐profit organisation. Thus, using unpublished data, we were able to investigate financial conflicts of interest related to any for‐profit organisation and the manufacturer for frequency of favourable conclusions.

For frequency of favourable conclusions, we found similar impact from financial conflicts of interest related to any for‐profit organisation and the manufacturer. In both cases, the effect estimate decreased slightly compared to our primary analysis (RR: 1.61, 95% CI: 1.16 to 2.24 for manufacturer; RR: 1.75, 95% CI: 1.16 to 2.63 for any for‐profit organisation; Analysis 5.12; Analysis 5.13).

5.12. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 12 Financial conflicts of interest related to manufacturer: frequency of favourable conclusions.

5.13. Analysis.

Comparison 5 Sensitivity analyses for primary outcomes, Outcome 13 Financial conflicts of interest related to any for‐profit organisation: frequency of favourable conclusions.

Sensitivity analysis for methodological quality (secondary outcome): restricting our analyses to studies assessed as low risk of bias in the comparability criteria

Two of the studies that investigated methodological quality used a matched design and were assessed as having low risk of bias in the comparability criteria (Jorgensen 2006; Lane 2013). Together, these studies provided information on 10 of the 11 dimensions of methodological quality.

Ten analyses based on these two studies showed similar results as our primary analysis. For five quality dimensions, the effect estimate increased; for four dimensions, the effect estimate remained the same; and for one dimension, the effect estimate decreased (Analysis 6.1‐Analysis 6.10). In the original analysis, three dimensions were statistically significant. In the sensitivity analyses, only one dimension (interpretation of results in light of risk of bias) was statistically significant (RR: 1.34, 95% CI: 1.03 to 1.75, Analysis 6.7).

6.1. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 1 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to search methods.

6.10. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 10 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to having conclusions supported by the data.

6.7. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 7 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to interpretation of results in light of risk of bias.

Data and analyses

Comparison 1. Frequency of statistically favourable results: systematic reviews with financial conflicts of interest versus systematic reviews without.

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Frequency of statistically favourable results	1	124	Risk Ratio (M‐H, Random, 95% CI)	0.84 [0.62, 1.14]

Comparison 2. Frequency of favourable conclusions: systematic reviews with financial conflicts of interest versus systematic reviews without.

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Frequency of favourable conclusions	7	411	Risk Ratio (M‐H, Random, 95% CI)	1.98 [1.26, 3.11]

Comparison 3. Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without.

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Low methodological quality related to search methods	4	302	Risk Ratio (M‐H, Random, 95% CI)	1.79 [1.09, 2.93]
2 Low methodological quality related to selecting studies	4	302	Risk Ratio (M‐H, Random, 95% CI)	1.53 [0.99, 2.36]
3 Low methodological quality related to assessing risk of bias	4	302	Risk Ratio (M‐H, Random, 95% CI)	1.83 [1.22, 2.74]
4 Low methodological quality related to addressing missing outcome data	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.1 [0.94, 1.29]
5 Low methodological quality related to addressing publication bias	1	121	Risk Ratio (M‐H, Random, 95% CI)	1.00 [0.85, 1.18]
6 Low methodological quality related to analysing individual studies appropriately and without avoidable bias	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.31 [0.97, 1.77]
7 Low methodological quality related to combining findings of relevant studies using appropriate meta‐analysis methods	4	302	Risk Ratio (M‐H, Random, 95% CI)	1.06 [0.78, 1.45]
8 Low methodological quality related to interpretation of results in light of risk of bias	2	247	Risk Ratio (M‐H, Random, 95% CI)	1.42 [1.13, 1.79]
9 Low methodological quality related to interpretation of results in light of reporting bias	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.08 [0.82, 1.43]
10 Low methodological quality related to interpretation of results in light of multiplicity	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.27 [0.82, 1.97]
11 Low methodological quality related to having conclusions supported by the data	3	181	Risk Ratio (M‐H, Random, 95% CI)	1.24 [0.87, 1.77]

3.4. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 4 Low methodological quality related to addressing missing outcome data.

3.5. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 5 Low methodological quality related to addressing publication bias.

3.6. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 6 Low methodological quality related to analysing individual studies appropriately and without avoidable bias.

3.7. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 7 Low methodological quality related to combining findings of relevant studies using appropriate meta‐analysis methods.

3.9. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 9 Low methodological quality related to interpretation of results in light of reporting bias.

3.10. Analysis.

Comparison 3 Methodological quality: systematic reviews with financial conflicts of interest versus systematic reviews without, Outcome 10 Low methodological quality related to interpretation of results in light of multiplicity.

Comparison 4. Subgroup analyses for primary outcomes.

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Frequency of favourable conclusions: studies with high risk of bias versus studies with low risk of bias	7		Risk Ratio (M‐H, Random, 95% CI)	Subtotals only
1.1 High risk of bias	6	397	Risk Ratio (M‐H, Random, 95% CI)	1.81 [1.21, 2.69]
1.2 Low risk of bias	1	14	Risk Ratio (M‐H, Random, 95% CI)	15.0 [1.02, 220.92]
2 Frequency of favourable conclusions: Cochrane Reviews versus non‐Cochrane systematic reviews	6		Risk Ratio (M‐H, Random, 95% CI)	Subtotals only
2.1 Non‐Cochrane systematic reviews	6	351	Risk Ratio (M‐H, Random, 95% CI)	1.32 [1.15, 1.52]
2.2 Cochrane Reviews	3	38	Risk Ratio (M‐H, Random, 95% CI)	2.17 [0.63, 7.44]
3 Frequency of favourable conclusions: recoding favourable conclusions	7		Risk Ratio (M‐H, Random, 95% CI)	Subtotals only
3.1 Conclusions in favour of the intervention	4	173	Risk Ratio (M‐H, Random, 95% CI)	1.94 [0.93, 4.07]
3.2 Interventions recommended without reservations	3	238	Risk Ratio (M‐H, Random, 95% CI)	2.11 [1.18, 3.79]

Comparison 5. Sensitivity analyses for primary outcomes.

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Re‐categorising industry reviews to systematic reviews with industry funding only: frequency of statistically favourable results	1	99	Risk Ratio (M‐H, Random, 95% CI)	0.82 [0.47, 1.42]
2 Re‐categorising industry reviews to systematic reviews with industry funding only: frequency of favourable conclusions	3	161	Risk Ratio (M‐H, Random, 95% CI)	1.60 [1.03, 2.48]
3 Re‐categorising industry reviews to systematic reviews with author financial conflicts of interest only: frequency of statistically favourable results	1	101	Risk Ratio (M‐H, Random, 95% CI)	1.12 [0.78, 1.63]
4 Re‐categorising industry reviews to systematic reviews with author financial conflicts of interest only: frequency of favourable conclusions	3	239	Risk Ratio (M‐H, Random, 95% CI)	1.27 [1.08, 1.49]
5 Excluding studies with financial conflicts of interest: frequency of favourable conclusions	6	399	Risk Ratio (M‐H, Random, 95% CI)	1.96 [1.23, 3.13]
6 Re‐analysing using fixed‐effect models: frequency of favourable conclusions	7	411	Risk Ratio (M‐H, Fixed, 95% CI)	1.68 [1.43, 1.98]
7 Excluding undeclared conflicts of interest: frequency of statistically favourable results	1	99	Risk Ratio (M‐H, Random, 95% CI)	0.79 [0.58, 1.07]
8 Excluding undeclared conflicts of interest: frequency of favourable conclusions	6	345	Risk Ratio (M‐H, Random, 95% CI)	1.57 [1.02, 2.41]
9 Excluding one atypical study (Yank 2007): frequency of favourable conclusions	6	287	Risk Ratio (M‐H, Random, 95% CI)	2.03 [1.56, 2.64]
10 Restricting to studies with low risk of bias in the comparability criteria or studies performing adjusted regression analyses: frequency of statistically favourable results	1		Odds Ratio (Random, 95% CI)	0.99 [0.44, 2.23]
11 Restricting to studies with low risk of bias in the comparability criteria or studies performing adjusted regression analyses: frequency of favourable conclusions	2		Odds Ratio (Random, 95% CI)	20.28 [0.57, 719.88]
12 Financial conflicts of interest related to manufacturer: frequency of favourable conclusions	1	162	Risk Ratio (M‐H, Random, 95% CI)	1.61 [1.16, 2.24]
13 Financial conflicts of interest related to any for‐profit organisation: frequency of favourable conclusions	7	411	Risk Ratio (M‐H, Random, 95% CI)	1.75 [1.16, 2.63]

Comparison 6. Sensitivity analysis for secondary outcome.

Outcome or subgroup title	No. of studies	No. of participants	Statistical method	Effect size
1 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to search methods	2	142	Risk Ratio (M‐H, Random, 95% CI)	2.64 [0.45, 15.60]
2 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to selecting studies	2	142	Risk Ratio (M‐H, Random, 95% CI)	3.08 [0.44, 21.29]
3 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to assessing risk of bias	2	142	Risk Ratio (M‐H, Random, 95% CI)	4.13 [0.60, 28.57]
4 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to addressing missing outcome data	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.1 [0.94, 1.29]
5 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to analysing individual studies appropriately and without avoidable bias	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.31 [0.97, 1.77]
6 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to combining findings of relevant studies using appropriate meta‐analysis methods	2	142	Risk Ratio (M‐H, Random, 95% CI)	2.05 [0.28, 14.89]
7 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to interpretation of results in light of risk of bias	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.34 [1.03, 1.75]
8 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to interpretation of results in light of reporting bias	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.08 [0.82, 1.43]
9 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to interpretation of results in light of multiplicity	1	126	Risk Ratio (M‐H, Random, 95% CI)	1.27 [0.82, 1.97]
10 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to having conclusions supported by the data	2	142	Risk Ratio (M‐H, Random, 95% CI)	1.69 [0.44, 6.55]

6.2. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 2 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to selecting studies.

6.3. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 3 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to assessing risk of bias.

6.4. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 4 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to addressing missing outcome data.

6.5. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 5 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to analysing individual studies appropriately and without avoidable bias.

6.6. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 6 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to combining findings of relevant studies using appropriate meta‐analysis methods.

6.8. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 8 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to interpretation of results in light of reporting bias.

6.9. Analysis.

Comparison 6 Sensitivity analysis for secondary outcome, Outcome 9 Restricting to studies with low risk of bias in the comparability criteria: low methodological quality related to interpretation of results in light of multiplicity.

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Chambers 2015.

Methods	To assess the methodological quality of published network meta‐analyses. All published network meta‐analyses in the Ovid‐MEDLINE database up to 30 July 2014.
Data	318 network‐meta‐analyses of pharmaceuticals.
Comparisons	Network meta‐analyses with financial conflicts of interest (defined as support of the study) and network meta‐analyses without financial conflicts of interest.
Outcomes	Methodological quality (assessed by criteria from the quote: “checklist of good research practices” in the ISPOR guidance for interpreting and conducting indirect‐treatment‐comparison and network‐meta‐analysis studies).
Funding source	No funding was received for the study.
Declaration of conflicts of interest	The authors declared no conflicts of interest related to any for‐profit organisation.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Yes	Two review authors assessed included studies
Adequate coding of financial conflicts of interest and outcomes?	Yes	Two review authors extracted data
Comparability of systematic reviews?	No	Compared systematic reviews of different interventions and different diseases

Dunn 2014.

Methods	To determine whether there is an association between financial conflicts of interest and the favourable presentation of evidence in systematic reviews. Systematic reviews identified through PubMed, PubMed In Process, Embase and the Cochrane Database of Systematic Reviews published between 1 January 2005 and 26 May 2014.
Data	26 systematic reviews on the use of neuraminidase inhibitors in the prophylaxis or treatment of influenza.
Comparisons	Systematic reviews with financial conflicts of interest (defined as employment, funding of grants, and the funding of medical writers) and systematic reviews without financial conflicts of interest.
Outcomes	Conclusions (defined as whether the systematic review was favourable or unfavourable towards the use of neuraminidase inhibitors).
Funding source	This study was funded by the Australian National Health and Medical Research Council and no additional funding related to any for‐profit organisation was declared.
Declaration of conflicts of interest	The authors declared no conflicts of interest related to any for‐profit organisation.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Unclear	Not described
Adequate coding of financial conflicts of interest and outcomes?	Yes	One review author extracted data on financial conflicts of interest and coded each systematic review. A second review author checked all of the identified financial conflicts of interest. Two review authors assessed conclusions
Comparability of systematic reviews?	No	Compared systematic reviews of the same drug class used for the same disease. However, large overlap in individual authors (e.g. single person is author of seven included systematic reviews), which may confound the association.

Ebrahim 2016.

Methods	To identify the impact of industry involvement in the publication and interpretation of meta‐analyses of antidepressant trials in depression. Meta‐analyses published from 1 January 2007 to 5 March 2014.
Data	185 meta‐analyses on antidepressants for depression.
Comparisons	Meta‐analyses with industry funding (defined as industry funding for a meta‐analysis that involves one or more of the drugs that it manufactures) and meta‐analyses without industry funding. Meta‐analyses with author financial conflicts of interest (defined as all the situations where one or more authors are either company employees of the industry or received any support from the industry for any of their work) and meta‐analyses without author financial conflicts of interest.
Outcomes	Conclusions (defined as whether the meta‐analysis included any negative statements expressing caveats about the effectiveness or safety/toxicity of the assessed antidepressant).
Funding source	No funding was received for the study.
Declaration of conflicts of interest	The authors declared no conflicts of interest related to any for‐profit organisation.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Yes	Four review authors assessed included studies
Adequate coding of financial conflicts of interest and outcomes?	Yes	Two review authors extracted data
Comparability of systematic reviews?	No	Compared meta‐analyses of the same class of drugs used for the same disease. However, compared meta‐analyses including both placebo and active control and meta‐analyses based on a systematic literature review and meta‐analyses without a systematic literature review

Gomez‐Garcia 2017.

Methods	To assess the methodological quality of systematic reviews published on psoriasis. Systematic reviews and meta‐analyses published up to 4 July 2016.
Data	121 systematic reviews on drugs or devices used for skin psoriasis.
Comparisons	Systematic reviews with industry funding (defined as pharmaceutical company funding) and systematic reviews without industry funding. Systematic reviews with author conflicts of interest and systematic reviews without author financial conflicts of interest.
Outcomes	Methodological quality (assessed by the 11‐item AMSTAR criteria).
Funding source	The study was partly funded by grants from the ISCIII‐Subdireccion General de Evaluacion and European Regional Development Fund (ERDF) and Plan Propio de movilidad para investigadores Del Instituto Maimonides de Investigacion Biomedica De Cordoba (IMIBIC) and no additional funding related to any for‐profit organisation was declared.
Declaration of conflicts of interest	FG‐G (lead author) has received honoraria from Pfizer, AbbVie, Janssen‐Cilag, and Novartis. JR (second author) and AVG‐N (ninth author) have received honoraria and financial benefits from Pfizer, Janssen‐Cilag, Novartis, and AbbVie.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Yes	Two review authors assessed included studies
Adequate coding of financial conflicts of interest and outcomes?	Yes	Two review authors extracted data and assessed methodological quality
Comparability of systematic reviews?	No	Compared systematic reviews of different interventions

Hartog 2012.

Methods	To examine recommendations and methodological quality of meta‐analyses published on hydroxyethyl starch fluid. Meta‐analyses identified in MEDLINE via Ovid and PubMed, Web of Science, Embase, and the Cochrane Library published up to June 2010.
Data	12 meta‐analyses on widespread clinical use of hydroxyethyl starch fluid.
Comparisons	Meta‐analyses with financial conflicts of interest (defined as financial relationships with or support from a manufacturer of commercially available intravenous fluids) and meta‐analyses without financial conflicts of interest.
Outcomes	Conclusions (defined as whether the meta‐analyses recommend hydroxyethyl starch over other fluids).
Funding source	The study was funded by the Intramural Research Program of the U.S. National Institutes of Health and no additional funding related to any for‐profit organisation was declared.
Declaration of conflicts of interest	KR (last author) has received research grants, speaker's fees, and consultancy fees from B. Braun, Melsungen, Germany.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Yes	Two review authors assessed included studies
Adequate coding of financial conflicts of interest and outcomes?	Yes	Two review authors coded the conclusions of each meta‐analysis
Comparability of systematic reviews?	No	Compared meta‐analyses of hydroxyethyl starch for different indications and different publication periods

Jorgensen 2006.

Methods	To compare the methodological quality and conclusions in Cochrane Reviews with those in industry‐supported meta‐analyses and other meta‐analyses of the same drugs. Pairs of a Cochrane Review and a similar review published in another journal comparing the same drug used for the same disease published within two years of each other.
Data	24 pairs of a Cochrane Review and another review of drug comparisons published between 1998 and 2003.
Comparisons	Systematic reviews with industry funding (defined as support by the pharmaceutical industry as provision of grants, authorship, or other major assistance such as help with the statistical analysis) and Cochrane Reviews without industry funding.
Outcomes	Estimated treatment effect (assessed by pooled comparative Z‐scores). Conclusions (defined as whether the experimental intervention was recommended without reservations, not recommended, or recommended only with reservations). Methodological quality (assessed by the 9‐item Oxman & Guyatt index).
Funding source	No funding was received for the study.
Declaration of conflicts of interest	The authors declared no conflicts of interest related to any for‐profit organisation.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Yes	Two review authors assessed included studies
Adequate coding of financial conflicts of interest and outcomes?	Yes	Two review authors extracted data
Comparability of systematic reviews?	Yes	Compared pairs of Cochrane Reviews and other reviews of the same drug, used for the same disease, and published within two years of each other

Jorgensen 2008.

Methods	To investigate whether meta‐analyses supported by the pharmaceutical industry are of poorer methodological quality and have conclusions favouring the experimental drug, compared to meta‐analyses with non‐profit or no support. Meta‐analyses published in 2004.
Data	39 meta‐analyses comparing different drugs or classes of drugs.
Comparisons	Systematic reviews with industry funding (defined as authorship, provision of grants to authors of the meta‐analysis, or other major assistance such as help with the statistical analysis) and systematic reviews without industry funding.
Outcomes	Conclusions (defined as whether the experimental intervention was recommended without reservations, not recommended, or recommended with reservations). Methodological quality (assessed by the 9‐item Oxman and Guyatt index).
Funding source	No funding was received for the study.
Declaration of conflicts of interest	The authors declared no conflicts of interest related to any for‐profit organisation.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Yes	One review author assessed eligibility of all potentially eligible meta‐analyses. A second review author assessed all included meta‐analyses and 10% of excluded meta‐analyses for eligibility
Adequate coding of financial conflicts of interest and outcomes?	Yes	Two review authors extracted data
Comparability of systematic reviews?	No	Compared meta‐analyses of different interventions and diseases

Lane 2013.

Methods	To compare the quality of pharmaceutical industry‐supported meta‐analyses with academic meta‐analyses. Pairs of industry‐supported meta‐analyses and non‐industry‐supported meta‐analyses matched on the basis of medical subject heading and publication dates. Meta‐analyses published in 2002 to 2004 and 2008 to 2009.
Data	63 pairs of industry‐supported meta‐analyses and non‐industry‐supported meta‐analyses of randomised trials assessing the efficacy or safety of any pharmaceutical intervention in humans.
Comparisons	Meta‐analyses with industry funding (defined as first or corresponding author having primary affiliation to a pharmaceutical company, two or more non‐lead authors having primary affiliation to the same pharmaceutical company, or any author having primary affiliation to the pharmaceutical company who manufactures the product under investigation) and meta‐analyses without industry funding.
Outcomes	Methodological quality (assessed by a 43‐item assessment tool that provides a qualitative assessment of statistical appropriateness and adequacy of interpretation developed and piloted by the authors).
Funding source	No funding was received for the study, but teleconference facilities of a pharmaceutical company were used for meetings regarding the study.
Declaration of conflicts of interest	PWL (first author) is an employee at GlaxoSmithKline. NFB (fifth author) is an employee at Amgen Ltd. JCC (sixth author) and SHa (seventh author) are employees at Pfizer. SHo (eighth author) is an employee at AstraZeneca. PM (tenth author) is an employee at Vifor Pharma.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Yes	Two review authors assessed included studies
Adequate coding of financial conflicts of interest and outcomes?	Yes	Multiple review authors extracted data
Comparability of systematic reviews?	Yes	Compared pairs of systematic reviews with similar publication date and medical subject heading

Wang 2010.

Methods	To explore a possible link between authors’ financial conflicts of interest and their position on the association of rosiglitazone with increased risk of myocardial infarction in patients with diabetes. Systematic reviews published between 2007 and 2009.
Data	11 systematic reviews commenting on rosiglitazone and the risk of myocardial infarction.
Comparisons	Systematic reviews with financial conflicts of interest (defined as pharmaceutical company funding of the article in question, author employment by a pharmaceutical company, pharmaceutical company funding of research other than that covered in the article in question, or the author acting as consultant, advisory board member, speaker, lecturer, or receives travel or honoraria from a pharmaceutical company, or owns stock) and systematic reviews without financial conflicts of interest.
Outcomes	Conclusions (defined as whether the authors present a favourable view of the safety of rosiglitazone (rosiglitazone does not increase the risk of myocardial infarction), a neutral view, or an unfavourable view).
Funding source	No funding was received for the study.
Declaration of conflicts of interest	The authors declared no conflicts of interest related to any for‐profit organisation.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	Unclear	Not described
Adequate coding of financial conflicts of interest and outcomes?	Unclear	Not described
Comparability of systematic reviews?	No	Compared systematic reviews of different outcomes, study designs, and inclusion criteria (based on assessment of abstracts of included systematic reviews)

Yank 2007.

Methods	To determine whether financial ties to one drug company are associated with favourable results or conclusions in meta‐analyses on antihypertensive drugs. Meta‐analyses published up to December 2004.
Data	124 meta‐analyses evaluating the effects of antihypertensive drugs on clinical outcomes in adults.
Comparisons	Meta‐analyses with financial conflicts of interest (defined as financial ties to one drug company) and meta‐analyses without financial conflicts of interest (defined as financial ties to multiple drug companies, no statement, financial ties to both drug and non‐profit companies, and financial ties to non‐profit companies).
Outcomes	Statistically significant results (defined as whether the results were statistically favourable towards the study drug or statistically unfavourable towards the study drug). Conclusions (defined as whether the conclusions were favourable towards the study drug or unfavourable towards the study drug). Methodological quality (assessed by the 9‐item Oxman and Guyatt index).
Funding source	The study was funded partly by the Eugene Garfield Foundation and no additional funding related to any for‐profit organisation was declared.
Declaration of conflicts of interest	The authors declared no conflicts of interest related to any for‐profit organisation.
Notes
Risk of bias
Item	Authors' judgement	Description
Adequate study inclusion process?	No	One review author assessed included studies
Adequate coding of financial conflicts of interest and outcomes?	Yes	One review author extracted data and coded all 124 meta‐analyses. A second review author coded 24 meta‐analyses. The degree of agreement between the two review authors was high
Comparability of systematic reviews?	No	Compared meta‐analyses of the same intervention used for the same disease. However, compared meta‐analyses including both placebo and active control. Performed adjusted analyses, but did not adjust for intervention type

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Ahmer 2006	No relevant outcomes
Kopelman 2013	No relevant outcomes
Lesser 2007	Wrong study domain (not related to drug or device companies)
Radecki 2011	Wrong sample of studies (no systematic reviews included)
Schuit 2016	No relevant outcomes
Sismondo 2008	Wrong sample of studies (no systematic reviews included)
Sismondo 2008a	Wrong sample of studies (no systematic reviews included)
Warner 2003	No relevant outcomes

Differences between protocol and review

We included one new subgroup analysis (referred to as a post hoc subgroup analysis): recoding frequency of favourable conclusions into conclusions in favour of the intervention and conclusions recommending the intervention without reservations. We also included five new sensitivity analyses (referred to as post hoc sensitivity analyses): 1) excluding systematic reviews with unclear or undeclared financial conflicts of interest, 2) excluding one atypical included study, 3) re‐analysing primary outcomes based on observational studies with either low risk of bias in the comparability of systematic reviews or adjusted regression analyses, 4) testing different definitions of financial conflicts of interest (i.e. related to the manufacturer and any for‐profit organisation), and 5) re‐analysing secondary outcome (i.e. methodological quality) based on included studies with low risk of bias in the comparability criteria.

We initially graded the included studies as providing low certainty in our assessment of the certainty of the evidence.

Contributions of authors

AL conceived the idea for the study. The protocol was developed by CH, AH, and AL. CH and AL developed the search strategy. CH and KR included studies, and CH and AL extracted data and assessed the risk of bias. CH performed the data analysis, and all authors participated in data interpretation. CH wrote the draft review, and all the co‐authors contributed in revising the review.

Sources of support

Internal sources

• Centre for Evidence‐Based Medicine Odense (CEBMO), Odense University Hospital and University of Southern Denmark, Denmark.

  CH, AL, and AH were personally salaried by the institution during the period of this review.

• Nordic Cochrane Centre, Rigshospitalet, Denmark.

  CH and KR were personally salaried by the institution during the period of this review.

External sources

• No sources of support supplied

Declarations of interest

The review authors have no relevant interests to declare.

New