Abstract
Objective
To assess the influence of industry sponsorship and authors’ conflicts of interest on the favorability of results and conclusions of systematic reviews regarding osteoarthritis of the knee.
Design
We searched MEDLINE, Embase, and the Cochrane Database of Systematic Reviews on June 26, 2020 for systematic reviews with or without meta-analyses focusing on treatment of osteoarthritis of the knee. Author COI were collected from the systematic review’s COI disclosure statement, the CMS Open Payments Database, Dollars for Profs, Google Patents, the United States Patent and Trademark Office (USPTO), as well as previously published COI disclosure statements. Study sponsorship was determined using information provided in each systematic review’s funding statement.
Results
Our study included 53 systematic reviews conducted by a total of 279 authors. Thirty-one authors (of 279; 11.1%) had one or more COI. Fourteen systematic reviews (of 53; 26.4%) had one or more conflicted authors. Of these reviews, only two (of 14; 14.3%) reported narrative results favoring the treatment group, whereas five (of 14; 35.7%) reported conclusions favoring the treatment group. Our results showed no statistically significant association between the presence of COI and the favorability of results (P= 0.11) and conclusions (P= 0.73). Because our sample only included one industry-sponsored systematic review, we were unable to adequately assess for a relationship between industry-sponsorship and favorability of results and conclusions.
Conclusion
We found no association between the favorability of systematic review results and conclusions with either the presence of author’s conflicts of interest or industry-sponsorship.
Keywords: Knee osteoarthritis, Systematic reviews, Conflicts of interest, Industry sponsorship
1. Introduction
Osteoarthritis (OA) is the most common chronic joint disease and cause of disability in countries with developed economies [[1], [2], [3]]. Knee OA is responsible for more than 80% of total OA cases [3]. In the United States, nearly one-fifth of the adult population over the age of 45 have chronic pain associated with OA of the knee [1]. This number is likely to increase with an aging population. In fact, the prevalence of knee OA has doubled over the last 70 years [1]. Currently, the American Academy of Orthopedic Surgeons (AAOS) Clinical Practice Guidelines (CPG) recommends conservative treatments — weight loss, exercise, and NSAIDs — as first-line intervention for mild to moderate cases [4]. More severe cases are treated surgically. Although there is limited evidence for osteotomy in younger patients, the end-stage treatment for this condition is total knee arthroplasty (TKA) [4]. This expensive intervention is projected to cost the U.S. greater than $13 billion annually by 2030 [5]. Accordingly, many avenues of active research for the management of knee OA are underway in hopes of improving existing procedures while creating novel ones. Considering the projected prevalence and economic cost, it is paramount that the orthopedic literature be patient-oriented, evidence-based, and unbiased.
High-quality systematic reviews (SRs) are traditionally regarded as the highest level of evidence [6]. In orthopaedics, evidence from SRs has widely been used by CPG panels in order to establish best treatment options for varying clinical conditions [7]. Take for example the AAOS 2013 CPG recommendation for the non-operative treatment of symptomatic knee OA. This recommendation overturned the 2008 level B (i.e. Moderate) recommendation for the use of acetaminophen for symptomatic knee OA [4]. The rationale for this change was attributed, in part, to the results from a SR conducted by Miceli-Richard et al. which found no statistically significant improvement of knee pain in the acetaminophen group when compared to placebo [8]. Based on outcomes of multiple SRs, the 2019 Rheumatology CPG statement recommends against the use of hyaluronic acid injections for the treatment of knee OA [9]. In fact, these CPGs panelists noted that only SRs with a high risk of bias reported beneficial effects of hyaluronic acid injections for the management of knee OA. Furthermore, when limited to primary studies with a low risk of bias, meta-analysis shows little to no difference in effect size between hyaluronic acid injections with placebo [10]. Considering the importance of SRs in clinical practice and health policy, it is essential they are free of potential sources of bias that may influence the direction of review outcomes. Potential sources of bias that pose a threat to the validity and reliability of SR outcomes are author conflicts of interest (COI) and industry sponsorship.
Previous studies have shown that author COI and industry sponsorship may influence the direction of results and conclusions of medical research. For example, a 2019 Cochrane review determined that SRs with conflicted authors were more likely to report positive outcomes of the drug being investigated compared to SRs without conflicted authors [11]. With respect to industry-sponsorship, Devji and Busse concluded industry bias was prevalent among a sample of orthopedic publications [12]. These authors found that industry-sponsorship was associated with an increased likelihood of reporting favorable efficacy results and conclusions, as well as higher rates of discordance between study results and conclusions compared to their non-industry-sponsored counterparts. Results from these studies highlight the potential influence author COI and industry bias may have of the accuracy of outcomes reported in the medical literature. However, the extent to which these issues affect the osteoarthritis literature is unknown.
To address this research gap, the aim of this investigation was to determine: (1) whether industry-sponsored SRs were more likely to contain results and conclusions favoring the intervention; (2) whether SRs with conflicted authors were more likely to contain results and conclusions in support of the intervention; and (3) the proportion of systematic review authors with undisclosed conflicts of interest.
2. Methods
2.1. Transparency, reproducibility, and reporting
We have provided study materials and protocol on Open Science Framework to increase transparency and reproducibility of our results [13]. While drafting this paper, we referred to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [14] and Murad and Wang’s guidelines for meta-epidemiological studies [15].
2.2. Search strategy
We searched MEDLINE (Ovid), Embase (Ovid), and the Cochrane Database of Systematic Reviews on June 26, 2020 to locate SRs and meta-analyses focusing on treatments of OA of the knee using the search strategies provided in the online protocol [13]. The resulting records were subsequently uploaded to a SR screening platform, Rayyan (https://rayyan.qcri.org/).
2.3. Screening
Two of us (B.H. and S·S.) screened search returns by title and abstract in a masked, duplicate manner [16]. Following title and abstract screening, full-texts were evaluated to determine final inclusion. Discrepancies were discussed between investigators until they reached an agreement. J.M.A. and M.W. were available for third party adjudication.
2.4. Eligibility criteria
Each study must have met the PRISMA-P definition of a SR or meta-analysis [17] and must have included a head-to-head comparison of one intervention to another intervention (or combination of interventions), placebo, or standard of care. The review must have investigated a treatment for OA of the knee.Finally, the SR must have been published between September 1, 2016 and December 31, 2019 in the English language and synthesized human data. This date range was selectedThis date range was selected because the International Committee of Medical Journal Editors (ICMJE) recommends financial interests be disclosed over the previous 36 months from the time of journal submission [18]. We chose the pre-specified date range from September 2016 forward to allow 36 months from the time the Open Payments Database appeared online in September 2013.
2.5. Training
All investigators completed online training modules which provided an overview of the study design and objectives, study materials, and data extraction from one example SR. Training was recorded and is available online for reference. [16].
2.6. Data extraction
Two of us (B.H. and S·S.) performed data extraction independently in a masked, duplicate fashion. Investigators extracted the following data items from each SR: (1) PubMed identification number and/or DOI; (2) journal name; (3) date of publication; (4) name of author(s); (5) treatment interventions being compared; (6) affiliation(s) for the first and last author; (7) funding source; (8) complete COI statement; (9) whether the SR or meta-analysis addressed risk of bias (RoB); (10) the verbatim RoB statement; (11) whether author(s) were also an author on one or more of the primary studies included in the review (yes/no); (12) total number of self-cited primary studies; (13) primary outcome; (14) whether an overall pooled effect estimate was calculated (yes/no); (15) statistical significance of pooled effect estimate; and (16) whether narrative results and conclusions favored the treatment or comparison group (e.g., placebo, standard of care, control). We used the term “conclusion” to represent a combination of the discussion and conclusion section of included reviews.
2.7. Favorability of narrative results and conclusions
Narrative results and conclusions were designated as “favorable”, “unfavorable”, or “mixed/inconclusive”. When appraising the results section, “favorable” was assigned to SRs with only positive results. “Unfavorable” was assigned when negative results were exclusively reported. “Mixed/inconclusive” was assigned to narrative results sections that included both positive and negative results. When appraising the conclusion sections, “favorable” was assigned to when authors stated or implied favorability towards the target intervention. “Unfavorable” was assigned when authors stated or implied favorability towards the comparison or control group. When neither “favorable” nor “unfavorable” applied to the conclusion, “mixed/inconclusive” was assigned (i.e., reporting negative population outcome but positive subgroup analysis).
2.8. Identification of undisclosed COI
Searches for undisclosed COI were undertaken using the algorithm provided in Fig. 1. This stepwise search was based on the methodology provided by Mandrioli et al. [19], with modifications. These modifications included the incorporation of 3 additional databases –– the Open Payments database, Dollars for Profs, and the United States Patent and Trademark Office (USPTO). Each database used in this search is described in Table 1. To ensure consistency between investigators, B.H. and S.S. created standardized search strings for PubMed, USPTO Database, and Google Patents using the Python programming language (Python Software Foundation, https://www.python.org/). If we were unable to verify a patent belonged to the author, we considered the search inconclusive and continued our process. In accordance with ICMJE standards of COI disclosure, PubMed searches were limited to 36 months prior to the publication of the original SR to determine if previously published studies included additional COI not disclosed in the SR from our sample. If this search yielded more than 10 publications, each investigator individually assigned random numbers to the resulting publications. The COI statement of the first 10 studies numerically were then examined. Each investigator individually generated random numbers to include wider search of publications and opportunities for authors to disclose a COI. This process was performed until an undisclosed COI was discovered, at which time the author was then counted as having an undisclosed COI. This stop-procedure is identical to that used by Mandrioli et al.[19].
Fig. 1.
Stepwise search for undisclosed COI.
Table 1.
Description of databases used to search for undisclosed COI among systematic review authors.
| Database | Description of Database |
|---|---|
| Open Payments Database (https://openpaymentsdata.cms.gov) | Open Payments Database – which was created on September 1, 2013 – is a Congressionally-mandated, openly accessible resource designed to increase the transparency within the United States healthcare system. This database collects and publishes information regarding industry relationships between healthcare providers (eg, physicians and teaching hospitals) and industry (eg, drug and device manufacturers). Physicians and teaching hospitals report industry payments received in the form of research, food and beverage, travel, and consulting or speaking fees. |
| Pro Publicas Dollars for Profs (https://projects.propublica.org/dollars-for-profs) | Dollars for Profs provides information from state universities and the National Institutes of Health (NIH) regarding industry payments and conflicts of interest of academic professors, researchers, and other support personnel. Rationale for including this database was based on the knowledge that searching for non-healthcare professionals listed as an author of a systematic review included in our sample would not return beneficial information. |
| United States Patent and Trademark Office (USPTO) (https://www.uspto.gov) |
The United States Patent and Trademark Office (USPTO) is responsible for the registration of US patents and trademarks in accordance with the commerce clause (Article I, Section 8, Clause 3) of the US Constitution. In addition, the USPTO “furthers effective intellectual property protection for U.S. innovators and entrepreneurs worldwide by working with other agencies to secure strong IP provisions in free trade and other international agreements.” (https://www.uspto.gov/about-us) |
| Google Patents (https://patents.google.com) | Google Patents is a database consisting of greater than 120 million patent publications from more than 100 different patent offices worldwide. In addition, Google Patents provides access to technical documents and books indexed in Google Scholar and Google Books, as well as documents included in the Prior Art Archive. |
| PubMed (https://pubmed.ncbi.nlm.nih.gov) | PubMed was launched in January 1996 and is one of the most widely used databases for academic research. The entire MEDLINE collection includes more than 30 million citations from biomedical literature. As part of the Enterz system of information retrieval, PubMed is maintained by The United States National Library of Medicine at the National Institutes of Health. (https://pubmed.ncbi.nlm.nih.gov/) |
2.9. Risk of bias evaluations
To evaluate the risk of funding bias, we applied the Cochrane Collaboration’s criteria for assessment, and the following 4 items from Mandrioli et al. [19]: (1) whether explicit and “well defined” criteria that could be replicated by others were used to select studies for inclusion/exclusion; (2) whether an adequate study inclusion method, with two or more assessors selecting studies, was used; (3) whether search strategies were comprehensive; and (4) whether methodological differences that may introduce bias were controlled for. Each item was designated as yes, no, or unclear. We considered the overall RoB to be low if at least 3 of the aforementioned criteria were sufficiently met. Otherwise, the RoB was considered to be high. Authors B.H. and S.S. performed an independent and masked evaluation of risk of bias items. Discrepancies were discussed between investigators until a consensus was reached. J.M.A. and M.W. were available for third party adjudication.
2.10. Statistical analysis
Results were quantified using descriptive statistics, and relationships were evaluated by Fisher’s exact tests, when possible. Stata 16.1 (StataCorp, LLC, College Station, TX) was used for all analyses.
3. Results
Our initial database search returned 1420 potential studies. After title and abstract screening, 146 records were included for full text analysis. Upon further review, an additional 93 records were excluded, the majority of which were non-head to head comparisons resulting in a total of 53 SRs meeting final inclusionary criteria (Fig. 2).
Fig. 2.
Flow diagram of study selection.
3.1. Systematic review characteristics
A total of 279 individuals authored the 53 SRs, which were published in 33 medical journals, most commonly in International Journal of Surgery (8/53; 15.1%) and The Journal of Knee Surgery (7/53; 13.2%). The majority of authors reported affiliations from China (151/279; 54.1%), followed by the United States (29/279; 10.4%) (Table 1). The majority of reviews (46/53; 86.8%) reported that the authors had no COI to disclose; four SRs (7.5%) did not provide a COI statement, and three SRs (5.7%) disclosed at least one COI among the authors. The most common target intervention under investigation was drug(s) (19/53, 35.8%), followed by medical device(s) (i.e. Robotic computer assisted surgery) (15/53; 28.3%) and surgical intervention(s)/technique(s) (15/53, 28.3%) (Table 2).
Table 2.
Systematic review characteristics (n=53).
| Characteristic | Form Response | N (%) |
|---|---|---|
| Journal in which systematic reviews were published (n= 53) | International Journal of Surgery | 8 (15.1) |
| The Journal of Knee Surgery | 7 (13.2) | |
| The Journal of Arthroplasty | 5 (9.4) | |
| Knee Surgery, Sports Traumatology, Arthroscopy | 4 (7.5) | |
| Journal of Orthopaedic Surgery and Research | 2 (3.8) | |
| Othera | 27 (50.9) | |
| Conflict of Interest Statement (n=53) | All authors report no COI | 46 (86.8) |
| No COI statement present | 4 (7.5) | |
| One or more authors report a COI | 3 (5.7) | |
| Intervention Type (n= 53) | Drug | 19 (35.8) |
| Device | 15 (28.3) | |
| Surgical Technique/Intervention | 15 (28.3) | |
| Other | 4 (7.5) | |
| Affiliation of First Author (n= 53) | Public Academic Institution | 38 (71.7) |
| Government | 7 (13.2) | |
| Private Academic Institution | 4 (7.5) | |
| Private/Industry | 3 (5.7) | |
| Multiple | 1 (1.9) | |
| Affiliation of Last Author (n= 53) | Public Academic Institution | 35 (66.0) |
| Private Academic Institution | 9 (16.9) | |
| Government | 6 (11.3) | |
| Multiple | 3 (5.7) | |
| Source of Funding (n= 53) | No funding received | 19 (35.8) |
| No statement listed | 17 (32.1) | |
| Government | 7 (13.2) | |
| Multiple | 7 (13.2) | |
| Public | 1 (1.9) | |
| Hospital | 1 (1.9) | |
| Private/Industry | 1 (1.9) | |
| Self-citation of primary studies (n=53) | No, did not include self-cited primary studies | 49 (92.5) |
| Yes, included one or more self-cited primary studies | 4 (7.5) |
Journals represented once in our sample: Archives of Orthopaedic and Trauma Surgery, Arthritis research & therapy, Arthroplasty today, BMC Musculoskeletal Disorders, BioDrugs, Cartilage, Clinical rheumatology, Der Orthopade, European Journal of Orthopaedic Surgery & Traumatology, Evidence-based complementary and alternative medicine: eCAM, Internal Medicine Journal, Journal of lasers in medical sciences, Journal of rehabilitation medicine, Knee, Osteoarthritis and cartilage, PLoS ONE, Pain Medicine, Rheumatology international, Systematic Reviews, International Journal of Clinical and Experimental Medicine, The Journal of bone and joint surgery, The Journal of international medical research, The Korean Journal of Internal Medicine, The surgeon: journal of the Royal Colleges of Surgeons of Edinburgh and Ireland.
3.2. Author characteristics and completeness of COI disclosures
Thirty-one authors (of 279; 11.1%) had one or more COI. Of these authors, 7 (21.9%) accurately disclosed all COI. The remaining 24 authors (of 31; 77.4%) had at least one undisclosed COI. Five authors (of 279; 1.8%) disclosed one or more COI within the review but had additional COI which were omitted from the disclosure statement (Table 3).
Table 3.
Characteristics of systematic review authors (n=279).
| Accuracy of author COI disclosure statement (n= 279)a | Reported no COI | 248 (88.9) |
| No COI disclosed in systematic review, found to have one or more undisclosed COI | 19 (6.8) | |
| All COI disclosed in systematic review | 7 (2.5) | |
| Disclosed one or more COI in systematic review, found to have additional undisclosed COI | 5 (1.8) | |
| Disclosed one or more COI in systematic review, found no COI | 0 (0) | |
| Country of affiliation for authors conducting the systematic review (n=279)a | China | 151 (54.1) |
| United States | 29 (10.4) | |
| Canada | 21 (7.5) | |
| United Kingdom | 17 (6.1) | |
| Korea | 17 (6.1) | |
| Germany | 13 (4.7) | |
| Japan | 12 (4.3) | |
| Mexico | 6 (2.2) | |
| Iran | 4 (1.4) | |
| New Zealand | 3 (1.1) | |
| Denmark | 2 (0.7) | |
| India | 2 (0.7) | |
| France | 1 (0.4) | |
| Israel | 1 (0.4) |
Number of individuals listed as an author on all systematic reviews within our sample (if authors were found to have authored more than one systematic review, we counted this author multiple times as each publication presented a unique opportunity to completely, or incompletely, disclose all COI).
3.3. Relationship between COI and favorability of results and conclusions
Two of 14 reviews (14.3%) with one or more conflicted authors reported results favoring the treatment group, whereas five SRs (35.7%) reported conclusions favoring the treatment group. Four of these SRs with favorable conclusions had mixed favorability of results (i.e. there were positive subgroup analyses). Of the 39 SRs without conflicted authors, three (7.7%) reported results favoring the treatment group, whereas 12 (30.8%) reported conclusions favoring the treatment group. Six of these 12 (50%) SRs with favorable conclusions had mixed favorability of results. Three (25%) SRs reported unfavorable results (i.e. negative or mixed results were reported within 1 or more of the populations). Our results showed no statistically significant association between the presence of COI and the favorability of results and conclusions (Table 4).
Table 4.
Frequency of favorability of results and conclusions by conflicts of interests disclosure (n= 53).
| Review Outcomes | COI Among Systematic Reviews |
||
|---|---|---|---|
| No COI (n= 39) | COI (n= 14) | Fisher’s exact | |
| Favorability of Results |
P= 0.11 |
||
| Results Favor Treatment Group | 3 (7.7) | 2 (14.3) | – |
| Results are Mixed/Inconclusive | 23 (59.0) | 4 (28.6) | – |
| Results Favor Placebo/Control Group |
13 (33.3) |
8 (57.1) |
– |
|
Favorability of Discussion/Conclusions |
P= 0.73 |
||
| Discussion Favors Treatment Group | 12 (30.8) | 5 (35.7) | – |
| Discussion is Mixed/Inconclusive | 14 (35.9) | 3 (21.4) | – |
| Discussion Favors Placebo or Control Group |
13 (33.3) |
6 (42.86) |
– |
|
Risk of Bias |
P=.74 |
||
| High risk of bias | 1 (2.6) | 0 (0.0) | – |
| Low risk of bias | 38 (97.4) | 14 (100.0) | – |
3.4. Relationship between industry sponsorship and favorability of results and conclusions
Seventeen SRs (of 53; 32.1%) reported funding support. One SR (of 17; 5.9%) reported industry sponsorship, whereas the remaining 16 (of 17; 94.2%) reported non-industry support. The single industry-sponsored SR reported results and conclusions favoring the treatment group. Of the 16 reviews with non-industry sponsorship, one (6.3%) reported results favoring the treatment group, and four (25%) reported conclusions favoring the treatment group (Table 5). The insufficient number of industry-sponsored SRs inhibited analysis of the association between industry-sponsorship and the favorability of results and conclusions.
Table 5.
Frequency of favorability of results and conclusions by funding sponsor.
| Review Outcome | Funding Sponsor |
||||||
|---|---|---|---|---|---|---|---|
| No funding received (n= 19) | No statement listed (n= 17) | Government (n= 7) | Multiple (n= 7) | Hospital (n= 1) | Public (n= 1) | Industry (n= 1) | |
| Favorability of Results | |||||||
| Results Favor Treatment Group | 2 (11.1) | 1 (5.9) | 0 (0.0) | 0 (0.0) | 1 (100.0) | 0 (0.0) | 1 (100.0) |
| Results are Mixed/Inconclusive | 11 (61.1) | 6 (35.3) | 6 (85.7) | 4 (57.1) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| Results Favor Placebo or Control Group | 6 (33.3) | 10 (58.9) | 1 (14.3) | 3 (42.9) | 0 (0.0) | 1 (100.0) | 0 (0.0) |
| Favorability of Discussion/Conclusions | |||||||
| Discussion Favors Treatment Group | 7 (38.9) | 5 (29.4) | 2 (28.6) | 2 (28.6) | 0 (0.0) | 0 (0.0) | 1 (100.0) |
| Discussion is Mixed/Inconclusive | 4 (22.2) | 6 (35.3) | 1 (14.3) | 4 (57.1) | 1 (100.0) | 1 (100.0) | 0 (0.0) |
| Discussion Favors Placebo or Control Group | 8 (44.4) | 6 (35.3) | 4 (57.1) | 1 (14.3) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| Risk of Bias | |||||||
| High risk of bias | 1 (5.6) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 0 (0.0) |
| Low risk of bias | 18 (94.7) | 17 (100.0) | 7 (100) | 7 (100) | 1 (100.0) | 1 (100.0) | 1 (100.0) |
3.5. Relationship between risk of bias and industry sponsorship or conflicts of interest
We found only one SR (of 53; 1.9%) to have a high RoB. With only one review in this category, we were unable to assess whether an association exists between review sponsorship and COI and RoB. Supplemental Table 1 provides a detailed summary of our RoB assessment.
4. Discussion
Our findings suggest that the favorability of results and conclusions of SRs and meta-analyses focusing on treatments of OA of the knee are not influenced by author COI. This finding differs from a substantial body of literature linking author COI with more favorable study outcomes [[20], [21], [22], [23], [24]]. Furthermore, over 10% of SR authors had at least one COI, most of which were omitted from the systematic review’s disclosure statement. This finding has significant implications regarding the enforcement of journal COI disclosure policies, as well as the extent to which authors can be trusted to accurately self-disclose. Here, we provide further discussion of our results within the context of the existing literature and offer recommendations for future research into the influence of COI and industry-sponsorship on outcomes of SRs.
Our results differ from previous research which has shown a potential link between author COI and the likelihood of reaching favorable study outcomes. For example, Printz and colleagues sought to determine whether an association exists between author COI and the favorability of qualitative conclusions of randomized controlled trials (RCTs) on hyaluronic acid injections for knee arthritis [25]. These authors observed an association between reported COI and the favorability of qualitative conclusions (P=0.01). Moreover, none of the RCTs with at least one conflicted author had unfavorable conclusions. In comparison, more than one-third of RCTs with no industry-affiliated authors indicated that hyaluronic acid knee injections were no more effective when compared to placebo injection. Other studies in surgical subspecialties have found a similar association between author COI and the favorability of study results. A cross-sectional analysis of 337 articles in surgical specialties found that when a surgeon author had at least one COI –– regardless of the amount or whether this COI was disclosed or undisclosed –– study outcomes were significantly more likely to reach favorable, pro-industry conclusion compared to studies without a conflicted author (75.2 vs 60.3%, Relative Risk 1.2, 95% Confidence Interval 1.1–1.4, p<0.00) [24]. Our results suggest that this relationship (between COI and favorability of study outcomes) does not seem to translate over to SRs focusing on treatment of OA of the knee.
Despite finding no association between COI and favorability of SR results and conclusion, we did find a sizable number of authors with undisclosed COI. Incomplete COI is not unique to our sample and has been shown to be problematic in the medical literature as a whole. Take for example a 2020 study published in the Journal of Hand Surgery which found that nearly 60% of authors had at least one undisclosed COI [26]. Moreover, these authors found that senior authors, authors who published in high impact orthopedic surgery journals, and authors receiving upwards of $500,000 in industry payments were less likely to accurately disclose all payments. A 2018 JAMA Surgery publication found similar inconsistencies between self-declared COI and reported payments received in the CMS Open Payments Database [27]. These findings, combined with the findings of our study, poses an area of improvement within the medical literature, as incomplete COI self-disclosure may threaten the integrity of research standards within the orthopedic literature.
Owing to an insufficient number of industry-sponsored SRs, we were unable to assess whether industry funding was associated with results and conclusions that favored the industry sponsor’s product. Though we were unable to assess potential industry bias at the SR level, a significant body of literature suggests outcomes of RCTs –– the primary study design from which data are synthesized in SRs –– are influenced by industry sponsorship. For example, Bhandari et al. found a statistically significant association between industry-sponsorship and pro-industry results from surgical trials [28]. Djulbegovic et al. found pharmaceutical trials funded exclusively, or in part, by for-profit companies were more likely to report favorable outcomes than the comparators [29]. Khan et al. found that 85% of industry-sponsored orthopedic clinical trials reported outcomes favoring the new treatment – a significantly higher rate than clinical trials receiving other non-industry support [30]. Based on this body of evidence and our limited sample, future investigation is warranted to explore industry’s direct and indirect influence on knowledge synthesis.
4.1. Implications in the field
Future policies on disclosures and competing interests must be to improve the quality of research evidence. The ICMJE produced a new disclosure form, which has been criticized for failing to address the need for quality improvement in research as well as having a limited scope of disclosure [31]. PLOS requires all involved in the peer review process to declare all potentially competing interests that occurred within 5 years of conducting the research under consideration or preparing the article for publication, as well as any interests outside the aforementioned 5 years if it could be considered a competing interest by the definition PLOS recognizes for competing interest [32]. The Cochrane Collaboration recently updated COI policies for all SRs. These policy updates include limiting the number of conflicted authors to 33% of the authorship team, disallowing the first and last authors from having a COI, and requesting authors to declare non-financial interests [33]. However, all such policies rely on accurate self-disclosure which continues to be a problem, as literature on disclosure accuracy continues to report inaccuracies in approximately 58% of studies [26], and our results — although based upon a small sample — show the same. JAMA has regularly published erratum because of inconsistent disclosures [34]. One such case of inaccurate disclosures is of the former Physician-in-Chief at the Memorial Sloan Kettering Cancer Center [35]. In 2018, he had failed to disclose “millions” in some of his research articles which resulted in his resignation from the Memorial Sloan Kettering Cancer Center. These examples seem to highlight the failed system of self disclosure. The American Journal of Sports Medicine instituted a policy requiring that disclosure forms for all physician authors based in the US are verified against Open Payments data, which is a good first step toward improvement. However, for those authors holding other terminal degrees such as PhDs as well as other research team members without a medical license, no record exists of their COIs. The Open Payments Program recently extended providers to include mid level practitioners, and this will increase transparency for multidisciplinary research teams.
4.2. Strength and limitations
Regarding strengths, all authors received extensive training prior to study commencement. Data was extracted in a masked, duplicate fashion, which the Cochrane Collaboration considers the gold standard [36]. The authors reconciled after both screening and data extraction to reduce errors. This has been shown to reduce error rates comparable to experienced experts [37].
Several methodological limitations were revealed upon completion of our study, each of which is discussed here. First, it is important to recognize that systematic reviews are susceptible to bias through biased primary studies. Our methods assessed whether a RoB assessment was performed by the authors of SRs, recorded the verbatim RoB statement, and evaluated if SR authors included studies they had previously authored. Bias introduced by primary studies with conflicted authors was not assessed by us. It is significant to note that COI among primary study authors may introduce bias in the SR as well — a form of bias compounding. To address issues related to financial bias in SRs, the Cochrane Collaboration established the TACIT work group as a means to evaluate the likely risk of bias associated with conflicted study authors. Second, there are inherent limitations in the databases used to search for undisclosed COI. For example, the Open Payments Database is restricted to the U.S.-affiliated authors, therefore, international authors were not searched using this platform, possibly contributing to a lower rate of COI non-disclosure. Dollars for Profs is a voluntary database and does not represent all public institutions across the U.S. or non-U.S.-institutions. Third, our study is cross-sectional and restricted to the English language, therefore our results may not be generalizable beyond these restrictions. Lastly, only one SR in our sample received industry funding support, thereby hindering us from determining the degree to which industry may influence the nature of results and conclusions of SRs focusing on treatment interventions of OA of the knee.
5. Conclusion
Based on the limitations of our COI search algorithm, our findings do not suggest that an association exists between author COI and the favorability of SR outcomes. Future studies should investigate the extent of industry influence on SR outcomes by evaluating a larger sample of industry-funded SRs.
Author statement
Ben Heigle: Data Collection, Screening, Formal analysis, Manuscript Composition and Approval; Samuel Shepard: Data Collection, Screening, Formal analysis, Manuscript Composition and Approval; J. Michael Anderson: Conception, Methodology, Data Collection, Screening, Formal analysis, Manuscript Composition and Approval; Michael Weaver: Data Collection, Screening, Formal analysis, Manuscript Composition and Approval; Micah Hartwell: Data Collection, Screening, Formal analysis, Manuscript Composition and Approval; Matt Vassar: Conception, Methodology, Data Collection, Screening, Formal analysis, Manuscript Composition and Approval
Declaration of competing interest
Vassar reports grant funding from the National Institutes of Health, the U.S. Office of Research Integrity, and Oklahoma Center for the Advancement of Science and Technology, all outside the present work. All other authors have nothing to disclose as confirmed on the CMS Open Payments Database, Dollars for Profs, Google Patents, the United States Patent and Trademark Office (USPTO), or on previously published COI disclosure statements.
Acknowledgements
None.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.ocarto.2021.100142.
Role of funding source
This study was funded by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship Grant. This funding source had no influence on the design of this study and did not have any role during its execution, interpretation of data, or decision to submit finalized results.
Appendix A. Supplementary data
The following is the Supplementary data to this article:
References
- 1.Wallace I.J., Worthington S., Felson D.T., et al. Knee osteoarthritis has doubled in prevalence since the mid-20th century. Proc. Natl. Acad. Sci. U. S. A. 2017;114(35):9332–9336. doi: 10.1073/pnas.1703856114. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Murray C.J.L., Atkinson C., Bhalla K., et al. The state of US health, 1990-2010: burden of diseases, injuries, and risk factors. J. Am. Med. Assoc. 2013;310(6):591–608. doi: 10.1001/jama.2013.13805. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Vos T., Flaxman A.D., Naghavi M., et al. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2012;380(9859):2163–2196. doi: 10.1016/S0140-6736(12)61729-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Jevsevar D.S. Treatment of osteoarthritis of the knee: evidence-based guideline, 2nd edition. J. Am. Acad. Orthop. Surg. 2013;21(9):571–576. doi: 10.5435/JAAOS-21-09-571. [DOI] [PubMed] [Google Scholar]
- 5.Delanois R.E., Mistry J.B., Gwam C.U., Mohamed N.S., Choksi U.S., Mont M.A. Current epidemiology of revision total knee arthroplasty in the United States. J. Arthroplasty. 2017;32(9):2663–2668. doi: 10.1016/j.arth.2017.03.066. [DOI] [PubMed] [Google Scholar]
- 6.Paul M., Leibovici L. Systematic review or meta-analysis? Their place in the evidence hierarchy. Clin. Microbiol. Infect. 2014;20(2):97–100. doi: 10.1111/1469-0691.12489. [DOI] [PubMed] [Google Scholar]
- 7.of Orthopaedic Surgeons AA, Others. AAOS Clinical Practice Guideline Methodology V2. 0. Published online 2017.
- 8.Miceli-Richard C., Le Bars M., Schmidely N., Dougados M. Paracetamol in osteoarthritis of the knee. Ann. Rheum. Dis. 2004;63(8):923–930. doi: 10.1136/ard.2003.017236. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Kolasinski S.L., Neogi T., Hochberg M.C., et al. American College of rheumatology/arthritis foundation guideline for the management of osteoarthritis of the Hand, hip, and knee. Arthritis Rheumatol. 2020. 2019;72(2):220–233. doi: 10.1002/art.41142. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Rutjes A.W.S., Jüni P., da Costa B.R., Trelle S., Nüesch E., Reichenbach S. Viscosupplementation for osteoarthritis of the knee: a systematic review and meta-analysis. Ann. Intern. Med. 2012;157(3):180–191. doi: 10.7326/0003-4819-157-3-201208070-00473. [DOI] [PubMed] [Google Scholar]
- 11.Hansen C., Lundh A., Rasmussen K., Hróbjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst. Rev. 2019;8 doi: 10.1002/14651858.MR000047.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Devji T., Busse J.W. Cochrane in CORR®: industry sponsorship and research outcome. Clin. Orthop. Relat. Res. 2017;475(9):2159–2164. doi: 10.1007/s11999-017-5421-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Evaluating conflicts of interest in systematic reviews concerning interventions for osteoarthritis of the knee. 2020. https://osf.io/dtvfx/ Published online July 1.
- 14.PRISMA. Accessed July 2, 2020. http://prisma-statement.org/.
- 15.Murad M.H., Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid. Base Med. 2017;22(4):139–142. doi: 10.1136/ebmed-2017-110713. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Heigle B., Shepard S., Anderson M., Weaver M., Hartwell M., Vassar M. Evaluating conflicts of interest in systematic reviews concerning interventions for osteoarthritis of the knee. 2020. https://osf.io/r7zp6/ Published online July 1.
- 17.Moher D., Shamseer L., Clarke M., et al. PRISMA-P Group Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 2015;4(1):1–9. doi: 10.1186/2046-4053-4-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.of Medical Journal Editors Ic, Others . 2015-09-06. ICMJE Form for Disclosure of Potential Conflicts of Interest.http://www,icmje,org/downloads/coi_disclosure,pdf Published online 2012. [Google Scholar]
- 19.Mandrioli D., Kearns C.E., Bero L.A. Relationship between research outcomes and risk of bias, study sponsorship, and author financial conflicts of interest in reviews of the effects of artificially sweetened beverages on weight outcomes: a systematic review of reviews. PloS One. 2016;11(9) doi: 10.1371/journal.pone.0162198. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Cherla D.V., Olavarria O.A., Bernardi K., et al. Investigation of financial conflict of interest among published ventral hernia research. J. Am. Coll. Surg. 2018;226(3):230–234. doi: 10.1016/j.jamcollsurg.2017.12.009. [DOI] [PubMed] [Google Scholar]
- 21.Cherla D.V., Viso C.P., Holihan J.L., et al. The effect of financial conflict of interest, disclosure status, and relevance on medical research from the United States. J. Gen. Intern. Med. 2019;34(3):429–434. doi: 10.1007/s11606-018-4784-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.DeGeorge B.R., Jr., Holland M.C., Drake D.B. The impact of conflict of interest in abdominal wall reconstruction with acellular dermal matrix. Ann. Plast. Surg. 2015;74(2):242–247. doi: 10.1097/SAP.0000000000000372. [DOI] [PubMed] [Google Scholar]
- 23.Dunn A.G., Arachi D., Hudgins J., Tsafnat G., Coiera E., Bourgeois F.T. Financial conflicts of interest and conclusions about neuraminidase inhibitors for influenza: an analysis of systematic reviews. Ann. Intern. Med. 2014;161(7):513–518. doi: 10.7326/M14-0933. [DOI] [PubMed] [Google Scholar]
- 24.Cherla D.V., Viso C.P., Olavarria O.A., et al. The impact of financial conflict of interest on surgical research: an observational study of published manuscripts. World J. Surg. 2018;42(9):2757–2762. doi: 10.1007/s00268-018-4532-y. [DOI] [PubMed] [Google Scholar]
- 25.Printz J.O., Lee J.J., Knesek M., Urquhart A.G. Conflict of interest in the assessment of hyaluronic acid injections for osteoarthritis of the knee: an updated systematic review. J. Arthroplasty. 2013;28(8 Suppl):30–33. doi: 10.1016/j.arth.2013.05.034. e1. [DOI] [PubMed] [Google Scholar]
- 26.Ross P.R., Wood S.M., Chung K.C. Industry funding and self-declared conflict of interest in Hand surgery publications. J Hand Surg Am. 2020;45(6):479–487. doi: 10.1016/j.jhsa.2020.02.017. [DOI] [PubMed] [Google Scholar]
- 27.Lopez J., Samaha G., Purvis T.E., et al. The accuracy of conflict-of-interest disclosures reported by plastic surgeons and industry. Plast. Reconstr. Surg. 2018;141(6):1592–1599. doi: 10.1097/PRS.0000000000004380. [DOI] [PubMed] [Google Scholar]
- 28.Bhandari M., Busse J.W., Jackowski D., et al. Association between industry funding and statistically significant pro-industry findings in medical and surgical randomized trials. CMAJ (Can. Med. Assoc. J.) 2004;170(4):477–480. [PMC free article] [PubMed] [Google Scholar]
- 29.Djulbegovic B., Lacevic M., Cantor A., et al. The uncertainty principle and industry-sponsored research. Lancet. 2000;356(9230):635–638. doi: 10.1016/S0140-6736(00)02605-2. [DOI] [PubMed] [Google Scholar]
- 30.Khan S.N., Mermer M.J., Myers E., Sandhu H.S. The roles of funding source, clinical trial outcome, and quality of reporting in orthopedic surgery literature. Am. J. Orthoped. 2008;37(12):E205–E212. discussion E212. [PubMed] [Google Scholar]
- 31.Shaw D. Withholding conflicts of interest: the many flaws of the new ICMJE disclosure form. J. Med. Ethics. 2020 doi: 10.1136/medethics-2020-106136. Published online July 1. [DOI] [PubMed] [Google Scholar]
- 32.PLOS Medicine: A Peer-Reviewed Open-Access Journal. Accessed July 21, 2020. https://journals.plos.org/plosmedicine/s/competing-interests.
- 33.A more rigorous conflict of interest policy is coming for Cochrane. Accessed July 25, 2020. https://www.cochrane.org/news/more-rigorous-conflict-interest-policy-coming-cochrane.
- 34.Bauchner H., Fontanarosa P.B., Flanagin A. Conflicts of interests, authors, and journals: new challenges for a persistent problem. J. Am. Med. Assoc. 2018;320(22):2315–2318. doi: 10.1001/jama.2018.17593. [DOI] [PubMed] [Google Scholar]
- 35.Downfall of Jose Baselga Respected Cancer Researcher Due to Careless Disclosure Mistakes Verified Through Open Payments. Policy & Medicine. Accessed July 25, 2020. https://www.policymed.com/2018/09/downfall-of-respected-cancer-researcher-due-to-careless-mistakes.html.
- 36.Cumpston M., Li T., Page M.J., et al. Updated guidance for trusted systematic reviews: a new edition of the Cochrane Handbook for Systematic Reviews of Interventions. Cochrane Database Syst. Rev. 2019;10:ED000142. doi: 10.1002/14651858.ED000142. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Horton J., Vandermeer B., Hartling L., Tjosvold L., Klassen T.P., Buscemi N. Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy. J. Clin. Epidemiol. 2010;63(3):289–298. doi: 10.1016/j.jclinepi.2009.04.007. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.