Abstract
Background:
Randomized controlled trials [RCTs] form the corner-stone of evidence-based medicine. RCTs published in high impact factor journals such as the New England Journal of Medicine [NEJM] are a key driver of clinical practice and policy decisions. RCTs are expected to report both efficacy and safety, however, safety reporting in many studies tends to be poor. The present audit was undertaken with the primary objective of evaluating safety reporting during a five-year period in all RCTs published in the NEJM.
Methods:
PubMed alone was searched for RCTs published in NEJM from 2013-17. Each RCT was searched for the following outcome measures –whether the trial was sponsored by pharmaceutical industry or investigator initiated, phase of trial, nature of intervention and therapeutic area in terms of reporting of safety outcomes [with ‘P values’ or ‘95% confidence interval’].
Results:
A total of n=623 articles reported safety outcomes of which 275/623 (44.1%) articles reported statistics for safety outcome. There was significant difference in reporting of safety statistics between investigator initiated studies and pharmaceutical industry sponsored studies, [cOR=4.0, 95% CI 2.8- 5.5 P < 0.001]; phase 3 and phase 4 trials, [cOR 0.67, 95% CI 0.5 - 0.9, P = 0.02]; trials involving drugs and surgery, [cOR 2.07, 95% CI 1.2-3.5, P = 0.01] and in therapeutic areas, cardiovascular and oncology [cOR 0.26, 95% CI 0.1-0.4, P < 0.0001].
Conclusions:
Safety reporting in RCTs continues to take a back seat relative to efficacy reporting and is worse for pharmaceutical industry funded studies. Safety reporting should be emphasized in the CONSORT guidelines.
KEY WORDS: Investigator-initiated trial, P-value, randomized controlled trial, safety outcome
Background
Well-designed randomized controlled trials (RCTs) are the gold standard for evaluating the efficacy and safety of a new treatment and form the corner-stone of evidence-based medicine.[1] RCTs published in high impact factor journals such as the New England Journal of Medicine (NEJM) are a key driver of clinical practice and policy decisions. Reporting of both efficacy and safety in these RCTs is equally important as benefit-risk assessment can be done only when both are present in the manuscript.[2]
Evidence shows that safety reporting often takes a backseat relative to the reporting of efficacy. Most studies in literature have only analyzed the safety reporting in terms of adequcacy of reporting of specific adverse effects and their severeity.[3,4,5] None of the studies have looked at specifics such as therapeutic area, funding support and type of intervention and phases of trials with regards to safety statistics.
The extension of the consolidated standards of reporting trials (CONSORT) statement for harm (2004) gives specific guidelines for the reporting of the harm related results of the clinical trials to address the shortcomings in analysis and reporting of harms. The idea is that the readers can assess benefit-harm ratio in the application of medical interventions.[6] to make informed decisions while treating individual patients.
Against this backdrop, the present study was undertaken with the primary objective of evaluating safety reporting (relative to efficacy reporting) of all RCTs published in the NEJM over a five-year period. We chose the NEJM for analysis as it multidisciplinary, weekly and with a large worldwide readership and an impact factor of 74.69 in 2019.[7]
Materials and Methods
Ethics: The study protocol EC/OA 140/2018 was granted exemption from review on 5th Jan 2018 by the Institutional Ethics Committee (IEC).
Search strategy and eligibility criteria: All articles published in the NEJM between 2013 and 2017 were electronically searched and extracted from the PubMed database using the keywords ‘N Engl J Med’ and Clinical trial. Inclusions were RCTs whereas observational studies, follow-up RCTs, review articles, case series and case reports were excluded.
Two authors MK and AM assessed the articles independently to confirm that they were truly RCTs. In addition, a hand search was carried out for each article by the same authors independently. Any discrepancy was resolved by consensus or by referral to senior authors (NG and UT).
Data extraction: A standardized data collection form was used to extract the following information from the studies- reporting of safety outcomes (with corresponding P values and 95% confidence intervals) both in the abstract and full-text paper, the type of sponsor (whether investigator-initiated or funded by the pharmaceutical industry), the phase of the RCT, nature of the intervention and therapeutic area. Additionally, data was also collected from regulatory sites of the FDA and EMA and clinical trial registries (clinicaltrials.gov and iscrtn.com) if not found in the published paper. A formal email was sent to the corresponding author where more information was needed or if it was missing.
Outcome measures
These were a) Proportion of RCTs reporting statistics for safety outcomes in the full-text article and/or in their abstract b) Nature/source of funding {Investigator-initiated or Pharmaceutical industry sponsored studies} c) The phase of the study (whether I, II, III, IV, I/II or II/III) d) Nature of the intervention e) Its therapeutic area f) Presence [or lack there of] of statistical parameters for safety and g) The association of safety information with i) Nature of funding ii) Phase of the study iii) Nature of Intervention and iv) Therapeutic area of the intervention.
Statistical analysis
Both descriptive and inferential statistics were applied to the data. Categorical data (for example therapeutic area) were expressed as proportions and quantitative data (total RCTs) as median (range). Normality of quantitative data was assessed using the Kolmogorov Smirnov test. Association of safety with respect to nature of funding, therapeutic area, with safety statistics (present or absent] was done at a significance level of P < 0.05 using the Chi-square test and a crude odds ratio (with 95% CI) was calculated. All statistical tests were done using PRISM version 7.
Results
Search findings between 1st January 2013 to 31st December 2017, a total of n = 995 articles were identified in the PubMed search. Of these, n = 239 abstracts were excluded as they did not meet the eligibility criteria allowing for 756 full-text articles for review. Of these 110/756 met the exclusion criteria and thus n = 646/756 (85.4%) RCTs were included for the analysis. [Figure 1]. Of these, in 23/646 RCTs, safety outcomes were not applicable because of the nature of the RCTs, for example-simulation-based trial[8] and thus n = 623/756 (82.4%) RCTs formed the final sample.
Figure 1.
Flow chart showing selection process of the studies
Reporting of safety outcomes
A total of 275/623 (44.1%) articles reported statistics for safety outcome in term of ‘P value’ (269/623, 43.2%) and/or 95% CI (6/623, 01%). Safety statistics were reported in 117/275 (42.5%) articles both in the abstract and the full text, while 126/275 (45.8%) articles reported only in the full text, and 32/275 (11.6%) articles reported only in the supplement.
Type of RCTs
A total of n = 299/623 (48%) RCTs were investigator initiated studies (IIS) and n = 324/623, (52%) were pharmaceutical industry sponsored studies (PISS). Most of the RCTs were in phase 3 (368/623, 57.8%), followed by phase 2 (60/623, 9.6%) and phase 4 (48/623, 7.7%). However, in 121/623 (19.4%) RCTs, the phase of the trial was not mentioned in the article or in the clinical trial registries. RCTs associated with drugs as a mode of intervention (n = 403/623, 64.7%) was highest followed by surgery (n = 63/623, 10.1%), medical devices (n = 23/623,3.7%) and vaccines (n = 22/623,3.5%). The largest numbers of the RCTs was in the therapeutic area of cardiovascular system (n = 172/623, 27.6%) which was followed by oncology (n = 101/623, 16.2%) and infectious disease (n = 94/623, 15.1%) [Table 1].
Table 1.
Demographics of randomized controlled trials
| Year | 2013 | 2014 | 2015 | 2016 | 2017 | Total | |
|---|---|---|---|---|---|---|---|
| Total RCT's | 106 | 116 | 152 | 116 | 133 | 623 | |
| Safety statistics | Present | 43 | 57 | 63 | 59 | 53 | 275 |
| Absent | 63 | 59 | 89 | 57 | 80 | 348 | |
| IIS/PISS | IIS | 57 | 54 | 77 | 66 | 45 | 299 |
| PISS | 49 | 62 | 75 | 50 | 88 | 324 | |
| Phase of trial | Phase 1 | 0 | 2 | 1 | 0 | 5 | 8 |
| Phase 1/2 | 0 | 0 | 3 | 0 | 1 | 4 | |
| Phase 2 | 10 | 11 | 24 | 5 | 10 | 60 | |
| Phase 3 | 55 | 73 | 81 | 70 | 89 | 368 | |
| Phase 4 | 8 | 12 | 6 | 13 | 9 | 48 | |
| Phase 2/3 | 3 | 1 | 4 | 5 | 1 | 14 | |
| Not mentioned | 30 | 17 | 33 | 23 | 18 | 121 | |
| Intervention | Surgery | 15 | 10 | 17 | 15 | 7 | 64 |
| Pharma | 60 | 83 | 94 | 71 | 95 | 403 | |
| Vaccine | 3 | 6 | 6 | 2 | 5 | 22 | |
| Device | 4 | 5 | 5 | 4 | 5 | 23 | |
| Radiotherapy | 1 | 0 | 1 | 0 | 0 | 2 | |
| Others | 20 | 10 | 25 | 20 | 17 | 92 | |
| Combination therapy | 3 | 2 | 4 | 4 | 4 | 17 | |
| Therapeutic Area | Cardiovascular | 42 | 20 | 42 | 29 | 39 | 172 |
| Oncology | 12 | 15 | 32 | 17 | 19 | 95 | |
| Infectious Disease | 11 | 33 | 23 | 14 | 9 | 90 | |
| Respiratory | 8 | 13 | 9 | 11 | 10 | 51 | |
| Metabolic | 4 | 5 | 11 | 6 | 9 | 35 | |
| Neurology | 6 | 6 | 1 | 5 | 17 | 35 | |
| Immunology | 2 | 4 | 5 | 10 | 11 | 32 | |
| Haematology | 4 | 4 | 10 | 5 | 7 | 30 | |
| Nephrology | 5 | 4 | 3 | 1 | 3 | 16 | |
| Musculoskeletal | 3 | 4 | 2 | 3 | 2 | 14 | |
| Gastrointestinal | 4 | 1 | 2 | 4 | 1 | 12 | |
| Others | 5 | 7 | 12 | 11 | 6 | 41 |
Subgroup analysis of different phase of trials, interventions and therapeutic areas in respect to the RCTs being an IIS or PISS is shown in [Table 2].
Table 2.
Subgroup analysis of randomized controlled trials
| Nature of RCTS | Subgroups | Total RCTs | Safety Statistics |
Type of Study |
||
|---|---|---|---|---|---|---|
| Present | Absent | IIS | PISS | |||
| IIS/PISS | IIS | 299 | 183 | 116 | ||
| PISS | 324 | 92 | 232 | |||
| Phase of trial | Phase 1 | 8 | 0 | 8 | 0 | 8 |
| Phase 1/2 | 4 | 0 | 4 | 1 | 3 | |
| Phase 2 | 60 | 20 | 40 | 30 | 30 | |
| Phase 2/3 | 14 | 5 | 9 | 10 | 4 | |
| Phase 3 | 368 | 148 | 220 | 138 | 230 | |
| Phase 4 | 48 | 33 | 15 | 30 | 18 | |
| Not mentioned | 121 | 69 | 52 | 90 | 31 | |
| Intervention | Surgery | 63 | 38 | 25 | 44 | 19 |
| Pharma | 403 | 150 | 253 | 153 | 250 | |
| Vaccine | 22 | 8 | 14 | 7 | 15 | |
| Device | 23 | 14 | 9 | 13 | 10 | |
| Radiotherapy | 2 | 1 | 1 | 1 | 1 | |
| Others | 93 | 53 | 40 | 6 | 87 | |
| Combination therapy | 17 | 11 | 6 | 8 | 9 | |
| Therapeutic Area | Cardiovascular | 172 | 120 | 52 | 83 | 89 |
| Oncology | 101 | 20 | 81 | 34 | 68 | |
| Infectious Disease | 94 | 35 | 59 | 51 | 43 | |
| Respiratory | 52 | 21 | 31 | 27 | 25 | |
| Metabolic | 37 | 13 | 24 | 19 | 18 | |
| Neurology | 34 | 15 | 19 | 18 | 16 | |
| Immunology | 32 | 5 | 27 | 7 | 25 | |
| Haematology | 23 | 7 | 16 | 8 | 15 | |
| Nephrology | 15 | 9 | 6 | 10 | 5 | |
| Musculoskeletal | 14 | 3 | 11 | 9 | 5 | |
| Gastrointestinal | 10 | 4 | 6 | 4 | 6 | |
| Others | 39 | 23 | 16 | 28 | 10 | |
Association of presence or absence of statistics for safety with respect to:
Investigator-initiated [IIS] or pharmaceutical industry sponsored studies [PISS]: There was higher reporting of safety statistics in IIS relative to PISS, (n = 183/299,61.2) versus (n = 92/324, 28.4%) [cOR = 4.0, 95% CI 2.8-5.5], P < 0.001.
Phase of trials: There was lower reporting of safety statistics in phase 3 trials (n = 148/368, 40.2%) [cOR 0.67, 95% CI 0.5-0.9], P = 0.02; whereas, more safety statistics were reported in phase 4 trials (n = 33/48, 68.7%) [cOR 3.03 95% CI 1.6-5.9], P < 0.01, as compared to other phases of the clinical trials.
Mode of intervention: RCTs involving drugs had poorer reporting of safety statistics (n = 150/403,37%) [cOR 0.45, 95% CI 0.3-0.6], P < 0.01 whereas, surgery as a mode of intervention had more reporting of safety statistics (n = 38/63, 60%) [cOR 2.07, 95% CI 1.2-3.5], P = 0.01, relative to other nature of interventions.
Therapeutic areas: Cardiovascular studies, n = 120/172 (70%) had better reporting of safety statistics (n = 120/172,70%) [cOR 4.4, 95% CI 3.0-6.4], <0.0001 whereas oncology trials had poorer reporting of safety statistics (n = 20/101,19.8%) [cOR 0.26, 95% CI 0.1- 0.4], P < 0.0001. relative to other therapeutic areas.
The associations of reporting of safety statistics is shown in Table 3.
Table 3.
Association of reporting of safety statistics
| Outcome Measures | Crude Odd's Ratio | P |
|---|---|---|
| Nature/source of funding | ||
| IIS vs. PISS | 4.0, (95% CI 2.8-5.5) | P<0.01 |
| Phases of trial* | ||
| Phase 2 | 0.60, (95% CI 0.3-1.1) | P=0.07 |
| Phase 2/3 | 0.70, (95% CI 0.2-2.3) | P=0.52 |
| Phase 3 | 0.67, (95% CI 0.5-0.9) | P<0.05 |
| Phase 4 | 3.03, (95% CI 1.6-5.9) | P<0.01 |
| Not mentioned | 1.90, (95% CI 1.2-2.9) | P<0.01 |
| Mode of intervention | ||
| Surgery | 0.45, (95% CI 0.3-0.6) | P<0.01 |
| Pharma | 2.07, (95% CI 1.2-3.5) | P<0.01 |
| Vaccine | 0.71, (95% CI 0.25-1.8) | P=0.45 |
| Device | 0.96, (95% CI 0.4-2.0) | P=0.9 |
| Radiotherapy | 1.26, (95% CI 0.02-99.6) | P=0.86 |
| Others | 1.83, (95% CI 1.1-2.9) | P<0.05 |
| Combination therapy | 2.3, (95% CI 0.8-7.9) | P=0.08 |
| Therapeutic area | ||
| Cardiovascular | 1.5, (95% CI 1.1-2.1) | P<0.05 |
| Oncology | 0.26, (95% CI 1.4-0.4) | P<0.001 |
| Infectious Disease | 0.7, (95% CI 0.4-1.1) | P=0.14 |
| Respiratory | 0.84, (95% CI 0.4-1.6) | P=0.57 |
| Metabolic | 0.67, (95% CI 0.3-1.4) | P=0.25 |
| Neurology | 1.0, (95% CI 0.5-2.1) | P=0.1 |
| Immunology | 0.22, (95% CI 0.1-0.6) | P<0.001 |
| Haematology | 0.54, (95% CI 0.2-1.4) | P=0.17 |
| Nephrology | 1.92, (95% CI 0.6-6.7) | P=0.21 |
| Musculoskeletal | 0.34, (95% CI 0.1-1.3) | P=0.08 |
| Gastrointestinal | 0.84, (95% CI 0.2-3.6) | P=0.8 |
| Others | 1.9, (95% CI 0.9-3.9) | P=0.054 |
*For Phase 1 and phase 1/2 trials, odd's ratio is not calculated, as no trials reported safety statistics.
Discussion
In the audit conducted by us on reporting of safety statistics in 623 RCTs published over a five year period, we found that only 43% of RCTs have reported safety statistics in the form of ‘P value’ or 95% confidence intervals. Moreover, 11% RCTs reported safety statistics only in the supplement indicating that safety reporting is not a priority. We also found higher reporting of safety statistics in Investigator initiated studies [IIS] relative to studies funded by the pharmaceutical industry.
About two-third IIS had better reporting of safety outcomes whereas less than one third PISS reported safety statistics. This indicates the efficacy centric approach of the latter whereas both safety and efficacy is paramount for investigators as they treat patients on day to day basis and have to explain both benefit and risk to the patients to enable them to make an informed choice. Additionally differences in such reporting highlights the lack of a consensus on the use of a statistical tool in reporting of adverse events. Although, the CONSORT statement extension for harm-related data published in 2004 provided guidelines in such reporting, the use of statistics in safety data is not clearly defined. The guideline highlighted common poor reporting practices for harm related data of which one of them is; ‘Reporting only the adverse events that reach a P value threshold in the comparison of the randomized arms (for example, P < 0.05)’.[7]
The answer to better reporting of safety statistics in phase 4 trials over phase 3 trials may lie in the objective of these trials. Phase 4 trials are more concerned about monitoring drug effectiveness in the general population (widespread use) while collecting information about adverse effects which may result in more reporting of safety ‘P value or 95% CI’. Phase 4 studies also have higher sample size compared to the phase 3 studies and hence led to greater reporting of safety signals.
We found that cardiovascular trials were more likely to report safety statistics. This may be likely because these trials largely used composite outcome (which included safety) as a primary endpoint. Only one-fifth of the oncology trials reported safety statistics. This is surprising, considering the high prevalence of adverse events associated with oncology drugs. One reason may be inability of the investigators to distinguish AEs caused by drugs and disease. In addition the patients may not report adverse events by themselves unless they are specifically requested to report them because they think it is a necessary part of the therapy.
Often, safety takes a backseat to efficacy and P value of safety are criticized and tend to be overinterrupted for secondary outcomes. Safety endpoint as a secondary outcome also faced criticism provided multiplicity issues are adjusted, else there could be over interpretation of the P value or the confidence interval adding potential concern for too many outcomes.
There is a responsibility of the authors regardless of whether they are from academia or pharmaceutical industry in being transparent in reporting of safety statistics (transparency in reporting). Responsibility of the journal editor or the peer-reviewers to see efficacy and safety are equally well reported in the manuscript. So, the CONSORT should consider revising their own guidelines and to give specific recommendation for reporting of safety statistics. Our study is limited by being a retrospective audit only from a single journal.
Conclusion
Safety reporting in RCTs continues to take a back seat relative to efficacy reporting and is worse for pharmaceutical industry funded studies. Uniformity in reporting statistics for both efficacy and safety outcomes across all studies will help physicians and patients to make an informed choice about treatments.
Financial support and sponsorship
A grant in aid was provided by Diamond Jubilee Society Trust, Seth GS Medical College and KEM Hospital.
Conflicts of interest
There are no conflicts of interest.
References
- 1.Sim I, Olasov B, Carini S. An ontology of randomized controlled trials for evidence-based practice: Content specification and evaluation using the competency decomposition method. J Biomed Inform. 2004;37:108–19. doi: 10.1016/j.jbi.2004.03.001. [DOI] [PubMed] [Google Scholar]
- 2.Pitrou I, Boutron I, Ahmad N, Ravaud P. Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med. 2009;169:1756–61. doi: 10.1001/archinternmed.2009.306. [DOI] [PubMed] [Google Scholar]
- 3.Garg SK, Henry RR, Banks P, Buse JB, Davies MJ, Fulcher G, et al. Effects of sotagliflozin added to insulin in patients with type 1 diabetes. N Engl J Med. 2017;377:2337–48. doi: 10.1056/NEJMoa1708337. [DOI] [PubMed] [Google Scholar]
- 4.Hussain M, Fizazi K, Saad F, Rathenborg P, Shore N, Ferreira U, et al. Enzalutamide in men with nonmetastatic, castration-resistant prostate cancer. N Engl J Med. 2018;378:2465–74. doi: 10.1056/NEJMoa1800536. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: An evaluation of 7 medical areas. JAMA. 2001;285:437–43. doi: 10.1001/jama.285.4.437. [DOI] [PubMed] [Google Scholar]
- 6.Ioannidis JP, Evans SJ, Gøtzsche PC, O'Neill RT, Altman DG, Schulz K, et al. Better reporting of harms in randomized trials: An extension of the CONSORT statement. Ann Intern Med. 2004;141:781–8. doi: 10.7326/0003-4819-141-10-200411160-00009. [DOI] [PubMed] [Google Scholar]
- 7.nejm.org [homepage on the Internet] United States: New England Journal of Medicine; 2021. Available from: https://www.nejm.org/about nejm /about nejm . [Last accessed on 2021 Apr 09] [Google Scholar]
- 8.Arriaga AF, Bader AM, Wong JM, Lipsitz SR, Berry WR, Ziewacz JE, et al. A simulation-based trial of surgical-crisis checklists. N Engl J Med. 2013;368:246–53. doi: 10.1056/NEJMsa1204720. [DOI] [PubMed] [Google Scholar]