Abstract
This study assesses whether research manuscripts with supplemental material submitted in 2016 to JAMA, JAMA Internal Medicine, and JAMA Pediatrics were more likely than those without supplements to undergo peer review, be accepted, and have more online views and downloads.
Journal article supplementary online content is an option for publication of data and materials that may not fit in the space allotted to a scientific article. The use of online supplements is increasing, and standards have been published to guide their format, length, and appropriateness. However, the value of supplements has been questioned. We assessed whether research manuscripts with supplements have a higher likelihood of peer review, acceptance, and online use after publication than research manuscripts without supplements.
Methods
This cohort study assessed the numbers and types of supplements with research manuscripts submitted between January 1 and December 31, 2016, to the 3 journals in The JAMA Network with the highest volume of research submissions: JAMA (general medical journal) and 2 specialty journals, JAMA Internal Medicine (JIM) and JAMA Pediatrics (JPED). Research manuscripts included randomized clinical trials (RCTs), meta-analyses, and other observational studies. Research Letters, which do not permit supplements, were excluded.
We compared the rates of obtaining peer review and acceptance for manuscripts with and without supplements and analyzed the peer reviewer comments about the supplements. In addition, we evaluated all research articles and the numbers and types of supplements published in these journals between January 1 and December 31, 2016. We compared online use (views and downloads) of articles vs the supplements between January 1, 2016, and March 31, 2017 (allowing time for viewing articles published later in 2016). Given that supplements are required for RCTs (for trial protocols) and most meta-analyses have supplements, we conducted a second analysis excluding these study types.
We calculated rate differences and 95% CIs and the χ2 test for 2-sided P values using Stata version 14.1 (StataCorp). P<.05 defined statistical significance.
Results
Overall, 7985 research manuscripts were submitted to the 3 journals (61.0% [n = 4868] with supplements and 39.0% [n = 3117] without supplements; Table 1). Manuscripts with supplements were more likely to be peer reviewed than those without supplements in JAMA (34.1% vs 16.5%, respectively; between-group difference of 17.6%, 95% CI, 15.1%-20.1%), JIM (17.0% vs 9.0%; between-group difference of 8.0%, 95% CI, 4.8%-11.1%), and JPED (32.9% vs 22.9%; between-group difference of 10.0%, 95% CI, 5.8%-14.3%). Of 1421 manuscripts with supplements that were sent for peer review, 484 (34.1%) were commented on by 538 reviewers and 155 (32.0%) of these comments were substantive (eg, addressed specific content in the supplement or suggested moving supplemental content to the main manuscript).
Table 1. Association of Supplement Inclusion With Likelihood of Research Manuscript Being Peer Reviewed or Accepted.
| Total No. | No. (%) | Between-Group Difference, % (95% CI) | ||
|---|---|---|---|---|
| With Supplement | Without Supplement | |||
| All 3 Journals | 7985 | 4868 (61.0) | 3117 (39.0) | |
| JAMA | ||||
| Research submissions in 2016 | 4416 | 2532 (57.3) | 1884 (42.7) | |
| Accepteda | 104 | 99 (3.9) | 5 (0.3) | 3.6 (2.9-4.4) |
| Peer reviewed | 1174 | 863 (34.1) | 311 (16.5) | 17.6 (15.1-20.1) |
| Supplement commented on by peer reviewers | 301 (34.9) | |||
| JAMA Internal Medicine | ||||
| Research submissions in 2016 | 1858 | 1325 (71.3) | 533 (28.7) | |
| Accepteda | 119 | 112 (8.4) | 7 (1.3) | 7.1 (5.4-8.9) |
| Peer reviewed | 273 | 225 (17.0) | 48 (9.0) | 8.0 (4.8-11.1) |
| Supplement commented on by peer reviewers | 82 (36.4) | |||
| JAMA Pediatrics | ||||
| Research submissions in 2016 | 1711 | 1011 (59.1) | 700 (40.9) | |
| Accepteda | 90 | 88 (8.7) | 2 (0.3) | 8.4 (6.6-10.2) |
| Peer reviewed | 493 | 333 (32.9) | 160 (22.9) | 10.0 (5.8-14.3) |
| Supplement commented on by peer reviewers | 101 (30.3) | |||
Recorded in February 2017.
Manuscripts with supplements were more likely to be accepted than those without supplements in JAMA (3.9% vs 0.3%, respectively; between-group difference of 3.6%, 95% CI, 2.9%-4.4%), JIM (8.4% vs 1.3%; between-group difference of 7.1%, 95% CI, 5.4%-8.9%), and JPED (8.7% vs 0.3%; between-group difference of 8.4%, 95% CI, 6.6%-10.2%). Similar results were seen in the analysis of the 6894 observational studies, which excluded 569 RCTs and 522 meta-analyses (Table 2).
Table 2. Association of Supplement Inclusion With Likelihood of Observational Study (Excluding Clinical Trials and Meta-analyses) Being Peer Reviewed or Accepted.
| Total No. | No. (%) | Between-Group Difference, % (95% CI) | ||
|---|---|---|---|---|
| With Supplement | Without Supplement | |||
| All 3 Journals | 6894 | 3859 (56.0) | 3035 (44.0) | |
| JAMA | ||||
| Observational study submissions in 2016 | 3790 | 1943 (51.3) | 1847 (48.7) | |
| Accepteda | 65 | 60 (3.1) | 5 (0.3) | 2.8 (2.0-3.6) |
| Peer reviewed | 925 | 625 (32.2) | 300 (16.2) | 15.9 (13.2-18.6) |
| JAMA Internal Medicine | ||||
| Observational study submissions in 2016 | 1549 | 1035 (66.8) | 514 (33.2) | |
| Accepteda | 90 | 83 (8.1) | 7 (1.4) | 6.7 (4.7-8.6) |
| Peer reviewed | 206 | 159 (15.4) | 47 (9.1) | 6.2 (2.9-9.5) |
| JAMA Pediatrics | ||||
| Observational study submissions in 2016 | 1555 | 881 (56.7) | 674 (43.3) | |
| Accepteda | 74 | 72 (8.2) | 2 (0.3) | 7.9 (6.0-9.7) |
| Peer reviewed | 421 | 270 (30.6) | 151 (22.4) | 8.2 (3.9-12.6) |
Recorded in February 2017.
Of 374 published research articles (162 in JAMA, 117 in JIM, and 95 in JPED), 372 had supplements. The medians (interquartile ranges) for online views and downloads of articles were higher compared with those of supplements in JAMA, 10 918 (1324-35 877) vs 37 (1-96), respectively; JIM, 3573 (568-17 654) vs 12 (1-53); and JPED, 1386 (407-5947) vs 5 (1-13). The majority of online views and downloads of articles (76%) occurred within 2 months of publication; for supplements, the majority (77%) took 7 months to occur.
Discussion
Research manuscripts with online supplements were more likely to be peer reviewed and accepted for publication vs manuscripts without supplements. These findings were shown for all research manuscripts and in the analysis that excluded RCTs and meta-analyses. However, peer reviewers only commented on about one-third of supplements. Online views and downloads of supplements were low compared with views and downloads of the associated articles. Although possibly less valuable to peer reviewers and readers, online supplements may be a marker for complexity and quality of research manuscripts submitted and published, facilitate complete and transparent reporting for studies, and may provide information for replication and future research.
This study was limited by its retrospective nature, the number of journals, and lack of detailed assessment of the methodological characteristics of the included studies and content analysis of the supplements. Future research could include assessment of the content and perceived usefulness of online supplements by authors and readers.
Section Editor: Jody W. Zylke, MD, Deputy Editor.
References
- 1.Schriger DL, Chehrazi AC, Merchant RM, Altman DG. Use of the internet by print medical journals in 2003 to 2009: a longitudinal observational study. Ann Emerg Med. 2011;57(2):153-160.e3. [DOI] [PubMed] [Google Scholar]
- 2.Marcus E. Taming supplemental material. Cell. 2009;139(1):11. [DOI] [PubMed] [Google Scholar]
- 3.National Information Standards Organization; National Federation of Advanced Information Services Recommended practices for online supplemental journal article materials, January 2013. http://www.niso.org/apps/group_public/download.php/10055/RP-15-2013_Supplemental_Materials.pdf. Accessed September 19, 2017.
- 4.Maunsell J. Announcement regarding supplemental material. J Neurosci. 2010;30(32):10599-10600. [Google Scholar]
- 5.Borowski C. Enough is enough. J Exp Med. 2011;208(7):1337. [Google Scholar]
- 6.Pop M, Salzberg SL. Use and mis-use of supplementary material in science publications. BMC Bioinformatics. 2015;16:237. [DOI] [PMC free article] [PubMed] [Google Scholar]
