Fate of Manuscripts Previously Rejected by the American Journal of Neuroradiology: A Follow-Up Analysis

Abstract

BACKGROUND AND PURPOSE: This is a follow-up article to “Fate of Submitted Manuscripts Rejected from the American Journal of Neuroradiology: Outcomes and Commentary.” The purpose of this study was to quantify differences in citation frequency between manuscripts published in the American Journal of Neuroradiology (AJNR) and those published after AJNR rejection and to understand citation frequency differences among rejected manuscripts.

MATERIALS AND METHODS: In this study, a MEDLINE search identified all manuscripts published in AJNR in 2005 and those initially rejected by AJNR in 2004 but subsequently published elsewhere. Once identified, the citation frequencies of both groups were determined by using SCOPUS. Citation data were used in comparative studies between AJNR rejected and accepted articles and in studies comparing citation frequency differences among rejected manuscripts as a function of journal and/or publication classifications.

RESULTS: Among 315 subsequently published rejections from AJNR in 2004, 696 citations accumulated between 2005 and 2007 (2.4 citations/journal year), whereas 441 AJNR articles published in 2005 accumulated 2490 citations between 2005 and 2007 (6.4 citations/journal year, P < .0001). One-way analysis of variance suggested that rejected manuscripts classified as technical reports and/or published in journals classified as either neuroradiology or general radiology had significantly higher citation frequencies than other submission types and journal classifications. Nonparametric analysis of citation frequency showed significant correlations with impact factors of respective publishing journals (ρ = 0.444).

CONCLUSION: Results from this study suggest that publications initially rejected from AJNR have a significantly lower citation frequency than those accepted by AJNR. Among rejected manuscripts, citations frequencies were highest in technical reports and among journals close to the neuroradiologic discipline.

The American Journal of Neuroradiology (AJNR) is a primary repository for the neuroradiologic disciplines, which may prompt authors of neuroradiology research initially to seek publication in this journal.^1,2 However, the limited yearly publication volume of AJNR in combination with the overspecialization of some submissions forces many authors to seek publication elsewhere. As we have previously reported, more than half of articles initially rejected by AJNR are subsequently published in other journals with publication and circulation volumes similar to those of AJNR.³ As a follow-up to our initial manuscript, we sought to characterize the citation frequency of these initially rejected manuscripts after providing sufficient time for these citations to accumulate. Using these data, we also sought to compare citation frequencies and impact factors between rejected and accepted AJNR publications, to determine how journal and manuscript characteristics are related to citation frequency, and to correlate citation frequency to other journal variables.

Materials and Methods

Of the 554 manuscripts received by the AJNR Editorial Office in 2004 that were initially rejected for publication, 315 (56%) were subsequently published. This cohort of 315 articles was the “initially rejected” cohort for this study.³ Of the 315 initially rejected submissions, 128 manuscripts were submitted as major scientific studies, 106 as case reports, and 81 as technical reports and/or reviews.³ For comparative purposes, 427 submitted manuscripts that were accepted for publication by AJNR in 2005 composed the initially accepted cohort.⁴ Of these 427 manuscripts, 244 manuscripts were submitted as major scientific studies, 140 as case reports, and 48 as technical reports and/or reviews.

The 315 manuscripts composing the initially rejected cohort were published in 116 different journals (impact factor of these journals: mean, 1.83; range, 0.43–9.05) and were cited a total of 696 times as of December 2007 (see “Methods” section below). For these 315 publications, 4 journal variables (circulation volume, publication volume, citations, and impact factor) were analyzed on the basis of journal classification (neuroradiology, general radiology, specialty radiology, neurology, neurosurgery, and other uncategorized biomedical journals) as well as manuscript classification (major study, case report, and technical report). For each submission, the author's name, article title, and date of receipt were obtained from the AJNR office. Institutional affiliations and coauthor names were not included for this study. In all cases, bias was minimized via omission of referee or reviewer comments.

Search Strategy

As previously described, a comprehensive search algorithm was used to best determine the publication outcome (publication journal, date) for the initially rejected and initially accepted cohorts.^3,4 Briefly, the 315 manuscripts comprising the “initially rejected” cohort were initially searched en masse, assuming no changes to author or title, using a custom PERL script search program. All remaining publications where either the author(s) or submission titles, or both were not identified using this program were searched using manually the PubMed data base through initial author search alone or in combination with title keyword searches.

Citation Frequency and Modified Impact Factor Calculation

Annual citation indices, which are the number of times an article is referenced by another manuscript, were determined by using SCOPUS Web-search software for all manuscripts in both cohorts (initially accepted and initially rejected).⁵ On the basis of our previous findings, a 15.6-month publication delay was observed for published manuscripts initially rejected by AJNR in 2004.³ Accordingly, we compared the citation frequencies of these rejected manuscripts with AJNR published manuscripts in 2005 to minimize overestimation of citation frequency in this group. After collection of total citation frequencies, we determined a normalized citation index (mean citations per manuscript). Using annual citation frequency data from SCOPUS, we determined “modified impact factors” for the rejected and accepted publication groups as follows:

Accordingly, modified impact factors for each group were determined for each year (2006, 2007) and as a composite sum of all years. This modified impact factor is a derivative of the original impact factor proposed by Garfield⁶ because the original impact factor uses the total number of citations in any given year for manuscripts published in the previous 2 years. Because this study was limited to the outcomes of publications rejected from AJNR in the year 2004, a modification to the impact factor calculation was deemed necessary.

Journal Study Criteria

Several characteristics of the journals that composed the initially rejected cohort were used for analysis: journal publication volume, journal circulation volume, and journal impact factor. The publication volume, which is the number of manuscripts published in the year of publication of the journal article, 2004–2007, was determined both by MEDLINE and information provided at the homepages of the journals. Individual journal circulation volume, or the number of electronic and hard copy subscriptions, was obtained by using Ulrich's Periodicals Directory.⁷ Journal impact factors, the mean frequency that published articles within a journal are referenced within 2 years, was determined from the Science Citation Index.^6,8–10

Statistical Methods

Continuous data were presented as continuous-range numeric data, whereas nominal or ordinal data were presented as discrete data or percentages. Simple statistical calculations of differences among data were performed by using the Student t test. Pair-wise continuous nominal/ordinal correlations were analyzed by using a simultaneous 1-way analysis of variance (ANOVA) F-test for normal data distributions and the Kruskal-Wallis χ² approximation for non-normal data distributions. Pair-wise nonparametric correlations between continuous datasets were interpreted by using Spearman ρ coefficient. Significance was estimated by P values (eg, the probability that 2 observations are not meaningfully different) reported at or below the fifth percentile. Confidence intervals were presented at the 95th percentile in all cases. Statistical analyses were performed by using JMP Version 7.0 software (SAS Institute, Cary, NC).

Results

Citation Frequency Comparison

The citation frequencies of the initially rejected cohort (rejected from AJNR after submission in 2004) were compared with the initially accepted cohort (published in AJNR in 2005). As of December 2007, the total number of citations of the 315 publications composing the initially rejected cohort was 696. For these previously rejected articles, the mean number of citations per manuscript was 2.43 ± 2.01 in 2006 and 2.42 ± 2.04 in 2007. As of December 2007, the normalized cumulative citation frequency of the 432 manuscripts composing the initially accepted cohort was 2490. For these published AJNR articles, the mean number of citations per manuscript was 6.32 ± 3.27 in 2006 and 6.48 ± 3.39 in 2007. The annual composite of all years, and the total number of citations of both groups are shown as a histogram in Fig 1, where the x-axis range represents the discrete number of citations per manuscript and the y-axis represents the cumulative number of citations in each discrete group.

Fig 1.

Citation frequency histogram of initially rejected and AJNR-published articles. Cumulative citation frequency distributions (eg, the number of times individual manuscripts were cited after publication from 1/1/2005 through 12/31/2007) of each cohort are shown in the 2 histograms. The x-axis represents the number of times any given publication was cited during the 3-year span, whereas the y-axis represents the cumulative citation frequency of each discrete group.

Statistical analysis revealed significant differences in citation frequencies between cohorts, with higher overall total citation frequencies in the initially accepted cohort when sorted by year or unsorted as a composite total (2006 citations: AJNR = 1223, rejected = 331, P > .0001; 2007 citations: AJNR = 1191, rejected = 365, P > .0001; total citations: AJNR = 2490, rejected = 696, P > .0001). Determination of modified impact factors of the 2 groups (AJNR and a hypothetic journal representing the AJNR-rejected manuscripts) revealed that the calculated modified impact factor (IF) of these accepted publications (2006: citations = 1223; IF = 2.83; 2007: citations = 1191 citations, IF = 2.76) was significantly higher than that in the hypothetic journal (2006: citations = 331; IF = 1.05; 2007: citations = 365 citations, IF = 1.15).

Journal Classification Analysis

One-way ANOVA analysis of the manuscripts initially rejected from AJNR revealed several statistically significant associations between journal variables and manuscript classification. Rejected manuscripts published in journals classified as either neuroradiology or general radiology had statistically higher citation frequencies (χ² = 18.9, P = .002), publication volumes (χ² = 75.6, P = .0001), and circulation volumes (χ² = 88.1, P = .0001) than manuscripts published in other journal classifications (eg, neurology, neurosurgery, specialty radiology, and other). One-way analysis failed to reveal a statistically significant difference among journal subtypes when sorted on the basis of impact factor, though journals arising from the general radiology and neuroradiology categories were tending toward, but failing to meet, threshold significance.

Manuscript Classification Analysis

Similar to journal-subtype analysis, 1-way ANOVA of AJNR-rejected manuscripts demonstrated significant relationships between journal variables and manuscript classification. Specifically, technical reports had statistically higher mean citation frequencies (χ² = 12.3, P = .006) and impact factors (χ² = 17.5, P = .0003) than case reports (t test, P ≤ .001) but not major studies (t test, P ≥ .31). Similar statistical differences were observed between different manuscript classifications within the pool of AJNR-published manuscripts (data not shown).

Correlation Analysis

Nonparametric correlation analysis demonstrated significant correlations between citation frequency and other journal variables (circulation volume, publication volume, and impact factor). Results from this analysis suggest that there was no significant correlation between citation frequency and either circulation (ρ = +0.018, P = .73) or publication volume (ρ = +0.044, P = .65), suggesting that citation frequency of individual journals is not related to overall journal readership. In contrast, a significant positive correlation was found between citation frequency and impact factor (ρ = +0.444, P = .0006), suggesting that publications with higher citations were published more often in journals with higher impact factors. It remains unclear if increased journal readership or enhanced manuscript “quality” within these journals is the primary determinant of this correlation.

Discussion

Our original study regarding the fate of articles rejected from AJNR was conceived to determine the outcomes of rejected manuscripts and provide assistance and guidance to authors looking to resubmit their work to another journal.³ We undertook the current follow-up study to expand our original analysis and to better characterize the outcomes of rejected manuscripts with respect to citation frequency. When we compared the citation frequencies of the rejected-manuscript population with articles published in AJNR, we discovered large statistically significant differences. Indeed, AJNR-published manuscripts have citation frequencies ∼3 times those of the rejected manuscripts. These citation-frequency differences manifest as large differences in the calculated modified impact factors between the 2 groups (rejected ∼1.1, accepted ∼2.7). Of note, our modified impact factor calculation, though a derivative of the impact factor equation, may differ slightly from a true impact factor calculation because the calculation differs slightly and we have excluded editorials and other articles within journals that affect the impact factor.

Although the modified impact factor of the group of rejected manuscripts was found to be ∼.1, the average impact factor of journals publishing the rejected manuscripts was 1.78. This suggests that after publication, these articles are less commonly cited even when compared with the average citation distribution of these 116 journals. The reasons for these differences are unclear, but it is possible that the lower citation frequency is somehow related to lower neuroradiologist readership in journals not directly relevant to the field of neuroradiology.

Within the rejected-publication population, we also hoped to better understand how journal variables, in particular citation frequency, were related to journal and manuscript classification to determine if certain publications were more likely to be read and/or cited. Of the rejected manuscripts, those published in journals categorized as general radiology and neuroradiology or those categorized as technical reports had the highest average citation frequencies, whereas publications in neurosurgery, general radiology, neuroradiology, or manuscripts classified as technical reports were published in higher impact journals. Publication within general radiology- and neurosurgery-classified journals had the highest circulation volume and publication volumes, presumably resulting in higher readership.

Among rejected manuscripts, we also hoped to find correlations between citation frequency and other previously studied journal variables. Our data show that neither publication nor circulation volume of the publishing journals was significantly correlated with citation frequency. This finding suggests that higher citation is not related to publication in journals with larger publication or circulation volumes. In contrast, a significant positive correlation was found between the citation frequency of the rejected manuscript population and the impact factor of the journal. These results suggest that impact factor is a significant predictor of future citation.

This study has several limitations. First, MEDLINE may not index rejected submissions subsequently published in some international medical journals, and this would underestimate publication and citation frequency. Second, SCOPUS may not index similar international medical journals, leading to an underestimate of the citation frequencies of these journals. Third, if the corresponding author changed or was no longer affiliated with the submission after rejection, it is possible that the publication would go undetected using our search criteria.

Conclusions

Our findings suggest that though >50% of the rejected manuscripts are eventually published, they accumulate significantly fewer citations than those published in AJNR and appear to have citation indices below the mean impact factor of the publishing journals. Among these rejected manuscripts, manuscripts classified as technical reports and publication in journals specialized in either neuroradiology or general radiology were associated with higher citation frequencies. Given these data, it is possible that the reduced citation frequencies of some of these rejected publications are a result of publication in journals that are less accessible or visited by their target audience, leading to diminished readership and hence citation frequency.