Perceptions and Patterns in Academic Publishing: A Survey of United States Residents in Radiation Oncology

Abstract

Purpose

We aimed to assess perceptions of, and training regarding, the publishing process among US radiation oncology (RO) residents, focusing on awareness and understanding of criteria for selecting appropriate and legitimate peer-reviewed journals for academic publishing. The growing challenge of predatory publication in the broader scientific realm and its relevancy to resident training is also briefly discussed.

Methods and Materials

A survey was opened to residents of all Accreditation Council for Graduate Medical Education–accredited RO programs in the United States, focusing on 3 categories: (1) demographics; (2) submission, peer review, and publication of academic research; and (3) subjective ranking of factors for choosing an appropriate publisher/journal. Results were stratified by level of training and number of publications.

Results

Overall, 150 of 690 residents (19.8%) responded, with a 98% (147 of 150) completion rate. Twenty of 150 residents (13.3%) reported formal training in manuscript preparation and choosing academic journals. Only 3.4% of residents reported departmental guidelines regarding publication in “predatory” journals; 57.7% were unsure. The 3 most important factors influencing publisher and journal choice were impact factor (ranked first for 59.0%), whether a journal is found in a major index (ranked first for 18.0%), and association with a reputable organization (ranked first for 17.0%). Importance of impact factor increased with number of publications (50% with 0 publications, 48.3% with 1-5, 63.9% with 5-10, 76.2% with 10-15, and 70.6% with >15). Cost considerations influenced journal choice at least once for 79 (52.7%) residents.

Conclusions

Impact factor was the most important consideration for residents when choosing an appropriate publisher, with increased emphasis with increasing number of publications. A minority had formal training in choosing appropriate academic journals and knowing how to identify so-called predatory journals or were aware if their department has proscriptions regarding publication in such journals. Additional emphasis on formal training for RO residents in manuscript preparation and choosing academic journals is warranted.

Introduction

The advent of online and open access (OA) publishing has caused a fundamental shift in the dynamics of academic publishing. This has created new opportunities and outlets for data, tempered with concerns over appropriate and fair peer review and publication.¹ In particular, concern is building over so-called predatory journals—those that use unethical solicitation and publication processes for financial gain in the absence of appropriate peer review. These costs may be incurred in the classic subscription-based model or alternatively in OA format via article processing costs.2, 3, 4 A number of factors and behaviors have been repeatedly identified as indicative of such activity, including unusually high or low associated costs, extremely short turnaround, unfamiliar impact factors, nontransparent processes, and e-mail solicitation.5, 6, 7

Published reports have characterized such activity within specific medical disciplines. One publication in emergency medicine categorized publications as legitimate or not using a set of criteria to provide a list of predatory journals and publishers—not without controversy.6, 8 Another study scrutinized the prevalence of specific behaviors in the context of OA publications in surgery, particularly e-mail solicitation as it relates to impact factor and article processing costs.⁹

In radiation oncology (RO), similar studies have not been reported. However, publication patterns have been quantitatively analyzed in detail on the level of resident contributions, and ethical issues associated with proper peer review and publication are of increasing interest.10, 11 Resident physicians in RO, although a relatively small group, are often heavily academically active and are often at the frontlines of data collection and manuscript preparation. Their research is published not only in journals specific to RO but also in myriad medical oncology, surgical oncology, radiology, general medical, and broader scientific journals. Many planned publications are suitable for a range of journals that may include not only traditional, subscription-based journals but also OA ones. We administered a nationwide survey to sample the experiences and perceptions among RO residents regarding academic research and publishing, thereby gaining insight into prevailing concerns and considerations in this context. Through this survey, we hope to elucidate strengths and weaknesses in programmatic education regarding appropriate journal selection and to identify areas for improvement in RO residency training.

Methods and Materials

On an institutional review board–approved protocol, a survey was disseminated by webmail invitation link to RO residents (postgraduate year 2 [PGY2]-PGY5), residency program directors, and residency program coordinators nationwide. E-mail addresses were obtained from the Association of Residents in Radiation Oncology Directory.¹² The survey was housed on a secure third-party survey website (surveymonkey.com). All participation was voluntary, responses were kept anonymous, and all participants consented to analysis of their responses before proceeding with the survey. A reminder e-mail was sent 1 month after the original e-mail to encourage participation/completion of the survey. A random drawing to award 10 participants with $25 Amazon gift cards, which were funded by the authors, was used to further incentivize participation.

Survey design

The survey (see Supplementary Material, available online at https://doi.org/10.1016/j.adro.2019.09.001)included 19 questions and focused on 3 categories: (1) demographic questions (level of training, location, number of publications); (2) specific questions pertaining to appropriate submission, peer review, and publication of academic research; and (3) subjective ranking of a number of factors important for choosing an appropriate publisher/journal. In this last category, residents were asked to rank the following factors in order of importance to them when considering a journal for submission of their research: impact factor, indexing in PubMed or other major scientific indices, association with a reputable professional or scientific organization, amount or quality of comments and suggestions provided by reviewers and editors, costs associated with submission and publication, duration of time indexed, name recognition of members on the editorial board, time to review/response/publication, and quality/appropriateness of journal transactions (eg, website, e-mail communication, phone calls). Participants were asked to provide e-mail addresses for forwarding of gift cards, and any repeated e-mail addresses were removed from analysis to ensure no duplication. To maintain anonymity, e-mail addresses were unlinked from responses before analysis.

Statistical analysis

Frequencies of responses calculated as percentages were used to represent the overall experience of residents regarding selection of journals and publication practices within their own training environment. Stratification was performed based on PGY to assess for increased exposure later in residency training.

Results

Overall, 150 of an estimated 690 residents (19.8%) nationwide responded, with 98% (147/150) completing the survey in full. Representation among all levels of training was equally distributed (PGY2, 21.3%; PGY3, 27.3%; PGY4, 28.7%; PGY5, 22.7%). The distribution of numbers of peer-reviewed journal publications for the respondents was 0 (8.7%), 1 to 5 (40.7%), 5 to 10 (25.3%), 10 to 15 (14.0%), and >15 (11.3%). Twenty of 150 (13.3%) residents reported formal training in manuscript preparation and selection of academic journals for submission. Only 3.4% of residents reported departmental guidelines regarding publication in “predatory” journals; 57.7% were unsure. Regarding questions specific to solicitation, 18.7% (28 of 150) reported submission to a journal in response to direct email solicitation at least once; 13.3% (20 of 150) reported submitting to a journal that send them copies of publications without ever having enrolled for subscription. Regarding questions specific to cost, 54% (79 of 160) reported choosing a journal over another journal because of cost at least once; 32.7% (49 of 150) reported publishing in a journal with higher publication fees after rejection by a first-choice journal; 57% (86 of 150) felt that high publication fees (>$500) indicated low or suspect quality journals, although 44% (63 of 150) did not feel this determined higher or lower quality in of itself (Table 1).

Table 1.

Basic demographic information of survey respondents

Characteristic	Respondents (N = 150), n (%)
Current PGY level
PGY2	32 (21.3)
PGY3	41 (27.3)
PGY4	43 (28.7)
PGY5	34 (22.7)
No. residents in program∗
1-6	31 (20.7)
7-10	56 (37.3)
11-16	45 (30.0)
>16	18 (12.0)
No. manuscripts in peer-reviewed journals
0	13 (8.7)
1-5	61 (40.7)
5-10	38 (25.3)
11-15	18 (12.0)
>15	17 (11.3)

Abbreviation: PGY = postgraduate year.

^∗Representation from 27 states.

The 3 most important factors influencing publisher/journal choice were impact factor (ranked first for 59%), whether a given journal is found in a major index (ranked first for 18%), and association with a reputable professional or scientific organization (ranked first for 17%) (Fig 1). Impact factor was most important overall, and its importance increased with number of publications (50% of residents with 0 publications, 48.3% with 1-5, 63.9% with 5-10, 76.2% with 10-15, and 70.6% with >15). Cost considerations influenced journal choice at least once for 79 (52.7%) and more than once for 54 (36.0%) residents. Cost consideration was ranked the fifth most important factor when selecting a journal.

Figure 1.

Percentage of respondents ranking a given factor as single-most important in choosing an appropriate journal or publisher.

Discussion

This study aimed to evaluate the exposure of residents in RO training to education/exposure to appropriate selection of journals for publication and to assess awareness regarding avoidance of so-called predatory journals. In this survey, the majority of polled residents considered impact factor most important when choosing an appropriate publisher, increasing in importance with increasing numbers of publications per resident. This makes intuitive sense because high-achieving or more senior and experienced residents may also view impact factor as the best indicator of a valued, oft-cited publication. Inclusion in a major scientific index and association with a major, reputable organization were ranked closely as second and third—and understandably so. Inclusion in a major scientific index, such as PubMed, likely reflects to trainees (as it does do their faculty mentors) external validation of a given journal’s legitimacy. Association with a well-known entity (a major society, publisher, or federal agency) connotes trustworthiness and implies transparent intention, purpose, and operations. These characteristics may also indicate a fair peer review critique and process—the core of publishing legitimacy.

The commonality among the 3 items indicated as most important in this survey—impact factor, association with a major organization, and inclusion in major indexing efforts—is that they are simple, public, and easily accessible. Interestingly, none represent specific behaviors or operating practices by publishers; instead they indicate a general community consensus on the merit of external validation. We believe this may indicate a baseline understanding of legitimately operating publishers among RO residents. However, other results of the survey suggest that detailed understanding may be lacking and may, in fact, have consequences. Although cost consideration was among the lower ranked of polled factors, it influenced choice of journal in more than half of respondents. Furthermore, only a small minority had formal training in choosing appropriate academic journals and knowing how to identify so-called predatory journals or were aware of whether their departments had proscriptions regarding publication in such journals. These overall may be attributed to a lack of awareness of the specific entity of predatory journals, despite the fact that most residents are likely aware of the existence of publishers with illegitimate practices and procedures. In particular, the financial and driving component of predatory publishing may be one that especially more junior residents may not appreciate, given their fledgling careers and generally not extensive prior publishing experience. Although many of the factors queried in this survey are extensively discussed in RO resident education programs and are well understood by faculty mentors, the specific topic of predatory publishing remains a gray zone. It highlights the challenge of distinguishing “legitimate” journals from those that may be less well validated and represents a crucial component of publishing ethics that is becoming increasingly salient and worthy of careful discussion.¹¹

Awareness and training in identifying predatory publishing, then, is critical for multiple reasons. First, the amount of predatory publishing in the broader scientific domain appears to be rapidly increasing, with total articles numbering 53,000 in 2010 and 420,000 in 2014.13, 14 These numbers are likely even higher as of 2019. This is widely believed to translate into a widespread “waste” of data, much of which is legitimately collected but does not undergo appropriate assessment and critique for publication and validation for use by the scientific community.¹⁵ The problem is endemic. The National Institutes of Health (NIH) has even “noted an increase in the numbers of papers reported as products of NIH funding which are published in journals or by publishers that do not follow best practices promoted by professional scholarly publishing organizations,” as given in their official public statement.¹⁶ This further underscores misuse of valuable resources. Second, the popular media and patients themselves have unprecedented access to primary medical literature, thus increasing the downstream effects of growing predatory publishing. This may (and likely has in many instances) lead to patients accessing literature that was not properly peer reviewed, may support erroneous understandings of disease processes, and may lead to delays in receipt of the most beneficial treatments. These could be just some of the consequences of this growing problem.¹⁷

To identify predatory publishing, it is important to become acquainted with certain well-established characteristics that have repeatedly resurfaced in the relevant literature. Lists of these characteristics and of specific journals determined to be “predatory” have been published online. This includes the famed Beall’s list, which for many years was a reference for scientists across all disciplines.6, 18, 19 It was eventually removed, partially as a result of its controversial nature as a “blacklist” (and because of threats of litigation). Many related articles have followed, most without specific blacklists but with descriptions of specific activities/characteristics that raise suspicion for predatory activity. These include, among others, higher than usual costs, e-mail solicitation, very rapid processing that raises suspicions of cursory or no peer review, no substantive feedback or critique, grammatically challenged or nontransparent communication, and baseless solicitations to become major contributors or editors. Further complicating the issue of blacklists is the fact that many legitimate journals, for instance, may be too young to have an impact factor or have higher publication costs for other reasons, although the core of their process is dedicated to fair and proper peer review and dissemination of data and literature.⁸ In particular, fees associated with publishing in well-recognized and legitimate peer-reviewed radiation oncology journals vary widely, from a few hundred dollars plus image costs to several thousand dollars to cover immediate OA—approaching the costs for some “predatory” publishers that instead do not provide appropriate review. This survey did not query presence or absence of impact factor, merely its overall value to the respondent in relation to other characteristics, but it did address the issue of cost with multiple question items. Residents were somewhat more inclined to regard higher cost as suspicious (54%), although an almost equal proportion were unsure to what extent it indicates quality (44%). This response pattern may reflect different viewpoints—that in some cases, high cost may be in the context of poor peer review and thus constitute predatory activity, whereas in other cases it may be a function of other logistic factors but in the presence of fair peer review and legitimate operations. It is then important not to categorically associate some of these factors with predatory activity. A more official “whitelist” of activities indicative of fair review process has been published by the Directory of Open Access Journals, which provides guidelines on proper and fair publishing of scientific data in an OA format that would result in a Directory of Open Access Journals “seal of approval.”²⁰ Much of the continued growth of predatory publishing has been via the OA format, and such guidelines are useful.¹⁷

Although such guidelines, with their descriptions and lists, are helpful, official guidelines by the scientific or medical communities at large have received much attention but have not yet been formulated. In the meantime, academic departments are recognizing the need to create their own specific publishing policies, to educate faculty and trainees on these policies, and to provide oversight and enforcement as data are collected and manuscripts are prepared. This ensures that data and ideas are given an optimal chance to contribute to the larger corpus of knowledge for the benefit of patients. RO research and publication, like many medical and scientific disciplines, has traditionally been heavily evidence based, with most residents being involved in research activities. It is our belief, then, that in the context of the modern publishing environment, specific training in avoiding predatory publishing is warranted and part of a comprehensive resident education. This would lay a foundation for these future leaders in RO to maintain high standards in the face of the growing threat of predatory publishing, thereby maintaining the integrity of the field in quality contribution and strong evidence-based practice.

Our survey had limitations. First, this is a sampling of residents, without complete inclusion of all residents across the country, and therefore does not incorporate all resident values and experiences. Residents who are more involved with publishing may have been more likely to proceed with the survey, possibly selecting for a specific set of views and biases. The relatively low response rate may be attributed to e-mail fatigue (ie, was not noticed or was ignored), lack of distribution to residents by program directors/coordinators (because these were the individuals to whom the survey was distributed), and resident-specific logistic factors (impending American Society for Radiation Oncology deadline, studying for the in-service examination). However, a broad representation of all regions of the country and PGY levels was achieved, yielding a general representation of the resident contingent. Second, creation of multiple-choice items in a survey may reflect unintentional bias. In this regard, a lack of free-text response questions may have limited the study.

Conclusions

When selecting a journal for publication of their work, residents valued impact factor the most, with increasing importance for more senior and prolific residents. This and other highly ranked factors indicate an understanding of well-validated publishing parameters among RO residents. Although ranked low in importance, cost considerations influenced journal selection in more than half of respondents. Very few residents undergo formal training in choosing appropriate academic journals or in knowing how to identify so-called predatory journals. Few were aware of their own departments’ rules and proscriptions regarding publication in such journals. Given the increasing use of predatory publishing and concern for appropriate peer review to protect integrity in the publication of unique research, formal training in manuscript preparation and appropriate journal selection for RO residents is warranted and could set a precedent for widespread dissemination and use across other specialties in medicine.