Engaging in the Publication Process Improves Perceptions of Scientific Communication, Critique, and Career Skills Among Graduate Students

Abstract

Reading and critiquing primary scientific literature is an important skill for graduate students, as reviewing literature is critical to advancing science. Prior research indicates that graduate students lack understanding of effective communication as well as basic experimental design, but also that graduate students are capable of growth in their experimental design abilities when given proper opportunities. The Journal of Emerging Investigators (JEI) provides graduate students with the opportunity to review and edit original research papers submitted by middle and high school student-authors. The purpose of this project was to determine whether participation in the primary literature process through JEI effectively aids in developing graduate students’ perceived abilities in the domains of communication, scientific critique, and career preparation. A 12-question survey was distributed using SurveyMonkey to 215 JEI reviewers and editors. Editors, whose role involves the synthesis of feedback from multiple reviewers and interaction with papers in their earliest stages, perceived that they benefited more than did reviewers in every domain assessed by the survey. Perceived impact on critiquing skills was only rated more highly by reviewers than by editors once the graduate students in question had reviewed 10 or more papers. The results of this research suggest that graduate students should participate early and often in the reading and reviewing of primary literature; furthermore, the study of flawed science writing can help to improve experimental design, critique, and science communication skills.

INTRODUCTION

The integrity of the scientific process depends on the ability of scientists to capably communicate the results of well-designed experiments to other researchers and the general public. Producing and reviewing academic literature are important responsibilities of a professional scientist (1, 2). These activities require a deep understanding of the scientific process, the nuances of effective experimental design, and appropriate science communication skills; notably, these skills are expected to be acquired before the end of a scientist’s graduate education.

While skills involved in science publication are essential to success in the sciences, they are not always given priority in graduate programs. Writing and communication are seen as essential parts of a career in the sciences (1), but many graduate programs choose to focus on research rather than communication skills (3, 4). Currently, graduate programs in the sciences tend to focus on teaching students to share results with other scientists through conferences and publications rather than with the general public, which diminishes the immediate impact of scientific discoveries (3, 5). This presents a problem for students who have chosen their career based upon their desire to contribute to society (3) as well as a missed opportunity for success, as scientists across disciplines who disseminate their work more widely were found to be more active academically as well (6).

The lack of a pedagogical approach to scientific communication results in many graduate students struggling to write at an appropriate level to effectively convey their work. For example, when graduate students in higher education were administered the SAT writing test, their scores did not significantly differ from those of a normal high schooler’s (7). When administered a basic writing assessment, 25% to 30% of graduate students in social work failed and had fundamental issues related to creating paragraphs and building an argument (8). Graduate students in clinical psychology likewise believe that they are receiving insufficient training in skills relevant to publication (9). Finally, faculty in STEM fields indicate that they themselves learned how to write without formal training, through trial and error, but also express frustration with their graduate students’ lack of writing ability (10). Clearly, the expectation that graduate students write as experts is not being met (7). While it was easy to find literature regarding writing difficulties in social science disciplines, fewer studies appear to have been done on the current state of communication skills among graduate students in the natural or bench sciences, further demonstrating the need for more research in this area (11).

Graduate students may not only have difficulty communicating their science, but also designing studies. Reviews of biological research have found flaws due to misunderstandings of basic statistics (12). Statistical models may be incorrectly applied, control groups forgotten, and random assignment neglected. These errors lead to wasted time and erroneous conclusions as researchers incorrectly interpret chance trends. Research design curriculum in PhD programs in psychology has stagnated since 1990 and leaves little time for advanced study (13).

Prior research on solutions

Evidence shows that graduate student research efforts in biology are rated as significantly improved after graduate students have critiqued papers lacking any significant flaws and papers with fatal experimental flaws (12). A 1999 study introduced experimental design and analysis to biology graduate students through lecture–discussion sessions and then required the students to regularly critique all aspects of one another’s experiments (12). Commenting on peers’ work in an undergraduate physics course can correlate with higher-quality papers by the reviewing students (14, 15). Students are quite capable of reviewing peers’ work; graduate students reviewing for a student-centered peer-reviewed journal returned similar decisions on papers as did their faculty peers (16), and undergraduate and graduate students across 16 courses and 4 universities produced very reliable grades when evaluating peers’ writing (17). Reviewing work that reaches outside the classroom walls can carry even greater benefits; graduate students in a chemistry course tasked with editing Wikipedia assessed information more critically than when simply studying for the class, perhaps because they knew it would be seen by a larger audience (18).

Despite the benefits of examining published primary literature, as well as investigating papers with serious problems (2), students are given limited opportunities to interact with seriously flawed studies; exposure to both good and bad experimental design and writing is simply not a standard part of graduate education. This fails to reflect the role of a professional scientist, which includes interacting with flawed literature as a reviewer and communicating effectively as an author.

The previously described issues in science communication and study design represent an alarming amount of wasted time, money, and potential. Solutions like the experimental programs discussed to address those concerns do exist, but such programs are not sufficiently widespread to match the present need (1). They target certain disciplines while excluding others and are not considered a standard part of graduate education. Graduate students need alternative ways to improve their skills by gaining early experience in the scientific review process.

The Journal of Emerging Investigators

Graduate students and postdoctoral scholars, in any STEM field and in universities across the world, have the opportunity to participate in the review and editorial processes of the Journal of Emerging Investigators ( JEI) (https://www.emerginginvestigators.org). JEI is an open-access, peer-reviewed scientific journal that publishes research papers in the fields of biology, psychology, chemistry, physics, math, and engineering by middle and high school student-authors (19). JEI was started as a way for young scientists to communicate their scientific research, regardless of the sophistication of the work. For example, JEI has published a variety of types of student research, from investigations of microbial presence on common household surfaces (considered less advanced) to the fabrication of solar cells by sulfurization of thermally evaporated metal stacks (considered more advanced) (Table 1). The ability to help young scientists effectively communicate their science in a clear and logical manner involves many skills that are actually required of professional scientists. These include understanding how to effectively construct and communicate one’s research, identifying flaws in communication and experimental design, and mentoring and inspiring the next generation of scientists. Working with JEI addresses the lack of opportunities for peer-review experience in existing graduate programs because it is open to students in any scientific discipline at any university and allows graduate students to critique papers that may have serious experimental flaws.

TABLE 1.

Examples of articles published by JEI student-authors arranged by field of study and level of sophistication.

Scientific Field	Article Title	Level
Biology	Isolation of Microbes from Common Household Surfaces	Beginner
	The Effect of the Stomatal Index on the Net Rate of Photosynthesis in the Leaves of Spinacia oleracea, Vinca minor, Rhododendron spp., Epipremnum aureum, and Hedera spp.	Intermediate
	The Protective Effects of Panax notoginseng Saponin on the Blood-Brain Barrier via the Nrf2/ARE Pathway in bEnd3 Cells	Advanced
Health Science	Does Music Directly Affect a Person’s Heart Rate?	Beginner
	Using a Risk Assessment Questionnaire to Identify Prediabetics and Diabetics in Tandag, Philippines	Intermediate
	The Impact of Age on Post-Concussive Symptoms: A Comparative Study of Symptoms Related and Not Related to the Default Mode Network	Advanced
Chemistry	Effectiveness of Biodegradable Plastic in Preventing Food Spoilage	Beginner
	Variation in Caffeine Concentration among Different Weight Loss Supplements Containing Green Tea and Green Coffee Extracts	Intermediate
	Fabrication of CuSbS2 Solar Cells by Sulfurization of Thermally Evaporated Metal Stacks	Advanced
Engineering	More Efficient Helicopter Blades Based on Whale Tubercles	Beginner
	Comparing the Voltage Output of Water in Drop and Flow Form Using a Piezoelectric Sensor and Hydroelectric Turbine	Intermediate
	The Development and Maximization of a Novel Photosynthetic Microbial Fuel Cell Using Rhodospirillum rubrum	Advanced

Graduate students may serve JEI as reviewers and/or editors. The responsibilities associated with each position differ. Editors have a broad role: they decide whether to reject or accept a manuscript, complete a preliminary review, select reviewers, compile all reviewer comments into an editor’s letter, and handle re-submission of the paper. Reviewers have a more focused role: they summarize the manuscript and their opinion of it, identify major changes needed in writing or experimental design, and submit their comments to the editor. While editors will see and manage manuscripts in any scientific field, reviewers are chosen to review manuscripts based on their field of research and will therefore see manuscripts that most closely relate to their own research. Both reviewers and editors must remember that they are critiquing and communicating with middle or high school students, and must balance improving the scientific integrity of the paper with encouraging the young scientist. Regardless of the level of sophistication in the science of the papers they work on, both reviewers and editors practice skills applicable to the publication process.

The present study

Although the science in papers submitted to JEI is not always as sophisticated as the science in other academic journals, the process of reviewing and editing flawed JEI papers mirrors the process shown to benefit students in literature discussed previously. The objective of our research was to determine whether participation in the review or editorial processes of JEI affected graduate students’ perception of their abilities in science communication, critique, and career development.

METHODS

Data collection and analysis

A 12-question survey created using SurveyMonkey was sent via e-mail to 215 individuals in the JEI reviewer and editor databases and had a response rate of approximately 47% (Appendix 1). This survey assessed students’ previous research experience, experience within JEI, motivation for participating, and impacts experienced as a result of participation. Questions about impact assessed growth in three domains: communication, critique, and career skills. Nine of the questions were constructed on a six-point Likert scale, and data were analyzed using IBM SPSS Statistics.

This project did not require IRB review. It was submitted to the Institutional Review Board and deemed a project evaluation, not human research.

Demographics

Graduate student editors and reviewers were asked about their age, the undergraduate and graduate institutions that they had attended, and the number of years they had been in their current PhD program in order to assess what categories of graduate students were most likely to participate as a reviewer or editor within JEI (Appendix 1). The majority (54.45%) of the reviewers and editors were in the 26 to 30 age range (Appendix 2). Editors and reviewers came from 67 unique undergraduate institutions across North America, Europe, Africa, Asia, and the Middle East.

Among the respondents, reviewers outnumbered editors, with 69% of students serving as reviewers and only 36% serving as editors. The percentages add to more than 100% because some students served in both roles, as editors were recruited from the existing pool of available reviewers. When reviewers and editors were compared in this study, individuals who had served in both roles were removed from analysis in order to isolate the effects of reviewing versus editing and ensure that respondents were not counted twice in subsequent analysis. Individuals who stated that they had not reviewed or edited any papers for JEI were also not considered in analysis based on JEI experience.

Perceived impact on skills

For each domain (communication, critique, and career), respondents were asked to assess the perceived impact of reviewing or editing for JEI on specific skills (Table 2). Editors and reviewers were asked to rank the change due to JEI in these skills on a 6-point Likert scale, with 1 being “significant decrease,” 2 being “moderate decrease,” 3 being “slight decrease,” 4 being “slight improvement,” 5 being “moderate improvement,” and 6 being “significant improvement.”

TABLE 2.

Perceived impact on skills by domain assessed.

Domain	Skills
Communication	Your ability to logically structure ideas in your own scientific writing
	Your ability to communicate your own research to a general audience
	Your ability to organize data and scientific information in your own scientific writing
Critique	Your ability to identify the strengths and weaknesses of experimental design
	Your ability to interpret scientific findings and draw conclusions
	Your ability to identify organizational strengths and weaknesses of scientific communications in your own field
Career	Your confidence in reviewing papers for your own field
	Your understanding of the roles and responsibilities of being a scientist
	Your ability to effectively manage time required by multiple projects

A broad, open-ended question elaborated on these numerical responses by asking students to comment on how serving as an editor or reviewer impacted their own research, education, and/or career goals (Appendix 1). The answers to the open-ended questions on this survey were not analyzed using a coding scheme but are used in this paper to illustrate the empirical results.

RESULTS

The objective of this research was to assess whether involvement with the Journal of Emerging Investigators as a reviewer or editor impacted graduate students’ perceptions of their abilities in three domains: science communication, scientific critique, and career development. The 12-question survey, which elicited 101 responses, was designed to assess the students’ experience within JEI and perceived benefits experienced through participation (Appendix 1).

The three domains

Editors consistently rated the perceived impact of JEI on their skills more highly than did reviewers in each of the three domains assessed (Fig. 1). Due to the small number of responses in some categories, the following analysis was based on descriptive comparison of the group means. Reviewers reported the greatest perceived impact in the domain of critiquing, while editors reported the greatest perceived impact in career skills (Fig. 1). For both editors and reviewers, the mean responses in all three domains were above the midpoint of 3.5.

FIGURE 1.

Mean perceived impact among reviewers and editors across three domains. Respondents were asked to rank the impact of their participation in JEI on three domains using a 6-point Likert scale.

Communication

We sought to determine whether reviewing and editing the work of middle and high school students enhanced the abilities of JEI participants to communicate their own scientific work. Overall, the mean ranking of perceived impact in this domain was 4.56 among editors and 4.25 among reviewers (Fig. 1). Moderate or significant perceived improvement in communication skills, signified by a score of 5 or 6 on the 6-point Likert scale, was reported by 47.50% of the editors and 26.55% of the reviewers (Appendix 2). Of the three skills that were ranked within this domain, editors and reviewers alike reported the largest perceived gains in their “ability to organize data and scientific information in [their] own scientific writing,” though editors’ perceived gains surpassed reviewers’ in this area (Fig. 2A). Student responses to open-ended questions also demonstrated this perceived impact, with students saying that the experience helped them better “communicate my science to an audience outside my field” and “gave me confidence in my writing.” It also helped them to understand the overall importance of communication as an “aspect of good science” and “how important it is to effectively communicate your own research to people with no scientific background.”

FIGURE 2.

Mean perceived impact for skills in the three domains. A) Scientific communication, B) scientific critique, C) career. For each domain, respondents were asked to assess the perceived impact of reviewing or editing for JEI on specific skills and to rank the change due to JEI in these skills on a six-point Likert scale, where 1 = significant decrease, 2 = moderate decrease, 3 = slight decrease, 4 = slight improvement, 5 = moderate improvement, and 6 = significant improvement.

Critique

In addition to assessing communication skills, we were interested in whether participation correlated with increased self-perceived critique skills. Within this domain, the three skills rated focused on experimental design, interpreting data, and identifying organizational strengths and weaknesses within their own scientific field. Here, the mean perceived impact was 4.67 among editors and 4.44 among reviewers (Fig. 1). Moderate or significant perceived improvement in critique skills was reported by 56.96% of the editors and 34.21% of the reviewers (Appendix 2). Both editors and reviewers perceived the greatest impact on their ability to identify the strengths and weaknesses of experimental design (Fig. 2B). Participants reported that working with JEI “prepared me for a lifetime of reading papers critically” and “helped me ID problem areas in my own writing” as well as in the writing they were assigned to review and edit. Participants also spoke about how they had learned about the scientific review process and changed the way they reported feedback, with one student specifically noting that JEI “helped me formulate critiques in a way that encourages young scientists.”

Career

Given that reviewing and editing papers is a responsibility of professional scientists, we wanted to determine whether participation in JEI as an editor or reviewer correlated with an increase in self-perceived career-readiness. The average overall mean ranking of perceived impact on career skills for was 4.84 among editors and 4.41 among reviewers (Fig. 1). A higher percentage of individuals in each group ranked the perceived impact of JEI on their career skills as moderate or significant compared with the other two domains, with 60.49% of editors selecting 5 or 6 on the scale and 37.93% of reviewers doing the same (Appendix 2). Both groups had the highest mean perceived-impact score (5.037 and 4.525 for editors and reviewers, respectively) in response to the question regarding their understanding of the roles and responsibilities of being a scientist (phrasing for this question came from results reported in the literature, 20) (Fig. 2C). One student said that, as a result of JEI, “I am now comfortable considering a career in publication,” and another called it “absolutely critical to the next step in my career.” Graduate students spoke positively about how JEI allowed them to experience the reviewing process early in their career; one student expressed that working with JEI was “an important experience for me … to be excited about research again.”

Effect of experience and age on the impact of JEI

Among both editors and reviewers, individuals with more experience reviewing and editing rated JEI as having had more of a perceived impact on their skills than did editors and reviewers with less experience (Fig. 3). Editors tended to rate the perceived impact of the JEI process on skills across each domain more highly than did reviewers of the same experience level. Editors generally reported the highest perceived growth in the area of career skills if they had worked on up to 10 papers; beyond 10, the highest perceived growth was reported in communication skills, followed by career and then critique. Reviewers across all experience levels consistently self-reported that critiquing skills benefited the most, though the disparity between scores in this domain and others became more pronounced with greater JEI involvement. Interestingly, reviewers who had reviewed 10 or more papers actually rated the impact on their critiquing skills more highly than editors with the same experience level; this was the only instance for which this was the case.

FIGURE 3.

Change in mean perceived impact on skills across three domains based on experience. A) Mean perceived impact reported by editors based on the number of papers edited at the time of the survey. B) Mean perceived impact reported by reviewers based on the number of papers reviewed at the time of the survey.

Older students generally held more favorable views of the impact of JEI on their skills in the three domains (Fig. 4). There was no clear relationship between age and number of papers reviewed or edited, and older students were not more likely to become editors than they were reviewers or vice versa. There was also no clear relationship between perceptions of impact of JEI and years of graduate school.

FIGURE 4.

Mean perceived impact on skills by age. Respondents were asked to rank the impact of their participation in JEI on three domains using a six-point Likert scale.

DISCUSSION

Prior research has demonstrated that graduate students lack key skills critical to success in science, such as communicating research to a general audience, structuring scientific writing, and correctly applying the principles of experimental design (3, 8, 12). The present study investigated whether working with JEI in the role of editor or reviewer increases graduate students’ perceptions of their abilities in communication, scientific critique, and career skills. Overall, our data demonstrate that both editors and reviewers report self-perceived improvement across all three domains. Editors, whose role involves the synthesis of feedback from multiple reviewers and interaction with papers in their earliest stages, perceived greater benefits than reviewers in every domain assessed by the survey. Reviewers and editors who had worked on more papers rated the perceived impact of JEI on their skills more highly than reviewers and editors who had worked on fewer papers. Perceived impact on scientific critique skills was rated more highly by reviewers than editors once the graduate student in question had worked on 10 or more papers. Older graduate students tended to rate the perceived impact of involvement with JEI more highly than younger students.

The greater self-reported success of editors relative to reviewers may be due to their role as synthesizers of information; editors consider all factors when selecting appropriate reviewers for a given paper and synthesize feedback from multiple reviewers into the editor’s letter. This responsibility requires that editors fully comprehend the author’s skill level, the manuscript, and each piece of reviewer feedback. Editors also have the opportunity to view the manuscript before it has passed pre-review. Previous research shows that evaluating papers with fatal experimental flaws improves future student research output (12), so exposure to the earliest stage of the peer review process may carry greater benefits than exposure to later stages.

While editors saw the highest perceived impact on their career skills, reviewers reported the highest perceived impact on their critiquing skills, even to the point of self-reporting higher impact than editors once they had reviewed 10 or more papers. The intense involvement of editors may lead to high initial impact, while reviewing many papers has a greater impact than does editing many papers. Repetition appears to play a role in reviewer success. The benefits of being a reviewer, a position with fewer but more focused responsibilities, also appear to be focused in the area most pertinent to the reviewer’s main responsibility: providing critical and constructive comments.

The primary limitation of this study was the use of self-reported data to assess the impact of JEI on graduate student abilities. Due to time restraints and the difficulty associated with administering skill tests for all domains before and after involvement with JEI, it was more effective to survey individuals who had already worked with JEI than to recruit and systematically test new individuals. While it was not possible to assess individuals both before and after their work with JEI, individuals were assessed at different points during their work with JEI. Individuals who had edited or reviewed more papers tended to rate the impact of JEI on their skills more highly and positively than individuals who had edited or reviewed fewer papers. These data suggest that work with JEI is correlated with increases in self-reported scores rather than these being a result of survey design. This relationship was not mediated by increased graduate school experience, as years of graduate school and overall perceived impact of JEI were not correlated. Additionally, prior research has demonstrated a relationship between students’ self-evaluations and actual performance in writing (21), which suggests that using self-perceptions can be a relevant tool for assessing actual skill.

The division of perceived-impact questions into three separate domains was necessary for the purpose of analysis; however, there is overlap between the categories. For example, the skill of “identifying organizational strengths and weaknesses of scientific communications in your own field” (Appendix 1), which was considered a critique skill, also involves communication skills. These overlaps reflect the fact that the skills required to be a scientist are interconnected and that growth in any given area may also boost other, related skills.

Our results suggest that flawed science writing can be used to improve skills currently lacking among graduate students. The results of this study align with those of previous studies: engaging with primary literature helps graduate students to attain skills relevant to the publication process (1, 2). However, our results uniquely demonstrate that to attain maximum overall growth in skills, graduate students of any age should take on the role of a journal editor. This includes interacting with a paper in its earliest stage, considering all relevant information in evaluating the paper, and thoughtfully presenting criticism in a format that is encouraging and helpful to the author of the paper. To improve abilities in scientific critiquing, graduate students can become reviewers who read a paper in detail to critique its writing and experimental design; they should review multiple papers for maximum effect. The results of this study not only attest to the impact of JEI, but also lend support for creating other opportunities for graduate students to review and edit; for example, perhaps existing journals could recruit graduate students to critique for a special student section of the journal.

Graduate student reviewers and editors represent only one part of the JEI process. This process also relies on middle and high school student-authors to create and submit papers and their teachers to introduce them to JEI and serve as intermediaries. Future research can assess the impact of JEI on these two groups. Research can also examine how best to implement programs like JEI in graduate schools so that students across many universities and disciplines have the chance to benefit.