Preprint Peer Review Enhances Undergraduate Biology Students' Disciplinary Literacy and Sense of Belonging in STEM

ABSTRACT

Education about scientific publishing and manuscript peer review is not universally provided in undergraduate science courses. Since peer review is integral to the scientific process and central to the identity of a scientist, we envision a paradigm shift where teaching peer review becomes integral to undergraduate science education. We hypothesize that teaching undergraduates how to peer review scientific manuscripts may facilitate their development of scientific literacy and identity formation. To this end, we developed a constructivist, service-learning curriculum for biology undergraduates to learn about the mechanisms of peer review using preprints and then to write and publish their own peer reviews of preprints as a way to authentically join the scientific community of practice. The curriculum was implemented as a semester-long intervention in one class and, in another class, as an embedded module intervention. Students' scientific literacy and peer review ability were assessed using quantitative methods. Student’s perceptions of their scientific literacy and identity were assessed using thematic analysis of students’ reflective writing. Here, we present data on the improvement in the peer review ability of undergraduates in both classes and data on the curriculum’s interrelated impact on students’ development of scientific literacy, identity, and belonging in peer and professional discourse spaces. These data suggest that undergraduates can and should be trained in peer review to foster the interrelated development of their scientific literacy, scientific identity, and sense of belonging in science.

INTRODUCTION

Undergraduate science education often focuses on how experiments are carried out and the knowledge generated by the resulting research literature, but it misses an opportunity to engage students in the critical validation process that translates one into the other, the peer review of primary scientific literature. Authentic laboratory experiences (e.g., course-based undergraduate research experiences, independent research) are known to be important to undergraduate science education as they enculturate students in the science community and increase understanding about the principles of experimental research (1). The value of early research experiences depends on them being authentic and within a community of practice (CoP) (see Table 1 for definitions of terms) (1–3). Moreover, implicit teaching of scientific inquiry through experimentation is insufficient for students to learn how scientists engage in inquiry (4, 5). As a result, there are demonstrable differences between what scientists experience and what students learn about regarding the process of science (6–8). Missing from many undergraduate research experiences are opportunities to learn how scientists communicate through scholarly publishing and peer review. We posit that providing undergraduates with explicit instruction in real-world forms of scientific communication, like peer review, will develop their scientific literacy and disciplinary literacy (Table 1).

TABLE 1.

Definitions of terms used

Term	Definition
Community of practice	A model for studying learning and identity development of individuals who share ways of thinking, communicating, or doing as they develop mastery of knowledge and skills through participation in the community (1).
Scientific literacy	The ability to know how scientific knowledge is generated and used to make evidence-based claims and how to make authentic scientific content.
Disciplinary literacy	What it means to think, read, communicate, and use information like an expert in a particular discipline
Authentic peer review	The process of writing critiques of scientific research manuscripts to evaluate and improve their scientific integrity and clarity. We use the modifier “authentic” to distinguish this process from when students evaluate other students’ classwork, which was not the focus of this study.
Constructivist	An approach to education based on learning through experience, which acknowledges that learning is an active and socially constructed process.
Service learning	The integration of academic activities with community needs, combining service with reflection in a structured learning environment.
Preprint	A scientific research manuscript that the authors openly share on a free, online server, usually prior to journal-organized peer review and curation.
Scientific identity	The composition of self-views as someone who knows about, uses, and contributes to science as part of the scientific community.

One way in which undergraduates can learn about and engage in authentic scientific conversation with a community of practicing researchers is by participating in the peer review of manuscripts. Peer review is integral to the scientific process, yet scholarship experiences for undergraduates (such as writing and publishing manuscript reviews) are rare, highlighting the novelty of our curriculum. The process of peer review is the backbone of scientific inquiry and a central component of the identity of a scientist (9). It justifies public confidence in scientific results and drives decisions about what research is published and funded. Therefore, education about peer review and participation in authentic peer review (Table 1) ought to occupy a central role in undergraduate science education, in the same vein as education about experimental research and participation in laboratory research (10, 11). Peer review is a form of disciplinary literacy (Table 1), and as such peer review is often reserved for those perceived as experts (e.g., faculty) and explicitly excludes students (12, 13). Yet, how can students develop disciplinary literacy, i.e., what it means to think, read, communicate, and use information like an expert in a particular discipline, if they are not provided instruction and practice in this essential skill? Therefore, peer review represents part of science, technology, engineering, and math (STEM) education’s “hidden curriculum” of unstated norms, values, skills, and expectations that are untaught yet required for success (14). Since peer review is integral to the scientific process and central to the identity of a scientist, we envision a paradigm shift that makes teaching peer review integral to undergraduate science education (15). Just as early research experiences help students form a scientific identity (16) and develop scientific literacy (17), so too can early scholarship experiences in peer review.

To this end, we developed and assessed a novel constructivist, service-learning curriculum for undergraduates to learn about the mechanisms of peer review, then write and publish their own peer reviews as a way to join the scientific community of practice. This contrasts with the traditional didactic model of peer review education in current practice, where a professor invited to review for a journal might engage a single trainee in one review exercise, with or without explicit instruction, feedback and/or disclosure to the journal editor (12). Our recent analysis of early career scientists revealed that formal, evidenced-based instruction in peer review is rare (12), so there is an unmet need to develop curricula on this topic. The new curriculum was designed with the goal of positively contributing to student learning, pedagogical research, and society (Fig. 1). We leveraged an innovation in scientific publishing, preprints, which are scientific manuscripts uploaded by the authors to a free, public server, often at the same time as submission to a peer-reviewed journal (18). Depositing articles as preprints on servers prior to journal submission has long been a normal practice in fields such as physics and mathematics, and it has recently grown in popularity in the biological sciences (19, 20). At the same time, experiments in open and pre- and postpublication peer review (21–23) have created preprint review platforms such as Review Commons (24), Early Evidence Base (25), Sciety (26), and PREreview (27). These platforms remove peer review from the exclusive realm of journals to increase participation in the peer review process. In contrast to participating in traditional journal club activities using already-finalized and published journal articles, undergraduate students in our curriculum now have the opportunity to engage in genuine peer review experiences, see work in progress, and experience the joy of working to improve the integrity and clarity of scientific manuscripts.

FIG 1.

Proposed impact of the curricular intervention (A) on student learning goals (B), research outcomes (C), and benefits to society (D). By explicitly teaching students about peer review and engaging them in it, we hypothesized that this curriculum would develop students’ disciplinary literacy (the ability to think, read, communicate, and use information like an expert in a particular discipline) and scientific identity (the composition of self-views as someone who knows about, uses, and contributes to science as part of the scientific community).

When students read published journal articles, they are not aware of the growth that occurs through peer review prior to publication. They see a retrospective narrative of scholarship, instead of the more realistic view of science as a constant work in progress in which failures and corrections are common. Undergraduates may struggle to reconcile final polished work with their personal experiences with science, such as failed experiments, negative data, or unsupported hypotheses. This disconnect could, in turn, negatively affect their sense of belonging in science and their understanding of the nature of science. To address this problem, our curriculum teaches students about peer review using preprints which are live, first-draft manuscripts. Reviewing preprints gives students an opportunity to help professional scientists improve their work by sharing their reviews with the authors. Another novel feature of our curriculum is that it provides students with the opportunity to publish their peer reviews of preprints on open-access, journal-independent internet platforms as a way to authentically engage with the scientific literature and the scientific community of practice.

Using an apprenticeship model from the pedagogy literature (16), our curriculum was designed to facilitate students’ self-development and self-expression within a community of practice, both within peer discourse spaces (e.g., by engaging with peers in the classroom) and professional discourse spaces (e.g., by engaging with preprint authors by publishing reviews in professional online forums). Here, we present findings from an exploratory study (n = 19 undergraduate upperclassmen) that used mixed methods to measure the interrelated impact of our curricular interventions on students’ sense of science identity, literacy, and belonging in peer and professional discourse spaces. We hypothesized that this curriculum, by explicitly teaching students about peer review and authentically engaging them in a community of practice, would improve students’ disciplinary literacy (specifically, peer review ability) and foster a sense of scientific identity and belonging in the scientific community.

METHODS

Context: intervention and participants

In a research liberal arts college for female, transgender, and nonbinary students (Mount Holyoke College), we implemented a curriculum on peer review in two different contexts: (i) as a full 14-week seminar course (Course 1, Peer Review in Biology) or (ii) as a single unit of peer review activities embedded within a disciplinary biology course (Course 2, Vaccines). Both courses were offered as upper-level electives and taught by the same instructor (R.S.L.) in the same semester (Spring 2022). In Course 1 (n = 9), peer review activities were scaffolded to facilitate students’ movement to a more legitimized science identity through four units, based on the clinician training paradigm of “see one, do one, teach one” (28) and the gradual release of responsibility model of literacy education that uses the framing “I do, we do, you do” (29). Figure 2 provides a conceptual overview of the full curriculum taught in Course 1. In Course 2 (n = 10), only one minimal unit (i.e., “do one,” where students perform peer reviews) was implemented to complement a discipline-specific course on vaccine biology. In both classes, the instructor and/or students selected biology preprints of interest to review, critically analyzed the preprints in writing and in discussion using guiding questions provided by the instructor, and then wrote peer review reports as a professional peer reviewer would do (see “Context: peer review activities,” below, for further detail). Students submitted weekly reflection journals in response to prompts about their perceptions of their performance, sense of self-efficacy, and understanding of disciplinary literacy in the context of peer review. A majority of students indicated their intention to publish their reviews publicly online to document their expertise and participate in the professional CoP. All students interacted with preprint authors and other experts in peer review through e-mail and video interviews.

FIG 2.

Peer review curriculum. In this constructivist service-learning curriculum, peer review activities are scaffolded to transition the student from an apprentice to a more legitimized science identity through 4 units, loosely based on the clinician training paradigm of “see one, do one, teach one” (28) and the gradual release of responsibility model of literacy education that uses the framing “I do, we do, you do” (29). Educators can choose to use a unit(s) alone or together depending on course needs and students’ previous experience. Throughout the curriculum, students review preprints freely available on servers and have the opportunity to publish their reviews to document their scholarship and serve the scientific community. (The curriculum schematic was adapted from our previous publication [15].)

All students provided informed consent to participate in the study, which was verified by the Mount Holyoke Institutional Review Board as exempt according to 45CFR46.101(b)(1, 2): (1) Educational Research, (2) Tests, Surveys, Interviews, on 12 October 2021.

Context: curriculum and peer review activities

Students in the full curriculum (Course 1, Peer Review in Biology) engaged in four peer review events throughout the semester (Fig. 2):

1. Review 1: individual review on a manuscript selected by the instructor. The initial, baseline event was assigned as individual homework after the first class meeting, after an initial discussion of the concept of peer review but before students had carried out any in-depth training in the course. All students reviewed the same manuscript, selected by the instructor for its accessibility to a general biology audience. The manuscript was written by precollege students, submitted to the Journal of Emerging Investigators (30), and ultimately published (31).

2. Review 2: individual review on a preprint selected by student groups. The next event took place in Unit 3, after ~6 weeks of explicit teaching about peer review (Fig. 2, Units 1 and 2) and after students cocreated a rubric to evaluate preprint peer reviews (32). Students were grouped by interest in biological topics (e.g., cancer, the 2019 coronavirus disease pandemic), and then each group selected one preprint to review, which was approved by the instructor as being accessible. Manuscript accessibility was determined as a collaboration between the student and instructor, based on the preprint’s usage of methodology, jargon, and statistical analyses. When most of the article was deemed accessible but specific components were not (e.g., advanced statistical analyses in an otherwise-approachable study), then students reviewed all but the inaccessible component and made a disclaimer at the top of the review. Each group member individually carried out peer review of the same preprint as homework. Students were not provided with detailed written instructions for how to write a peer review; instead, they were asked to create their review based on their learning from Units 1 and 2. Guiding questions for peer review were provided as an optional resource, but answers were not required to be provided in the assignment.

3. Group review: group review on a preprint selected by student groups. Students shared their individual review 2 with their group members, then spent time in class discussing their individual reviews and the instructors’ feedback on them. Then, they synthesized their individual reviews into one group review (the third review event). After completing the group review, students reread their individual reviews and self-graded using the rubric they cocreated (32).

4. Review 3: individual review on a preprint selected by individual students. The fourth review event was assigned to students as individuals, as a final assessment after the completion of the curriculum. Without any input from the instructor, each student selected a preprint of interest and wrote an individual review.

Students in Course 2 (Vaccines) engaged in a single module of the curriculum (Fig. 2, Unit 3) and three peer review events:

1. Review 1: individual review on a manuscript selected by the instructor. The baseline event was assigned as individual homework after a 30-min discussion of the concept of peer review and after ~6 weeks of disciplinary lessons on vaccinology. All students reviewed the same preprint, selected by the instructor for its accessibility to a general vaccinology audience (33). Because this course did not involve explicit lessons on peer review in class, students were instead provided with detailed written instructions for how to perform a peer review, including guiding questions which were required to be answered as part of the assignment.

2. Review 2: individual review on a preprint selected by student groups. Students were grouped by interest in vaccine topics, and then each group selected one preprint to review, which was approved by the instructor as being accessible. Each group member individually carried out peer review of the same preprint as homework, using the same detailed instructions and guiding questions as Review 1.

3. Group review: group review on a preprint selected by student groups. Students shared their individual reviews (Review 2) with their group members, then spent time in class discussing their individual reviews and the instructors’ feedback on them. Then, they synthesized their individual reviews into one group review (the third review event).

Assessment of disciplinary literacy: peer review quality

All peer reviews written by students were deidentified by the instructor (R.S.L.) and provided to the independent researcher (G.S.M.). The researcher generated four metrics of peer review ability (instruments) using three unique tools:

• The Review Quality Instrument (RQI) (34), which consists of eight Likert-scale questions (ranging from 1 to 5). One question asks the evaluator’s overall opinion of the review, and this question is reported here as the RQI (range, 1 to 5). The other seven questions ask about components of the review, and we have combined these and report them as the “RQI total” (range, 7 to 35).

• The PREreview review assessment rubric (35).

• A rubric for evaluation of preprint reviews was generated by the students in course 1 (32). This consisted of a series of scores of 0 to 4 being awarded to different sections of the review, which were then converted into a percentage, reported here as “MHC”.

Scores on each instrument were normalized to percentages (i.e., divided by the total possible score for that instrument and multiplied by 100) to allow for comparisons between the instruments, since the maximum score varied between each instrument. A repeated-measures two-way analysis of variance (ANOVA) was performed to evaluate the impact of instrument and each chronological review event (independent variables) on normalized peer review scores (dependent variable). Tukey’s multiple-comparisons posttests were used to make pairwise comparisons between review events. All statistical analyses were performed using GraphPad Prism version 9.4.

Additionally, scientific literacy more broadly defined was assessed using Gormally et al.’s TOSLS survey (36) administered before and after each of the two interventions (see Appendix S1 in the supplemental material for more details).

Thematic analysis

Students completed weekly reflection journals, which were deidentified by the instructor (R.S.L.), assigned pseudonyms to retain anonymity, and provided to the independent qualitative researchers (J.L.O., M.M.B.). Deidentified reflection journal entries were uploaded to MaxQDA (ver. 22.2.1) and coded by thematic analysis (37) (see Appendix S2). Thematic analysis used students’ science literacy, science identity, and sense of belonging within the scientific community to inform initial latent codes. Initial themes corresponding to science literacy included the following: understanding science content, using science skills to help others, talking about science with others, and practicing science now and in the future. These codes were then collapsed into the following overarching themes: knowledge (understanding and communication about science), practice (applying or performing science skills and knowledge), and value of practice (understanding the use and need of science). We also identified personal and environmental variables corresponding to a student’s identity that were then organized as professional identity (pursuing a science career or internship and interacting with science professionals) and personal identity (systemic and structural barriers and access to resources). Finally, we divided codes related to belonging into presence, absence, or facultative (i.e., a sense of belonging in some contexts but not others). To identify differences between each course context, we compared codes across courses. To establish trustworthiness, two team members separately coded 20% of the students’ reflections and ensured a minimum 90% interrater reliability. We also engaged in expert debriefing to discuss the alignment between our themes and our theoretical framing.

Inclusion and exclusion criteria

Only data from students who completed all assignments were included in the analysis. This represented all 9 students in Course 1 and 10 of 11 students in Course 2. In Course 2, one student did not complete reflection journal entries, and so that student’s data were excluded from the thematic analysis. Another student did not complete the peer review exercises on time or the postsurvey for TOSLS, and so this student’s data were excluded from those analyses.

RESULTS

Peer review curriculum enhances undergraduates’ disciplinary literacy

Student’s peer review ability, a form of disciplinary literacy, improved in a statistically significant and dose-dependent manner as a result of the full curriculum (P < 0.0001 by ANOVA) (Fig. 3A) and the embedded module (P < 0.0001 by ANOVA) (Fig. 3B). Baseline levels of peer review ability were established by the first review event in each course, where students wrote an initial review with absolutely no prior instruction (full curriculum) or with only 30 min of introduction to peer review (embedded module). Increases in peer review ability were observed regardless of which assessment tool was used (Fig. 3 and Tables 2 and 3). Multiple assessment tools were used as a way to reduce the subjectivity of the assessment, since each tool places different emphasis on elements of the review such as importance, originality, ability to identify of strengths and weaknesses in various sections of the manuscript, constructiveness of comments, ability to interpret results, balancing of positive and negative comments, and ability to distinguish and articulate major and minor issues. For example, the PREreview assessment focuses on tone and constructiveness of critique, and since no undergraduate students wrote harmful or offensive comments and since all made clear efforts to provide constructive feedback and readable prose, scores in their initial reviews were already approaching a saturation point. When using the RQI, an overall score out of 5 is given after evaluating a range of specific components; as this adds an extra layer of subjective opinion (i.e., the overall impression of the evaluator), both this final overall score and an average of the combined scores across all individual components were reported (Tables 2 and 3 and Fig. 3).

FIG 3.

Improvements in peer review quality as a result of the full curriculum (A) (n = 9) and embedded module (B) (n = 10). Students’ deidentified peer reviews were assessed by an independent researcher using four metrics: RQI, the single question in the Review Quality Index (34), where the researcher gives an overall assessment of the review; RQI total, the combined score for all questions in the RQI; PREreview’s assessment tool (35); and MHC, the grading rubric created by students in Course 1 (32). Review events are presented on the x axis in chronological order in the curriculum. Scores on each instrument were normalized to percentages to allow for comparisons between the instruments, since the maximum score varied between each instrument. Data are presented as means ± standard errors of means and were analyzed by two-way repeated measures ANOVA (P < 0.0001 for review event in both the full curriculum and embedded module) and by Tukey’s multiple-comparisons posttests for pairwise comparisons between review events. All statistically significant comparisons are indicated: *, P < 0.05; **, P < 0.01; ***, P < 0.001; ****, P < 0.0001.

TABLE 2.

Changes in peer review quality as a result of the full curriculum (Course 1)^a

Measure	Review(s) analyzed	RQI	RQI total	PREreview	MHC
Score range		1–5	7–35	1–5	0–100%
Avg score	Review 1	2.33	19.4	3.52	50.8%
	Review 2	3.22	21.1	4.18	73.2%
	Group review	3.56	23.7	4.33	81.0%
	Review 3	4.33	28.1	4.84	92.9%
Change per event	Δ(1→2)	0.18	0.05	0.13	0.22
	Δ(2→3)	0.22	0.2	0.13	0.2
	Δ(1→3)	0.4	0.25	0.26	0.42

^aStudents carried out 3 individual reviews as described in Methods, with review 2 being used to generate the group review. Changes in scores between different reviews are reported as Δ[Review#]→[Review#] and were normalized to give a range of change, from −1 (maximum decrease) to +1 (maximum improvement), with 0 signifying no change in peer review quality. Review quality was assessed by an independent evaluator using four metrics: RQI, the single question in the review quality index, where the evaluator gives an overall assessment of the review; RQI total, the combined score all questions in the RQI; PREreview reviewer’s assessment rubric score; and MHC, the grading rubric created by students in Course 1.

TABLE 3.

Changes in peer review quality as a result of the embedded module (Course 2)^a

Measure	Review(s) analyzed	RQI	RQI total	PREreview	MHC
Range of scores		1–5	7–35	1–5	0–100%
Avg score	Review 1	2.78	21.0	4.17	72.6%
	Review 2	3.00	21.0	4.35	69.9%
	Group review	4.78	27.6	4.87	93.9%
Change per event	Δ(1→2)	0.04	0.00	0.04	−0.03
	Δ(2→group)	0.36	0.19	0.10	0.24
	Δ(1→group)	0.40	0.19	0.14	0.21

^aStudents carried out 2 individual reviews as described in Methods, with review 2 being used to generate the group review. Changes in scores between different reviews are reported as Δ[Review#]→[Review#] and were normalized to give a range of change from −1 (maximum decrease) to +1 (maximum improvement), with 0 signifying no change in peer review quality. See Table 2 for description of the instruments used.

Overall and depending on the measurement tool used, the full curriculum resulted in a 25 to 42% increase in students’ peer review ability (Table 2, bottom row), which was comparable to the 14 to 40% improvements seen as a result of the embedded module (Table 3, bottom row). Measurement of improvement was lowest on the PREreview assessment, because many reviews were close to the ceiling on this tool. At the final peer review event, which occurred after completion of the interventions, students' reviews in both classes earned 80 to 97% of the total maximum score possible for each assessment tool. This result is remarkable because the RQI and PREreview tools were designed to assess the quality of reviews written by experts, not undergraduates or other learners. Many students elected to publish their reviews (38–42), implying that they were proud of the final products (see next section on perceptions). These data suggest that while the baseline quality of peer reviews written by an untrained undergraduate is mediocre as might be expected, these disciplinary literacy skills can be developed through an intentional curriculum that offers explicit instruction, iterative practice, and opportunities to authentically participate in a community of practice.

Peer review curriculum enhances undergraduates’ perceptions of science literacy, identity, and belonging

Student’s perceptions of their own development were captured in both courses through weekly reflection writing in response to specific prompts (see Appendix 3). Results from this analysis demonstrated that students developed an affiliation with the science CoP over the course of the interventions (Fig. 4). While many students were science majors (hence, early career scientists), a few used the course as an opportunity to engage in and learn about science as nonpractitioners. Therefore, prior to the intervention, students identified as novices (termed “apprentices” in the CoP model) (Fig. 4) or even as outsiders (in the periphery) (Fig. 4). As they participated in the peer review curriculum, students’ reflections on their literacy, identity, and belonging within the CoP demonstrated progression toward mastery (Fig. 4). Table 4 provides a componential analysis of the overarching themes of literacy, identity, and belonging achieved by students upon completion of an intervention, with each discussed in detail below.

FIG 4.

Community of practice conceptual model (based in part on ideas from reference 1). Students' progression from “apprentice” toward “master” is influenced by their sense of belonging within both peer and professional discourse spaces. Belonging is seen to bridge literacy (knowing and applying science) and identity (feeling like a scientist).

TABLE 4.

Componential analysis of the overarching themes of literacy, identity, and belonging achieved by students upon completion of the intervention, after either the full curriculum (Course 1) or the embedded module (Course 2)^a

Course no.	Student	Science literacy			Science identity		Science belonging
		Knowledge	Practice	Value	Personal	Professional	Academic	Professional
Course 1	Kasper	NO	✓	✓	−, +	✓	✓	NO
	Kori	✓	✓	✓	−, +	✓	✓	✓
	Gianna	NO	✓	✓	−	✓	✓	NO
	Niki	✓	✓	✓		✓	✓	✓
	Dani	✓	✓	✓	+	✓	✓	✓
	Aaliyah	✓	✓	✓		✓	✓	✓
	Kiara	✓	✓	✓	−	✓	✓	✓
	Zara	✓	✓	✓	−	✓	✓	✓
	Madi	✓	✓	✓	−	✓	✓	✓
Course 2	Sam	✓	✓	✓		✓	✓	✓
	Alex	NO	✓			✓	✓	NO
	Amara	✓	✓	✓	−	✓	✓	✓
	Isabella	✓	✓			✓	✓	NO
	Sofie	✓	✓	✓		✓	✓	✓
	Kris	✓	✓	✓	+	✓	✓	✓
	Riley	✓	✓			✓	✓	✓
	Jocelyn	✓	✓	✓		✓	✓	✓
	Aneta	✓	✓	✓		✓	✓	✓
	Lotte	✓	✓			✓	✓	✓

^aA checkmark indicates a student was coded as being scientifically literate, possessing a professional science identity, or having a sense of belonging within the science community. NO indicates student was coded as not possessing scientific literacy, identity, or belonging. Cells are left empty when student responses did not address aspects of literacy, identity, or belonging. −, indicates aspects of a student's personal identity that the student perceived as a disadvantage; +, indicates aspects of a student's personal identity that the student perceived as an advantage. Pseudonyms are used to maintain students’ privacy.

(i) Literacy. In both courses, the peer review curriculum improved students’ perceptions of their own scientific literacy. Scientific literacy included the students’ understanding of how scientific knowledge is generated, their engagement in science practices, and how they perceived the value of these practices. For example, students coded as scientifically literate were able to describe the peer review process and its value, as well as feel confident about performing a peer review. This was seen in Sam’s journal entry when she commented on her confidence in performing peer review after the intervention: “After thoroughly reviewing the standard requirements for a publication, as well as the scientific theory supporting the article, I felt surprisingly well-equipped to offer constructive feedback on the assigned preprint.”

A nuanced development and then strengthening of scientific literacy was seen in the full curriculum (Course 1) over each of the four units (Fig. 2). In Units 1 and 2, students were asked to reflect on their new understanding of what peer review is, who engages in peer review, and how to conduct one as part of establishing a baseline measure of their disciplinary literacy. In their early journal entries, students shared many of the misconceptions that they held about peer reviews, including that reviews were done after publication and that authors were responsible for recruiting reviewers themselves. This was seen in Kiara’s reflections when she stated, “I had this idea that there was not an editor involved, but rather that the author of the study themselves was responsible for reaching out and facilitating the process.” Students went on to refute these misunderstandings, even explaining why they were wrong.

Within this unit, students also shared their perceived value of the peer review process. Most students suggested superficial benefits, such as the ability to proofread articles prior to publishing them, but did not yet demonstrate a sophisticated understanding of the value of peer review plays for science (43). In Unit 3, during which students completed Review 2, the first review after instruction, all students discussed feeling confident, comfortable, or prepared for the assignment. Students explained that the course activities and materials helped to clarify what was expected of their review. In Unit 4, when students evaluated others’ reviews, there were obvious improvements in students’ depth of knowledge of the value of peer review. Several students were able to elaborate on the importance of reviewers in producing high quality publications. Kiara shared the following: “This feature is incredibly valuable to the process as it allows for the most amount of feedback for the author … as the more feedback that is provided for the review, the better the review can potentially be.” Kiara’s comments demonstrated an understanding that the review process contributes to how scientific knowledge is disseminated. Students in Course 2 were also seen to strengthen their literacy through critique of discipline-specific papers (e.g., on vaccines) and offering constructive feedback on peer presentations that occurred outside the embedded peer review module.

Finally, students in both courses mentioned that through reading and critiquing reviews made by their peers, they were able to sharpen their own skills in preparation for their final reviews. Many students discussed their plans for their final review, each noting unique areas of improvement. Kiara was inspired by her classmates’ use of a peer review rubric to keep track of necessary comments, stating that she would remember to use it in the future. Over the course of the semester, students first developed their knowledge, value, and practice of peer review and then further strengthened these points through repeated writing and critiquing of reviews.

(ii) Identity. In both courses, students’ perception of their science identity was influenced by their own personal and professional identities. Students who demonstrated a strong science identity talked about science with others in and out of class, described the ways in which their newly learned skills could be applied to their academic and postgraduate careers, and expressed their validation in the science field because of others who looked like them. Madi wrote, “The critical thinking skills I gained from this peer-review course have really started kicking in. I also feel confident that if my experiment works, I will be able to write about it for a manuscript.” Earlier in her reflection, Madi explained how learning to analyze and critique research data helped her to overcome a roadblock in her research lab: “I am also considering writing a thesis for my senior year because of the confidence I have gained in research, reading, writing, and reviewing.” Madi demonstrated that through reflection of her own experiences, she planned to engage in future disciplinary literacy activities that will likely further solidify her professional science identity. Although Madi was noted as developing her professional science identity, she also shared perceived disadvantages of her personal identity. As a first-generation international student, Madi acknowledged underrepresentation of those with marginalized identities in science and the bias they face due to their nationality and race. Similarly, Gianna shared, “Being a non-native English speaker, I know how [language could] be the largest barrier in science research.” Both students, among others, recognized systematic and structural barriers that inhibit individuals with certain identities from participating in or contributing to the scientific community.

(iii) Belonging in the peer CoP. Students reported that their sense of belonging changed as a result of the intervention in what we identified as two “discourse spaces.” The peer discourse space consisted of in-person and virtual classrooms, where students interacted with one another. The professional discourse space was where students interacted with science professionals (e.g., preprint authors, journal editors, expert reviewers). Engaging with classmates through discussion and cowriting peer reviews reinforced students' sense of belonging within the peer discourse space. A few students felt they belonged in some contexts, but not in others. For example, Isabella explained, “I think discussing the paper in class with people who are on the same level as me really helped.” Here, Isabella explained that the context of a classroom felt safe, but she went on to explain that in other contexts, where she perceived that there were some members who were different from her, she did not feel as confident. “In the past, I have done journal clubs with graduate students, which have been much more difficult for me to feel confident in sharing my thoughts. Today's class really helped me gain more confidence about my ability to review primary research independently.” Hence, some students expressed confidence in peer discourse spaces but not in professional ones prior to the intervention and implied that the confidence gained through the intervention could improve feelings of belonging in professional discourse spaces in the future.

(iv) Barriers to belonging. Students in both courses most often identified illiteracy (i.e., inadequate knowledge and practices) as a barrier to belonging. This was evident in Alex’s journal entry when they mentioned feeling like a “wallflower,” or on the periphery (Fig. 4), because of their lack of science experience: “I still don't really feel I am part of the scientific community. It is a huge field and looking at a few papers makes me feel like I am more of a wallflower than anything else…with my very limited experience I don't think it would be right to say that I am part of the community. I am still going through the initiation rites.” Although Alex described feeling like they were on the periphery of the CoP, they described their experience as part of an “initiation” period, which implies that they may have seen themselves as belonging sometime in the future.

(v) Belonging in the professional CoP. Emailing with preprint authors, interviewing editors, and expert reviewers during the interventions allowed for students in both courses to feel they belonged in the professional discourse space. Madi described how experts helped to foster her belonging within a group of scientists when she wrote, “… firstly I was able to make a conversation with a professional and deliver my question in a way that did make sense to them. The points they were making made me feel familiar with other jargon and issues surrounding the process of peer review …. I did not feel inadequate or less knowledgeable during these conversations.” Madi found the experience of interacting with others valuable in shaping her sense of being a member of a CoP. Likewise, Kiara wrote about how she perceived others to perceive her: “She [professional scientist] saw me as a student and as a scientist …. I so often do not feel seen or understood in the STEM field so having this moment to talk to her allowed for me to feel valid in our field.” In other words, Kiara’s sense of belonging was strengthened through positive reinforcement by those who she saw as experts in the professional science CoP. In their responses, Madi and Kiara pointed to their science literacy and identity being validated through engaging with experts within the CoP. Overall, these data demonstrated that belonging in peer and professional discourse spaces contributes to the development of students’ scientific literacy and identity, which then facilitates their progression from “apprentice” toward “master” in the scientific CoP (Fig. 4).

DISCUSSION

This study demonstrates that undergraduates are capable of being taught how to perform effective peer review of scientific manuscripts, a critical scientific skill and a form of disciplinary literacy often overlooked in STEM education. Our novel peer review curriculum resulted not only in improvements in disciplinary literacy, but also, importantly, in undergraduates’ perceptions of their scientific literacy, scientific identity, and belonging in STEM. These results were generated by both formats of the curriculum: a peer review course (full curriculum) and a short peer review module embedded into a disciplinary course. Therefore, instructors who are not able to offer a full course on peer review might still consider incorporating a module into a preexisting science course as a way to intentionally develop students’ disciplinary literacy, scientific identity, and sense of belonging in STEM.

We attribute the efficacy of the peer review intervention to three critical features: (i) explicit instruction that unmasks part of STEM’s hidden curriculum, (ii) iterative practice, feedback, and peer collaboration, and (iii) opportunities to authentically participate in a community of practice. Our curriculum explicitly teaches undergraduates how professional scientists evaluate primary literature and engage in discourse with the community of scientists through peer review and publishing, two features of STEM’s hidden curriculum that are rarely taught even to graduate students. Instead of reserving this knowledge and skillset for the privileged few who have quality independent research mentorship, our classroom-based instruction provides access to this information at an earlier stage, so that any science undergraduate can become savvy about professional practice. Explicit instruction can be effective in the form of class lessons (as was done in the full curriculum) or written, granular guiding questions (as was done in the embedded module). Both formats allowed students to see how manuscripts evolved through peer review (Fig. 2, Unit 2), instilling a growth mindset in students that contrasts with the crystalline truths portrayed in textbooks and published articles. Students can then reconcile their imperfect lived experience with science and better identify their potential to be scientists themselves.

A second feature of the curriculum’s success derives from iterative rounds of practice, feedback, peer collaboration, and revision. This was evident from the dose-dependent improvement in disciplinary literacy with each review event (Fig. 3), and the progression revealed in the thematic analysis of students’ reflection writing. A particularly beneficial exercise appears to have been the group synthesis activity, where students wrote individual reviews on a shared preprint, received instructor comments on their written work, read each other’s reviews, and then synthesized their findings into one group review, akin to an editorial letter in journal review. We attribute this benefit to the combination of individualized expert feedback (e.g., instructor comments on the review) and the safe spaces for growth created by peer discourse spaces (e.g., group revision and synthesis). The need for specific, written feedback on the reviews aligns with the findings of Houry et al. (44), who showed that the current model of pairing an expert and novice reviewer fails to train the novice when explicit feedback and/or a structured curriculum is not provided. The merit of student group work aligned with established knowledge on collaborative learning (45) and likely derived from the fact that groups were composed of peers engaged in the same training, removing hierarchical power dynamics based on career stage or previous review experience. Future work could focus on framing the group synthesis intentionally within a collaborative learning framework to examine contrasts to the traditional didactic training paradigm for peer review. When group work creates a peer community of practice, it may support students’ transition to self-perceptions of a more legitimized scientific identity (i.e., apprentice to journeyman) and as valuable members of the professional community of practice, which may explain why these self-beliefs are predictors of persistence in STEM careers, especially for minoritized students (46, 47). One implication for these data is that individual-feedback-then-group-synthesis exercises could benefit peer review training in contexts beyond the undergraduate classroom, e.g., for graduate student education and for onboarding new invited reviewers at academic journals.

Our findings are consistent with prior work on the pedagogical benefits of peer review in the classroom (e.g., calibrated peer review, CREATE systems) (reviewed in reference 12), which found that students learn best by participating themselves in the review process and when there are multiple rounds of review followed by feedback and revising (48–51). Students in the CREATE model, which uses published articles (not preprints) for review, commented on the importance of personal connections with scientists, e.g., through interacting with the authors of the published papers that they read or viewing footage of a prior group’s interaction (48). These students developed literacy about the practices of a CoP and expressed an increased interest in becoming scientists (48). Similarly, students in our curriculum reported increases in identity and belonging as a result of engaging with professionals, e.g., by e-mailing reviews to the preprint authors. Authors frequently replied to the students, expressing surprise and gratitude at the students’ labor and encouragement for students to stay in STEM. Some authors said that students’ reviews agreed with reviews from journals and would be implemented. Students reported that these interactions with the professional CoP contributed to a sense of joy and being valued members of STEM.

Given the importance of belonging in shaping students’ science identity, and possibly also persistence in STEM (52), one implication of this work is an appreciation of the nuances between different levels of belonging (53). Our data suggest that belonging in a CoP is composed of two parts: a sense of belonging in the student’s peer discourse setting (i.e., the classroom), in combination with a sense of belonging to the community of scientists (i.e., professional settings). The sense of classroom belonging appears to be important in creating a safe space for students to develop their literacy and explore their identity as scientists and their place in the wider scientific community. Other work has shown that a strong science identity predicts higher grades, with manipulation of belonging in college impacting the relationship between science identity and academic performance (47). Students have a clear understanding of what they are required to do to succeed academically, but what professional success looks like, and how they can achieve it, may be less clear (53). Addressing in an academic context how to move from the periphery to mastery in a professional CoP may be an important foundation in building professional scientific identity. This may have implications for retention in academia: moving instruction on how to be a practicing scientist away from later career stages and into earlier academic settings may allow earlier establishment of belonging in STEM. When greater attention is given to intentionally fostering a sense of belonging in the scientific community of practice, as was done in this novel peer review curriculum, undergraduate education may be more effective at developing a broader diversity in the next generation of STEM professionals.

A limitation of our study is that evaluating peer review quality in a standardized way is challenging. Reviews are subjective assessments of a manuscript, and so evaluations of a review become a subjective assessment of a subjective assessment. This is further complicated by comparing reviews of different manuscripts, as the quality of the review depends on how much (or little) there is to critique. We endeavored to make up for this limitation by using four assessment tools representing all that are publicly available and allow for quantification. Though the tools vary in their emphases, all define a quality review as having a respectful tone that balances strengths with weaknesses, which speaks to the trend of harsh journal reviews (12). In contrast, students’ reviews were never rude and frequently pointed out strengths, earning high marks for tone on the assessments. This may have been biased in our sample of Mount Holyoke College students, whose gender and/or early career stage identities are socialized to use respectful tone. Another major limitation of the assessment tools is that none evaluate the sophistication of the critiques (surface-level versus deep scientific analysis). For example, students requested that a clinical trial be rerun due to small sample size, without considering feasibility nor justifying what sample size would be required. One future direction of this work is to develop clearer standards for evaluation of peer reviews and, by extension, a rubric for peer review learning outcomes.

Despite seeing significant changes in disciplinary literacy (Fig. 3), we observed no significant changes in overall science literacy using Gormally’s test of science literacy survey (TOSLS) (36) with a pre- and posttesting methodology similar to that of Cartwright et al. (54) (see Appendix S1). We attributed this to a combination of factors similar to the findings of Cartwright et al. (54). Correct response rates were high at baseline (in agreement with what Gormally et al. reported for private liberal arts colleges (36)) and left little room for demonstration of improvement. It may be that in populations with lower initial TOSLS scores, our peer review intervention could still improve science literacy. TOSLS may not capture changes over the course of only one semester and may be more appropriate for longer interventions or our future cross-site comparisons.

In ongoing work, we are investigating the transferability of the peer review curriculum to other educational contexts (e.g., a large land grant university, a 2-year college), more diverse student populations, different instructors, and larger class sizes. For example, the modular intervention is currently being tested in a 200-student lecture and laboratory course, where the review activities take place in the lab because student-to-instructor ratios are lower. This introduces a new variable of multiple lab instructors with various comfort levels with peer review (e.g., graduate student teaching assistants who need peer review training themselves). Focus groups with instructors will be used to identify pinch points and develop appropriate training for instructors. Ultimately, this work will create evidenced-based open educational resources on peer review that will be transferable to diverse educational settings and enhance students’ interrelated development of scientific literacy, identity, and belonging in STEM classrooms and professions.