A guide to peer reviewing medical education scholarship: Advice from editors of AEM Education and Training

INTRODUCTION

Peer reviews in medical education can seem intimidating and the quality of reviews can vary widely. ¹ , ²  Many journals (including Academic Emergency Medicine Education and Training [AEM E&T]) provide scores for peer reviewers based on the quality of their reviews. While we share our reviewer scores each year, reviewers have asked how they could improve their reviewer scores and the strength of their reviews. As members of the editorial board for AEM E&T, we sought to share our insights and advice for creating a high‐quality peer review of medical education research. To address this need, we identified some of the common reasons for lower scoring reviews and features of higher scoring reviews to guide reviewers in the spirit of education and continual improvement.

While several guides and reporting guidelines already exist to help peer reviewers hone their craft, ³ , ⁴ , ⁵ , ⁶ we feel that it is our duty as members of the editorial board to share our journal's values and attitudes within the review process. This article describes a framework for peer reviewers to guide them while reviewing an education research paper. Our goal is to elevate the peer review process to assure that we are publishing high‐quality literature and advancing the field of emergency medicine medical education. The quality of scholarship published by AEM E&T is dependent not only on the authors of the papers but also on the quality of reviews by our peer reviewers. The journal is dependent on our community of peer reviewers for the success of the journal. High‐quality reviews benefit all stakeholders in the development and dissemination of useful scientific work. Before we delve into the specific tactics for being a good reviewer, it is important to first review why peer reviews are important to both you and our field.

Why perform peer review

There are a number of reasons why being a peer reviewer is a worthwhile experience. We have broken down these reasons into five main tenets:

1. You are an important voice within our field—Those who engage in editing and reviewing have an important role to play in shaping the journal's message and published literature. Moreover, medical education is a fairly new domain of scholarship; however, it is becoming increasingly professionalized and, therefore, can be difficult for those new to the field. Having reviewers who help both our editorial team make decisions (i.e., help us see the blind spots and hidden angles) and the authorship teams develop in their scholarly careers can be of the utmost importance. This can also be valuable for ensuring that the ultimate paper meets the scientific rigor for publication in the journal.

2. “Academic Karma”—Each time you publish a paper, at least two (or sometimes up to six) volunteers have worked alongside you to help your paper become publishable. Between the editors (editor‐in‐chief, decision/associate editor) and one to four peer reviewers, most papers have been screened, reviewed, and/or edited by many others. Therefore, it is important to be mindful of your submission‐to‐review ratio, aiming for 1:1 as opposed to 5:1.

3. Gaining expertise—Altruism is all well and good, but in the competing sea of work, it is worth noting that peer reviewing can also help you become a better scholar. We fully believe that you can learn from others’ mistakes, and therefore by being a reviewer you will see more than just your own foibles and follies. As such, if you aim to improve your writing based on the errors or problems you see in others’ papers or you get a chance to see some new ways of writing or displaying data, then peer review can be a great learning experience. You can also gain insights into new methods, conceptual frameworks, and a new content domain. Thinking critically on a regular basis about others’ work can help you become a better researcher and writer. Reading papers in the final form will preclude you from seeing some of the common slip‐ups that authors make. By reviewing the earlier versions, you can gain insight into how to prevent those in your own work.

4. Gaining insight into the field—Reviewing papers regularly can help you gain a foothold on the conversations and advancements within the field of medical education. While our journal is fairly quick in terms of turnaround time, there is still much to be done to help learn more about the field and to get to know what other researchers might be in your area of interest. You also gain a better understanding of the “behind the scenes” about what editors and reviewers expect from submissions to a journal.

5. Academic advancement—While peer‐reviewed publications remain one of the primary currencies of academic advancement, many promotion and tenure committees will count peer review as part of research or service. Additionally, new editors are often selected from peer reviewers, which can provide additional advancement opportunities. Finally, some journals (like this one) have awards for those who are top peer reviewers, which can be beneficial because they are often seen as national or international awards by your institution.

HOW TO PERFORM A PEER REVIEW OF MEDICAL EDUCATION SCHOLARSHIP

When performing a peer review, we recommend beginning by performing a quick read of the paper in entirety. The first read‐through should focus on understanding the big picture of the paper and developing your overall gestalt. This will help with contextualizing the individual sections and comments during the more in‐depth secondary review. Often, this will also save you from adding comments and asking questions that are answered in a subsequent section.

When drafting the reviewer comments, you should begin with an overall assessment of the paper. You do not need to summarize the paper in your review. Rather, you should highlight the major strengths and weaknesses of the submission and how this adds to the existing literature on the topic.

The review should then address major and minor edits. When proposing a revision, you should include the location within the manuscript (e.g., section or line number) where this corresponds to, your rationale, and the proposed changes when possible. This will help to better guide the author in understanding and addressing the revisions. Make sure the bulk of the comments are addressable. Refrain from suggesting that the authors completely redesign their study because that would not be feasible. In those cases, you should decide if this is a fatal flaw (i.e., the paper is simply not publishable in the journal as a result of this) or merely a limitation that needs to be discussed. If this is a minor flaw, you may want to encourage authors to identify and name the particular flaw within the limitations section. You should avoid focusing on spelling and grammar in the review (as there are dedicated copy editors to address this) and instead focus on clarity of thought and the logical flow of ideas. You should also consider whether the authors have adequately reviewed and situated the study in the context of the existing literature and whether the authors have included a conceptual or theoretical framework to inform the study. If you are a novice within the field and these terms are unfamiliar to you, consider asking a mentor to allow you to co‐review an article with them to gain experience with conceptual or theoretical frameworks. It is also valuable to review the journal's author instructions as well as relevant reporting guidelines. ⁵  While we have not made any specific reporting guideline mandatory at AEM E&T, high‐quality papers will generally have all the elements from the relevant reporting guidelines and reviewing the guidelines can help strengthen your review (see Table 1). ⁵ You should also ensure that there is alignment between the research question, the research methodology, and the outcomes that have been reported. Again, if you are a new reviewer or someone who is reviewing a methodology that is not within your skill set, then make sure to declare your limitations within the comments to the editor section of the review. For a summary of key components and resources for each section, please review Table 2. Table 3 provides examples of poor‐quality and high‐quality peer review comments.

TABLE 1.

Examples of guidelines for each study design

Study design	Reporting guideline	Citation for reporting guideline
Randomized controlled trial	CONSORT	Schulz KF, Altman DG, Moher D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med. 2010;152(11):726‐732. 7
Observational design	STROBE	von Elm E, Altman DG, Egger M, et al. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573‐577. 8
Systematic reviews	PRISMA	Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. PLoS Med. 2021;18(3):e1003583. 9
Scoping reviews	PRISMA‐ScR	Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA‐ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467‐473. 10
Online surveys	CHERRIES	Eysenbach G. Improving the quality of web surveys: the Checklist for Reporting Results of Internet E‐Surveys (CHERRIES). J Med Int Res. 2004;6(3):e34. 11 Erratum concerning guideline: Eysenbach G. Correction: improving the quality of web surveys: the Checklist for Reporting results of internet E‐Surveys (CHERRIES). J Med Int Res. 2012;14(1):e8. 12
Qualitative research	SRQR, COREQ	O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245‐1251. 13 Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32‐item checklist for interviews and focus groups. Int J Qual Health Care. 2007;19(6):349‐357. 14
Quality improvement	SQUIRE 2.0, SQUIRE‐EDU	Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf. 2016;25(12):986‐992. 15 Ogrinc G, Armstrong GE, Dolansky MA, Singh MK, Davies L. SQUIRE‐EDU (Standards for QUality Improvement Reporting Excellence in Education): publication guidelines for educational improvement. Acad Med. 2019;94(10):1461‐1470. 16

TABLE 2.

Key questions you can ask about each section

Section of paper	Questions you may ask	Suggested resources for reviewers and authors for this section
Overall (allegiance to prior work, conceptual frameworks)	Is the work situated well within prior work that has come before? Does the work present as very one‐sided in a controversial area? Does it adequately address skepticism or acknowledge the other side of a controversy? Do the authors articulate their theory, theoretical framework, and/or conceptual framework clearly? (Usually will be woven within introduction, methods, and discussion) Is the study original? Does the study address a need or gap in medical education?	Bordage G. Conceptual frameworks to illuminate and magnify. Med Educ. 2009;43(4):312‐319. 17 Bordage G, Lineberry M, Yudkowsky R. Conceptual frameworks to guide research and development (R&D) in health professions education. Acad Med. 2016;91(12):e2. 18 Varpio L, Paradis E, Uijtdehaage S, Young M. The distinctions between theory, theoretical framework, and conceptual framework. Acad Med. 2020;95(7):989‐994. 19
Title	Does the title contain the key elements and describe the study? Will the title compel the reader to choose to read the article? Does the title adhere to the standards set forth by the journal for length?	Lingard L. Bonfire red titles. Perspect Med Educ. 2016;5(3):179‐181. 20
Abstract	Does the abstract familiarize the readers with the essential background? Does this reflect the key elements of the study methods? Does it report the main findings? Does it synthesize the findings into a conclusion or application?	Ruffell D. Writing a great abstract: tips from an editor. FEBS Lett. 2019;593(2):141‐143. 21
Introduction	Did the authors do their homework to understand the conversations within the literature around their topic? Are there key studies or papers the authors missed? Do they articulate the gap within the literature that they chose to fill and why? Do they clearly identify the need/niche for this study? Do they outline their objectives for the study succinctly?	Lingard L. Joining a conversation: the problem/gap/hook heuristic. Perspect Med Educ. 2015;4(5):252‐253. 22 Lingard L. Writing an effective literature review. Perspect Med Educ. 2018;7(2):133‐135. 23
Methods	Do they describe the study in sufficient detail such that it could be replicated? If a survey or tool is used, do the authors provide validity evidence?	https://www.equator‐network.org/ 5 Sullivan GM. A primer on the validity of assessment instruments. J Grad Med Educ. 2011;3(2):119‐120. 24
Results	Do they describe the participant characteristics? Do they focus on the findings and avoid interpreting the results in this section?	Lang TA, Lang T, Secic M. How to report statistics in medicine: annotated guidelines for authors, editors, and reviewers. ACP Press; 2006. 25
Discussion	Do they situate the study in light of the existing literature? Do the authors explain how their study's findings align or differ with prior works? Do the authors cite highly relevant works? Are they missing any key contributors of a field?	Lingard L. Does your discussion realize its potential? Perspect Med Educ. 2017;6(5):344‐346. 26
Limitations	Do they truly identify the weaknesses and explain the implications? Do they explain the rationale for their study design? Do they discuss how the weaknesses were or were not active concessions that resulted from methodological choices of the research team?	Lingard L. The art of limitations. Perspect Med Educ. 2015;4(3):136‐137. 27
Conclusions	Does this align with the findings of the study? Do they overreach with their conclusion? Is this short and succinct?	N/A
Figures	Do the figures add significantly to the results? Are the figures clear and of appropriate image resolution? Did the authors use color effectively?	Tufte E. The Visual Display of Quantitative Information. Cheshire, CT: Graphics Press; 2001. 28 Duarte N. Data Story: Explain Data and Inspire Action Through story. Ideapress Publishing; 2019. 29
Tables	Is the material presented clearly? Are abbreviations spelled out in the legend?
Appendices	Are the resources necessary for the reader to understand the paper? Does the data or tables make sense? Is the information clear? Can the appendices stand alone?	N/A
Grammar and Style	Do the authors need substantive copy editing before publishing? Does the paper read well and logically flow? Is the voice consistent throughout the paper? Is there an overuse of passive voice?	Lingard L. From semi‐conscious to strategic paragraphing. Perspect Med Educ. 2019;8(2):98‐100. 30 Watling C. The three ‘S’s of editing: story, structure, and style. Perspect Med Educ. 2016;5(5):300‐302. 31 Lingard L. The academic hedge part I: modal tuning in your research writing. Perspect Med Educ. 2020;17:1‐4. 32 Lingard L. Pace, pause & silence: creating emphasis & suspense in your writing. Perspect Med Educ. 2020;9(1):57‐59. 33 Lingard L. Mastering the sentence. Perspect Med Educ. 2017;6(1):51‐53. 34 Lingard L. Avoiding prepositional pile‐up. Perspect Med Educ. 2015;4(4):186‐187. 35 Lingard L. Get control of your commas. Perspect Med Educ. 2016;5(1):39‐41. 36 Lingard L. Enlisting the power of the verb. Perspect Med Educ. 2015;4(2):53‐54. 37 Sullivan GM. Writing education studies for publication. J Grad Med Educ. 2012;4(2):133‐137. 38

TABLE 3.

Examples of a poor‐quality and high‐quality review delineated by section

Section of paper	Example of a poor‐quality review	Example of a high‐quality review
Overall	This was a study evaluating the role of ultrasound in residency training.	Overall, this is a fundamentally sound study. You provide a reasonable conceptual framework and situate the study well within the existing literature. The study addresses a clear gap in the literature. Strengths of the study include the large sample size across multiple institutions and statistical analyses accounting for multiple potential confounders. The study could benefit from clarifications regarding several areas of the methods and a more conservative conclusion. My specific comments and concerns are outlined below.
Introduction	The authors should better copyedit this submission. For example, “it's” should not have an apostrophe.	The manuscript has an appropriate conceptual framework to inform the study. You clearly delineate the importance and relevance of the overall problem. However, it is less clear to me where the gap lies within the literature. I would recommend reviewing the following studies and discussing how the present study builds upon this prior work, while addressing the important lack of literature on faculty‐specific training in this application.
Methods	The authors should have performed a randomized controlled trial.	There are several areas that could benefit from clarification and expanded description. As a qualitative study, it is important to describe both the population selected and the authors’ experiences and potential biases. I would recommend including how you assessed for reflexivity. Please also expand on whether there were efforts made for triangulation of findings. Moreover, you should expand on the specific methodology that informed your approach (e.g., grounded theory, phenomenologic, ethnographic).
Results	The authors should shorten the results by one‐third.	The tables are high‐yield and informative. In several areas (e.g., lines 108–115 and lines 123–130), you repeat data directly from the table. Since it is already in the table, I would recommend condensing and summarizing it to streamline the results. I would also consider removing the p‐values, because you already are presenting 95% confidence intervals, which is a more useful measure of the size of the effect.
Discussion	The discussion provides a good summary of the work.	While your discussion nicely builds upon your results, I feel the rationale for these findings could be strengthened by linking back to the conceptual framework you mentioned in the introduction section. I would also recommend discussing how your study findings aligned with the prior study and where this built upon it (e.g., inclusion of faculty, utilization of both immediate and delayed outcomes). In the last paragraph, I would include a discussion of how this study should inform current practice and future directions for research.

When writing comments to the author, consider the tone of the comments and place yourself in their position. Remember that a large part of peer review is to help enhance the paper and feedback is more likely to be received if received as well intentioned. ⁶ Consider using first‐ and second‐person (e.g., “I found this section overly technical and difficult to follow. You should consider better expanding upon this and using plain language.”) as opposed to third person (e.g., “The authors use an overly technical description which limits readability”). ⁶

The comments to the editor section provides a unique space for you to discuss or expand on major concerns not appropriate for the comments to the author. You should use this space to advocate for or against publication. However, this should remain consistent with your review, and you should avoid writing an overly positive review only to recommend rejection in the editor comments. ⁶ You can also propose whether a commentary may be warranted and suggest potential authors (including yourself) for particularly interesting or controversial manuscripts.

MARKERS OF RIGOR FOR A REVIEW: UNDERSTANDING REVIEWER SCORING

While there are no hard and fast scoring tools for quality of reviews, each journal typically asks the decision editor to score the quality of a review. Mostly, these scores are used to help stratify reviewers for awards. Occasionally, they are used by journal editors to decide whether or not to invite you to review a paper again. At AEM E&T, we disclose the average score each reviewer receives on an annual basis, mostly as a personal benchmark; however, we have occasionally received emails to better understand the scoring system. We currently use a rating system with the following scores: 4 (outstanding), 3 (good), 2 (adequate), and 1 (inadequate). While our rating system is holistic, we have some guiding questions that help us to decide on the quality of reviews:

• Does this help the editor make a decision?

• Does this help the authorship team find their blind spots?

• Is the review cordial, civil, and will it convince the authorship team to revise their paper (regardless of outcome)? ⁶

• Is the review accurate and trustworthy in its assessment of the paper?

• Is it submitted on time?

LIMITATIONS

The peer review process is not perfect, but it is our hope that by assembling several smart and informed individuals with multiple points of view that the reviewed paper can ultimately be improved. ³⁸ , ³⁹ However, as has been noted previously, peer review in and of itself is often insufficient to prevent erroneous or falsified data from being published. ⁴⁰  Moreover, many institutions have asymmetrical reward systems that do not always value the contributions to the “backstage” processes of publishing (e.g., editing, reviewing). ⁴¹ As such, there is room for increased advocacy from chairs and academic leaders to consider assigning academic merit for good scholarly citizenship, such as peer review.

CONCLUSION

Peer review is a critical step for contributing to and advancing the field of medical education. It is also valuable to reviewers for gaining experience and insights into the field. This paper summarized strategies, pearls, and pitfalls for performing a high‐quality review of medical education research. We hope that this provides a valuable guide for both novice and experienced reviewers alike to enhance their approach to peer review.

Gottlieb M, Chan TM, Promes SB. A guide to peer reviewing medical education scholarship: Advice from editors of AEM Education and Training . AEM Educ Train. 2021;5:e10652. 10.1002/aet2.10652