The Reviewer Academy of the Society of Critical Care Medicine: Key Principles and Strategic Plan

Abstract

The Society of Critical Care Medicine (SCCM) Reviewer Academy seeks to train and establish a community of trusted, reliable, and skilled peer reviewers with diverse backgrounds and interests to promote high-quality reviews for each of the SCCM journals. Goals of the Academy include building accessible resources to highlight qualities of excellent manuscript reviews; educating and mentoring a diverse group of healthcare professionals; and establishing and upholding standards for insightful and informative reviews. This manuscript will map the mission of the Reviewer Academy with a succinct summary of the importance of peer review, process of reviewing a manuscript, and the expected ethical standards of reviewers. We will equip readers to target concise, thoughtful feedback as peer reviewers, advance their understanding of the editorial process and inspire readers to integrate medical journalism into diverse professional careers.

The Society of Critical Care Medicine (SCCM) Reviewer Academy seeks to address a gap in skills training for peer reviewers. To create the Reviewer Academy, we have brought together an international, multidisciplinary, and interprofessional group of contributors representing the breadth and scope of the Society. These include physicians from multiple disciplines, allied health professionals, nurses, and the Editors-in-Chief (EIC) of SCCM journals. This initiative seeks to train and establish a community of trusted, reliable, and skilled peer reviewers with diverse backgrounds and interests to promote high-quality reviews for each of the SCCM journals: Critical Care Medicine, Pediatrics Critical Care Medicine, and Critical Care Explorations. Goals of the Academy include building accessible resources to highlight the qualities of excellent manuscript reviews; educating, mentoring, and supporting a diverse group of healthcare professionals who can serve as reviewers; and establishing and upholding standards for insightful and informative reviews.

This manuscript will map the mission of the Reviewer Academy with a succinct summary of the importance of peer review, the process of reviewing a manuscript, and the expected ethical standards of reviewers. We will equip readers to target concise and thoughtful feedback as peer reviewers, advance their understanding of the editorial process, and inspire readers to integrate journalism into diverse professional healthcare careers.

THE DEVELOPMENT AND FRAMEWORK OF THE REVIEWER ACADEMY

Individuals representing a broad multidisciplinary, interprofessional, and diverse set of volunteers from practicing critical care clinicians formed a working group to envision and implement an educational initiative that would come to be called the Reviewer Academy. The group has set out a framework for an enduring product that will be accessible to all individuals interested in participating in the review process. This includes a summary manuscript, an in-person workshop, a structured mentoring pathway, and a series of online educational modules. These resources will be collated for online delivery as a toolkit for new reviewers, as well as those joining the Editorial Boards (EBs) of the SCCM journals. Participation in the Academy is envisioned as voluntary and open to any member of the critical care community.

This manuscript summarizes the core concepts delivered at the 2023 SCCM Critical Care Congress Reviewer Academy Hands on Workshop and builds the framework upon which the Reviewer Academy Online Modules will expand. The online modules will serve as an accessible, and ongoing, opportunity to reinforce the educational goals of the Academy and content will be extended to include topics such as conflict of interest, academic integrity, issues related to bias, geopolitical influences, and resource limitations.

THE PEER-REVIEW PROCESS

Academic journals are 1) accountable to the scientific community, 2) have a duty to maintain high standards of integrity, and 3) should only publish and disseminate valid and appropriate content (1, 2). A thorough editorial and peer-review system helps to ensure this process by promoting public and academic community trust. High-quality peer review assists the Editor in determining scientific validity, originality, and appropriateness for publication, while also helping authors to improve their reporting and to identify and correct any errors of analysis, omission, or interpretation (2, 3). Through this process of article improvement, appropriate translation into clinical practice can be informed (3). Despite the obvious importance of the peer-review process, there is little available to the new reviewer in the form of formal education on what constitutes a quality product. This serves as the basis for the establishment of this initiative.

Attempts to bypass or accelerate the peer-review process, as seen in the current culture of social media, predatory journals, and preprints that lack the checks and balances of peer review, has significant risk to the scientific enterprise and may introduce the risk of patient harm (4). For example, the desire for urgent evidence in an uncertain time, such as the COVID-19 pandemic, led to a proliferation of nonpeer reviewed manuscripts, with the dissemination of false findings that detrimentally changed clinical practice (5-7). In contrast, the peer-review process teaches us to learn to be slow in a hurry and appreciate that hurried or unvetted data may be more damaging for patients than no data at all (8). In recent years, there have been some lessons learned from high-profile retracted and presumed fraudulent publications (9). For reviewers awareness, some key red flags include lack of transparency of data sources with unnamed contributing centers, collaborations amongst apparently disparate coauthors (i.e., those without prior co-authorships or shared institutional affiliation), and apparent overstating of the perceived impact of a study. Genuinely fraudulent intent remains challenging to identify and prove, even with high-quality review (10).

Although the actual process of peer review should not be shrouded in mystery, confidentiality and some degree of opaqueness between reviewers and the article authors is often maintained. The confidentiality of the peer reviewer is not universal, with some journals advocating for an open peer-review process or one in which the reviewer are identified postacceptance; the impact of this system should not lead to a lack of transparency in peer review of an article. Indeed, it is important for every author or potential reviewer to understand both the typical editorial structure within academic journals as well as each step in the process as a manuscript progresses from submission to potential publication.

The editorial team is led by the EIC, a highly experienced clinician or scientist charged with upholding the journal’s mission, adapting to the evolving needs of both the journal and the scientific community, and overseeing the peer-review process (Fig. 1). At most journals, the EIC is supported by a small number of deputy editors and/or associate editors (AEs) with domain expertise. These AEs then are supported by an EB composed of experienced individuals in the scientific community who provide routine peer reviews and guide journal decisions. Members of the EB are often among the first individuals considered for promotion to AE when positions become available.

Figure 1.

Standard editorial board composition. AE = associate editor.

The peer-review process begins with the submission of the manuscript components, generally through web-based portals (Fig. 2). Authors are required to adhere to journal-specific guidelines for manuscript preparation to ensure alignment with the journal’s mission and audience and with reporting and formatting requirements. Although there is enthusiasm for a “universal format” for manuscripts submitted to peer-reviewed journals, this has not been broadly agreed by the scientific community (11). Once submitted by the authors, manuscripts typically undergo a cursory administrative evaluation by a journal managing editor; this review is specifically in accordance with journal submission guidelines and formatting requirements. If a manuscript is returned to the submitting author at this stage, revisions are needed before any content review is undertaken.

Figure 2.

Peer review process for a submitted manuscript.

In the case of the SCCM journals, manuscripts that have passed through administrative review are then considered by the EIC for general appropriateness for the journal. If a manuscript is deemed potentially suitable by the EIC, it is assigned to an AE based on the individual expertise of the AE. The AE assesses the article’s quality and potential for publication, and, if adequate, assigns individual reviewers for an in-depth review. An in-depth review typically involves 2–4 peer reviewers. The number of reviewers and reviews is determined by the EIC and/or AEs, and with the reviewers are selected from among members of the EB or external reviewers. This diverse group of reviewers is selected based on clinical and methodologic expertise, as well as the potential to provide a timely and high-quality review (see below). Peer reviewers provide an evaluation of the manuscript that broadly addresses several questions about a manuscript: 1) Is this important? 2) Is this new? and 3) Is this true? (12, 13) Finally, reviewers offer a general recommendation for article disposition (e.g., reject, major revision, minor revision, accept). The peer reviewer assessment of the manuscript and recommendations are considered by the AE and EIC and a final disposition is rendered for the manuscript. If revisions are suggested and the authors opt to revise and resubmit, the process is repeated. Authors submitting revised manuscripts usually submit at least three files—a revised manuscript with changes marked, a “clean” revised manuscript, and a Response to Reviewers document with point-by-point responses to reviewers’ critiques.

THE REVIEW

Constructive feedback is the cornerstone of good peer review. The most useful (i.e., desirable) feedback addresses both content and style, includes concrete, actionable steps toward improvement, and uses compassionate language that improves the manuscript and does not demean the work or the authorship team. The level of detail, number of comments, and length of the peer-review report are positively correlated with the author perception of constructive feedback, while harsh comments are negatively correlated with constructiveness (14). Optimal peer review utilizes statements that focus on actionable steps without making assumptions regarding why the authorship team formulated a manuscript a certain way. For example, if the reviewer believes that an inappropriate statistical test was used, they could comment that “On page 4, line 36, the authors applied Chi-square to the primary outcome; however, the t-test may be more appropriate given that the outcome is a continuous variable.” Unhelpful feedback includes global statements like “Elementary errors were committed in the statistical analysis making me question the authors’ expertise.” Ideal peer reviewer critiques (Table 1) are both specific and clear but also allow for the possibility that the reviewer may not have all the relevant information. Notable elements of the above example statement include its specificity of comments and softer wording (e.g., “may be more appropriate”).

TABLE 1.

Examples of How to Potentially Transform “Sub-Optimal” Reviewer Comments Into a “Highly Useful” Reviewer Comments

Aim of the Reviewer	Suboptimal Review	Highly Useful Review
Improve study methods used	“The regression analysis utilized in this study is flawed.”	“Although the regression analysis used in this study included some covariates, important known confounders such as severity of illness scores and initial lactate levels were not included. The authors should consider adding important clinical covariates in their multivariable analysis.”
Improve results reporting	“The results of the study were reported in a haphazard manner.”	“The current reporting of the study’s findings lacks organization or structure. The results should be presented according to the primary and secondary aims of the study as outlined by the authors in the introduction.”
Improve discussion section	“The discussion section is too lengthy and should be summarized further.”	“The discussion section is too lengthy and lacks focus to help the reader interpret the results. Some discussion points mentioned (e.g., paragraphs 2 and 3) do not discuss the impact of age and the use of steroids in severe pneumonia in relation to the current study findings. It is important for the authors to compare and contrast the current study’s findings in relation to specific available medical literature/data and offer/postulate explanation of these similarities and/or differences. To ensure that the discussion section remains not too lengthy, the authors should only include discussion on what they considered as the top 3 risk factors of poor outcomes in their cohort and not include a detailed discussion of every risk factor identified in their study.”
Highlight missing key references	“The manuscript lacks references in this topic.”	“Two key publications were not included in the introduction and discussion of this manuscript. Reviewer proposed reference 1 and reference 2 as these are important studies in this area. The authors should examine these two reports and include them in their manuscript appropriately.”
Improve reporting of the abstract	“The abstract lacks pertinent details and is incomplete.”	“The abstract can be improved by incorporating necessary important details in the results section. For example, when comparing between the two groups, instead of just putting the p values, the authors should include the means (standard deviations) of the two groups so that the reader can know the effect size of the comparison.”

Applying a Systematic Process as a Reviewer

Utilizing a systematic process to provide comprehensive feedback can be helpful to reviewers, the journal EB, and the manuscript authors themselves. Before agreeing to conduct a review, potential reviewers should read the provided manuscript abstract both to determine if they are qualified to perform the review based on their own expertise and to identify any potential conflicts of interest (COI). In the scientific community, expertise in certain specialized areas can be limited to a few individuals and thus, it is not uncommon or inappropriate to review the manuscript of a known colleague. Ultimately, however, there should not be any COI that could impact the reviewer’s ability to provide an unbiased review (15).

Once the reviewer accepts the review, performing a thorough review is a time-consuming process that can take many hours to complete (Fig. 2) (16). Many reviewers will complete an initial, brief screen of the manuscript for any major or “fatal” flaws in the study design (15, 17, 18). Fatal flaws include unethical procedures (e.g., failure to obtain consent when needed) or methodologies that may have compromised the results (e.g., failing to account for significant inherent biases), or content that is of minimal importance or relevance to the journal. Following this brief screen, the manuscript can then be read in detail with the reviewer composing high-level commentary on three key manuscript domains: scientific merit, contribution to the literature, and publication recommendation (typically only seen by AE and EIC) (17). Moving then to a more granular assessment of content, the reviewer can determine if each section of the manuscript imparts the appropriate information (i.e., results are solely in the Results section and not scattered in other sections, etc.). Assessment by the reviewer of whether or not the manuscript makes a contribution to the existing knowledge base is helpful for both the authors and editors (18). Finally, the reviewer can compose private comments addressed to the editor to aid in article disposition (17).

Effective reviewers’ commentaries to authors and editors often follow a general outline. After a brief summary of the manuscript’s study aims, design, and results (2–3 sentences), one approach, but certainly not the only approach, divides the remainder of the review into major and minor comments (15, 17). Major comments include methodologic concerns, similar work published in the topic area that has not been acknowledged, and misrepresentation of results. Minor feedback may include points of clarity, missing references, and the incorrect assignment of measurement units to results (18). Another approach is to offer section-by-section input on the manuscript; this formatting for the review can be useful for the authors during revision of the manuscript. Moreover, providing further thoughtful limitations of the manuscript’s conclusions that were not initially expressed by the authors can further aid in both framing the study’s impact in the literature as well as providing future directions. If reviewers identify straightforward solutions to the problems detected, providing these may assist the authors in a revision. Recommendations should not be based merely on the reviewer’s preference and ideally would not require infeasible ancillary studies. Finally and importantly, reviewers should also identify positive attributes to the manuscript, as this aids the AE and EIC to better understand the unique strengths of the study and to support a decision about publication.

Common Pitfalls

A defining element of a low-quality review is the lack of constructive feedback (14). Ineffective reviews lack concrete, actionable discussion of the three key domains (i.e., scientific merit, contribution to the literature, and publication decision). Moreover, tone and style of delivery can have a major negative impact on the quality of a review (17). Two common pitfalls are to be too short or too lengthy. Reviews that are too brief tend to lack structure, offer nonspecific global suggestions, do not review the tables and figures in detail, and make suggestions that can be perceived as rude and irritating. Review of supplemental materials is important and often provides critical information for an optimal review. On the other side of the spectrum, reviews can be overly detailed and appear nitpicky to stylistic considerations that do not necessarily enhance the quality of the work; these unhelpful reviews may also make requests that are well outside the scope of the research question (14, 19, 20). Although reviewers may be content experts or prolific authors themselves, they should not demand that their own work is cited unless absolutely fundamental to the study context. Reviewing for readability, spelling, and grammar can be helpful, but is not strictly necessary, and should take the place of constructive feedback on the more substantive aspects of the work. Manuscripts by nonprimary English-speaking authors may have worthy content but need revision by an English-fluent medically-knowledgeable resource as part of the consideration process. Peer review should never be rude or unhelpful; the primary goal of a review is to provide thoughtful, useful critique to improve the quality of scholarly work, which is a service to the healthcare profession and society at large.

THE REVIEWER

Expectations of a Reviewer

Formal competencies of the peer reviewer have yet to be outlined. However, many characteristics of a good reviewer have been discussed in the literature (21-23). Common attributes include having knowledge of the content area, being able to provide timely feedback, remaining unbiased and ethically sound, and having the ability to provide objective, constructive feedback which will enhance the quality of the manuscript. Reviewers should account for grammatical issues in the manuscript, but should not assume the role of copy editor and attempt to rewrite the work under review. Additionally, reviewers should be able to communicate respectfully, highlighting both strengths and opportunities for improvement (21, 23). It is also imperative for the reviewer to identify and disclose potential COI before accepting an invitation to review. Reviewers who decline should explain why and preferably provide a suitable replacement for the review (22). Effective reviewers should know the journal guidelines for review and understand fully the scope and criteria for acceptance into the journal for which they are providing a review (22). Finally, understanding that reviewers act as advisors to the EIC and AE, reviewers should provide robust feedback on the content rather than deliberating solely on technical aspects of the paper such as spelling or grammar (21, 23).

Role of Peer Review in Professional Development and Academic Promotion

Authors, editors, and journals primarily benefit from peer-review services. However, there is a reciprocal advantage to the reviewer through the opportunity for professional development. First, providing peer-review services allows the reviewer to hone their skills in critically evaluating the scientific literature. The amount of published literature is steadily increasing, on average by 2.6% annually, but more rapidly during the COVID-19 pandemic (24, 25). Peer reviewers have an opportunity to evaluate several publications, serving as the gatekeepers of the research literature. This process adds to their knowledge of a given disease state and the scientific methods used in each manuscript. It is a privilege and honor to be a part of a scientific community, and contributing to that community is part of the moral obligation one has to their profession (17). Peer-review services may lead to future opportunities for individual professional development, like aiding in idea generation for potential research, the potential for collaboration with other authors, potential authorship of editorials/commentaries, and positions on EB. Peer reviewers may also be rewarded for their services by being included in the journal’s annual list of peer reviewers or with complementary access to the journal for a specific period of time (26-28). There is also a role for individual mentorship in this process as junior faculty may be asked to participate in the process by a more senior and experienced reviewer. This, as well as reaching out to specific review partners with expertise, must be acknowledged by the reviewers.

Finally, peer review is an important part of scholarly work and peer-review activities can often be included as part of materials for academic promotion and tenure at some institutions and in some departments. In our experience, targeting 5–8 peer reviews annually can demonstrate sustained contribution to the profession.

Path to the Editorial Boards

Demonstrating a consistent, high-quality pattern of peer review is the best way to progress toward EB membership. Board members are typically identified and chosen from a population of excellent reviewers (29). Excellent reviewers ideally provide cohesive, concise yet comprehensive a guide to the editorial staff and complete reviews in a timely manner. Additionally, the reviewer should be an expert that publishes both in the journal in question as well as in other journals in the field.

Recommending an Editorial

Editorials provide perspective and enhance comprehension of scientific work published, typically in the same issue of the journal. Editorials may synthesize data and compare the paper at hand to standard practice or other previously published work, enhancing scientific validity, thereby enhancing science communication. A few main themes can help guide reviewers when determining whether an editorial is appropriate (Table 2). Reviewers should make editorial recommendations directly to the editor to ensure editorials fit the scope of the journal. It is acceptable for reviewers to suggest themselves or their colleagues as editorial authors if they possess the requisite expertise.

TABLE 2.

Guidance on Whether an Editorial Should Accompany an Article in Publication

Construct	Rationale
Innovation	The manuscript displays a significant amount of novelty or innovation. An editorial can explain and compare with pre-existing views
Controversy or contradictions	Manuscripts containing data that challenge previously accepted notions often benefit from an editorial to synthesize the disconnect
Clinical implications	Manuscripts with significant implications for clinical practice may immediately benefit from an editorial to put the findings in context
Landmark scientific discovery	Scientific discoveries that have research and clinical implications should be celebrated

SPECIAL TOPICS

Promoting Academic Integrity and Addressing Research Misconduct

Publishing research in academic journals is challenging and highly competitive, with a bias toward novel studies, especially those reporting positive findings with large effect sizes (30-33). This may tempt researchers to commit a wide range of ethical transgressions or violations, such as withholding undesirable research results (“cherry-picking”), submitting duplicate publications (self-plagiarism and “salami slicing”), and failing to disclose potential COI including potential personal, professional, or financial gain (34). Although the reviewer is considered and expert in the field, self-citation of work should be viewed as a valuable addition to the work rather than a path to self-promotion. Further violations can include flagrant misconduct such as plagiarism and falsification or fabrication of research findings (2, 32, 35). The culture of biomedical science is based on trust; where end-users of research—healthcare professionals and patients—rely on scientists’ truthfulness and integrity to inform safe and effective clinical practice. Lapses of research integrity and misconduct across the spectrum impact the trustworthiness and reproducibility of research findings, thus affecting the entire culture of science and society (35). Education in the responsible conduct of research is therefore considered to be a fundamental element of research (36).

Appropriate authorship is also an important consideration for academic integrity and can have professional, academic, social, and financial implications. Authorship implies inclusion, responsibility, and accountability for the published work. “Guest” or “gift” authorships confer undeserved benefits, while also holding the person accountable for work in which they did not have a substantial part (37, 38). The integrity of the work is also impacted when deserving researchers are omitted from the authorship, for example, when senior faculty take credit for junior faculty’s work or there is discriminatory exclusion based on cultural, gender, professional discipline, or other biases (32, 39, 40). Many journals now request to state the contributions of each named author, following the International Committee of Medical Journal Editors criteria (38).

Reviewers should also be aware of any explicit (prejudice) or implicit biases in submitted manuscripts, including selective or exclusionary recruitment strategies and inappropriate language and/or definitions for gender, race, and ethnicity, for example (40-42). Inclusive research practice informs better healthcare for marginalized communities, who already may often have worse healthcare outcomes (40). Additionally, EICs and AEs must be aware of the biased selection of reviewers. One recent publication demonstrated women are underrepresented in the peer review process and editors of both genders operate with substantial same-gender preference (43). This observation highlights the need for a diverse pool of knowledgeable reviewers as well as authors.

Journal reviewers have the responsibility not only to evaluate the scientific validity of the manuscript but also to identify any potential research misconduct and inherent biases. Authors can be asked to provide necessary clarification or explanation as part of the manuscript review. Reviewers should report all suspected breaches of research integrity to the EIC or AE.

Predatory Publisher/Journals

In the last decade, there has been a stark increase in the number of predatory publishers and journals, which engage in author-funded publishing of manuscripts with fraudulent, fake, absent, or minimal peer review (44, 45). Predatory journals use the open-access or the author-pay model for their own profit with little to no regard for science, leading to unethical practice and scientific misconduct (44, 45). Various manuscripts have brought attention to predatory journals and have outlined techniques to distinguish predatory from reputable journals (44-47). Undoubtedly, the number of predatory journals in critical care will continue to grow as there are limited international policies to prevent and regulate the creation of new open-access journals. We strongly recommend that reviewers verify information and perform due diligence before agreeing to participate in the peer-review process for a journal. It is important for reviewers to allocate their valuable volunteer time to reviewing quality work for respected journals.

METRICS

Metrics for Journals—Impact Factors

Journal impact factor (IF) provides an objective metric that is intended to convey how important, impactful, or relevant a journal is to its respective field. The IF, developed by Eugene Garfield (48), is calculated based on the number of citations received in one calendar year for articles published in the journal in the preceding 2 years. Given their higher readership, general interest journals have higher IF than those focused on a particular field. The IF metric provides reviewers with one measure of the significance of the journal. Journals without an IF may be newly established (it may take years for an initial IF) or have articles that are below a meaningful threshold for citations; lack of an IF may also be an indication that the journal is in the “predatory” category (see above). IF has received some criticism including the potential for skewed calculations from a few highly cited manuscripts in an otherwise low-quality journal. Regardless, IF provides immediate objective data about a journal as a whole. Unfortunately, authors often misconstrue a journal’s IF with the impact of their own publication; promotion committees are now attuned to this anomaly and will look to how many citations an individual article has received. As such, resources for determining the impact of an individual article are searchable using the Clarivate Web-of-Science database of Journal Citation Reports and SCImago Journal and Country Rank (SJR) (24, 48-51).

There are a growing number of alternatives to journal’s impact factor (25). These include the Eigenfactor score, the Article Influence Score, the Journal Citation Indicator as well as CiteScore, SJR, and Source Normalized Impact per Paper. Each represents variations in determining readership, impact, and quality of the journal’s publishing patterns. Additionally, Altmetric measures the media impact of scientific publications. It includes data on news outlet mention and social media posts on Twitter or Facebook. This is now becoming a valuable gauge of impact across the entire media spectrum, rather than just the scientific community.

Metrics for the Reviewer

Academic institutions and accrediting bodies may require individuals to perform peer reviews for academic promotion and accreditation. To demonstrate sustained contribution through peer review, reviewers should track their activities. Common logging techniques include author curriculum vitae and electronic services like Publons, which record reviewer activity as a measurable research output, and ensure credit is assigned for completion of peer review (24, 26). Additionally, authors can sign up for Open Researcher and Contributor ID (ORCID), which provides a digital identifier that is owned by each author, distinguishing them from other researchers. Peer review activities can be linked to ORCID through Publons, to credit reviewers. The SCCM journals offer reviewers direct communication with Publons for this purpose.

Metrics for the Reviewer Academy

As with any training program, specific metrics should be identified to evaluate the effectiveness of the Reviewer Academy in achieving the desired outcomes. With such a program, the aim was to achieve benefits at two levels: one at the level of the participants and the other at the level of the journals and society. Through this program, we aim to improve the knowledge and skills of the participants in reviewing manuscripts, and we expect this mechanism to have an impact on the number of available reviewers as well as the quality of reviews for the SCCM journals. Furthermore, the long-term impact of the program is expected to result in the progression of graduates from the Reviewer Academy into committee and leadership roles within the SCCM journals and progression to roles on the EBs.

We propose to track the effectiveness of the Reviewer Academy using the New World Kirkpatrick model (NWKM), which is a modified version of the well-known Kirkpatrick model (52). Similar to the original Kirkpatrick model, the NWKM is an outcome-focused model evaluating the outcomes of an educational program at four levels: reaction, learning, behavior, and impact. Table 3 outlines the metrics we will evaluate for each level. The criteria/metrics outlined in the table relate to both the participants of the program and those related to the journals. By utilizing specific criteria to evaluate each level, areas of strength and improvement can be identified, and if necessary, certain elements of the program may be revised.

TABLE 3.

The New World Kirkpatrick Model to Evaluate the Reviewer Academy

NWKM Evaluation Level	Criteria Related to Participants/Program	Criteria Related to the Journals
Level 1: Reaction
The degree to which participants find the training favorable, engaging, and relevant to their jobs	Postcourse evaluation of the content and the instructors.	N/A
Level 2: Learning
The degree to which participants acquire the intended knowledge, skills, attitudes, confidence, and commitment based on their participation in the training	Proportion of participants who completed all requirements of the program	N/A
	Successful completion of assessments for each module
	Successful completion of assignments for manuscript reviews
	Time required for completion of modules
Level 3: Behavior
The degree to which participants apply what they learned during training when they are back on the job	Online access to the Reviewer Academy toolkit	Quality of completed reviews, using journal-specific quality assessment measures
	Number of completed reviews
	Quality of completed reviews (using pre-defined criteria)
Level 4: Results
The degree to which targeted outcomes occurs as a result of the training and the support and accountability package	Reviewer for an SCCM journal	Number of reviewers recruited
	Appointed to the editorial board of an SCCM journal	Number of reviews completed by the trained reviewers
		Quality of the submitted reviews

N/A = not applicable, SCCM = Society of Critical Care Medicine.

DISCUSSION

The science and practice of clinical care rely on observation, imagination, hypothesis generation, experimentation, and repetition. Over time and through an iterative process, the result yields what we regard as scientific truth and help shape our practice to optimize patient care and outcomes. However, any result not widely shared cannot be scaled to make an impact. Healthcare journals serve as primary means for dissemination of reliable information. Peer review is fundamental to the reporting of medical and scientific discoveries. It includes a process through which experts review data and results to judge the veracity, quality, integrity, and clarity of the work. Additionally, peer review should determine whether findings impact patient care or spur further investigation. Although the importance of peer review cannot be understated, formal educational training in such an endeavor has been uncommon.

To standardize training of reviewers and better ensure that reviewers reflect the multidisciplinary, interprofessional, gender, racial, and ethnic diversity of practicing critical care clinicians, SCCM is developing a Reviewer Academy. The Reviewer Academy will consist of resources for training reviewers (such as this summary publication and planned web-based tutorial modules) as well as a mentorship program where experienced reviewers will be paired with junior reviewers so as to mentor and coach over a period of a few months. The Reviewer Academy is currently in formative stages with full implementation planned by 2024.

CONCLUSIONS

In summary, peer review of scientific manuscripts is a learned process contributing to the reviewer’s professional development, improving the author’s knowledge, elevating the quality of work published, and benefiting the overall scientific community. SCCM is advancing this goal through the development and support of the Reviewer Academy.