The JAMIA Student Editorial Board: Peer Review Education in Biomedical Informatics

Peer review is defined as “an evaluation by experts of the quality and pertinence of research or research proposals of other experts in the same field.”¹ Peer review is a key component of the process by which an academic journal retains its quality and scientific rigor. Peer review serves three main purposes: providing a quasi-objective metric for the quality of journal submissions, serving as a mechanism to improve the quality of the content, and providing a mechanism for informing and educating journal contributors (authors). Reviewers help editors to determine whether a manuscript is worthy of publication in the specific journal. Not only do reviewers give their opinion of the merits of the manuscript on a “publishability” scale, they explain to the Editorial Office why they ranked the submission as they did and provide suggestions for how to improve the manuscript to both editors and authors. This process allows a discipline to maintain and improve the quality of its published papers. This process secondarily affects what research is conducted, what methodologies are employed, and what messages are disseminated to professional and lay audiences and, to some extent, rightly or wrongly, provides a mechanism to assist in evaluating the work of the authors who submit publications.

Peer review is at first glance a thankless job for the anonymous reviewer. It takes time and effort to do well, for which the primary reward is the contribution the reviewer makes to the academic and professional communities served by the journal. However, one gains skill and knowledge from participating in the peer-review process. Reviewers learn effective methods for organizing and presenting scientific content from the authors who submit manuscripts. They also learn from the scientific ideas in the manuscripts they review (even though they cannot act on them or discuss them with others until the manuscript is published in the public domain). Reviewers gain knowledge, understanding, and perspective from reading the confidential reviews (distributed by many journals, including JAMIA, among all reviewers) of other peer reviewers who have submitted opinions regarding the same manuscript. Reviewers also learn from observing the process through which authors iteratively respond to (or fail to respond to) critiques in successive revisions of a manuscript.

Despite the longstanding tradition of this approach to manuscript quality control, the medical literature is replete with concerns about the peer-review process. A 1994 survey of authors submitting papers to Journal of Clinical Anesthesia noted that unclear comments, judgmental reviews, discrepant reviews, and disorganized management of the peer review process (including timely return of manuscripts and selection of knowledgeable reviewers) were primary reasons for dissatisfaction.² These concerns are not limited to specific journals; in fact, prescriptions for constructing an acceptable review have been published for many leading journals.^3,4,5

Given the importance of this task, and the widespread need for peer reviewers in all scientific disciplines, it would seem prudent to incorporate formal training about peer review into the training of academic professionals.⁶ In fact, some journals, such as the Annals of Emergency Medicine,⁵ have established training programs for new reviewers. It is of interest, however, that training and guidelines generally are provided after a reviewer has been given peer-review responsibility. Ideally, these practices would be more valuable if provided to new reviewers before they participate in official reviews, rather than through “on-the-job” training.

This year, through the combined efforts of its editor, associate editors, and assistant editor, JAMIA designed and implemented a Student Editorial Board (SEB). This board was formed by selecting outstanding trainees who applied from the National Library of Medicine Training Program sites. Each program was invited to submit the names of up to two trainees to the Assistant Editor. Applicants were all well qualified, and had interests and experience spanning the breadth of biomedical informatics. After discussion and voting, the Associate Editors selected six trainees to form the inaugural Student Editorial Board. The current SEB members are:

• Tricia A. Thornton, BA (Vanderbilt University)

• Michael F. Chiang, MD (Columbia University College of Physicians and Surgeons)

• Peter Mork, MS (University of Washington, Seattle)

• Adam Rothschild, MD (Johns Hopkins University School of Medicine)

• Roderick Y. Son, MS (University of California, Los Angeles)

• Lisa J. Trigg, MN, ARNP (University of Washington, Seattle)

Student Editorial Board Objectives

The SEB is designed to provide hands-on experience to prospective biomedical informatics journal reviewers. We envision that this experience will serve a variety of purposes. First and foremost, the SEB experience will allow prospective reviewers to learn how to construct a well-written, critical, and constructive review. To accomplish this objective, each SEB member is given a monograph written by the JAMIA Editor and Assistant Editor, citing previous work of Stead et al.,⁷ Friedman,⁸ and Aydin.⁹ This monograph describes the elements of a good review, the ethical responsibilities of peer reviewers, and the generally acceptable tone and style of a critical review. Each time an SEB member completes a review, the lessons contained within the monograph are reinforced by a careful critique of each SEB review by the Assistant Editor . In addition, SEB members receive copies of the other JAMIA-assigned reviewers' comments once the editorial process has been completed on the submitted manuscript they reviewed.

The SEB experience also will help SEB members understand how to publish work done at any stage of an informatics project. We will ask SEB members to review all types of materials submitted to JAMIA—including books, original research articles, viewpoint documents, model formulation papers, and case reports. We also will provide SEB members with revised and resubmitted manuscripts, so that they might see how a paper evolves from its first reviewed form all the way through to its final draft.

Finally, we expect SEB members to learn the time management skills necessary to incorporate ad hoc, but time-critical work into their busy schedules, so that they do not find the process as onerous during their early years as independent investigators. Members will have an opportunity to complete six reviews (provided at times when their own publications or presentations should not compete for their attention). They are expected to adhere to the timeline provided to other reviewers.

The SEB Review Process

We will accomplish these objectives by integrating SEB members into the existing JAMIA review process. Each SEB member has provided the JAMIA Assistant Editor his or her areas of interest as a part of the application process. Students are expected to construct a timely, professional review of the manuscript with the same turnaround as regular reviewers. Student reviews are returned to the JAMIA office, where the Assistant Editor reviews them, provides constructive feedback, and accepts revisions by the SEB member. Once approved, the JAMIA Editor includes the student reviews as a part of the overall critique of the manuscript returned to authors. A label informs the author that the review is from an anonymous SEB member. The student reviews are intended to complement the reviews by JAMIA Editorial Board members and outside reviewers that will continue to occur. Students also have an opportunity to turn back manuscripts (not review them) when the timing of a JAMIA request is not convenient for them.

Student Editorial Board members—like other editorial board members—are not financially compensated for their work. The major benefits to SEB members are education about the publication process and the construction of a useful review, as well as the notoriety from being selected. Student Editorial Board members also are recognized through a listing in JAMIA on the same page/e-location where the Editorial Board membership is listed.

Thus far, the SEB process has proceeded smoothly. Our six SEB members recently completed their first set of reviews. A careful critique of these reviews reinforced the potential impact that the SEB will have on the written critiquing skills of its members. For example, critiques of these reviews were able to use specific textual examples to teach:

• The fine line between providing constructive insight to the author and clouding insight with condescending statements

• The importance of including an evaluation of prior work in the introduction of an original research manuscript

• The impact of poor author writing style on the ability of a reviewer to critique a potentially groundbreaking research project

• How the author's receptivity toward a review may be affected by the style of the review (particularly when the review addresses the author as “you”)

Although it is too early to determine whether this process makes a difference in either the quality of our reviews or the skills of potential reviewers, comments from current SEB members suggest that the process is both enjoyable and instructive. They have been uniformly responsive to critiques of their reviews, and have made revisions that have generally produced a much more effective review. We intend to evaluate the effects of the SEB process more carefully over the year.

We will soon expand the SEB from six to 12 members, with a plan to provide each SEB member with at least six reviews during his or her two-year tenure on the Board. The next round of applications will extend beyond trainees in U.S. training programs. In the meantime, we will evolve the SEB to meet the needs of our initial cohort of members, while striving to keep the process both fun and educational.