Improving the peer review skills of young rheumatologists and researchers in rheumatology: the EMEUNET Peer Review Mentoring Program

Abstract

Although peer review plays a central role in the maintenance of high standards in scientific research, training of reviewing skills is not included in the common education programmes. The Emerging EULAR (European League Against Rheumatism) Network (EMEUNET) developed a programme to address this unmet need. The EMEUNET Peer Review Mentoring Program for Rheumatology Journals promotes a systematic training of reviewing skills by engaging mentees in a ‘real world’ peer review experience supervised by experienced mentors with support from rheumatology journals. This viewpoint provides an overview of this initiative and its outcomes, and discusses its potential limitations. Over 4 years, 18 mentors and 86 mentees have participated. Among the 33 participants who have completed the programme, 13 (39.3%) have become independent reviewers for Annals of the Rheumatic Diseases after the training. This programme has been recently evaluated by a survey and qualitative interviews, revealing a high interest in this initiative. The main strengths (involvement of a top journal and learning opportunities) and weaknesses of the programme (limited number of places and insufficient dissemination) were identified. Overall, this programme represents an innovative and successful approach to peer review training. Continuous evaluation and improvement are key to its functioning. The EMEUNET Peer Review Mentoring Program may be used as a reference for peer review training in areas outside rheumatology.

Introduction

Peer review is the process by which research findings are evaluated for quality, significance and originality,¹ and it is the basis of the scholarly publishing system and the maintenance of high standards in research.

Peer review represents an important step for professional development. However, access to peer review is challenging for early-career researchers, due to limited exposure as authors and² and the need of specific skills,¹ which require instruction and practice.

A recent survey on educational needs from young European clinicians and researchers in the field of rheumatology revealed a strong interest in reviewing skills by young professionals.³ However, this training is not included in formal educational programmes. Although a number of journals provide resources to address this issue (from checklists and instructional materials, to brief workshops),^4–7 these initiatives are insufficient to gain robust reviewing skills.

Taking the above into account, the Emerging EULAR (European League Against Rheumatism) Network (EMEUNET) started in 2012 a collaboration with the top-leading journal in rheumatology, the Annals of the Rheumatic Diseases (ARD), aimed to enhance the peer reviewing skills of young rheumatologists and researchers by means of supervised manuscript reviewing. This initiative, now named the EMEUNET Peer Review Mentoring Program for Rheumatology Journals, has been under continuous evaluation and improvement. Since 2017, it has been expanded to a second rheumatology journal: RMD Open.

The objective of the present viewpoint is (1) to outline the characteristics of the EMEUNET Peer Review Mentoring Program, (2) to discuss its potential challenges and directions to further improvement, and (3) to provide guidance for the implementation of similar programmes.

The EMEUNET Peer Review Mentoring Program: overall structure

The EMEUNET Peer Review Mentoring Program for Rheumatology Journals is a peer mentoring programme targeting young rheumatologists and researchers in order to develop and improve their reviewing skills.

The EMEUNET Peer Review Mentoring Program is a joint initiative that requires the coordinated work of all parties involved: organisational core (EMEUNET Peer Mentoring Subgroup and Steering Committee), the journals’ editorial offices (ARD and RMD Open), senior reviewers (mentors) and junior reviewers (mentees). The mentors are established researchers in the field of rheumatology, who are frequently invited to perform reviews, and chosen to cover a broad range of research topics. Based on these criteria, the mentors are selected in collaboration with the editorial office. Thereafter, the mentors are asked to provide keywords that best reflect their research interests.

The mentees are selected in an open call based on a competitive process. Applications are formally scored, including the assessment of the research background and motivation for the programme. Some basic academic experience is valued but not mandatory. The candidates are assigned to the mentors (five mentees/mentor) based on their preferences and according to their scores. As confidential information is handled, mentees are required to sign a confidentiality agreement.

This programme provides training under ‘real world’ peer review, so its procedures follow the same steps as common peer review (figure 1). Once mentors receive a manuscript from the journal editor, they circulate it among their mentees indicating the deadline to complete the review. Thereafter, mentees work independently in their reviews. After this step, the mentors assess the reviews performed by the mentees and provide educative feedback (related to the structure of the reviews, scientific adequacy, decision recommended and any other aspect relevant for the integrity of the peer review process), as well as the final review submitted to the journal, as a reference for the expected quality level. This can be done as a group discussion (eg, teleconference) and/or independently for each mentee (via email). Importantly, it remains the responsibility of the mentor to submit the definitive review before the deadline set by the journal. The mentor can decide to either include or not any comments provided by the mentees in the definitive review. Finally, the mentors can share with their groups the final decision letter sent from the editorial office.

Figure 1.

Flow chart of the steps in the EMEUNET Peer Review Mentoring Program. The procedures followed in the programme are indicated in the flow chart, including the parts involved in each step, as well as the description of the tasks to be completed in each step. EMEUNET, Emerging EULAR (European League Against Rheumatism) Network.

It is important to note that throughout the whole training, the mentees do not have direct contact with the editorial office, and the training does not interfere with the normal editorial process.

The supervised peer review is repeated four to five times within a year by the whole group of mentees. When the mentor decides that the reviewing skills of the mentee have reached the standard to independently conduct a good quality review, the training of the mentee finishes and a certificate of completion is issued. Of note, not all mentees are able to develop sufficient skills to qualify to become an independent reviewer.

During the whole training period, the members of the EMEUNET Peer Mentoring Subgroup are in close contact with all parties involved to ensure adequate progress and to timely solve any issue.

Outcomes of the EMEUNET Peer Review Mentoring Program

The main outcome of the programme was defined as the number of mentees who performed independent reviews for the corresponding journal after completion of their training. The official status as reviewers of previous participants was confirmed by the ARD Editorial Office. The number of mentees becoming official reviewers for ARD is naturally an underestimation of the true outcome of the project as mentees might not have become reviewers for ARD, but for other journals.

Since its inception in 2012, four consecutive editions of the programme have been organised. Over this period, 18 senior reviewers (14 men) from 10 countries have served as mentors for 86 mentees (53 men) from 25 countries. Mentees enrolled in the current and ongoing edition (n=34) and participants from the previous edition still in training (n=19; expected date of completion: 2017) were excluded from the analysis of the outcome as they were still under training. Among the 33 participants who had completed the training period at the time of writing of this manuscript, 13 (39.3%) have already performed independent reviews for ARD. Only 5 (15.1%) mentees had acted as ARD reviewers before starting the training.

Evaluation of the programme: assessment of strengths and limitations

Such an initiative requires a constant, close monitoring to ensure its proper functioning and to adjust to the needs of the target population. Thus, an online survey to gain insight into the strengths and flaws of this programme was conducted. The survey was defined by consensus within the EMEUNET Peer Mentoring Subgroup and Steering Committee. The survey was sent by email to all EMEUNET members (1291 members in September 2016) and advertised in social media platforms. The survey was open between 22 September and 13 October 2016.

From a total of 179 anonymised responses collected, 163 (91%) were complete and analysed. The majority of respondents were within the age groups of 31–35 (41%) and 36–40 (37%) years old, as expected for the EMEUNET community. Regarding the position of the respondents, a balanced distribution was observed among fellows in clinical training (23%), consultants (26%), PhD students (20%) and postdocs (16%). Of note, 13% of the respondents reported a combined position. Most of the respondents had previously acted as a reviewer, either in rheumatology journals (46%) or in other journals (17%).

Overall, the initiative was considered as useful by most of the respondents (156; 96%), and a high rate of interest to apply in future editions was observed (96/117; 82%).

When participants were asked about how they had heard about this initiative, the EMEUNET newsletter was the most reported (45%), followed by EMEUNET emails (37%), colleagues (12%) and the EMEUNET website (9%). EMEUNET profiles in social media (Facebook and Twitter, launched in 2014) were less reported (5% and 2%, respectively).

Treatment, immunology and epidemiology ranked the top choices in the respondents’ research interests (figure 2A). Rheumatoid arthritis was considered the disease of highest interest, followed by spondyloarthritis/psoriatic arthritis and systemic lupus erythematosus (figure 2B).

Figure 2.

Main fields of interest. Summary of the main areas (A) and diseases (B) of interest registered in our survey (n=163) (multiple responses were allowed). The vertical axis represents the number of responses collected for each item. Myos, myositis; OA, osteoarthritis; RA, rheumatoid arthritis; SLE, systemic lupus erythematosus; SpA/PsA, spondyloarthritis/psoriatic arthritis; SSc, systemic sclerosis; Vasc, vasculitis.

Finally, the analysis of the weaknesses and strengths of the programme, according to the results of the survey, is summarised in table 1. The limited number of available places and the strict eligibility criteria were seen as the main weaknesses. A quarter of the respondents found no weaknesses in the programme. Concerning the strengths, the participants emphasised the involvement of leading journals and reviewers, together with the valuable learning experience. Results were similar when participants were stratified by their current position (data not shown). No differences in strengths and weaknesses perceived were observed when PhD students were compared with postdocs, or fellows compared with consultants, either.

Table 1.

Analysis of the main weaknesses and strengths of the EMEUNET Peer Review Mentoring Program according to the results from the survey*

	Responses, n (%) (N†=163)
Main weaknesses
Dissemination not enough	49 (30)
Few places available	59 (36)
Too strict eligibility requirements	27 (17)
Strict deadlines to meet	11 (7)
Area of interest not covered	8 (5)
Too time-consuming	20 (12)
Nothing from the above	41(25)
Other	18 (11)
Main strengths
Highly reputed senior reviewers	104 (64)
Involvement of a top journal	122 (75)
Opportunities to scientific discussions	80 (49)
Valuable learning experience	109 (67)
Opportunity to become an independent reviewer	81 (50)
Expanding the research network	56 (34)
Nothing from the above	5 (3)
Other	6 (4)

Multiple responses were allowed.

*The survey was conducted among EMEUNET members, including previous mentees of the programme and potential future applicants.

†The number of respondents was used as the denominator to calculate each percentage.

EMEUNET, Emerging EULAR  (European League Against Rheumatism) Network.

Evaluation of the programme: qualitative  assessments

In order to gather additional insights from previous participants, semistructured telephone interviews were conducted by one of the authors (PP) (interview guide available on request). Ten out of 17 randomly invited previous mentees participated.

All interviewees had at least some experience of reviewing for scientific journals with medium impact factor prior to the start of the programme, and several had done so together with a senior colleague. The expectations from the programme were to gain more systematic skills in how to perform a good review and what is expected from a reviewer. Most participants noted that their reviewing skills have improved in terms of structure of the review and confidence about their role. Few respondents also mentioned the additional positive effects of the programme, such as expanding the professional and social network. Having seen multiple reviews of the same manuscript from different angles by several people was recognised to have a positive learning effect.

The programme workload was perceived as reasonable, although two interviewees suggested that a longer duration would have been desirable. Respondents expressed other concerns such as an insufficient number of manuscripts being reviewed or a poor overlap of expertise with the mentor. These issues were effectively addressed in the later editions. Despite these concerns, all respondents remained positive about the programme.

Suggestions to improve the programme included (1) promotion of ‘face-to-face’ meetings between mentor and mentees, (2) an introductory video-conference meeting, to discuss a ‘practice’ manuscript and literature on peer review, and (3) organising webinars with experienced reviewers. Two participants expressed a wish to have a formal examination or assessment to formalise the outcomes of the programme and increase quality and commitment.

Some of these ideas have already been implemented. In fact, at EULAR Annual Conference 2017, meetings were organised between the mentors and the mentees to mark the start of the new edition of the programme. The organisation of these meetings was appreciated by all the participants.

Strengths of the programme

Early-career researchers represent a source of qualified and highly motivated potential reviewers. However, their access to the peer review system is challenging. On the other hand, there is a continuous growing number of manuscripts submitted to peer review worldwide.^{8 9} Therefore, the number of manuscripts requiring a review may overburden the available qualified referees and present a threat to timeliness and quality of their evaluations,⁸ hence emphasising the increasing need for new well-trained reviewers. Then, the implementation of peer review training initiatives may be useful in this scenario. The EMEUNET Peer Review Mentoring Program is, to the best of our knowledge, a pioneer initiative aiming to address this unmet need.

The evaluation of the programme has confirmed that the programme achieves its goals. First, a significant proportion of previous participants have become independent reviewers for ARD. Of note, these were young researchers (<40 years), with limited review experience, who became reviewers in the top journal of the field of rheumatology. Finally, the results of the survey and the qualitative interviews revealed positive feedback about this programme. Therefore, the facts presented pave the ground for the implementation of similar initiatives in other areas. This has been one of the motivations to write such a viewpoint, so that our pioneering programme is described in the literature and others can make use of it, and even contribute to its improvement.

Peer review is a very demanding task.¹ While educational resources to perform effective peer reviewing are provided by several journals,^{4 10 11} mentorship could contribute to the development of reviewing skills by trainees, similar to the other aspects of academic work.^{12 13} The innovative programme herein described, enabling systematic training of reviewing skills by engaging mentees in a ‘real world’ peer review experience, represents a win-win scenario for all parties involved. Mentees benefit from (1) easier access to peer review, (2) participating in scientific discussions and (3) getting a valuable learning opportunity from leading experts and colleagues. Mentors also benefit, as the participation of mentees provides different perspectives to the peer review and enables constructive criticism and the possibility to rethink their approach. Moreover, by transferring their knowledge and skills, mentors help in maintaining high standards of publications while decreasing their workload. Finally, journals benefit in the short and medium term from the availability of an increasing number of well-trained and highly motivated reviewers. Furthermore, it may represent an additional publicity for journals.

Lessons learnt

An ambitious initiative as this programme faces important challenges that have been identified and are being addressed in a process of a continuous assessment and improvement. Based on our experience, we propose a list of recommendations that can help in the implementation of similar programmes in other areas (table 2). Most of these difficulties are related to communication issues. It is important to promote fluent communication among all parties involved. Due to the characteristics of the programme, long periods without any updates from the programme may have a negative impact on the perception of the mentees. A standardised protocol for periodic communication for a tight monitoring of the groups is useful in this aspect.

Table 2.

Recommendations to consider for the implementation of a peer review mentoring programme*

Area	Recommendations
Dissemination	Disseminate the programme as much as possible using different, complementary ways: emails, efficient use of social media platforms, advertising in websites, national societies and young organisations.
Communication	Promote fluent communication among all parties involved, mainly between the critical interactions: editorial office–mentors, organisational core–mentors, and mentors–mentees. Actively liaise with the editorial technical staff, in addition to and in collaboration with the editor. Facilitate the personal acquaintance between mentor and mentees. Keep mentees updated on the progress of the programme. Develop standardised templates including all the information requested from the editorial offices to submit a ‘real world’ manuscript review to facilitate the work of mentors and mentees.
Editor/Editorial office	The editor should prioritise the assignment of reviews to the mentors involved in the programme. The editor should also have a focus on assigning reviews to mentees after their approval as qualified reviewers.
Mentors	Consider mentors who are frequent reviewers (high number of manuscripts per year) for the journal to meet the aims of the training. Clearly explain the aims, duration and their expected roles to the potential mentors so that they can have a clear idea of the commitment related to this programme.
Mentees	If applications outnumber places available only in some groups (allocated according to area of interest), offer mentees with high scores the possibility to join other groups, provided that research interests are compatible. Emphasise at the beginning of the training the obligation to stick to deadlines proposed by mentors.
Evaluation of the initiative	Perform close monitoring on a periodic basis to evaluate the progress of the different groups. Give priority to the periodical, objective evaluation by the organisational core of the initiative. Perform a careful analysis of the findings of the evaluation and use conclusions to tailor the programme to the needs of the target population.

*These recommendations were elaborated based on the experience of the working group members involved in the organisation and management of this programme.

Bearing in mind the above-mentioned continuous assessment principle, the results from the survey and from the interviews were used thereafter to launch the latest edition of the programme, where the areas of interest, mentors and number of places were tailored accordingly.

Notably, the results from our survey did not show relevant differences between participants on their assessment of the programme, regarding their education background, status of clinical training or current position. This suggests that a wide range of professionals can be enrolled within the same initiative, provided that a broad range of research topics is covered. Heterogeneity may be seen as an advantage, since it allows different perspectives to be shared in group discussions, thus enriching the scientific debate.

Concluding remarks

We described an innovative approach developed by EMEUNET to address the long-lasting unmet need of peer review training for young researchers. The success of the programme highlights the key relevance of peer mentoring and may hopefully inspire similar future initiatives in other areas outside rheumatology. European societies of different specialties, or ideally even their young organisations, can use this programme as a starting point for a fruitful mentoring initiative, which can contribute to improving peer reviewing skills of young clinicians and researchers.