Perspectives on involvement in the peer-review process: surveys of patient and public reviewers at two journals

Abstract

Objective

In 2014/2015, The BMJ and Research Involvement and Engagement (RIE) became the first journals to routinely include patients and the public in the peer review process of journal articles. This survey explores the perspectives and early experiences of these reviewers.

Design

A cross-sectional survey.

Setting and participants

Patient and public reviewers for The BMJ and RIE who have been invited to review.

Results

The response rate was 69% (157/227) for those who had previously reviewed and 31% (67/217) for those who had not yet reviewed. Reviewers described being motivated to review by the opportunity to include the patient voice in the research process, influence the quality of the biomedical literature and ensure it meets the needs of patients. Of the 157 who had reviewed, 127 (81%) would recommend being a reviewer to other patients and carers. 144 (92%) thought more journals should adopt patient and public review. Few reviewers (16/224, 7%) reported concerns about doing open review. Annual acknowledgement on the journals’ websites was welcomed as was free access to journal information. Participants were keen to have access to more online resources and training to improve their reviewing skills. Suggestions on how to improve the reviewing experience included: allowing more time to review; better and more frequent communication; a more user-friendly process; improving guidance on how to review including videos; improving the matching of papers to reviewers’ experience; providing more varied sample reviews and brief feedback on the usefulness of reviews; developing a sense of community among reviewers; and publicising of the contribution that patient and public review brings.

Conclusions

Patient and public reviewers shared practical ideas to improve the reviewing experience and these will be reviewed to enhance the guidance and support given to them.

Strengths and limitations of this study.

• The involvement of patients and the public in the peerreview process of journal articles is a novel initiative.

• This is the first study to evaluate the experience of patient and public reviewers.

• We explore the perspectives of patient and public reviewers at two journals with different publishers.

• We involved patients and carers in the design, conduct and dissemination of this study.

• We report only on the early experiences of patient and public reviewers, and this may change over time.

Introduction

Peer review has been central to the scientific process for over 300 years.^{1 2} It plays a key role in funding and publication decisions and in improving the accuracy and clarity of reported published research. Peer review contributes to scientific integrity by helping identify robust research and weeding out methodologically weak studies. While the value and effectiveness of peer review continues to be debated vigorously, it is still regarded as an essential quality assurance mechanism for funding organisations and journals, and it is the only widely accepted method for research validation.^{1 3–6} It is therefore a deeply entrenched process in international funding organisations and high-quality journals. By playing a key part in determining what research gets funded and published, peer review has a pivotal role in determining the treatment and interventions patients receive and how medicine is practised.

Despite its pivotal role, peer review is criticised on many fronts.^{1 7} Traditionally, it has been a closed process, and the majority of journals still practice blinded review. It is only recently that there have been calls to include the views of all relevant stakeholders, including patients and the public. The role of patients and the public is also being stepped up as coproducers of health and social care research not least because their involvement has potential to increase the quality and value of the research and reduce waste.⁸ Many grant funding organisations now incorporate members of the public in their review processes or panels, for example, the UK’s National Institute for Health Research (NIHR) and the Patient-Centered Outcomes Research Institute (PCORI) in the USA. By contrast, biomedical journals still do very little to involve patients and the public in their peer-review processes. To address this imbalance, The BMJ and Research Involvement and Engagement (RIE) journals have adopted innovative strategies aimed at coproducing their content with patients and the public, and both journals have patient editors.^9–12 The systematic incorporation of patient and public review of research manuscripts alongside traditional scientific peer review is a key component of these journal patient partnership strategies. Patient and public involvement in peer review of research manuscripts represents a landmark shift in the concept of peer review at journals, reflecting a desire to ensure research is appropriate and relevant to end users. These journals also practise open peer review where the authors and reviewers know each other’s identities, and reviews of published articles are published online beside the article for readers.

While some patient and public reviewers are able to comment on research methodology, their strength collectively is in addressing the relevance and importance of the research to patients and carers and whether the treatment or intervention studied or guidance given is practicable and acceptable to patients. They can also identify challenges patients and carers face; whether the outcomes measured in the studies described are important to patients and carers and if other outcomes or issues should have been considered. They can also suggest ways to strengthen papers to make them more useful for shared decision making. Contributions will vary according to the background, skills and experience of the patient and public reviewer, as it does with academic reviewers. Some patients and carers may comment on the relevance of a study for a particular group, while those with academic training and research knowledge might comment on methodology. Others may focus on the clarity of the reported research and its interpretation to a lay audience. All these contributions and others have the potential to enhance the quality of the research and its uptake in practice.

Academic researchers are encouraged and motivated to publish their research as part of career advancement in a ‘publish or perish’ environment,¹³ and reciprocal peer reviewing of articles is considered to be a professional duty.^14–17 In marked contrast, patient and public reviewers have no professional implicit or explicit obligations to review. With open access publishing and non-blinded review, the responsibility of reviewing papers may be compounded by the challenge of being identified as a person with an illness or disability. Both The BMJ and RIE offer guidance for patient and public reviewers, developed with patient editors, but these have not been informed by the experience of patient and public reviewers and might not provide the level of support some need. In this collaborative survey carried out by The BMJ and RIE, with embedded patient involvement, we explore the early experiences of our patient and public reviewers. Reviewers were asked what motivates them to review research articles, if and why they have declined to review, their confidence in reviewing, their perceptions of open review, their satisfaction with the process, how they would like to be acknowledged for their contributions and their views on how the process could be improved. This knowledge will be used to inform evidence-based guidance for patient and public reviewers.

Methods

Sampling

Patients and public reviewers who were registered as reviewers for The BMJ or RIE and were invited to review for one of the journals at least once between January 2015 and May 2017 were eligible to participate. The reviewer pool was split into two samples based on review history: those who had already completed a review and those who had been invited but not yet completed a review (by April 2017). This was so that questions that were not relevant to those who had not yet reviewed could be excluded.

Reviewers’ contact details were extracted from the relevant manuscript tracking systems and uploaded to SurveyMonkey. We excluded those who were registered as reviewers but whom had not been invited to review and those who have previously asked not to be contacted by The BMJ for marketing purposes.

Questionnaire development and pretesting

Survey items were developed by the research team, which included patient and public reviewers from both participating journals (AD, RH and JE) and The BMJ and RIE patient editors (TR, AP and RS). The questionnaires were pretested before launching the survey with a further four BMJ patient and public reviewers to check the survey content was appropriate and addressed issues of importance to patient and public reviewers and that questions were not ambiguous.

Survey administration

Reviewers were sent a personalised email invitation to complete a survey (online supplementary appendix 1) administered by SurveyMonkey. There were two versions of the questionnaire: those who had already completed a review received all the questions and those who had not received only a subset. Participants were informed that they could choose not to take part and withdraw at any time; survey completion was used to indicate consent to participate. They were told that the results of the survey would be submitted for publication and that we may choose to quote from their responses but individuals would not be identified. All survey data were treated confidentially, and only SS and EF saw the individual responses for their own journals. All data were deidentified before sharing with the wider research team. Non-respondents were sent up to two reminders to take part at 2 and 4 weeks after the initial mailing.

Patient involvement

Patient and public reviewers were involved in multiple stages of the research. Three patient and public reviewers (JE, RH and AD) are coauthors on this paper as they were core members of the research team and substantially contributed to the development of the questionnaire, refinement of the questionnaire, interpretation of results and manuscript revision. These three patient partners coproduced an abstract and poster of the findings for the ‘NIHR-INVOLVE’ Conference 2017¹⁸ to aid dissemination of the research to academic and industry research professionals, patients, policy makers, editors and advocacy groups. We pretested the questionnaire with a further four BMJ patient and public reviewers (outside of the research team) and refined the questions based on their feedback. Three other members of the research team are patient editors with long-term conditions committed to involving patients in research. More details can be found in the completed GRIPP2 reporting checklist for improving the reporting of patient and public involvement in research (see online supplementary material 2).

Results

Response rate

One hundred and twenty-two (74%) of the 164 BMJ and 35 (56%) of the 63 RIE reviewers who had previously reviewed responded and 51 (38%) of the 133 BMJ and 16 (19%) of the 84 RIE reviewers who had not yet reviewed responded. We pooled the results for the overlapping content and report for both samples combined where relevant.

Respondent characteristics

Table 1 shows the respondent characteristics for reviewers completing the surveys. The majority of reviewers lived in the UK. Reviewers who had reviewed had completed a median of two reviews. Overall reviewers who had reviewed reported spending a median of 5 days a month on other patient involvement or engagement activities and those who had not yet reviewed reported this as 7 days a month. However, there were differences between the journals with RIE reviewers reporting more days spent doing these activities than BMJ reviewers.

Table 1.

Respondent characteristics

Had already reviewed	BMJ (n=122)	RIE (n=35)	All (n=157)
Median (LQ, UQ) number of reviews completed	2 (1, 4)	2 (1, 3)	2 (1, 4)
Country of residence, n (%)
UK	63 (52)	28 (80)	91 (58)
Europe (excluding UK)	20 (16)	3 (9)	23 (15)
North America	36 (30)	2 (6)	38 (24)
Australia/New Zealand	3 (2)	2 (6)	5 (3)
Median (LQ, UQ) number of days per month spent on patient involvement and engagement activities	5 (2, 11)	11 (3, 24)	5 (2, 15)

Had not already reviewed	BMJ (n=51)	RIE (n=16)	All (n=67)
Country of residence, n (%)
UK	22 (43)	5 (31)	27 (40)
Europe (excluding UK)	10 (20)	10 (63)	20 (30)
North America	14 (27)	0 (0)	14 (21)
Australia/New Zealand	0 (0)	0 (0)	0 (0)
Asia	1 (2)	0 (0)	1 (1)
Median (LQ, UQ) number of days per month spent on patient involvement and engagement activities	6 (2, 14)	21 (10, 26)	7 (3, 21)

Percentages do not sum to 100% because of missing data.

LQ, lower quartile; RIE, Research Involvement and Engagement; UQ, upper quartile.

Motivation to review

Patient and public reviewers who had already reviewed described in free-text comments how they are motivated to review by the opportunity to include the patient voice in the research process; the opportunity to influence the quality of the literature and to help ensure it meets the needs of patients and is understandable by them; helping make research more relevant and useful to researchers, clinicians and patients; the intellectual challenge; and learning more about research on topics of interest to them.

Data are presented as the mean (SD) of responses on a 5-point Likert scale for the n=157 who had reviewed. The factor rated most highly in importance to the decision to accept a paper to review was the opportunity to add the patient or carer perspective (4.49 (0.71)) (figure 1), but more than half reported the following as 'very' or 'extremely important': the contribution of the paper to the subject area (3.8 (0.9)), relevance of the topic to own experience/interests (3.8, (1.0)), desire to keep up-to-date on current research (3.6 (1.0)) and opportunity to learn something new from the paper (3.5 (1.0)).

Figure 1.

Importance of specific factors in the decision to accept to review (n=157 who had previously reviewed).

Reasons for declining

Data are presented as the mean (SD) of responses on a 5-point Likert scale. Across both samples, 101 reviewers reported that they had declined a request to review at some stage. The most highly rated factors important in the decision to decline to review was conflict with other workload (3.7 (1.4)), a tight deadline for completing the review (3.3 (1.4)) and conflicts with family commitments or responsibilities (2.8 (1.6)), figure 2. Dislike of the open peer-review process (1.1 (0.4)), having to use the online manuscript review system (1.3 (0.7)) and lack of formal recognition of reviewer contribution (1.3 (0.7)) were the least important factors. Other reasons reviewers mentioned for declining to review included a lack of experience or interest in the topic, the irrelevance of the topic for patient and public review in general and not understanding the paper.

Figure 2.

Importance of specific factors in the decision to decline to review (n=101 who had previously declined an invitation to review).

Confidence in reviewing

Of the 157 who had reviewed, 43 (27%) reported feeling either 'not at all' or only 'slightly' confident when doing their first review, but this proportion fell for those who had done more than one review to 11/141 (8%) when doing their last review. Six (4%) reported they were 'extremely confident' when doing their first review and 19 (13%) when doing their last review.

Reviewers explained that the following had helped to improve their confidence: practice and gaining familiarity with the process; seeing other peer reviewers’ comments on the paper; having access to sample reviews; receiving feedback from editors; knowing their review was valued; being asked to review again; improved journal guidance and provision of frequently asked questions; seeing their own comments in letters to authors requesting revisions; and generally reading more peer reviewed articles and reviewing for other organisations simultaneously.

"It takes time in my estimation to get into a place where as a patient I am confident reviewing the work of academic scholars. At first I took the stance of ‘who am I to challenge them,’ but recognized after the first review I am the patient with the experience and perspective that can help to inform the practice of research."

"Initially I was worried that my lack of in-depth knowledge of statistical methods would make my comments seem petty or too simplistic. Yet when I read expert reviewers comments I saw that they picked up on some of my points which gave me the confidence to stick to my guns and write anything I felt was important to say."

Perceptions of open review process

At both journals, all reviewers’ named comments are published alongside the article on the journals’ websites. When asked if they had concerns about open review, 181/224 (81%) said no, 16 (5%) said yes and 25 (7%) were unsure. Of the 16 who said yes, 15 had already reviewed a manuscript. Reported concerns included some which were generic to the concept of open review (for example worries about being completely critical of a paper or misunderstanding something) and some which were specific to being a patient (for example embarrassment or being perceived negatively by others, patient confidentiality, misuse of personal information published online, implications for future job applications and impact on receiving disability benefits).

"I am cautious about who I share information with in terms of my health. Just putting my name out there as a patient reviewer exposes some of that. I have decided that the benefits of being a BMJ peer reviewer are greater to me than the risk of exposing myself that way, but it was a consideration."

Reviewer satisfaction

Likeliness to recommend to other patients and carers

One hundred and twenty-seven (81%) of the 157 who had reviewed (82% BMJ, 77% RIE) would recommend being a patient and public reviewer to other patients and carers, 3 (2%) would not and 26 (17%) were unsure. Reasons for potentially not recommending to others included the lack of a sense of community among patient and public reviewers, non user-friendly submission software and disappointment that the paper reviewed was accepted despite their comments that the paper was not methodologically strong or well presented. Hesitancy to recommend focused on the fact that reviewing is not for everyone and that it is a highly subjective exercise.

Should other journals adopt patient and public review?

Based on their reviewing experience so far, 144 (92%) of the 157 reviewers (91% BMJ, 94% RIE) thought more journals should adopt patient and public review.

"We need to show maturity in academic working and co-creation with patients and carers. Patients and carers already influence whether grants are awarded (millions of pounds) so this is another step in closing the evidence to practice loop."

"Patients bring a unique perspective to scientific articles. Far from being a luxury, patient views are an important factor in research relevance."

"… [I]t should prove mutually beneficial and give researchers a ’patient’s eye view of potential/existing treatments as well as suggesting further avenues for research."

"Very good scheme and needs to be developed to become a beacon for other general medical journals."

Hesitancies included worry that patient and public review could become a tick box exercise, needing reassurance that patient and public reviewers’ comments were being considered and the appropriateness of journal content for patient and public review.

Perceptions of authors and other reviewers

Thirty-five (22%) reviewers indicated that they had reviewed a paper for which the editors later shared the peer reviewers’ comments. Twenty-nine of 35 (83%) ‘strongly agreed’ or ‘agreed’ that the authors of the paper addressed the points they had raised in their reviews and 28/35 (80%) ‘strongly agreed’ or ‘agreed’ that the authors were courteous when addressing their points. After having seen the peer reviewers’ comments on the paper they reviewed, 22 (63%) felt they had been able to include points important to patients or carers in their own review(s) that were not raised by the peer reviewers, 10 (29%) could not remember or felt it was difficult to judge and only three (9%) felt that they were unable to do so.

Helpfulness of journal instructions

Overall, 95/157 (61%) who had reviewed perceived the current instructions received on what to include in the review as ‘extremely’ or ‘very helpful’ (BMJ: 63%, RIE: 51%); 49 (31%) found them only 'moderately' helpful; and 13 (8%) 'slightly' or 'not at all' helpful. Reviewers made constructive suggestions on how the guidance could be improved.

How to acknowledge patient and public reviewers’ help

Patient and public reviewers at both journals are acknowledged annually on the journal websites, and analysis of the free-text comments about how reviewers should be acknowledged showed this was well received by participants. All BMJ reviewers receive an annual free subscription to the online version of the journal (bmj.com) as only research articles are open access, and this was popular with BMJ respondents. RIE is already open access for anyone to read. Having access to information was important and respondents were keen to receive longer term subscriptions for the journal, the ability to pass it on to a family member or subscriptions or access to articles in other journals published by BMJ. Participants were also keen to have access to more online resources, conferences and training to improve their reviewing skills.

"I think I am adequately ‘compensated’ for my work and I appreciate the subscription. As an advocate who rarely gets paid for my work, journal access is a big deal to me. I do this because it’s a passion and I want to see the dynamic change. I’m doing my part and that’s what gratifies me most."

RIE offers patients or service users who have reviewed a certain number of articles within any period of 12 months an Article Processing Charge (APC) waiver for an article submitted to the journal. Only three RIE reviewers commented on the value of the waiver for the APC. One was very positive of its value and had used the APC waiver to publish the experience of their patient group in the journal, another felt all patient and public reviewers should have the APC waived and another that it should be transferable to another person.

The majority of free-text comments did not focus on monetary incentives, but some reviewers did suggest honorariums, payment, gift vouchers, a prize draw and donations to charity after accruing reviews. There were suggestions that the lack of any form of compensation for reviewing might put off some potential patient and public reviewers. Others wanted something physical that they could show others, for example: a certificate of participation; a pin badge or keyring for long-standing reviewers; a gadget; an annual award for the best reviewer and an announcement of this; graded awards (eg, gold, silver and bronze) based on reviewer performance; a reward system tied to Publons (https://publons.com/home); a formal letter by post after a fixed number of reviews; and something to state in their CVs.

In free-text comments, reviewers expressed interest in being named as patient and public reviewers on the published articles themselves rather than just in the accompanying published reviews or for an indication in the article that it has been reviewed by patients. Others wanted more publicity over the contribution that patient and public review brings to the journals, the opportunity to contribute patient opinions on published articles or to be included as coauthors in commentaries and blogs. They suggested a patient and public reviewer TweetChat led by journal staff, more feedback about how their reviews were beneficial and to schedule patient and public reviewer meetings in order to build community and to develop expertise.

How to improve the experience

Suggestions on how to improve the overall experience included: allowing more time to review; improving and more frequent communications, for example, by newsletter; making the submission process and automated communications more user friendly; adding more and improved guidance on how to review including videos of the submission process and explaining what to expect and when; better matching of papers to reviewers’ experience; providing more varied sample reviews; more support, encouragement, training and forums to answer questions; editors providing (brief) feedback on usefulness of reviews and how papers have changed as a result of the reviews; better sharing of information around decisions made on manuscripts reviewed; developing a sense of community (an online network) among patient and public reviewers; peer-to-peer mentoring; raising the profile of patient review by illustrating the value added and changes made; and treating patient and public reviewers as equal partners in the peer-review process.

Discussion

This is the first study to explore the experience of patient and public reviewers involved in the peer-review process of academic journals. A large majority of patient and public reviewers were enthusiastic about their experience, would recommend others to act as reviewers and thought other journals should adopt patient and public review. Few reported concerns about being identified through open review. Reviewers identified areas where the journals could improve and how they would like to be acknowledged for their contributions. Annual acknowledgement of reviewers’ contributions on the journals’ websites was welcomed. BMJ reviewers welcomed the free subscription to the journal as a reward for reviewing (all content in RIE is open access). Only three RIE reviewers commented on the APC waiver. Some reviewers cautioned about tokenism and underlined the need for more feedback on how patient and public review is valued by authors and editors. Fostering a sense of community among patient and public reviewers was advocated. Despite the different approach and guidelines for patient and public review between The BMJ and RIE, the reviewers’ views on most issues were similar.

While the motivating factors for academics to review papers are known,^14–17 this study assessed what motivates patients and carers to review. It establishes that the overarching motivating factor is the opportunity to add patient or carer perspectives. It is of interest to note that motivating factors reported by academic reviewers were also considered important to patient and public reviewers, for example, the contribution of the paper to the subject area, relevance of the topic to their own experience/interests, a desire to keep up-to-date on current research and the opportunity to learn something new from the paper. Similar to academic reviewers, patient and public reviewers valued being given an online subscription to the journal^{14 17} and indicated a desire for access to more information from journals and publishers. Again, similar to academic reviewers, the most important factor in declining to review was conflict with other workload,¹⁴ but a higher proportion of patients reported the tight deadline for completing the review affected their decision to decline than academics.

The results suggest that it is feasible to introduce patient and public review alongside peer review of research papers. Patients and the public greatly appreciate the opportunity to be involved and see value in it. However, they also indicated areas for development and questioned some of our practice. This emphasises the importance of seeking feedback and making the subsequent changes to demonstrate that we value their opinion. Some areas are easily fixed, while others are restrained by the software, expense or demands of a fast publication process designed to meet the needs of authors and researchers. Journals do not typically pay peer reviewers. While the majority of reviewers did not focus on monetary compensation or incentives, some suggested lack of monetary compensation was a barrier if the goal was to solicit views from a fully representative sample of patients. Some organisations offer patients honoraria for reviewing, for example, NICE’s Health Technology Appraisal, although these requests are to review documents substantially longer than a research paper. Responses to this survey identified a range of ways that patient and public reviewers can be thanked and acknowledged other than through financial compensation.

We received a good response rate of 69% overall for those who had previously reviewed, reflecting a high level of ongoing engagement and support for this initiative. The respondents were involved in other patient involvement activities for an average of 5 days a month reflecting the increasingly strong commitment to include the patient’s voice in health research and healthcare provision. RIE reviewers reported spending more time on these activities than BMJ reviewers. This finding reflects the fact that many of the patient and public reviewers for RIE have been recruited from groups already working in research, such as the National Cancer Research Institute Consumer Forum in the UK or are European Patients' Academy Fellows, and thus likely to have experience and expertise in critical appraisal of protocols and research studies (especially on the patient-related aspects of the research). BMJ reviewers are asked to respond to a broader patient-specific set of questions and are not expected to have any research experience. Despite these differences, responses to the surveys were so similar for both journals that we needed to report only the combined response.

Our study is limited by including the experiences of patient and public reviewers at only two journals. However, these two journals were the first to introduce patient and public review alongside academic peer review and have sufficient experience of implementing it to enable the survey. At the time of the survey, we were not aware of any other journals systematically including patients in reviewing research papers. The two journals have different approaches and guidance, but their reviewers report similar experiences and challenges. Some respondents had only limited experience of reviewing and their perceptions may change with time. As this is a recent and novel initiative, it is important to survey patient and public reviewers again as we make changes to the guidance and seek to provide them with more support. While respondents did not focus on the need for monetary rewards and financial incentives, it is important to acknowledge that their views may not be representative of all potential patient and public reviewers. Similarly, while open review did not appear to be a concern for most patients, we have no way of knowing how many have been deterred from reviewing because of this.

Patient and public review of health research is a development in response to the call for greater patient and public involvement in research and healthcare in general. Peer review of research papers happens at the end of the research process, and we recognise and agree on the importance of involving patients early on at the start of the research process and throughout it to increase its value and minimise waste in research^{19 20} and to help make evidence more relevant and accessible to end users.^21–25 The BMJ and RIE see partnering with patients, their carers, wider support networks and the public as an ethical imperative essential to improving the quality, safety, value and sustainability of health systems.^{9–12 26} Patient and public review is just a single initiative under these partnership strategies the journals have introduced. Both journals also have patients and carers as members of their editorial boards and patient editors to bring the patients’ perspective to internal discussions and decision making. The results of this survey will be used to help improve the guidance, support and ultimately the experience of patient and public reviewers for the participating journals and other journals that adopt it. Further research is planned to identify where and how patient and public reviewers add value to the peer review process.