Analysis of peer reviewers’ response to invitations by gender and geographical region: cohort study of manuscripts reviewed at 19 biomedical journals before and during covid-19 pandemic

Abstract

Objectives

To describe gender and geographical inequalities in invitations to review and the response to these invitations and to assess whether inequalities increased during the covid-19 pandemic.

Design

Retrospective cohort study.

Setting

17 specialist medical journals and two large general medical journals from BMJ Publishing Group.

Population

Reviewers invited to review manuscripts submitted between 1 January 2018 and 31 May 2021. The cohort was followed up to 28 February 2022.

Main outcome measures

Reviewer’s agreement to review.

Results

A total of 172 278 reviewer invitations were sent (40.0% (68 827/172 278) women), and 61 660 (35.8%) were agreed. Invitations were sent to potential reviewers mainly (160 588; 93.2%) affiliated with high income countries: Europe (93 112; 54.0%), North America (47 177; 27.4%), Asia (13 920; 8.0%), Oceania (12 461; 7.2%), Africa (3220; 1.9%) and South America (2388; 1.4%). Independent factors associated with agreement to review were gender (odds ratio 0.92, 95% confidence interval 0.89 to 0.95, for women compared with men), geographical affiliation (3.62, 3.25 to 4.03, for Africa; 2.91, 2.75 to 3.08, for Asia; 3.36, 2.98 to 3.79 for South America; 1.02, 0.98 to 1.06 for North America and 1.35, 1.27 to 1.43, for Oceania compared with Europe), and country income (3.36, 3.13 to 3.60, for upper middle income; 5.16, 4.73 to 5.63, for lower middle income; and 4.69, 3.84 to 5.72, for low income compared with high income country). Agreement was also independently associated with last author’s geographical affiliation (0.82, 0.76 to 0.88, for Africa, 0.79, 0.77 to 0.82, for Asia, 0.94, 0.91 to 0.98, for North America, 0.88, 0.84 to 0.93, for Oceania compared with Europe), impact factor (1.62, 1.20 to 2.20, for >10 compared with <5), and type of peer review process (0.42, 0.30 to 0.60, for open compared with anonymised). During the first and second phases of the pandemic, agreement was lower than in the pre-pandemic period (P<0.001). The interaction between time periods and covid-19 related topic and reviewer’s gender was non-significant. However, significant interaction was found between time periods and covid-19 related topic and reviewer’s geographical affiliation.

Conclusions

To reduce bias and improve diversity, editors need to identify and implement effective strategies and continually evaluate progress against these to ensure that more women and researchers from upper middle income and low income countries are involved in review.

Introduction

Gender and geographical inequalities have been widely reported across the biomedical publication process, including the underrepresentation of women and ethnic minorities on editorial boards, authoring research publications, and holding of key authorship positions on manuscripts.^{1 2 3 4 5 6} Research from higher income countries is also favoured over lower income countries.⁷

Peer review, pivotal to the publication process, is under increasing pressure and prone to many potential biases,^{8 9 10 11} but the extent of these biases has been less investigated than other areas of the publication process as most peer review is done anonymously and reviews are not usually in the public domain.^{12 13} Some studies have analysed the selection of peer reviewers and their response to invitations to review submissions to biomedical journals; however, as this information is confidential, these studies have largely been done within single journals and may not be representative beyond their communities.^{14 15 16} Analyses of published peer reviews and public acknowledgment statements to named reviewers of biomedical journals have shown gender and geographical bias in the contributions.^{17 18 19 20}

The main reason researchers give for declining to review is lack of time.^{14 21} Peer reviewing is not a formally rewarded academic task and is often considered as outside the scope of researchers’ duties and done outside of work hours.^{8 22} The covid-19 pandemic and associated lockdowns led to significant and sudden global changes to researchers’ work patterns; many experienced an increase in their professional duties combined with additional family responsibilities.^{23 24 25 26} Women were shown to be more affected than men, particularly in their research activity and production.^{3 23 24 25 26 27 28 29 30 31} Few studies have evaluated the impact of the pandemic on biomedical peer review activity. Bro et al and Perlis et al compared peer review invitations and agreement patterns in single journals with no focus on gender or geographical inequalities and reported no significant differences in reviewer activity during the pandemic.^{15 32} Hupalo et al found no effect on women’s peer review activity during the pandemic compared with previous years,³³ but this was explored in a single journal. However, a large study across 2329 Elsevier health and medicine journals found that women were less involved in peer review, with an agreement rate lower than that of men in the very early phase of the pandemic compared with two previous years; they did not report any differences by geographical areas.²³

The objectives of our cohort study were to describe gender and geographical inequalities in invitations to review and the response to these invitations and to assess whether inequalities increased during the early and later stages of the covid-19 pandemic in 19 biomedical journals.

Methods

Participants and data sources

Using ScholarOne databases, we identified all original research manuscripts (including systematic reviews) submitted to one of 19 participating BMJ Publishing Group journals (see supplementary table A) between 1 January 2018 and 31 May 2021. We included all manuscripts that were reviewed by at least one peer reviewer and for which a final decision had been made by 28 February 2022 in the cohort with complete information for the studied variables.

Factors and outcomes

We collected data at three levels: manuscript, journal, and reviewer. At the manuscript level, we recorded manuscript ID; journal name; date of original submission; manuscript title; last authors’ first, middle, and last names, salutation, institutional address, and affiliation (department, institution, city, country);first and last editorial decisions and dates; and total numbers of reviewers invited who agreed to review and who completed reviews. At the journal level, we recorded the journal impact factor (JCR year 2019) (categorised as <5, 5-10, and >10) and the type of peer review process (open or anonymised (single anonymised)) (supplementary table A). At the reviewer level, we recorded the reviewer’s first, middle, and last names, salutation, institutional address, and affiliation (department, institution, city, country); response to the invitation (“unavailable,” “no response,” “late response,” “conflict of interest,” “decline,” “agreed,” “email problem”); dates of the invitation to review, response to the invitation, and completion of the review where applicable. We excluded invitations to reviewers who were unable to receive the invitation.

Our primary outcome was the reviewer’s agreement to review—that is, whether the response to the review invitation was “agreed” not “declined.” We considered response to peer review invitation as declined if the response to the invitation was “decline,” “conflict of interest,” “unavailable,” or “no response.”

The predictors assessed in our study were geographical affiliation, wealth of country of affiliation, invitation period, and manuscript topic. Common practice in biomedical research is for the most senior author to go last, and we hypothesised that the last author was likely to be more influential than the first. We extracted geographical affiliations of the last author, and reviewers from the country of their institutional affiliation and classified them into one of six groups (Europe, Africa, Asia, South America, North America, or Oceania). We further classified geographical affiliation by the wealth of the country by using the four levels defined by the World Bank Atlas method according to 2021’s gross national income per capita (low income by annual inhabitant $1045 or less, lower middle income by $1046 to $4095 per inhabitant, upper middle income by $4096 to $12 695, and high income by $12 696 or more).³⁴ We defined three invitation time periods—“pre-pandemic period” (1 January 2018 to 31 December 2019), “first phase of the pandemic” (1 January 2020 to 31 January 2021), and “second phase of the pandemic” (1 February 2021 to 28 February 2022)—to obtain periods of relatively similar duration before and during the pandemic.³ For each manuscript reviewed, we further identified whether covid-19 was the research topic by using a combination of keywords in the title and the abstract, as previously described.³

Gender determination

We identified the gender of reviewers and the last authors by using the same sequential four step procedure as in a previous study, using an accuracy for gender determination above 80%.³ Where this procedure did not identify gender, we recorded gender as “undetermined.”

Statistical analysis

Our sample size was fixed (26 040 reviewed manuscripts and 172 278 reviewer invitations), and the number of events evaluated (primary outcome) provided sufficient power to test several predictors in multivariable models. We described continuous variables as mean (standard deviation) or median (interquartile range) and presented categorical variables as frequencies and relative proportions. We graphically verified the linearity assumption of logit for the continuous variable (impact factor) and, as it was not verified, we further reported associations with a categorical variable. We performed mixed effects logistic regression models with response to the invitation to review as the dependent variable. To account for some clustering in the data (several reviewers were invited to review different manuscripts in different journals), we included three random factors (reviewers and manuscripts nested within journals) on the intercept by using the journal, manuscript ID, and reviewer ID variables. As the random effect for the manuscript was null, it was removed from the models. We prespecified two interactions: time period and manuscript topic with reviewer’s gender, and time period and manuscript topic with reviewer’s geographical affiliation. We performed this regression model on complete cases, and we report unadjusted and adjusted odds ratios, 95% confidence intervals, and P values from univariate and multivariable models. We used the glmer function from the “lme4” package with convergence optimizer “bobyqa” and a maximum of 20 000 iterations. We did a post hoc sensitivity analysis by using multiple imputation for missing gender determination (for the main model without interactions). Missing gender of last author and reviewer was considered at random and M=5 imputations were applied; adjustment of the imputation model was done for last author’s and reviewer’s geographical affiliation, time periods, and journal with the “ mice” package. We used R software (version 4.0.2) for analyses.

Patient and public involvement

We partnered and co-authored with the chair of the BMJ’s LGBTQ+ network (MR). MR contributed to almost every aspect of the study (data collection, gender determination, manuscript revision, and final approval).

Results

Characteristics of reviewed manuscripts

Among the 90 609 manuscripts submitted during the study period, 26 040 (28.7%) were reviewed, eligible for inclusion in our cohort and the analysis. Table 1 shows the characteristics of included manuscripts. The mean difference in the number of invited and agreed reviewers was 6.0. Most reviewed manuscripts were from journals with single anonymised peer review (n=13 515; 51.9%) and with impact factors <5 (n=16 483; 63.3%). Reviewed manuscripts more frequently had a man as last author (17 365/26 040; 66.7%) and a last author affiliated with a high income level country (n=21 463; 82.4%)—Europe (n=12 399; 47.6%), Asia (n=5906; 22.7%), or North America (n=4837; 18.6%)—with only a few affiliated to Africa (n=863; 3.3%) or South America (n=406; 1.6%).

Table 1.

Characteristics of selected reviewed manuscripts^* (n=26 040) submitted to 19 BMJ Publishing Group journals between 1 January 2018 and 31 May 2021 and followed up until 28 February 2022. Values are numbers (percentages) unless stated otherwise

Characteristics	Reviewed manuscripts (n=26 040)
Mean (SD); median (IQR) No of invited reviewers	8.8 (8.9); 6 (4-11)
Mean (SD); median (IQR) No of reviewers who agreed	2.8 (1.2); 3 (2-3)
Mean (SD); median (IQR) No of reviews completed	2.4 (0.9); 2 (2-3)
Type of peer review process:
Single anonymised	13 515 (51.9)
Open	12 525 (48.1)
Impact factor:
<5	16 483 (63.3)
5-10	3997 (15.3)
>10	5560 (21.4)
Last author’s gender:
Woman	8675 (33.3)
Man	17 365 (66.7)
Last author’s geographical affiliation†:
Europe	12 399 (47.6)
Africa	863 (3.3)
Asia	5906 (22.7)
South America	406 (1.6)
North America	4837 (18.6)
Oceania	1629 (6.3)
Income level of last author’s country of affiliation‡:
High income	21 463 (82.4)
Upper middle income	3006 (11.5)
Lower middle income	1163 (4.5)
Low income	408 (1.6)

IQR=interquartile range; SD=standard deviation.

^*Manuscripts with at least one review completed by 28 February 2022.

^†Country of institutional affiliation.

^‡Income level by country of affiliation according to United Nations classification (1 July 2021).

Reviewers’ response to invitations by characteristics of reviewers and time period

Among the total number of 172 278 invitations to review, 35.8% (n=61 660) were agreed. Most (n=53 433; 86.7%) reviewers who agreed to review went on to complete it. Table 2 shows the characteristics of reviewers invited, agreeing, and declining to review. A total of 68 827 (40.0%) women were invited, and women agreed to review manuscripts 6.3% less often than men (P<0.001). More than half of invitations were sent to reviewers affiliated to Europe (54.0%) and around one-quarter to North America (27.4%); only a small proportion of invited reviewers were affiliated to Asia (8.0%), Oceania (7.2%), Africa (1.9%) or South America (1.4%). Agreement to review was highest among invited reviewers affiliated to Asia (52.0%), Africa (50.3%) followed by South America (50.0%), North America (34.3%), Oceania (35.4%) and Europe (33.3%). More than nine out of 10 (93.2%) invited reviewers were affiliated with a high income country, whereas agreement to review was highest among reviewers affiliated with lower middle, low income countries and upper middle income compared with high income countries (respectively, 60.0%, 54.7%, 51.6%, and 34.4%).

Table 2.

Proportion of review invitations agreed and declined by key characteristics of reviewers (n=172 278 review invitations)

Reviewer characteristics	No (column %) invited (n=172 278)	No (row %) agreed (n=61 660; 35.8%)	No (row %) declined (n=110 618; 64.2%)	P value
Mean (SD); median (IQR) interval between reviewer’s invitation and response, days	2.0 (2.5); 1 (0-4)	2.0 (2.4); 1 (0-4)	2.0 (2.6); 1 (0-4)	P<0.001*
Reviewer’s gender:				P<0.001*
Woman	68 827 (40.0)	22 040 (32.0)	46 787 (68.0)
Man	103 451 (60.0)	39 620 (38.3)	63 831 (61.7)
Reviewer’s geographical affiliation†:				P<0.001*
Europe	93 112 (54.0)	31 024 (33.3)	62 088 (66.7)
Africa	3220 (1.9)	1619 (50.3)	1601 (49.7)
Asia	13 920 (8.0)	7234 (52.0)	6686 (48.0)
South America	2388 (1.4)	1195 (50.0)	1193 (50.0)
North America	47 177 (27.4)	16 171 (34.3)	31 006 (65.7)
Oceania	12 461 (7.2)	4417 (35.4)	8044 (64.6)
Income level of reviewer’s country of affiliation‡:				P<0.001*
High income	160 588 (93.2)	55 223 (34.4)	105 365 (65.6)
Upper middle income	6380 (3.7)	3289 (51.6)	3091 (48.4)
Lower middle income	4558 (2.6)	2737 (60.0)	1821 (40.0)
Low income	752 (0.4)	411 (54.7)	341 (45.3)

IQR=interquartile range; SD=standard deviation.

^*Univariate mixed effect logistic regression with journal and reviewer as random effects.

^†Country of institutional affiliation.

^‡Income level by country of affiliation according to United Nations classification (1 July 2021).

Looking in more detail, more than three times fewer reviewers with an affiliation to Asia were invited than manuscripts reviewed with a last author affiliated to Asia (fig 1, top). Conversely, more invited reviewers were affiliated to Europe and North America than manuscripts reviewed with a last author affiliated to those regions. Similarly, more invitations were sent to reviewers affiliated with high income countries than manuscripts reviewed with a last author affiliated with a high income country (fig 1, bottom).

Fig 1.

Distribution of geographical affiliation (top) and income level group of country of affiliation (bottom) for reviewers invited to review versus last author of reviewed manuscripts in cohort (n=172 278 review invitations and n=26 040 reviewed manuscripts)

Of all invitations sent during the pre-pandemic period, the agreement rate was 35.8%. Agreement rates seemed to be higher during the first phase of the pandemic for covid-19 and non-covid-19 related manuscripts (respectively, 2609/7184; 36.3% and 18 602/50 927; 36.5%) compared with the second phase of the pandemic (respectively, 1448/4493; 32.2% and 6766/20 421; 33.1%) (supplementary table B). Reviewers were faster to respond to invitations to review covid-19 related manuscripts in the first phase of the pandemic compared with pre-pandemic. The percentage of agreement to review was higher for men than for women in all time periods, both for covid and non-covid related manuscripts.

Multivariable analysis of response to peer review invitations

Table 3 shows the univariate analyses. The time period and manuscript topic, the reviewer’s gender and geographical affiliation, the last author’s geographical affiliation, , and the type of peer review process were all significantly associated with agreement to review; these associations all remained significant in the multivariable model in addition to the journal’s impact factor (table 3, fig 2).

Table 3.

Agreement (n=172 278) to review manuscripts^* submitted to 19 BMJ Publishing Group journals between 1 January 2018 and 31 May 2021: univariate^{† ‡} and multivariable models^{† ‡}

Independent variables	Univariate analysis (n=172 278) ‡ †			Multivariable analysis (n=172 278)† ‡
	Odds ratio (95% CI)	P value		Odds ratio (95% CI)	P value
Time period and topic § (reference=pre-pandemic)
First phase of pandemic (covid-19)	1.03 (0.97 to 1.10)	<0.001		0.98 (0.92 to 1.05)	<0.001
Second phase of pandemic (covid-19)	0.89 (0.82 to 0.97)			0.84 (0.77 to 0.92)
First phase of pandemic (non-covid-19)	0.94 (0.91 to 0.97)			0.92 (0.89 to 0.95)
Second phase of pandemic (non covid-19)	0.83 (0.79 to 0.86)			0.80 (0.76 to 0.83)
Reviewer’s characteristics
Woman (reference=man)	0.86 (0.84 to 0.89)	<0.001		0.92 (0.89 to 0.95)	<0.001
Reviewer’s geographical affiliation¶ (reference=Europe):		<0.001			<0.001
Africa	3.39 (3.05 to 3.75)			3.62 (3.25 to 4.03)
Asia	2.76 (2.61 to 2.92)			2.91 (2.75 to 3.08)
South America	3.18 (2.83 to 3.57)			3.36 (2.98 to 3.79)
North America	1.01 (0.97 to 1.04)			1.02 (0.98 to 1.06)
Oceania	1.33 (1.25 to 1.41)			1.35 (1.27 to 1.43)
Manuscript’s characteristics
Woman as last author (reference=man)	0.99 (0.97 to 1.02)	0.57		0.99 (0.96 to 1.02)	0.42
Last author’s geographical affiliation¶ (reference=Europe):		<0.001			<0.001
Africa	0.94 (0.88 to 1.01)			0.82 (0.76 to 0.88)
Asia	0.85 (0.83 to 0.88)			0.79 (0.77 to 0.82)
South America	0.97 (0.88 to 1.08)			0.91 (0.81 to 1.01)
North America	0.94 (0.91 to 0.97)			0.94 (0.91 to 0.98)
Oceania	0.89 (0.85 to 0.94)			0.88 (0.84 to 0.93)
Journal’s characteristics
Impact factor (reference= <5):		0.06			0.002
5-10	1.00 (0.71 to 1.40)			0.88 (0.64 to 1.21)
>10	1.49 (1.07 to 2.09)			1.62 (1.20 to 2.20)
Open peer review (reference=single anonymised)	0.44 (0.31 to 0.61)	<0.001		0.42 (0.30 to 0.60)	<0.001

CI=confidence interval.

^*Manuscript with at least one review completed by 28 February 2022.

^†Mixed effect logistic regression with journal and reviewer as random effects.

^‡Model performed on completed values (n=172 278)

^§First phase of pandemic was defined as 1 January 2020 to 31 January 2021, second phase as 1 February 2021 to 28 February 2022, and pre-pandemic as 1 January to 2018 to 31 December 2019.

^¶Country of institutional affiliation.

Fig 2.

Response of reviewers invited (n=172 278) to review manuscripts (with ≥1 review completed) submitted to 19 BMJ Publishing Group journals between 1 January 2018 and 31 May 2021. Summary of multivariate model: mixed effect logistic regression with journal and reviewer as random effect. Reference categories of variables used in model are specified horizontally below interrupted black line. Model presented in table 3. IF=impact factor

Time period and manuscript topic was significantly associated with agreeing to review a manuscript in the multivariable analysis; all associations were significantly lower compared with pre-pandemic, except for covid-19 related manuscripts during the first phase of the pandemic, for which the association was not significant. We observed a decreasing trend in agreement from pre-pandemic to the second phase of the pandemic.

Reviewer’s gender and geographical affiliation—Agreement to review a manuscript was significantly lower among women compared with men (adjusted odds ratio 0.92, 95% confidence interval 0.89 to 0.95). Compared with Europe, the adjusted odds ratio was significantly higher for all regions (except for North America) (table 3). When high income was the reference (supplementary table C; supplementary figure A), the odds ratio was drastically higher for reviewers affiliated to upper middle income (3.36, 3.13 to 3.60), lower middle income (5.16, 4.73 to 5.63) and low income countries (4.69, 3.84 to 5.72).

Journal impact factor and type of peer review process—Agreement to review was significantly associated with impact factor (higher agreement to review for journals with impact factors >10 compared with <5). The type of peer review process was also independently associated with agreement to review: lower for journals with open peer review compared with single anonymised peer review (table 3, fig 2).

We confirmed similar findings when we used multiple imputation, except that the odds ratios for the associations with the reviewer’s geographical affiliation were slightly lower (supplementary table D).

Reviewers’ response to invitations by geographical affiliation and gender during and before the pandemic

The interaction between the reviewer’s gender and the time period and topic was not significant (P=0.06; supplementary table E). Women agreed to review less often than did men, with no difference across the time periods.

The interaction between the reviewer’s geographical affiliation and the time period and topic was significant (P=0.002; supplementary table F). Reviewers affiliated with Asia and Oceania agreed to review less often for covid-19 and non-covid-19 related manuscripts in the first and second phases of the pandemic compared to pre-pandemic. Reviewers affiliated to Africa or South America agreed more often to review covid-19 related manuscripts in the first phase and the second phases of the pandemic than in the pre-pandemic.

Discussion

We found wide inequalities in the gender and geographical affiliation of researchers invited to peer review, and also in their response to the invitations. Similarly to other studies,^{8 15 16} just over a third of all invited reviewers were women, but we also found that women were less likely to agree to review than men. Overall agreement to review fell as the pandemic unfolded, but gender inequalities did not widen. The vast majority of invited reviewers were affiliated to high income countries and, similarly to other studies, some regions were underrepresented in invitations to peer review despite being large contributors as last authors on the same cohort of submissions and having high review agreement rates.^{8 15} Factors associated with higher agreement to review were invitations to reviewers affiliated to upper middle income, lower middle income or low income countries (compared with high income countries); invitations to reviewers affiliated to Africa, Asia, South America, or Oceania (compared with Europe); journals with higher impact factors (>10 compared with <5); and journals using an anonymised peer review process (compared with open). Factors associated with lower agreement to review were invitation to review sent during the pandemic (except for covid-19 related manuscripts during the first phase), reviewer being a woman, last authors’ geographical affiliation (Africa, Asia, North America, or Oceania compared with Europe), and last authors’ country income level (low or lower middle income or upper middle income compared with high income).

Comparison with other studies

Previous works on publicly available datasets have also shown gender and geographical bias in peer review.^{7 17 18 19 20} Our dataset enabled us to look at both invitations and responses to these same invitations, and we found that women were not only less frequently invited to review but also agreed less often than men, compounding this gender inequality. Although some gender inequality in peer review activity has been reported in the early pandemic period,²³ other studies found no difference in overall patterns of review activity and women’s peer review activity.^{15 32 33} To the best of our knowledge, our study was the first to look at invitations and responses to invitations by gender in the later stages of the pandemic compared with pre-pandemic. We did not find a significant change in women’s agreement to review covid-19 or non-covid-19 related manuscripts in either pandemic time period compared with pre-pandemic. This is not in line with our previous study of women’s authorship in the same group of journals and other studies of reduced research productivity during the pandemic.^{3 23 24 25 26 27 28 29 30 31} We can only speculate that this might be because peer review is less onerous than other aspects of research productivity that were affected in the pandemic.

We found several factors that were associated with agreement to review at manuscript, journal, and reviewer level. We did not identify other studies exploring factors associated with agreement to review. However, our finding that reviewers with North American/European/Oceanian affiliations had lower rates of agreement compared with reviewers affiliated with Africa, Asia, South America, or Oceania has previously been reported.¹⁵ Reviewers also seem to be more motivated to agree to invitations from journals with high impact factors; others have found that review reports are longer and completed more quickly for journals with higher impact factors.⁸

Strengths and limitations of study

We included almost 100 000 submitted research manuscripts and 172 278 responses to peer review invitations from 19 biomedical journals in a range of specialties attracting submissions and worldwide reviewers. This unique dataset gave us the opportunity to explore some hypotheses on the factors associated with agreement to review and to include all review invitations (including those that were declined—the hidden part of the peer review process, which has been explored little until now). This is the first study to explore gender and geographical diversity in invitations to review and responses to these invitations and to consider factors associated with this agreement at several levels (journals, manuscripts, and reviewers). We included all review invitations at multiple journals, whereas other studies have been limited to online peer review reports of published manuscripts, focused on a single journal, or used aggregated metadata.^{14 15 16 17 18 19 20 23 32 33 35} We explored geographical diversity by using multivariable analyses with both geographical affiliation and the income level of the country, which increases the understanding of the identified associations. Other researchers have investigated geographical diversity, but they provided only a descriptive analysis.⁸ Undetermined reviewers’ and last authors’ genders had no effect on our results, as multiple imputation resulted in similar findings. Future studies should go one step further and investigate the role of ethnicity and seniority of reviewers and their association with the different steps of the peer review process.

Our study has some limitations. In the absence of self-identified data from authors and reviewers, we determined gender as binary by using software based on first name and country of affiliation. Although we found that open peer review was associated with lower agreement than anonymised review, most (87%) open review invitations were from one journal (BMJ Open) and we should be cautious in our interpretation of the generalisability of this result. Retrospectively, we had to exclude the variables related to the editors owing to the identification of inconsistencies in the way editors’ names and roles were documented on the tracking systems across the journals (see supplementary file for changes made).

Policy implications

Lack of gender and geographical diversity in the peer review process has important implications. Involvement in peer review is important, as it not only exposes researchers to current and new research ideas but can also help with career progression through the development of writing and critical appraisal skills and foster relationships with other research networks.³⁶ Non-inclusive practice in invitations to review thus has the potential to exacerbate existing gender and geographical bias. Relying on a narrow pool of reviewers may lead to the opinion of the most productive researchers being overrepresented.^{8 37 38} Lack of diversity in peer review also has the potential to imbalance scientific knowledge with an underrepresentation of opinion from women and non-high income countries, which could impair global health policies delivered to implement research findings. Also, gender and geographical inequalities in peer review may deprive the scientific community of more innovative research,^{39 40} as we already know that collaboration in gender diverse teams leads to more innovative and high impact scientific ideas.⁴¹ Bibliographic evidence also shows that papers with more ethnically diverse authorship teams generate more citations and can outperform teams made up of similar people.^{39 42}

Journals need to improve diversity when recruiting and inviting reviewers and monitor the success of their approaches across time to highlight barriers and propose new effective methods for improving inclusion. For example, a policy whereby underrepresented groups are oversampled to account for their lower level of agreement might be considered for reviewer invitations. Our finding that women were not only invited to review less often than men but also declined more often needs further attention. We need to better understand the reasons why women are less likely to agree to review to find ways to support them to become more active peer reviewers. The counter argument to this is that men might agree to review manuscripts more often than is appropriate. Men have been shown to be more confident; they more positively frame their own research findings,⁴³ and they might approach invitations to review differently than women. Fixing the systems that support bias should be the focus.⁴⁴ Academic institutions need to better support women involved in research to increase their overall productivity, which has been shown by others to have fallen in the pandemic, and avoid a backslide in gender equality.^{24 45} More equal sharing out of pastoral and teaching roles that deprive women of time to commit to other scholarly activities,⁴⁵ including peer review, could help to redress some of the imbalance.

Overall reduction in the agreement to peer review during the pandemic will have had an undesirable effect, but the widening of geographical inequalities during the pandemic is of particular concern, especially as reviewers from some underrepresented regions were more likely to accept invitations to review. Researchers from these regions were not given the opportunity to contribute to scientific debate as the pandemic unfolded, despite the fact that some of these regions were the most affected by the pandemic. Researchers from minority ethnic groups have also faced further challenges to academic work during the pandemic, and these need to be tackled.⁴⁵ Editors can help to correct imbalances in reviewer invitations going forward, but this will not influence the identified bias of reviewers not agreeing to review manuscripts with last authors from underrepresented regions. These findings could be interpreted as a disinterest in research conducted in Asia and more globally for the research conducted in low and middle incomes countries. To overcome potential bias related to the name and/or affiliation of authors, journals might consider removing author information from the reviewer’s invitation email. However, anonymising reviewers is often not effective,¹¹ and it could detract from moves towards greater transparency in peer review.

Conclusions

Calls for increased diversity and inclusion in editorial boards of journals have been issued,^{18 42 46 47} but they are not enough. Editors need to identify and implement effective peer review strategies and continually evaluate these over time to ensure that more women and researchers from middle income and low income countries are involved in peer review and that global opinion is better represented in research.

What is already known on this topic

• Gender and geographical disparities have been reported in editorial processes, among editorial boards, and in key authorship positions of submitted and published biomedical research

• Surveys and studies based on samples of published reviewers’ reports or reviewers’ names have shown lower participation of women and researchers from emerging countries

• In the early pandemic period, gender inequalities widened for attainment of key authorship positions

What this study adds

• Gender inequality among invitations to peer review is compounded by fewer women agreeing to peer review invitations, and this persisted during the pandemic

• Reviewers with African, Asian and South American affiliations and those from low and middle income countries are underrepresented, even when they were more likely to agree to review

• These geographical inequalities widened during the pandemic

Web extra.

Extra material supplied by authors

Web appendix: Supplementary materials

Web appendix: File showing changes made

Ethics statements

Ethical approval

This cohort study is ancillary to the ATHENA study, a broader research project funded by the SNSF, which is assessing whether gender bias exists during the editorial decision making process. The ATHENA project received ethical approval from the Institutional Review Board of the Geneva Canton on 16 April 2018 (2019-00540). All studies are being conducted under a confidentiality agreement between BMJ Publishing Group and the medical school of Geneva University represented by the Department of Health and Community Medicine.

Data availability statement

Relevant anonymised reviewer level data are recorded in the public repository at the University Bern, BORIS portal (https://doi.org/10.48620/92251), and data are available on reasonable request from the corresponding author. The study was conducted under a confidentiality agreement between the BMJ Publishing Group and the medical school of Geneva University, represented by the Department of Health and Community Medicine.