Geographical diversity of peer reviewers shapes author success

Significance

Peer review is central to allocating resources in science. However, if reviewers typically favor work from their own country (homophily) and most reviewers come from just a few countries, then authors from those countries enjoy a structural advantage that we call “geographical representation bias.” Using administrative data from 60 STEM journals published by Institute of Physics Publishing, we find evidence of this bias. Models with manuscript fixed effects ensure that this result is not confounded by differences in quality or topic. Anonymizing manuscripts does not substantially reduce reviewers’ homophily. The lack of geographical diversity among reviewers limits the diversity of successful authors, so investments in reviewer diversification may be necessary to reduce blind spots in published knowledge.

Abstract

Scientific institutions like funding agencies and journals rely on peer reviewers to select among competing submissions. How does the geographical diversity of reviewers affect which authors are selected? If reviewers typically favor submissions from their own countries, but reviewers from only some countries are well represented in the reviewer pool, this can create a “geographical representation bias” favoring authors from those well-represented countries. Using administrative data on 204,718 submissions to 60 STEM journals from the Institute of Physics Publishing, we find support for representation bias. Reviewers from the same country as the corresponding author are 4.78 percentage points more likely to review positively compared to other reviewers of the same manuscript. Authors from the United States of America, China, and India are 8 to 9 times more likely to be evaluated by same-country reviewers compared to less-represented countries with similar incomes. Furthermore, an instrumental variables analysis of an anonymization policy shock shows that anonymizing submissions does not significantly reduce same-country homophily. Thus, investments in reviewer diversification may be necessary to mitigate the structural advantage of authors from major science-producing countries and avoid blind spots in collective knowledge.

Publishing in peer-reviewed journals is crucial to making new ideas visible to the scientific community and other audiences. However, authors from low- and middle-income countries often have lower acceptance rates in academic journals across various fields (1, 2), leading to lower attention and impact (3, 4). These publishing disparities can affect not only individual careers—given growing reliance on publication metrics in evaluations (5)—but may also have collective epistemic consequences for their fields.

Scientists’ choices of which problems to research are frequently correlated with their geographic location (6). Findings produced in one location are often not salient or applicable globally—as has been observed with diseases (7, 8), agricultural technologies (9), and economic development programs (10). Consequently, barriers to geographic diversity in publishing may create blind spots in the corpus of published knowledge and shape what problems are viewed as pursuit-worthy.

Some causes of global disparities in publishing are well understood. First, authors from lower-income countries may have less access to the equipment, networks, and other resources needed for cutting-edge work in resource-intensive fields (11, 12) and face language barriers (13, 14). As a result, their papers may have lower quality or fit for English-language outlets. Second, authors from underrepresented demographic groups may study topics that are viewed less favorably by evaluators in overrepresented groups (15, 16), and such topic mismatch may affect geographically underrepresented scientists as well. Third, some research finds that peer reviewers and editors show bias toward high-status authors and work from wealthier countries (17–22).

While these causes are related to individual-level characteristics of authors, like resources, status, and country, and individual-level evaluator biases, here we focus on a different and fundamentally structural cause related to peer reviewer diversity. Specifically, we investigate geographical concordance between authors and reviewers and how it may result in what we call “geographical representation bias.” This bias occurs if two conditions are met. The first is homophily—reviewers favor work from authors that are similar or close to them in some dimension. We focus on homophily by country, sometimes referred to as “home bias” (23–25). Reviewers from the same country as authors, i.e., same-country reviewers (SCRs), may exhibit homophily through mechanisms including having more information about the submission or its authors, similar tastes to the authors, and identity-based biases like nationalism (SI Appendix, Mechanisms). The second condition is differential access to SCRs—systematic differences in how likely authors are to be evaluated by SCRs. Both conditions are necessary for a lack of evaluator diversity to result in the geographical representation bias. If reviewers exhibit homophily (condition 1 is met) but all authors have equal access to SCRs (condition 2 not met), then no group is disadvantaged relative to another. And a lack of demographic diversity among reviewers (condition 2 is met) does not by itself create a representation bias if they review similarly (condition 1 not met).

Geographical representation bias, if it exists, is a structural bias in the sense that it is not attributable to malfeasance by any individual actor (26). Instead, it is the system-level product of institutional practices, policies, and historical path dependence. Specifically, peer reviewers for particular outlets are likely selected disproportionately from those publishing in these outlets. Historically, those authors hailed from major science-producing and relatively wealthy countries (27). Consequently, authors from these wealthier countries who submit work to the outlets now will have greater chances of being reviewed by SCRs and benefit from their homophily more often. In this way, past success systemically improves future success.

Despite widespread belief that representation bias affects science, existing evidence for it is mixed and inconclusive. The key challenge has been that peer review data are typically sensitive and proprietary. In cases where data were available, some analyses found suggestive evidence but could not control for relevant confounding factors like submission quality because of a lack of granularity (28–30). Others found null evidence but lacked sufficient statistical power (31). Without strong evidence that reviewers review differently depending on their geographic concordance with authors, it is possible that diversifying reviewers, as many have called for, would do little for the publication prospects of underrepresented authors. And as difficult as it has been to establish geographical representation bias in the first place, to our knowledge, no studies have tested methods for reducing it.

Here, we overcome the confounding and statistical power challenges of prior work using large-scale administrative peer review data on 204,718 submissions to 60 English-language journals from January 2018 to December 2022 from the Institute of Physics Publishing (IOP). Submissions were matched to countries using the self-reported country of the corresponding author. Most submissions were reviewed single-anonymously, where authors’ identities were visible to reviewers. The granularity of the data enabled us to fully control for manuscript quality and other attributes using manuscript fixed effects, effectively comparing different reviewers of the same manuscript. We found strong, and arguably conclusive, evidence of country homophily: Reviewers from the same country as authors (SCRs) gave positive recommendations on manuscripts about 4.78 p.p. more often than non-SCRs on the same manuscript. Authors from low- and lower-middle-income countries were less than half as likely as those from high-income countries to be assigned to SCRs, proportionate to their representation in the reviewer pool. Taken together, we conclude that geographical representation bias is present in our data, even when fully controlling for differences in manuscript quality and other attributes.

How can institutions remedy geographical representation bias? Since there are two necessary conditions for the bias, a remedy could work by equalizing access to homophily through diversification (2, 30, 32, 33), which would in effect spread homophily out more evenly. An alternative method is to reduce country homophily in the first place. Numerous studies have shown that double-anonymization (i.e., hiding authors’ identities from reviewers) is effective at reducing a variety of reviewer biases (17–22, 34). However, to our knowledge, no study has conclusively shown whether such policies reduce country homophily or considered them as a remedy for low reviewer diversity. The effectiveness of anonymization is important not only practically but conceptually. Double-anonymization should affect evaluations only through identity-related mechanisms, such as a nationalism bias. To test for the effect of anonymization on homophily, we leverage a policy change among IOP journals that enabled authors to select into double-anonymous peer review (35). The rollout of the anonymization policy was staggered across journals in an arbitrary order, making the policy an attractive instrumental variable for anonymization (see Materials and Methods and SI Appendix). In the instrumental variables analysis, the effect of anonymization on homophily was substantively small and statistically nonsignificant. In other words, hiding authors’ identities (including geographic information) did little to reduce country homophily.

Existing Evidence

Country Homophily.

Country homophily—evaluators favoring products, people, or ideas from their own country—is well-studied outside of science (36). For example, Wright et al. (24) find that judges of global startups are systematically more likely to recommend startups from their home region and provide evidence that this preference is due to information differences rather than differences in taste or evaluator biases. Verlegh (37) discusses several studies finding that consumers prefer domestic over foreign products, arguing that the homophily stems from concerns for the domestic economy and an in-group identity bias.

In science, Rubin et al. (25) provide some evidence that editors from civil-law European countries are more favorable toward work from their own country, while those from the common-law countries are not. They attribute this pattern to a difference in individuals’ orientation to institutions in these regions. It is also suggested that the European editors’ same-country preference is not merit-based, since those same-country manuscripts they publish receive fewer citations than their common-law counterparts.

Our closest intellectual antecedents are Link (29), Murray and colleagues (30), Gaston and Smart (28), and Tomkins (31). Link explored whether peer reviewers favor US vs. non-US submissions to a medical journal. She found that, while both USA and non-USA reviewers favored submissions from the United States of America over those from non-USA, USA reviewers favored them more strongly. Murray et al. studied reviewer–author gender and country homophily in a setting where reviewers evaluate manuscripts as a team. They found higher acceptance rates for manuscripts where at least one member of the reviewer team was from the same country as the last or corresponding author. Gaston and Smart analyzed over 100 journals across two disciplines and found that reviewers from the same region as the corresponding author are more likely to accept invitations to review. They also show that manuscripts with reviewers and authors from the same region have higher acceptance rates. While higher acceptance may be related to positive reviews specifically from SCRs, Link, Murray et al., and Gaston and Smart are unable to confirm whether this is the case since their data are at the manuscript level rather than at the level of individual reviews. While the evidence is suggestive, it is inconclusive because of two plausible confounders—differences in submission quality and differences in reviewers’ standards. Manuscripts of a higher quality may come from countries with more reviewers, creating an association between SCRs and positivity. Furthermore, reviewers’ standards of evaluation may differ systematically by country. Thus, we argue these studies are insufficient in establishing the existence of country homophily among reviewers.

Tomkins et al. conducted an experiment anonymizing submissions to a computer science conference. They find that, controlling for manuscript quality using scores given by double-anonymous reviewers, there is no statistically significant association of SCRs with positivity. However, only 146 submissions in their data had SCRs, so the null effect could have been due to low statistical power. In sum, nonscience contexts have shown country homophily, while in science, the evidence is mixed and inconclusive.

Homophily may be driven by some combination of information, tastes, and identity-related biases (SI Appendix, Mechanisms).Information refers to reviewers’ prior knowledge of the topic and methods, related work, or the work being reviewed. For instance, reviewers may be more familiar with same-country authors’ prior work and give them the benefit of the doubt when the claims in a manuscript are incompletely described or subjective (38). This self-reinforcing, cognitive mechanism has been documented in the physical sciences and is theorized to make peer review systems risk averse (39). Tastes refers to reviewers’ preferences for certain methods, theoretical frameworks, or stylistic choices. For example, qualitative work shows that US and Italian research communities studying gravitational radiation have distinctly separate epistemic cultures, with different norms for data sharing and standards of evidence (40). Identity-based biases refer to reviewers’ favoring of work by authors with particular identities.

Differential Access to SCRs.

Authors from different countries are likely to be assigned to SCRs at different rates. First, much discussion surrounding the diversification of editors and reviewers centers around homophily in the formation of professional networks. Editors, who are typically from a small set of wealthier countries (2, 41, 42), may rely on their professional networks to fill editorial boards and find reviewers (30, 43). Consequently, the pool of evaluators may be disproportionately composed of researchers based in those countries. This result may occur entirely unintentionally. If a journal’s pool of reviewers is composed primarily of individuals who have submitted and published in the journal in the past, that pool will overrepresent countries submitting the most work, such as China, the United States of America, and India. Consequently, if editors assign reviewers by choosing them from the existing pool and based only on expertise—ostensibly the most fair way—authors from poorly represented countries will have lower chances of being assigned to SCRs. To summarize, there is increasing evidence that the geographical distributions of those who submit manuscripts to academic journals and those who evaluate them are diverging. Attention has focused primarily on editors rather than reviewers, although it is implied that reviewer demographics will correlate with editor demographics. The present study focuses on reviewers directly.

Empirical Setting.

Partnering with the IOP, which is based in the United Kingdom, we use metadata on all submissions and reviews from January 2018 to December 2022. The metadata comprise 63 English-language journals in the physical sciences (three journals were ultimately excluded from the analytic sample due to insufficient complete observations). The data did not include the content of the submissions or the peer reviews. The analysis was undertaken with a data use agreement and approved by the University of Michigan Institutional Review Board, Protocol #HUM00194927.

Authors and reviewers from roughly 150 countries are represented in the analytic sample. Thus, the setting for this study is truly global. SI Appendix, Tables S1 and S2 display summary statistics for variables used in the model specifications for the data on realized reviews and invited reviewers, respectively. Then, 30.25% of the 787,055 invitations to review resulted in a realized review. SCRs were 4.74 p.p. more likely to respond to invitations to review compared to non-SCRs invited for the same submission (SI Appendix, Table S3). Further, 113,232 of 204,718 (55.31%) submissions received at least one review (many were desk-rejected). 70.40% of 238,097 first-round reviews were positive recommendations (i.e., accept or revise). The analytical sample is restricted to submissions that underwent external peer review (i.e., not desk rejected). Of the invited and realized reviewers in the first review round, 11.94 and 14.67% were SCRs, respectively. Then, 24.49% of submitted manuscripts involved at least one SCR.

SI Appendix, Fig. S1 shows the number of manuscripts and journals in the analytic sample by year. The number of journals increased each year until peaking in 2021 with 60 journals. Country-level distributions for number of submissions, number of unique realized reviewers, acceptance rates, and reviewer positivity and response rates are available in SI Appendix, Fig. S2. The top three countries by both number of submissions and unique reviewers were China, India, and the United States of America. Acceptance rates were highest for countries in Europe, Northern America, and Oceania. Of the 204,718 submissions in the raw data, 68,905 (33.66%) were from high-income countries (HIC), 84,222 (41.14%) were from upper-middle-income countries (UMIC), and 51,591 (25.20%) were from low- and lower-middle income countries (LLMIC) (see Income Categories in Measures and Methods). SI Appendix, Fig. S3 aggregates raw acceptance rates by the country income groups. Submissions from HIC authors were accepted more than twice as often as those from LLMIC authors, corroborating work finding large geographic disparities in acceptances.

Results

Is There Country Homophily in Peer Review?

We find that SCRs had a 4.78 p.p. (P < 0.0001) higher likelihood of positive review compared to non-SCRs evaluating the same submission. Fig. 1 shows the “Overall” estimate and estimates for each of the three country income groups separately. SI Appendix, Table S3 shows the full regression table.

Fig. 1.

SCRs are more likely to give positive reviews compared to non-SCRs on the same manuscript. Marginal probability that a review is positive in overall data and for each country income group. The line at zero represents the positivity of non-SCRs. See Eqs. 2 and 3 for the model specifications. SE were clustered at the level of the fixed effects and error bars are 95% CI.

The intensity of SCR homophily varied by country income group, with the upper-middle income country (UMIC) group showing the largest estimates, but SCR homophily estimates were positive and significant for all (SI Appendix, Table S4). We emphasize that these relationships between SCR status and positivity are not confounded by manuscript quality, because including manuscript fixed effects in the models ensures that comparisons are between reviewers on the same manuscript. Nor are they confounded by reviewers’ standards, since reviewer fixed effects absorb individual reviewers’ baseline positivity rates. These results establish the first necessary condition for geographical representation bias—homophily.

Are There Geographical Differences in Access to Homophily?

First, we investigate how SCR access is associated with the geographical composition of the reviewer pool. In principle, editors could assign reviewers to avoid or seek out SCRs, leading to a low or high SCR frequency for all countries and, therefore, a low correlation between reviewer pool representation and SCR frequency. In contrast, assigning reviewers without regard to their country would lead to a high correlation, which is what our data reflect. Fig. 2A shows that SCR access for a country’s authors is almost entirely predictable from that country’s representation in the overall reviewer pool (Pearson corr. = 0.9761). The correlation between a country’s number of invited reviewers and SCR access was similarly high (Pearson corr. = 0.9592). Three countries—United States, China, and India—have outsized presence in the reviewer pool (and high SCR probabilities), so we consider them separately in subsequent analyses.

Fig. 2.

Authors from countries well-represented in the reviewer pool are most likely to be reviewed by SCRs. (A) Points are countries of corresponding authors. Countries with less than 100 submissions were excluded. SCR access (y-axis) estimated as described in Eq. 4. Estimates with P > 0.05 (some negative) were set to zero. (B) Wealthier country groups have higher SCR access, excluding the largest submitter from each group (India, China, USA). Among those largest submitters (lighter bars), the wealthier ones have higher SCR access. See Eq. 5 for the model specification, and SI Appendix, Table S9 for regression tables.

The high correlation is consistent with editors inviting reviewers on criteria unrelated to their country, e.g., expertise, resulting in as-if random sampling from the pool of previous reviewers. This suggests that differential access to country homophily is a structural outcome of the collective behavior of editors, reviewers, authors submitting works, as well as organizational policies and procedures like reviewer selection and recruitment practices.

Fig. 2B shows that access to homophily varies substantially across and within country income groups. HIC authors receive SCRs over twice as often as LLMIC authors. USA-, China-, and India-based authors have the highest chance that a reviewer is SCR—30.27%, 26.59%, and 13.69%, respectively. The SCR rates of these well-represented countries are roughly eight to nine times higher relative to other countries in their respective income groups. For instance, LLMIC authors not from India have a 2.66% likelihood a review is from their country, the lowest rate among the country income categories (SI Appendix, Table S9). Similar disparities can be seen in reviewer invitations (SI Appendix, Fig. S4). These results establish the second necessary condition for geographical representation bias—differential access to homophily.

Does Anonymization Reduce Geographical Representation Bias?

Using instrumental variables, we find that hiding author identities did not cause a significant reduction in homophily (among those submissions complying with the anonymization option). The estimated effect of anonymization on SCR positivity, relative to non-SCRs, was to decrease it by 0.67 p.p. (P = 0.0742) (see β3 in Eq. 10 and IV in the regression SI Appendix, Table S6). For robustness, we also estimated SCR positivity separately for voluntarily anonymized and nonanonymized submissions, using manuscript and reviewer fixed effects, and found that the two estimates were not statistically distinguishable (β2 = −0.0263 in Eq. 6, P = 0.1381, OLS in SI Appendix, Table S6). The introduction of the anonymization option itself (i.e., the ITT effect) did not change SCR positivity significantly either (β3 = −0.0100 in Eq. 11, P = 0.3213, ITT in SI Appendix, Table S6). The unresponsiveness of country homophily to the visibility of author information suggests that it is not driven by identity-based biases, pointing toward other mechanisms.

Substantive Significance Toward Geographic Disparities in Publishing.

Having established that geographical representation bias exists, we now perform a back-of-the-envelope analysis of how mitigating this bias would affect publication prospects for different authors. SCRs are associated with 6.78 p.p. (P < 0.0001) higher acceptance rates per SCR on the reviewer panel. If USA authors’ SCR access was moved down from the observed 30.57% (Fig. 2A) to the dataset average of 14.68% (SI Appendix, Table S1), they can be expected to experience a drop in acceptance rates of roughly 1.09 percentage points, i.e., (0.3071 to 0.1468) × 0.0678, from 70.71% to 69.63%. Conversely, consider Vietnam, an LLMIC with little representation in the reviewer pool. Raising Vietnamese authors’ SCR access from the empirical 0.4% to the hypothetical 14.68% is expected to increase its acceptance rate from 60.92 to 61.88%, a 0.96 p.p. increase.

This exercise suggests that substantial reviewer diversification would have fairly modest effects, increasing acceptance by roughly 1 percentage point for the most disadvantaged groups. Importantly, diversification does not make everyone better off, as it would reduce acceptance rates of advantaged groups by the same amount.

Discussion

This study investigated geographical representation bias in scientific peer review—a structural bias hypothesized to result from the lack of geographical diversity among reviewers. There are two necessary conditions for geographical representation bias: Reviewers need to systematically favor research from their own countries, i.e., “country homophily”, and authors from different countries need to have different chances of being assigned to such favorable reviewers, i.e., “access to homophily.” We found strong evidence of homophily. Reviewers from the same country as authors (SCRs) were 4.78 p.p. more likely to review positively compared to non-SCRs on the same manuscript. This study demonstrates author–reviewer country homophily conclusively by ruling out key confounders like submission topic or quality. Data in previous attempts to establish country homophily lacked the granularity to examine individual reviewer behavior, requiring those studies to assume higher acceptance rates were due to more positive reviews from SCRs. Our data enabled us to compare different reviewers of the same manuscript using manuscript fixed effects, thereby controlling for key confounders and directly observing higher reviewer positivity in SCRs.

We also found strong evidence that authors from different countries had dramatically different access to homophily. Authors in USA, China, and India had the highest chance that a reviewer is an SCR—30.27%, 26.59%, and 13.69%, respectively. Low- and lower-middle-income countries (LLMIC) authors not from India had only a 2.66% chance of SCRs, the lowest rate among the country income categories (SI Appendix, Table S9). Although editors appear to invite reviewers based on criteria unrelated to geography, which is arguably a fair, nondiscriminatory practice, the practice advantages authors from countries with many reviewers in the reviewer pool, as it results in them being reviewed by SCRs more often. By analogy with the racial discrimination literature (26), the representation bias is structural and not directly linkable to any individual editor or other actor.

While the country homophily we observed in peer review was substantial, it is important to note that mitigating representation bias is not a “great equalizer” of geographical disparities in publishing. Our back-of-the-envelope calculations suggest that representation bias accounts for roughly 2 percentage points of the up to 30-percentage point disparity in acceptance rates between country income groups (SI Appendix, Fig. S3).

We also considered double-anonymization as a strategy to reduce representation bias. Double-anonymization has proven effective in reducing prestige bias (17, 22, 31, 34, 44, 45), but evidence on whether it reduces country homophily was lacking until now. A policy enabling authors to choose double-anonymous review (instead of the usual single-anonymous) was rolled out in a staggered and arbitrary manner, allowing us to conduct an instrumental variables analysis and thus control for self-selection into double-anonymization. We used policy availability and availability interacted with SCR status as instruments for anonymization and anonymization interacted with SCR status. We found that anonymization had no significant effect on SCRs’ positivity relative to non-SCRs (SI Appendix, Table S6).

The ineffectiveness of double-anonymization suggests that country homophily is not largely driven by identity-based biases, leaving information and tastes as possible explanations. To give some evidence of which of the remaining mechanisms is the primary explanation for our results, we conduct a supplementary analysis of SCR positivity using the COVID-19 disruptions to in-person research dissemination events in 2020 and 2021 as a shock impacting reviewers’ information (SI Appendix, Analyzing SCR Positivity During and After COVID-19 Travel Restrictions). Results are suggestive that reviewers’ private information contributes only a small part to SCR positivity.

Several limitations and open questions remain. First, an overwhelming majority of submissions are from teams of researchers, but we focus on the country of the corresponding author’s institution, assuming that the corresponding author is most relevant to how the submission is evaluated. We find our results are robust when focusing on first authors, single-country teams, and regions instead of countries (SI Appendix, Tables S7, S8, and S10 and Submission Country Robustness Checks). Nevertheless, the measure does not account for researchers who have moved countries. If authors’ and reviewers’ self-reported countries are not reflective of their information and tastes, our estimate of country homophily might be conservative (i.e., downwardly biased). Future analyses with more detailed data on researchers’ geographic trajectories may help elucidate the nature of country homophily.

Second, as large as our data are, they comprise only one area of science and one language. For example, if non-English-language journals attract authors and reviewers who are more geographically concentrated, geographic homophily among reviewers may not get activated and this type of representation bias may not occur. Intensity of representation bias may also vary across fields based on their level of consensus. The physical sciences, the setting of this study, have relatively high consensus on establishing scientific validity, so the “room” for representation bias is arguably smallest there. In fields with less consensus, such as the social sciences and humanities, representation bias may be larger.

Third, our study strongly suggests, but cannot conclusively establish, a causal connection between reviewer diversity and author success. One threat to this connection is that editors may already account for homophily when weighing recommendations from SCRs and non-SCRs. If so, increasing the diversity of reviewers may have little effect on final decisions. Another threat is that achieving a diverse pool of realized reviews can be challenging: Underrepresented countries may lack the expert capacity to supply qualified reviewers, and even if editors invite diverse reviewers, those reviewers must ultimately decide to submit reviews. Settings with less self-selection, such as grant review panels where panelists do not choose which proposals to review, may have a tighter connection between the composition of selected reviewers and successful authors and may show less homophily.

Last, our analyses offer only suggestive evidence of the mechanisms driving SCR homophily (SI Appendix) and rely on several strong assumptions. The instrumental variables analysis of anonymization suggests that identity-based bias does not contribute significantly to homophily, while analysis of COVID-19 disruptions to in-person events suggests a similarly small role for the information mechanism. Further research is needed to establish the mechanisms more decisively.

In sum, our results establish geographical representation bias. To put it simply, the success of scientific papers is determined not only by what is in them but by who evaluates them. Peer reviewers who are geographically close to a paper’s authors are systematically more favorable, and authors from wealthier countries that produce more science are more likely to be reviewed by such favorable reviewers. Anonymizing submissions, which appears to reduce other types of biases, was ineffective at reducing country homophily, and therefore representation bias, in our setting. This suggests that scientific institutions are unlikely to reduce geographical representation bias with “quick fixes,” and instead need to invest in long-term reviewer diversification efforts. However, our back-of-the-envelope calculation shows that even such efforts would do little to equalize the sizable disparities in acceptance rates by geography, as even giving all authors similar access to favorable reviewers would change the rates by about 1%. Other contributors to global disparities in science publishing, such as access to higher-quality training and infrastructure, may be higher priorities for policy interventions.

Materials and Methods

Measures.

Review Positivity.

Different journals gave reviewers slightly different options for recommendations, resulting in 36 distinct options. Following previous literature (46), we recoded these options into a binary indicator Review Positivity where 0 indicates Reject and 1 indicates Accept or Revise-and-resubmit. See SI Appendix, Positive and Not-Positive Codes for Reviewer Response Categories for the exact mapping. Following previous literature, our analyses of reviewer positivity focus on only the first round of review. This is because in the second and subsequent rounds, reviewers may influence each other, which would violate assumptions of independence in our statistical analyses. Additionally, the objective of the reviews may shift, for example, from filtering to improving a submission likely to be accepted.

Income Categories.

Author and reviewer self-reported countries were aggregated into three income categories based on GDP per capita according to the 2023 fiscal year World Bank Country and Lending Groups. The groups are High-income countries (HIC, e.g., USA), UMIC (UMIC, e.g., China), and Low- and Lower-middle-income countries (LLMIC, e.g., India). The full list of countries is displayed in SI Appendix, Countries by Income Group.

Country of submissions.

To assign submissions, which are mostly team-authored, to a single country, we use the self-reported country of the corresponding author. In SI Appendix, Submission Country Robustness Checks, we consider the sensitivity of our results to two other plausible operationalizations. First, we rerun the analysis after defining SCR as a match between reviewer and lead author (as opposed to corresponding author). Second, we restrict our original analysis to submissions with authors all from the same country or region. Regions are defined according to the UN geoscheme as shown in SI Appendix, Countries by Region. We find that estimates of SCR positivity (SI Appendix, Tables S7 and S8) and access to SCRs (SI Appendix, Table S10) in these robustness checks are highly similar to those in our main analysis.

Methods

For simplicity and to enable estimation with the rich set of fixed effects, we use linear probability models with cluster-robust SE. In the specifications included below, manuscripts are indexed by i, reviewers by j, and time by t. Fixed effects are denoted as journal γ, manuscript α, reviewer invite or submission year-month τ, reviewer ρ, and ε is the error term. For regressions without manuscript fixed effects, we include a control for team size, as prior work has shown it to be highly predictive of positive reviews (46, 47). SE are clustered at the level of the fixed effects in each case.

Establishing Homophily Using Linear Models with Manuscript and Reviewer Fixed Effects.

A key challenge in prior work was potential confounding of the association between reviewers’ geography and positivity by submission quality or other characteristics. To address this challenge, we rely on manuscript fixed effects. This research design effectively compares reviews from different reviewers of the same manuscript, accounting for any differences in manuscript quality. Additionally, we use reviewer fixed effects to account for potentially idiosyncratic reviewing standards.

Eqs. 1 and 2 below are used to estimate SCR positivity relative to non-SCRs (SI Appendix, Table S3). Eq. 1 does not include manuscript fixed effects and therefore does not control fully for manuscript quality and other unobserved characteristics. It does include journal and reviewer fixed effects and controls for logged team size, anonymization status, and corresponding author country. This specification is used as a baseline for comparison with prior work and our preferred specification—Eq. 2. Eq. 2 includes manuscript and reviewer fixed effects. Manuscript-level fixed effects absorb the effects of characteristics that do not vary across reviewers of a manuscript, such as team size, anonymization status, and corresponding author country. Estimates from Eq. 2 are reported as our preferred estimate of reviewers’ country homophily. Additionally, we use similar specifications to measure the association between SCR status and whether reviewers accept the invitation to review (SI Appendix, Table S3).

$$\begin{array}{c}{\mathit{Positivity}}_{\mathit{ij}}={\gamma }_i+{\rho }_j+{\beta }_1{\mathit{SCR}}_{\mathit{ij}}+{\beta }_{2}ln(TeamSize{)}_i+{\beta }_3{\mathit{Anon}}_i\hfill \\ +{\beta }_4{\mathit{AuthCountry}}_i+{\epsilon }_{\mathit{ij}},\hfill \end{array}$$ [1]

$${\mathit{Positivity}}_{\mathit{ij}}={\alpha }_i+{\rho }_j+{\beta }_1{\mathit{SCR}}_{\mathit{ij}}+{\epsilon }_{\mathit{ij}}.$$ [2]

Furthermore, we explore whether country homophily is exhibited by all or just some groups of reviewers with a secondary analysis that interacts SCR status with corresponding author country income group. Eq. 3 below adds an interaction term to Eq. 2 to estimate the heterogeneity of SCR positivity across country income groups. Country income categories were coded via dummy coding with high-income countries (HIC) as the reference category, and all categories were included in a single regression. The results of this subanalysis are in SI Appendix, Table S4, and the total effects (main plus interaction) are presented in Fig. 1.

$${\mathit{Positivity}}_{\mathit{ij}}={\alpha }_i+{\rho }_j+{\beta }_1{\mathit{SCR}}_{\mathit{ij}}+{\beta }_2{\mathit{SCR}}_{\mathit{ij}}\ast {\mathit{IncomeCat}}_i+{\epsilon }_{\mathit{ij}}.$$ [3]

Establishing Differential Access to Homophily Using Linear Models.

We operationalize an author’s “access to country homophily” or “SCR access” as the probability that a reviewer of their paper is an SCR. Eqs. 4 and 5 below are specifications that estimate geographic differences in invitation rates and SCR access. We estimate the likelihood of this outcome by corresponding author country (Fig. 2A) or country income group (Fig. 2B and SI Appendix, Fig. S4 and Table S9), expressed as categorical variables, with journal fixed effects and controls for demeaned log team size and anonymization status. This specification was used for the set of invited reviewers in the case of invitation rates (SI Appendix, Fig. S4) and of realized reviewers in the case of access to SCRs (Fig. 2). Statistically significant differences among the geographic categories would indicate an uneven distribution of access to country homophily.

$${\mathit{SCR}}_{\mathit{ij}}={\gamma }_i+{\beta }_1{\mathit{AuthCountry}}_i+{\beta }_{2}ln(TeamSize{)}_i+{\beta }_3{\mathit{Anon}}_i+{\epsilon }_{\mathit{ij}},$$ [4]

$${\mathit{SCR}}_{\mathit{ij}}={\gamma }_i+{\beta }_1{\mathit{IncomeCat}}_i+{\beta }_{2}ln(TeamSize{)}_i+{\beta }_3{\mathit{Anon}}_i+{\epsilon }_{\mathit{ij}}.$$ [5]

Establishing the Causal Effect of Double-Anonymization Using Instrumental Variables.

Physical sciences journals have historically been single-anonymous. In 2020, IOP announced a shift to offer authors double-anonymized peer review at journals in their portfolio (35). A handful of IOP journals were already offering double-anonymization at the time of the announcement. The policy encouraged but did not force authors to make their submissions anonymous. In our dataset, 8,742 (7.72%) submitters selected the double-anonymous option. In consultation with the last author (M.T.), IOP staff made the anonymization policy available across its journals in an arbitrary order, i.e., unrelated to author and reviewer characteristics, on different days in almost all cases. In a few cases, journals were rolled out at similar times if submitted manuscripts were commonly transferred across them. Transferring journals were grouped so that, if a manuscript was submitted for double-anonymous review and was not accepted at one journal, authors could be recommended to submit at a similar journal that also offered double-anonymous review.

To identify the effect of anonymization, it is tempting to compare evaluations across anonymized and nonanonymized submissions. Eq. 6 below makes this comparison between subsets of SCRs, while still controlling for the rich set of fixed effects used earlier, particularly manuscript fixed effects.

$${\mathit{Positivity}}_{\mathit{ij}}={\alpha }_i+{\rho }_j+{\beta }_1{\mathit{SCR}}_{\mathit{ij}}+{\beta }_2{\mathit{SCR}}_{\mathit{ij}}\ast {\mathit{Anon}}_i+{\epsilon }_{\mathit{ij}}.$$ [6]

However, this naive comparison, presented in the second column of SI Appendix, Table S6 (“OLS”), may yield a biased estimate of the effect because certain types of authors may have systematically chosen or avoided anonymizing.

To overcome the endogeneity challenge, we leverage the staggered and quasirandom anonymization option rollout and instrumental variables (IV). IV accounts for the self-selection problem in actual anonymization (the endogenous treatment variable) by using the quasirandom variation generated in it by an auxiliary, non-self-selected variable (the instrument), which in our case is the anonymization option. Intuitively, IV estimates the effect of being offered the anonymization option (intention-to-treat effect, or ITT) and scales it by the proportion of authors who choose to anonymize when given the option, thereby isolating the causal effect of anonymization among those who comply.

We restricted our analysis to papers that received reviews (i.e., editors did not desk-reject; 55.72% of all submissions). Although this restriction means that our analysis is conditioning on a posttreatment outcome, it is unlikely to bias our results because neither the instrument nor the treatment affected the information editors saw and therefore should not have affected their desk-rejection decisions. Consistent with this argument, we found that the instrument did not affect desk-rejection probability (SI Appendix, Table S5).

There are several other assumptions required for IV estimates to be valid in this setting (SI Appendix, Instrumental Variable Approach). The most critical is the exclusion restriction, which states that the instrument (anonymization option) should influence the outcome (review positivity) only through the treatment (anonymization), and not through other channels, such as influencing authors’ decisions to submit or how editor’s desk reject or assign reviewers. This assumption cannot be validated directly. Eq. 7 below displays the specification used to test for some threats to the exclusion restriction as discussed in the SI Appendix. This specification examines the effect of anonymization option availability on four outcomes separately and includes fixed effects for journal γ and submission year-month τ. The first three outcomes pertain to the submissions that went to review: logged team size, probability of a submission being reviewed by at least one SCR, and the total number of realized reviewers. The fourth outcome is the probability of not being desk rejected (i.e., going to review), which includes submissions not included in the rest of our analyses. The results from these validity tests are presented in SI Appendix, Table S5.

$${\mathit{Outcome}}_{\mathit{ijt}}={\gamma }_i+{\tau }_t+{\beta }_1{\mathit{PolicyAvail}}_{\mathit{it}}+{\epsilon }_{\mathit{ij}}.$$ [7]

If policy availability is a valid instrument for anonymization status, then policy availability interacted with SCR status is a valid instrument for anonymization interacted with SCR status (48), which is our effect of interest. Thus, our instrumental variables approach addresses two endogenous variables, i.e., anonymization and anonymization interacted with SCR status with two instruments, i.e., policy availability and policy availability interacted with SCR status. This is implemented in a two-stage least squares (2SLS) procedure as described in Eqs. 8, 9, and 10 below. Since anonymization status does not vary within-manuscript, comparisons must be made across manuscripts, thus precluding the use of manuscript fixed effects in this analysis. Eqs. 8 and 9 are the first stages, where each of the two endogenous variables—anonymization status and anonymization interacted with SCR status—are separately regressed on two instruments, i.e., policy availability and the interaction between policy availability and same-country status, journal and reviewer fixed effects, a reviewer invite year-month fixed effect τ, and controls for logged team size and author country. Eq. 10 is the second stage, where review positivity is regressed on the predicted values of the endogenous variables and the same covariates as in the first-stage regressions. The results of this secondary analysis are in the third column of SI Appendix, Table S6 (“IV”).

First Stages:

$$\begin{array}{c}{\mathit{Anon}}_{\mathit{ijt}}={\gamma }_i+{\rho }_j+{\tau }_t+{\beta }_1{\mathit{PolicyAvail}}_i+{\beta }_2{\mathit{SCR}}_{\mathit{ij}}+{\beta }_3{\mathit{PolicyAvail}}_i\ast {\mathit{SCR}}_{\mathit{ij}}\\ +{\beta }_{4}ln(TeamSize{)}_i+{\beta }_5{\mathit{AuthCountry}}_i+{\epsilon }_{\mathit{ijt}},\end{array}$$ [8]

$$\begin{array}{c}Anon\ast SC{R}_{\mathit{ijt}}={\gamma }_i+{\rho }_j+{\tau }_t+{\beta }_1{\mathit{PolicyAvail}}_i+{\beta }_2{\mathit{SCR}}_{\mathit{ij}}+{\beta }_3{\mathit{PolicyAvail}}_i\ast {\mathit{SCR}}_{\mathit{ij}}\\ +{\beta }_{4}ln(TeamSize{)}_i+{\beta }_5{\mathit{AuthCountry}}_i+{\epsilon }_{\mathit{ijt}}.\end{array}$$ [9]

Second Stage:

$$\begin{array}{c}Positivit{y}_{ijt}={\gamma }_i+{\rho }_j+{\tau }_t+{\beta }_{1}SC{R}_{ij}+{\beta }_2{\widehat{Anon}}_{ijt}+{\beta }_3{\widehat{Anon\ast SCR}}_{ijt}\\ +{\beta }_{4}ln{(TeamSize)}_i+{\beta }_{5}AuthCountr{y}_i+{\epsilon }_{ijt}.\end{array}$$ [10]

From a policy perspective, the effect of the anonymization option on evaluations is interesting in itself. Double-anonymization can be costly to enforce, so a voluntary policy represents a potential approach for balancing goals of bias reduction and enforcement effort. The policy’s effect can be interpreted as the intent-to-treat (ITT) effect, which only requires that the anonymization option is randomly assigned. Eq. 11 below is the specification for the ITT analysis, which takes a similar form to Eq. 10. Here, we still have journal, reviewer invite year-month, and reviewer fixed effects, as well as controls for team size and author country, but we replace anonymization with the availability of the voluntary anonymization policy to estimate its effect on SCR positivity (β3). Results for this subanalysis are in the first column of SI Appendix, Table S6 (“ITT”).

$$\begin{array}{c}{\mathit{Positivity}}_{\mathit{ijt}}={\gamma }_i+{\rho }_j+{\tau }_t+{\beta }_1{\mathit{PolicyAvail}}_i+{\beta }_2{\mathit{SCR}}_{\mathit{ij}}+{\beta }_3{\mathit{PolicyAvail}}_i\ast {\mathit{SCR}}_{\mathit{ij}}\\ +{\beta }_{4}ln(TeamSize{)}_i+{\beta }_5{\mathit{AuthCountry}}_i+{\epsilon }_{\mathit{ij}}.\end{array}$$ [11]

Back-of-the-Envelope Analysis on Contribution of Geographical Representation Bias to Disparities.

We estimate how diversifying the review pool would affect inequality in acceptance rates using a simple back-of-the-envelope calculation. Eq. 12 is the specification used to estimate the association between the number of SCRs on a review panel and likelihood of acceptance. Note that the outcome and SCR are indexed at the manuscript-level i. SCR is not a binary variable in this specification, but rather the number of SCRs associated with a submitted manuscript. This specification includes journal fixed effects and controls for logged team size and corresponding author country.

$${\mathit{Accepted}}_i={\gamma }_i+{\beta }_1{\mathit{SCR}}_i+{\beta }_{2}ln(TeamSize{)}_i+{\beta }_3{\mathit{AuthCountry}}_i+{\epsilon }_i.$$ [12]

Taking the estimates from this specification, we present a thought experiment proxying reviewer diversification policies where all submissions have an average level of SCR access. We do not assume that this policy is feasible to implement, but using such an extreme example can give a sense of the bounds of diversification’s effectiveness.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

Deidentified data and replication code can be retrieved from the University of Michigan’s Deep Blue Data repository (49). This anonymized data and replication code have been deposited in Deep Blue Data (https://doi.org/10.7302/VVJ6-X579). The following steps were taken to ensure anonymity. Journal, manuscript ID, reviewer name, year-month of manuscript submission, year-month of invitation, and year-month of submitted review were deidentified using randomly generated, 10-character strings. Unique countries in the dataset were gathered from the union of corresponding and lead authors’ and reviewers’ countries. Countries were then deidentified using randomly generated, 10-character strings, so that they are consistent across variables (i.e., USA code is same for author and reviewer country columns). For each numerical variable, a random 1% of values were shuffled around (including missing values). Team size, the only non-binary numerical variable, was logged and demeaned in the main paper analysis. This variable was also converted to quintiles for deidentification.