Surgical research journals - Under review: An assessment of diversity among editorial boards and outcomes of peer review

Abstract

Background

The COVID-19 pandemic exposed racism as a public health crisis embedded in structural processes. Editors of surgical research journals pledged their commitment to improve structure and process through increasing diversity in the peer review and editorial process; however, little benchmarking data are available.

Methods

A survey of editorial board members from high impact surgical research journals captured self-identified demographics. Analysis of manuscript submissions from 2016 to 2020 compared acceptance for diversity, equity, and inclusion (DEI)-focused manuscripts to overall rates.

Results

25.6% of respondents were female, 2.9% Black, and 3.3% Hispanic. There was variation in the diversity among journals and in the proportion of DEI submissions they attract, but no clear correlation between DEI acceptance rates and board diversity.

Conclusions

Diversity among board members reflects underrepresentation of minorities seen among surgeons nationally. Recruitment and retention of younger individuals, representing more diverse backgrounds, may be a strategy for change. DEI publication rates may benefit from calls for increasing DEI scholarship more so than changes to the peer review process.

1. Introduction

Disparate health outcomes for patients from racial and ethnic minority backgrounds have been consistently demonstrated across specialties, insurance coverage, genders and at all socioeconomic levels,1, 2, 3, 4 However, it took the global COVID-19 pandemic to irrefutably confirm that racism is a public health crisis in America.^{3 , 5 , 6} Patients of color experienced higher rates of infection, hospitalization, and death due to COVID.^{3 ,} 5, 6, 7 Asian Americans faced additional discrimination and hate crimes associated with public sentiment that they were responsible for spreading the virus.^{2 , 8} Meanwhile, highly publicized examples of ongoing police brutality against Black Americans, such as the killings of Ahmaud Aubery, Breonna Taylor, and George Floyd, inspired protests around the world, including “White Coats for Black Lives.“^{1 , 9} Healthcare providers and researchers increasingly acknowledged their role in structural racism and that choosing to remain agnostic perpetuates these historical norms. Calls were put forth from all levels of academic medicine to increase organizational efforts to improve diversity, equity, and inclusion (DEI) in medical research and publication.1, 2, 3 ^{, 5 , 6 , 9 , 10}

The case for promoting diverse perspectives in the peer review and editorial process was further strengthened when the Journal of the American Medical Association (JAMA) promoted a new podcast episode with a tweet that read: “No physician is racist, so how can there be structural racism in healthcare?“¹¹ The tweet and the podcast itself, which inaccurately attributed healthcare disparities to socioeconomic status rather than racism, faced immediate backlash. Over 10,000 individuals signed a petition to restructure JAMA's editorial leadership with input from black, indigenous, and people of color (BIPOC) patients and healthcare providers.¹² Demands for increased representation among journal leaders and peer reviewers were not limited to race and ethnicity. The publication of a paper titled “Prevalence of unprofessional social media content among young vascular surgeons”¹³ also triggered a campaign for increased gender diversity. The manuscript, which used subjective criteria that was disproportionately critical of women such as “wearing provocative Halloween costumes” and “posing in bikinis,” was published in a journal with predominantly male leadership; this publication was regarded as a failure of the peer review process. The physician community briskly responded on social media with posts emblazoned with the viral #MedBikini tag, featuring photographs of themselves in swimsuits and accompanied by calls for increasing diversity in the peer review process.¹⁴

In the wake of these events, “A Call to All Surgery Editors for Diversity in the Editorial and Peer Review Process” initiated by Drs. Melina Kibbe and Julie Freischlag, was signed by the editors of 121 surgical research journals as a pledge to “strive for diversity in the peer review process and among the editorial boards and editorial leaders”.^{15 , 16} Likewise, several individual journals published their own statements of support for increasing DEI within their own organizations.17, 18, 19

In order to evaluate whether progress is being made regarding these pledges, empirical study is necessary. There is currently a dearth of objective data about diversity among journal leadership and in the peer review process. Prior studies have quantified gender diversity among editorial boards for surgical research journals, revealing that as of 2020 an estimated 14.8% of editorial board members were female²⁰ along with 6.7% of editors-in-chief.¹⁵ To our knowledge, no studies in the surgical literature have quantified diversity among editorial boards based on race, ethnicity, or other demographics nor are there studies examining the outcomes of the peer review process for DEI literature.

We therefore undertook this study to ascertain two aspects of diversity in peer-reviewed surgical literature:

• 1.What is the current level of diversity among editorial boards for surgical research journals?
• 2.Over the past five years, what has been the acceptance rate for DEI-related manuscripts, and does that rate vary from surgical research in general?

2. Methods

This study was reviewed by the University of Michigan Institutional Review Board and granted exemption status (Protocol #HUM00186872).

The study consisted of two components:

• 1)A demographic survey of editorial board members
• 2)An analysis of DEI content among manuscripts submitted for review

The journals targeted for this study were selected by reviewing the surgery journal ranking lists published by Clarivate (jcr.clarivate.com), Scopus CiteScore (https://www.scopus.com/sources), and Scimago Journal & Country Rankings (https://www.scimagojr.com) in conjunction with insight from the research team's senior investigators (GS, HC) to identify the highest impact research journals with a broad focus in general surgery. The study protocol was also discussed with two editors of journals based in Canada and the United Kingdom, respectively, who independently raised concerns that the terminology used to categorize race and ethnicity in the United States is not uniform internationally. Therefore, the decision was made to include only journals with editorial boards based predominantly in the United States.

Seven journals met these criteria, and an invitation to participate was emailed to their editors-in-chief. Nonrespondents received a follow up email, and an invitation was also emailed to their journal's managing editor. Though journal editors consented to have their journal named here as participants, results were reported as Journal A, Journal B, etc. to enhance anonymity. Each journal's editor-in-chief(s), however, was provided with a copy of their own results to distribute at their own discretion.

2.1. Survey

A ten-question survey (Supplemental 1) was electronically distributed using Qualtrics (Provo, UT). The survey queried about demographics, including position on the editorial board, age, gender, and race/ethnicity. The survey also asked respondents whether they identified as underrepresented in medicine based on any demographic including race, ethnicity, sexual orientation, gender identity, religion, or other. The method of distribution of the survey to editorial board members was tailored to the preferences of each journal's editor-in-chief(s); they could either distribute the survey themselves using an anonymous link to the survey or have the research team email editorial board members directly. A second reminder email was sent one week after the initial invitation. Results were described using summary statistics to evaluate journals individually and en masse.

2.2. Manuscript analysis

Each journal provided the research team with a report generated by Editorial Manager® submission management system that included the title, keywords, date of submission, and final editorial decision for all manuscripts submitted 2016 to 2020.

The manuscripts were then classified as DEI based on the protocol described in Fig. 1 . The initial screen was conducted by an automated search in Microsoft Excel using a list of terms based on the “MEDLINE®/PubMed® Health Disparities and Minority Health Search Strategy.“²¹ In order to capture potential variations of these terms, we drafted a list of 104 single or partial-word terms that fully overlapped with the MEDLINE® list but would produce a more sensitive search (Supplemental 2). For example, the terms “gender diversity” and “sexual diversity” on the MEDLINE list were represented on our modified-MEDLINE search protocol with four separate terms: “gender”, “sex”, “diversity”, and “diverse”. The terms “Medicaid”, “Medicare”, and “Affordable Care Act”, which were not included in the MEDLINE search protocol, were included in our modified-MEDLINE protocol.

Fig. 1.

Manuscripts were screened using a robust screening protocol based on a MEDLINE®/PubMed® search strategy²² to identify DEI-related manuscripts.

Manuscripts that did not include a modified-MEDLINE term in the title or keywords were designated as “Non-DEI”. Those containing a search term were then subjected to manual screening by one of two researchers (EMW or DB). In this stage, manuscripts were designated as either “DEI” or “Non-DEI” at the discretion of the screeners. For example, no variations on the term “bias” are included in the MEDLINE keywords, but “racial bias”, “gender bias” and “unconscious bias” are all clearly associated with DEI literature and were therefore designated as DEI during this screen.

Additional clarifying criteria were defined based on consensus discussion among the research team for categories of manuscripts that were frequently flagged as potentially DEI by the modified-MEDLINE screen but were not felt by the researchers to address DEI issues. These included:

• •Dimorphisms in outcomes secondary to female gender were considered DEI, however manuscripts describing predominantly gender-specific diseases, complications, or treatments (ex. breast cancer, erectile dysfunction, transvaginal cholecystectomy) were not.
• •Global Health research received a DEI designation only if some additional aspect of DEI was also described (ex. underserved population, resource limited setting). Merely conducting research outside of the United States was insufficient to be deemed DEI.
• •Dimorphisms due to old age received a DEI designation only if some additional aspect of DEI was also described or if the manuscript specifically investigated instances of “ageism”

For any manuscripts that a researcher felt might be borderline or questionable, a designation was assigned based on their best judgment and the manuscript was sent for review by a second researcher to independently make their own designation. If the two screeners disagreed on the designation, the manuscript was sent to a third research team member (RM) as the tie-breaker.

Frequencies were described using summary statistics and Chi-square tests were applied to determine the significance of any differences in acceptance rates observed. Because the overall acceptance rate was likely to vary between journals, it would not be appropriate to directly compare DEI acceptance rates. Therefore, acceptance rates were also reported as an odds ratio comparing DEI manuscripts to non-DEI manuscripts.

3. Results

Editors from six journals (Annals of Surgery, American Journal of Surgery, JAMA Surgery, Journal of Surgical Education, Journal of Surgical Research, and Surgery) out of seven responded to the invitation and consented to participate in the demographic survey component of the study. Two distributed the survey to editorial board members via an email from their office while four opted for the survey to be emailed to potential respondents from the research team directly. Five of these journals also consented to participate in the manuscript analysis component of the study.

3.1. Demographics

240 (of 449, 53.5%) individuals responded to the demographic survey. The response rate between journals varied from 38.5% to 97.6%. The average time to complete the survey was less than 1 min.

The demographic distribution for gender and race/ethnicity is shown in Fig. 2 for each journal as well as for the entire cohort. When stratifying responses by age group, we observed increasing diversity among younger respondents, both for gender and race/ethnicity (Fig. 3 ).

Fig. 2.

Variations in gender and racial/ethnic diversity across six surgical research journals.

Fig. 3.

Decreasing Age is associated with increasing diversity for both gender and race/ethnicity. This is consistent with changing demographics over time among surgeons nationwide based on data available from the AAMC.23, 24

Respondents were asked whether they personally identified as underrepresented in medicine based on any demographic criteria, and were asked to select from a list of choices: race, ethnicity, gender identity, sexual orientation, religion, or other. While the majority of respondents did not identify as underrepresented, additional diversity was contributed in all categories, including two individuals who wrote in “disability” (Fig. 4 ). Among female respondents, 50% (30/60) identified as underrepresented in medicine compared to 20.1% of men. 100% (7/7) of Black/African American respondents, 75% (6/8) of Hispanic/Latinx respondents, 33.3% (2/6) of Middle Eastern/North African respondents, and 25% (9/36) of Asian respondents stated that they identified as underrepresented based on their race or ethnicity.

Fig. 4.

Race, ethnicity, gender identity, sexual orientation, religion, and disability all contribute to diversity on editorial boards, however most respondents do not identify as underrepresented.

3.2. DEI analysis

After excluding all manuscripts which did not meet inclusion criteria, a total of 41,810 manuscript titles and accompanying keywords were analyzed for DEI content. 18,618 (44.5%) of these manuscripts were flagged for manual review. 828 manuscripts were ambiguous and sent for a second review, of which only 86 yielded a disagreement (Cohen's κ = 0.78) requiring a tie-breaking review. The results for each journal and the pooled data are described in Table 1 . For all journals, the DEI acceptance rate was significantly higher than the non-DEI acceptance rate (χ²-test, p < 0.001) and the odds of acceptance for DEI manuscripts was not significantly different among journals. However, the proportion of DEI among the manuscript submissions to each journal did significantly vary as seen in Table 1. This input resulted in the final proportion of DEI manuscripts among accepted manuscripts also being significantly different among journals.

Table 1.

Results of peer review demonstrates significantly higher odds of acceptance for DEI manuscripts as compared to non-DEI.

Journal	Total Submissions	Overall Acceptance Rate	Proportion of Submissions That Were DEIa	DEI Acceptance Rate	Non-DEI Acceptance Rate	DEI vs. Non-DEI Acceptance Odds Ratio [95% CI]	Proportion of Accepted Manuscripts That Are DEIa
A	n = 3541	28.47% (n = 1008)	2.80% (n = 99)	44.44% (44/99)	28.01% (964/3442)	2.06 [1.37–3.08]	4.37% (44/1008)
B	n = 8801	25.78% (n = 2269)	2.01% (n = 177)	37.29% (66/177)	25.54% (2203/8624)	1.73 [1.27–2.36]	2.91% (66/2269)
C	n = 10,357	27.79% (n = 2878)	2.42% (n = 251)	52.99% (133/251)	27.16% (2745/10,106)	3.02 [2.35–3.89]	4.62% (133/2878)
D	No Data Available
E	n = 14,832	20.67% (n = 3066)	2.92% (n = 433)	37.64% (163/433)	20.16% (2903/14,397)	2.39 [1.96–2.92]	5.32% (163/3066)
F	n = 4279	42.63% (n = 1824)	7.10% (n = 304)	61.84% (188/304)	41.16% (1636/3975)	2.32 [1.82–2.95]	10.31% (188/1824)
All	n = 41,810	26.42% (n = 11,045)	3.02% (n = 1264)	46.99% (594/1264)	25.78% (10,451/40,544)	2.55 [2.28–2.86]	5.38% (594/11,045)

^aChi-square analysis shows significant variation between journals p < 0.05.

Evaluating the data for any trends over time (Fig. 5 ), we observed that the acceptance rate for DEI has remained fairly stable, while the acceptance rate for non-DEI work has decreased. As a result, the overall proportion of DEI represented in the accepted literature has increased, particularly over the 2019–2020 period.

Fig. 5.

The proportion of accepted manuscripts that are DEI-related has increased over time in the context of decreasing acceptance rates overall.

4. Discussion

This analysis of diversity among six high impact American general surgery research journals demonstrates disparities in minority race/ethnicity and gender representation when compared to the American patient population but are consistent with the current surgeon workforce. However, some journals did show higher (and lower) rates of diversity for certain demographics. While an explanation for variation among journals falls outside the scope of this study, this observation suggests recruitment or retention practices may exist to generate levels of diversity that transcend national surgeon demographics.

Acceptance rates for DEI-related manuscripts exceeded non-DEI manuscripts at all journals, with an overall odds ratio of 2.55 (CI 2.28–2.86). When compared to their baseline acceptance rates for non-DEI work, no journals stood out as having a significantly higher or lower DEI acceptance rate. When looking at accepted manuscripts, however, there was a significant difference in the final proportions that were DEI-related. This variation appears to result not from the peer review process, but rather from variations in the proportions of manuscripts initially submitted for review.

With respect to the composition of the editorial boards, our study found only 25.6% of participating editorial board members are women compared to approximately half (50.8%) of the U.S population.²⁵ Similarly, 13.4% of the U.S population identifies as black race, and 18.5% as Hispanic,²⁵ whereas our study population was only 2.9% and 3.3% respectively. The demographics we observed are highly reflective of demographics within the current surgery workforce in America. The 2019 American Association of Medical Colleges (AAMC) report of physician demographics notes that 22% of active general surgeons are female,²⁴ while 5.4% identify as black and 5.8% Hispanic.²³ Therefore, this disparity likely arises predominantly from the poor representation of racial/ethnic minorities and women in the surgeon population from which board members are recruited rather than being solely attributable to the journals themselves. That said, gender diversity at some journals notably exceeded the proportion of women in the population of general surgeons, and likely indicates intentional efforts to improve diversity in this regard.

The observed trend of increasing gender and racial diversity among younger board members is also reflective of national surgeon demographics (Fig. 3). Coupled with a study of similar surgical journals that showed the proportion of women on editorial boards increased from 5% in 1997 to 19% in 2017,²⁶ we would anticipate this trend of increasing diversity to continue as junior members become more senior. The decrease in diversity among older individuals, however, may represent a retention problem similar to the “leaky pipeline” described frequently in academic surgery.²⁷

We could not establish a clear correlation between increased editorial board diversity and DEI acceptance rates, in part due to the sample size of only six journals, but also due to the lack of an accepted metric for quantifying diversity. Some strategies exist within the literature, for example, the Blau Index provides a measure of diversity by calculating the probability that two randomly selected individuals from the group share the same demographics.²⁸ Alternatively, a more simplistic approach would be to report the proportion of all minorities (i.e. women, people of color) as compared to the majority group (i.e. white men). These strategies, however, rely on assigning individuals to a discrete set of mutually exclusive categories. Applying these strategies to our data set would omit the contribution of other potentially diverse viewpoints, for example, that of a white male who also identifies as LGBTQ or disabled. Asking participants directly whether they self-identify as a minority did not provide an accurate marker for diversity either though, as half of our female respondents – clearly an underrepresented demographic – did not themselves identify as such.

The sample size may also raise concerns about poor generalizability, however, it is not clear that expanding this study to additional journals would have actually improved the generalizability of the results. These journals were selected due to their broad scope of surgical topics, and it may well be appropriate for the proportion of DEI manuscripts to be considerably different for journals with a narrow practice scope. Given the high readership of these journals, and an analysis of over 41,000 submissions and 11,000 accepted manuscripts, this study does represent a large proportion of the high impact literature in the field.

The generalizability of the demographic survey component of this study, however, would likely benefit from the inclusion of additional journals. Prior studies of gender diversity among editorial boards have included larger numbers, both in surgery and in other medical fields, however, they predominantly relied on the application of gender norms to names or photographs and did not explore additional demographic data.^{14 , 20 , 26 ,} 29, 30, 31 A major strength of this study's design is that all demographic data was self-reported, which is considered best practice,³² although this does negatively affect response rate. With a response rate of 53.5% there exists potential for a selection bias, particularly because respondents from underrepresented demographic groups may have a more vested interest in DEI and be more likely to respond. However our findings are fairly consistent with both the prior work examining gender diversity in this population^{20 , 26} and national surgeon demographics,^{23 , 24} so significant bias in these results seems unlikely. Furthermore, when attempting to quantify individuals' potential to bring diverse beliefs and perspectives to the table, our research team felt it was more important to capture how those individuals self-identify, even though it was at the expense of capturing a complete data set.

An additional limitation of this study was inherent to the manuscript analysis protocol. Because the protocol relied on the detection of certain keywords, it is likely some DEI manuscripts were missed. However, using the MEDLINE list exactly as written would have only yielded 64.2% (826/1286) of the DEI manuscripts we identified, suggesting our search protocol was considerably more sensitive than the proposed standard. Of additional concern is that searching for specific keywords does not consider the full content of those manuscripts. It is possible that a third category of “anti-DEI” manuscript may exist, which would include racist or sexist manuscripts. Because keywords for such papers would have significant overlap, these theoretical manuscripts may have been miscategorized as DEI. No accommodations for this could be reasonably made given the magnitude of the data set being screened; however, neither researcher noted any manuscripts they suspected of being anti-DEI.

The results of this study present an optimistic future for DEI in surgical research literature. The comparatively high DEI acceptance rates and an overall trend showing an increasing proportion of DEI among accepted manuscripts reflects a commitment from across the field to publish on DEI issues. Our findings also reveal some potential strategies to assist surgical journals in making good on their pledges. The increasing diversity found among younger surgeons and trainees may provide the clearest opportunity to increase diversity among editorial boards: a policy of recruiting junior editorial board members and providing support and mentorship to retain them in the academic pipeline would more rapidly increase diversity on boards because they provide a more diverse pool of candidates.³³ Additionally, journals seeking to increase their commitment to DEI should not just focus on recruiting more diverse board members but should also make efforts to recruit additional DEI-focused manuscript submissions.

These conclusions may be further strengthened through additional study. A survey of self-identified demographics that includes race and ethnicity and involves a larger number of journals should be pursued. An exploration is also needed to identify what factors drive the observed variation among surgical journals in board member demographics and in rates of DEI submissions.

5. Conclusion

Compared to the American patient population, the demographics among editorial board members for high impact general surgery research journals reflects the same patterns of underrepresentation historically seen in surgery, although some journals have demonstrated the ability to cultivate higher rates of diversity among their boards. Across all journals, the outcomes of peer review invariably demonstrated higher odds of acceptance for DEI manuscripts as compared to other works. Additionally, discrepancies in the proportions of DEI-focused manuscripts submitted to the different journals resulted in varying levels of DEI representation among the published manuscripts. The findings of this study can be used to guide editorial board recruitment practices and emphasize the importance of intentionally pursuing scholarly work in DEI.

Declaration of competing interest

We have no other conflicts of interest or financial disclosures.

Appendix A. Supplementary data

The following are the Supplementary data to this article: