Promoting inclusive metrics of success and impact to dismantle a discriminatory reward system in science

Abstract

Success and impact metrics in science are based on a system that perpetuates sexist and racist “rewards” by prioritizing citations and impact factors. These metrics are flawed and biased against already marginalized groups and fail to accurately capture the breadth of individuals’ meaningful scientific impacts. We advocate shifting this outdated value system to advance science through principles of justice, equity, diversity, and inclusion. We outline pathways for a paradigm shift in scientific values based on multidimensional mentorship and promoting mentee well-being. These actions will require collective efforts supported by academic leaders and administrators to drive essential systemic change.

This Essay argues that success and impact metrics in science are based on a system that perpetuates sexist and racist ‘rewards’ by prioritizing citations and impact factors; the authors advocate shifting this outdated value system to advance science through principles of justice, equity, diversity, and inclusion.

Overview

“The most dangerous phrase in the language is: We’ve always done it this way.”

—Rear Admiral Grace Hopper

Experiencing challenges related to justice, equity, diversity and inclusion in science is universal across disciplines [1]. Strong evidence highlights the breadth of biases, yet action-based solutions have not been broadly adopted, and systemic change remains elusive. Under the pressure for “objective” metric-based “success” and “impact,” multiple biases are perpetuated in science. For example, flawed interpretations of data with damaging conclusions are published [2,3], including papers requiring retraction [4]. Here, our interdisciplinary, international team of women scientists publicly acknowledges and denounces the pervasive sexist and racist structures persisting within the value systems, which typify science. We further advocate to accelerate the pace of positive change in science by building on the advancements made through systemically marginalized groups, including the prior and ongoing efforts of women, Black people (we recognize this language may not be used commonly internationally and use it here to explicitly acknowledge that systemic racism disproportionately affects the lives of Black people, particularly within the United States), indigenous people, people of color, lesbian, gay, bisexual, transgender, queer, and others (LGBTQ+), and their allies (e.g., [5–9]). We (1) highlight long-standing problems associated with narrow definitions of success and impact in science; (2) advocate for expanding measures of success beyond citations to value the multifaceted nature of scientific impact (Fig 1); and (3) propose a model that values the recruitment and retention of scientists from diverse backgrounds through building safe and healthy work environments (Fig 2).

Fig 1. Science is suffering from observational bias in our value system.

This bias is analogous to the streetlight effect where (A) citations are valued because that is where we look, despite the fact that they perpetuate gender and racial biases as metrics of success. We advocate for (B), an expanded view of success and impact that is multifaceted and includes critical areas of mentorship, inclusion, and diversity.

Fig 2. Here, we show 2 models for the disciplines of STEMM.

We argue that the narrow mentorship model based on the top-down “publish or perish” approach to success and impact facilitates processes that lead to a reduction in diversity and innovation (illustrated by the inverse gray pyramid) and a detrimental STEMM culture that supports a limited subset of scholars. By contrast, a multidimensional mentorship model supported by those in leadership roles (e.g., by university and college presidents, chancellors, and provosts) working across academic institutions will incorporate diverse measures of success and impact to create system-wide change (illustrated by the purple pyramid). We argue that the latter approach can lead to increased innovation that will transform STEMM culture where processes, which support the 2 models, and outcomes of each, are side highlighted within the oval shapes. STEMM, Science, Technology, Engineering, Mathematics, and Medicine.

It is imperative for those holding positions of power, privilege, and visibility to take informed and strategic action rather than only engage in a performative manner. Strong actions that support justice, equity, diversity, and inclusion are essential for the accelerated evolution of the value system in science. Collectively, these changes are key to generating a greater capacity for innovation, which is essential for addressing the challenges of the present and future, such as pandemics and climate change (Fig 2).

Pivoting the paradigm to ensure equitable evaluation in science

(1) Citation counts are biased

One of the many detrimental constructs underpinning academic science is the “publish or perish” model that celebrates the quantity of publications, citation rates, and impact factor scores as the primary, and often sole, indicators of success and impact [10–12]. Citation metrics, which have been widely used across most research areas due to their quantitative nature and easy estimation, influence career advancement at all levels including graduate opportunities, funding success, career positions, awards, distinctions, and tenure and promotion. However, a lack of diversity among the most cited scientific authors is driven by historical demographics of faculty and those in academic leadership positions [13–16]. While there have been recent successes in increasing diversity among trainees and early-career researchers [8,9], differential recruitment, retention, and promotion rates with respect to age, sex, gender, race, and ethnicity continue to perpetuate the lack of diversity among all career levels of scientists [14,17–19]. This issue is self-perpetuating due to reliance on citation metrics, which reflect deeply entrenched biases and exclusionary networks that disadvantage systemically marginalized groups, and these citation metric biases continue to rise globally [20].

Sexism in science publishing is ubiquitous. Women (throughout this manuscript, we use the term “women,” by which we intend to respectfully include and acknowledge the experiences and challenges of all who identify as women and/or womxn and also acknowledge that these and other challenges also exist for nonbinary individuals) are uniformly less cited than men, even though this issue is consistently well acknowledged [21–24]. Recently, the citation gap between genders was found to be as large as 30% across 13 Science, Technology, Engineering, Mathematics, and Medicine (STEMM) disciplines [16], and this gap has been documented across a breadth of journals [25]. These patterns are partially explained by men exhibiting higher rates of self-citation [22,23] and women having shorter career lengths than men [16]. However, decisions on whom to cite may also reflect exclusionary scientific networks that coalesce at scholarly meetings and conferences that, despite recent efforts in improving diversity among participants [26–28], primarily cater to established white men from privileged universities [28–31]. In addition, in comparison to men, women receive more manuscript rejections [32–34], are less likely to be published in prestigious journals (which typically have high citation rates) [35], and are less likely to be invited to write commentaries [36]. These issues may stem from women’s scholarly writing being held to a higher standard than men’s by editors and peer reviewers, placing penalties on women’s productivity, with excessive time spent reworking old research at the cost of conducting new research [37]. Sex-specific differences in manuscript decisions may also arise from conscious and unconscious biases that can impact reviewer assignment [38] and peer review scores [27]. Moreover, the impact of unprofessional peer reviewer comments, defined as “any statement that is unethical or irrelevant to the nature of the work” [39], have disproportionately negative effects for women and nonbinary people relative to men [39]. We also would like to acknowledge that biases experienced by women are likely to be exacerbated for nonbinary individuals, and, so far, little attention has been given to the effects on these groups (but see positive change [40]).

Pervasive racism in science also drives substantial and systemic biases in publication rates, citation rates, and editorial positions [24]. Publication-related metrics show distinct patterns of bias against racially and/or ethnically diverse scientific teams, which experience more than 5% lower acceptance rates and fewer citations than less diverse author teams [41]. Citational segregation—where authors prefer citing authors from the same racial/ethnic group(s)—has been demonstrated with white authors citing other white authors more frequently [24]. This particular bias further reduces the circulation and intellectual acknowledgement of nonwhite scholars’ work and the diversity of viewpoints they bring. Additionally, “high-quality” research is implicitly associated with high-income countries [42], thereby limiting the dissemination of research by scientists from lower-income countries. Moreover, because 98% of scientific journals are published in English, success is related to English proficiency or access to additional editorial support. Scholars who are not fluent in English are at a distinct disadvantage in the publication process, further exacerbating the global gap in citations and research dissemination [43].

Together, gender, racial, and other biases interact and accumulate, often elevating cisgender white males to much higher status than deserved given their contributions to science [44,45]. As such, the unwavering focus on citation-based metrics as indicators of success ignores the breadth of scientific evidence showing these metrics are unreliable, inaccurate, and damaging. While the history, outcome, and current treatment of these biases can vary across disciplines, it is clear that continued use of these metrics perpetuates substantial gender, racial, and ethnic biases, as well as reduced representation of diverse scholarship.

Many efforts to improve diversity in science disciplines have not yet been successful [46–49]. In fact, gender and racial citation biases remain stable or have even worsened over the last half century [16,21–24,41,50], highlighting that efforts to change the system have, by and large, failed to remove systemic biases. Clearly, assessing scientific impact, and thereby assigning value to an individual’s scientific contribution, exclusively—or even primarily—through citations of peer-reviewed literature reflects and amplifies the existing numerous biases that remain embedded within science. Reliance on citation metrics as the primary gauge of impact will continue to limit the advancement of marginalized groups and diminish their scientific contributions [24], representing a loss of diverse talent, perspectives, and approaches.

(2) Expanding scientific impact beyond citations

Ignoring the breadth of areas where scientists have strong impacts creates an unduly narrow view of the many avenues through which scientists can contribute to intellectual advances, applied science, and equitable communication and translation of science to the public (Fig 1A). This narrow view excludes the real-world impacts within the scientific system (Fig 1B). Even if citation metrics were not biased, using citations as a proxy for success supports the false paradigm that scientists lack impact if they do not (or cannot) publish and/or have chosen “alternative” career paths—a phrase that falsely suggests that academic roles are the only dominant or valued careers for scientists [51,52]. Notably, scholars holding academic positions with high teaching loads, mentoring responsibilities, service requirements, and/or administrative work, as well as those who have chosen careers outside of academia, make critical and diverse contributions to science. Nonacademic careers often place less emphasis on publishing or allow less time to lead or contribute to publications, yet nonetheless provide influential routes for training new scientists, move science broadly into the public realm, and inform critical policy and decision-making [53,54].

Beyond the university classroom and research group, valuing the broader impacts of research is also critical. The cocreation and dissemination of scientific knowledge through collaboration with industry, implementation of government policy, public outreach and media engagement, societal service through science communication, and deferring to the guidance of the communities in which science takes place have the potential to center communities outside research institutions in critical topics [55,56]. Additionally, these intentional actions can aid in restoring public trust in science and promoting the advancement of diverse groups in science careers (e.g., [57]). In fact, funding agencies (e.g., Natural Sciences and Engineering Research Council in Canada, The National Science Foundation in the United States, and the Research Excellence Framework in the United Kingdom) are now including these contributions in the evaluation criteria of the quality of researchers, demonstrating that funding bodies are beginning to play a critical role in normalizing and rewarding the work that scholars do to connect to communities and are key contributors to the valuing of this work. Together, this shift in evaluation criteria indicates that quantifying these impacts is possible and meaningful to science and society more broadly.

Another key avenue of impact on the scientific system is through training upcoming generations of scientists. This role necessitates diverse mentoring and pedagogical skills essential to attract, engage, retain, and elevate scientists in training from different geographies, social–cultural and socioeconomic backgrounds, and career paths. Mentorship is fundamental to efforts to drive a transformation to a fair and safe scientific culture, and we explore the value of mentorship in more detail below in Section 3.

(3) Broadening the system to value mentorship, diversity, and well-being

Broadening the definitions of success and impact provides an essential foundation to shift the academic system and scientific culture. Through valuing multidimensional mentoring, we can promote decisive actions to improve justice, equity, diversity, and inclusion (Fig 2).

Valuing the impact of multidimensional mentorship

Mentorship is a bidirectional relationship that changes as the relationship evolves, and these relationships may vary from being highly formal, structured, and with very specific goals, assignments, and timelines to less formal or clearly articulated relationships [58]. While a traditional mentorship relationship can be between an academic supervisor and a mentee (graduate student, postdoctoral scholar, undergraduate researcher, etc.), mentorship can come in a variety of forms that include peer, supervisor, career development mentor, and/or personal mentor [58,59]. For the purposes of this perspective, we use a broad definition of mentorship that encompasses dynamic mentorship and diverse relationships. We follow the frameworks of scholars who have established the ideas of mentor networks or webs, with mentorship evolving as the needs and aspirations of mentees change through each career stage [60–63].

High-quality mentorship greatly benefits mentees, since mentors are essential in determining career outcomes [64,65]. Research examining a wide range of mentoring relationships (e.g., in government, hospitals, and business) demonstrates that deep engagement in mentorship also leads to a greater sense of job satisfaction, higher commitment to the institution, and higher career success for mentors [66]. Cultivating these outcomes within science could reduce attrition rates often associated with low levels of job satisfaction [67] and a lack of institutional community [68]. The benefits of multidimensional and networked mentoring across career stages, especially by mentors with multiple identities from marginalized groups (i.e., intersectional identities), are critical to increasing representation, recruitment, and retention in the scientific system [58,69–74]. Good mentors can foster a sense of belonging for mentees with diverse backgrounds [75], especially if the mentor belongs to, or strongly associates with, a particular identity, which further emphasizes the importance of inclusive representation in science.

Within academia, outstanding mentorship is invaluable [76,77]. However, this mentorship is traditionally quantified by mentee productivity, which is assessed by the same traditional metrics (e.g., [10]) that have significant biases (see Section 1 above). These metrics fail to acknowledge the diverse value of mentorship, and, thus, reevaluating mentoring practices and how impact is measured will benefit a diverse and intersectional group of early-career scientists [78,79]. We propose that a broader lens of mentorship quality be acknowledged and employed by institutions and funding agencies, which would provide a more holistic measure of scientific impact and reward high-quality mentorship (Figs 1 and 2).

Holistic valuation of mentorship quality includes the contributions from mentors and the achievements of mentees [61,70,80]. In addition to research productivity, metrics encompassing the breadth of mentorship dimensions can incorporate the mentee’s acquired skills, tools provided to the mentee, mentee retention, career commitment, self-efficacy, mentee satisfaction, and overall group culture [81–83]. Mentorship quality could then be quantitatively tracked by institutions throughout an individual’s career within or outside of academia using surveys such as the Global Measure of Mentorship Practices as adapted for STEMM postsecondary education [84]. These metrics could be compared empirically against institutional or national statistics to gauge scientific impact.

Institutions should also elevate strong mentorship by both establishing internal awards for mentor excellence and increasing the weight of such awards in promotions or tenure assessments. Awards such as the National Science Foundation’s Presidential Award for Excellence for Science Math and Engineering Mentoring (PAESMEM), the Australian Museum Eureka Prize, and the Nature Research Awards for Mentoring in Science already exist to recognize outstanding mentors. In addition, placing value on mentorship by funding agencies (e.g., National Science Foundation’s Broader Impacts [85]; Natural Sciences and Engineering Research Council of Canada (NSERC) contributions to training [86]; and Athena Scientific Women’s Academic Network (SWAN) Award Scheme [87]) creates further incentives to achieve mentorship excellence. These prestigious recognitions, coupled with funding and incentives to support mentees from marginalized groups, represent strong steps forward in valuing mentoring and highlighting the efforts and impacts of individuals working to support the next generation of researchers from diverse backgrounds and/or identities.

Mentorship can promote justice, equity, diversity, and inclusion in science

An important avenue in promoting justice, equity, diversity, and inclusion in science is through effective mentorship strategies beyond the traditional dyadic and top-down relationships, such as creating mentorship networks as we discussed above [60,88]. The idea of a comprehensive, singular mentor suggests that one person can meet all mentee needs; however, each individual has unique needs—especially those who identify as members of systemically marginalized groups [89–91]. For example, some individuals may seek mentorship for academic and career advice alone, while others may pursue mentors with similar personal identities or experiences for support with the unique situations these individuals face in science. Mentoring networks are effective in allowing individuals to seek out appropriate mentorships to meet their identified goals and associated needs. By implementing the use of mentorship networks for researchers at all levels, institutions can better connect individuals with appropriate mentors that can support the success of one another [60,90]. Because mentorship networks support the long-term career goals of the mentees [92], institutions that support these networks are investing in higher retention of scientists from marginalized groups [93].

Mentoring should not be limited to students or early career stages. Mentoring throughout one’s career provides important mechanisms for learning new skills, broadening career path options, and attaining life goals. For example, formal mentoring could be available for individuals transitioning toward administration. This is particularly important for helping to build a more equitable community as administrators often hold the most power for implementing change at institutions. If such leaders are trained in how to build inclusive cultures, change will likely be quicker and broader.

While multidimensional mentoring will facilitate a more inclusive culture, specific strategies are also needed to change the systemic sexism and racism that pervade institutions [93,94]. A first step is to identify barriers to diversity, followed by policies and training designed to support transformative institutional change [93]. These include shifting community culture through communication, collaboration, and training to support interventions and leadership. For example, transitioning from a “gatekeeping” to a “groundskeeping” approach at all levels of the academic hierarchy is a key component of the required shift in culture to address pervasive obstacles to justice, equity, diversity, and inclusion [95].

For these efforts to be achieved, faculty and researchers need to be educated and supported with structured programs that embed these principles in teaching, research, and mentoring (e.g., [96]). For example, training in inclusive pedagogical approaches (i.e., inclusive or deep teaching [97,98]), bystander intervention training, and anti-bullying and antiracist mentoring and teaching practices can be made part of the job expectation for those in supervisory and mentoring roles. This training may also include critical pedagogy that examines and challenges the systems of oppression that shape society [99–101] and promotes both the intellectual growth and well-being of students and mentees [102]. To ensure that training opportunities become valued by participants, institutions may consider implementing mandatory participation by requiring training for career advancement or as prerequisites for recruiting mentees. However, training programs should be mindfully designed to engage those who may complete training for inauthentic reasons. Discussions of topics covered in training should become standard practice at regular events including faculty meetings and retreats and graduate student association meetings. Undergraduate programs can include discussions of unconscious bias and how such biases influence classroom dynamics. When it comes to hiring, candidates should be assessed across multiple axes to ensure the recruitment of individuals who are dedicated to building a stronger and more equitable community. Leaders need to work hard to develop creative ways to promote justice, equity, diversity, and inclusion in science to dismantle the barriers that prevent healthy and innovative science workplaces. We therefore advocate for the continuation and development of awards and incentives that recognize and reward authentic efforts to do so.

Unfortunately, large gaps in the implementation of effective strategies to dismantle discriminatory systems still persist. Over the last decade, a range of initiatives in academia, industry, and government have been implemented to support the attraction, retention, and progression of people from systemically marginalized groups at national and international levels. To normalize and move these initiatives forward, we must leverage the many recommendations that have already been made for justice, equity, diversity, and inclusion in science [5–7,96,103,104]. Evaluating these actions and policies within a scientific framework and developing best practices is a start to implementing effective strategies [105]. Scientific institutions and funding agencies must implement initiatives that address the systemic discrimination and biases in student admissions, recruitment, grant funding, and promotions [93,106–109]. Institutional commitment is needed to strategically implement meaningful equity and inclusion approaches with effective accountability mechanisms in place [110].

Transformed science culture: Supporting a safe and healthy environment

The role of inclusive mentoring practices (e.g., sponsoring, counseling, networking, and advocating; Fig 2) is unequivocal in providing essential tools to foster justice, equity, diversity, and inclusion for mentees, preventing toxic mentor–mentee relationships, and overcoming barriers and access in STEMM careers [111,112]. Social belonging and valuing of multiple identities in science reinforces achievement [9,96,113,114], and diverse teams have been shown to increase the rate of innovation and collective creativity [115–117]. While good mentorship can foster a sense of belonging in science for the mentee, relationships of many mentees from marginalized groups with their mentors—who are often from the majority group—are not always positive, leading to health issues, such as insomnia and anxiety [118], and lower retention of these groups in science (reviewed in [93,104]). In order to effectively mentor, all mentors—particularly those who are not familiar with the experiences and perspectives of systemically marginalized scholars—should engage with cultures, communities, and perspectives that differ from their own, connect with communities that are working toward creating justice, equity, diversity, and inclusion, and support institutional change already underway. In addition, increasing representation from marginalized communities throughout institutional hierarchies provides greater opportunities for mentees to find mentors with which to build meaningful relationships.

Of particular concern is the recently highlighted decline in mental health of many academics and a growing crisis at the graduate level [119]. Graduate students are at least twice as likely to experience mental health challenges, such as anxiety and depression, compared to the general population with equivalent education [120]. This trend is even more striking for women of color in STEMM, who are facing systemic sexism and racism, along with daily microaggressions and safety concerns [121]. Sexual minorities and LGBTQ+-identifying people are also subject to discrimination that adversely affects their well-being, mental health and, ultimately, retention in STEMM fields [74,104]. Laboratory work, field work, and simple existence in the academy can often place marginalized groups, including those with disabilities, at risk of injury, harassment, bullying, and assault (e.g., [103,122–124]). To combat these challenges, specific strategies for safety and well-being [125,126] must be supported at the research group, departmental, institutional, and funding organization levels.

Moreover, destructive mentoring has frequently gone unchecked in academia [123], often because of the appearance of a superficially productive, well-functioning, or supportive working environment. This is in large part due to power dynamics within the formal mentor–mentee relationship, as academia was constructed on a model with a top-down hierarchy (Fig 2). Key future directions to redress this issue include proactive policies at the institutional and departmental levels, which could include formalizing mentee and advisor responsibilities and expectations [127,128]. Initiatives can be tailored to empower mentees to manage their relationship with their research mentors and for faculty to advise, educate, and supervise using inclusive techniques [112]. Further, there should be clear procedures to change behaviors displayed by potentially abusive mentors and significant consequences to ensure the prevention of negative impacts on future mentees (e.g., [129]). Actions such as facilitating safe ways in which mentees can provide feedback to their mentor—whether positive or negative—is a start to empowering mentees and aligning expectations [130]. Institutional oversight in developing a strong mentorship culture, support for mentor–mentee training, and responsibility for administrative interventions are critical aspects of ensuring a safe environment for all.

Institutions are at the foundation of creating a culture that promotes community wellness, beginning with a clear mission that aligns with the safety and health of mentees and mentors, especially those from marginalized groups [131]. Indeed, it is the institution’s responsibility to ensure there is a specific training focused on effective mentoring practices and modeling wellness for mentees [93,132]. Mentees and mentors need to be trained to appropriately flag, assess, and address mental health and safety concerns using targeted and early-intervention roadmaps in safe spaces. This training should be made readily available via a variety of modalities, such as mental health first aid training (e.g., [133]). An enhanced focus on health, safety, and accessibility in science, as well as institutional support for mentorship assessment and growth, will lead to improved retention of scientists from diverse backgrounds and increased community health and wellness (Fig 2). While it has become increasingly standard for institutions to publicly profess commitments to justice, equity, diversity, and inclusion, without sufficient investment of time, energy, and funding, these commitments will remain performative [134].

Conclusions

To dismantle the problematic reward system and create an inclusive and innovative community, the scientific system needs to move beyond the current narrow measures of success and impact to focus on holistic assessment (Fig 1). Acknowledging that there is a diverse range of contributions and career pathways will broaden the value system in science. By embracing inclusive approaches and not forcing people to assimilate into sexist and racist norms, we can grow a more equitable model for science that addresses and actively works to counter injustices. The challenges associated with changing a deeply embedded institutional history, culture, and structure toward a different inclusive value system will require institutions to champion a “new norm” to bring change at a global scale. Such a shift must be embraced by all and led by those currently in positions of power and privilege. This shift requires not only specific proactive actions and reforms to institutionalize change but also mechanisms to monitor implementation and provide feedback optimizing an adaptive and dynamic structure.

Abbreviations

LGBTQ+

lesbian, gay, bisexual, transgender, queer, and others

NSERC

Natural Sciences and Engineering Research Council of Canada

PAESMEM

Presidential Awards for Excellence in Science, Mathematics and Engineering Mentoring

STEMM

Science, Technology, Engineering, Mathematics, and Medicine

SWAN

Scientific Women’s Academic Network

Funding Statement

The authors received no specific no specific funding for this work.