The PULSE Diversity Equity and Inclusion (DEI) Rubric: a Tool To Help Assess Departmental DEI Efforts

ABSTRACT

In an attempt to redesign science, technology, engineering, and mathematics (STEM) departments to be more inclusive of all student populations, institutions of higher learning are reviewing their programs, policies, and the ways they engage students. The Partnership for Undergraduate Life Sciences Education (PULSE) has been working with STEM departments over the past 10 years to improve the student experience by incorporating evidence-based teaching practices and creating curricula with a deeper focus on conceptual understanding of scientific principles, competencies, and the process of science. PULSE created the PULSE rubrics, a set of five rubrics designed to assist life sciences departments in assessing their implementation of the recommendations of the American Association for the Advancement of Science Vision and Change report in the areas of curriculum, assessment, faculty practice and faculty support, infrastructure, and climate for change. An additional rubric, on diversity, equity, and inclusion (DEI), is described in this paper. Each of the 13 criteria of the PULSE DEI rubric begins with a context section of background information with references and a scale of 0 to 4 (baseline to exemplar) with descriptors for each score. The PULSE DEI rubric has been added to allow departments to determine the starting point for their DEI work and reveal areas that require attention. All PULSE rubrics can be accessed from the PULSE Community website (https://www.pulse-community.org/rubrics).

PERSPECTIVE

Throughout the history of higher education, scholars of color have led the dialogue about educational inequality, antiracism, and fostering pedagogies of social justice and cultural responsiveness (1–4). Their leadership has led to a refocusing of this dialogue to address inequities in higher education, compelling colleges and universities to rethink their policies and procedures at all institutional levels (5, 6). This recent “call to action,” stimulated by the Black Lives Matter and other movements, has motivated higher education communities to reconsider their interactions with students, implement initiatives that introduce student-centered inclusive teaching practices, develop inclusive curricula at the departmental level, and modify promotion and tenure criteria to value equity work (7, 8). Ultimately, this collective, cross-campus work will lead to a re-envisioning of the policies that often marginalize underserved students, faculty, and staff. Because student diversity is expected to increase over the next few decades (9), it is even more crucial for institutions to respond to this call to action and intentionally address inequities by removing the barriers to academic momentum and advancement experienced by underserved students. This work has been supported by the actions of professional societies, such as the American Association for the Advancement of Science and the American Society for Microbiology, which have published position statements and calls for equality and unity. In addition, federal agencies and private foundations have provided funding opportunities, such as the National Science Foundation’s Racial Equity in STEM and Howard Hughes Medical Institute’s Inclusive Excellence Initiative, which focus on addressing systemic inequities and promoting practices designed to increase the number of individuals from “historically excluded communities” (10) in science, technology, engineering, and mathematics (STEM).

No doubt, colleges and universities are concerned about equal access to higher education, discriminatory admissions policies, and ways to address inequities in teaching and learning. Conversations about diversity, equity, and inclusion (DEI) are taking place, starting at the level of instructor self-awareness to improve individual’s efforts to address disparities in teaching and learning (11–15). While change at the individual level is an essential step, DEI efforts need to be a core value of an institution and coordinated across all its levels, from admissions policies and classroom practices to hiring, promotion, and tenure policies. As programs and initiatives with the goal of creating a more inclusive academy are implemented, it will be important to assess their impact in order to ensure they are effective. Currently, there are few measurement tools that assess the success of departmental DEI efforts on those underserved in STEM (16, 17). The creation of such measurement tools will allow various institutional departments and units to determine the current status of their DEI work and identify specific areas for improvement.

PULSE AND THE PULSE RUBRICS

The Partnership for Undergraduate Life Sciences Education (PULSE) is a nonprofit organization focused on empowering the transformation of life sciences departments to embrace evidence-based educational practices through development of a shared vision, creation of an action plan to achieve the vision, and routine self-assessment of its activities. PULSE was launched in 2012 by the National Science Foundation, the Howard Hughes Medical Institute, and the National Institute for General Medical Sciences. Forty Vision and Change Leadership Fellows were selected from a pool of applicants that had demonstrated collaborative leadership experience as well as experience as change agents in STEM education at Associate’s, Baccalaureate, Master’s, and Doctoral or research universities. Since 2012, PULSE has recruited new fellows, for a total of about 60 fellows, and has engaged with more than 300 departments and institutions through our programs.

PULSE has established three major programs: Ambassadors Workshops, Recognition Program, and Regional Institutes. The Ambassadors Program facilitates discussions in life sciences and STEM departments that lead departments to create a shared vision and an action plan to guide the department’s work in implementing the recommendations of Vision and Change and other evidence-based practices. The Recognition Program uses the PULSE rubrics to engage departments in the assessment of their programs in accordance with the recommendations of Vision and Change. Participating departments are recognized for their achievements using a progression-level model similar to the Leadership in Energy and Environmental Design rating system for green construction. PULSE Regional Networks host workshops throughout the United States, allowing neighboring institutions to build communities of practice and work together to accomplish the goals of Vision and Change. More information on PULSE programs can be found on the PULSE community website (https://pulse-community.org/home).

The PULSE rubrics for departmental assessment were released in 2013 (18) and validated in 2016 (19) in response to the call for action promulgated in the 2011 Vision and Change in Undergraduate Education report (20). The report delineated recommendations on ways to improve undergraduate education with the ultimate goal of increasing student success as life sciences majors. The Vision and Change report emphasized the importance of students “learning about science by doing science” and highlighted a movement away from acquisition of information to conceptual understanding, with a focus on scientific competencies underlying the process of science, all ideas that could potentially lead to more diversity in STEM disciplines. The PULSE rubrics provide life sciences departments with a tool to determine their level of implementation of the recommendations of the Vision and Change report. The rubrics are divided into five rubrics with criteria in the areas of curriculum (11 criteria), assessment (16 criteria), faculty practice and faculty support (20 criteria), infrastructure (10 criteria), and climate for change (8 criteria). The PULSE rubrics, available on the PULSE community website (https://www.pulse-community.org/rubrics), give departments opportunities to have conversations around evidence-based practices. Additionally, the PULSE rubrics can be used by departments as a tool to identify programmatic areas needing improvement, request institutional resources to enact these changes, and make data-informed decisions about policies and practices that improve students' educational experiences and track their progress over time.

In response to the calls to action noted above and the lack of tools that allow departments to reflect, implement, measure, and self-assess their results of DEI efforts, PULSE has created a new addition to its rubrics that focuses on departmental DEI work. The PULSE DEI rubric is intended to assist departments in determining the success of their previous DEI efforts, monitor their ongoing work, and develop future strategies to increase diversity, equity, and inclusion within their department. Similar to the other five PULSE rubrics, the DEI rubric is intended to be a departmental self-assessment tool that supports dialogue within a department to determine what inclusive excellence looks like (21) and guides the department’s work in building learning environments that intentionally increase diversity, value all individuals, and foster an inclusive environment where all members of the department can grow.

The DEI rubric is intentionally aspirational, and departments may find some of the items difficult to address. By completing the rubric, departments will be able to recognize early outcomes of their DEI efforts, such as determining what inclusive excellence looks like and/or more advanced outcomes, such as determining the success of their DEI initiatives. PULSE recommends departments complete the DEI rubric every 5 years as part of an iterative assessment process. Improvement on some of the rubric items will require institutional support; other items may be easily implemented by a department if motivated to do so. In this way, the rubric serves as a starting point to guide difficult conversations within departments. It can also be a tool for change, as it provides evidence for the need for institutional support to enact change.

DEI RUBRIC DEVELOPMENT PROCESS

In Spring 2020, PULSE decided to expand its commitment to supporting departmental diversity, equity, and inclusion efforts. PULSE created a statement of solidarity as well as an Anti-Racism Resource Page for Biology Departments on the PULSE community website. In addition, PULSE reviewed its programs as well as the PULSE rubrics to see how they could better support departmental DEI work. PULSE Fellows that lead the Recognition Program determined that PULSE needed a way to encourage departments to actively reflect, implement, measure, and self-assess their efforts. The group recognized that while the initial five PULSE rubrics did include some DEI components, the need for a rubric focusing on only DEI would be an essential tool to support DEI efforts. In addition, the DEI rubric development team decided all the DEI items should appear in a separate rubric so that disciplines beyond life sciences and STEM could use the rubric.

In July 2020, a group of five PULSE Fellows began the DEI rubric development process. Since diversity, equity, and inclusion are broad terms that can involve many dimensions, a decision was made to focus the PULSE DEI rubric on persons excluded due to ethnicity or race (PEERs) (22). Without a specific focus, it would be difficult for departments to interpret, generate fruitful discussions, and determine their score if multiple identities were used for the rubric items. In addition, DEI work addressing biases, policies, and practices that disenfranchise PEERs will likely lead to the simultaneous mitigation of bias against other identities due to the intersectional nature of identity (23).

The DEI rubric development team looked at existing rubrics, including the New England Resource Center for Higher Education Self-Assessment rubric for the institutionalization of diversity, equity, and inclusion in higher education (24), the University of California, Berkeley rubric to assess candidate contributions to diversity, equity, and inclusion (25), the University of Wisconsin Whitewater diversity learning and intercultural competence rubric (26), University of Rhode Island diversity and inclusion general education rubric (27), and the Peralta Community College District online equity rubric (28). While these rubrics address various aspects of advancing DEI work, they do not focus on departmental-level practices as described in PULSE’s initial five rubrics. The PULSE DEI rubric was therefore designed to supplement the initial five PULSE rubrics and focus on curriculum, assessment, faculty practice and faculty support, and climate for change.

After an initial draft DEI rubric was completed, a series of focus groups was conducted so that PULSE fellows could provide comments to improve the DEI rubric. Once these comments were incorporated, an updated draft version of the rubric was sent to eight scholars in the field of antiracism, who graciously volunteered to provide suggestions for improvement. In addition, three departments field tested the DEI rubric and provided feedback. The final 13-item PULSE DEI rubric is now available on the PULSE Community website (https://www.pulse-community.org/rubrics) for departments to use.

ANATOMY AND PURPOSE OF THE PULSE DEI RUBRIC

As previously mentioned, the PULSE DEI rubric addresses four of the five categories present in the initial set of PULSE rubrics: curriculum, assessment, faculty practice and faculty support, and climate for change. We chose not to include infrastructure items in the DEI rubric because departments sometimes do not have sole control over their infrastructure and frequently rely on institutional or state-level capital improvement projects to make changes. In addition, our decision was informed based on the rubric data we have collected with the departmental average score at the level of accomplished (a score of 3 on a scale of 0 to 4) on the infrastructure rubric. Each of the four categories in the DEI rubric includes two to four criteria, designed to address key aspects of that category. Each criterion includes a detailed context statement explaining the criterion and how it should be interpreted and cites references to clarify the meaning of the criterion and support for departmental DEI work. Each criterion is scored on a scale of 0 to 4: baseline = 0, beginning = 1, developing = 2, accomplished = 3, and exemplar = 4. Each performance level includes detailed descriptors; these descriptors are related to the information in the criterion’s context, so that a department can determine the current status of their DEI efforts (Fig. 1A).

FIG 1.

(A) Anatomy of the PULSE DEI rubric (https://www.pulse-community.org/rubrics). Each criterion of the PULSE rubrics contains a context section that explains the criterion with related references. Descriptors for each scoring level, 0 to 4, baseline to exemplar, are described. (B) PULSE diversity, equity, and inclusion rubric criteria. The PULSE DEI rubric is one of the six rubrics categories of the PULSE rubrics. The DEI rubric is divided into four subcategories: curriculum, assessment, faculty practice/faculty support, and climate for change. Each subcategory has two to four criteria.

All PULSE rubrics, including the DEI rubric, require departments to determine a consensus score (not an average score) for each rubric criterion. Departments develop consensus scores by coming together, discussing the rubric criteria, and determining as a group their overall scores for the department. Working through the DEI rubric to determine consensus scores will likely involve deep, challenging conversations. The process of engaging in those conversations is an important step toward implementing antiracist actions within the department.

DEI RUBRIC CRITERIA

The following section briefly describes the criteria in each of the four DEI rubric categories and why they were selected to be included in the rubric. A summary of the DEI rubric items is displayed in Fig. 1B.

Curriculum criteria

The four curriculum criteria consider the following: (i) the incorporation of high-impact practices (HIPs) and inclusive pedagogies; (ii) student access to course materials; (iii) incorporation of racially diverse perspectives; and (iv) considering bias in the curriculum.

(i) Incorporation of HIPs and inclusive pedagogies. This criterion examines the use of HIPs, such as undergraduate research, internships, service learning and civic engagement, writing-intensive courses, first-year seminars, capstone courses, learning communities, common intellectual experiences, e-portfolios, diversity and global learning, and collaborative assignments and projects (29), since HIPs have been shown to improve student learning (30) and have a positive impact on PEER students’ perception of learning (31). It has been reported that participation in HIPs has not been equal, with certain PEERs not having access to these transformative educational experiences (32). Therefore, it is important to find ways to modify HIPs to reach as many students as possible and to consider the quality of HIPs being offered (33, 34). An examination of inclusive pedagogies is included in the DEI rubric, since they are teaching practices known to foster an environment where varied backgrounds are considered so that all students feel valued and included. Inclusive strategies focus on maximizing student participation, building community for all students, monitoring behavior, and cultivating divergent thinking. Such strategies support all students in the classroom so they can think, talk, and learn effectively. Tanner provided a rich resource outlining 21 quick-to-implement strategies to improve equity in the classroom (35).

(ii) Student access to course materials. This criterion was included in the rubric because it addresses the importance of making courses and course materials available to all students, regardless of their socioeconomic status. It assesses the use of open educational resources (OERs); courses designed to intentionally consider bandwidth issues needed to view and use digital course materials; considerations for costs and use of additional software students may need; and the ability to be on campus or at specific off-campus sites for assignments and activities required in the course. This item encourages instructors to be intentional in their course design choices so that the needs of students who work, care for others, or may not have the finances to purchase learning materials are considered.

(iii) Incorporation of racially diverse perspectives. It has been established that students become more engaged when they can recognize themselves within the curriculum and when they make connections between the curriculum and their lives, which increases their sense of belonging (35–37). This item examines whether courses highlight contributions from a broader body of underrepresented scientists and reflect the racial diversity of the student population.

(iv) Considering bias in the curriculum. This criterion focuses on the implicit biases that are part of scientific studies. Departments can use this rubric item to review their curricular content to determine if biases addressing those who serve as subjects in research studies and who benefit from scientific research findings are included in courses (10, 38).

Assessment criteria

The two Assessment criteria have departments explore the following: (i) the extent to which disaggregated student data are analyzed and (ii) assessment of perceptions of equity and inclusion.

(i) The extent to which departments analyze disaggregated student data. This criterion is included in the rubric so that departments can consider the success of specific groups of students. Analyzing disaggregated data is important because it assists departments in identifying equity gaps and developing specific strategies to improve student performance (39). Ample evidence exists that PEERs generally underachieve in STEM courses compared with non-PEERs (40), but this is not the case in every institution (41). Therefore, it is necessary for each department to disaggregate student achievement data to determine whether there are disparities in outcomes that need to be addressed.

(ii) Assessment of perceptions of equity and inclusion. This item allows a department to consider the use of climate surveys and other internal or external instruments to evaluate perceptions of equity and inclusion. These types of surveys have proven to be significant in revealing hidden feelings of exclusion and provide evidence of the effectiveness of actions taken to improve equity and inclusion (42, 43).

Faculty practice and faculty support criteria

The four criteria for faculty practice and faculty support consider the following: (i) faculty awareness of the terminology and knowledge of the history of institutional racism in higher education; (ii) the availability of faculty professional development on DEI-related topics (antiracism, equity, inclusion, and culturally responsive teaching); (iii) opportunities for faculty to engage in antiracism work; and (iv) opportunities for faculty to develop mentoring skills that are inclusive of PEER students.

(i) Faculty awareness of the terminology and knowledge of history of institutional racism in higher education. This item was included to have departmental faculty self-reflect and assess their knowledge of racism in higher education. Developing this knowledge requires an understanding of a variety of terms commonly used in the history of racism in the United States. Sources have been provided in the context to help faculty become familiar with this terminology (44–47).

(ii) Availability of faculty professional development on DEI-related topics. This criterion allows departments to assess the range of professional development available to faculty, including implicit association tests, articles that present frameworks to develop curricula addressing diversity, ways to include inclusive practices in the curriculum (48), and conferences and training focused on diversity, equity, and inclusion in STEM (49, 50; https://crossroadsantiracism.org/).

(iii) Opportunities for faculty to engage in antiracism work. This criterion was included in the rubric so that a department could reflect on opportunities for faculty to participate in national initiatives, such as the education division of professional societies and scholarship that has traditionally not been considered appropriate for STEM faculty.

(iv) Opportunities for faculty to develop mentoring skills that are inclusive of PEER students. This criterion allows departments to consider specific strategies to mentor PEER students, as it has been shown that mentoring of PEERs can lead to a greater sense of belonging in STEM (51).

Climate for change criteria

The three climate for change items consider the following: (i) addressing biases in academic policies; (ii) recruiting, hiring, and retaining PEER faculty and staff; and (iii) ensuring equity for all department members with particular attention on the intersectionality of marginalized identities with PEER identities.

(i) Addressing biases in academic policies. This criterion was incorporated into the rubric so that departments could determine whether their policies are part of their continuous improvement planning. Policies that are commonly reviewed include pre- and corequisites, grading policies, withdrawal, pass-fail options, attendance policies, readmission, and credit for prior learning.

(ii) Recruiting, hiring, and retaining PEER faculty and staff. This criterion has departments measure the extent to which strategies, policies, and transparent efforts that support and advance PEERs at all stages of their career (from recruiting, hiring, transition, retention, and advancement) have been implemented. Utilizing the ideas from Stewart and Valian’s An Inclusive Academy (52) can assist in making progress with this rubric criterion.

(iii) Ensuring equity for all department members with particular attention to the intersectionality of marginalized identities with PEER identities. This rubric criterion was added because the primary focus of the DEI rubric is PEERs. To fully support PEERs, one must consider the intersectionality (23) of their PEER identities with their other identities. By supporting PEERs, non-PEERs who identify with the other marginalized identities will also be elevated (53–55).

RUBRIC LIMITATIONS

There are several limitations to the PULSE DEI rubric. Because there are so many identities that are marginalized, the DEI rubric authors needed to make several choices about the scope of the rubric. The DEI rubric focuses on PEER identities, except for one rubric item which deals with intersectionality. The rubric does not focus on other marginalized identities, including gender identity, sexual preference, and persons with disabilities. The focus on PEERs was an intentional design choice, following the example of Asai (40). However, many of the rubric items promote equity and inclusion in general, and a department may choose to amend or add rubric items to focus their efforts on other identities. In addition to choosing to focus on the PEER identities, the rubric authors also chose to focus on aspects of DEI that departments have the most control over. Therefore, we did not include any items related to infrastructure in this rubric. While not exhaustive, the PULSE DEI rubric will help departments begin to properly assess the current state of their DEI efforts and help support departmental growth.

Finally, another rubric limitation is the challenging nature of the rubric itself. Departments are meant to work through the rubric’s complex criteria and descriptors and reach consensus. This may lead to difficult conversations and may also expose complex power dynamics within a department. Departments who have used the rubric have navigated these difficult conversations by setting up ground rules for conversations; some departments have used an external facilitator to guide these conversations. Like the other PULSE rubrics, the DEI rubric is not static. Over the next several years, as more departments use the rubric and as we collect and analyze more DEI rubric scores, the DEI rubric will be revisited and revised as needed.

FUTURE EFFORTS AND NEXT STEPS

PULSE plans to continue its work on departmental DEI efforts by collecting DEI rubric scores to create a national data set to determine the status of DEI efforts in the United States. Based on these data, PULSE will be able to modify the rubrics and its programs to address ways it can better serve departments and improve their DEI efforts.

For departments, using the PULSE DEI rubric can be a pathway to develop strategies to diversify the student body, faculty, and staff and ultimately foster greater participation of underserved groups in the STEM workforce. Departments that desire to build more inclusive environments can begin by selecting certain rubric items to focus on and expand their efforts as they proceed. Departments that are successful in creating equitable and inclusive communities can serve as models for others within their institutions and for STEM departments nationally.