Supporting Biomedical Research Training for Historically Underrepresented Undergraduates Using Interprofessional, Nonformal Education Structures

Abstract

Research experience provides critical training for new biomedical research scientists. Students from underrepresented populations studying science, technology, engineering, and mathematics (STEM) are increasingly recruited into research pathways to diversify STEM fields. However, support structures outside of research settings designed to help these students navigate biomedical research pathways are not always available; nor are program support components outside the context of laboratory technical skills training and formal mentorship well understood. This study leveraged a multi-institutional research training program, Enhancing Cross-Disciplinary Infrastructure and Training at Oregon (EXITO), to explore how nine institutions designed a new curricular structure (Enrichment) to meet a common goal of enhancing undergraduate research training and student success. EXITO undergraduates participated in a comprehensive, 3-year research training program with the Enrichment component offered across nine sites: three universities and six community colleges, highly diverse in size, demographics, and location. Sites’ approaches to supporting students in the training program were studied over a 30-month period. All sites independently created their own nonformal curricular structures, implemented interprofessionally via facilitated peer groups. Site data describing design and implementation were thematically coded to identify essential programmatic components across sites, with student feedback used to triangulate findings. Enrichment offered students time to critically reflect on their interests, experiences, and identities in research; network with peers and professionals; and support negotiation of hidden and implicit curricula. Students reported the low-pressure setting and student-centered curriculum balanced the high demands associated with academics and research. Core curricular themes described Enrichment as fostering a sense of community among students, exposing students to career paths and skills, and supporting development of students’ professional identities. The non-formal, interprofessional curricula enabled students to model diverse biomedical identities and pathways for each other while informing institutional structures to improve diverse undergraduate students’ success in academia and research.

Students from racial and ethnic minorities or low-income backgrounds as well as disabled students are underrepresented in health and science professions (Boekeloo, Jones, Bhagat, Siddiqui, & Wang, 2015; Duffus et al., 2014; Valantine & Collins, 2015), yet together these groups are becoming the majority of the U.S. population (U.S. Census Bureau, 2015). The National Institutes of Health (NIH) highlights that a more diverse group of students must be attracted to biomedical research careers, or the scientific workforce may not be fully prepared to address the increasingly complex nature of biomedical research (Scientific Management Review Board, 2015). Underrepresented minority (URM) scientists produce higher rates of scientific innovation, yet their careers are more likely to end prematurely (Hinton et al., 2020). Further, URM students are less likely to receive undergraduate and graduate degrees in science, technology, engineering, and math (STEM), including the fields of biological sciences, chemistry, and physics (Bonham et al., 2012; Hussar et al., 2020a; U.S. Department of Education, 2019). While many URM students enter undergraduate institutions with the express intent to pursue a STEM career, few emerge with STEM degrees (Hrabowski et al., 2011; Scientific Management Review Board, 2015). Potential root causes for this inequity include factors associated with low socioeconomic status and inconsistent access to relevant curricula (Scientific Management Review Board, 2015), as well as lack of mentorship, limited research internship opportunities (Cohen & Garcia, 2008), and lack of community due to being in environments that are not representative (Chang, Sharkness, Hurtado, & Newman, 2014; Clark Blickenstaff, 2005; Harrison & Tanner, 2018; Moss-Racusin et al., 2012). In 2010, URM individuals made up 29.3% of the U.S. population yet 8.3% of STEM doctoral degree recipients, and 7.3% of faculty positions (Estrada et al., 2016). In 2018, URM individuals made up 14% of doctoral degree recipients (National Center for Science and Engineering Statistics, 2018d) and ~26% of full-time faculty positions (Hussar et al., 2020b). Thus, while STEM and health disciplines are moving in the right direction, there is still a long way to go. Institutions and programs must continue to find ways to support, engage, network, and retain historically underrepresented students throughout their schooling.

To meet the need for training a diverse biomedical workforce, the NIH has funded 10 BUILD (Building University Infrastructure Leading to Diversity) sites nationwide to encourage and evaluate innovative approaches for undergraduate research training (Valantine & Collins, 2015; Valantine, Lund, & Gammie, 2016). The largest of the BUILD sites, EXITO, is a consortium of institutions representing five universities and six community colleges across the Pacific Rim. Richardson and colleagues (2017) described programmatic components of its intensive 3-year biomedical research training program, with its mentorship outlined by Keller and colleagues (Keller & Lindwall, 2020; Keller, Logan, Lindwall, & Beals, 2017). An important feature of the program is that many students start their training at partnering community colleges prior to transferring to the primary university. As 35% of the nation’s 16.6 million undergraduates in 2018 were attending 2-year institutions, which typically enroll higher proportions of historically underrepresented students (Hussar et al., 2020a), this program’s findings may inform the efforts of other institutions serving similarly diverse populations.

To enhance retention of historically URM students pursuing undergraduate biomedical research, the EXITO program recognized that URM students face systemic barriers and inequities whose negative effects require additional skills and resources to negotiate (Richardson et al., 2017), consistent with prior work by Ladson-Billings (1995). While STEM interest, motivation, self-efficacy, and identity may have brought these students to STEM fields and play an important role in their professional development (Boekeloo et al., 2015; Chemers, Zurbriggen, Syed, Goza, & Bearman, 2011; Jenson, Petri, Day, Truman, & Duffy, 2011; Maltese & Tai, 2011; Marriott et al., 2019; Usher & Parajes, 2008), additional factors contribute to STEM persistence. For example, both academic and scientific cultures can give mixed messages to URM students about their path to success. Harris and Gonzalez (2012; as cited in Trinidad, 2014) noted that “people of color and first-generation college students often face a contradictory culture of academia, and experience contentious places and processes” (Harris & Gonzalez, 2012). To support students, the EXITO program involves several hierarchical training components, including a multifaceted mentoring model (Keller & Lindwall, 2020; Keller et al., 2017), supervised research placements, and a Gateway to Research course. These interventions provide undergraduates access to research training by academic professionals with experience navigating academic and scientific cultures. However, cultural knowledge about best practices for supporting historically URM students may vary since few faculty have URM status themselves (Hussar et al., 2020b).

When considering interventions aimed at increasing URM students in STEM, Trinidad noted that academia itself “serves as a gatekeeper in accessing occupations, careers, and professions” (p. 17). For example, unique authority relationships are embedded in academic courses (professor–student), academic mentoring (faculty mentor–student mentee), and scientific research supervision (principal investigator–assistant). The authority figure controls critical factors for advancement such as grades, recommendations, and research access. Cultural norms in academia and research about what is wrong to say, do, and ask are unknown to students from families and communities where academic and research backgrounds are absent. Underrepresented students must also learn how to sustainably navigate academic and research cultures as well as represent themselves in the ways required of professionals in those areas, to enter and succeed in advanced degree programs in biomedical sciences.

Authors of this study are or were EXITO staff and faculty who provided support for undergraduate biomedical research trainees, including students from underrepresented backgrounds. We were involved in the development and implementation of a group support structure, termed Enrichment, at EXITO partner sites and engaged in reporting, analyzing, and/or summarizing the information presented in this case study. Predefined training models and procedures for students are typically documented, but support activities that emerge from direct work with students (around their needs) may leave no official data on what was done, how they were implemented, or why they were envisioned. As part of program development in the EXITO consortium, Enrichment activities had to be defined to allow consortium evaluation and to share best practices. The diversity of both students and sites in this consortium created an opportunity for a multisite case study to identify what activities, developed independently, support underrepresented undergraduate researchers. This article describes how Enrichment was designed and implemented, including how it evolved over 30 months to better meet student needs.

Materials and Methods

Setting

The EXITO program is overseen by Portland State University’s Institutional Review Board (IRB). Primary data for this case study were collected from faculty and staff participating in the NIH-funded EXITO program who were involved in undergraduate training at partner sites. Portland State University (University 1) serves as the primary institution with 10 partner sites (Figure 1). Partner sites included Oregon Health & Science University (University 2; a research-intensive graduate institution), three Pacific Rim universities (University of Guam; University of Alaska-Anchorage; University of Hawai’i at Manoa; Universities 3–5), and six community colleges. Four community college partners were located in the Portland metropolitan area (Portland Community College, Clark College, Clackamas Community College, Chemeketa Community College) and the other two were located in the north and south Pacific (American Samoa Community College; Northern Marianas College). In 2018, the University of Hawai’i at Manoa and Chemeketa Community College stopped recruiting students and were not included in this study’s follow-up data collection about Enrichment. In addition, Oregon Health & Science University is graduate-focused and was not included in this study, though serves as the research training site for some EXITO scholars at PSU. Secondary data were collected from undergraduate students participating in the NIH-funded EXITO program, with evaluations approved by the IRB.

Figure 1. Location of EXITO sites representing 11 institutions (five universities and six community colleges).

Data were collected from nine sites that had implemented Enrichment activities, with complete data from eight sites. Map adapted with permission from (Trapido-Lurie, 2010a, 2010b). Note. EXITO=Enhancing Cross-Disciplinary Infrastructure and Training at Oregon (EXITO)

Study Participants

Authors are or were faculty and staff serving undergraduates from diverse backgrounds pursuing biomedical research fields, including majors in biological sciences (e.g., biology, neuroscience, environmental health sciences), physical sciences (e.g., chemistry, engineering, physics), and social and health sciences (e.g., psychology, sociology, public health, social work, counseling, and education). Students aspired to research and/or practice careers in a range of biomedical disciplines and health professions (e.g., nursing, dentistry, medicine, pharmacy, physician assistant, radiation therapy, occupational therapy, mental health counseling, genetic counseling, etc.). At the primary university, EXITO biomedical majors had a 98% continuation rate and a 3.18 grade point average (compared to 82% and 2.94, respectively, for all biomedical majors at the primary university; institutional data from spring 2017). The program required 3 years of full-time enrollment, with community college students transferring to the primary university after the 1st year to engage in concurrent research internships for the last 2 years, with research intensives in both summers. Overall, EXITO retention was 82% (for biomedical students across all sites). For a national comparison, in fall 2017, the retention rate for all undergraduates across all majors, not limited to STEM, was 81% at universities and 62% at community colleges (McFarland et al., 2019a, McFarland et al., 2019b). For STEM majors alone, national retention rates have averaged 48% across all sites with the lowest retention rates (30%) at community colleges (Chen, 2013; Snyder & Cudney, 2017).

Demographics of EXITO program participants (Table 1) were obtained from students’ program applications. URM students were defined using NIH definitions, including (a) racial/ethnic backgrounds of Black/African American, Native Hawaiian/Pacific Islander, Native American/Alaskan Native, and Hispanic and Latino/Latina/Latinx; (b) students with a disability; and (c) students with a disadvantaged background (NIH, 2019). Disadvantaged background was estimated from application data using two approaches: (a) scholar self-report (question independent of demographics) and (b) through the calculation of a composite variable that replicated NIH’s 2019 definition of disadvantaged backgrounds for underrepresented populations (NIH, 2019). Criteria included (a) self-reported disadvantaged background, (b) first-generation college student; (c) eligibility for need-based financial aid (e.g., Pell grant); (d) foster care experience; or (e) previously or currently homeless. Fewer EXITO students self-reported their race as White (29%) compared to either the primary institution (67% in 2017) or 1st-year college students nationally (72% in 2016) attending public universities (Eagan et al., 2017). The EXITO evaluation team also provided scholar retention data across each of the 10 sites that enrolled scholars at any time.

Table 1.

Demographics of students in the EXITO program.

Demographic category	n (%)
Gender (N = 427)
Female	294 (69%)
Male	124 (29%)
Other/nonbinary/third gender	9 (2%)
Age (N = 406)a
19 years or younger	212 (52%)
20–25 years	114 (28%)
26 years or older	80 (20%)
Race (N = 427)
African American/Black	26 (6%)
Asian	73 (17%)
Native American or Alaskan Native	11 (3%)
Native Hawaiian or Pacific Islander	47 (11%)
More than one race	72 (17%)
White	125 (29%)
Other	53 (12%)
Unknown or not reported	20 (5%)
Ethnicity (N = 427)
Hispanic or Latino/Latina/Latinx	99 (23%)
Not Hispanic or Latino/Latina/Latinx	247 (58%)
Other or decline to answer	81 (19%)
Other demographics
Disability (N = 424)	48 (11%)
Disadvantaged background, self-report (N = 424)	225 (53%)
Disadvantaged background, composite score (N = 427)	369 (86%)
First generation college student (N = 426)	254 (60%)
Foster care experience (N = 427)	39 (9%)
Need-based financial aid (N = 427)	302 (71%)

Note. Cumulative demographics (Cohorts 1–5; application data). EXITO=Enhancing Cross-Disciplinary Infrastructure and Training at Oregon (EXITO)

^aAge collected in program orientation surveys, with some missing responses.

Development of Enrichment at the Primary Site

The original EXITO training plan included these defined elements: an introduction to research course, research internship experiences, assigned faculty mentors throughout the program at all sites, assigned near-peer mentors at the primary site for students in their 1st year of the program, and dedicated advising for program students. Rather than attempting to predefine what key needs would be at each site, how they should be met, or how students would interact and respond, activities and support systems were allowed to evolve with the students and programs at each site. Recognizing the benefit of student support (Appendix A), EXITO leadership defined Enrichment as an official program intervention in Year 2, with all sites needing some type of student enrichment, though its form, content, and definition was left up to each site. Activities at the primary site occurred on a regular schedule of required sessions (Appendix A). Student evaluation of Enrichment at the primary site began in Year 3.

EXITO Conference and Baseline Data Collection

An annual EXITO conference brought teams from each site together in person to plan curricular and programmatic components. This collaborative, nonhierarchical approach defined a set of consortiumwide shared learning objectives for all Gateway to Research courses in the early years of the BUILD EXITO grant and its annual conference, despite courses being created by faculty in different fields and taught in different formats at diverse institutions. By 2018 (Year 3), program leadership supported using this collaborative approach to define and assess other program areas, including Enrichment, a focus area for the summer 2018 conference. Enrichment offered an opportunity for sites, and for the program as a whole, to identify any common features (i.e., structure, content) in what their institutions had created to support the success of underrepresented undergraduate researchers pursuing biomedical research.

To understand how Enrichment was designed and implemented at partner sites, the Enrichment coordinator emailed instructional leads at each institution in April 2018. A five-question digital survey included one question each about instruction, format, and goals plus two questions about learning objectives (Appendix B). All partner sites were emailed except the research partner (OHSU), as it provided research placements only. To support data collection, the Enrichment coordinator met with instructional leads or teams from all local-area partners. All sites reported the same barrier to returning data: They had not created an Enrichment program. However, when asked to describe what they did with their EXITO students outside of the required course and mentor meetings, every site immediately described specific organized and intentional group activities for EXITO students, often held on a regular schedule. When asked why they had not reported these activities as their Enrichment, all sites independently gave one or both of the following responses. First, they had not believed that informal activities could be formally documented or that a research training program would be interested in documenting informal activities. Second, they did not think that activities that were not direct research skills training could “count” as supporting student research training. One faculty member from an underrepresented background in academia and research who was experienced with URM students in STEM summed up many comments we received across sites: “This is just what we do. Because our students need it. We know they need it because we’ve been there ourselves.” Following in-person meetings, each local site quickly returned baseline data. Following these conversations, the Enrichment coordinator emailed nonlocal sites to clarify that Enrichment could include any type of activity and did not have to be predefined as a formal intervention. All sites except University 5 returned information.

At the June 2018 EXITO conference, results of the baseline data collection on Enrichment were shared. The survey had asked about learning objectives, given the consortium’s previous success in defining shared learning objectives for diverse EXITO Gateway to Research courses. However, the consortium concluded that Enrichment did not have shared instructional learning objectives. Instead, Enrichment shared a curricular structure across sites: nonformal education using facilitated instruction with peer and near-peer group activities. The consortium defined a preliminary list of Enrichment focus areas based on the activities at the sites, which sites could use to define and focus their content goals for Enrichment.

Follow-Up Data Collection

To understand how Enrichment was implemented across sites, a follow-up survey was emailed to active Enrichment leads in November 2019 (18 months later). The survey included the same prompts from the baseline survey along with three additional prompts (Appendix B): “What have you learned about offering enrichment to your students? What feedback have you received from students about enrichment thus far (course evaluations, word of mouth, anecdotes, etc.)? Anything else you want us to know?” All sites returned documentation within 2 weeks without in-person meeting support.

Qualitative Data Analysis

Text from baseline and follow-up surveys were uploaded to Dedoose (version 8.1.8; Los Angeles, CA) for thematic content analysis (Green & Thorogood, 2014). A faculty member who teaches graduate-level qualitative methods completed the coding, with a review of code definitions and themes for face validity by the EXITO lead evaluator and Enrichment program coordinator. To protect site privacy, data from the nine participating institutions were coded using the following designations: primary university (i.e., University 1), university (i.e., Universities 3 and 4); urban community college (i.e., Community Colleges 1–3 and 6), or distant community college (i.e., Community Colleges 4 and 5, both located over 5,000 miles from the primary university). Code applications for each site type were compiled in a data matrix within Dedoose and exported to Excel for secondary analysis of themes. The number of sites that reported each theme as well as the average number of mentions of that theme by site type were computed.

Student Evaluation Data Collection

While Enrichment programmatic features were coded from instructor and team perspectives, student perspectives were collected using informal “course evaluations” implemented at the primary site at the end of students’ final term of Enrichment (Appendix B). Because all students from partnering community colleges finished the program at the primary institution, this Enrichment data included students from seven of the nine participating partner sites. The evaluation asked about overall feelings about Enrichment, thoughts on it being mandatory, key gains/gaps, and instructor ratings. Survey administration occurred at the end of winter term (last term of EXITO) in both 2019 and 2020, with each senior cohort comprising over 55 students. Questions were identical across surveys with two items added in 2020: (1) institution where the scholar started EXITO (checkbox option of primary university or transfer institution; and (2) “Enrichment connected me with my peers”(Appendix C). A 6-point Likert scale captured overall feelings about Enrichment, with responses scored from 1 (strongly disagree) to 6 (strongly agree). Scholars’ percentage agreement was computed from those who answered 4 (slightly agree), 5 (agree), or 6. Data was compiled in Excel before transfer to SPSS (IBM, version 26) for statistical analyses.

Member Checking

A final survey was sent to all prior Enrichment leads in October 2020 (about 30 months after baseline data collection), which presented summarized themes from the qualitative analysis (Table 2) and asked each lead to rate the frequency and impact of these themes at their site on a 0–100 scale (prompts in Appendix B). Frequency data that were left blank were recoded as 0. Impact data were not recoded.

Table 2.

Themes observed for the Enrichment program’s goals and core responsibilities.

Code	Definition	Example quote
Research exposure	Students learn about different fields of research and topic areas within research	“Ensure students have exposure to areas of science that align with their short-term and long-term educational and career goals.” — urban community college
Path representation & networking	Students meet professionals in their desired area of interest and learn what it takes to pursue a particular career path. Students can network and talk with professionals who may have been previously unknown to them	“Increase the breadth of students' experience of who does science, what science looks like, and the range of scientific work.” — urban community college
Research skills	Students develop specific research skills (e.g., learning statistics, writing an abstract, understanding research ethics, conducting a literature search, etc.) Also includes exposure to and practice with basic activities done in research (e.g., fundamental bench laboratory techniques)	“Critical thinking and problem-solving development, intellectual development, shows understanding of ethical conduct, demonstrates advanced level of content knowledge skills and methodologies, and applies feedback from mentor effectively.” — partner university
Build professional identity	Students talk about becoming a professional/researcher in their field of interest, what it looks like, what gaps may exist. This code also includes portfolio development, which creates and reflects the existence of a professional self	“Supporting development of research identity and self-efficacy (I can do science, scientists are real people, I have the same chance as anyone else to be a scientist)” — urban community college
Knowledge & building self-efficacy	Students learn about research components (could be path specific or path agnostic); helps build the competence to apply that knowledge	“Increase student self-efficacy (through practice of activities and communication related to EXITO core themes: awareness-building, responsibility, sustainability, ethics)” — urban community college
Research relevance	Describes making research more personal and relevant for students, typically by considering uses for specific populations in their own community, community service to understand needs in others, or other components that build relevance	“Homeless Outreach — To expose students to this vulnerable population which also suffers from health and socioeconomic disparities. Students also gain an increased awareness of issues surrounding homelessness on [island], and services available to assist homeless individuals and families. — partner university
Sociocultural dynamics	Describes talking about dynamics that affect marginalized populations in academic and research environments (such as first-generation college students, Black people, women in “hard science,” people with disabilities). Often includes strategies for responding to microaggressions, cultural bias, systemic racism	“Provide clear understanding of common expectations, and common issues or pitfalls that underserved and underrepresented students may experience on the educational and career path of their choice.” — urban community college
Professional development	Describes skills, growth, and access that support the ability to add research responsibilities to academic and personal responsibilities (may include time management skills, resources, self-reflection, i.e., what do I need to function as a professional?)	“As students become scholars, we discovered that there was a great demand for students to learn other hidden skills such as effective note taking, study habits, and skills for long-term retention of information through retrieval practice. There was also a demand for developing etiquette for professional environments such as conferences and meetings.”— primary university
Self-care	Similar to professional development, but focused more on mental health, social and emotional pressures related to work–life balance, and strategies that mitigate burnout in professional settings (i.e., what are the advanced skills required for me to flourish in a professional life, particularly in research and academia?)	“Life skills for underserved and underrepresented professionals (work–life balance; accessing support; managing high biosocial demands and stresses with high academic and professional expectations)” — urban community college
Cohort & team building	Provides opportunities for students to get to know each other, share stories/issues/concerns, and help each other in their path development	“Cross-cohort interaction is very helpful: peer mentoring, normalizing interest in science, information-sharing on transfer preparation and academic success.” — urban community college
Support	Describes helping students navigate supports available at institutions as well as responding to emergent needs. Also includes description of mentorship when it is described as a support for students.	“Regular in-person contact with Scholars: to listen and notice student needs, support access to resources, help identify the need. The student won't go to an advisor or student services until they recognize the need.” — urban community college
Institutional infrastructure	Describes institutions learning from students to improve instruction or discover student needs and create or improve resources/access. Often describes how sites institutionalized courses or student supports	“Give local EXITO core faculty a chance to “take the pulse” of students on a regular basis, identify red-flag issues before they become problems” — partner university

Statistical Analyses

Statistical analyses compared whether student responses differed between years and whether transfer status influenced ratings. “Course evaluation” data were descriptively analyzed within SPSS for means and standard errors, reported in graphical displays. Data were tested for normality using the Shapiro–Wilk test of normality and Levene’s test for homogeneity of variance within SPSS, with significance values above .05 describing normally distributed data. Nonparametric independent samples t tests (Mann–Whitney U test) were used to analyze Likert-scale data for statistical differences (Norman, 2010), including student data between survey years (i.e., 2019 vs. 2020) and starting institution (e.g., primary university vs. transfer institution) as well as instructor data (frequency and impact for university vs. community college sites).

Results

Qualitative Analysis of Themes

Sites described Enrichment across the following core areas: goals and core responsibilities; learning objectives; structure and format; measurement and outcomes; barriers; implementation solutions; and key changes over the 30-month study period. Data definitions and example quotes for all coded themes are described in Appendix D.

Goals and core responsibilities.

Sites were asked to describe the thematic goals of Enrichment as separate prompts (“goals” and “core responsibilities”), though responses were interchangeable and merged to facilitate reporting of themes. Twelve goals and core responsibilities emerged (see Table 2). Subthemes are described in Appendix E, with the prevalence of each theme across site type shown in Table 3. In October 2020, sites were presented with the summarized list of themes and asked to rate each theme’s frequency of inclusion and perceived impact on scholars at their site (Table 4), thereby highlighting core elements needed across university and community college environments to support URM students in biomedical research training programs.

Table 3.

Enrichment goals and core responsibilities reported across sites, sorted prevalence described across sites.

Prevalence	Core theme	Total sites n (%) describing theme; mentions/site	Primary university n (%) describing theme; mentions/site	Partner universities n (%) describing theme; mentions/site	University subtotal n (%) describing theme; mentions/site	Urban community colleges n (%) describing theme; mentions/site	Distant community colleges n (%) describing theme; mentions/site	Community college subtotal n (%) describing theme; mentions/site
1	Cohort/team building	9 (100%); 2.2	1 (100%); 32.0	2 (100%); 9.0	3 (100%); 13.7	4 (100%); 3.5	2 (100%); 1.0	6 (100%); 2.3
2	Professional identity	8 (89%); 12.5	1 (100%); 59.0	2 (100%); 5.0	3 (100%); 23.0	4 (100%); 6.5	1 (50%); 5.0	5 (83%); 6.2
3	Research skills	8 (89%); 11.4	1 (100%); 37.0	2 (100%); 13.5	3 (100%); 21.3	3 (75%); 3.0	2 (100%); 9.0	5 (83%); 5.4
4	Path representation/networking	8 (89%); 10.4	1 (100%); 44.0	2 (100%); 4.5	3 (100%); 17.7	4 (100%); 6.5	1 (50%); 4.0	5 (83%); 6.0
5	Professional development	8 (89%); 5.4	1 (100%); 19.0	1 (50%); 3.0	2 (67%); 11.0	4 (100%); 3.5	2 (100%); 3.5	6 (100%); 3.5
6	Research relevance	7 (78%); 5.4	1 (100%); 3.0	1 (50%); 9.0	2 (67%); 6.0	3 (75%); 3.0	2 (100%); 3.5	5 (83%); 3.5
7	Research exposure	7 (78%); 4.1	0 (0%); 0	2 (100%); 4.0	2 (67%); 4.0	3 (75%); 5.7	2 (100%); 2.0	5 (83%); 4.2
8	Institutional infrastructure	7 (78%); 3.3	1 (100%); 6.0	2 (100%); 3.0	3 (100%); 4.0	3 (75%); 3.0	1 (50%); 2.0	4 (67%); 2.8
9	Sociocultural dynamics	6 (67%); 6.7	1 (100%); 20.0	1 (50%); 3.0	2 (67%); 11.5	4 (100%); 4.3	0 (0%); 0	4 (67%); 4.3
10	Support	6 (67%); 5.7	1 (100%); 19.0	0 (0%); 0	1 (33%); 19.0	3 (75%); 4.0	2 (100%); 1.5	5 (83%); 3.0
11	Knowledge/self-efficacy	6 (67%); 2.2	1 (100%); 3.0	1 (50%); 1.0	2 (67%); 2.0	3 (75%); 2.3	1 (50%); 2.0	4 (67%); 2.3
12	Self-care	5 (56%); 2.2	1 (100%); 3.0	1 (50%); 1.0	2 (67%); 2.0	2 (50%); 3.0	1 (50%); 1.0	3 (50%); 2.3

Note. Instructors at each site were asked, “What are your goals [and] core responsibilities?” The number and percentage or each institution type mentioning that theme were calculated as frequency of sites mentioning a theme divided by total sites in that institutional setting. Data included one primary university, two partner universities, four urban community colleges, and two distant community colleges, for a total of nine sites. Mentions/site were calculated by dividing the number of mentions of the theme by the sites describing that theme. Of note, the primary university data reflect an increased number of instructors and mentions compared to other sites, because the primary university had three sections of students (sophomores, juniors, seniors) with separate leads, coded together to represent the site’s approach to enrichment.

Table 4.

Post hoc perception of Enrichment themes across sites, ranked by overall average (frequency covered and perceived impact on students).

Theme	Frequency of coverage (M; SD; n)				Perceived impact on students (M; SD; n)				Overall average rank
	University	Community college	Overall	Rank	University	Community college	Overall	Rank
Cohort & team building	87.3, 14.7, 8	73.8, 22.3, 6	81.5, 18.8, 14	1	87.6, 12.1, 8	84.2, 11.1, 6	86.1, 11.4, 14	1	1.0
Professional development	86.3, 16.9, 8	23.5, 26.6, 6	59.4, 38.3, 14; ***	3	91, 10.3,8	40, 26.5, 3	77.1, 27.9, 11; *	4	3.5
Self-care	79.4, 25.6, 8	28.7, 31.2, 6	57.6, 37.5, 14; *	6	84.4, 30.9, 8	67.8, 33.3, 4	78.8, 31.2, 12	2	4.0
Build professional identity	86.5, 18.6, 8	21.7, 40.2, 6	58.7, 43.8, 14; *	5	80.8, 23.5, 8	65, 49.5, 2	77.6, 27.3, 10	3	4.0
Support	81, 13, 8	60, 39.5, 6	72, 28.4, 14	2	69.4, 21.8, 8	76, 26.7, 6	72.2, 23.2, 14	6	4.0
Research exposure	70.5, 41, 8	43.3, 38.3, 6	58.9, 40.8, 14	4	74.4, 18.3, 7	58, 32.7, 5	67.6, 25.4, 12	9	6.5
Sociocultural dynamics	78.5, 25.7, 8	8.3, 13.3, 6	48.4, 41.5, 14; ***	9	84.1, 31.4, 8	91, 10.3, 1	77, 36.3, 9	5	7.0
Path representation & networking	79.5, 21.1, 8	18.3, 19.4, 6	53.3, 37, 14; ***	8	85.3, 19.7, 8	32.5, 29.9, 4	67.7, 34.1, 12; *	8	8.0
Knowledge & building self-efficacy	70.3, 33, 8	40, 43.4, 6	57.3, 39.4, 14	7	67, 32.7, 8	67.5, 34, 4	67.2, 31.6, 12	10	8.5
Research relevance	55.4, 31.7, 8	31.7, 41.2, 6	45.2, 36.6, 14	11	76.6, 21.4, 7	55, 63.6, 2	71.8, 30.7, 9	7	9.0
Research skills	52.6, 41.5, 8	36.7, 22.5, 6	45.8, 34.5, 14	10	65.9, 32.2, 7	49.2, 29, 5	58.9, 30.7, 12	11	10.5
Institutional infrastructure	52.9, 29.3, 8	26.7, 41.8, 6	41.6, 36.3, 14	12	56, 25.7, 8	50, 70.7, 2	54.8, 32.8, 10	12	12.0

Note. Instructors at each site were asked to rate each theme by frequency of coverage (0 = not at all; 100 = core component discussed frequently) and its impact on scholars at their site (0 = no impact; 100 = maximal impact). Statistical differences in means were analyzed by nonparametric independent samples t tests.

^*p < .05.

^***p < .001.

Learning objectives.

The stated learning objectives for Enrichment at each site were coded according to Bloom’s taxonomy of cognitive learning objectives (Adams, 2015) describing six ascending levels of cognitive order (1–6). Table 5 indicates these different levels, the proportion of learning objectives at each level, and a summary description of the types of learning objectives represented at each level (Table 5). Appendix F further describes thematic analysis of learning objectives as well as context, barriers, solutions and key changes.

Table 5.

Bloom’s taxonomy of cognitive learning objectives applied to Enrichment learning objectives across sites.

Bloom’s level	Proportion of responses	Description
1. Knowledge (recall)	10%	Describes the learning and repetition of information, such as identifying research methodologies or resources.
2. Comprehension (explanation)	26%	Centered on students talking with peers and professionals, since these activities emphasized understanding and explaining points of view that may be different from their own.
3. Application (information use)	32%	Describes students applying knowledge and using information to access student support services, apply rational decision making to their research, and implement ways to handle the demanding expectations of their educational and career plans.
4. Analysis (connection)	21%	Most often referred to evaluating scientific sources, making comparisons across scientific fields of study, and self-identifying gaps in professional preparation.
5. Evaluation (justification)	12%	Described using scientific information to make conclusions and describe limitations within research, as well as the evaluation and articulation of advantages and disadvantages of research careers so students could justify whether a path would be right for them.
6. Creation (production)	14%	Referred to development of professional materials, such as personal statements, professional portfolios, oral conference presentations, visual poster presentations, and other materials that showcased students’ development and identity as professionals and researchers.

Structure and format.

This core area referred to logistics and common practices for running an Enrichment program. Enrichment generally consisted of 60- to 90-minute sessions offered weekly or biweekly, although some sites held them less frequently (e.g., 1–2 times/term). Depending on the size and capacity of the site, Enrichment was facilitated by one or more faculty and/or academic professionals. A thematic analysis of program components is summarized in Appendix F, including instruction, frequency and duration, audience and attendance, approach to content, activities, instructional supports, and online resources.

Measurement and outcomes.

Sites reported measurement outcomes and measurement opportunities, as well as areas where they would feel strong or weak in evaluating learning objectives (Appendix F). Few sites implemented “course” evaluations, though several cited a desire to work with the program’s evaluation team to design and implement Enrichment evaluation with their students. Subthemes around professional development and student engagement were cited as measurement outcomes of interest. In 30-month follow-up surveys, evaluation had increased at one distant community college site (e.g., positive impact of research symposia and students describing “how…the peer mentoring they receive and support from their cohort and previous cohorts have had such a great impact on their success in the program”). Another urban community college site reported an easy metric for estimating cohort engagement:

One of my hallmarks of whether a cohort has been established is when students begin to ask how other students are doing or where they are at if they miss a class or meeting. That indicator that they are looking out for and supporting each other is why I ranked the cohort and team-building component relatively high for our students.

Barriers.

Barriers refer to implementation challenges (e.g., travel logistics, limited access to scientists at distant sites, busy student schedules) as well as larger systemic barriers that intersect to impact students’ progression in biomedical research training programs (Appendix F). For example, 71% of the program’s students are on need-based financial aid. An urban community college partner cited the NIH requirement that students must be continuously enrolled with full-time credits to participate in the program. However, since students are unable to reduce their course load without losing all their program funding for that term, extra support is required to keep students enrolled full time through a crisis (e.g., illness, death in family) or relocation to a new institution.

Implementation solutions.

Implementation solutions describe strategic considerations and effective solutions for designing and delivering Enrichment’s nonformal, interprofessional structure (Table 6). These include approaches to both delivering content (e.g., mentored support, peer sharing, core logistics) and expanding delivery or reach (e.g., institutionalization and partnerships; sharing across sites). Enrichment served as a way to identify student needs as well as refer students to resources, when needed (Appendix F). Appendix G describes additional lessons learned for implementing Enrichment with biomedical research trainees.

Table 6.

Enrichment implementation solutions across sites.

Solution	Proportion of responses	Description
Mentored support of professional development	40%	Described the value of providing support to students to help them navigate their professional paths, identify training gaps, understand cultures, and integrate new skills (e.g., time management by online scheduling) into their larger identities. Student support and success solutions consistently included peers, program alumni, and diverse professionals in flexible mentoring processes. Several sites described offering support to students for professional writing.
Peer sharing and student-led content	35%	Described the high value of including peer sharing and student feedback in the development of content and goals for Enrichment. Given students’ busy schedules, supporting their autonomy in how they meet their needs is key. Sites described that as scholars became networked with each other, they learned to support and rely on each other in times of strength, weakness, success, and failure.
Logistical supports	30%	These included consistent day, time, and meeting locations to improve attendance, as well as the use of course management software to help share announcements and resources with students. Sites described integrating evaluation into their processes, particularly for new events, as important logistics to consider to ensure activities are working for students.
Institutionalization and partnerships	25%	Referred to processes for institutionalizing Enrichment with or without additional research instruction (i.e., a Gateway to Research course that teaches research ethics and methodology). As sites recognized the importance of a peer base for Enrichment, they became creative in how to increase the number of students accessing Enrichment. Two sites hoped to offer Enrichment as a noncredit course to recruit students beyond the program, and others implemented formal and informal partnerships with other STEM programs that supported undergraduate research (e.g., NASA URISE) to implement Enrichment across a wider student population. This strategy likely benefits both institutions and students since Enrichment excels as an interprofessional environment. However, as Enrichment becomes institutionalized, one urban community college said it best: “While institutionalization of the [site’s] Gateway class has been our goal for a number of years, now that it appears to be coming to fruition we need to ensure that the move from “unofficial” to official class with greater numbers of students won’t lead to a loss of informality. It is this atmosphere that has encouraged and allowed open and honest discussions. It would be a shame to lose this environment that enriches all participants, not just the scholars.”
Sharing across sites	14%	Referred to sites’ desire to share resources, instructional materials, and lessons learned with greater intent and frequency. Structures to share resources and activities online was encouraged, as was establishing instructional development around Enrichment as a regular part of the annual EXITO conference.

Note. STEM=Science, Technology, Engineering, and Mathematics; NASA=National Aeronautics and Space Administration; URISE=Undergraduate Research Internship in Science & Engineering); EXITO=Enhancing Cross-Disciplinary Infrastructure and Training at Oregon (EXITO)

Key changes over the study period.

Baseline and follow-up data were coded to understand significant changes that sites made to Enrichment over the 30-month study period. Key changes included a greater emphasis on professional development, implicit curricula about social norms and dynamics of academic and research cultures, and self-care in science (Appendix F). Over the 30-month follow-up, instructional teams became more diverse, including not only staff and faculty, but postdoctoral fellows and program alumni who returned as hired staff to serve as peer mentors and develop scholar-facing content (e.g., web content, management of Slack/Discord channels, and topic tutorials) for the Enrichment program. A community college site also recommended effective resources for their students, such as the At the Bench book on laboratory practices (Barker, 1998), which they reported “helped [students] feel more competent and confident entering the research setting” and “reduced feeling[s] of imposter syndrome.” Team-building (with peers, peer-mentors, faculty, and staff) was essential; sites’ reported that it helped scholars’ build academic self-efficacy and confidence, constructs that can be measured among students as a future direction of this work.

Effective curricular arcs for biomedical enrichment.

Summarized themes for 3-year instructional arcs were successfully implemented at the primary university over the 30-month study period (Table 7). Core arcs describe areas of work such as developing identity, finding fit, communicating identity as a professional, and gaining access to advanced academic and professional systems. When examined with Table 4, institutions should integrate peer discussion (cohort/team-building) as a best practice when discussing these themes.

Table 7.

Effective curricular arcs and goals for Enrichment across a 3-year training program.

Area of work	Core questions	Program Year 1 (Sophomore)	Program Year 2 (Junior)	Program Year 3 (Senior)
Developing identity	Who am I? What am I good at? What do I want to do?	Strengths assessment; cohortbuilding activities; exposure to academic and scientific fields	Identifying field(s) of interest; identifying academic/research interests; identifying a good program/mentor fit	Networking, interview, and presentation practice; creating a professional identity online and in person (e.g., individual development plan, e-portfolio);
Finding fit (integrating identity and role)	What are the roles and groups within academia? In research? What role(s) do I have? What group(s) am I part of? What roles in academia and research are a good fit for me? What groups (sociocultural, professional, academic) are a good fit for me?	Exposure to diverse academic and scientific professionals and communities; cultures of science across natural, social, and clinical fields; understanding systemic barriers and imposter syndrome; degree/job paths across natural, social, clinical, and professional fields; roles in research teams	Small-group talks; deepening exposure to diverse professionals and communities; identifying academic pathway(s) of interest; building network using professional associations and conferences; cultures of research (e.g., academia, industry, government settings); negotiating strategic self-disclosure as an underrepresented professional	Negotiating underrepresentation; transition strategies for the following year (gap year, postbaccalaureate programs, work-to-graduate-school paths); completion of remaining steps (e.g., GRE, MCAT, interviews); defining a personal timeline and needs in a job or program; demythologizing the “straight track” to a terminal degree
Communicating identity as a professional	How do I represent my identity to others as a professional? How do I recognize and describe, in a professional way, roles and groups that fit me well?	Developing documents used for research placement (e.g., resume, short interest statement); networking using student groups; professional tools and skills (e.g., business cards, presentation practice, negotiation skills)	Building an application portfolio (for postbaccalaureate, graduate school, professional programs, internships, jobs), including personal statement, curriculum vita, cover letter, statement of research interests, asking for letters of recommendation	Revising the portfolio, with poster(s), publication(s), conference experience, updated curriculum vita and recommenders; creation of professional digital identity; networking and planning skills
Gaining access	What do I need to do to join/create these groups and communities? What is my next step? What do I need to do to gain access to/qualify for these roles? What is my next step?	Building relationships with career and research mentors; research laboratory placement; networking with peers, faculty, and peer mentors	Graduate school, professional program, postbaccalaureate, internship, and job list; informational interviewing; funding sources for advanced schooling	Professional association membership(s); “offboarding” plan describing graduate program choice or gap plan with funding goals, target employers, or postbaccalaureate application; transition to paid support role in the path of interest

Student Evaluation of Enrichment

To understand student perceptions of Enrichment, students at the primary university were given “course” evaluations at the end of the academic year. Data were collected from students at all community colleges and the primary institution, but not from partner universities. A total of 57 seniors completed Enrichment course evaluations (Figure 2), representing 33 Cohort-2 scholars (of 57 in March 2019, 58% response rate) and 24 Cohort-3 scholars (of 56; in March 2020, 43% response rate). The response rate averaged 50%, representing 13% of EXITO’s total student population (N = 427; Table 1). Scholars agreed with most statements about Enrichment (Figure 2; percentage agreement calculated from Likert scale ratings >3), including that it made them feel supported (98% agreement), had a large impact on [their] professional development (95%), and helped [them] understand career path options (91%). Responses were similar across the 2 years (p ≥ .05), with higher scores observed in 2020 for organization (4.6 to 5.2; p < .01) and resources/materials (4.6 to 5.0 p < .03). Cohort-3 course evaluations asked scholars if they started EXITO at the primary institution or transferred, with no statistical differences in any ratings based on scholars’ starting institution (p > .07–1.0). Scholars grew in their appreciation of Enrichment being mandatory by 1 point, from 5.6 ± 0.3 at the start of Enrichment to 6.6 ± 0.3 at the end (1 = strongly dislike; 10 = strongly like). Senior scholars were asked if they would attend if Enrichment was not mandatory, with 33.3% indicating yes, 45.6% indicating their attendance was dependent on the session topic, 14% reporting no, and 7% unsure. At the 30-month check-in, the primary institution had implemented additional engagement options to accommodate students with busy schedules and diverse needs, including online cross-cohort sessions, supplemental enrichment, and weekly writing workshops offered virtually to support students across sites.

Figure 2. “Course evaluation” data from seniors completing EXITO at the primary university.

A total of 57 seniors completed course evaluations and rated Enrichment on a Likert scale of 1 (strongly disagree) to 6 (strongly agree). Mean and standard error are shown, with scholar percentage agreement (rating >3) inset. Items denote 57 responses except for “career path” and “writing” questions (n = 56) and “connecting with peers” (n = 24; asked of Cohort 3 only).

Discussion

Enrichment served as an important nonformal support structure for historically underrepresented undergraduates pursuing biomedical research training. Enrichment is defined as a learning structure that enabled students to have regular connection with peers, faculty, and staff over time for the specific purpose of addressing the needs of students navigating academic and biomedical research training paths. Understanding what supports are needed to retain historically URM students in research training is critical to enhance representation in the biomedical workforce (Duffus et al., 2014; Estrada et al., 2016; Hinton et al., 2020; Valantine & Collins, 2015; Valantine et al., 2016). The nonformal, interprofessional structure differs from other academic and training environments and fulfills a needed role. For example, while course instruction enables group contact with students about a topic, it is limited in time (i.e., term) and predefined by instructional learning objectives. Mentoring provides regular contact over time and is driven by student needs, but each mentor sees a limited subset of students; patterns that affect a larger group or minorities within a group may be invisible to both mentor and mentee. Advising offers an individual lens; the advisor sees the pattern of needs in many students over time, but interactions are typically 1:1 without peer interactions or group feedback to advisors. The Enrichment role was unique. This study of Enrichment implementation represents a multisite case study of how regular group interaction over time with an interprofessional population of historically URM students can inform faculty and staff understanding of the diverse population’s needs, and the kinds of program or institutional activities that can address them. While URM refers to students from historically underrepresented racial and ethnic minority groups, the term is also inclusive of students with disability and disadvantaged backgrounds, including those in the foster care system and from low socioeconomic backgrounds. EXITO serves a diverse population of students, including historically URM students, gender and sexual minority students, and Asian/Pacific Islanders who grew up in socioeconomically-disadvantaged environments in American Samoa, Guam, Hawai’i, or Northern Mariana Islands.

Enrichment instructional team members had very diverse job roles in higher education, came from different demographic backgrounds, had attained degrees in diverse STEM and non-STEM fields, and had no common training beyond research ethics (Appendix F). Based on their regular group contact with diverse underrepresented undergraduates pursuing biomedical research training, all sites independently developed Enrichment with the same features: peer- and near-peer-based facilitated interprofessional learning, with access to diverse professionals in contexts where they did not have direct authority over the student’s grades or work, using nonformal education structures.

Enrichment’s ability to meet needs across nine diverse institutions (three universities and six community colleges across the Pacific Rim) suggests high potential generalizability and applicability to other sites nationwide. In nonformal structures, instruction responds to learners in situations rather than adhering to predefined instructional objectives (Melnic & Botez, 2014; Yasunaga, 2014), with students’ voices and available opportunities used to develop instructional goals. The Enrichment process was highly valued by students, as 98% reported in course evaluations that they felt supported and 91% agreed that Enrichment helped them understand career path options. Enrichment provided facilitated time for students to explore biomedical paths, visualize themselves as scientists, understand systemic barriers, add academic and research languages and cultures to their tool kits, and find ways to access further education and research that fit their life situations, interests, values, and goals. Supporting students’ informed choices and autonomy about their career decisions is vital for students who face barriers to biomedical research careers, such as URM students (Chang et al., 2014; Hrabowski et al., 2011; National Science Foundation [NSF], 2017), particularly since advanced training in these fields may take a decade or more.

Interprofessional Setting of Enrichment Mirrors Biomedical Research

All sites reported that the interprofessional setting of Enrichment, an important feature of biomedical programs given their diversity of majors and degree paths, was also important for student development. Interprofessional education refers to students from different disciplines learning with, from, and about each other (Health Professions Accreditors Collaborative, 2019) and has become an increasingly common component of scientific training programs (Averill et al., 2019; Health Professions Accreditors Collaborative, 2019). Enrichment’s interprofessional setting supported student-led modeling and exploration of different professional degrees (e.g., M.D., R.N., M.S.E., Pharm.D.), research paths (Ph.D., M.D./Ph.D., M.S.), and career options after graduation. As diverse students explored these paths together, they self-defined paths, programs, and timelines that would allow them to successfully pursue their goals given their situation and interests. The interprofessional setting enabled students across the physical, biological, clinical, and social sciences to share similar experiences and feelings about professional identity development, which helped others realize they were not alone when questioning how they could become a scientist and what it means to be one. Identifying common experience was key for underrepresented professionals to distinguish personal barriers from systemic barriers and identify when solutions needed more individual effort or better access to systems. Enrichment provided a space and time in which URM individuals were the visible majority and could serve as role models, informal mentors, and search images for “scientists like me.” The inclusion of program alumni in Enrichment was particularly meaningful, as these students could share proven strategies for negotiating underrepresentation and balancing research, academics, and personal life. While formal faculty mentor meetings have their own unique value, our study data show that student support and success solutions identified across sites consistently included peers, program alumni, and professionals from a variety of backgrounds, including underrepresented populations.

Enrichment Provides the Reflection Time for Students to Develop Their Professional Identity

While a desire exists among grantors and training programs to help students find career paths and access graduate programs as quickly as possible, underrepresented students may need more time to observe and evaluate the academic and research landscapes. Most program students (71%) received need-based financial aid, 86% reported a disadvantaged background, and 60% were first-generation college students. Graduate fields and degree programs differ in their financial support for students (National Center for Science and Engineering Statistics, 2018b, 2018c) and students’ path selection can influence accrued student debt (National Center for Science and Engineering Statistics, 2018a). Financial and class barriers compound the challenges of less access to mentorship and research (Cohen & Garcia, 2008). URM students receive doctorate degrees at lower rates than non-URM students (National Center for Science and Engineering Statistics, 2018d; NSF, 2015) and they are more likely to have graduate debt and higher amounts owed across STEM fields, especially in social science-related STEM fields (National Center for Science and Engineering Statistics, 2018e). Hoppe and colleagues (2019) recently showed that African American/Black scientists pursue community and population health fields at higher rates than white scientists and that these fields have lower funding levels despite higher impact of publications. Moreover, Muslim scientists may practice riba, an Islamic practice in which loans with interest are avoided, which restricts how a scholar can pay for graduate schooling. Hinton and colleagues (2020) highlighted funding opportunities for scholar training across trajectories. Together, these studies suggest that giving underrepresented students the time and support to plan feasible paths and make informed decisions may help increase the percentage who complete advanced degrees.

Enrichment helped students self-define their own professional interests, paths, and goals, critical for developing a professional identity that integrates their personal and professional selves (Kasperiuniene & Zydziunaite, 2019). This is a large job when historical exclusion from a field limits representation and modeling of paths. Universities’ Enrichment programs discussed themes of professional development, sociocultural dynamics, path representation, professional identity, and self-care more frequently than those at community colleges (Table 4). Perceived impacts of themes for students were comparable across university and community college sites, with the exception of professional development and path networking, which were rated higher at university sites where they were also more frequently discussed. Community colleges underscored the importance of team building (for building comfort with research and personnel) and supports for their students (facilitating students’ learning about how and where to ask for help at their institutions). By placing these themes in context for students, Enrichment’s reflection time and activities enabled students to relate research to their lives and identify practical strategies for negotiating barriers to biomedical careers that fit their interests and goals. Students at the primary site described high interest in gap years and funded postbaccalaureate programs for this reason, as they would provide additional facilitated time to explore their desired field while increasing research experience in that area. Our findings are consistent with those of Hinton and colleagues (2020), who underscored the importance of this transition time for URM students.

Enrichment Surfaces Implicit Curricula

In academia, personal relationships across authority lines provide the recommendations required for further education, as well as access to research jobs and key information on graduate programs and funding. To advance, undergraduates must build good relationships with people who have direct authority over them. However, we found formal scientific relationship expectations to be largely unknown to students, especially for first-generation college students. When asked in Enrichment to identify potential individuals they could ask to write them a letter of recommendation, several students did not identify their laboratory mentors despite working in their research laboratory for over 2 years. Likewise, students asked whether they must say “no” to coffee with a graduate student in their field who was their project supervisor and offered to talk with them about graduate programs. As academia has the second highest rate of sexual harassment of all workplace types, behind the military (Ilies, Hauserman, Schwochau, & Stibal, 2006), it is reasonable for students to question what constitutes an appropriate work environment. Coe and colleagues (2020) highlighted that mentoring and training regarding potential challenges that may arise in the workplace, including potential harassment, are needed for leadership development in academic medicine, particularly for URM individuals and women who face higher rates of harassment (National Academies of Sciences, Engineering, and Medicine, 2018). While sexual harassment was not reported by sites in our study, it often goes underreported in academic medicine due to concerns that it may threaten careers (Bates et al., 2018). Helping students recognize appropriate contexts for their research training and how to respond to potential challenges is critical for training in any field. For underrepresented undergraduates facing imposter syndrome (Bravata et al., 2020), academia places intense stress on becoming what is perceived to be desired by authority figures. Sites reported that their students consistently self-identified imposter syndrome as a problem, even as they gained research experience and credentials, including first-author publications. Our observations underscore a need to explicitly address hidden and implicit curricula with undergraduates as part of biomedical research training, which was also identified as a need for professional development training by both junior and senior biomedical faculty (Rubio et al., 2019) as well as undergraduate programs (Hinton et al., 2020; Merolla & Serpe, 2013; Toven-Lindsey, Levis-Fitzgerald, Barber, & Hasson, 2015). Together, these findings highlight a common need for instruction across all biomedical training levels. By creating space for students to talk together about navigating complex barriers and issues, we believe Enrichment structures are able to strengthen the professional development URM students need for advanced scientific careers.

Enrichment Highlights the Accessibility of Supports for Institutional Development

Underrepresented undergraduate researchers are both high-needs and high-performing students. Enrichment offers institutions insight about supports their students need, how accessible those supports are, and what institutional development efforts could better serve students—all factors highlighted by Estrada and colleagues (2016) as needed for improving URM student persistence in STEM. The structure of Enrichment typically incorporated instructional teams including faculty, staff, peer mentors, program alumni, and sometimes graduate or clinical students from nearby programs. Consistent staffing within instructional teams helped programs optimize Enrichment curricula over time and build trusting relationships with program students as well as relevant institutional faculty and staff. Staff often serve high-needs students, whereas faculty typically serve “high-performing” students, roles that were bridged by instructional teams in Enrichment. By using nonformal education structures to engage students as partners in the design of Enrichment content, students were able to have their needs met while programs gained rapid feedback about emergent student needs, available resources, accessibility of supports, and strategies for training diverse students in STEM and research. Enrichment also identified cross-institutional support options, such as accessible remote resources online, and potential strategies at the federal level, such as grantor flexibility for part-time enrollment to enable undergraduate trainees pursuing biomedical research programs to better cope with added life stressors that may arise without losing that term’s financial support. One instructor described Enrichment as a “flashlight that illuminates the landscape” by pointing out challenges that all students face when trying to flourish in academia and research but are particularly prominent for underrepresented students. Therefore, examining supports that work for underrepresented populations in undergraduate biomedical research training programs may yield important insights for supporting more biomedical research trainees, akin to universal design supports in education (Al-Azawei, Serenelli, & Lundqvist, 2016).

Access Barriers to Enrichment

Common barriers to participating in Enrichment included intractable schedule conflicts (biomedical major prerequisite courses, solo caregiver role for children or disabled adults, survival-income work) and lack of transportation access. Sites reported off-campus events that required transportation and schedule coordination had lower attendance. The consistency of required Enrichment sessions at a regular time enhanced students’ ability to plan their academic and research schedules. However, the balancing act of Enrichment was highlighted in data from student evaluations, where 45.6% of scholars indicated that if Enrichment was not mandatory, their participation in sessions would depend on the topic or the day/time. As most students agreed that Enrichment helped connect them with resources and that its materials and resources enhanced their learning, barriers surrounding access should be considered by future sites planning similar activities. Sites recommended that if attendance was required, it should be optional in midterm and finals weeks. All sites agreed that requiring students to produce material to be graded or judged was outside the nonformal structure and goals of Enrichment. At the primary site, universal access was provided through regular sessions, supplemental small groups at a different time, and a limited number of individual online make-up activities (supported by in-person appointment scheduling and online Enrichment resources). As online makeup work lacks community-building and peer-sharing components cited as helpful by program students, it was allowed to supplement, but not replace, in-person participation in peer sessions. Options added since the 30-month check-in included cross-cohort virtual sessions and weekly writing workshops.

Access barriers also existed for institutions: Small, distant community college sites reported limited direct exposure to research settings or well-equipped libraries of professional biomedical literature in diverse fields. They also reported access to fewer professionals to meet with their students. Partnerships across programs, departments, campuses, and institutions helped improve access. When students have equitable access and exposure to research training, programs and institutions can choose the best applicants rather than the applicants who had the best access (often those from the most privileged backgrounds).

Self-Care as a Strategy for Systemic Trauma

Self-care was given increased focus across sites as Enrichment evolved over time. Self-care is typically described as strategies for stress management and health (Butler, Carello, & Maguin, 2017; Myers et al., 2012; Roulston, Montgomery, Campbell, & Davidson, 2018). For underrepresented student researchers, self-care was discussed in the context of professional development, navigating the combination of systemic trauma (e.g., exclusion and abuse in systems; imposter syndrome; generational poverty; health disparities), and the typically poor work–life balance of academic, medical, and research cultures. In this population, students must recognize that needing self-care is not a personal failure and identifying strategies to cope is important for their professional development. Students in research training programs may stay in these programs despite high stress and lack of health supports because the opportunity to advance their career goals is too important to pass up. Students completing Enrichment reported that it made them feel supported, addressed their core needs, and had a large impact on their professional development.

Future Directions

Online sharing of resources across sites and greater use of teleconferencing is desired to increase access and reach to distant sites. Students at the primary site have requested podcasts, while instructional teams have requested digital platforms for sharing materials. Program alumni have asked to maintain contact with the program; a subset of alumni work for the primary institution’s Enrichment on key tasks, such as building an online alumni network, creating online resources for cross-institutional remote chat, and curating access to a library of support materials. Students at multiple sites have piloted student-led projects as part of Enrichment, which offer new opportunities for developmental, cross-institutional networking. Students help define Enrichment, alumni develop its resources, while faculty and staff provide support and mentorship. Together, sustainability is improved and these developing professionals gain creative opportunities for publication, presentation, teaching, and other forms of curriculum vitae development.

Conclusion

Nonformal, interprofessional curricular structures support underrepresented students by offering a flexible and responsive environment for enhancing the biomedical research training of undergraduates. These findings are consistent with those from undergraduate and faculty STEM development programs that implement peer-enhanced supplemental instruction (Merolla & Serpe, 2013; Rubio et al., 2019; Toven-Lindsey et al., 2015). Enrichment engages diverse students in peer spaces and facilitates activities that give them a sense of ownership over the development of their professional identities, as well as practical tools to succeed in the academic and professional path of their choosing, on their own terms. The bidirectional benefits can be extended to programs and institutions that can use Enrichment-like structures to gain a greater understanding of how to support diverse student success in biomedical fields at their institutions. These practices align with recommended approaches for ultimately improving URM student persistence in STEM (Estrada et al., 2016).

Appendix

Appendix 1. Origin and early structure of Enrichment.

Start of student experience.

A program orientation event at the primary site welcomed new student participants from all sites. This event responded to the recognized need to create a supportive environment for students, allow them to connect with peers, learn about campus services and supports, explore professional development opportunities, and engage in activities and events fostering a sense of shared purpose and community.

Origin of enrichment.

At the primary site, an EXITO academic professional and the program’s academic advisor created and led optional group activities in the first year (cohort 1) to help students network with campus professionals of interest, provide a forum to share information conveniently, and provide a sense of structure and belonging for the program, especially in its first year. Early student feedback suggested these activities were very important, but that most program participants had too many competing responsibilities to attend optional events. In year 2, EXITO leadership defined Enrichment as an official program intervention, with all sites needing some type of student Enrichment, though the form, content, and definition of Enrichment were left up to each site. Activities at the primary site were converted into a regular schedule of required weekly sessions for cohorts 1 and 2, run by a faculty lead. A small working group of faculty and staff at the primary institution and a transfer partner community college provided support for this transition; the group had come together to identify common barriers for underrepresented undergraduates, and members were experienced in direct support roles with URM students. The chair of the working group, a non-faculty academic professional with teaching experience, became the coordinator of Enrichment. As university faculty availability varies year to year, including an instructor/coordinator in an Academic Professional staff role provided program consistency. During year 2, Enrichment sessions at the primary site were on Fridays for both cohorts, sometimes run as a single session. By the start of year 3, the primary site instituted separate Enrichment sessions for each cohort with three new faculty leads and the coordinator.

Appendix 2. Prompts used for site evaluation of Enrichment.

Sites were sent a digital document asking about Enrichment implementation at their institution, with five prompts (April 2018):

1. Who does Scholar Enrichment at your institution?

2. What are the “nuts and bolts” of your Scholar Enrichment? (how often, format(s), etc.)

3. What are your goals?”; “What do you think are the core responsibilities of Enrichment?

4. If those are the core responsibilities, what are your primary learning objectives (L.O.)?;

5. If you were going to be assessed on these L.O.s, what would you change or what would you need to feel confident?.

Follow-up prompts (November 2019):

1. What have you learned about offering enrichment to your students?

2. What feedback have you received from students about enrichment thus far? (course evaluations, word of mouth, anecdotes, etc.)

3. Anything else you want us to know?

Member checking prompts (October 2020):

1. Which areas do you cover in your site's enrichment (0 = not at all; 100= core component discussed frequently)

2. Please rate each Enrichment theme for its IMPACT on scholars at your site. 0 = no impact; 100 = maximal impact.

3. Any feedback on the above?

Appendix 3. Enrichment “course” evaluation used to measure student impact at the primary university.

Appendix 4. Data definitions and example quotes of observed themes reoorted by Enrichment sites.

Code	Data Definition	Example Quote(s)
Instructor	Describes who does Scholar Enrichment at the institution (for example, solo instructor, team instruction, is the instructor level a faculty member or staff)	“Teaching teams: [one lead] for each cohort, one coordinator on all the teams, and peer mentors (1st year) or grad students (years 2&3).” “Teaching teams meet every other week to plan the next sessions; all teams meet together once a term.” -Urban University
Structure and Format	Describes what the “nuts and bolts” of each site’s scholar Enrichment, such as frequency per term, duration, and general format through inclusion of speakers, etc. Subthemes observed:	Frequency and Duration: “Meet weekly for 1.5 hours; First 30 minutes: students present on their work; other students critique the presentation in writing [students not yet placed in labs present on interests]. Remaining hour: interactive discussions on “Hidden curriculum” topics, such as how to communicate research with potential funders, faculty, conference attendees, friends and family; where to find funding and research opportunities outside of [institution]; designing and updating CV; writing manuscripts; presentation tips; code switching; identifying research passion; balancing life/work/research/classes. –Urban university Audience and Attendance “Program stability may require a minimum of 10-15 students; goals are difficult to meet with <6 students in a session.” –Urban Community College Approach to content “As the only regular in-person meeting point for Scholars at [site], Enrichment tends to handle a variety of other program needs/activities.” – Urban university Format and Activities “Content: lecture, group activities, panels/mixers. Students prefer interactive formats--practice skills and talk with professionals.”-Urban university Instructional supports/ online resources “[Three scientists] conducted a panel-style presentation on their respective research followed by a question and answer session. [Scholars at another distant institution] were able to view the presentation via zoom video conferencing. – Partner university
Learning objectives	Sites defined primary learning objectives for their students based on defined goals. Learning objectives often mirrored the goals, therefore, the code “learning objectives” was applied to individual child codes under goals and objectives to identify where alignment most frequently took place.	“Engage in collaborative and interdisciplinary approaches and teamwork for improving population health.”-Partner university “Collaborate with others from diverse backgrounds in addressing health disparities and inequities.”-Partner university “Explain the influence that science and technology have on individual and population health.”-Partner university “Critically analyze implicit bias and the barriers to equity” –Primary university
Measurement and Outcomes	Sites described approaches for evaluating learning objectives and outcomes observed. They also described strength of evidence and what support would be needed to feel confident in their evaluation.	“We have incorporated a lot of formative evaluation, for each special event, for important topics we are introducing for the first time, or at the end of many terms.[…] We use this data to make improvements to enrichment and address remaining gaps in the next session. One of the things most important is that students feel that their voices are being heard, which makes the data quality better (defined by more descriptive open ended responses). Students knowing that we care about their responses and are using their data has increased their feedback to us.” – Primary university “Gather pre-Enrichment and post-Enrichment data on Scholar use of: faculty office hours, accessing [disability resource center], accessing institutional support services/campus resources.”-Urban community college
Barriers	Barriers described to implementing enrichment (student, institutional, geographical, etc.)	“Access to speakers has been very sparse on island so we typically have 1 or 2 presentations per semester.” – Distant community college “We also talk about the transfer process often and what it looks like when a student is juggling a full course load and [program] tasks, along with outside responsibilities. Our demographic at the community college includes students that have a wide range in their preferred number of credits per term, either due to work, family, or educational needs. Transitioning into [training program] and the required continuous full time (FT) status does cause anxiety in some students, so we try to talk about that in advance. We also find that having early information on campus resources (financial aid, [disability research center], childcare, etc.) at both [community college] and [primary university] has been critical for students. – Urban community college site. “While the focus is still on both (i) enrichment and (ii) academic/curricular goals, the weekly hour-long sessions have organically grown to be more focused on enrichment rather than the material in the class modules (that is, a lesser focus on the academic and curricular goals of the class). This has been the result of a better understanding of what would most benefit scholars at this early stage of their research careers.” – Urban community college Scheduling and transportation are still challenging. Enthusiasm seems higher so far.” –Urban community college
Implementation Solutions	Describes approaches that programs have used or ideas that could be tested. Includes description of other existing opportunities that could be synergistic or help enrichment take hold institutionally.	In the fall semester, most enrichment activities have been incorporated into the Gateway [to Research] course. – Distant community college Considering institutionalization [of Enrichment] as a student club; the student clubs organization has funding but consistent leadership is a challenge. – Urban community college “Students won’t attend unless they are held accountable for attendance. Through a [program] meeting at [a scientific] meeting, [instructor] found that other schools offer 0-credit classes that go on the transcript but are essentially free; students receive a pass/fail grade. We will explore this.”-Partner University In terms of outcomes, student satisfaction with enrichment is a low-hanging but critical outcome. If they don’t feel that their needs are being met, enrichment can feel like another thing they have to do, which is hard when scholars are already so busy. Building in time for scholars to talk with each other is one easy way for increasing satisfaction, another is helping with their writing, as well as giving opportunities to meet professionals. Focusing solely on skills development and attainment can leave some scholars who have not solidified their professional identity feeling left behind. Interspersing professional identity development discussions is helpful, particularly when time horizons are discussed in the context of important considerations (e.g., financial, family in town, caring for elders, etc.), which can influence time until pursuing graduate school and if that path is feasible for them in the next five years. Many students are highly interested, but the financial barriers are significant. Focusing on post-bacc programs is great because it’s a way for students to gain protected time in a mentored environment, especially when placements are funded.”-Primary university “[S]tudents are motivated to continue in their academic programs. These activities create a strong learning environment where interaction among a diverse group of college students takes place. This builds a sense of belonging and sense of community among the [program] Scholars.” – Distant community college“
Change over time	Child codes of “change” or “no change” applied to components that evolved over the study period.	“In the last consortium meeting held [at primary university] in March 2019, we learned that Enrichment activities don’t have to be solely focused on research – but rather being a successful scholar. Up until that point, we felt that the lack of opportunities to provide exposure to research really held us back from providing what we thought were “enrichment” activities. We realized that we did in fact provide enrichment in other ways that we weren’t accounting for. For example, all of our scholars have to take a prerequisite course prior to the Gateway Course called College Success. College Success covers many Enrichment activities like time management, being professional, etc. However, after the Gateway Course, we have now decided to have them participate in at least 3-4 of [site’s] Student Success Series. – Distant community college “Allowing scholars to attend other cohorts’ sessions (and being explicit which are open to other scholars) has been reviewed favorably. Transfer students get more of the professional development skills that sophomore Enrichment covers. Students get more autonomy in picking sessions that would be beneficial to them. – Primary university “Offering students flexibility in how to receive the content is highly impactful. Regular sessions, supplemental sessions (smaller ~4-8 students with facilitator), online make up. Students want to get out of enrichment because it frees up time, but after going through it, students report that the time spent was valuable in helping to put their training and research experiences in perspective. The cohort building piece is significant.”-Primary university

Appendix 5. Core themes and sub-themes described by sites.

Theme 1. Cohort and team-building.

All sites described building an interactive group environment that would foster a “sense of relatedness”, “sense of belonging”, or “sense of community.” Of the 50 quotes in this theme, 78% mentioned ‘Group conversations to support students’ professional and personal growth.’ They referred to students participating in conversations together, reflecting with each other on their academic and scientific experiences. ‘Program and/or discipline cohort building’ was mentioned in 24% of code applications and described supporting students’ sense of community/belonging with their program, with others in their cohort year (year of training program), or with their professional area of interest. The primary site reported contact across cohort years was highly meaningful for students, particularly when student numbers in an area may be limited. Less common academic and research disciplines (e.g., speech-language pathology, physical therapy, community-based participatory research) and demographics (e.g. Muslim, trans person, intersectionality between race/gender) were not always represented within a student’s official cohort, even in the largest program in the consortium. Similar students with different levels of experience (near-peers) could overcome this barrier in cross-cohort sessions or informal discussions, where they would share information about successful strategies.

Proxy for support’ was described in 24% of this theme’s code applications and referred to sites using cohort/near-peer networking and group activity as a proxy for missing or inadequate support structures. Codes described peer and near-peer discussions in which a specific common struggle was revealed, identifying a previously unrecognized need for URMs in the program. The students would then support each other, share information on available options, and/or advocate with the program or institution for a specific improvement. Institutions and the EXITO program were able to use Enrichment to identify common challenges faced by their underrepresented student populations rather than responding to these issues as individual personal problems.

Theme 2: Professional identity.

A total of 93 quotes described professional identity. It was viewed from one external perspective (negotiation) and four internal perspectives that described self-development. ‘Presenting oneself,’ the external perspective, was the most common professional identity sub-theme, cited in 51% of these quotes. It was most often described in the context of students’ professional portfolio materials (i.e., personal statements, cover letters, curriculum vitae, resumes, and application essays that ask students to describe their identity, experience, and goals). Activities in this area addressed implicit cultural information and technical skills to self-represent as an underrepresented professional in academia and research, particularly in the contexts that control access to advanced degrees. For example, they included discussion and practice of negotiation, conference networking, graduate and professional interviews, and technical information and strategic disclosure skills for the written documents required to access graduate and professional programs, secure funds to finish college, and apply to research jobs.

Internal perspectives (i.e., ‘What could I look like as a professional or researcher?’) described how scholars viewed themselves, identified their goals, and understood the training they wanted from the educational system. ‘Modeling who does science’ was represented in 49% of quotes. It described helping students understand the diversity of people involved in science, what researchers care about, and what limitations in science and research careers may exist, even for successful scientists. “Reflection time” was represented in 39% of professional identity quotes. Sites reported that helping students understand that identity development takes years of reflection was important. Enrichment provided this time to help students integrate their developing research skills (e.g., ethical considerations, data management), with their goals and values into self-directed professional identities. ‘Recognizing gaps in professional preparation’ was described in 35% of this theme’s quotes. It referred to supporting students’ ability to identify gaps in their knowledge and skills as opportunities for growth and future training (rather than reinforcing imposter syndrome often experienced by URMs (Bravata et al., 2020). Emphasizing strategies for persistence after failure, which occurs frequently in science, was identified as critical. ‘Sense of Belonging’, described in 24% of quotes, referred to a focus on showcasing similarities among program students across biomedical fields. Sites reported that sharing common experiences helped students understand norms and expectations. For example, asking questions when having limited information or concerns was emphasized as an expected norm in biomedical research training and careers, rather than inappropriate behavior. Students helped each other understand these expectations.

Theme 3: Research skills.

General research skills were cited in 43% of the 91 code applications. Specific research skills were categorized into six areas: communication skills, skills for navigating research careers, critical thinking, research methodology, information literacy, and research ethics. ‘Communication skills’ were cited in 44% of code applications and referred to improving scientific writing, both research communication (e.g., scientific presentations, informal discussions about research) and professional documentation (e.g., personal statements, curriculum vitae, business cards for research event networking). ‘Skills to navigate research’ was cited in 32% of quotes and described skills that students could use to navigate research careers, such as how to find mentors or additional training in an area. ‘Critical thinking’ was cited in 16% of quotes and referred to helping scholars’ interpret research literature and findings, identify limitations, and consider next steps in the experimental process. ‘Research methodology’ skills were also cited in 16% of quotes, and referred to access to learning quantitative and qualitative methodologies, most often statistical and data management skills to support scholars in their research. ‘Information literacy’ skills (15%) described teaching students how to conduct literature searches, evaluate the credibility of information sources, and how to search for information to inform their career development. For example, online searches of graduate programs, training opportunities (e.g., postbaccalaureate programs, summer research programs), and funding opportunities (e.g., scholarships, grants) all were considered ‘information literacy’. Finally, ‘research ethics’ (8% of code applications) was cited as an integration of research ethics and principles into scholars’ identities as researchers and biomedical professionals.

Theme 4: Path representation and networking.

A total of 76 code applications described path representation and networking in the context of allowing students to identify scientific professionals and career paths of interest to support students’ autonomy in making their decisions. ‘Available paths’ was most commonly cited (64% of code applications) and referred to supporting student networking to meet professionals who could help students identify or solidify their possible career paths. ‘Social connections’ was equally mentioned (62% of code applications) and referred to social interactions independent of a desired path, which helped students strengthen communication skills inside and outside of their desired field. Professional networking was described as being intimidating for some students, particularly those who may be introverted or have social anxiety. In such cases, Enrichment sites offered strategies and tips (such as example email prompts or small group work) that helped students to overcome networking anxiety. Code applications for ‘path representation’ were categorized into internal versus external components for understanding a path. Approximately 22% of code applications described networking in the internal context of helping scholars “build confidence in pursuing their desired path;” such as conversations with professionals and near-peers who could represent what a desired path could look like, allowing scholars to better visualize their own career in that area. In contrast, 17% of quotes described using networking in the external context, as a way for scholars to understand what portfolio items would be needed in order to pursue that path of interest (e.g. GRE/MCAT, clinical shadowing, personal statement, etc.). Since underrepresented students face systemic oppression in educational systems, both internal and external supports helped students visualize their belonging within a field. Sites reported that networking with diverse professionals aided students’ understanding of how their values could be integrated within that career path, and how their own understanding related to the external steps required to get there.

Theme 5: Professional development.

This theme included both personal growth (beliefs) and accrual of professional skills. Professional development was described in 43 quotes, both generally (44% of code applications) and across specific sub-themes. For example, ‘Specific skills’ (e.g., time management, study skills, financial skills, leadership skills) were referenced in 21% of quotes. These skills also included work-life balance and professional etiquette (such as conference dress codes, use of business cards, etc.). ‘Self-care and emotional intelligence’ were described in 21% of quotes, with emotional intelligence referring to the context of navigating interpersonal relationships in professional environments. As underrepresented students face more barriers and inequities, the personal toll when navigating professional environments may be higher for these students, which is supported by our data showing that emotional intelligence and self-care were often mentioned together. Also described in the context of professional development, ‘Support of self-beliefs’ referred to raising students’ self-awareness and belief that they could pursue and be successful in biomedical research paths. Imposter syndrome was described in this context, which together with self-beliefs comprised 21% of professional development code applications. Finally, ‘Self-advocacy’ was described in 16% of quotes and referred to negotiation skills and how to navigate conflict on behalf of oneself. Self-advocacy may serve as one output of scholars’ gaining emotional intelligence, though these sub-themes were only mentioned together once when approaches for self-care were described in the context of “life skills that will help the student cope with the demanding expectations of their educational and career paths, given the additional responsibilities and pressures specific to their underrepresented status.”

Theme 6: Research relevance.

Societal and personal impacts were described across 28 code applications describing research relevance. Societal impacts (64% of ‘research relevance’ code applications) described helping scholars expand their perspectives to see populations in need of additional research, including those facing economic inequities and health disparities. Sites also described how to address real problems facing their communities and the world today. Social responsibility was particularly impactful for one university that had students complete outreach activities to better understand health and economic disparities faced by homeless individuals living in their region, as they cited the work supported students’ visualization of the diverse, multidisciplinary factors involved in research. Approximately 50% of ‘research relevance’ code applications described helping students see the personal relevance of research, including what research could look like for them as scientists.

Theme 7: Research Exposure.

Sites described exposure to “types of research” and “research culture” across 29 quotes. ‘Types of research’ was described most frequently in these quotes (79%). It referred to exposing students to the different types of research that they could potentially do. Sites provided research exposure through visiting scientist presentations, journal clubs, and general discussions. Of note, access to scientists/speakers was more challenging for distant community colleges. Exposure to research culture (45% of research exposure quotes) described providing scholars with exposure to research environments and addressed questions such as who does the work, what laboratories look like, and the range of skills and roles typically needed. Approximately 24% of quotes applied both type and culture codes when describing ‘research exposure’. When planning Enrichment, students’ research exposure and desired career paths were often informed together, which influenced instructors’ sense of what additional activities and exposures were needed.

Theme 8: Institutional infrastructure.

Referenced in 21 quotes, sites described this code from two perspectives. ‘Improving student/faculty/institutional relationships’ was described in more than 2/3 of quotes (67%) and highlighted that Enrichment supported relationship-building between students and faculty, and “often fulfill[ed] a mentorship role.” Sites also reported that their program’s students helped to build connections across their institution’s scientific disciplines or research programs, which was visible during Enrichment’s peer sharing sessions. Sites offered that the program’s students served as a ‘scientific pollinator’ across scientific areas in ways that enhanced networking and communication at their site. This sharing is important for institutions, as 19% of code applications cited that the credibility of their institution improves when program students are successful, highlighting the mutual benefits for both students and institutions. ‘Facilitating access to institutional supports’ was described in 62% of institutional infrastructure quotes. It referenced sites’ use of Enrichment to understand early indicators of student needs. Sites reported that Enrichment helped institutions change (or plan for change) responsively and in an evidence-based manner. Some sites were working to institutionalize instructional components to make Enrichment available to more students at their institution.

Theme 9: Sociocultural dynamics.

Sites described a focus on the sociocultural dynamics within academia and research across 40 quotes. ‘Navigating academic/research cultures and teaching hidden curriculum’ was most prevalent (70% of code applications) and described processes for underrepresented students to learn about academic, research, and workplace cultures, as well as their often unspoken norms, values, and expectations. ‘Inequities faced by underrepresented students’ comprised 35% of code applications and included ‘Systemic barriers and disparities’ (22.5%) that people of color and other marginalized groups disproportionately face, as well as strategies for ‘Responding to Bias/Microaggressions’ (12.5%) in successful ways that support students’ personal and professional selves. ‘Diverse representation’ was described in 23% of code applications and referred to the diversity of research teams (or lack thereof) and the benefits of including diverse experiences and cultural perspectives for strengthening teams. Finally, “Supporting communication across diverse perspectives” was cited least frequently (15%). It referred to understanding sociocultural perspectives within a scientific discipline or across academic levels by talking with diverse professionals and peers. The quote below offers an overarching reason why it matters to help underrepresented students navigate these sociocultural dynamics and academic cultures.

“Professional development skills are part of a hidden curriculum that is not taught traditionally in the classroom and is essential for career development. To provide skills in scientific and professional development, sophomore enrichment is divided into 3 areas in the sophomore year each with its own learning outcomes. Area 1 focuses on student development and self-efficacy as biomedical researchers. Here students are taught skills in overcoming personal barriers such as self-care, imposter syndrome and microaggressions as well as systematic barriers that impedes success for underrepresented students. They work on deconstructing the idea of what a scientist should look like and there is an emphasis on the importance of diverse people in STEM and the values they bring into research that has a global impact on human health and the community. We work with students to understand the importance of mentors, advisors, and advocates in the development of their career and professional identity. Students also incorporate their own personal life experiences into constructing their personal identity and we work with them to build resilience through skills aimed at improving their self-efficacy, confidence, and strong emotional intelligence. Lastly, we provide a space to discuss career pathways to support students as they align their personal goals with their development in biomedical research. Area 2 focuses on the development of their professional identity through skills that are part of a hidden curriculum. These skills include effective communication, time management, networking, and negotiation strategies. Area 3 focuses on skills that can be applied in a research learning community such as presentation skills, database management, literature reading and writing. In addition, there is a strong emphasis on development of interpersonal skills that are necessary for working in a team lead research learning community. These interpersonal skills include working effectively in teams, building group cohesiveness, and handling conflict within group dynamics. Over the year enrichment and peer mentors were able to help provide these essential skills for student success and development as professionals in biomedical research. ” –Primay University

Theme 10: Supports.

A total of 32 quotes described helping students navigate supports available to them. ‘Checking in for support and emergent student needs’ was a major subtheme, representing 74% of code attributions. It described Enrichment as a place for the program to check in with students, understand needs, offer support, and refer students to resources. Several sites described Enrichment as a way for their institutions to learn about emergent student needs so they could proactively develop structures to help current and/or future students. “Accessing supports and normalizing the process’ referred to helping students learn how to identify, access, and navigate supports (35% of code applications), such as financial aid, student health, and disability accommodations. Approximately 15% of all “Support” quotes described using Enrichment to help scholars learn coping skills and stress management techniques. Likewise, 15% of these quotes described an expressed intention by sites to normalize the idea of people in academia asking for help and needing support at times, with all sites who described normalization also providing guidance on how to access support resources.

Theme 11: Knowledge and self-efficacy.

The development of scholars’ knowledge and self-efficacy was described by 6 of the 9 sites in only 10 quotes over two equally referenced subthemes. ‘Knowledge of paths or resources’ (70% of quotes) referred to the accrual of knowledge that supported scholars’ career path decisions, content knowledge in specific paths, or procedural knowledge about how to pursue paths. The accrual of knowledge was viewed by one instructor in the context of students’ role shift into developing professionals. ‘Building competence of program students’ (60% of quotes) was described in the context of self-efficacy. It referred to helping students believe that they could pursue their desired goals (e.g., academics, research, etc).

Theme 12: Self-care.

Though described in only 11 quotes, self-care was a growing emphasis for sites over the 18-month study period and was identified by 5 of 9 sites as a critical need for supporting student professional development in research environments. Sites described a need for instructors to model self-care and work-life balance, as it helped inform students how to manage self-care in academia and whether a research career could work for them. Sites also described that helping students incorporate self-care into their routines was important for preventing burnout while supporting mental health. Finally, self-care included practices that students could use themselves, as well as teaching them peer-to-peer, and recognizing when additional support may be needed.

Appendix 6. Thematic analysis of learning objectives, structure and format, barriers, implementation solutions, and key changes to Enrichment across sites.

Learning Objectives

Bloom’s taxonomy describes six levels of learning objectives that range from lowest (level 1) to highest (level 6) cognitive order: remember, understand, apply, analyze, evaluate, and create (Adams, 2015). This taxonomy was used to sub-code 115 quotes describing diverse learning objectives to find common elements. In learning objectives, sites described four knowledge types described by Adams (2015) including “factual (terminology and discrete facts); conceptual (categories, theories, principles, and models); procedural (knowledge of a technique, process, or methodology); and metacognitive (including self-assessment ability and knowledge of various learning skills and techniques).”

Structure and format

Structure and format describes logistics and common practices for running an Enrichment program, including instruction; frequency and duration; audience and attendance; approach to content; activities; and instructional supports and online resources.

Instruction.

The facilitation of Enrichment varied across sites, with “instructor” comprising both faculty and academic professional staff. Most common was a faculty instructor ‘lead’ working with a team (5 of 9 sites cited this approach; 2 of 3 universities, 2 of 4 urban community colleges, and 1 of 2 distant community colleges). “Team” referred to any combination of staff, other faculty, undergraduate peer mentors, and graduate students involved with Enrichment activities; typically 1-2 people, with one section at the primary university having 6 peer mentors. Teams without a defined “lead” were used at two sites (1 distant university had 4 team members and 1 urban community college had 2 team members). A single instructor led Enrichment at both distant community colleges. Consistency in instructors was a challenge at some sites and changed frequently as a result, particularly in cases where Enrichment depended on faculty with other teaching and/or research responsibilities. Financially supporting a dedicated person’s effort (Full Time Equivalent, FTE, or portion of a team’s FTE) enabled sites to build consistency in instruction, and identify and implement best practices based on participant feedback. Consistency also helped to build trust between students and instructors. Students built long-term relationships with stable instructors that often served as the basis for obtaining letters of recommendation. One partner university site was able to leverage funding from another research grant to support instructor FTE to implement Enrichment for students across both research training programs.

Frequency and duration.

Two universities held 90 minute sessions. Three of four urban community colleges and one distant community college offered 60 minute weekly sessions. Duration at the remaining sites varied based on the event. Instructors who used 90 minute sessions identified them as highly valuable for workshops (e.g., informational, discussion, or practice time). Two universities with 90 minute sessions offered them weekly with attendance required as part of program participation. One urban community college held enrichment events once or twice per quarter, and one partner university held events once or twice per semester. Two different structural challenges appeared:

1. Large required programs. The primary university served more than 100 participants per term. It established three separate cohorts, and tied frequency to student level (i.e., sophomore, junior, senior), requiring weekly sessions (10/quarter) in the first year of training (sophomore) and every other week (5/quarter) for students in the last two years of training since students also had required research hours to complete in laboratories. Supplemental options were offered for universal access and contact across levels.

2. Small isolated programs. Enrichment at both distant community colleges was variable in frequency and varied in the types of on-campus sessions and off-campus activities included. Programs had limited access to professionals beyond the core instructor/team. Some smaller sites with limited numbers of students and faculty/staff leveraged resources by integrating aspects of Gateway course instruction, Enrichment, and Mentoring. Student issues and programmatic findings were consistent across sites, regardless of whether Enrichment activities were combined with other program elements or offered as stand-alone options.

Audience and attendance.

All sites identified their Enrichment as interprofessional. Biomedical research training programs serve majors in natural science, social science, and technical STEM fields, who aspire to both academic and professional degree paths. Sites reported interprofessional instruction facilitated students’ self-discovery while making it easier to implement Enrichment at their institutions. The interprofessional setting provided a critical mass of student peers, while keeping implementation and staffing costs low by serving multiple majors and degree paths. Both university and community college partner sites reported a minimum number of students attending Enrichment was needed for sessions to feel worthwhile to students. Student attendance was particularly important for institutions with smaller or fewer cohorts, such as community colleges where attendance lasts for two years; these institutions never had more than two cohorts at once. Partnering with other STEM research training programs can improve student attendance and provide a critical mass of students.

Approach to content.

Overall, sites described Enrichment as a “flexible container” that could offer a range of programmatic content that was responsive to emerging issues and needs. Enrichment content included knowledge transfer through didactic sessions, skill development, portfolio creation, communication about the research training program logistics (i.e., announcements, expectations, deadlines, resources), and served as a dedicated time/place for students to talk with each other and meet professionals. On rare occasions, it offered a group setting for trauma support (e.g., student death, natural disaster). Most sites highlighted that the content of their Enrichment was student-driven or student-informed. Students co-created what they wanted to learn (informing both choice and timing of topics) and identified existing gaps in their training as a group. Students used Enrichment to raise emergent issues for themselves and their peers, such as barriers faced in academia and research. As instructional leads were situated to document issues and support collective change, Enrichment enabled students to get their needs met as individuals while informing programmatic and/or institutional support gaps. At the primary site, content became scaffolded to programmatic year over the 18-month study period, enabling topic areas to differ for sophomores, juniors, and seniors while still being responsive to the needs of students who rarely have an academic and career “straight track.” Cross-cohort sessions supported students’ ability to form groups based on other factors than program year (i.e. identity, degree/career interest), and share skills, resources, and path options.

Format and activities.

Formats varied but were highly interactive. Key activities focused around two themes: ‘dedicated time for personal reflection/student sharing’ (32% of’format’ code applications) and ‘specific scientific activities’ (28% of’format’ code applications). The latter, scientific activities, included scientist talks, panels and mixers, science pubs, journal clubs, ethics discussions, and “lab boot camps” that gave opportunities to practice basic lab techniques. Enrichment activities often integrated both themes. For example, one partner university offered a science non-fiction book club and student-led movie nights; another used a community service outreach project with a local homeless coalition to understand health disparities in practice; other institutions gave portfolio writing workshops (i.e., personal statement, curriculum vitae, resume) and tutor training. Sites reported that institutional collaborations across their scientific training programs, and leveraging existing resources, were effective to create Enrichment activities. Partnering with other institutional programs (grant-funded or institutionally funded) enabled sites to extend offerings to students (e.g., tutoring, professional development supports, and writing support). For example, an urban community college and a distant community college both offered aspects of program Enrichment through attendance at events outside the program: research symposia/conferences, and an institution’s existing ‘Scholar Success” workshop series. Two distant sites, a university and a community college, collaborated with each other to create a scientist panel [on one island] that was video-conferenced for students at another island. An example of how these formats and activities evolved is described below:

“The structure of enrichment during our sessions have also evolved from passive teaching methods which include lecture and reading to participatory methods that involved student engagement through role playing, small group discussion, and practice by doing. We curated opportunities for students to practice professional development skills such as active listening, emotional intelligence, and communication skills through EXITO sponsored panels, networking events, and community building through engagement with each other within the cohort and with peer mentors. Overall the cohort community building over these years since 2017 has strengthened and Enrichment attendance also improved dramatically.” -Primaiy university

Instructional supports and online resources.

Online platforms (e.g., Google Docs, online course management platforms) enabled sites to share announcements, materials, resources, and writing/portfolio examples with students. Other sites used email as their primary communication method. To meet access needs, one urban community college used calendar polling software to find group meeting times, while the primary university used meeting booking software to support 1:1 student meetings. Several sites described lack of these instructional supports as barriers to implementing Enrichment (see Barriers below). While online platform resources were created for immediate needs, teams across sites were interested in their potential to share content to support student, alumni, and faculty/staff development across institutions.

Measurement and outcomes

Sites reported measurement outcomes as well as areas where they would feel strong or weak in evaluating learning objectives. Of the 47 quotes, over half (51%) described evaluation approaches for measuring impact of their Enrichment activities, such as event evaluations for formative feedback, and term evaluations for summative feedback. Sites offered suggestions for other evaluation approaches, such as written student reflections that could be analyzed for learning objectives and development of professional identity. ‘Professional development’ was identified as a desired measurement outcome in 32% of quotes. It included the perceived value of supporting students’ professional development, particularly professional development skills such as cultural knowledge of academic and research cultures and their norms, professional writing skills to articulate a professional identity, and implicit curriculum on how to access support systems. ‘Authentic conversations with faculty, staff, visiting scientists, and peers’ was described as an important outcome in 19% of code applications. It referred to Enrichment providing dedicated time and space to have meaningful conversations with others about science, research, and path progression. ‘Student satisfaction and appreciation’ was also referenced in 19% of quotes and described student satisfaction as an important early indicator for gauging the effectiveness of Enrichment for students. Partner sites sent student quotes, which referenced student satisfaction and appreciation for Enrichment. One student from an urban community college described the personal impact of Enrichment:

“The enrichment sessions at [my urban community college] have been very valuable to me! It is like a check in with reality to think about the future and realize the things I need to do now to prepare for that and increase my success rate thanks to the support team I have. Sometimes life gets really busy but meeting every Friday reminds me that the school/life stresses is a part of the journey that I am on as I am working towards my career. Like a light at the end of the tunnel. I believe the honesty and openness of the team is very helpful, being able to have discussions about what to expect, good or bad, helps me not worry as much. I do think that the structure…could be a little bit better. Maybe a little less ethics and a little more personal development, although we are learning/growing through our discussions, maybe more CV writing or negotiation techniques. There is just not much time through this class to learn everything but I think our campus EXITO program should keep in contact with the EXITO Enrichment program at Portland State University to make sure that we are all on the same page and communicating with each other so we don't learn things next year when we transfer that we've already learned this year and so we don't miss anything that they've covered this year.”

Finally, ‘logistics and student-driven approaches matter’ was mentioned least frequently (13%), but was described by sites as being very important for Enrichment and its outcomes. For example, barriers to attendance (e.g., travel to events) or methods to boost the number of students attending (e.g., attendance requirements or other options) were cited as measurement opportunities for evaluation.

Barriers

Across the nine sites describing Enrichment over two time points, a total of 24 quotes described barriers experienced by program students which surfaced through Enrichment. ‘Access to scientists/challenges with travel logistics’ was cited exclusively by community college partners (38% of ‘barriers’ quotes). It referred to students having inadequate access to scientists who could speak with them. At distant sites, Enrichment addressed the barrier by recruiting individuals who were already coming to the island (for symposia, research, vacation, etc.). Urban community college sites used Enrichment to move the students to events in their region where they could meet scientists (i.e., Enrichment visits to science pubs/nights, conferences, special talks, etc.). However, this approach revealed financial and access barriers, since transportation was often needed to attend. ‘Staff time/staff support’ was also referenced in 38% of quotes and described the value of having dedicated staff who could work with students (cited as a barrier when absent). Staff time was critical at the primary site for providing students with writing support for scholarships, graduate/professional applications, and internships, with abstracts and scientific writing cited to a smaller extent. Application anxiety was high, with students typically asking for review and revision support very close to deadlines, requiring dedicated flexible hours from an experienced professional. “Staff support” also referred to comments that mentioned Enrichment faculty/staff wanting a way to share materials among sites, or wanting guidance from trained evaluators on how to evaluate student outcomes of Enrichment activities. ‘Unmet needs of program students’ was referenced in 29% of barriers quotes. Cited as a barrier when absent, students needed professional writing support to secure scholarships, apply to academic and professional next steps, or submit scientific work. ‘Busy student schedules’ was cited as a barrier in 25% of quotes; students are full-time enrolled in biomedical majors, pursuing research, and often must work additional hours at outside jobs or care for family members. Finding consistent times where all students can meet was a common challenge. ‘Funding’ was mentioned as a barrier in 17% of quotes, in reference to lacking dedicated staff time or financial support for students to attend conferences/events to network with professionals as part of Enrichment.

Implementation Solutions

Sites offered solutions for implementing Enrichment across 57 quotes (Table 6). Sites described Enrichment as a way to identify and understand student needs at their institutions, often helping students to navigate and advocate for institutional support (e.g., disability and childcare resources, financial aid navigators, culturally competent mental health, financial wellness services, etc.). These solutions offer guidance for institutions wanting to implement programs like Enrichment with their students.

Significant Changes over Study Period

Baseline and follow-up data were coded to understand significant changes that sites made to Enrichment over the 18-month study period. A total of 44 quotes described these changes, with ‘Moving beyond research-specific activities to include broader relevant skills’ being most prominent. Approximately 57% of quotes described changes to incorporate more professional development skills, and greater emphasis on teaching implicit curriculum around social norms and dynamics of academic and research cultures. Many sites identified that it was valuable to use student input to identify instructional needs and effective curricula. Another major change was ‘Interprofessional instructional teams,’ described in 27% of quotes. It referred to needing to replace faculty instructors who had other duties, and developing teaching teams that leveraged instructors across programs and faculty/staff roles, including peer mentors on the teaching team, and leveraging program alumni and graduate students as part of the diverse team. ‘Formats and flexible options for students’ was cited as an important change in one quarter of quotes (25%), referring to changes in Enrichment formats (e.g., activities, structures, etc.). The most dramatic change in format occured at the primary site, where students were given more opportunities to attend other cohorts’ enrichment sessions, which was reviewed positively by scholars since it allowed them to meet their needs and goals. Other sites also adjusted formats to some extent, responding to meet student needs. ‘Increased use of data and evaluation’ to understand student needs occurred over the study period (20%). Sites often used informal evaluation to understand student satisfaction with events, access barriers to instructional supports, or how the term was perceived by students. The primary site’s evaluation mirrored course evaluation questions at the university, but also included questions to understand student needs for the coming year. Finally, an ‘Increased emphasis on self-care to support research and professional careers’ was represented in 11% of quotes and described additional focus around self-care and mental health, including normalizing those needs, skills to support work-life balance, and how “to identify resources to support them when they need help being successful in research.” Student needs and instructional goals can be interwoven to support instructional arcs across program levels (Table 6).

Appendix 7. Additional lessons learned for implementing Enrichment with research trainees.

Student Development

1. Students balancing full-time college, intense life demands, and research training have extreme pressure on their schedules. Requiring their presence brings with it the ethical obligation to make the activity worth their time.

2. What is worth their time is their decision, so instructors need ways to learn and respond to their priorities. All sites created strong feedback loops within Enrichment, using methods from informal conversation to polls and surveys to giving students the authority to create structures they want.

3. While cohort effects can produce strong impacts on student development in the scientific literature, informal and near-peer mentorship are also documented as high-value. 24% of code applications related to cohort and near-peer bonding were “proxy for support.” Program participants were coming together through Enrichment, across cohorts, to find solutions for issues that institutions and programs had failed to effectively address.

Program and Institutional Development

1. When program participation is tied to access to education (funding or tuition remission), requiring Enrichment as part of program participation also requires the program create universal access to Enrichment. Doing so requires dedicated work hours.

2. Students underserved and underrepresented in academia and research must be unusually competent to access academia and research. With appropriate support, they often work at a higher level than is typical for undergraduates.

3. Appropriate support requires connecting across the faculty-staff divide. Academic professional staff or faculty-staff teams were key to Enrichment at multiple sites. Academic systems tend to categorize students in an either-or binary: high-needs (served by non-academic staff) or high-performing (served by academic faculty). A healthy future for biomedical research in the United States depends on engaging students who are both. Serving them means creating systems that bring together faculty and support staff. Asking individual underrepresented faculty mentors and staff advisors to substitute in isolation for connected networks is another aspect of “proxy for support.” The need is common; it can be addressed if it is not seen as an enormous number of isolated individual problems.

4. Positions for program alumni support effective Enrichment and the careers of program alumni. Relevant biomedical interests include science education, mentorship, social research, and assessment. Research training program graduates are trained research professionals, and best placed to understand program needs and resources at their site. Professionals should be paid.

5. Today’s undergraduate is tomorrow’s graduate student or junior professional. Models where support is imagined as an institutional cost without benefit, tied to the idea that underrepresentation makes people more needy or less skillful, do not function for professionals who must be recognized as skillful contributors to advance. By making its Scholars supported partners in developing new structures and curricula to meet their own needs, to improve academic and research systems that were not originally built to serve them, Enrichment offers a model for self-defined professional development as institutional development.

Access

All sites showed interest in remote resources and connections to improve access. Several barriers were identified:

1. Where instruction is unstable and institutional course shells expire, materials may not be easily available across years at the same institution.

2. Online learning management systems (e.g., Sakai, Blackboard, Canvas) are easily available at most institutions, but hosting resources can block cross-institutional and alumni access. One instructor at the host site developed a Google Docs folder for a collection of curricular resources and de-identified professional portfolio materials, which was shared with partner sites.

3. Some instructors felt that recording Enrichment would compromise its basic function as a low-risk “safe space” for conversation and practice.

4. Interactive activity is a key feature of Enrichment. Aspects of live interaction are possible online through video conferencing. Web-based activities may support distant sites that report less access to research-related materials and professionals. However, video conferencing was cited only once, by a distant community college.