Broadening and Diversifying the Behavioral and Biomedical Research Workforce through Early Access to an Undergraduate Research Training Program

Abstract

There is a broad need to support the early educational trajectories of underrepresented students pursuing behavioral and biomedical research, particularly at large, comprehensive institutions. The Building Infrastructure Leading to Diversity (BUILD) initiative at California State University Long Beach (CSULB) created an Associates Program designed to provide undergraduates with early exposure to research and foster a sense of belonging and interest in a research career during their sophomore year. Our Associates Program had high retention rates (> 90%) and served as a pathway to other research opportunities on campus, with over half of the students entering an intensive, upper-division research training program at CSULB upon completion. Analysis of evaluation data gathered at multiple points throughout the training program provided preliminary evidence that our early intervention program resulted in student trainees’ growth in a number of key areas, including their sense of belong to the BUILD Program, interests in science and research, and understanding of what research entails and of the skills necessary for conducting research (e.g., scientific writing, oral presentation, data analysis). More importantly, comparisons of the students who continued on to an upper-division research training program to those who did not continue revealed that students who continued reported generally higher levels of science/research interests regardless of the time points of the survey, and a greater increase in their perception of gains made in some areas of research during the second half of the training program. Lastly, our results also showed that the Associates Program is similarly effective for trainees across behavioral and biomedical disciplines, underrepresented minority status, and gender. Based on these findings, we conclude that an early intervention program for undergraduate students results in development of research skills for students exploring research and serves as an effective pipeline for diverse students into more intensive upper-division training programs.

Introduction

Broadening and Diversifying the Behavioral and Biomedical Research Workforce through Early Access to an Undergraduate Research Training Program.

There is a longstanding need to support the early educational trajectories of underrepresented students pursuing behavioral and biomedical research. Indeed, the COVID-19 global pandemic has demonstrated how critical diversification of the research workforce is to understanding health disparities and addressing public health challenges. An increase in representation of historically excluded communities brings a much-needed perspective to health-related research fields (Hurtado et al., 2008). For example, given their lived experiences, students of color can bring a social justice approach to their research which can further advance critical perspectives in the study of behavioral and biomedical sciences (Garibay, 2015; McGee & Bentley, 2017). Unfortunately, there continues to be an underrepresentation of students of color, LGBTQIA+ students, and women as well as first-generation college students in the sciences with an overrepresentation of White, cisgender men (McGee, 2021; Trapani & Hale, 2019).

Access to undergraduate research is a widely recognized intervention providing multifaceted support for underrepresented students pursuing STEM fields across critical transitions, such as high school to college, college to graduate school, and graduate school to profession (Hurtado et al., 2008; Kitchen et al., 2018; Murray et al., 2016). Moreover, early exposure to research has been shown to benefit STEM and non-STEM students alike (Stanford et al., 2017). There are examples of research or bridge programs that offer support to underrepresented students during these critical transitions into college, including the American Chemical Society’s (ACS) Summer Experiences for the Economically Disadvantaged (SEED), the National Science Foundation (NSF) Bridge Programs, and the Leadership First Year Research (FYRE) Programs. Scholars who have explored the outcomes and effectiveness of these transitional programs highlight how early exposure to authentic research experiences helps foster growth in career aspirations, knowledge and technical skillsets that are necessary for pursuing and attaining a STEM degree (Hathaway et al., 2012; Hernandez, 2016; Kezar & Holcombe, 2017; Kitchen et al., 2018).

Early structured undergraduate research has also been found to help underrepresented students with equally important personal and psychosocial development. Early exposure to research can cultivate students’ science self-efficacy and science identity, as well as promote a sense of belonging and community to research and science (Hurtado et al., 2008; Jones et al., 2010; Maton et al., 2012; Ovink & Veazey, 2011). Despite these benefits, early access to structured and/or funded research opportunities is still limited on most campuses for lower division students (Hurtado et al., 2008). Being left with limited options, students may face the challenge of having to initiate and develop rapport with faculty (i.e., overcome the fear of asking a professor to join their research lab) while taking large introductory courses, which often makes it hard for students to get to know and identify potential mentors (Buffalari et al., 2018). From an equity perspective, this is concerning, considering that researchers have well documented the isolating climate that underrepresented students of color already feel in their STEM classrooms (Museus et al., 2011).

Campus programs that support matching student participants with a faculty mentor can help them overcome these barriers. In addition to providing essential experience in learning experimental design and techniques, mentored research activities in the lab provide invaluable opportunities for faculty guidance and support that can shape the educational trajectories for students, especially those who are new to the university and the research endeavor (Hernandez et al., 2018). Faculty mentors play a particularly important role for early research exposure as their pedagogical approaches and individual lab cultures set the tone for new students in such a way that could either be welcoming or unintentionally turn students away from research and even the sciences. Thus, providing ongoing development opportunities for improving research mentoring, such as culturally responsive and inclusive mentoring, is also a vital component for successful early research experiences for underrepresented students (Haeger & Fresquez, 2016; Young et al., in press).

Practices that focus on student assets such as peer support and community-based learning environments can help to create a sense of belonging among students that can further promote retention in their respective majors. For example, one of the hallmarks of successful training programs is the use of cohort-models that bring together students to create a sense of community and camaraderie that can help combat the racial stigma students experience as part of being in underrepresented student-targeted science programs (Hurtado et al., 2008). Multi-tier mentoring from near-peer mentors (undergraduate or graduate), program staff and faculty mentors provides an additional layer of support to underrepresented students (Abeywardana et al., 2020). This type of wrap-around support was found to be important for alumni in a biology training program, many of whom often credited program staff, directors and peers for providing personalized advice on ways to navigate their academic program and a community of like-minded peers (Ovink & Veazey, 2011). Considering then, that undergraduates in their first and second years are getting acclimated to the campus environment, building a multi-layered, supportive scientific and research community can help ensure that they succeed academically and develop confidence in continuing to pursue research as a career.

Last, but not least, the importance of having adequate financial resources to dedicate the necessary time for an immersive research experience cannot be overemphasized. Studies with undergraduate research programs that provide financial support have noted that this support plays an important role in alleviating students’ stressors, so they are able to stay engaged in program activities (Hurtado et al., 2008; Hurtado et al., 2007; Maton et al., 2016). This is particularly true for underrepresented minority (URM) students at large, comprehensive Minority-Serving Institutions, many of whom have been shown to have significant financial and basic needs (Crutchfield & Maguire, 2019). These multi-faceted supports highlight how undergraduate research programs can be particularly beneficial for URM students early in their academic career, as they adjust to both a new college environment and their rigorous major-related courses.

Rationale and Background for Present Study.

This paper describes the outcomes of a National Institutes of Health (NIH) funded training opportunity aimed at exposing second-year students underrepresented in STEM to research. One of the goals of this initiative was to determine how effective early research exposure is for strengthening the pipeline to the more intensive upper division research training programs. The present study took place at California State University, Long Beach (CSULB), a large, public, comprehensive, Hispanic-Serving Institution (HSI) and Asian American and Native American Pacific Islander-Serving Institution (AANAPISI). With a large pool of talented underrepresented students, this institution was well situated to address the lack of diversity and inclusion of historically underrepresented (excluded) students in health-related research fields. CSULB has a long history of student training awards funded by federal and state governments (e.g., NIMH Career Opportunities in Research, NIH Maximizing Access to Research Careers Undergraduate Student Training in Academic Research [MARC U*STAR], NASA University Research Center; HSI STEM), industry, and private organizations. However, most were aimed at upper-division level students. The majority of these programs also resided in the College of Natural Sciences and Mathematics where the undergraduate student research culture was already well established at the time the BUILD Program began. A key goal of the CSULB BUILD Program was to go beyond this focus on traditional biomedical disciplines by broadening the pipeline of underrepresented students pursuing health-related research across behavioral and biomedical science disciplines (Urizar et al., 2017). Accordingly, this allowed for students from the Colleges of Health and Human Services and Liberal Arts to be eligible to participate along with biomedical students from the Colleges of Engineering and Natural Sciences and Mathematics.

The CSULB BUILD Student Training Program consists of a one-year, lower division Associates Program and a two-year, upper division Scholars Program (Urizar et al., 2017; see Figure 1). The Associates Program was established with the goal of giving rising sophomores, who were interested in research but not certain about their career goals yet, a less intensive, exploratory introductory experience. This approach is particularly important for members of historically underrepresented backgrounds who may have had less exposure to career options in health-related research, compared with applied health careers such as medicine, nursing or physical therapy. The goal of the Scholars Program was then to provide the more intensive research training that would help strengthen their path toward a doctoral program and ultimately a research career in the biomedical or behavioral sciences (this program will be discussed in a subsequent scholarly contribution, see Vu et al., submitted).

Figure 1.

Undergraduate research training pipeline for BUILD Associates

Because the main goal of the Associates Program was to help students make an informed decision about whether to pursue the more intensive research training opportunities at the upper-division level, the Associates trainees were encouraged in their second semester to apply to continue in one of our competitive, NIH-funded, two-year, upper-division research program including BUILD Scholars, the Research Initiative for Scientific Enhancement (RISE) and MARC U*STAR programs. While each of these programs had a set of unique requirements, they shared goals of increasing representation of students pursuing doctoral degrees in health-related disciplines. To maximize coordination and participation in these programs, we utilized a joint application process where students could apply to one, two or all three of the programs at once with a single application. The three upper-division programs also conducted the application reviews and trainee selection jointly to streamline the process for the student applicants and the programs.

Methods

Programmatic Components.

The CSULB BUILD Associates Program was a one-year program designed to provide culturally responsive research training to lower division students in behavioral and biomedical science disciplines. Informed by the literature on undergraduate research training, the Associates Program was built on best practices from past and current undergraduate research training programs at CSULB while connecting psychosocial assets to cultivate research persistence in URM students. The training goals of the Associates Program were for students to: (1) acquire basic research methods and statistical skills via required coursework and mentored research experience, (2) learn about current directions and cutting-edge behavioral and biomedical research with an emphasis on identifying the underlying causes of health disparities, (3) gain faculty-mentored, hands-on research experience, (4) learn to disseminate research findings, and (5) strengthen science interests and sense of belonging to the undergraduate research community that would lead to continuation on to upper division level research experiences. A comprehensive training curriculum was developed and implemented to achieve these goals, described below and in Figure 2. The training program curriculum was implemented by faculty training directors and graduate mentors from behavioral and biomedical disciplines.

Figure 2.

BUILD Associates Program training curriculum

Individual Development Plan.

Each Associate trainee worked with their faculty research mentor, training director and graduate mentor to develop an individual development plan. Students were guided through the creation of their individual development plan by considering their academic and personal goals, priorities, and needs and resources as they developed their immediate, short-term, intermediate, and longer-term strategies. They updated their individual development plans each semester, which gave them an opportunity to reflect on their progress and revise their goals and plans as needed.

Mentored Research Training.

Every Associate trainee was assigned to a BUILD faculty mentor at the start of the program. To help ensure quality mentoring, BUILD mentors were required to participate in a year-long mentor training program (see Young & Stormes, 2019) within a year of taking on a BUILD trainee to continue as a mentor in the program. This year-long program featured training in the use of inclusive, culturally relevant practices to work with diverse learners. BUILD Associates were expected to develop a mentor-mentee contract during the summer and engage in faculty mentor-directed research activities between 10 to 15 hours a week during the academic year. Being new to the research process, though, Associates often started with a literature review and/or shadowing research being performed by other students in the laboratory with the goal that they would eventually be able to take part more directly in a research project of their own later in the year. The specific research experience was at the discretion of their faculty mentor and varied by discipline and mentor. This allowed for faculty to also assess trainees’ specific needs, ‘meet them where they are’ and create appropriate research tasks/experiences for them.

Research-infused Curriculum.

The program also sought to cultivate students’ science and research interests by helping them increase knowledge and skills in research methods and scientific writing through a required curriculum of research courses (see Taing et al., 2022). Associates were required to take one such course during the year. Specifically, they were encouraged to take the Introduction to Research Methods course, with separate sections offered for the behavioral (e.g., Psychology, Health Sciences) and biomedical (e.g., Chemistry, Biomedical Engineering) disciplines. They also had the option of taking an Interdisciplinary Approaches to Health Disparities course that examined the disparities in health status and access to care across various communities defined by race, gender, socioeconomic status, and geography (e.g., urban vs. rural) and introduced students to unique and collaborative approaches to understanding and eliminating disparities across various behavioral and biomedical disciplines.

Learning Community Seminar.

Associates participated in a faculty-led learning community that started with an intensive, two-week summer program (Preparing for Research Excellence Program, PREP) and continued as a weekly one-hour course during the academic year. The goal of PREP was to provide an opportunity for students to connect with members in their cohort and program faculty and staff (i.e., establish a sense of belonging to BUILD) and to boost their interests in a science career. During the academic year, the goal of the learning community was to help Associates gain a better understanding of what everyday research is like and develop a range of research skills such as reading, analyzing, writing, and presenting.

Summer PREP Component.

PREP provided an intensive jumpstart for the program where students met with BUILD training faculty and staff daily from 9 am-3pm for two weeks. PREP activities included guidance in necessary activities for trainees to get started with research via required safety trainings and ethics trainings, sessions on the research process (e.g., library literature searches, proper lab etiquette and how to read a research article), and a basic introduction to different research methodologies. To facilitate integration into their research groups, students were required to meet with their mentors and discuss their joint expectations for their laboratory experience including how lab or research meetings work in their group so they could prepare for what they would experience during the school year. In preparation for their participation in research on top of their academic coursework during the school year, BUILD provided trainings and discussions centered around topics such as time management, stress management, conflict resolution as well as addressed issues such as challenging feelings of imposter syndrome and stereotype threat. PREP culminated in students attending the Summer Research Symposium where they were expected to network and interact with upper division BUILD trainees who were presenting posters on their research.

Learning Community Seminar for Academic Year.

During the academic year, students enrolled in a one-hour (one-unit) academic course that met every Friday morning. Some of the activities continued themes from PREP (e.g., time management, reading scientific articles, and professional etiquette) as well as offering students additional opportunities to explore careers in research. Associates learned about careers in behavioral and biomedical research disciplines via panel discussions with researchers in various fields and case study discussions. Several learning community sessions were also dedicated to fostering professional development and science identity including guided group discussions of “What is science?” and “What is a scientist?” and how students’ cultures and identities fit into these science identities. At least one session per semester was a joint session with the upper division Scholars trainees to foster the development of the overall BUILD community and provide Associates with an even closer level of near peer mentors.

Associates were also required to attend four research colloquia that took place during the learning community with outside speakers from diverse fields in social sciences, natural sciences and engineering. The talks were given by early career researchers (doctoral candidates, postdoctoral fellows, assistant professors) who not only presented their research but also shared their career journeys. The speakers, who often shared identities similar to the Associates, were asked to share the highs and lows of their journeys and how they drew from their cultural assets and backgrounds to navigate their academic and research careers in the sciences as persons of color. This helped to model the possibilities for how students could identify and draw on their own cultural assets to persist in a behavioral or biomedical research career. The speakers would often share how their work, where applicable, could help advance the conditions for URM communities. Afterwards, students were able to have an informal “meet n’ greet” with speakers over coffee where they could ask candid questions of the speakers, especially those in their specific discipline in a comfortable environment.

One of the major goals of Associates was to teach students how to present their research to both the general public and a scientific audience. During the fall semester learning community, students were required to develop a brief “elevator speech” about their research with separate versions suitable for both “friends and family” and a scientist or engineer in their field once they had begun to experience research with their mentor. During the spring semester, BUILD Associates were assigned a research report describing their project and presented their work at the end of spring learning community.

Multi-tier Mentoring.

Associates were guided and supported by several mentors that varied by roles and social distance. As described above, each Associate was assigned to a faculty research mentor from their research discipline of interest. The faculty research mentors are tenured/tenure track faculty at CSULB with an active research lab. Associates had regular one-on-one meetings with their faculty research mentor throughout the year. Associates also were mentored by the two training directors of the Associates Program. The training directors are also research active faculty who led PREP and the learning community seminars during the academic year. The training directors helped with a variety of professional development activities and provided support whenever trainees experienced any challenges with the training activities, academic coursework, and/or personal matters. Training directors also served as a liaison when difficulties arose between the trainees and their faculty research mentors. Graduate mentors’ role included tracking trainees’ progress through monitoring of their weekly activity logs and checking in with their assigned trainee if there were any unusual activities such as a sustained drop or increase in reported research hours. As near peer mentors, they also met with their Associates individually several times throughout the semester and held office hours. As aforementioned, the Associates were also given several opportunities throughout the year to meet with the upper division BUILD Scholars trainees to hear about their experiences in the more intensive training program.

Financial Support.

Associates were given financial support through hourly pay for the research with their faculty research mentors during the academic year. In addition, each trainee was provided with a research supplies fund for up to $1500 for the year, which could be used to purchase materials/supplies needed to run the research project or pay for participant incentives.

Participant Selection.

A joint selection committee that consisted of program directors for the BUILD, RISE and MARC Programs and faculty research mentors was formed. Faculty from these programs were selected because they were all involved in upper-division research training programs for which the Associates Program would be a pipeline. The use of a joint committee streamlined the application process for students interested in an NIH-funded upper-division research training opportunity and reduced the burden on students for having to select which program to apply to or prepare essentially duplicate materials to apply to the different programs. While all three programs offer training in health-related research to undergraduate students, only BUILD and RISE offer training to lower-division students. Thus, the committee selected students for the BUILD and RISE programs based on the students’ preferences and the needs of each program, such as the number of slots available and disciplines supported by each program. For example, only BUILD offered training to students from the College of Health and Human Services.

A holistic evaluation of each applicant, through a written application and interview with the student, was conducted using a rubric that clearly articulated factors under consideration to reduce the impact of implicit bias in our selection. The rubric included traditional metrics such as academic record, robustness of faculty reference, clarity of academic and career goals, and interest in health-related research as described in the student’s statement. In addition, non-traditional metrics were included such as the ability to enhance diversity of perspectives among the BUILD trainees and demonstrated resilience in the face of challenges. The rubric had scores assigned to pre-defined levels of each criterion, and an overall score was computed from the sum of scores from each component. Part of the selection process also required matching of applicants with research mentors if they did not already have one.

Applicants and Participants.

Table 1 shows the distributions of applicants and BUILD Program participants by discipline (behavioral vs. biomedical), URM status (URM vs. non-URM), and gender (male vs. female vs. non-binary) over the 4 recruiting cycles of BUILD I (2015–2019). For this paper, we utilize the established underrepresented minority acronym, URM, to include historically unrepresented minority students in STEM fields, including Latinx, Black, and Native American students. However, we also recognize the emerging critiques that argue that compiling such a diverse group of students into an acronym can further obscure and perpetuate certain racial disparities (Zinshteyn, 2021). To this end, we attempted to disaggregate data wherever possible while maintaining the confidentiality of student participants.

Table 1.

Number of BUILD Applicants and Participants by discipline, URM and Gender

			Academic Discipline				URM Status						Gender
	Applicant and Participant Data	Overall	Behavioral Sciences		Biomedical Sciences		URM		Non-URM		Uknown/Declined		Male		Female		Gender Non-Binary or Declined to State
Associates (n=127)		N	N	%	N	%	N	%	N	%	N	%	N	%	N	%	N	%
2015–2016	Applicants	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
	Participants	39	8	20.5	31	79.5	21	53.9	18	46.2	0	0.0	13	33.3	26	66.7	0	0
2016–2017	Applicants	57	9	15.8	48	84.2	33	57.9	24	42.1	0	0.0	29	50.9	28	49.1	0	0
	Participants	32	6	18.8	26	81.3	17	53.1	15	46.9	0	0.0	15	46.9	17	53.1	0	0
2017–2018	Applicants	63	21	33.3	42	66.7	27	42.9	35	55.6	1	1.6	21	33.3	42	66.7	N/A	N/A
	Participants	26	11	42.3	15	57.7	12	46.2	14	53.9	0	0.0	10	38.5	16	61.5	0	0
2018–2019	Applicants	76	23	30.3	53	69.7	18	23.7	47	61.8	11	14.5	34	44.7	41	53.9	1	1.3
	Participants	30	11	36.7	19	63.3	18	60.0	12	40.0	0	0.0	15	50.0	14	46.7	1	3.3
Total	Total Applicants	196	53	26.9	143	73.1	78	39.8	106	54.1	12	6.1	84	42.8	111	56.6	1	<1.0
	Total Participants	127	36	28.4	91	71.7	68	53.5	59	46.5	0	0.0	53	41.7	73	57.5	1	<1.0

Notes: N/A = data for 2015–2016 are incomplete or not available. As a result, the total number of applicants is higher than that indicated in the table; BUILD application was a joint effort with MARC and RISE starting in 2016–2017.

The number of applications per year increased from 57 to 76 over the four cycles. These applications were dominated by students in biomedical disciplines which resulted in higher representation of students from those colleges in the program (up to 81.3% of trainees in 2016–2017), but the representation in both applicants and participants of the two behavioral colleges increased in the last two years, due in part to intentional outreach strategies targeting those colleges. In terms of the demographic distributions, the percent of URM Associates remained relatively stable over the four cycles, with a slightly higher number of URM students in the last two years. For gender, the number of female participants varied from a high of 67% in 2015–2016 to a low of 47% in 2018–2019, but overall was skewed toward female (57.5% vs. 41.7% male; see Table 1).

To increase transparency regarding the disaggregation of race/ethnicity categories in research, we provide further details of the BUILD URM/non-URM applicant and participant data in Table 2. The URM aggregated number in Table 1 shows slightly higher numbers than those in Table 2 since participants who self-identified as White or Asian as racial group and Hispanic as ethnicity (i.e., they answered ‘yes’ to the Hispanic category) were included in the total URM category of Table 1. Tables 2a and 2b show that the percentage of participants were similar or even higher than the percentage of applicants for those identifying as African American/Black and Hispanic/Latinx. While we had high participation rates of students who identified as White or Asian, we note that over 90% of our entire sample identified with having at least one underrepresented status as a racial/ethnic minority student, female gender, financial aid eligible, or a first-generation college status.

Table 2a.

Number of BUILD Applicants and Participants by Race/Ethnicity

ProgramYear	Applicant and Participant Data	Overall	African American/Black		Asian American		American Indian		White		Native Hawaian Pacific Islander		More than one race		Declined to State, Unknown or Other
Associates (N =127)		N	N	%	N	%	N	%	N	%	N	%	N	%	N	%
2015–2016	Applicants	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
	Participants	39	1	2.6	14	35.9	0	0.0	7	18.0	0	0.0	6	15.4	11	28.2
2016–2017	Applicants	57	4	7.0	16	28.1	2	3.5	6	10.5	0	0.0	8	14.0	21	0.0
	Participants	32	2	6.3	13	40.6	1	3.1	7	21.9	1	3.1	0	0.0	8	25.0
2017–2018	Applicants	63	2	3.2	27	42.9	0	0.0	8	12.7	0	0.0	5	7.9	21	1.6
	Participants	26	2	7.7	15	57.7	0	0.0	2	7.7	0	0.0	2	7.7	5	19.2
2018–2019	Applicants	76	6	7.9	21	27.6	4	5.3	24	31.6	0	0.0	10	13.2	11	14.5
	Participants	30	1	3.3	8	26.7	3	10.0	9	30.0	0	0.0	5	16.7	4	13.3
Total	Total Applicants	196	12	6.1	64	32.5	6	3.1	38	19.3	0	0.0	23	11.7	53	26.9
	Total Participants	127	6	4.7	50	39.4	4	3.2	25	19.7	1	0.8	13	10.2	28	22.1

Notes: N/A = data for 2015–2016 are incomplete or unavailable. Thus, the total number of applicants is higher than that indicated in the table; BUILD application was a joint effort with MARC and RISE starting in 2016–2017. In two cases from the 2016–2017 cohort, students reported their race and ethnicity differently in the application than later identified in other university and program records.

Table 2b.

Number of BUILD Applicants and Participants by Hispanic/Latinx Category

Applicant and Participant Data		Overall	Hispanic/Latinx
Associates (N=127)		N	N	%
2015–2016	Applicants	N/A	N/A	N/A
	Participants	39	15	38.5
2016–2017	Applicants	57	21	36.8
	Participants	32	14	43.8
2017–2018	Applicants	63	20	31.8
	Participants	26	9	34.6
2018–2019	Applicants	76	29	38.2
	Participants	30	15	50.0
Total	Total Applicants	196	70	35.5
	Total Participants	127	53	41.7

Notes: N/A = data for 2015–2016 are incomplete or not available. Thus, the total number of applicants is higher than that indicated in the table; BUILD application was a joint effort with MARC and RISE starting in 2016–2017.

Analytic Sample of Associates.

Despite best attempts to obtain trainee data, we do not have complete data for all trainees. Thus, for all analyses, the sample of Associates is smaller than the total Associate population presented in Section 2.1. The analytical sample also excludes 11 trainees who left the Associates Program before completing the full duration of training. For demographics and outcome variables, our analytic sample included 109 Associates. The majority of Associates majored in a biomedical discipline (68%, n = 74) compared to a behavioral discipline (32%, n = 35). Also, 55% identified as URM (n = 60) and 45% identified as non-URM (n = 49); 58% as female (n = 63) and 42% as male (n = 46); and 46% (n = 50) as first-generation college students. The majority were eligible for financial aid (74%, n = 81) and were non-transfer students (93%, n = 101). The number of Associates trainees in each cohort ranged from 24–35 per year. Overall, these demographic data suggest that we were able to provide early research opportunities to a diverse group of students. For the survey data, the number of participants in our analytic sample was different by analyses because not all students completed all surveys or survey items.

Program and Survey Data Sources.

Program and evaluation data were gathered at various points during the Associates Program as described above and below in Figure 3. Informed consent was obtained from each trainee at the beginning of their participation in BUILD for research dissemination. Program data included trainees’ background information, progress in the program, and outcome data such as whether participants completed the Associates Program and whether they continued to an upper division program. Trainees’ demographic information was gathered through their application forms and university records and academic performance (i.e., cumulative GPA) via transcripts.

Figure 3.

Variables by data source and data collection timeline

Continuation to Upper Division Research Training Program.

Whether an Associates trainee continued to an upper division research training program (i.e., BUILD, MARC or RISE) upon completing the Associates Program was recorded as a dichotomous variable (continued vs. did not continue).

Academic Performance.

The academic performance of trainees was determined with cumulative GPAs at two time points: (a) at the end of the semester prior to entering the Associates Program (baseline GPA) and (b) at the completion of the Associates Program (end GPA). We used cumulative GPAs rather than Science GPAs since our sample consisted of majors outside the traditional sciences.

Retention Rate of Associates Program.

We recorded the number of participants who did not complete the Associates Program as a dichotomous variable (completed vs. did not complete). Exit interviews with the students who did not complete the program were conducted by an external program evaluator and responses were coded into four major categories: Personal reasons, change in career goals, academic challenges, and other/unspecified.

Evaluation data consisted of self-reported, confidential survey data on trainees’ self-reported motivations for joining the Associates Program and growth in their research knowledge and skills. The surveys were administered by an external program evaluator at the beginning and end of PREP and at the end of the fall and spring semesters. Below we describe the measures that were administered for the Associates Program.

Motivations for Joining the Associates Program.

Trainees’ motivation for joining the Associates Program was assessed at the beginning of the Associates Program using a ten-item measure from the Undergraduate Research Student Self-Assessment (URSSA; Weston & Laursen, 2015). Trainees indicated “yes” or “no” to each of the ten possible reasons for doing research, which ranged from wanting to have a good intellectual challenge, exploring interests in science, and getting clarification for their future directions to wanting to develop a stronger research portfolio for resume and letters of recommendation.

Sense of Belonging.

Associates’ sense of belonging to BUILD was measured at the beginning and end of summer PREP (pre/post) using a single item: “I see myself as part of the BUILD community” (Hurtado & Ponjuan, 2005). Associates indicated on a 6-point scale their level of agreement with the statement from 1 indicating “strongly disagree” to 6 indicating “strongly agree”.

Science Interests.

Associates’ general interest in science was assessed at the beginning and end of the summer PREP (pre/post) with 13 items. Five items were drawn from the Specific Interest subscale developed to measure math interests in students (Marsh et al., 2005). References to mathematics or mathematician from that scale were replaced with science or scientist to measure the Associates’ general interest and passion for science. Sample items were “I enjoy working on science problems” and “Science is one of the things that is important to me personally.” Additionally, eight items were developed by our external evaluator for the purpose of program evaluation. Four items tap into interest in and familiarity with a scientist’s career such as “I am interested in pursuing a career as a scientist,” and “I believe a career as a scientist would be enjoyable.” The remaining four items measured knowledge about, and skills related to research. Associates rated their level of agreement on these statements on a 6-point scale from 1 indicating “strongly disagree” to 6 indicating “strongly agree.” A mean rating was computed for each participant.

Research Understanding and Skills.

Five URSSA items (Weston & Laursen, 2015) were used to measure trainees’ understanding about the research process and foundational research skills. During the academic year (i.e., the end of fall and spring semesters), trainees rated on a 6-point scale (1 indicating “strongly disagree” to 6 indicating “strongly agree”) how much they agreed with statements such as: “Understanding what everyday research work is like,” “Writing scientific reports,” “Making oral presentations,” “Using statistics to analyze data,” and “Understanding journal articles.” These skills items were selected because they cover the foundational research skills common across behavioral and biomedical disciplines. We examined the individual ratings of these items.

Data Analyses.

Both descriptive and inferential analyses were performed on specific program data to evaluate the impact of the Associates Program. To assess whether the Associates Program was an effective pipeline to more intensive upper-division training programs, the number of students continuing to an upper-division program was provided, along with percentages and frequency distributions by cohort and upper-division program. Chi-Square analyses were performed to determine if there was a relationship between the frequencies as a function of three group characteristics (i.e., discipline: behavioral or biomedical, URM status: URM or non-URM and gender: male or female). Students’ academic progress while in the Associates Program was also evaluated by examining changes in GPA over time. An analysis of variance (ANOVA) was conducted with time (pre- and post-) as a variable. In addition, to see whether there were group differences in GPA, separate ANOVAs were conducted with the between-subjects factor of group (continuation to upper-division training program, discipline, URM status and gender). Finally, we determined the number of students who dropped out of the Associates Program before completing it (i.e., attrition rate).

For the survey data, the Associates were evaluated at four time points: (a) before PREP (i.e., baseline); (b) after PREP (i.e., end of summer); (c) mid-academic year (i.e., end of fall semester); and (d) end of academic year (i.e., end of Associates Program). Analyses were performed separately for the summer component (PREP) and the academic year. For the summer component (i.e., sense of belonging to BUILD and science interests), separate ANOVAs were performed for each grouping variable (continuation to upper-division training program, discipline, URM status and gender) with time (beginning of PREP vs. end of PREP) as the within-subjects factor. For the academic year, we examined whether there was a difference in growth in their research understanding and research skills for each grouping variable. That is, separate repeated measures ANOVAs with time (mid-academic year vs. end of academic year) for each group (continuation to upper-division training program, discipline, URM status and gender) were performed on each of the five surveyed items described earlier. To ensure that trainees were not identifiable, we excluded trainees from analyses when their group sample size was less than five.

Results

Continuation to an Upper-division NIH-funded Research Program on Campus.

Upon completion of the Associates Program, 77% (n= 84 of 109) of the students applied to an upper division program and over half (62.5%, n= 69 of 109) of all trainees in the sample were accepted into one of our three NIH-funded research programs at CSULB (BUILD Scholars, MARC, and RISE), with 54% (54/109) of the sample continuing in the BUILD Scholars Program (see Table 3 for percentage of students by program year). Of the 84 Associates who applied to an upper-division training program, the majority (77%; n= 65) were accepted (i.e., a total of 19 Associates who applied were not accepted across all four years). Typical reasons for non-acceptance included lack of plans to pursue a research career (e.g., pursuing an advanced professional degree such as M.D. or Pharm.D.) or poor academic performance during their time as Associates trainees.

Table 3.

Associates who continued in an upper division NIH-funded Research Program

Academic Year	BUILD Associates (n)	BUILD Scholars (n)	MARC (n)	RISE (n)	BUILD Scholars Only (%)	All NIH-Funded Programs (%)
15–16	35	19	3	2	54	68
16–17	25	16	4	1	64	84
17–18	24	13	0	0	54	54
18–19	25	11	0	0	44	44
Total	109	59	7	3	54	62.5

Chi-square analyses comparing Associates who did not continue and Associates who continued to upper division NIH programs with trainees’ discipline and demographic profiles showed no significant associations of continuation with discipline, URM status, and gender. The finding that nearly two-thirds of Associates chose to continue with their research training and were admitted to an intensive, upper-division training program regardless of discipline, URM status, and gender demonstrates an important successful outcome of early exposure to a research program. The early exposure can help create pathways to more research-intensive opportunities and ultimately strengthen the preparation of a broad range of students for graduate school and post-baccalaureate research (Hernandez, 2016; Kezar & Holcombe, 2017; Kitchen et al., 2018).

Because the Associates Program entailed a significant commitment in terms of students’ time and effort, we compared the Associates’ GPAs at the beginning and completion of the program to evaluate whether participating in it impacted the trainees’ academic performance. We found that the Associates’ GPAs changed very little from the time the students entered the Associates Program to when they completed it (M = 3.56 at beginning and 3.51 at the completion of Associates, F(1,101) = 2.57, p = 0.112, ns). However, we did find that the GPAs of Associates who were accepted to an upper-division research training program (M = 3.52, SEM = .042) were significantly higher than Associates who were not accepted (M = 3.35, SEM = .058), F(1,108) = 6.13, p = .015, η² = .054.

Overall, the non-completion rate was low (8.7%, n = 11 out of 127 total participants) for the Associates Program. More non-URM participants (n = 7) did not complete the Associates Program compared to URM participants (n = 4). In terms of gender, more male participants (n = 6) did not complete the Associates Program compared to female (n = 4) and non-binary (n =1) participants. The disciplinary comparisons revealed that more biomedical students (n = 10) left the program early compared to their behavioral counterpart (n = 1). Most trainees stated that the reason for leaving was personal (e.g., personal health/medical leave or caring for family member; n = 5), followed by a change in career goals (n = 3) to non-research major such as nursing or veterinary medicine. Two Associates left due to academic challenges (e.g., not passing course in major), and one did not provide a reason.

Gains in Belongingness, Science Interest, and Research Understanding and Skills.

We examined different areas of gains for the summer PREP and the academic year because they had different program goals. The two-week summer PREP was designed to promote a sense of community within the BUILD Program and reinforce their excitement for science. During the academic year, the program focused on fostering their understanding of the role of research in science and developing foundational research skills through their research and learning community activities.

Impact of the Summer PREP Component.

For BUILD Belonging, the main effect of time, F(1,63) = 29.59, p < 0.001, was significant. BUILD Belonging scores were higher at the end of PREP (M = 5.46, SEM = .080) than at the beginning (M = 4.88, SEM =.104). No other main effects or higher-level interactions were significant. For Science Interests, the main effect of time was significant, F(1,72) = 21.34, p < 0.001, as was the main effect of group, F(1,72) = 5.43, p = 0.023. Science Interests scores were higher at the end of PREP (M = 5.33, SEM = .062) than at the beginning of PREP (M = 5.04, SEM = .078), and higher for students who continued to an upper-division program (M = 5.04, SEM = .096) than students who did not continue (M = 5.33, SEM = .084). The interaction between time and group was not significant. It should be noted, though, that the Associates’ Science Interests scores were high overall (M = 5.19 on a 6-point scale).

Gains in Research Understanding and Skills During the Academic Year.

We examined the Associates’ reported gain ratings on 5 items relating to research career understanding (item 1) and research skills (items 2–5), as shown in Table 4. The main effect of time was significant for all items except gain in understanding journal articles, which approached statistical significance. Students reported significant gains in their understanding of what everyday research work is like from mid-academic year to end of the Associates Program. Similarly, they showed gains in writing, oral presentation, and data analysis skills over the academic year. Overall, these gains reported did not differ significantly from those who continued to an upper-division program versus those who did not continue (i.e., no main effects of group). Moreover, these gains also did not differ by discipline, URM status, and gender (i.e., no significant interactions of time with URM status, gender, and discipline were observed; data not shown).

Table 4.

Research Understanding and Skills Gains during Academic Year identified by Associates

Research Understanding and Skills	Mean Rating Pre (SEM)	Mean Rating Post (SEM)	Main Effect of Time	Main Effect of Group	Time × Group Interaction
1. Understanding what everyday research work is like	4.63 (.147)	5.05 (.108)	F(1,63) = 6.98, p = .010	Not significant	F(1,63) = 3.60, p = .062
2. Writing scientific reports or papers	3.91 (.148)	4.81 (.135)	F(1,62) = 27.73, p <.001	Not significant	Not significant
3. Making oral presentations	3.94 (.159)	5.02 (.125)	F(1,62) = 34.18, p <.001	Not significant	F(1,62) = 3.99, p = .050
4. Using statistics to analyze data	3.09 (.228)	4.05 (.193)	F(1,54) = 16.27, p <.001	Not significant	Not significant
5. Understanding journal articles	4.64 (.141)	4.93 (.135)	F(1,64) = 3.08, p = .084	Not significant	Not significant

In addition, the time × group interaction was significant (see Table 4 for F-ratios) for gains in making oral presentations and approached statistical significance for gains in understanding of what everyday research work is like. Students who continued to an upper-division training program indicated more gain (M = 5.26 post vs. 3.83 pre; 1.43 mean difference) in making oral presentations over the academic year than students who did not continue (M = 4.77 post vs. 4.06 pre; 0.71 mean difference). For understanding what everyday research work is like, students who continued to an upper-division training program showed a trend of more gain (M = 5.26 post vs. 4.52 pre; 0.74 mean difference) than students who did not continue (M = 4.85 post vs. 4.72 pre; 0.13 mean difference).

A series of post hoc analyses was conducted to better understand the main effect of continuation to an upper-division program on science interests. This main effect means that students who went on to an upper-division program reported generally greater levels of science interests than those students who did not go on. Given that students who continued to an upper-division program also reported making greater gains in the research domain over time than those who did not, we examined whether greater science interests reported at the end of the Summer PREP was correlated with greater research gains at the end of the Associates Program. This analysis provided a way of exploring a possible path that links science interests to continuation in research training through greater gains in research understanding and skills. Spearman’s rho correlation coefficients were computed due to the positively skewed distribution of all variables (range: 3.06–5.49). As shown in Table 5, the level of science interests reported at the end of Summer PREP was positively and significantly correlated with four of the five indicators of research gain, including the two that the Associates who continued were significantly greater gains over time than those who did not (i.e., “understanding what everyday research work is like” and “making oral presentation”).

Table 5.

Spearman’s Rho Correlation Coefficients Between Science Interests and Gains in Research Understanding and Skills

Research Understanding and Skills at End of Spring	Science Interests Post-PREP (N=62)
Understanding what everyday research work is like	.338*
Writing scientific reports or papers	.285*
Making oral presentations	.347*
Using statistics to analyze data	.164
Understanding journal articles	.260*

What Motivated Students to Join the Associates Program?

Because the Associates program was intended to be an early intervention program, we examined whether the reasons reported by Associates for joining the program included exploratory aspects. Among the 10 possible reasons for joining the Associate Program (see Table 5), “Explore my interest in science,” “Gain hands-on experience in research,” “Have a good intellectual challenge,” and “Clarify whether I wanted to pursue a science career” were the top four reasons endorsed by the trainees. Overall, these reasons are consistent with exploratory nature of students at this early stage in college as they seek to clarify interests in their fields and disciplines. This pattern was similar across disciplines, URM status, and gender (data not shown).

Discussion

The one-year BUILD Associates Program was established with the goal of providing lower division students an introductory research experience. The training activities of the Associates Program were specifically designed around this goal and ultimately helped Associates make an informed decision about whether to pursue the more intensive research training opportunities at the upper-division level. Over three quarters of Associates who completed the program applied to an upper division research training program, illustrating their interest in continuing to pursue research. Moreover, nearly 63% of all Associates, and 77% of those who applied, were accepted into an upper division NIH-funded research training program. Thus, this study demonstrates that the Associates Program was an effective, early intervention for promoting and sustaining student interest in science-related research at a comprehensive, Minority Serving Institution. Also, the Associates Program attracted students in lower division levels who were eager to learn and explore scientific research as a potential pathway as we anticipated early interventions would (and should).

Another way to evaluate the effectiveness of the Associates Program was to examine its retention rate. Overall, the percentage of students who did not complete the Associates Program was only 9%, which is lower than the national figure of 22% for attrition in science and engineering and non-science and engineering majors (Trapani & Hale, 2019). The non-completion rate was larger for non-URM students than URM students, which suggests that our approach to structuring the Associates Program provided the support needed to maintain URM students’ interest in research. This retention finding also reflects the fact that some of our students were still exploring their interest in science and were in the process of defining their career goals. For example, in the case of the three URM students who had a change of career goals, two pursued other professional careers (e.g., nursing or veterinarian medicine) and the third no longer wanted to pursue a Ph.D. program that made them ineligible to continue in the program. Though they are categorically ‘non-completers,’ we do not view this as a negative programmatic outcome if it means that students were able to clarify their career goals and go on to pursue their academic and professional aspirations.

Our evaluation survey findings provide preliminary evidence that the Associates Program was effective in fostering personal and professional growth in students over time. Specifically, we found that the summer PREP component cultivated growth in an early interest in science and research, and a sense of belonging in BUILD, reinforcing the cohort model, by the end of the two-week program. Over the academic year, the Associates showed significant gains in their understanding of what everyday research work is like, and gains in their writing, oral presentation, and data analysis skills. Understandably, Associates who continued on to an upper-division research training program showed greater gains in the level of interests in science, understanding of research, and oral presentation skills than the Associates who did not continue. Furthermore, how much the Associates were interested in science and research by the end of summer PREP was associated with a broad range of research gains they perceived to have made by the end of Associates Program, including those that associated with continuation to an upper division program. Although preliminary, our results suggest that students with greater science interests are more motivated to engage in research and thereby acquire better understanding of what research is and greater research skills over time. Future research should examine possible mechanisms for such research gains in early entry research trainees.

Another persistent finding in this study is the lack of significant patterns and differences in outcomes associated with discipline, URM status, or gender. Our results showed that participating in the Associates Program did not take away time and attention from doing coursework as indicated by its lack of impact on academic GPA at the time of completion. As expected, strong academic performance was associated with continuation to an upper division training program. Associates who were accepted to continue in an upper division research training program had higher GPAs than those who did not continue. However, the GPA did not differ significantly with respect to discipline, URM status, and gender, suggesting the pipeline was similarly effective across behavioral and biomedical disciplines, for URM and non-URM as well as male and female students. We conclude that the absence of any observed group differences suggests that the early intervention program had a broad-based positive effect by supporting historically underrepresented and underserved students in behavioral and biomedical research and academic careers.

Although successful, the costs associated with this intervention program can be a barrier for wide implementation at universities without external funding. This is likely a major reason that the majority of existing programs on campuses focus on supporting upper division students. The cost of funding the Associates was about $10–12K per student. This cost estimate includes direct support to students (i.e., stipend/hourly pay, research supplies, and travel), but does not account for the indirect support (e.g., training director/graduate mentor/staff salaries, speaker fees, event costs) that is critical for a successful operation of a research training program. Thus, universities that wish to implement these programs without sufficient internal funds to cover the costs should anticipate securing funds from external sources such as federal and state agencies, industry, foundations, or private donations.

At CSULB, we determined a practical way to institutionalize program elements of the Associates Program. We have partnered with CSULB’s Office of Undergraduate Research Services to create a Health Research Peer Group in an existing Undergraduate Research Opportunity Program (UROP). UROP was originally designed for freshman and sophomores who have federal work study to explore their interest in research and work with a faculty mentor to gain hands-on research experience. In addition, half of the student population at CSULB has transferred from another institution or local community colleges. For these transfer students, such an early exposure to research is not as feasible. To address this need, the UROP Health Research Peer Group was opened to transfer students at the junior level. Like the Associates Program, UROP is a structured program that matches students and faculty for mentored student research and employs a learning community model, although it is of a narrower scope than what was possible in BUILD Associates with NIH funding. This collaboration with an existing, lower cost campus resource is intended to sustain and institutionalize the early intervention (although at a less intensive level) and pipeline benefits of the Associates Program.

Conclusion

Taken together, our findings suggest that the promising outcomes of early interventions can be achieved in less research-resourced institutions such as CSULB that is both an Hispanic-Serving Institution (HSI) and Asian American and Native American Pacific Islander-Serving Institution (AANAPISI), and a comprehensive, public, broad access institution. We demonstrated that a focus on early exposure to undergraduate research training is a valuable high impact practice that, when designed well, can provide mentored research experiences to undergraduate students from a wide range of disciplines in a comprehensive institution and prepare them to continue in the pipeline and be competitive for more intensive upper division research training programs. In the long run, this early exposure to research can help set students on a promising academic career path to graduate school and beyond. Our BUILD Associates Program demonstrated that early intervention research training can help level the playing field for URM, female, and other underrepresented students in their pursuit of research careers in biomedical and behavioral disciplines.

Table 6.

Research Motivations identified by Associates

Reasons	Yes (%)
Explore my interest in science	100.0
Gain hands-on experience in research	100.0
Have a good intellectual challenge	100.0
Clarify whether I wanted to pursue a science research career	95.2
Participate in a program with a strong reputation	90.3
Enhance my resume	90.3
Clarify whether graduate school would be a good choice for me	90.2
Clarify which field I want to study	90.0
Get good letters of recommendation	80.6
Work more closely with a particular faculty member	66.1