The CSULB BUILD Scholars Program: A Research-Intensive Upper-Division Program to Broaden and Diversify the Behavioral and Biomedical Research Workforce

Abstract

Engaging undergraduates in research is a high impact practice shown to increase underrepresented students’ persistence in Science, Technology, Engineering, and Mathematics (STEM) fields and entry into research careers. The California State University Long Beach (CSULB) BUilding Infrastructure Leading to Diversity (BUILD) Scholars Program is a 2-year, upper-division research training program. Although similar research training programs exist, most admit relatively few students a year, primarily from the natural sciences. The BUILD award from the National Institutes of Health (NIH) allowed us to broaden research training to a wider range of health-related disciplines across four different colleges to have more even representation across the behavioral and biomedical science disciplines. Our Scholars Program builds upon best practices of programmatic mentoring, assets-based and cohort-based training, financial and educational support, and intensive research training by faculty in the students’ disciplines. In this paper, we present the outcomes and evaluation of our training program with data from the first phase of the BUILD award (2015-2019). Findings demonstrate that our Scholars Program was effective at recruiting and retaining underrepresented students from a broad range of disciplines. Moreover, our trainees demonstrated a high level of research engagement through off-campus summer research experiences, conference presentations, and publications. The intensive training in the Scholars Program also yielded high graduate school acceptance rates for our trainees. Most importantly, our findings show that it is possible to broaden an intensive undergraduate research training program that is similarly effective for trainees across behavioral and biomedical disciplines, underrepresented minority status, and gender. While we highlight several elements of our training program, we emphasize these components likely work together interactively, and institutions wanting to establish a similar training program need to ensure sufficient resources for its successful implementation.

Introduction

Broadening and Diversifying the Behavioral and Biomedical Research Workforce through a Research-Intensive, Upper-Division Program.

Women, racial and ethnic minorities (e.g., Latinx, African American, Native Alaskan, and Native American), and people with disabilities are underrepresented in the science and engineering workforce (National Science Foundation, 2017). Studies have found that cultural barriers negatively impact their persistence in academic and research careers (e.g., Seymour & Hewitt, 1997). Underrepresented students (URS) who do not identify with science or see themselves reflected in their professors or course content may switch to majors and career paths perceived to be more congruent with their cultural identities. Seymour, Hunter, and Weston (2019) noted that the percentage of students switching from STEM (Science, Technology, Engineering and Mathematics) to non-STEM disciplines decreased from 44% (1997) to 28% (2013-2014) over 25 years. While these trends seem promising in meeting the increased need for STEM workforce, the percentage of switches remained larger for women (especially women of color), first generation college students, and underrepresented students (URS) from families with lower socio-economic status (e.g., PELL-eligible) than for their respective counterparts leading to greater inequity in representation (Seymour et al., 2019).

Cromley, Perez, and Kaplan (2016) found that cognitive (i.e., knowledge and skills), motivational, and institutional factors influence undergraduates’ persistence in STEM. Engaging students in undergraduate research is an intervention shown to be effective in retaining URS in STEM fields, increase students’ graduation rates, and enable students to pursue graduate degrees (see e.g., Bayliss et al., 2018). Specifically, the duration and level of involvement of research experiences have been found to strengthen students’ research and academic skills, clarification of research and career goals, and socialization into graduate school and the research enterprise. This is accomplished through activities such as summer research experiences (SREs), conference participation, and opportunities to produce academic publications (Gilmore et al., 2015; Hathaway et al., 2002; Hunter et al., 2007; Jones et al., 2010; Nnadozie et al., 2001). Unfortunately, Rodríguez Amaya et al. (2018) found that many URS (e.g., Latinx and first-generation students) did not engage in research even when they were aware of research activities being available on their campus. They note that a misconception still exists among students that research is only for those who seek to become scientists who work in isolation in a lab. Moreover, among those URS who do benefit from resources and opportunities created by intensive research training programs, a significant number of them select alternative career paths (Hurtado et al., 2009; Hall et al., 2016).

In addition, URS may experience barriers such as lack of access to funding, programmatic mentoring, and other institutional practices (e.g., course structure/timing, use of “weed out” courses, lack of academic and career development support; Cromley et al., 2016). The lack of broad engagement in research by URS may also be a result of the fact that the small size and narrow disciplinary focus of many intensive research training programs limits the number of students that can be reached. Combined, these issues highlight the importance of demystifying research for URS, making it more accessible for those who may not traditionally seek such opportunities, and support them in navigating graduate school opportunities. Thus, the need to diversify the pipeline of URS pursuing STEM remains strong. Health-related research has been the central focus of the Diversity Program Consortium funded by the National Institutes of Health (NIH; see Norris et al., 2020) because increased representation of URS in biomedical and behavioral research is imperative to best serve the nation’s diverse population and its complex health challenges. This fact has been especially highlighted by the major health disparities observed during the COVID-19 pandemic, where counties in the US with more diverse demographics were at a higher risk of COVID-19 infections (Abedi et al., 2021).

In 2015, NIH awarded California State University Long Beach (CSULB) one of 10 BUilding Infrastructure Leading to Diversity (BUILD) awards to test education interventions that would increase the number and diversity of students entering Ph.D. programs in health-related disciplines, with the goal to increase the biomedical and behavioral research workforce. The CSULB BUILD Student Training Programs provided undergraduates with research exposure at the sophomore level through an Associates Program and intensive research training at the upper-division level through a Scholars Program. Background on the CSULB BUILD award and description of the outcomes of the one-year Associates Program are described in detail by Kingsford et al. (in press). The Associates Program served as a pipeline for the Scholars program, but students can be accepted into the Scholars Program without participating in the Associate Program. The Scholars Program is a two-year program that focused on preparing juniors for graduate study in health-related disciplines with the long-term goal of obtaining a Ph.D. and pursuing health-related research careers. We referred to Scholars 1 as those who were in their first year of the program, and Scholars 2 as those who were in their second year. One unique aspect of the CSULB BUILD student training program is that it broadened access to research training to 20 majors in four different colleges [College of Engineering (COE), College of Health and Human Services (CHHS), College of Liberal Arts (CLA), and College of Natural Sciences and Mathematics (CNSM)] with faculty conducting health-related research (see Appendix A for listing of department listing of the student participants). The focus of the present paper is to describe the Scholars Program and provide evidence documenting its outcomes relating to broadening and diversifying the number of underrepresented minority (URM) students and, more broadly, URS entering graduate programs in health-related disciplines. We define URM students as those who self-identify as African American or Black, Native American, Native Hawaiian or Pacific Islander, Multiple Races, or Hispanic. URS in our programs are students who are URM students, first-generation college students, students from low socio-economic status (i.e., financial aid eligible), students with disabilities, or women in certain STEM fields.

URS who feel competent in their field (i.e., have knowledge and skills), feel that their work makes a meaningful contribution to society (i.e., have motivation to continue), have institutional support (e.g., academic resources), and financial support are more likely to persist in STEM disciplines (Cromley et al., 2016). The Scholars Program built upon successful practices of previous and existing URS training programs at CSULB (e.g., National Institute of Mental Health Career Opportunities in Research Program in Psychology and Maximizing Access in Research Careers in the College of Natural Sciences and Mathematics) by providing students with (a) programmatic mentoring to enhance their scientific research knowledge and sense of belonging, and (b) financial support and educational resources to reduce socio-economical and institutional barriers. In addition, we incorporated (c) assets-based training to increase students’ motivation to persist in research careers (Johnson & Bozeman, 2012). These components represent evidence-based practices that are vital to promoting higher graduation rates and pursuit of graduate education at the Ph.D. level (Bayliss et al., 2018; Johnson & Bozeman, 2012).

Multi-Tiered Programmatic Mentoring.

According to Bayliss et al. (2018), successful programmatic mentoring is provided from multiple sources. These mentors work together synergistically to develop student knowledge, skills, and abilities in conducting research and to provide career and professional development opportunities that promotes student growth as a researcher. This approach expands upon traditional student research mentoring by faculty in their field by providing students access to a network of mentors who can provide the student with different types of support based on their individual needs.

The Scholars Program incorporates programmatic mentoring by providing students with access to mentors in their respective research fields and mentors within the BUILD Program. In the research lab or research group, the mentors include not only the faculty member, but also may include post-docs, graduate students, and near-peer mentors such as more senior undergraduate students working with the faculty mentor. Having multiple mentors within a research group allows students to be exposed to a variety of individuals who can provide different types or levels of support. For example, working with more advanced and experienced near-peer mentors allows URS to see and learn from other students with whom they are more likely to identify. In addition to research training in their discipline, faculty or post-doc mentors can provide students with more career and professional development support for academic and professional success.

Prior research has shown that the quality of research mentoring is a key aspect of a successful undergraduate research experience (Pfund et al., 2016). Studies on mentoring have identified practices that lead to more positive student outcomes (see Byars-Winston et al., 2015; Haeger & Fresquez, 2016, for example). Despite the importance of quality research mentoring, models or guidelines to support faculty in creating such successful mentoring relationships have been lacking (Shanahan et al., 2015). This can leave faculty without formalized training in any kind of mentoring, including mentoring URS, and as a result, many may rely on outdated practices that do not support the needs of today’s students. Mentor training at CSULB was not formalized prior to BUILD (i.e., methods/approaches of mentoring undergraduates in research was based on personal styles or experiences of individual faculty). Thus, we developed and provided formal mentor training to our faculty through a BUILD Mentoring Community (BMC).

The BMC was a 2-semester program where participants met as a learning community over the course of 10 weeks during one semester to discuss topics such as their mentoring philosophy, how to align faculty and student expectations, effective communication, issues of equity and inclusion, and how to foster mentee independence, among other topics. These topics and their associated activities were based on the Entering Mentoring (Pfund et al., 2016) curriculum that was developed at the University of Wisconsin, Madison. The content was validated by the National Research Mentoring Network (NRMN) for post-doctoral fellows and modified to be appropriate for faculty at our university. BMC participants also attended a face-to-face intercultural communication workshop and developed a project where they performed and evaluated a new mentoring approach, activity or skill in a subsequent semester (see Young & Stormes, 2020, for a complete description and evaluation of the program). The BMC met all the learning objectives of the Entering Mentoring training and all 93 of our BUILD faculty mentors (across 24 different disciplines) who completed the program earned a certificate of completion for NRMN Research Mentor Training. Having all CSULB BUILD program faculty mentors complete the BMC ensured that the mentors had common knowledge of best practices known to positively impact undergraduate research experiences of students, especially URS.

Programmatic mentoring was also provided through the BUILD Training Directors and staff, as well as near-peer mentoring via Graduate Mentors and second-year Scholars. Because BUILD Program mentors are usually different from their research lab/group mentor, students can bring up questions and concerns about their research training in a safe and supportive environment. Many of the BUILD Principal Investigators served as Training Directors along with other faculty mentors to implement the BUILD research training curriculum, which will be described in Section 1.2. Graduate Mentors were master’s students from a range of behavioral and biomedical disciplines who provided near-peer mentorship and support for the training curriculum (e.g., tracking of students, grading and facilitation of small group discussions; see Abeywardana et al., 2020, for a detailed description of the Graduate Mentor’s role and training). BUILD staff also provided students with instrumental and logistical support for their program participation as well as assistance and encouragement with various student training activities. The BUILD Training Directors, Graduate Mentors, and staff had weekly contact with the BUILD Scholars in the learning community and during office hours, providing them with access to an additional network of support. The Training Directors also met with students as needed to discuss their progress in the BUILD Program, inquire about the adequacy of meetings with their research faculty mentors, answer questions about research, research training and/or graduate school preparation, and provide general advice and guidance as the students progressed in the program. On occasion, BUILD Training Directors and staff also served as a mediator between the student and their faculty research mentor.

Lastly, we encouraged students to extend their opportunities for networking and mentoring by providing opportunities to interact with colloquia speakers who are researchers, faculty, and doctoral students from underrepresented groups, as well as BUILD alumni. We believed that facilitating these connections with a broader group of individuals could also expose students to potential role models from around the country.

Financial Support and Educational Resources.

Most BUILD trainees qualified for financial need as defined by financial aid eligibility or came from families below the poverty level. Thus, from the beginning, providing financial resources both personally and for their research was a critical aspect of participation in the BUILD Program. The Scholars received monthly stipends, partial tuition support, as well as funds for research supplies via their faculty mentors, and conference travel. They also received priority registration to ensure that they were able to enroll in their required courses for timely graduation. One of the challenges we faced was that the BUILD training years did not always align with a student’s graduation timeline. Some trainees, especially those in high unit majors in natural sciences and engineering, took an extra semester or two to graduate after they completed the Scholars Program (i.e., 4.5-5 year graduates). For other students, it was to delay their graduation to improve their academic performance while they wrapped-up research projects that would make them more competitive for graduate school applications in the next academic year. One of the major challenges for these students was that they were not able to apply to graduate school at the same time as the rest of their cohort and they did not get the full support of the student training and financial interventions during that crucial semester. To address this issue, in year 4 of BUILD (2017-2018), we extended the Scholars Program for a semester in their third year, which enabled Scholars who were not graduating to continue in the program for an additional semester with financial support (i.e., stipends, partial tuition, research supplies, and conference travel support). Year 3 Scholars continued to work with a BUILD Training Director in their own learning community as they prepared and submitted their graduate school applications during the 5th semester in BUILD. They also continued to do research with their faculty mentors and were strongly encouraged to disseminate their research at professional conferences and in publications.

Assets-based Approach Combined with Research Training.

Scientific efficacy, interest in science, and science identity are assets associated with URS persistence in STEM (Estrada et al., 2016). In addition, Johnson and Bozeman (2012) identified five asset bundles (educational endowments, science socialization, network development, family expectations, and material resources) that interact with each other in influencing URS persistence in STEM fields. According to an assets-based approach, training programs are designed to develop students’ assets by focusing on their capabilities rather than trying to rectify their deficiencies (Johnson & Bozeman, 2012). Educational endowments refer to the training curriculum and how it enhances student scientific efficacy. Science socialization and network development occur through interactions with the BUILD Program mentors and faculty research mentors, presentations at conferences, and engagement in the research and publication processes. These two asset bundles can increase students’ interest in science and their science identity. Family expectations refer to the fact that there are differences in role expectations (especially by gender) and the value of different career paths held across racial and ethnic communities. For example, our students reported that their families were more supportive of them pursuing careers in medicine or nursing, careers perceived as having tangible financial rewards and connections to the community and that they had little knowledge about or appreciation for research careers. Understanding and encouraging students to form culturally congruent science identities were key components of the Scholars Program that attempted to allow students to capitalize on both their scientific and cultural identities and assets.

Summary.

The Scholars Program incorporated well-established research training components such as research experience, weekly program meetings, scientific writing workshops/courses, GRE preparation, academic/career advising, and programmatic mentoring (Bayliss et al. 2018). These are known to enhance URS’s educational endowment, scientific and research efficacy, engagement in research through increasing science interest, and science identity through science socialization and network development (see e.g., Johnson & Bozeman, 2012; Cameron et al., 2020). The program also provided URS the financial and education resources for competitive application to graduate school. We also included family engagement activities as a best practice (Maton et al., 2012). Family members of BUILD trainees often were not aware of research careers and had understandable concerns about their student participating in activities such as traveling to national conferences or SREs or moving away for a graduate program. To address these concerns, we engaged family members at multiple points in the program so that they could learn more about what their students were doing in the program and ask BUILD Program faculty and staff questions about the students’ research training and/or address concerns about family expectations. In the next section, we describe the main research training components and timeline of the Scholars Program.

Overview of The Scholars Program.

The two-year Scholars Program began with an 8-week summer research training program called Summer Undergraduate Research Gateway to Excellence (SURGE) in the students’ first year. SURGE was a full-time commitment that included two weekly 3-hour BUILD learning communities and weekly research activities in their faculty research mentor’s projects to develop research efficacy. The learning community focused on culturally relevant community building activities (e.g., the sharing of a culture box with meaningful items to each student), an introduction to research and research careers, development of an individual development plan (which enabled students to articulate short-, medium- and long-term goals), responsible conduct of research training, and field trips to our R1 partner campuses at the University of California, Irvine and the University of Southern California. All these activities were designed to increase students’ research efficacy and interest and foster the growth of their science identity.

SURGE also helped students learn how to speak publicly about their research, a key skill for researchers. Scholars first presented their projects as succinct “elevator speeches”, with one version geared towards the academic community and a less technical version geared towards family and friends. The elevator speeches not only developed students’ scientific efficacy in terms of communication skills, but also promoted their science interest and asset bundle of science socialization. Students from each Scholars cohort also competed for prizes to enhance a sense of fun and community. SURGE culminated in a Summer Symposium where Scholars presented a poster of the research they had completed that summer and/or proposed work that would be continued during the academic year. An important element of this event was inclusion of family or other loved ones. The symposium started with an acknowledgement of the family members and loved ones in the academic journey of the BUILD trainees, followed by an orientation to the BUILD Scholars Program. During the research poster session, family members and loved ones were able to see the fruits of the long hours the students had put into their projects and had opportunities to meet their students’ network of mentors. In this way, the Summer Symposium fostered the students’ researcher identities, both for themselves and their families, whose buy-in to the research career is key to their success.

During the academic year, all Scholars participated in faculty-mentored research and the BUILD learning community, which was structured as a 1-unit, graded course in each semester of both years of the training program, increasing their research efficacy and educational endowments. They also received support in preparing for the GRE exam via workshops available on campus. Students were required to complete at least one practice GRE exam prior to completion of their first year in the program. Scholars were financially supported to attend at least one national conference in each of their two years in the program. At a minimum, Scholars typically attended a student-focused conference [e.g., Annual Biomedical Research Conference for Minority Students (ABRCMS) or Society for Advancement of Chicanos/Hispanics and Native Americans in Science (SACNAS)] chaperoned by BUILD Training Directors and staff in their first year and discipline-specific professional conferences with their research faculty mentors in their second year. Attendance at these conferences enhanced the scientific socialization asset bundle.

In preparation for their second summer in the program, Scholars were required to apply for an off-campus SRE at an R1 institution or industry setting. Scholars also had the option of making informal arrangements for summer research projects with researchers at external sites, which was especially common in certain behavioral disciplines. Those that did not have an off-campus research placement or informal research arrangement participated in an on-campus SURGE 2 learning community where they met with a Training Director once a week and continued their research training with their respective research faculty mentor. SURGE 2 focused on helping students begin their graduate school applications. All Scholars were required to take the GRE before the start of fall semester so that they could retake it if their scores were not competitive for their field/discipline. The second year of the Scholars Program primarily focused on supporting Scholars’ research into graduate school options and the application process and conducting advanced research with their faculty mentors, including conference presentation and scientific writing.

Scholars also engaged in a curriculum of research courses during their two-year program, including courses developed by the CSULB BUILD Program in collaboration with on-campus departments in Interdisciplinary Approaches to Health Disparities, Introduction to Research Methods, Scientific Research Communication, and Advanced Research Methods (see Taing et al., 2022, for details on the implementation and evaluation of the research courses). In the latter three courses, students learned important skills in experimental design and approach, grant and manuscript writing, and additional presentation skills (enhancing scientific efficacy and education endowments). Trainees were required to take at least one of these courses (or a suitable substitution within their own major) for each year of their participation in BUILD. Lastly, during their time in the Scholars program BUILD trainees were required to complete at least 8 hours of face-to-face training in Responsible Conduct of Research (RCR), in addition to completing the online Collaborative Institutional Training Initiative (CITI) training at the beginning of their participation in BUILD. Scholars celebrated the completion of the two-year program with their faculty mentors, family members, and loved ones at the BUILD Commencement Ceremony.

Present Study.

NIH’s goal for BUILD is to increase the number of URM students in health-related disciplines that go on to pursue doctoral degrees and enter research careers. Thus, the goal of the present study was to answer two major research questions through a series of subquestions:

1. Did BUILD attract, select, and retain a diverse group of students to the Scholars Program?

   1. What was the diversity of the applicant pool and the participants selected for the Scholars Program?
   2. What were the top reasons (i.e., motivators) given by Scholars for applying to the program?
   3. What was the retention rate for the Scholars Program? What were the reasons given by students for leaving the program?
2. How effective was the Scholars Program?

   1. Did the BUILD programmatic requirements affect the Scholars’ academic performance?
   2. What were the areas of growth for Scholars over the two years in the training program in terms of scientific efficacy?
   3. What were the intermediate and final training outcomes of the Scholars Program?
   4. How well did the Scholars Program serve as a pipeline to graduate programs in health-related disciplines?

To answer these questions, we utilized data collected by our program for evaluation purposes following an approved Institutional Review Board protocol from CSULB. Due to the live implementation of the Scholars Program, we acknowledge that data collection was not uniform across all four years of the program, as the instruments changed to accommodate new priorities, and data collection methods varied over the years depending on level of available support personnel. We also acknowledge that our evaluation of the Scholars Program is at the entire program-level and not by the specific components. However, given that these challenges occur during development and implementation of most real-world training programs, our findings should still be informative to researchers and practitioners implementing student training programs.

Moreover, as with many programs, success can be defined in multiple ways. Estrada et al. (2021, p.2) noted one common definition of success is that “students do something as a consequence of their involvement in a training program”, such as the student entering graduate school after completing the program. While we acknowledge that there are multiple indicators of success, we define it as achieving the goals and hallmarks set by the NIH Diversity Program Consortium (McCreath et al., 2017). These include cultivating academic and scientific self-efficacy, science/researcher identity, ensuring retention and persistence in a biomedical science discipline relevant to BUILD, and participation in an undergraduate/summer biomedical research training, among several other hallmarks. Below we describe our methods, results, and discussion for each of these questions in separate sections for clarity.

Did BUILD Attract, Select, and Retain a Diverse Group of Students to the Scholars Program?

NIH’s goal for BUILD is to increase the number of URM students in health-related disciplines that go on to pursue doctoral degrees and enter research careers. To achieve that goal, we broadened the applicant pool by including health-related disciplines across four different colleges on campus. Moreover, we were intentional in our outreach and recruitment efforts to reach URM students, and used more inclusive metrics (e.g., evaluation of diverse experiences and resilience) in our selection criteria. Details on the Outreach, Recruitment, and Selection process of the CSULB BUILD Program is described in detail by Kingsford et al. (submitted) and will only be briefly covered below.

Our outreach and recruitment efforts were combined with MARC U*STAR (Maximizing Access to Research Careers Undergraduate Student Training in Academic Research) and RISE (Research Training Initiative for Student Enhancement) Programs, two NIH-funded research training programs for undergraduates focusing on increasing the number of URM students in the biomedical workforce on our campus. Outreach included flyers and marketing materials. In-person outreach and recruitment efforts included information sessions, class visits and presentations to specific groups and student organizations. BUILD staff, Graduate Mentors, and students also hosted information tables at campus events. For off-campus recruitment, we worked with local community colleges to recruit transfer students to the Scholars Program. In the first two years of BUILD, the recruitment consisted of providing our community college partners with flyers about the BUILD Program as well as hosting information sessions at their campuses. Starting in Year 3, these efforts were supplemented with online recruitment videos, customized for each campus, and BUILD student ambassador visits (i.e., current BUILD Scholars who were transfer students would return to their respective community colleges to recruit the next cohort). Reports on recruitment activities and their respective student ambassadors were also sent to individual campus presidents and college deans to increase awareness of our BUILD Program.

To minimize competition among and maximize student access to training programs, a common application was developed for the BUILD, MARC U*STAR and RISE Programs, and the selection committee consisted of program directors and faculty mentors from all three programs. Students were selected to only one program, using a holistic evaluation of each applicant based on traditional metrics (i.e., academic record, faculty reference, students’ research and personal statements) and non-traditional metrics (i.e., diversity of perspectives among the BUILD trainees; resilience in the face of challenges). A rubric was used to list factors under consideration and articulate the scoring process to avoid implicit biases. After each recruitment cycle, we modified the weighting of the scores for certain criterion on the rubric to help increase the diversity of trainees selected for our programs. Part of the selection process included matching of the selected student trainees with faculty mentors if they did not already work with a specific faculty member.

Using applicant data specific to the Scholars Program, we addressed whether BUILD attracted and selected a diverse group of students to the Scholars Program. Specifically, we examined the disciplinary and sociodemographic backgrounds of our applicants to determine the diversity of our applicant pool. Since our selection process was intentionally designed to be more inclusive of racial/ethnic groups historically marginalized in STEM, we examined whether the program participants reflected the applicant pool or were more diverse than the applicant pool. For participants who entered the Scholars Program, we also examined the motivators for joining and, if they left before completing the program, reasons for leaving to determine whether there were external or programmatic barriers that affected our students’ program completion.

In this section we addressed the first research question, Did BUILD attract, select, and retain a diverse group of students to the Scholars Program? by asking the three sub-questions:

1. What was the diversity of the applicant pool and the participants selected for the Scholars Program?

2. What were the top reasons (i.e., motivators) given by Scholars for applying to the program?

3. What was the retention rate for the Scholars Program? What were the reasons given by students for leaving the program?

Methods.

Information on the applicants’ demographic characteristics and majors were collected from the application materials that they submitted in the joint application to the three NIH training programs. In addition, informed consent was obtained from each trainee to use their program and evaluation data for research dissemination. All participants could decline to answer specific questions on the surveys/questionnaires used to collect data.

Demographic and majors of applicants.

Applicants’ demographic data included Race (African American/Black, Asian American, American Indian, Native Hawaiian/Pacific Islander, White, or more than one race) and Ethnicity (Hispanic or non-Hispanic), using categories consistent with NIH reporting requirements. We grouped students into categories of URM (i.e., African American, or Black, Native American, Native Hawaiian or Pacific Islander, Multiple Races or Hispanic) or non-URM (Asian American or White) to designate students who have been historically underrepresented in STEM. Applicants reported their gender as male, female, or non-binary.

Applicants also reported their college major, and we coded those data into “behavioral” or “biomedical” disciplines. Behavioral disciplines include majors in selected departments in CLA (e.g., Anthropology, Linguistics, Sociology, Psychology) and in CHHS (e.g., Family and Consumer Sciences, Gerontology, Kinesiology, Health Care Administration, Health Science). Biomedical disciplines included majors in two departments in CNSM (Biological Sciences and Chemistry & Biochemistry) and selected departments in COE (e.g., Biomedical Engineering, Chemical Engineering, Computer Science, Electrical Engineering, and Mechanical and Aerospace Engineering).

Reasons for Joining BUILD.

We assessed selected participants’ motivation for joining the Scholars Program at the beginning of the program using a ten-item measure from the Undergraduate Research Student Self-Assessment (Weston & Laursen, 2015). Trainees indicated “yes” or “no” to each of the ten possible reasons for doing research, which ranged from wanting to have a good intellectual challenge, exploring interests in science, getting clarification for their future directions to wanting to develop a stronger research portfolio for resume and letters of recommendation.

Reasons for Attrition.

Scholars who left the training program early were asked to participate in an exit interview conducted by a BUILD Program Evaluator who inquired about their experience in the BUILD Program, reasons for leaving the program early, and changes in their academic and/or professional goals. The responses were coded into categories reflecting major themes for leaving the program.

Results and Discussion.

Disciplinary and Sociodemographic Distributions of Applicants and Participants.

Table 1 shows the distributions of applicants and BUILD Program participants by discipline [behavioral (CHHS/CLA) vs. biomedical (COE/CNSM) sciences, see Appendix A for majors of participants] and gender (Male vs. Female vs. Non-Binary) over the 4 recruiting cycles (2015-2019). To increase transparency regarding the disaggregation of race/ethnicity categories in research, we provide further details of the BUILD URM/non-URM applicant and participant data in Table 2a. The URM aggregated number in Table 1 shows slightly higher numbers than those in Table 2a since participants who self-identified as White or Asian as racial group and Hispanic as ethnicity (i.e., they answered ‘yes’ to the Hispanic category) were included in the total URM category in Table 1.

Table 1.

Number of BUILD Applicants and Participants by Discipline, URM, and Gender

			Academic Discipline				URM Status						Gender
	Applicant and Participant Data	Overall	Behavioral Sciences		Biomedical Sciences		URM		Non-URM		Unknown/ Declined		Male		Female		Gender Non- Binary
Scholars (N=183)		N	N	%	N	%	N	%	N	%	N	%	N	%	N	%	N	%
2015-2016	Applicants	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
	Participants	47	17	36.2	30	63.8	28	59.6	19	40.4	0	0.0	19	40.4	28	59.6	0	0.0
2016-2017	Applicants	88	29	33.0	59	67.0	43	48.9	37	42.0	8	9.1	28	31.8	58	65.9	2	2.3
	Participants	45	21	46.7	24	53.3	29	64.4	16	35.6	0	0.0	16	35.6	28	62.2	1	2.2
2017-2018	Applicants	113	42	37.2	71	62.8	56	49.6	54	47.8	3	2.7	38	33.6	75	66.4	0	0.0
	Participants	49	17	34.7	32	65.3	25	51.0	24	49.0	0	0.	17	34.7	32	65.3	0	0.0
2018-2019	Applicants	110	48	43.6	62	56.4	16	14.5	67	60.9	27	24.5	30	27.3	79	71.8	1	<1.0
	Participants	42	25	59.5	17	40.5	25	59.5	17	40.5	0	0.0	13	31.0	29	69.1	0	0.0
Total	Total Applicants	311	119	38.3	192	61.7	115	37.0	158	50.8	38	12.2	96	30.9	212	68.2	3	<1.0
	Total Participants	183	80	43.7	103	56.3	107	58.5	76	41.5	0	0.0	65	35.5	117	63.9	1	0.6

Notes: N/A = data for 2015-2016 are incomplete or not available. As a result, the total number of applicants should be higher than that indicated in the table; BUILD application was a joint effort with MARC U*STAR and RISE starting in 2016-2017; applicants may have been reviewed and accepted for the other NIH programs.

Table 2a.

Number of BUILD Applicants and Participants by Race/Ethnicity

	Applicant and Participant Data	Overall	African American/ Black		Asian American		American Indian		White		Native Hawaiian Pacific Islander		More than one Race		Declined to State/ Unknown
Scholars (N=183)		N	N	%	N	%	N	%	N	%	N	%	N	%	N	%
2015-2016	Applicants	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
	Participants	47	3	6.4	11	23.4	2	4.3	12	25.5	0	0.0	2	4.3	17	36.2
2016-2017	Applicants	88	4	4.5	23	26.1	0	0.0	11	12.5	0	0.0	17	19.3	33	37.5
	Participants	45	2	4.4	13	28.9	0	0.0	8	17.8	0	0.0	9	20.0	13	28.9
2017-2018	Applicants	113	9	8.0	34	30.1	0	0.0	20	17.7	0	0.0	9	8.0	41	36.3
	Participants	49	6	12.4	16	32.7	1	2.0	17	34.7	0	0.0	0	0.0	9	18.4
2018-2019	Applicants	110	8	7.3	37	33.6	4	3.6	26	23.6	0	0.0	8	7.3	27	24.5
	Participants	42	3	7.1	17	40.5	1	2.4	6	14.3	0	0.0	2	4.8	13	31.0
Total	Total Applicants	311	21	6.8	94	30.2	4	1.3	57	18.3	0	0.0	34	10.9	101	32.5
	Total Participants	183	14	7.7	57	31.2	4	2.2	43	23.5	0	0.0	13	7.1	52	28.4

The number of applicants per year increased from 88 to 311 over the four cycles. We also saw a skew in Scholars’ applicants toward biomedical disciplines early on (see Figure 1), but over time the number of behavioral applicants increased, approaching almost half of the applicant pool by the 2018-2019 academic year. In terms of participants (i.e., applicants that were admitted and enrolled in the program), a similar trend was observed with a greater biomedical representation in the first three years (varying from 53.3%-65.3% biomedical) but the behavioral representation exceeded biomedical in the 2018-2019 academic year (59.5% behavioral vs. 40.5% biomedical, see Figure 1).

Figure 1.

Percentage of Applicants and Participants as a Function of Discipline.

URM representation in Scholars varied over time, but it was always greater than non-URM across all four years, reaching a high of about 64% URM participants in the 2016-2017 academic year (Figure 2). Note that the number of non-URM applicants was higher than the URM applicants overall, but this was mainly due to the large number of non-URM applicants in the 2018-2019 academic year. Tables 2a and 2b show that the percentage of participants were similar or even higher than the percentage of applicants for those identifying as African American/Black and Hispanic/Latinx. While we had high participation rates of students who identified as White or Asian, we want to note that over 90% of our entire sample identified with having at least one underrepresented status as a racial/ethnic minoritized student, female, eligible for financial aid, or having a first-generation college status.

Figure 2.

Percentage of Applicants and Participants as a Function URM Status

Table 2b.

Number of BUILD Applicants and Participants by Hispanic/Latinx Category

Applicant and Participant Data		Overall	Hispanic/Latinx
Scholars (N=183)		N	N	%
2015-2016	Applicants	N/A	N/A	N/A
	Participants	47	23	48.9
2016-2017	Applicants	88	25	28.4
	Participants	45	19	42.2
2017-2018	Applicants	113	38	33.6
	Participants	49	19	38.8
2018-2019	Applicants	110	49	44.5
	Participants	42	21	50.0
Total	Total Applicants	311	112	36.0
	Total Participants	183	82	44.8

In terms of gender, the Scholars Program was dominated by participants who identified as female, increasing each year to 69% the 2018-2019 academic year cycle (Figure 3).

Figure 3.

Percentage of Applicants and Participants as a Function Gender.

Reasons (Motivators) for Joining the Scholars Program.

Among the 10 possible reasons presented to trainees for joining the Scholars Program, “Explore my interest in science,” “Gain hands-on experience in research,” “Have a good intellectual challenge,” and “Enhance my resume” were the four most endorsed by the Scholars. Table 3 below lists the reasons in the order of most to least commonly endorsed. This pattern was similar across Discipline, URM Status, and Gender. The motivators reflect the exploratory nature of undergraduate students seeking to clarify academic and career interests via their participation in a research training program. Further, these top-rated reasons highlight the need for expanding access to research opportunities, particularly for students who are rising upper-division students. Scholars rated “Enhance my resume” as the fourth most common reason for joining BUILD, which likely reflects the fact that Scholars are beginning to think more pragmatically about co-curricular opportunities that can enhance future post-baccalaureate opportunities.

Table 3.

Research Motivations identified by Scholars

Reasons	Yes (%)
Gain hands-on experience in research	99.3
Explore my interest in science	98.6
Have a good intellectual challenge	97.2
Enhance my resume	95.8
Participate in a program with a strong reputation	90.0
Get good letters of recommendation	86.0
Clarify which field I want to study	84.2
Work more closely with a particular faculty member	81.2
Clarify whether I wanted to pursue a science research career	77.4
Clarify whether graduate school would be a good choice for me	76.8

Program Attrition.

Overall, the non-completion rate was 17% for the Scholars Program, which is lower than the attrition rate of 22% for STEM majors at CSULB and national attrition rates of 22% of science and engineering majors (Trapani & Hale, 2019). In Table 4, we provide the number of trainees who did not complete the program as a function of Discipline, URM Status, and Gender, and the percentage of non-completion based on the total number of trainees from the same category. Given that there were 25% more URM trainees than non-URM trainees in our programs, we focus on the rate of non-completion among the two groups rather than the raw number of students. Group comparisons revealed a greater percentage of URM participants did not complete the Scholars Program (18.69% of URMs vs. 14.47% of non-URMs). The difference in non-completion rate between URM and non-URM students was 4%, a relatively small gap, but we recognize that there is room for program improvements to help retain URM students. In terms of gender, the non-completion rates for male and female participants were similar. Disciplinary comparisons revealed that the non-completion rate for the behavioral science majors was much lower than those for their biomedical counterparts, which is supported by research suggesting that students in biomedical fields may often face unique barriers towards persistence and retention.

Table 4.

Reasons for non-Completion of the Scholars Program

Scholars Program (N=31)
			URM Status				Discipline				Gender
	Top Reasons		URM		Non-URM		Behavioral		Biomedical		Male		Female
Program Completion		N	n	% (n/107)	n	% (n/76)	n	% (n/80)	n	% (n/103)	n	% (n/65)	n	% (n/117)
Did not Complete		31	20	18.7	11	14.5	11	13.8	20	19.4	12	18.5	18	15.4
Top Reasons		N	n	% (n/20)	n	% (n/11)	n	% (n/11)	n	% (n/20)	n	% (n/12)	n	% (n/18)
	Personal Reasons	16	11	55.0	5	45.5	5	45.5	11	55.0	8	66.7	7	38.9
	Academic Challenges	9	4	20.0	5	45.5	2	18.2	7	35.0	2	16.7	7	38.9
	Change in Career Goals	3	3	15.0	0	0	2	18.2	1	5.0	1	8.3	2	11.1
	Unknown	3	2	10.0	1	9	2	18.2	1	5.0	1	8.3	2	11.1

Note: The data for one participant were omitted from the Gender analysis. This accounts for the discrepancy in overall numbers in this category.

We also include the frequency for the top reasons that students reported for leaving the Scholars program in Table 4. The most common reason trainees reported for leaving was related to personal circumstances, with most being health-related (e.g., medical leave or caring for sick family members). Academic challenges included having a low GPA in the learning community or, more commonly, in their degree program was the second most common reason. Fewer than 5 Scholars reported financial difficulties (categorized as a personal reason) or poor fit with program/faculty mentor as reasons for leaving (categorized as academic challenge). Change in professional career goals included the desire to pursue other careers (e.g., applying to medical school rather than doctoral programs). While a departure from a research career does not meet our hallmark goals, we recognize that clarification of career goals overall is an important outcome for students.

Summary.

Given that a major goal of BUILD was to support a diverse group of student researchers in health-related disciplines at CSULB, we wanted to know whether BUILD attracted, selected and retained a diverse group of students that addressed the disparities among health science disciplines. Our data show that BUILD was successful in the recruitment and selection of a diverse group of student trainees and, more importantly for our program’s goal of increasing disciplinary diversity, the outreach and recruitment efforts improved over time in attracting behavioral researchers. In general, URM representation in participants was greater than non-URM for all four cohorts. Moreover, the percent of URM participants in our programs increased over the four cohort years, reaching a high of 68% in the 2018-2019 academic year. The selection of more diverse trainees can be attributed to changes in the selection process, where a greater weight was placed on the required diversity statement in the student applications. Furthermore, a majority of trainees were women, which represents an important increase in diversity for certain disciplines such as chemistry and engineering.

In terms of reasons for applying to the program, the top motivators for students who joined the Scholars Program included those that we would expect from research trainees in general, such as wanting to explore their career interest in science and research, to be intellectually challenged, and to enhance their resume (or Curriculum Vitae), which would make them more competitive for post-baccalaureate opportunities such as graduate school. The non-completion rate for the Scholars Programs was about 5% less than the national attrition rates in science and engineering majors (Trapani & Hale, 2019); however, the non-completion rates were higher for the URM, male, and biomedical trainees. More than half of the student attrition was due to person reasons, including health-related departures. Moreover, some students struggled with their academic performance, which may reflect institutional barriers that could be mitigated through additional academic support programs.

How Effective Was the Scholars Program?

The primary training goal of the Scholars Program was to provide an intensive research experience in health-related research to upper division undergraduate students and foster their professional development as a budding researcher. Recall that the two-year Scholars Program began with the 8-week SURGE component, during which the Scholars participated in a 3-hour learning community twice a week and conducted 30+ hours of research weekly with their respective faculty mentor. During the first academic year, they continued with the weekly 1-unit course that focused on developing foundational research skills such as scientific research communication and preparing off-campus SRE applications for their second summer in the Scholars Program. The second year began with the summer research experience/internship, either off campus or at CSULB (i.e., SURGE 2), and continued with more advanced research activities throughout the year that culminated in research presentations at professional conferences for most Scholars and even research publications for some. The learning community during the second year was primarily devoted to graduate school application preparation and process (e.g., interviewing and unpacking the financial aid package).

In this section we examined the second research question, How effective was the Scholars Program? by answering four sub-questions:

1. Did the BUILD programmatic requirements affect the Scholars’ academic performance?

2. What were the areas of growth for Scholars over the two years in the training program in terms of scientific efficacy?

3. What were the intermediate and final training outcomes of the Scholars Program?

4. How well did the Scholars Program serve as a pipeline to graduate programs in health-related disciplines?

The first two sub-questions allowed us to evaluate how the intensive research training and programmatic requirements of the Scholars Program impacted students’ educational endowments in terms of academic performance in general and development of scientific efficacy in particular. For the third sub-question, we defined intermediate outcomes as placed in an off-campus SRE after completing the 1^st year of the Scholars Program, and final training outcomes as the number of research presentations and publications produced over their two years in the program. Finally, in terms of assessing how the program served as a pipeline to graduate programs in health-related disciplines, we examined program acceptance by Discipline, URM Status, and Gender, and survey items most associated with likelihood of graduate school acceptance.

Methods.

Data Sources and Limitations.

The Scholars’ data were drawn from the BUILD program data and evaluation data collected by the Center for Evaluation and Educational Effectiveness (CEEE). The analytical sample excluded 31 trainees who left the Scholars Program before completing the full two years of training (descriptions of these trainees are presented in Section 2). In addition, to ensure that trainees were not identifiable, we excluded trainees from certain analyses when their group sample size was less than 5.

Measures.

Research Understanding and Skills

Scholars’ growth as a researcher was assessed at Pre- and Post-SURGE with five items from the survey of Student Perception of Skills and Knowledge for Academic and Research Success (Enriquez, et al., 2015) and at the end of each academic year with five items from the URSSA (Weston & Laursen, 2015). Understanding research process was measured with “I understand the research process in my field” Pre- and Post-SURGE and with “Understanding what everyday research works like” at the end of Year 1 and Year 2.

To account for the differences in survey questions used during the summer and academic year evaluation surveys, we used inter-rater consensus building to identify comparable items. Using this method, we identified from the two measures four foundational research skills that were (a) comparable in types of understanding and skills the items measured and (b) relevant to all behavioral and biomedical disciplines. Ability to read and understand journal articles was measured with “I have an ability to read and understand primary literature” at Pre- and Post-SURGE and with “Understanding journal articles” at the end (Spring semester) of Years 1 and 2. Data analyses and/or statistical skills was measured with “I have the ability to analyze data and other information” at Pre- and Post-SURGE and with “Using statistics to analyze data” at the end of Years 1 and 2. Oral presentation skills was measured with “I have skill in how to give an effective oral presentation” at Pre- and Post-SURGE and with “Making oral presentation” at the end of Years 1 and 2. Finally, scientific writing skills was measured with “I have skill in science writing” at Pre- and Post-SURGE and with “Writing scientific reports or papers” at the end of Years 1 and 2. All of these items were rated on a 5-point Likert scale, 1 indicating “strongly disagree” and 5 “strongly agree”.

Participation in Summer Research Experience

Scholars’ participation in an SRE between their first and second year in the program was coded as off-campus at an R1 institution or industry setting vs. on-campus at CSULB (i.e., SURGE2) (Off-campus = 1, On-campus = 0).

Research Productivity

Trainees’ research productivity during their participation in the Scholars Program was measured with two indicators: (a) total number of professional conference presentations and (b) having authored/co-authored any research publications. Total number of professional conference presentations was constructed as an ordinal variable, coded as “1”, “2”, “3”, “4”, or “5 or more” presentations. Given the small number of publications, authoring/co-authoring research publications was coded as a dichotomous variable (Yes = 1, No = 0).

Graduate School Acceptance

Scholars’ graduate school acceptance was coded in three ways: (a) whether Scholars were accepted into any graduate program; (b) whether Scholars were accepted into a master’s program, and (c) whether Scholars were accepted into a doctoral program. All three acceptance variables were coded as dichotomous variables (Yes = 1, No = 0).

Procedure.

The program and evaluation data were gathered at various points throughout the Scholars Program. Informed consent was obtained from each trainee to use their program and evaluation data for research dissemination. As described in Section 2, trainees’ demographic and background information was gathered from their applications to the BUILD Program. Cumulative GPAs were collected from transcripts for the semester prior to beginning the Scholars Program and at the end of each semester throughout the training program. Trainees’ research activities and training outcomes (i.e., SRE completion, conference presentations, research publications, and acceptance to graduate programs) were collected twice a year. Learning community evaluation surveys were administered by CEEE at the beginning and end of each summer program (SURGE) and at the end of each year during the learning community courses.

Results and Discussion.

Disciplinary and Demographic Characteristics of the Scholars Sample.

Data included in this analysis were from cohorts who started the Scholars Program in the 2015-2016, 2016-2017, 2017-2018, and 2018-2019 academic years. Of the total sample of Scholars (N = 152), 45% of Scholars (n = 69) majored in behavioral sciences and 55% in biomedical fields (n = 83). In terms of demographic characteristics, 57% identified as URM (n = 87) and 43% identified as non-URM (n = 65), 65% (n = 99) identified as female and 35% as male (n = 53), and a majority (73%; n = 111) were eligible for financial aid. Only 45% (n = 69) of Scholars were classified as being first generation. See Table 5 for the complete breakdown of Scholars’ characteristics.

Table 5.

Demographics of the Analytic Sample of Scholars

Disciplinary and Demographic Information	Scholars Program N=152
	n	%
Discipline
Behavioral Sciences	69	45.4
Biomedical Sciences	83	54.6
URM Status
URM	87	57.2
Non-URM	65	42.8
Gender
Male	53	34.9
Female	99	65.1
First-Generation Status
Yes	69	45.4
No	83	54.6
Financial Aid Eligibility
Yes	111	73.0
No	27	17.8
Unsure	14	9.2
Transfer Student Status
Transfer	50	32.9
Non-Transfer	102	67.1

Impact of BUILD Programmatic Requirements on Scholars’ Academic Performance.

We compared the Scholars’ GPAs at the beginning and completion of the Scholars Program to evaluate the potential burden of participating in an intensive research training program. In addition, we compared whether any impact varied by Discipline, URM Status, or Gender. Scholars’ GPAs were submitted to a 2 (Time: GPA at the beginning vs. GPA at completion of Scholars Program) x 2 (Discipline: Behavioral vs. Biomedical Majors) x 2 (URM Status: non-URM vs. URM) x 2 (Gender: Male vs. Female) mixed ANOVA. Time was the within-subjects factor and Discipline, URM Status, and Gender were between-subjects factors. Overall, the Scholars’ GPAs increased slightly, but significantly, from the start until the completion of the BUILD Program (M = 3.46 at beginning and 3.49 at the completion of the Scholars Program, F(1,144) = 4.59, p = 0.034, ${\eta }^2=.031$). In addition, the main effects of Discipline, F(1,144) = 17.65, p < 0.001, ${\eta }^2=.109$, and URM Status, F(1,144) = 8.65, p = 0.004, ${\eta }^2=.057$, were significant. GPAs were higher overall for students in the behavioral majors (M = 3.60, SEM = .05) than those in the biomedical ones (M = 3.36, SEM = .03) and for non-URM students (M = 3.56, SEM = .04) than URM students (M = 3.40, SEM = .04). No other main effects or interactions were significant. These findings indicate that Scholars Program was able to enhance the Scholars’ scientific efficacy through intensive research training without hurting other educational endowments.

Research Growth for Scholars.

The Scholars were evaluated for gains in scientific efficacy in terms of research understanding and skills at four time points: (a) before SURGE; (b) after SURGE; (c) end of Year 1 of Scholars Program; and (d) end of Year 2 of Scholars Program. The aim of the Scholars Program was to strengthen the Scholars’ understanding of the role of research in science and develop foundational research skills through their research activities with their research faculty mentors and learning community activities with BUILD Training Directors and Graduate Mentors. We examined their self-rated growth in research understanding and skills as a function of Time (i.e., length of participation in the Scholars Program) and trainees’ Discipline, URM Status, and Gender. Because of the ordinal nature of Likert-like scales, we also report a non-parametric (Friedman) statistic for the effect of Time.

Understanding of Research Process in the Field

For the items “I understand the research process in my field” (Pre- and Post-SURGE) or “Understanding what everyday research work is like” (end of Years 1 and 2), there was a significant effect of Time, F(3,177) = 8.13, p < .001 (Friedman’s ${\chi }^2(3)=28.48$, p < .001; see Figure 4). Bonferroni pairwise comparisons showed an increase in understanding of their field’s research process from Pre-SURGE (baseline) and all subsequent data points (Post-SURGE, end of Years 1 and 2). The remaining pairwise comparisons were not significant, indicating that trainees had the most gain in understanding their field’s research career process during SURGE.

Figure 4.

Growth of Scholar’s Understanding of the Research Process.

Ability to Read and Understand Journal Articles

For the items “I have an ability to read and understand primary literature” (Pre- and Post-SURGE) or “Understanding journal articles” (End of Scholars Years 1 and 2), there was a significant effect of Time, F(3,180) = 2.85, p = .039 (Friedman’s ${\chi }^2(3)=10.70$, p = .013; see Figure 5). However, Bonferroni pairwise comparisons indicated no significant changes in ratings across pairs of time points. This finding likely indicates general fluctuations in trainee’s confidence in their ability to read and understand research articles, in general, due to the types of articles they may be reading at the time of the assessments.

Figure 5.

Growth of Scholar’s Ability to Read and Understand Journal Articles.

Data Analysis and/or Statistical Skills

For the items “I have the ability to analyze data and other information” (Pre- and Post-SURGE) or “Using statistics to analyze data” (End of Scholars Year 1 and 2), there was a significant effect of Time, F(3,159) = 5.42, p < .003 (Friedman’s ${\chi }^2(3)=11.01$, p = .012; see Figure 6). Bonferroni pairwise comparisons indicated that there was no change in trainee’s ratings of their ability to analyze data from Pre-SURGE (baseline) to Post-SURGE, but there was a decrease in ratings from Pre-SURGE to Spring of Year 1 Scholars. A similar decrease in ratings between Post-SURGE and End of Year 1 Scholars was also significant. None of the other comparisons was significant, indicating that trainees may have over-estimated their data analysis skills at the start of the Scholars Program, then realized they had much to learn during the program, but returned to baseline confidence levels by the end of the Scholars Program.

Figure 6.

Growth of Scholar’s Data Analysis Skills.

Oral Presentation Skills

For the items “I have skills in how to give an effective oral presentation” (Pre- and Post-SURGE) or “Making oral presentations” (End of Scholars Years 1 and 2), there was a significant effect of Time, F(3,183) = 20.48, p < .001 (Friedman’s ${\chi }^2(3)=48.22$, p < .001; see Figure 7). Bonferroni pairwise comparisons showed an increase in oral presentation skills from Pre-SURGE (baseline) and all subsequent data points (Post-SURGE, End of Scholars Years 1 and 2). The difference between Post-SURGE and end of Year 1 was not significant, but the increase from Post-SURGE to the end of Year 2 was significant. The difference from the end of Year 1 and Year 2 was significant. Both increases in confidence coincide with the research presentation activities during SURGE (Elevator Speech Contest and Summer Symposium) and Year 2 (professional conferences). In sum, the trainees showed an increase in their oral presentation skills during their summer research training and from Year 1 to Year 2.

Figure 7.

Growth of Scholar’s Oral Presentation Skills.

Scientific Writing Skills

For the items “I have skill in science writing” (Pre- and Post-SURGE) or “Writing scientific reports or papers” (End of Scholars Years 1 and 2), there was a significant effect of Time, F(3,183) = 19.49, p < .001 (Friedman’s ${\chi }^2(3)=51.22$, p < .001; see Figure 8). Bonferroni pairwise comparisons showed an increase in writing skills from Pre-SURGE (baseline) and all subsequent data points (Post-SURGE, Spring of Year 1 Year 2 Scholars). The difference between Post-SURGE and End of Year 1 Scholars was not significant, but the increase from Post-SURGE to the End of Year 2 Scholars was significant. The difference from the End of Year 1 and End of Year 2 of Scholars was not significant. The growth in scientific writing skills coincides with the BUILD learning community activities which require a great deal of writing about their research and the program requirement of the writing-intensive Scientific Research Communications course during the academic year. In sum, the trainees showed improvements in their writing skills during SURGE and over the full two-year Scholars Program.

Figure 8.

Growth of Scholar’s Scientific Writing Skills.

Off-Campus SRE Participation and Research Productivity.

Scholars’ progress and success in the program was assessed in terms of their participation at an off-campus SRE and number of research presentations at a professional conference and research publications they produced during their two-year participation in the Scholars Program. Scientific productivity has been shown to predict science identity through the development of scientific efficacy (Cameron et al., 2020). We examined these intermediate and final program outcomes as a function of the trainees’ Discipline, URM Status, and Gender (see Table 6).

Table 6.

Scholars RI SRE Participation and Research Productivity

				Discipline				URM Status				Gender
SRE Participation and Research Productivity	Scale	Overall		Behavioral Sciences		Biomedical Sciences		URM		Non-URM		Male		Female
		(N=152)		(n=69)		(n=83)		(n=87)		(n=65)		(n=53)		(n=99)
		N	%	N	%	N	%	N	%	N	%	N	%	N	%
Participation in Off-Campus Summer Research Experience (N=152)	Yes	94	61.2	42	60.9	52	62.7	55	63.22	39	60	29	54.7	65	65.7
	No	58	38.8	27	39.1	31	37.3	32	36.78	26	40	24	45.3	34	34.3
	Subtotal	152	100	69	100	83	100	87	100	65	100	53	100	99	100
Number of Off-Campus Presentations (N=152)	0	5	3.29	3	4.35	2	2.41	3	3.45	2	3.1	1	1.9	4	4
	1	22	14.47	7	10.14	15	18.07	15	17.24	7	10.8	8	15.1	14	14.1
	2	32	21.05	11	15.94	21	25.3	17	19.54	15	23.1	15	28.3	17	17.2
	3	32	21.05	12	17.39	20	24.1	18	20.69	14	21.5	12	22.6	20	20.2
	4	25	16.45	11	15.94	14	16.87	12	13.79	13	20	9	17	16	16.2
	5+	36	23.68	25	36.23	11	13.25	22	25.29	14	21.5	8	15.1	28	28.3
	Subtotal	152	100	69	100	83	100	87	100	65	100	53	100	99	100
Published (N=152)	Yes	34	22.4	17	24.6	17	20.5	18	20.7	16	24.6	13	24.5	21	21.2
	No	118	77.6	52	75.4	66	79.5	69	79.3	49	75.4	40	75.5	78	78.8
	Subtotal	152	100	69	100	83	100	87	100	65	100	53	100	99	100

Participation in SRE

A little more than 60% of Scholars participated in an SRE at an R1 institution or in industry. A logistic regression analysis was performed to determine the effects of Discipline, URM Status, and Gender on the likelihood that the Scholars would participate in an off-campus SRE. The logistic regression model was not statistically significant, $\chi 2(3)=2.38$, p = 0.497. The results showed no evidence that Discipline, URM Status, or Gender were associated with the likelihood of a Scholar participating in an off-campus SRE.

Number of Presentations

Overall, 96.7% of Scholars made at least one research presentation off campus during their time in the Scholars Program. The mean and median number of presentations was 3 for the Scholars. The number of presentations (0, 1, 2, 3, 4, or 5+) was submitted to a univariate ANOVA with Discipline, URM Status, and Gender as between-subjects factors. The only effect to approach statistical significance was the main effect of Discipline, F(1,44) = 3.42, p = 0.066, ${\eta }^2=.023$, as students in the behavioral disciplines (M = 3.27, SEM = .228) tended to have more off-campus research presentations than students in the biomedical disciplines (M = 2.75, SEM = .163).

Number of Publications

Approximately 22% of the Scholars published one or more papers based on the research they conducted in the program. A logistic regression was performed to determine the effects of Discipline, URM Status, and Gender on the likelihood that the Scholars would produce one or more publications. The logistic regression model was not statistically significant, $\chi 2(3)=1.23$, p = 0.746. The results showed no evidence that Discipline, URM Status, or Gender influenced the likelihood of whether a Scholar would produce one or more publications.

Acceptance to Graduate Schools.

To evaluate how well the Scholars Program served as a pipeline to graduate programs in health-related disciplines, we examined the acceptance rates of Scholars in three ways: (a) any graduate school acceptance, (b) master’s program acceptance, and (c) doctoral program acceptance. Table 7 presents the overall Scholars’ graduate school application and acceptance rates for application attempts as of Spring 2021 and the breakdown by their Discipline, URM Status, and Gender. Using hierarchical logistic regression analysis, we also tested the significance of the association of research experience and research productivity (step 1: off-campus SRE placement, number of conference presentations, publication status, and Cumulative GPA) and trainee characteristics (step 2: Discipline, URM Status, and Gender) with likelihood of graduate school acceptance, separately for overall, master’s program, and doctoral program acceptance (see Appendix B). The hierarchical model allowed us to determine whether any trainee characteristics are associated with the likelihood of graduate school acceptance above and beyond trainees’ research experience and productivity.

Table 7.

Scholars Graduate School Application and Outcomes

				Academic Discipline				URM Status				Gender
Final Outcomes	Scale	Overall		Behavioral Sciences		Biomedical Sciences		URM		Non- URM		Male		Female
		(N=152)		(n=69)		(n=83)		(n=87)		(n=65)		(n=53)		(n=99)
Graduate School Application		N	%	N	%	N	%	N	%	N	%	N	%	N	%
	Applied	121	79.6	57	82.6	64	77.1	70	80.5	51	78.5	42	79.3	79	79.8
	Did Not Apply	14	7.9	5	7.3	9	10.8	9	10.3	5	7.7	5	9.4	9	9.1
	Subtotal	135	88.8	62	89.9	73	88.0	79	90.8	56	86.2	47	88.7	88	88.9
	Missing	17	11.2	7	10.1	10	12.0	8	9.2	9	13.8	6	11.3	11	11.1
Graduate School Acceptance	Accepted	109	71.7	55	79.7	54	65.1	63	72.4	46	70.8	35	66.0	74	74.7
	Not Accepted	12	7.9	2	2.9	10	12.1	7	8.1	5	7.7	7	13.2	5	5.1
	Subtotal	121	79.6	57	82.6	64	77.2	70	80.5	51	78.5	42	79.2	79	79.8
Master's Program Acceptance	Yes	62	40.8	39	56.5	23	27.7	31	35.6	31	47.7	19	35.9	43	43.4
	No	14	9.2	6	8.7	8	9.6	11	12.6	3	4.6	3	5.7	11	11.1
	Subtotal	76	50.0	45	65.2	31	37.3	42	48.3	34	52.3	22	41.5	54	54.5
Doctoral Program Acceptance	Yes	71	46.7	31	44.9	40	48.2	43	49.4	28	43.1	26	49.1	45	45.5
	No	34	22.4	16	23.2	18	21.7	20	23.0	14	21.5	11	20.8	23	23.2
	Subtotal	105	69.1	47	68.1	58	69.9	63	60.0	42	64.6	37	69.8	68	68.7

Note: The doctoral and master’s acceptance rates are not mutually exclusive. Some Scholars were accepted to both programs. Also, while not included in the table count, there were Scholars who were not accepted into a graduate program when they initially applied, but later re-applied and were accepted.

Any Graduate School Acceptance

Overall, 71% of the 152 Scholars who completed the training program were accepted to a graduate program. This percentage is similar to the 70% reported for the MARC U*STAR program between 2001-2005 (Hall et al., 2016). Of those who applied to any graduate program (n = 121), 90% were admitted. The hierarchical logistic regression analysis on the 121 trainees who applied to any graduate program showed that the regression model at step 1 with off-campus SRE placement, number of off-campus research presentations, publication status, and end cumulative GPA was statistically significant for the overall model fit, $\chi 2(4)=12.135$, p = .016; Nagelkerke R Square was .20 and the overall correct classification was 90.9%. Of the four trainee variables, only the end cumulative GPA variable was statistically significant. The odds ratio for the end cumulative GPA coefficient was 12.744 (p = .018) with a 95% confidence interval of [1.543, 105.247], confirming that higher cumulative GPA was associated with much greater likelihood of students’ graduate school acceptance. Off-campus SRE participation approached significance with an odds ratio of 3.546 (p = .065, 95% confidence interval of [0.926, 13.582], suggesting that students who attended an off-campus SRE were 3.5 times more likely to be accepted to graduate school. The model at step 2 that included the student characteristic variables of Discipline, URM Status, and Gender did not significantly increase the overall model fit, $\chi 2(3)=3.89$, p = .274.

Acceptance to Master’s Programs

Overall, 41% of the 152 Scholars that completed the training program were accepted into a master’s program. Of those who applied to a master’s program (n = 76), 83% were admitted. The hierarchical logistic regression analysis on the 76 trainees who applied to a master’s program showed that the regression model at step 1 that included the variables of off-campus R1 SRE placement, number of research presentations, publication status and end cumulative GPA was not statistically significant, $\chi 2(4)=.848$, p = .932. The model at step 2 that included the additional variables of trainee characteristics significantly improved the model fit, $\chi 2(3)=12.32$, p = .003, but the overall model was still not statistically significant, $\chi 2(7)=13.172$, p = .068.

Acceptance to Doctoral Programs

Overall, 46% of the 152 Scholars were accepted into a doctoral program. There were a few students who pursued a doctoral program in medicine or physical therapy, but the majority pursued a Ph.D. program. Our percentage is favorable when compared to national acceptance rates for doctoral programs, which was only 23.3% (Okahana, Zhou & Gao, 2020). Of the 105 Scholars who applied to a doctoral program (n=105), 67.6% were accepted. The hierarchical logistic regression analysis on the 105 trainees showed that the regression model at step 1 that included the variables of off-campus R1 SRE placement, number of research presentations, publication status and cumulative GPA was not statistically significant, $\chi 2(4)=5.092$, p = .278. The model at step 2 that included trainee characteristics variables did not improve the model fit, and the overall model was not significant, $\chi 2(7)=7.729$, p = .357.

Summary.

For the second research question, we sought to examine the productivity and outcomes of trainees in the Scholars Program and had four sub questions of interest. The first question was, did the BUILD programmatic requirements affect the Scholars’ academic performance? Although the Scholars Program was an intensive, research training program requiring a significant time commitment from students, the analysis of trainees’ GPAs showed that participating in the program did not negatively impact their GPAs over time. Thus, the time-intensive Scholars Program was able to enhance the Scholars’ scientific efficacy without hurting other educational endowments.

The second question was, in what areas did Scholars show growth as a researcher during the two years in the training program? According to our survey data, scholars showed growth in their scientific efficacy, both their understanding of a research career and in the skills that are necessary for conducting research (i.e., writing skills, oral presentation skills, data analytical skills, and the ability to understand research articles). The timing of the growth in specific areas tended to coincide with the timing of the research and training activities and experiences implemented, providing evidence for their intended outcomes. Fortunately, these gains were generally similar for all trainees across Discipline, URM Status and Gender.

The third question was, what were the intermediate and final outcomes of the Scholars Program? Scholars engaged in several activities that aimed to enhance their graduate school application and be more competitive for admission into graduate school, particularly for doctoral programs. For example, more than 60% of BUILD Scholars participated in an SRE at an off-campus R1 institution or in industry. Participating in an SRE is a great way to enhance scientific socialization and network development by becoming familiarized with a doctoral-granting institution and the graduate school environment and having the opportunity to get a letter of recommendation from a faculty mentor or researcher at another institution. For students who applied to a Ph.D., we found that 9% entered a graduate program from the SRE institution. For those that go to a location far from home, it can also boost their self-confidence in being able to move to a distant location for graduate school and possibly preview how family expectations may shift as a result of these moves. Our results also showed that the SRE opportunity was made equally accessible to our BUILD trainees, in general. Note that the remaining 40% of Scholars who did not attend an off-campus SRE continued their research at CSULB over that summer which provided similar summer experiences consisting of professional development training in graduate school application preparation and faculty mentored research experience.

In terms of presentations and publications, 97% of BUILD Scholars made at least one research presentation off campus (median number of presentations was 3), and about 22% of the Scholars published a paper based on the research they conducted in the program. These are strong indicators of research productivity and their potential for success in graduate school (see Cameron et al., 2020). Moreover, authorship on publications for 1 out of 5 Scholars indicates that the trainees are receiving a high-level of research training and that the research they produce makes substantive contributions to the literature in their areas. We believe that this authorship is also indicative of the collaborative and supportive nature of the research faculty mentors who aim to demystify the publication process for these budding researchers. These findings did not differ by students’ URM Status or Gender. Students’ Discipline was associated with the number of conference presentations they gave, with students in the behavioral disciplines giving more conference presentations than their biomedical counterparts. However, Discipline was not associated with number of publications. Here we must acknowledge that the two-year duration of the Scholars Program may have also supported students with such notable research productivity, as it takes time and resources to apply to off-campus SREs and produce research for presentations and publications. These findings highlight that it is not only about the exposure of research, but the duration and quality of the research experience that can yield the best outcomes for students. Our results suggest that the two-year Scholars Program was highly beneficial to a diverse group of student trainees.

The final question was, how well did the Scholars Program serve as a pipeline to graduate programs in health-related disciplines? Nearly 80% of the Scholars applied to master’s and/or doctoral programs in line with BUILD Program requirements. Overall, 71% of the Scholars who completed the training program were accepted to a graduate program, with 46% accepted to a Ph.D. program. Of those who applied to a master’s program, 83% were admitted, and of those who applied to a doctoral program, 67% were admitted. We are proud to say that these acceptances rates far exceed the national acceptance rates for master’s and doctoral programs from 2009-2019, which were 52.4% and 23.3%, respectively (Okahana, Zhou & Gao, 2020). Our percentage for acceptance to a Ph.D. program for all program participants of 46% is less than the 59% reported for the MARC U*STAR alumni between 2001-2005 (Hall et al., 2016). Note, though, that the MARC U*STAR program provides a broader definition of doctoral programs, which include not only Ph.D. programs but also professional/clinical doctorates. Additionally, the MARC U*STAR program has smaller cohorts and may have more stringent admissions requirements (e.g., on our campus, only 4-6 students are admitted each year and those students are required to participate in an honor’s program to be eligible). BUILD admitted students in large cohorts of 42-49 students per year and did not employ as stringent academic standards for admission. In addition, students who matriculated to doctoral programs after a master’s program are also included as part of the MARC U*STAR program, whereas our alumni are too recent to have this number included.

Our data showed that academic performance measured as cumulative GPA at the time of BUILD completion or graduation, and possibly having completed an off-campus SRE at an R1 institution, are associated with graduate program acceptance. One reason for why presentations and publications were not more predictive is that almost all BUILD scholars had given conference presentations and only a small number had publications. None of the trainee characteristics, their research experience, or productivity was associated with acceptance to a master’s or doctoral program, but we need to keep in mind the much smaller sample sizes for the master’s and doctoral program acceptance analyses which reduced their statistical power. While these results highlight the importance of academic performance and possibly off-campus SRE, they also indicate that our BUILD Scholars Program was successful in supporting a diverse group of students for acceptance into graduate school including students in non-traditional STEM disciplines, URM students, and women.

Conclusions and Implications

The findings from the first phase of the BUILD Program (2014-2019) demonstrate that the Scholars Program was successful in preparing students to enter graduate school. It further helped to broaden access to undergraduate research at CSULB by expanding the number of fields and disciplines that are traditionally considered biomedical science. The inclusion of a more diverse array of disciplinary approaches will allow for a more complex and critical examination of health disparities in research. We attribute our success to employing best practices for a research training program that includes programmatic mentoring, assets-based and cohort-based training, financial and educational resources, and inclusion of family members. These components work together interactively as part of the Scholars Program to promote student success.

We encourage other institutions to use our Scholars Program as a model for research training as its program elements are based on best practices that can be adopted to any institution that is committed to providing resources for its implementation. We recognize that the costs associated with the program are a barrier for wide implementation without adequate funding. The cost of the Scholars was about $20-25K per student per year. This cost estimate includes direct support to students (i.e., stipend/hourly pay, research supplies, and travel), but does not account for the indirect support (e.g., Program director/staff salaries, speaker fees, event costs) that is critical for a successful operation of a research training program. Thus, universities that wish to implement these programs without sufficient internal funds to cover the costs should anticipate leveraging their existing campus research partnerships, staff expertise, and local networks, in addition to securing funds from external sources such as federal and state agencies, industry, foundations, or private donations.

At CSULB, we are currently evaluating the specific programmatic components of the Scholars Program in the second phase of BUILD (2019-2024) to determine whether similar outcomes can be obtained in a more economical manner, as the external funding for this program ends in 2024. Specifically, we are currently collecting data on a 1-year Fellows Program to compare to the 2-year Scholars Program to determine whether the program goals can be met with a 1-year rather than 2-year program. If so, then the number of students impacted by the program can be increased by our ability to train more students at the same cost. Moreover, we have examined whether components of the program can be implemented through online modules to reduce the cost of administering the program. So far, we have found that some components of the program, such as content relating to applying to graduate schools and summer research experiences, can be successfully implemented as online modules (Vu et al., 2021). We also found that the use of online mentor training modules, with facilitated discussions, is an effective and cost-efficient way to formalize mentor training on campus for both faculty and staff (Young et al., in press). Finally, we are exploring a non-degree, research certificate option where students do not need to be enrolled in a formal research training program but engage in faculty-mentored research, complete online professional development modules, and take research-focused courses in their majors.

Overall, we are hopeful that others interested in implementing a similar model of a structured undergraduate research program learn from our practices in broadening and diversifying the biomedical research enterprise. It is important to continue to increase access to a larger number of students to research opportunities, and equally as important, to diversify and be intentional about the demographic representation of student participants. Our Scholars Program builds upon best practices of programmatic mentoring, assets-based and cohort-based training, financial and educational support, inclusion of family members, and intensive research training by faculty in the students’ disciplines. Through the description of our programming and illustration of key intermediate and final outcomes, we demonstrate how these markers, when combined, create the necessary supports to engage our students and ultimately promote their persistence and retention in biomedical and health related majors and subsequently, towards research careers.

Appendix A. BUILD Participant Majors by Discipline and College

Discipline/College	Majors
Behavioral Sciences
College of Health and Human Services (CHHS)
	Family & Consumer Sciences Health Care Administration Health Science Kinesiology Nutrition & Dietetics Speech-Language Pathology
College of Liberal Arts (CLA)
	Anthropology Communication Studies International Studies (Dual Track) Linguistics Psychology Political Science
Biomedical Sciences
College of Natural Sciences and Mathematics (CNSM)
	Biological Sciences Chemistry & Biochemistry Physics
College of Engineering (COE)
	Biomedical Engineering Chemical Engineering Civil Engineering Electrical Engineering Mechanical Engineering

Appendix B. Predictors for Graduate Program Acceptance

Dependent Variable	Variables	Model 1			Model 2
		B	OR	P	B	OR	p
(All) Graduate School Acceptance	SRE Off-Campus	1.266	3.546	0.065	1.410	4.097	0.053
	Off-Campus Presentations	−0.262	0.769	0.288	−0.323	0.724	0.235
	Publications	1.565	4.781	0.156	1.297	3.660	0.245
	End GPA at Completion	2.545	12.744	0.018	1.741	5.702	0.188
	URM vs. Non-URM				0.307	1.359	0.677
	Gender				0.741	2.097	0.284
	Discipline				−1.218	0.296	0.224
Master’s Acceptance	SRE Off-Campus	0.102	1.108	0.871	0.685	1.984	0.352
	Off-Campus Presentations	−0.151	0.860	0.486	−0.303	0.739	0.272
	Publications	0.431	1.539	0.550	0.181	1.198	0.820
	End GPA at Completion	−0.048	1.049	0.960	−1.829	0.161	0.162
	URM vs. Non-URM			0.633	−2.022	0.132	0.015
	Gender				−1.392	0.248	0.122
	Discipline				−2.379	0.093	0.015
Doctoral Acceptance	SRE Off-Campus	0.393	1.482	0.395	0.355	1.426	0.458
	Off-Campus Presentations	−0.115	0.891	0.450	−0.067	0.935	0.667
	Publications	0.386	1.472	0.457	0.569	1.767	0.299
	End GPA at Completion	1.336	3.805	0.053	2.023	7.557	0.017
	URM vs. Non-URM				0.321	1.379	0.494
	Gender				−0.223	0.800	0.647
	Discipline				0.729	2.073	0.176