Fine-Tuning Summer Research Programs to Promote Underrepresented Students’ Persistence in the STEM Pathway

This study documents the impact of Leadership Alliance Summer Research Early Identification Program components on students’ research skills knowledge and career pathway planning in relation to student demographic and academic characteristics. These experiences contribute to the ongoing formation of confidence, goals, and motivational behaviors that shape academic/career pursuits.

Abstract

Although the importance of undergraduate research experiences in preparing students for graduate study and research careers is well documented, specific examination of program components is needed to assess the impact of these programs on underrepresented (UR) students. The Leadership Alliance, a consortium of leading PhD-granting and minority-serving institutions (MSIs), has leveraged its diverse partnership to place UR students from MSI and non-MSI institutions in competitive research environments through its national Summer Research Early Identification Program. Using longitudinal pre/post data collected from student surveys, we applied social cognitive career theory as a conceptual framework to examine how research engagement, skill development, and mentorship aspects of a summer research program affect students’ commitment to pursue research careers. Self-reported knowledge of research skills, time engaged in research activity, and students’ understanding of and attitudes toward pursuing graduate study were measured in relation to the classification of students’ home undergraduate institution, level of students’ pre-existing research experience, and demographic factors. Our results provide evidence of specific programmatic components that are beneficial for UR students from varying academic and cultural backgrounds. This study describes important aspects of summer research programs that will contribute to students’ ability to persist in science careers.

INTRODUCTION

Summer undergraduate research experiences have been leveraged to address the significant educational disparities across racial and ethnic groups in degree attainment in the sciences (Chang et al., 2008). In 2011, students from underrepresented (UR) backgrounds (blacks, Hispanics, and American Indians and Alaskan Natives) accounted for 17% of students enrolled in science and engineering graduate programs, compared with 65% for whites (National Science Board [NSB], 2014). National data also reveal an increase from 6% in 2000 to 8% in 2011 in the number and proportion of doctoral degrees in science and engineering fields earned by students from UR backgrounds (NSB, 2014). While this increase is welcome, it remains substantially disproportionate when compared with 43% of doctoral degrees in science and engineering fields being awarded to white U.S. citizens and permanent residents (NSB, 2014). Further, the demographics of the U.S. population are rapidly changing, such that traditionally UR minority groups are rapidly becoming the collective majority (U.S. Department of Labor, 2007). Thus, efforts to prepare students from UR backgrounds for graduate training are critical to increasing diversity in the U.S. scientific community.

The Leadership Alliance, a long-standing national partnership among minority-serving institutions (MSIs), PhD-granting institutions, and private industry, was established to train and mentor students from diverse cultural and academic backgrounds into competitive graduate training programs and professional research-based careers. The mentoring begins at the undergraduate level with the Summer Research Early Identification Program (SR-EIP), a national summer research program that exposes students to the world of research-based careers by providing hands-on research experiences and mentorship in all academic disciplines. The culminating event of the SR-EIP is the Leadership Alliance National Symposium, where ongoing professional development, mentoring, and networking occurs among a community of undergraduates, graduate students, SR-EIP alumni who have obtained doctoral degrees, faculty, and administrators.

A substantial body of literature provides evidence that summer research experiences promote the development of critical research and communication skills and help clarify career paths (Lopatto 2004a, 2007; Seymour et al., 2004). Longitudinal evaluations and outcome data of the SR-EIP demonstrate the efficacy of these programs in building students’ confidence in their research and communication skills; clarifying students’ knowledge about the steps needed to apply for and succeed in various training pathways; and exposing students to the different career paths in academia, industry, and the public and private sectors (Ghee et al., 2014). Although these data support the overall importance of a summer research experience, analysis of the specific components of the summer program is critical to understanding the impact of these activities on career clarification and research skill development.

In this article, we sought to identify program components of the SR-EIP that enhanced students’ research self-efficacy and increased students’ awareness and commitment to pursuing a research career. The following research questions were developed to help frame this study: What aspects of research skills and career pathway showed the largest pre- to postprogram change? Did particular subgroups of students show significantly larger benefits of participation based on student characteristics? What aspects of the program were significantly associated with change in research skills and career pathways? Given our interest in research self-efficacy and career pathways, we adopted elements of the social cognitive career theory (SCCT) as a conceptual framework. Developed from Bandura’s (1986) social cognitive theory, SCCT provides a framework to understand interrelated aspects of career development through three primary tenets: self-efficacy, outcome expectation, and goals (Lent et al., 1994). We posit that students’ self-efficacy or their belief in their ability to successfully perform a task is positively associated with their outcome expectations, which ultimately influences career training goals.

We present data on pre- and postsurvey measures of research skills and students’ understanding of and attitudes toward pursuing graduate study and research-based careers. We examine these changes further in relation to three contextual factors: 1) classification of students’ home undergraduate institutions, 2) level of students’ pre-existing research experiences, and 3) demographic factors such as gender, racial/ethnic status, and first generation attending college status. This study describes specific programmatic components of a summer research program that are designed to enhance students’ knowledge and skill development and increase their awareness of scientific research careers. In addition, the summer research program is designed to expose students to the graduate school research site and to enhance their understanding of the expectations of graduate study and research. The mentoring relationship is emphasized as a central aspect in developing both their awareness of the pathway to scientific research careers and knowledge and skill development.

METHODS

Data sources for this study include a longitudinal database of demographic and outcome data collected on all participants, across all SR-EIP sites. Surveys were administered to participants before and after their summer research experiences. Student registrations provide self-report data on race, ethnicity, gender, income level, first-generation status, undergraduate enrollment, and academic discipline. Tracking data for each participant includes information on educational and training activities pursued after completion of baccalaureate degree but before entry into graduate school; graduate school enrollment; graduate degree(s); postgraduate training via postdoctoral, fellowship, or residency appointments; and initial job placement. For the current study, only 17 and 31% of sample members completed their undergraduate degrees by 2014 and 2015, respectively. As a result, data on postbaccalaureate training and employment outcomes are limited at this time.

Survey Development

The Leadership Alliance evaluation activity was initiated in 2000 and involved a consortium-level committee whose charge was to outline evaluation goals and scale of the evaluation effort. Ten experts, including social science and science, technology, engineering, and mathematics (STEM) faculty, program coordinators, and evaluation specialists, convened to draft a student survey instrument focused initially on program effectiveness and implementation. This committee reviewed program-specific data-collection instruments from undergraduate programs housed at member institutions and extant instruments shared by other programs through the Council of Independent Colleges (CIC). Together, this committee drafted the initial 2001 survey instrument that was then administered as a postsurvey to students participating in summer research programs across 13 Leadership Alliance institutions. The overall purpose stated for the evaluation at that point in time was to “assess the extent to which the SR-EIP meets its goals: to provide students with research experience; and to encourage students to pursue academic careers in the sciences, engineering and humanities.” The 2001 survey explored the level of engagement of students in various facets of scientific research and with the faculty, postdocs, and graduate students who served as their mentors. The survey also examined outcomes of this engagement, both in areas of student learning over the summer and impact on students’ long-term academic and career goals.

Since that time, annual evaluation findings prepared by external evaluators were reviewed by the consortium collectively, with the Leadership Alliance’s Data Policy Committee charged to review survey instruments and administration for responsiveness to program development and improvement needs. Differentiation of results by race, ethnicity, gender, disciplinary field, and program site was subsequently incorporated into the data collection and analysis of survey results. Over time, changes to the evaluation instrument were pursued in response to emerging research findings about effective undergraduate research programs and Leadership Alliance coordinators’ inquiries related to the same. Instruments developed by the Higher Education Research Institute (2006) and David Lopatto (2004a,b, 2007, 2008) were reviewed for primary constructs, and items were adapted for use in the evaluation given alignment with program goals and intent. By 2011, the postsurvey instrument had expanded to include students’ self-assessment of knowledge development, future academic and career plans, mentoring experience, professional development activities, and program impact. Background or context information related to students’ previous and/or current academic enrichment experiences was also collected.

In 2012, a review of the Leadership Alliance postsurvey evaluation instrument was undertaken by an external evaluator for the Leadership Alliance and the Leadership Alliance’s Data Policy Committee. As a result of this review, a pre- and postsurvey strategy was initiated in 2013 to streamline data collection across students’ period of engagement with the summer program and provide more appropriate measures of preprogram versus postprogram student perspectives on knowledge development and future plans. See the Supplemental Material. As a result, items concerning preprogram communication and engagement with students’ mentors and a wider range of preprogram research experiences (e.g., course work on theories/methods, independent study, research internships at home institution) were introduced in the presurvey. The impact of mentoring and support received by others was differentiated postprogram by faculty, postdoc, and other types of mentors encountered during the summer program.

Presurvey Administration

The presurvey is administered online within the first week of arrival at the host institution. Online administration includes a unique link for each participant, which facilitates follow-up and survey response rates. Presurvey response rates for each cohort exceed 90%. The presurvey collects information on participants’ prior undergraduate research activities, either through a formal program at their home or other institutions or through informal undergraduate research learning experiences offered by their home institutions. In addition, presurvey respondents identify specific communication and preparation activities initiated by the program or their Leadership Alliance mentors and their perceptions of these activities. Specific to the aims of this study, students are asked to rate their knowledge of the research process and specific methods/approaches used in research before starting the summer program. They also indicate their current status with regard to plans for graduate school training and their careers.

Postsurvey Administration

During the last week of the summer program, students receive an invitation to the online postsurvey. Individual links are issued to assure efficient, targeted follow-up, with postsurvey response rates for each cohort exceeding 90%. In addition to collecting general perceptions of the program, the postsurvey assesses students’ satisfaction with their mentors and specific aspects of mentoring. The latter includes the availability of the mentor, the provision of constructive feedback, help with establishing relationships, introduction to a range of methodological techniques, and how well the mentor helped the student see how his or her project fit within the overall research effort. Questions repeated from the presurvey include self-report levels of knowledge about the research process and specific methods/approaches and plans for graduate school training and their careers. Differences between pre- versus postsurvey responses provide indicators of change.

Participants

The participants in this study were undergraduates from the Leadership Alliance SR-EIP in 2013 (N = 157), 2014 (N = 164), and 2015 (N = 175). While the SR-EIP engages students from all disciplines, this study is limited to analysis of participants from STEM disciplines. STEM participants make up 70–75% of each cohort and constitute a robust sample for this study. Of the 496 individual students in the full sample, 16 did not complete the preprogram survey, 24 did not complete the postprogram survey, and six completed neither survey, leaving an analytical sample of 450 respondents.

Participants represent a wide range of undergraduate student characteristics. Given the goals of the program, most were UR minorities, as shown in Table 1 (42.7% were Hispanic and 31.3% were black). Other demographic indicators outlined in Table 2 show that less than half of program participants were enrolled at minority-serving institutions (MSIs; 10.9% at historically black colleges and universities [HBCUs] and 24.9% at Hispanic-serving institutions [HSIs]). Nearly one-third (30.2%) were first-generation college students. The overwhelming majority was near the end of their undergraduate programs; 57.1% had 1.5 or fewer years until graduation, 30.9% had 1.6–2.5 years until graduation, and only 12.0% had more than 2.5 years until graduation. For most participants, this was their first participation in a formal research experience program outside their home institutions. Only 16.9% reported previously participating in a research experience program.

Table 1.

Race and ethnicity of participants (total sample = 450)

	Ethnicity
Race	Hispanic (N)	Not Hispanic (N)
Black/African American	10	141a
Asian	4	30b
Native American/Alaskan Native	16	2c
Pacific Islander/Native Hawaiian	1	0
White	76	57d
Biracial/multiracial	20	22c
Decline to indicate	65	6c
Subtotal	192e	258

^aAll classified as black/African American.

^bAll classified as Asian.

^cAll classified as other.

^dAll classified as white.

^eAll classified as Hispanic.

Table 2.

Demographic characteristics of participants

Variable	N	%
HBCU or HSI
HBCU	49	10.89
HSI	112	24.89
Neither	289	64.22
Undergraduate institution type
Baccalaureate	90	20.00
Master’s	93	20.67
Doctorate–Low Research	37	8.22
Doctorate–High Research	230	51.11
Previous summer research
Yes	76	16.89
No	374	83.11
Years until graduation
1	257	57.11
2	139	30.89
3	54	12.00
First-generation status
Yes	136	30.22
No	289	64.22
Did not indicate	25	5.56
Gender
Female	267	59.33
Male	183	40.67
Discipline area
Biological/agricultural sciences	314	69.78
Physical sciences	67	14.89
Engineering	51	11.33
Math and computer sciences	16	3.56
Other	2	0.44
Total sample	450

As expected, because undergraduates are still in an exploratory phase of identifying various career pathways and developing flexible preprofessional résumés, participants reported interest in a variety of graduate training options. The lowest level of interest was reported for planning to attend graduate school to pursue a master’s degree; only 6.0% “strongly agreed” with that statement. The highest level of interest was reported for planning to attend graduate school to pursue a PhD; 46.9% “strongly agreed” with that statement. A moderate level of interest was reported for planning to attend graduate school to pursue an MD–PhD; 23.6% “strongly agreed” with that statement.

Outcome Measures

To approximate students’ research self-efficacy, we examined survey items focused on students’ gains across several research skills. Individual elements of this indexed measure, as well as the index itself, were explored to assess how summer research activities impacted students’ self-reported sense of research self-efficacy. Pursuant to our SCCT framework, the enhanced learning experiences would also lend themselves to heightened expectations for future research training and career plans. Thus, a second outcome measure was developed to address changes in students’ career pathway plans.

Research Skills Variables

Students reported their self-rated knowledge of several aspects of conducting research, which included overall research process, literature and basic concepts in research area, and data gathering and analysis. (Item wording can be seen in Figure 1.) These ratings were obtained on both the pre- and postprogram surveys. A research skills index combining all 16 items was also calculated (index alpha = 0.89).

Figure 1.

Students’ self-assessment of their knowledge of research skills. Distribution or responses to each research skills question are presented. Each bar represents the percent that reported “intermediate” or “high” from participants who completed the pre- and postsurvey (2013–2015 summer cohorts; n = 450). Students were asked to report their level of knowledge of each item on a scale of 1 through 4 (1 = none; 2 = beginning; 3 = intermediate; 4 = high). An asterisk indicates a significant difference at p < 0.01.

Career Pathway Variables

Career pathway variables measure knowledge of steps on the pathway to and interest in research careers. Students reported the extent of their agreement with regard to the following five survey items: 1) “I have a solid understanding of the graduate school application process”; 2) “I have a solid understanding of graduate school life”; 3) “I have a solid understanding of the careers available to me in my discipline”; 4) “I plan to pursue an academic career”; and 5) “I plan to pursue a research career outside academia.” All questions were asked on both the pre- and postprogram surveys.

Program Component Measures

Leadership Alliance summer research programs include a mix of intense hands-on research experiences inclusive of formal research presentations and professional development activities. Engagement in research and professional development activities is addressed through student self-report estimates of time on task. Another program component integral to each student’s summer experience is mentorship, which is addressed here in terms of quality and actual type of mentor.

Time Use during the Summer Program

The amount of time that students spent on three categories of activities (research preparation skill development, conducting research, and professional development) is based on their reports of “how many hours each week did you typically spend on” a list of 13 activities. Time spent on social events or gatherings and self-reported other activities was not included in these analyses. Time spent on research preparation includes the following six items: lectures, formal seminars and symposia, small-group discussions, study groups, one-on-one tutorials, and library/museum work (index alpha = 0.77). Time spent conducting research includes the following four items: laboratory work, data gathering, data analysis, and preparation of research presentation. These items did not hold together as an index (index alpha = 0.52); therefore, each item was examined separately in the regression analyses. Time spent on professional development is based on students’ response to one item: “professional development activities (e.g., GRE exam preparation, financial aid for graduate school, graduate school application process, etc.).” For each item, students could choose from five response options: 20 h or more, 10–19 h, 5–9 h, <5 h, and not applicable. Not applicable is coded as having spent zero time on that activity.

Quality and Type of Mentoring Relationship

On the postprogram survey, students rated the quality of their interactions with their mentors on 10 items in response to the question “Based on the contact with your primary research mentor, how would you rate him/her on the following qualities?” (Table 3). A quality of mentoring relationship index combining all 10 items was calculated (index alpha = 0.94). Students also reported, on a four-point scale, “overall satisfaction with your primary research mentor.” The correlations between each mentoring relationship quality item and satisfaction with mentor are shown in Table 3. This shows that the close interpersonal interactions (e.g., “Showing interest in your research and being supportive of your ideas” and “Making you feel like an integral part of the research project”) had the strongest correlations with satisfaction, and broader interaction experiences (e.g., “Helping you make the distinction between undergraduate research and large-scale projects in a research university” and “Helping you establish relationships by introducing you to other faculty, graduate students and researchers”) had the weakest correlations with satisfaction.

Table 3.

Correlations with mentor satisfaction

Mentor quality variables	Satisfaction with mentor
Showing interest in your research and being supportive of your ideas	0.733
Making you feel like an integral part of the research project	0.731
Being available to discuss and respond to questions about your research	0.721
Offering guidance and advice on your research	0.710
Giving you a good sense of how your project fit within the overall goals of the project	0.715
Introducing you to a range of methodological techniques in research	0.691
Providing you with constructive feedback on your academic career development	0.604
Demonstrating knowledge/expertise in your field	0.592
Helping you make the distinction between undergraduate research and large-scale projects in a research university	0.586
Helping you establish relationships by introducing you to other faculty, graduate students and researchers	0.567

Students were asked on the postprogram survey to identify what type of individuals served as their primary research mentors. Choices included: faculty member, postdoctoral researchers, graduate student, and other. Forty-one percent reported having faculty members as their primary research mentors, with 28 and 27% reporting postdocs or graduate students, respectively, as their primary research mentors. Only 4% (n = 18) reported “other” persons as their mentors; these mentors were described by students as administrators and/or program directors.

Additional Covariates

Several factors representing student characteristics and their home undergraduate institutions were measured. Race and ethnicity were self-reported in two questions. In one question, students were asked to report their race. In a separate question, students were asked to report whether they were of Hispanic heritage. Because there were too few students who identified as both Hispanic and black (11 students), black and Asian (five students), or Hispanic and Native American (16 students) to allow for separate analysis, they were all coded as Hispanic. Additionally, 87 students who identified as Hispanic did not self-identify with a specific racial category. Essentially, all students who identified as Hispanic were coded as Hispanic, and all students who identified as not Hispanic were coded based on their self-identified racial category. The final coding includes five categories: Asian, black, Hispanic, white, multiracial, and other (see Table 1).

First-generation student status was measured by whether students reported that at least one parent had a college degree. The Carnegie Classification of Institutions of Higher Learning was used to categorize students’ home undergraduate institutions into four categories: baccalaureate granting, master’s degree granting, PhD granting–high research intensity, PhD granting–very high research intensity. Undergraduate home institutions were also classified as HBCUs or HSIs as defined by the U.S. Department of Education and the Hispanic Association of Colleges and Universities, respectively. Previous participation in an undergraduate research program was measured by students reporting prior participation in any of the following six programs: Leadership Alliance SR-EIP, Leadership Alliance Mellon Initiative, CIC Summer Research Opportunities Program, Amgen Scholars Program, Mellon Mays Undergraduate Fellowship, and Minority Access to Research Careers. Based on student surveys administered since 2007, these six programs were most often indicated when students were asked about previous research experience. Open-ended responses further indicate a number of research experiences provided by the students’ home institutions (e.g., LSAMP, RISE, HBCU-UP, HHMI, McNair, a given professor’s lab) or provided through programs at other institutions (Broad Institute, SURP, LSAMP, NASA, REUs, Jackson Laboratory, etc.).

Data Analysis

First, t tests were used to compare the post- versus preprogram values to examine whether participation in the summer research experience resulted in significant increases on each outcome measure (career pathway and research skill variables). Next, for each outcome measure, analyses of variance were used to determine whether different subgroups of students had significantly higher preprogram values; and whether different subgroups showed significantly greater postprogram change, controlling for preprogram values. These analyses focused on race/ethnicity, undergraduate institution type, first-generation student status, and whether they had a previous summer research experience. Finally, regression analyses were used to determine the programmatic impact of each summer research experience measure on each postprogram outcome measure. All regression analyses control for preprogram values on the specific postprogram outcome measure.

RESULTS

What Aspects of Research Skills and Career Pathway Showed the Largest Pre- to Postprogram Change?

Research Skills.

All research skills knowledge items evidenced significant positive postprogram change (Figure 1). The full range of the four-point scale was used to test for pre- versus postprogram change; however, to facilitate readability, Figure 1 shows responses dichotomously coded based on whether students reported “intermediate” or “high” (=1) versus “beginning” or “none” (=0) for each knowledge of research skills item. The highest baseline levels were reported for broader conceptual factors, such as overall research process (80.2%), ethical aspects of research (77.8%), critical reading skills (77.6%), and professional aspects of research (70.4%). Students reported the lowest baseline levels for knowledge of more technical aspects of both qualitative and quantitative data collection and analysis, such as statistics software (20.9%), data-mining techniques (21.6%), techniques for conducting archival research (31.6%), and techniques for conducting interviews (33.6%). Correspondingly, students reported the greatest knowledge gains on the technical aspects of research. The research skills items that revealed the highest degree of mean rating change included bibliographic software (49.8% increase), statistics software (30.2% increase), and approaches to quantitative data analysis (26.7% increase). The lowest degree of change was for critical reading skills (8.5% increase), one of the highly rated broader conceptual factors noted earlier. Two other areas evidencing low degrees of change were techniques for archival research (11.0% increase) and ethical aspects of research (12.3% increase). The small change for archival research is understandable, because no students were working on projects for which archival research techniques were the primary research methods used. Continued exploration into the scope and content of research ethics is needed to understand what programs are offering and how students perceive the issue of research ethics.

Career Pathway.

As indicated in Figure 2, all career pathway measures, with the exception of “planning to pursue a research career outside academia,” evidenced significant positive postprogram change. Again, based on analysis of the four-point scale to test for pre/postprogram change, Figure 1 shows responses dichotomously coded based on whether students reported “agree” or “strongly agree” (=1) versus “disagree” or “strongly disagree” (=0) for each career pathway measure. Presurvey respondents were most likely to agree that they understood the careers available in their discipline (72.9%) and least likely to agree that they had an understanding of graduate school life (60.4%) and understood the graduate school application process (62.0%). The majority of postsurvey respondents (96.0%) agreed that they understood the graduate school application process, understood graduate school life (91.8%), and understood the careers available in their disciplines (90.4%). Regarding career plans, at preprogram, 68.2% agreed that they plan to pursue an academic career, which increased to 70.9% postprogram. In contrast, at preprogram, 63.3% agreed that they plan to pursue a research career outside academia, and this decreased slightly to 62.4% postprogram.

Figure 2.

Students’ self-assessment of their knowledge of several career pathway indicators. Distribution or responses to each career pathway question are presented. Each bar represents the percent that “agreed” or “strongly agreed” from participants who completed the pre- and postsurvey (2013–2015 summer cohorts; n = 450). Students were asked to report the extent of their agreement with each item on a scale of 1 through 4 (1 = strongly disagree; 2 = disagree; 3 = agree; 4 = strongly agree). An asterisk indicates a significant difference at p < 0.01.

Did Particular Subgroups of Students Show Significantly Larger Benefits of Participation Based on Student Characteristics?

To determine whether there were differential impacts of participation in SR-EIP based on students’ demographic and educational characteristics, we analyzed subgroup differences of pre- and postsurvey respondents. Overall the student characteristics and undergraduate institution variables had small or no significant associations with students’ preprogram (Table 4) and postprogram (Table 5) measures. Only the factors with significant subgroup differences are shown in Tables 4 and 5.

Table 4.

Subgroup differences in preprogram measures

	Understand graduate school application process	Understand graduate school life	Understand careers available in discipline	Plan to pursue academic career	Plan to pursue research career outside academia	Research skills index
Subgroups	Mean (SD)	Mean (SD)	Mean (SD	Mean (SD	Mean (SD)	Mean (SD)
Previous summer research
Yes	3.07 (0.72)	3.01 (0.68)	3.05 (0.69)		2.57 (0.74)	2.82 (0.46)
No	2.65 (0.77)	2.60 (0.73)	2.85 (0.71)		2.77 (0.79)	2.52 (0.48)
Years until graduation
1.5 or fewer	2.85 (0.79)	2.76 (0.72)				2.64 (0.48)
1.6–2.5	2.61 (0.71)	2.64 (0.71)				2.52 (0.49)
2.6–4	2.37 (0.76)	2.33 (0.78)				2.36 (0.49)
HBCU or HSI
HSI	2.91 (0.78)	2.90 (0.72)				2.69 (0.48)
HBCU	2.67 (0.69)	2.55 (0.65)				2.54 (0.58)
Neither	2.65 (0.78)	2.60 (0.74)				2.53 (0.47)
Gender
Male	2.81 (0.79)		2.99 (0.70)	2.99 (0.81)
Female	2.66 (0.77)		2.81 (0.70)	2.81 (0.84)
Discipline areaa
Biological/agricultural sciences				2.89 (0.84)
Physical sciences				2.88 (0.73)
Engineering				2.59 (0.85)
Math and computer sciences				3.56 (0.51)

Only factors with significant associations with the research skills and career pathway variables are shown.

^aOther discipline areas excluded because there were only two students.

Table 5.

Subgroup differences in postprogram measures

	Postprogram
	Understand graduate school application process	Understand graduate school life	Understand careers available in discipline	Plan to pursue academic career	Plan to pursue research career outside academia	Research skills index
Subgroups	Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)	Mean (SD)
HBCU or HSI
HSI		3.57 (0.56)		3.29 (0.81)
HBCU		3.43 (0.58)		2.86 (0.94)
Neither		3.30 (0.67)		2.93 (0.86)
Race/ethnicity
Asian			3.43 (0.57)			3.27 (0.57)
Hispanic			3.40 (0.65)			3.14 (0.54)
Black			3.33 (0.65)			3.01 (0.60)
Multiracial or other			3.33 (0.66)			3.01 (0.39)
White			3.04 (0.73)			2.94 (0.45)

Only factors with significant associations with the research skills and career pathway variables are shown. Analyses control for preprogram values of each measure.

Previous summer research experience was significantly and positively associated with all preprogram measures, with the exception of planning to pursue an academic career. However, having had a previous summer experience was not significantly associated with any of the postprogram outcomes, controlling for the preprogram value.

Students with fewer years until graduation reported significantly higher levels of understanding of the graduate school application process, graduate school life, and research skills knowledge. However, subsequent analyses (unpublished data) found that this was due to students with fewer years until graduation having a significantly higher likelihood of having had a previous summer research experience. Twenty-five percent of students with 1.5 or fewer years until graduation had a previous summer experience, compared with 8.6% of those with 1.6–2.5 years until graduation, and 1.9% of those with 2.6 −<4 years. The number of years until graduation was not significantly associated with any of the postprogram measures, controlling for preprogram values.

Students attending HSIs reported significantly higher levels of understanding of the graduate school application process, graduate school life, and research skills index, compared with students attending HBCUs and traditional universities. However, subsequent analyses (unpublished data) found that this was due to HSI students having a significantly higher likelihood of having had a previous summer research experience. Thirty percent of students from HSIs had a previous summer experience, compared with 20.4% of those from HBCUs and 11.1% of those from traditional universities. Students from HSIs also reported the highest levels of postprogram understanding of graduate school life and plans to pursue an academic career, controlling for preprogram values. Again, this can be attributable to the fact that many of the HSI students had previously participated in Leadership Alliance Summer Programs. Students’ undergraduate discipline area was significantly associated only with preprogram plans to pursue an academic career. Students in math and computer science reported the highest desire to pursue an academic career, and those in engineering reported the lowest.

Race/ethnicity was not significantly associated with any of the preprogram measures. However, race/ethnicity was significantly associated with postprogram understanding of careers available in their discipline and the research skills knowledge index, controlling for preprogram values. White students reported the lowest levels on both measures; Asian and Hispanic students reported the highest levels on both measures, but only the Hispanic students were significantly different from the other four racial/ethnic groups. It is important to note that the Asian category contains the fewest number of students, and the lack of significance for the Asian students versus the other racial/ethnic groups may be due to a lack of statistical power.

Male students reported significantly higher preprogram levels of understanding of the graduate school application process and careers available in their disciplines and plans to pursue an academic career. Gender was not associated with any of the postprogram measures, controlling for preprogram values.

Undergraduate institution Carnegie classification and first-generation student status were not significantly associated with any of the pre- and postprogram measures.

What Aspects of the Program Were Significantly Associated with Change in Research Skills and Career Pathway?

Regression analyses show that time spent on research preparation learning activities was positively associated with significant gains on the research skills index and all career pathway variables (Table 6, row 1). Time spent on professional development activities was positively associated with significant gains on the research skills index and all career pathway variables, except plans to pursue an academic career (Table 6, row 2). Time spent on data analysis was positively associated with the research skills index and knowledge of careers available in a discipline (Table 6, row 3). Time spent gathering data was positively associated with the research skills index (Table 6, row 4). Time spent on laboratory work and on preparation of research presentation was not significantly associated with any of the dependent variables (Table 6, rows 5 and 6). Mentor quality was positively associated with significant gains on the research skills index and all career pathway variables, except plans to pursue an academic career, (Table 6, row 7). Tests for whether type of mentor (faculty member, postdoctoral researcher, graduate student, or other) mattered found no significant differences, except for a significantly lower rating on quality of mentoring for the 18 students who reported “other” as their mentor. “Other” mentors included administrators who were assigned to students before the program.

Table 6.

Association between programmatic elements and change in career pathway and research knowledge variables^a

	Understand graduate school application process	Understand graduate school life	Understand careers available in discipline	Plan to pursue academic career	Plan to pursue research career outside academia	Research skills index
Predictor variable	Beta (SE)	Beta (SE)	Beta (SE)	Beta (SE)	Beta (SE)	Beta (SE)
Time spent in preparation for research	0.10 (0.04)*	0.12 (0.05)**	0.16 (0.05)***	0.13 (0.06)**	0.18 (0.06)***	0.16 (0.04)***
Time spent in professional development	0.12 (0.03)**	0.13 (0.03)**	0.12 (0.03)**	0.02 (0.04)	0.10 (0.04)*	0.18 (0.03)***
Time spent doing data analysis	0.03 (0.02)	0.05 (0.03)	0.10 (0.03)*	0.04 (0.03)	−0.03 (0.04)	0.11 (0.02)*
Time spent gathering data	0.06 (0.02)	0.05 (0.02)	0.07 (0.02)	−0.01 (0.03)	0.02 (0.03)	0.09 (0.02)*
Time spent on laboratory work	0.09 (0.05)	0.07 (0.03)	0.08 (0.03)	0.01 (0.03)	−0.06 (0.04)	0.03 (0.02)
Time spent on preparation of research presentation	−0.03 (0.03)	−0.01 (0.03)	0.06 (0.03)	0.04 (0.04)	0.09 (0.04)	0.05 (0.03)
Mentor quality	0.11 (0.03)*	0.17 (0.04)***	0.14 (0.04)**	0.08 (0.04)	0.16 (0.05)**	0.22 (0.03)***

^aCoefficients from the regression model of impact of each aspect of the summer research experience on each career pathway variable and the research skills index. Coefficient estimates are in terms of student ratings. Because each regression model controls for the preprogram measure of the dependent variable, coefficients represent change in the outcome variable.

*p < 0.05.

**p < 0.01.

***p < 0.001.

For knowledge of the graduate school application process, time spent on professional development activities had the largest effect (β 0.12 ± 0.03 SE, p < 0.01). For knowledge of graduate school life, mentor quality had the largest effect (β 0.17 ± 0.04 SE, p < 0.000). For knowledge of careers available in a discipline, plans to pursue an academic career, and plans to pursue a research career outside academia, time spent on research preparation skill development activities had the largest effect (β 0.16 ± 0.05 SE, p < 0.000; β 0.13 ± 0.06 SE, p < 0.01; β 0.18 ± 0.06 SE, p < 0.000). Finally, for the research skills index, mentor quality had the largest effect (β 0.22 ± 0.03 SE, p < 0.000).

DISCUSSION

This study documents the impact of Leadership Alliance’s SR-EIP programmatic components on students’ research self-efficacy and career pathway planning in relation to student demographic and academic characteristics. Before participants’ summer experiences, they report comparatively lower levels of understanding with regard to the graduate school application process and graduate school life. Controlling for these preprogram measures, professional development activities are shown to have a positive and significant impact on these career pathway measures. Further analyses of the preprogram data revealed that students with fewer years until graduation who had completed a previous research experience reported significantly higher levels of understanding of research skills and the graduate school application process and graduate school life. Students’ consideration of research careers outside academia is relatively unchanged after their summer experiences. While student demographics and institutional characteristics were not significantly associated with preprogram measures, modest significant differences were observed postprogram with regard to research skill development and academic career planning.

SCCT Implications for Programmatic Interventions

Adopting an SCCT framework, we earlier posited that students’ research self-efficacy or belief in their ability to perform a task successfully is positively associated with their outcome expectations and, ultimately, career goals. Our analyses bear this out, revealing that activities related to research preparation, professional development, and mentorship were significantly associated with positive changes in students’ knowledge of careers available in their disciplines, plans to pursue research-based careers, and knowledge of the graduate school application process and graduate school life. Time spent immersed in symposia, small-group discussions, study groups, and lectures provided learning experiences that increased their overall research knowledge skills. Specific time spent gathering and analyzing data was also positively associated with students’ research skills. Activities related to data analysis also had a positive effect on students’ knowledge of careers in their disciplines. These findings are consistent with previous studies on engaging students in academic-year research, which demonstrated a positive effect on student’s grade point averages (Fechheimer et al., 2011). In addition, evaluations of summer research programs demonstrated learning gains from these experiences (Lopatto, 2004a,b 2007). Our data confirm that research-specific, skill-building activities complemented with professional development activities are important factors that contribute to developing students’ research self-efficacy and, in turn, informing students’ outcome expectations and career goals.

This study also examined the importance of mentoring in the summer undergraduate research experience as it pertains to changes in research skills and career pathways. While a previous study by Russell et al. (2007) reported little evidence of the impact of mentoring on student outcomes, we found that the quality of the mentoring experience was positively associated with changes in research skills and career planning, given mentors were identified as faculty members, postdoctoral researchers, and graduate students. In fact, mentor quality had the largest effect on the research skills index. This is consistent with data in Table 3 that show significant correlations with mentor satisfaction, with the strongest correlations including some of the following items: showing interest in students’ research and being supportive of their ideas, making students feel like an integral part of a research project, offering advice and guidance on research, being available to discuss research, and providing constructive feedback on academic career development. In contrast, assigning students to mentors whose primary academic appointment was in administration revealed lower quality of mentoring. These data suggest effective mentoring practices promote a productive interpersonal relationship between the discipline-specific mentors and their mentees and emphasize the importance of a holistic mentoring approach that combines research skills with discussions about graduate school and careers.

Taken together, our findings suggest several implications for enhancing summer research programs. First, program activities that promote research skill development in combination with professional development mentoring on graduate school and research strengthens students’ belief in their ability to perform, heightens their outcome expectations, and clarifies the graduate school process and their career planning goals. Second, providing opportunities for undergraduates as early as their first year to be exposed to research concepts and careers is important for scaffolding students’ course work, increasing retention in their STEM majors, and ensuring their competitiveness for subsequent research opportunities. Finally, identifying and developing effective mentors is critical to creating essential support systems for students as they navigate their academic and career pathways. Mentorship and advising play a key role in engaging, recruiting, retaining, and training students in the STEM disciplines. This is especially true when the mentor–mentee relationship is discipline based and inclusive with respect to different backgrounds, cultures, and experiences. Providing multiple moments for discussion and reflection on issues of diversity is a key principle of building mentorship expertise.

The constructs of research self-efficacy, outcome expectations, and goals are posited by SCCT theorists to be reciprocal influences that inform and modify each in turn as interests and academic/career decisions are evidenced (Lent et al., 2002). Moreover, these attributes exercise considerable influence on the development of career interests throughout students’ participation in or exposure to a range of learning experiences or contexts. Our data reveal significant positive postprogram change for all research skill and career pathway items, with the exception of planning to pursue a research career outside academia. Taken together, the acquisition of research skills and increased awareness of the graduate school application process and career pathways contribute to the ongoing formation of confidence, goals, and motivational behaviors that shape subsequent academic and career pursuits.

Limitations

A temporal limitation of this study is its focus on recent program participants between 2013 and 2015. As many of these students are still enrolled in their baccalaureate programs or are just entering graduate training or the workforce, outcome data for these students are either unavailable or limited. As indicated in the section on Future Research, documentation of outcomes would further our understanding of the longitudinal impact of programmatic activities on students’ progression into academic and professional research careers.

As indicated earlier, our survey instruments have not yet incorporated a specific scale on self-efficacy, be it research, science, or general self-efficacy. Rather, items related to the impact of the research experience on knowledge development, self-confidence, and future plans were measures considered as viable for close approximation of research self-efficacy for the purposes of this study. Plans for ongoing development of Leadership Alliance instruments will include consideration of extant measures of general self-efficacy, such as that developed by Chen et al. (2001), or derivation of a valid and reliable research self-efficacy measure from a composite of items already included in the Leadership Alliance surveys.

Future Research

This study builds on findings that demonstrate the effectiveness of an undergraduate research experience in clarifying career decisions (Lopatto, 2007) by identifying specific programmatic components that increase students’ research knowledge skills and heighten their expectations for graduate training and research careers. As the Leadership Alliance continues to track students throughout their academic and career pathways, future research will focus on the study of participants’ postbaccalaureate outcomes to ascertain the impact of SR-EIP activities on their matriculation into graduate programs and the research workforce. Learning that significant gains in research knowledge skills are associated with continued or repeated undergraduate research experiences, the Leadership Alliance will focus on earlier interventions for first-year students. Whether or not such interventions will promote retention in STEM disciplines or research training is key to our ongoing efforts to increase diversity throughout the academic pathway.

Further, we are interested in pursuing two initiatives that strengthen research efforts. The first initiative involves introducing perspectives on engagement of mentees in future iterations of our mentor survey to investigate the correlation of participants’ perceptions of their attitudes and skills with mentors’ perceptions. Second, we would like to learn more about the impact of the Leadership Alliance’s plan to provide academic-year skill building for first- and second-year students at partner institutions. These skill-building workshops are intended to prepare students for summer undergraduate research experiences and continue their mentoring after the summer. Again, using our SCCT framework as a guide, we argue that early introduction to and supportive experiences in research are key to forming positive expectations for success in STEM training and career planning.