Factors Influencing Student Gains from Undergraduate Research Experiences at a Hispanic-Serving Institution

When factors relevant to student gains from undergraduate research were examined, results revealed that students who reported receiving higher-quality mentorship, spending more hours caring for dependents, and receiving more programmatic resources experienced significantly greater gains in science competency, personal growth, and skills.

Abstract

Undergraduate research experiences (UREs) confer many benefits to students, including improved self-confidence, better communication skills, and an increased likelihood of pursuing science careers. Additionally, UREs may be particularly important for racial/ethnic minority students who are underrepresented in the science workforce. We examined factors hypothetically relevant to underrepresented minority student gains from UREs at a Hispanic-serving institution, such as mentoring quality, family income, being Latino/a, and caring for dependents. Data came from a 2013 survey of University of Texas at El Paso students engaged in 10 URE programs (n = 227). Using generalized linear models (GzLMs) and adjusting for known covariates, we found that students who reported receiving higher-quality mentorship, spending more hours caring for dependents, and receiving more programmatic resources experienced significantly greater gains from their URE in all three areas we examined (i.e., thinking and working like a scientist, personal gains, and gains in skills). In two of three areas, duration of the URE was positive and significant. Being Latino/a was positive and significant only in the model predicting personal gains. Across the three models, quality of mentorship was the most important correlate of gains. This suggests that providing training to faculty mentors involved in UREs may improve student outcomes and increase program efficacy.

INTRODUCTION

Participation of undergraduate students in faculty-mentored research has been documented as a “high-impact” educational practice, which means that it has been widely tested and found to be beneficial for undergraduate students from a variety of backgrounds (Kuh, 2008). Undergraduate training programs centered on faculty-mentored research projects have been shown to promote gains among student participants in active learning, self-confidence, and pursuit of science careers (Lopatto, 2007; Lopatto and Tobias, 2010). Research experiences also increase students’ critical-thinking and communication skills (Bauer and Bennett, 2003; Seymour et al., 2004; Hunter et al., 2007). Students are more likely to acquire critical-thinking and communication skills from participating in out-of-class research projects than they are from other activities, such as internships or course-based research projects (Thiry et al., 2011). Undergraduate research participants are thus better prepared to become science professionals than their peers (Laursen et al., 2010; Thiry et al., 2011).

Undergraduate research experiences (UREs) may be particularly important for racial/ethnic minority students (Russell et al., 2007; Jones et al., 2010), who are underrepresented in higher education at the undergraduate level, in graduate school enrollment, and in terms of degree completion. UREs have been shown to have a bigger impact on underrepresented minority (URM) students than non-URM students in terms of aspirations to attend graduate school (Eagan et al., 2013), and URM students participating in research programs are more likely to persist in a science career than are URM students not doing research (Schultz et al., 2011). Qualitative evidence collected from high-achieving URM alumni of an undergraduate biology enrichment program demonstrated that UREs were critical in guiding their career choices; they pinpointed the URE experience as increasing their commitment to science in general and to their interest in pursuing a PhD (Villarejo et al., 2008).

A number of federally and privately funded undergraduate educational intervention programs focused on UREs have been implemented in the United States over the past three decades to increase the success of URM students in higher education and eventually in the science, technology, engineering, and math (STEM) labor market. This study examines factors that predict student gains from UREs among a sample of STEM undergraduate research program participants at the University of Texas at El Paso (UTEP). This study’s focus on a Hispanic-serving institution (HSI), wherein nearly 90% of students are Hispanic/Latino, enables examination of factors that might be particularly important for low-income URM student success. These factors include the duration of students’ engagement in research programs, the quality of faculty mentoring provided to students, student ethnicity, English language deficiencies among students, caretaking responsibilities of students, student household income, student sex, and resources received by students through research programs.

Many studies have demonstrated an association between longer duration of UREs and greater student gains. For example, students who participated in undergraduate research programs over one summer reported gains in fewer areas than those completing a yearlong research experience (Adedokun et al., 2014). Additional studies have shown that, after completing 1 year in an undergraduate research program, students have read scientific literature, talked with senior scientists, and participated in regular lab meetings. These activities are hypothesized to help students integrate their understanding of underlying theories and concepts (Thiry et al., 2011). While benefits have been assessed for shorter-duration UREs, it has been argued that it takes three semesters in an undergraduate research program for students to build an identity as a scientist; after this much time, students begin to adopt the traits, habits, and temperament necessary to succeed along a scientific research path (Thiry et al., 2011).

Research and theory on mentoring suggest that the quality of mentoring is important for student gains, but there has been limited focus in the undergraduate research literature on the quality of the mentoring relationships in which students engage. Previous studies on URE student gains have focused primarily on quantifying the amount time students spend in faculty-mentored research programs or on assessing the gains associated with having versus not having a mentor (e.g., Schultz et al., 2011), rather than on the quality of the mentoring that occurs during this formative time (e.g., Hunter et al., 2007; Thiry et al., 2012). Outside the context of UREs, having a mentor (as opposed to not having one) has been associated with greater persistence and higher grade point average among college students (Jacobi, 1991; Campbell and Campbell, 1997; Crisp and Cruz, 2009; Crisp, 2010). In one of the few studies to examine the impact of mentoring over time, “hard science” doctoral students were asked to rate their advisers in terms of psychosocial mentoring, career-related mentoring, and research collaboration (i.e., invited coauthorship of research products). When participants were surveyed 3 years later, receiving higher levels of psychosocial mentoring predicted greater research self-efficacy (Paglis et al., 2006). However, few studies have assessed benefits associated with mentoring quality in student research experiences, and none to our knowledge have done so at the undergraduate level.

Apart from the duration of the URE and mentoring quality, other factors may be important correlates of student gains. Research indicates that HSIs are an important point of entry for Latino/a students in STEM fields (Crisp et al., 2009). Students at HSIs, including UTEP, are likely to be Latino/Hispanic, to not speak English at home, to be responsible for caring for dependents, and to have low incomes. These factors hypothetically influence gains from UREs, but they have not been comprehensively studied.

Racial/ethnic minority status is a factor that is likely to influence gains from UREs. At the University of California–Davis (a non–minority-serving institution [MSI]), Jones et al. (2010) found that Hispanic and African-American students (relative to other racial/ethnic groups) exhibited the largest gap in the probability of graduating with a biology degree between those who participated in URE versus those who did not, suggesting that URE participation may have provided increased benefits to students from those minority groups. By the fourth year of college, among students who had participated in a URE, Latino/as had higher aspirations for attending graduate programs than did whites (Eagan et al., 2013). In terms of student-reported gains, among summer research participants across 41 universities, URM students reported greater learning gains than their white/Asian peers, but rated their satisfaction with the overall experience the same (Lopatto, 2007).

Many students at MSIs face additional challenges, such as having family members who do not speak English and having to care for dependents. It is plausible that students who do not speak English at home might exhibit worse performance in UREs due to language challenges; however, we hypothesize that they stand to gain more from UREs than English as a first language students. This is because previous studies have demonstrated that students who did not speak English at home were more likely to persist in college and to receive more “A’s” and fewer “F’s” than those who grew up in families in which English was frequently spoken (Lohfink and Paulsen, 2005; Breckler et al., 2011). Many students attending HSIs have complicated family lives that include caretaking for dependents like children and elderly relatives. Among graduating UTEP seniors between 2008 and 2012, the average student provided 20 h of dependent care per week; counterintuitively, more student-reported caretaking hours were significantly and positively related to greater student–faculty interaction, more perceived institutional support, and more self-reported gains in knowledge and skills (Collins et al., in press, 2016). Conversely, among Indiana University students, caring for dependents was associated with lower odds of retention (Fiorini et al., 2014). While prior research results provide mixed evidence regarding the relationship between caring for dependents and student gains, it is an important variable to consider when examining URE outcomes at an HSI.

Socioeconomic status (SES) is hypothetically important to student gains from UREs. Not surprisingly, some studies have found lower SES among students to be associated with worse educational outcomes (Walpole, 2003; Engle and Tinto, 2008; Stuber, 2011). However, among Indiana University students, having an unmet financial need increased both the likelihood of first-year retention and 4-year graduation. While this seems counterintuitive, the authors asserted that students with unmet financial needs may value their educational experiences more than students from higher-SES backgrounds and may expend greater efforts to complete their university educations (Fiorini et al., 2014). This suggests that student participants of lower SES may stand to gain more than their higher-SES counterparts from UREs. Given the low SES of many students at MSIs, it is logical that providing more resources to URE program participants would result in greater gains. URE programs often provide resources such as monthly stipends, travel money for conference presentations, and funds for conducting the research project. While these factors are rarely examined, research has shown that providing resources (e.g., financial aid) to college students plays an important role in degree completion (Cerna et al., 2009).

Students’ sex may also relate to gains from UREs, as female sex has been linked to better academic outcomes more generally (DeBerard et al., 2004; Conger and Long, 2010; DiPrete and Buchmann, 2013) and among Latinas specifically (Cole and Espinoza, 2008). In a study of summer URE participants, women reported greater learning gains than men (Lopatto, 2007), but men and women did not differ in terms of their intentions to continue their education (Lopatto, 2004). These findings suggest that student sex is an important variable to include in any study of student gains through UREs.

In sum, while prior studies have found the duration of the URE to positively predict student-reported gains from UREs, other potentially relevant influences on student gains in the context of UREs (e.g., mentoring quality, household income, and programmatic resources) remain largely unexamined. To address this, we examine the influence of multiple variables upon self-reported gains (i.e., thinking and working like a scientist, personal gains and gains in skill) among a sample of STEM undergraduate research program participants.

MATERIALS AND METHODS

Institutional Context

UTEP provides an illustrative context to examine student gains associated with undergraduate research programs, especially among a majority URM study body. UTEP is one of the nation’s leading contributors to diversity in higher education, specifically through the provision of educational opportunities for the rapidly growing albeit socially and economically disadvantaged U.S. Hispanic population. UTEP ranks second among all universities in the continental United States in the number of bachelor’s degrees conferred to Hispanics (UTEP CIERP, 2014). Washington Monthly’s 2013 rankings place UTEP as the seventh-best U.S. university overall, which is largely attributable to UTEP’s number one ranking in the “social mobility” category. According to institutional records, 57% of enrolled undergraduates at UTEP from 2009 to 2013 were from the first generation in their families to attend college. UTEP has a Hispanic majority undergraduate student body. As of 2013, just under 20,000 undergraduate students were enrolled at UTEP; among those of known race/ethnicity, 87.3% of those were Hispanic, 2.8% were black (non-Hispanic), and 8.7% were white (non-Hispanic). UTEP is regarded as being highly supportive of student engagement. In Kuh et al.’s (2010) Student Success in College: Creating Conditions That Matter, UTEP was examined as a high-impact university in terms of student engagement, since it exhibited both higher than predicted graduation rates and higher than predicted scores on the National Survey of Student Engagement (NSSE).

Participants

An Institutional Review Board–approved structured survey (UTEP IRB No. 503441-1) based largely on the Undergraduate Research Student Self-Assessment (URSSA) and the NSSE was administered to participants using Qualtrics Survey Software. The URSSA instrument evaluates student outcomes of UREs. The URSSA instrument has been repeatedly tested, and the validity and reliability of the items within the instrument have been continuously verified (e.g., Weston and Laursen, 2015). Through the College Student Report survey instrument, the NSSE examines students’ participation in programs and activities that institutions provide for their learning and personal development; the survey items can be used to obtain reliable estimates of how undergraduates spend their time and what they gain from attending college (NSSE, 2015). Using these valid and reliable measures allows for comparison of the outcomes of students in our study to outcomes measured in the same way in other student populations. The general aim of our survey was to collect information on undergraduate research program participants at UTEP from a variety of different programs in order to generally assess current program strengths and future needs.

The survey was distributed to students who were enrolled in an undergraduate research program at the time of the survey (the end of the Fall 2013 semester) and to students who had been involved in an undergraduate research program within the past 5 years (2009–2013), including those who had already graduated from the university. All students who were solicited had participated in their programs for a minimum of one summer or one semester. The names of the 408 students meeting those criteria were obtained from program directors of 10 programs at UTEP. These 10 programs included MARC (Minority Access to Research Careers), RISE (Research Initiatives for Scientific Enhancement), BRIDGES (Bridges to the Baccalaureate Program), SMARTS (Student Mentoring to Achieve Retention: Triads in Science), LSAMP (Louis Stokes Alliance for Minority Participation), COURI (Campus Office for Undergraduate Research Initiatives), STEM-RTI (STEM Research Teaching Integration), UPBIT (Undergraduate Participation in Bioinformatics Training), SMART:MIND (Summer Mentoring and Research Training: Methods in the Neuroscience of Drug-abuse), and the provost office’s summer research assistant program. These programs vary in the duration of engagement with trainees, lasting between a minimum of one summer to a maximum of several years with yearly reappointment. Many of these programs are supported by either the National Institutes of Health (NIH) or the National Science Foundation.

In terms of sample demographics, 89.4% of respondents identified themselves as Latino/a, Hispanic, or of Spanish origin, which closely reflects UTEP’s demographics. Of the respondents, 70% reported speaking a language other than English at home and 59.4% were female. Nearly 40% of the respondents had graduated at the time of the survey, while the other 60% were still enrolled as undergraduates.

Data Collection

The survey was active for 4 weeks (beginning in the middle of November 2013 and ending in the middle of December 2013). Our strategies to enhance the response rate involved sending a prenotification message via email to invite participation in the study and sending three follow-up email reminders to nonrespondents (Manfreda and Vehovar, 2008). In addition, two waves of postcards were sent to permanent addresses of the students. Participants were awarded with a $10 gift card to incentivize survey completion. The response rate was 53%, and a total of 227 individuals completed the survey.

Variables

We used three URSSA constructs as dependent variables: 1) thinking and working like a scientist, 2) personal gains, and 3) gains in knowledge and skills (URSSA, 2009). These reported gains relate to students’ most recent undergraduate research program experience. 1) “Thinking and working like a scientist” refers specifically to student reports of growth in applying scientific knowledge and skills, understanding the scientific research process, and improving their intellectual understanding of the field. 2) “Personal gains” relates to student reports of improvement in comfort and ability working within the scientific field. 3) “Gains in knowledge and skills” measures student reports of acquisition of new skills and knowledge within the field and expansion of their existing knowledge outside the field. As per URSSA, the gains scales items on the survey were rated on a five-point Likert scale ranging from 1 = no gain to 5 = a great gain. The three composite-gains items were created by taking the mean response of the specific questions under each gains item. Cronbach’s alpha scores for each construct revealed that they were reliable. The Cronbach’s alpha scores were 0.923 for thinking and working like a scientist, 0.929 for personal gains, and 0.914 for gains in knowledge and skills. Table 1 reports the specific survey questions used for each gains measure and the justifications for using each selected gains measure.

Table 1.

Variables used in analysis, survey questions, survey instrument used, justification, and coding

Variable	Survey questions	Surveya	Justification	Coding	Recode
Thinking and working like a scientist	How much did you GAIN in the following areas as a result of your most recent undergraduate research program experience? (See Table 2 for the “areas.”)	URSSA	Gains in understanding the nature of scientific knowledge is an essential aspect of students cognitive development (Thiry et al., 2012).	Scale: 1 = no gain, 2 = a little gain, 3 = moderate gain, 4 = good gain, 5 = great gain, 6 = not applicable	1 and 6 = no gain, 2 = a little gain, 3 = moderate gain, 4 = good gain, 5 = great gain
Personal gains	How much did you GAIN in the following areas as a result of your most recent undergraduate research program experience? (See Table 2 for the “areas.”)	URSSA	Personal gains contribute to students’ confidence in their ability to do research and contribute to the discipline overall (Thiry et al., 2012).	Scale: 1 = no gain, 2 = a little gain, 3 = moderate gain, 4 = good gain, 5 = great gain, 6 = not applicable	1 and 6 = no gain, 2 = a little gain, 3 = moderate gain, 4 = good gain, 5 = great gain
Knowledge and skills gains	How much did you GAIN in the following areas as a result of your most recent undergraduate research program experience? (See Table 2 for the “areas.”)	URSSA	Gains in students’ knowledge and skills come from greater exposure and personal experience with research (Thiry et al., 2012).	Scale: 1 = no gain, 2 = a little gain, 3 = moderate gain, 4 = good gain, 5 = great gain, 6 = not applicable	1 and 6 = no gain, 2 = a little gain, 3 = moderate gain, 4 = good gain, 5 = great gain
Months in a research experience	1. How many times have you participated in a research program during the academic year? and	URSSA	More time spent in research program contributes to more overall gains (Thiry et al., 2011; Adedokun et al., 2014).	Academic year: 1 = never participated during academic year, 2 = 1 semester, 3 = 2 semesters, 4 = 3 semesters, 5 = 4 or more semesters	Converted semesters and summer categories to months; academic year: 1 = 0 mo, 2 = 4.5 mo, 3 = 9 mo, 4 = 13.5 mo, 5 = 22.5 mo
	2. How many times have you participated in a research program during the Summer?			Summer: 1 = never participated during the summer, 2 = 1 summer, 3 = 2 summers, 4 = 3 summers, 5 = 4 or more summers	Summer: 1 = 0 mo, 2 = 2.5 mo, 3 = 5 mo, 4 = 7.5 mo, 5 = 10 mo
Quality of mentor	In relation to your most recent undergraduate research program experience, please rate the following: My working relationship with my research mentor	URSSA	Individuals who report a higher quality of mentor report higher gains (Thiry and Laursen, 2011).	Scale: 1 = poor, 2 = fair, 3 = good, 4 = excellent	No recoding needed
Latino/a	Are you of Hispanic, Latino, or Spanish origin?	ACS	Minorities report greater gains in undergraduate research programs (Lopatto, 2007; Lopatto and Tobias, 2010).	1 = yes and 2 = no	1 = yes and 0 = no
English at home	Do you speak a language other than English at home?	ACS	Students who face language barriers may gain more from undergraduate research (Lohfink and Paulsen, 2005; Breckler et al., 2011).	1 = yes and 2 = no	1 = yes and 0 = no
Providing for children/dependents	During your most recent undergraduate research program experience, about how many hours did you spend in a typical 7-d week doing each of the following: Providing care for dependents living with you? (Hours per week)	NSSE	Individuals who report caretaking responsibilities may report higher gains (Collins et al., in press, 2016).	Number of hours per week: 1 = 0, 2 = 1–2, 3 = 3–5 4 = 6–10, 4 = 11–15, 5 = 11–15, 6 = 16–20, 7 = 21–25, 8 = 26–30, 9 = 31–35, 10 = 36–40	Converted using midpoint of hour categories: 1 = 0 h, 2 = 1.5 h, 3 = 4 h, 4 = 8 h, 5 = 13 h, 6 = 18 h, 7 = 23 h, 8 = 28 h, 9 = 33 h, 10 = 38 h
Income	What was your total HOUSEHOLD income for the year 2012 before taxes?	ACS	Economically disadvantaged students may gain more from undergraduate research (Fiorini et al., 2014).	Scale: 1 = under 10,000, 2 = from $10,000 to just under $20,000, 3 = from $20,000 to just under $30,000, 4 = from $30,000 to just under $40,000, 5 = from $40,000 to just under $50,000, 6 = from $50,000 to just under $75,000, 7 = from $75,000 to just under $100,000, 8 = from $100,000 to just under $250,000, 9 = $250,000 or over
Sex	What is your sex?	ACS	Females may gain more than males from UREs (Lopatto, 2007).	1 = male and 2 = female	1 = male and 0 = female
Resources received	Which of the following things were provided: 1) stipends; funding for: 2) conferences, 3) loan repayment, 4) discretionary expenses, or 5) books; 6) equipment, or 7) supplies; 8) an organized learning community composed of other undergraduate researchers; 9) workshops on undergraduate research and professional development; 10) training experience before you began your research project; 11) a computer	Created by research team	The more resources students receive as part of their research program, the more they may gain (Cerna et al., 2009).	Sum of items 1–11
Graduated	Have you graduated from UTEP or another university with your bachelor’s degree?	Created by research team	Used as a control variable	1 = yes and 2 = no	1 = yes and 0 = no
Participation in multiple programs	Which of the following undergraduate research programs have you participated in? (Please check all that apply): 1) Minority Access to Research Careers; 2) UTEP Research Initiative for Scientific Advancement; 3) EPCC Research Initiative for Scientific Advancement; 4) Border Bridges to the Baccalaureate; 5) Student Mentoring to Achieve Retention: Triads in Science; 6) Louis stokes Alliance for Minority Participation; 7) Campus Office of Undergraduate Research Initiatives Research Awards Program; 8) STEM Research-Teaching Integration; (9) Undergraduate Participation in Bioinformatics Training; 10) Provost’s summer scholars program; 11) other programs	Created by research team	Used as a control variable	Sum of items 1–11	1 = responses > 2 and 0 = 1 program or less

^aURSSA is the Undergraduate Research Student Self-Assessment; NSSE is the National Survey of Student Engagement; ACS is the American Communities Survey.

We used eight independent variables in our analysis that are relevant to each student’s most recent research program experience (as it is possible for a student to have taken part in more than one program sequentially) along with two control variables. 1) “Months in a research experience” was calculated by converting the reported semesters and summers spent in their most recent research program into total months. 2) The “quality of mentoring” variable was rated on a four-point Likert scale (1 = poor, 2 = fair, 3 = good, and 4 = excellent) for the most recent research program experience. 3) Being Latino/a or Hispanic was coded as 1 for yes and 0 for no. 4) “Speaking a language other than English at home” was coded as 1 for yes and 0 for no. 5) “Providing for children or dependents” was calculated by taking the midpoint of the hours reported doing that activity each week during their most recent research program, for example, if a student reported spending 26–30 h providing care, the recoded variable would reflect 28 h. 6) Income was measured on a nine-point scale ranging from 1 (less than $10,000 per year) to 9 (more than $250,000 per year). 7) Sex was coded 1 for male and 0 for female. 8) “Resources received” as part of the student’s most recent research program was created based on summing responses from a checklist of resources developed based on a review of what each program provided.

The study used two control variables. “Having already graduated” at the time of the survey was coded as 1 for yes and 0 for no. “Participation in more than 1 program” was created by summing responses from a checklist of programs students could have participated in; students who had been in two or more programs were then recoded as 1, and students who had been in 1 program were recoded to 0.

Table 2 reports the descriptive statistics for the thirteen variables. Students on average reported approximately “good gain” on the three gains measures (with the average score ranging from 4.09 to 4.27 out of 5). Approximately 11.6% of students participated in their research program for one summer, 36.6% participated in their research program for 3–12 mo, and 49.1% did research through their program for more than 12 mo. The mean length of a research experience was nearly 15 mo. The mean student ranking for the quality of research mentor was 3.21 on a four-point scale. Thirty percent of the students reported providing for children or dependents, and, across the entire sample, the mean time spent doing this was 6–10 h each week. The median value for household income reported was 3, which is from $30,000 to just less than $40,000 (median value not shown in Table 2). The mean number of reported resources received by students was 4.1, with the most commonly received items being stipends to support living expenses followed by supplies to conduct research, equipment to conduct research, and workshops for professional development. Approximately 24.7% of students had participated in more than one research program.

Table 2.

Descriptive statistics for all analysis variables including those variables comprising the composite indicators

Continuous variables	n	Minimum	Maximum	Mean	SD	% Missing
Thinking and working like a scientist	211	1	5	4.19	0.818	7
• Analyzing data for patterns	214	1	5	4.05	1.125	5.7
• Figuring out the next step in a research project	214	1	5	4.23	0.959	5.7
• Problem solving in general	214	1	5	4.28	0.901	5.7
• Formulating a research question that could be answered with data	214	1	5	4.09	1.091	5.7
• Identifying limitations of research methods and designs	214	1	5	4.11	1.071	5.7
• Understanding the theory and concepts guiding my research project	212	1	5	4.27	0.939	6.6
• Understanding the connections among scientific disciplines	214	1	5	4.18	1.082	5.7
• Understanding the relevance of research to my course work	213	1	5	4.32	1.037	6.2
Personal gains	210	1	5	4.28	0.836	7.5
• Confidence in my ability to contribute to science	213	1	5	4.07	1.086	6.2
• Comfort in discussing scientific concepts with others	213	1	5	4.15	0.998	6.2
• Comfort in working collaboratively with others	212	1	5	4.31	1.024	6.6
• Confidence in my ability to do well in future science courses	213	1	5	4.23	1.090	6.2
• Ability to work independently	212	1	5	4.40	0.956	6.6
• Developing patience with the slow pace of research	213	1	5	4.24	1.097	6.2
• Understanding what everyday research work is like	213	1	5	4.49	0.877	6.2
• Taking greater care in conducting research procedures	212	1	5	4.32	1.054	6.6
Gains in skills	206	1	5	4.03	0.811	9.3
• Writing scientific reports or papers	213	1	5	4.10	1.081	6.2
• Making oral presentations	212	1	5	4.21	1.082	6.6
• Defending an argument when asked questions	212	1	5	3.96	1.080	6.6
• Explaining my project to people outside my field	212	1	5	4.28	0.975	6.6
• Preparing a scientific poster	213	1	5	4.13	1.248	6.2
• Using statistics to analyze data	213	1	5	3.71	1.331	6.2
• Working with computers	212	1	5	3.86	1.249	6.6
• Understanding journal articles	212	1	5	4.26	0.951	6.6
• Conducting database or Internet searches	213	1	5	4.16	1.018	6.2
• Managing my time	212	1	5	4.10	1.006	6.6
• Using conventional methods, techniques, and technologies to conduct research in my field	211	1	5	4.10	1.108	7
Months in a research experience	216	0	31.5	14.77	9.927	4.8
Quality of mentor	212	1	4	3.21	0.912	6.6
Providing for children/dependents	216	0	38	4.65	10.243	4.8
Income	203	1	9	3.81	2.114	10.6
Resources received	200	0	10	4.10	2.820	27

Dichotomous variables	n	Yes		No		% Missing
Latino/a	227	203		24		0
English at home	212	149		63		6.6
Sex	227	92 (male)		135 (female)		0
Graduated	216	85		131		4.8
Participation in more than one program	227	56		171		0

Analytical Methods

Generalized linear models (GzLMs) were used to examine relationships between the independent variables and the three different gains items. GzLMs support analysis of nonnormal distributions and employ multiple link functions (Nelder and Wedderburn, 1972). On the basis of visual inspection of the histogram of our dependent variable (unpublished data), we ran the GzLMs with six different specifications: inverse Gaussian distribution with log and identity link functions, gamma distribution with log and identity link functions, and normal distribution with log and identity link functions. The identity link function means the dependent variable is predicted directly and is not transformed, while the log link function models the natural log of the dependent variable (Garson, 2012). We selected normal distribution with identity link function for the three final GzLMs, since these specifications yielded the lowest Akaike information criterion (AIC) values for each of the models (see Table 3). A normal distribution with identity link function GzLM is equivalent to an ordinary least-squares (OLS) regression model, which is, in fact, a special case of GzLM. Findings for a normal distribution with identity link function GzLM can be interpreted as though the model was an OLS regression model. Our dependent and independent variables are normally distributed, based on skewness and kurtosis statistics. Because AIC values for the normal distribution with log link function models were close to the normal distribution with identity link function models, we also specified and examined the three GzLMs using normal distribution with log link function; findings between the two sets of models fully agreed in terms of the direction and significance of relationships between the independent and dependent variables. We tested for possible multicollinearity among the analysis variables included in each GzLM. According to variance inflation factor, tolerance, and condition index criteria (Belsley et al., 1980), inferences from GzLM results were not affected by multicollinearity problems.

Table 3.

Model fit statistics (i.e., AIC scores) for the six specifications considered for the GzLM models

	A. Thinking and working like a scientist	B. Personal gains	C. Knowledge and skills gains
Inverse Gaussian with log link	489.221	1299.376	1363.404
Inverse Gaussian with identity link	510.832	1294.638	1430.123
Gamma with log link	453.188	1250.250	1335.108
Gamma with identity link	449.235	1245.456	1332.247
Normal with log link	403.180	1185.805	1298.929
Normal with identity linka	399.729	1181.255	1296.610

^aResults reported in Table 4.

Multiple imputation (MI) techniques address potential bias associated with missing data in statistical analysis. They involve creating multiple sets of values for missing observations using a regression-based approach. MI is used to avoid the bias that can occur when missing values are not missing completely at random (Penn, 2007) and is appropriate for self-reported survey data (Enders, 2010). We used MI to impute the missing values of all analysis variables before running the GzLM. Twenty imputed data sets were specified, which is a current “rule of thumb” in MI, as it maximizes power (as opposed to using 3–5 data sets, which used to be the convention) and improves the validity of multiparameter significance tests, and 200 between-imputation iterations were used to ensure the resulting imputations were independent of each other (Enders, 2010). MI techniques appropriately adjust standard errors for missing data as part of the pooling phase (Enders, 2010), wherein separate model results for each of the 20 imputed data sets were combined into pooled model results (which we report here).

MI was used for the GzLM, while univariate analyses (i.e., descriptive statistics) utilized original data only. Analyses of the MI data sets did not change the significance or direction of any of relationships between the independent and dependent variables in the pooled GzLM results, as compared with model results from analyses employing the original data only. Table 2 displays the percent missing for each analysis variable. The nondichotomous independent variables, including those based on ordinal measures (e.g., quality of mentoring), were standardized and analyzed as continuous predictors in the GzLM. This approach is considered a best practice in MI when imputing missing data and estimating model parameters, since rounding off imputed values based on discrete categorical specifications has been shown to produce more biased parameter estimates in analysis models (Horton et al., 2003; Allison, 2005; Enders, 2010; Rodwell et al., 2014). IBM SPSS (version 21) was used to conduct all analyses.

RESULTS

Table 4A reports pooled results for the model predicting gains in thinking and working like a scientist. In this model, both having higher-quality mentoring and spending more months in a research experience were significantly associated (p < 0.05) with greater self-reported gains. Receiving more resources through the program and spending more time providing for children or dependents were also significantly associated (p < 0.05) with greater self-reported gains in thinking and working like a scientist. Income, speaking a language other than English at home, having already graduated at the time of the survey, being Latino/a, sex, and engagement in more than one program were not found to be significant predictors of self-reported gains in thinking and working like a scientist.

Table 4.

Results of generalized linear models predicting student gains in three areas (n = 227)^a

	A. Thinking and working like a scientist			B. Personal gains			C. Gains in knowledge and skills
Independent variables	B	SE	p Value	B	SE	p Value	B	SE	p Value
Months in a research experience	0.012	0.0053	0.019	0.011	0.0054	0.034	0.008	0.0054	0.117
Quality of mentor	0.333	0.0527	<0.001	0.343	0.0533	<0.001	0.248	0.0528	<0.001
Latino/a	0.214	0.1631	0.189	0.348	0.1646	0.035	0.110	0.1659	0.507
English at home	−0.025	0.1099	0.817	−0.009	0.1110	0.939	0.079	0.1119	0.483
Providing for children/dependents	0.011	0.0048	0.023	0.010	0.0048	0.037	0.014	0.0048	0.005
Income	−0.029	0.0233	0.213	−0.034	0.0237	0.152	−0.033	0.0237	0.158
Sex	0.056	0.0966	0.562	−0.036	0.0976	0.714	0.023	0.0978	0.810
Resources received	0.093	0.0202	<0.001	0.082	0.0203	<0.001	0.091	0.0204	<0.001
Graduated	−0.105	0.1088	0.333	−0.045	0.1090	0.683	0.058	0.1098	0.596
Participation in multiple programs	−0.124	0.1130	0.273	−0.009	0.1141	0.939	−0.026	0.1147	0.817

^aResults reported are pooled across the 20 imputed data sets, and each model uses a normal distribution with an identity link function. Quality of mentor, months in a research experience, resources received as part of research program, providing for children/dependents, and income were included as standardized variables.

Table 4B reports results for the model predicting self-reported personal gains. Within this model, having higher-quality mentoring as well as spending more months in a research experience were significantly associated (p < 0.05) with greater self-reported gains. Acquiring more resources through the program and spending more time providing for children or dependents were significantly associated (p < 0.05) with greater self-reported personal gains, as was being Latino/a. However, speaking a language other than English at home, having already graduated at the time of the survey, sex, and participation in more than one program were not significant predictors of self-reported personal gains.

Results for the model predicting gains in knowledge and skills are presented in Table 4C. As with the other two GzLMs, having higher-quality mentoring was significantly associated (p < 0.05) with greater self-reported gains, but spending more months in a research experience was not significant, although the effect was positive. Receiving more resources through the program and spending more time providing for children or dependents were significantly associated (p < 0.05) with greater self-reported gains in knowledge and skills. The rest of the variables were not statistically significant.

DISCUSSION

Most of the literature on undergraduate research programs has focused on the effect of time on student gains; however, the results of this study indicate that quality is more important when predicting student gains and fostering the development of knowledge and skills. The quality of mentoring was a significant predictor of student gains across all three GzLMs. Comparing the parameter estimates (Table 4B), the effect size for quality of mentoring is more significant and nearly triple the effect size for months in a research experience in the thinking like a scientist, personal gains, and gains in knowledge and skills GzLMs.¹ Our findings in regard to mentoring quality have not been found in previous studies of undergraduate research training. Spending more months in a faculty-mentored research program was significantly and positively related to personal gains and thinking and working like a scientist, but not to knowledge and skills gains. Thus, the length of the research experience may be most important for psychosocial gains, including the formation of a science identity, which is in line with findings from previous studies (Thiry et al., 2011, 2012). In contrast, results suggest that gains in knowledge and skills may be more closely linked to quality mentorship, and that such gains may accrue over relatively short time frames through focused, high-quality interaction and engagement with a faculty mentor. Our results indicate that the duration of engagement in faculty-mentored research and the quality of mentorship each contribute independently to student gains,² but that high-quality mentoring may be more important on the whole than having a URE of long duration. In practical terms, this suggests that students may stand to gain in the three areas measured here from relatively short-duration UREs, assuming that they engage in high-quality interactions with their mentor. Our results also indicate that undergraduate research programs might generate greater impacts on student development by providing training to faculty mentors in the practice of undergraduate student mentoring.

In terms of the other factors considered, there were several notable findings. Spending more hours caring for children or dependents was found to be a significant positive predictor of gains in all three GzLMs. We believe that this may be attributed to students with family responsibilities prioritizing their time differently than their peers who do not have this extra obligation. These students have limited amounts of time and may invest more energy into their time spent doing undergraduate research and therefore get more out of the experience than students who do not report caretaking obligations. This finding is not unique to these research participants; prior research on graduating seniors at UTEP showed that more student-reported caretaking obligations significantly and positively predicted more student–faculty interaction, more perceived institutional support, more self-reported gains in knowledge and skills, and greater overall satisfaction with the UTEP experience (Collins et al., in press, 2016).

Providing more resources as part of research programs was also associated with significantly greater gains in all three areas. This suggests that programs that provide more resources (e.g., money to support travel to conferences, repayment of student loans, funds to support living expenses, equipment for research, and skills-building opportunities like workshops and seminars) are highly beneficial in terms of student gains, even when accounting for students’ household income. Future research should disentangle which types of resources would be most beneficial to provide within URE programs. Being Latino/a was significantly associated with greater self-reported personal gains. Being Latino/a was not significant in the other two GzLMs, but the effect was positive, which suggests that Latino/a students may gain more from participation in faculty-mentored UREs than non-Hispanics. The small numbers of non-Latino/a students in the sample may have increased our SE and reduced our ability to find a statistically significant association. Our results suggest that Hispanic students may gain more than non-Hispanic students from participation in undergraduate research programs at this HSI. This finding is in line with previous studies finding that URM students benefit disproportionately from UREs (Lopatto, 2007; Russell et al., 2007; Jones et al., 2010; Eagan et al., 2013). Student sex, income, participating in more than one program, and speaking a language other than English at home were not found to be significant in any of the GzLMs, indicating they are not important correlates of self-reported gains.

Future research would be advanced through the use of a multidimensional scale to measure the quality of research mentorship. In past qualitative research, students have been asked to describe mentor quality in ideal terms (Ishiyama, 2007; Thiry et al., 2011). Their assessments shed light on what students are likely considering when asked to rate the quality of the mentorship they have received. Undergraduate students defined a quality mentor as one who regularly interacts with them, helps with their understanding of the values and norms of their profession, and provides intellectual as well as personal or emotional support (Thiry et al., 2011). An examination of students in the McNair postbaccalaureate program found that black students were more likely than white students to describe a good mentor as one who is personally supportive (Ishiyama, 2007). When defining the “mentor concept” based on a thorough review of the literature, Crisp and Cruz (2009) assert that academic mentoring involves four domains: psychological and emotional support, support for setting goals and choosing a career path, academic subject knowledge support aimed at advancing a student’s knowledge of his or her chosen field, and specification of a role model (Crisp and Cruz, 2009). This qualitative work supports the assertion that a multidimensional scale is needed.

While our reliance on URSSA meant we used a unidimensional scale, multidimensional scales to assess URE mentor quality are available. For example, the Mentoring Competency Assessment (MCA), developed by researchers the University of Wisconsin–Madison, enables research mentees to evaluate six competencies of their mentors. This instrument consists of 26-items which are rated on a seven-point Likert-type scale, and it addresses the following domains: maintaining effective communication, aligning expectations, assessing understanding, addressing diversity, fostering independence, and promoting professional development (Fleming et al., 2013). Because there are versions available for the mentee and mentor, this instrument can also be used to identify areas in which mentor and mentee assessments diverge, which would provide information useful for faculty development interventions (Pfund et al., 2014). The use of a multidimensional scale designed to measure more aspects of the student–mentor relationship would provide an improved basis for analyzing the role of mentoring quality in student development via undergraduate research participation.

A second limitation is our examination of only the most recent research program experiences of participants. However, only one-quarter of students had participated in more than one program, and this variable was not a significant predictor of self-reported gains in any of the three GzLMs. We also assumed, by summing up the number of resources provided by the students’ programs, that each resource was equally important to students’ self-reported gains, when it may be the case that some resources are more valuable than others.

Furthermore, our analysis is based on student self-reports for all gains items. Self-reports may be subject to social desirability responses. However, there is evidence that self-report measures are less susceptible to this type of response when the accuracy of the item responses is verifiable (Chan, 2009). Self-reports can also suffer from recall bias (Hassan, 2006). When studying something as subjective as one’s own personal gains, self-reports are the best source of data, since they are related to how one feels about oneself. In terms of knowledge, skills, and science gains, there are limitations to self-reported measures, as documented by recent studies (Gordon et al., 2008; Bowman, 2010, 2011). Those studies suggest that self-report–based gains measures should not be used as simple proxies for actual student learning. We believe that it is still important to examine self-reported gains in the context of UREs. The standardized test–based measures used to assess learning in prior studies are not designed to gauge objective growth in highly specialized research skills/techniques that typically define URE experiences; but this growth can be captured by the URSSA gains scales. The self-reported gains we examine may reflect the acquisition of highly specialized research skills not readily captured by other, more objective measures, although no extant studies substantiate (or refute) that claim.

CONCLUSION

Our results suggest several practical implications when designing and implementing undergraduate research programs. More training for research mentors may improve the quality of the experiences they provide for students within undergraduate research programs. Previous studies have demonstrated the benefits of targeted training to improve research-mentoring practices among faculty members (Pfund et al., 2006, 2013; Balster et al., 2010). For example, research employing the MCA scale described above has demonstrated that, after mentors complete a structured mentoring curriculum, their mentees report an improvement in mentor competency (Pfund et al., 2014). Mentors who take part in training are usually satisfied with the experience; they self-report measurable skills gains as well as positive behavioral changes (Pfund et al., 2013). For example, trained mentors are significantly more likely than untrained mentors to discuss their expectations with their undergraduate research protégés, to consider issues of diversity, and to discuss mentoring with peers and faculty members (Pfund et al., 2006).

Training opportunities for mentors are accessible. For example, Optimizing the Practice of Mentoring is a free online curriculum developed by the Clinical and Translational Science Institute at the University of Minnesota to prepare faculty in higher education to be effective research mentors (www.ctsi.umn.edu/education-and-training/mentoring/mentor-training). Entering Mentoring is a curriculum developed by a team from the University of Wisconsin–Madison for use across STEM disciplines. This program helps mentors explore issues that affect their research-mentoring relationships and enables them to practice more reflective and effective mentorship. These materials are available online at no cost (www.researchmentortraining.org). The National Research Mentoring Network (NRMN) was funded by NIH beginning in 2014 and has developed best practices for mentoring. The NRMN also provides training opportunities for mentors, mentee–mentor matching, and networking and professional development opportunities for mentors (http://nrmnet.net).

Additionally, we found that students who received more resources as part of their program reported greater gains. This suggests that providing resources such as tuition reimbursement, living expense stipends, and supplies such as computers would improve outcomes for students engaged in UREs, especially for those who are economically disadvantaged. There are many research programs available to students currently, but without all the necessary components, students may not be gaining as much from these programs as they could be.

Overall, the quality of mentoring received by students and the months that the students spent in their research experience both matter in predicting student gains. Our results indicate, however, that quality may be a more important predictor than quantity. Length of research experience may influence gains from UREs, but not all time spent in a research program is productive or contributes to student development. Raising the quality of mentorship within research programs will enable students to benefit from both longer experiences that begin earlier in their undergraduate careers and short-duration research engagement, for example, summer experiences. Mentoring training for faculty members within undergraduate research programs should be a top priority to increase the quality of mentoring provided to student participants and, in turn, to improve outcomes defined in terms of student gains.