Impact and Outcomes of the Oklahoma IDeA Network of Biomedical Research Excellence Summer Undergraduate Research Program

Abstract

Participating in research under the guidance of faculty mentors can increase undergraduate students’ skills, knowledge, and confidence in conducting scientific research and pursing a scientific career. The Oklahoma IDeA Network of Biomedical Research Excellence (OK-INBRE) in the US state of Oklahoma has established an infrastructure to develop future researchers and healthcare professionals by providing students with summer internships. However, long-term benefits have not typically been well quantified, and most prior investigations examining benefits and outcomes of undergraduate summer research experiences have been descriptive and/or observational in nature. To assess OK-INBRE summer program participants, educational and career outcomes were systematically tracked. Data for evaluation included 1) primary sources containing responses to learning surveys from OK-INBRE and national cohorts over a three-year period, and 2) secondary sources from the Oklahoma State Regents for Higher Education (OSRHE), that track educational outcomes of Oklahoma students over a 15-year period. Frequency statistics were compiled and bivariate analyses were conducted to examine participant academic and career outcomes. Survey responses reported higher satisfaction in various dimensions of learning among OK-INBRE students than among corresponding national cohorts, particularly on dimensions of knowledge, skills, and understanding of sciences. The OSRHE data showed that, compared with those in the control group, there was a 100% increase in OK-INBRE participants who enrolled in or had completed a professional degree (e.g., MD/DO) and a 175% increase in students attending a biomedical science graduate program. These findings demonstrate the contribution of the OK-INBRE program in promoting science education and professions.

INTRODUCTION

Participation of undergraduate students in mentored research projects under the guidance of faculty members has been characterized as a high-impact experiential practice that confers a multitude of benefits. Examples of these benefits include positive effects on students’ decisions to pursue postgraduate education and/or a research career in science (1–4), the ability to think and work like a scientist (5, 6), self-efficacy in science research (7, 8), and identification as scientists (5, 8, 9). These qualities have been identified as indicators of students’ growth as scientists as well as their pursuit of research-related careers (10).

More importantly, undergraduate research can help bridge the gap and increase the number of women and underrepresented minority students who are exposed to the sciences as a viable career option (11–13). Collaboration with a mentor and participation in research facilitate retention of underrepresented students in science majors (11, 14–17) by providing culturally appropriate role models and mentoring structures that have been shown to be associated with higher scientific identity, greater intentions to pursue a doctorate degree in STEM fields, and higher scholarly productivity (11).

As the summer typically offers an uninterrupted block of time in which students may engage in undergraduate research full-time, various institutions have developed summer research programs structured to train and motivate undergraduate students in research (18). Additionally, studies have shown that summer research programs can dramatically affect students’ commitment to research careers, self-reported knowledge of research skills, time engaged in research activity, and understanding of and attitudes toward pursuing graduate study (11, 19, 20). All of these benefits were also shown to persist even when a student’s home undergraduate institution, level of preexisting research experience, and demographic factors were taken into account (11, 18–20). A number of publications have described components of summer research programs and reported on outcomes. For example, Slattery et al. described a 10-week summer research program that immerses undergraduate nursing students in a broad spectrum of clinical and translational research. At the conclusion of the program, students reported increased ability to participate in research, effectively interact with academic medical center researchers, and incorporate elements of evidence-based practice into future nursing interventions (18). Frantz et al. compared two models within a summer research program for undergraduate students, a team-based collaborative learning model and the traditional apprenticeship model, and examined their outcomes. While both models produced similarly positive results, they found no difference in either short-term (e.g., progress towards STEM degree) or long-term (e.g., retention in science-related field) outcomes between the two models (21). Therefore, while current literature reported on a number of models in which summer research programs are delivered, these programs have largely demonstrated positive outcomes regardless of the techniques or methodology utilized. Moreover, some summer research programs can enhance their interaction with specific underserved populations by addressing the ethnic/racial and gender gaps in science career opportunities using specific programmatic components to better accommodate students of varying academic and cultural backgrounds (22).

While a number of studies have reported positive outcomes in terms of summer programs reinforcing degree attainment and enhancing commitment to a career in biomedical fields, the evidence base for these claims has largely been weak. Linn and colleagues conducted a review of 60 studies and reported that more than half of these studies relied largely on self-report surveys or personal interviews. Moreover, the reports on outcomes have mostly been descriptive in nature, with samples being of insufficient size to determine statistical significance. More rigor and quantitative analyses are needed in the research methodology, and better and more defined control groups are required to truly examine impact and outcomes from these multifaceted summer research programs. These important issues must be addressed for results to accurately inform and improve future design to increase research summer program effectiveness. Processes such as documenting student progress, distinguishing effective and ineffective aspects of the experiences, and illustrating student interpretation of their research experiences exemplify components of a study design that can offer more powerful and generalizable assessments (23).

Over the last decade, the Oklahoma IDeA Network of Biomedical Research Excellence (OK-INBRE), funded by an Institutional Development Award (IDeA) from the US National Institutes of Health/National Institute of General Medical Sciences, has continued to build an infrastructure that creates a pipeline to develop the next generation of researchers and healthcare professionals in Oklahoma. The OK-INBRE network is diverse, comprised of The University of Oklahoma Health Sciences Center (OUHSC) and 11 partner institutions throughout the state of Oklahoma. In addition to promoting research careers, OK-INBRE also helps to increase the state’s research capacity and infrastructure by providing support to biomedical research scientists, which includes enhancing the statewide research equipment infrastructure and promoting workforce development in Oklahoma. The scientific themes of OK-INBRE are multidisciplinary, targeting the fields of infectious diseases, cancer, and developmental biology. These thematic foci are aligned with the strategic research plans of its two major biomedical research-intensive institutions, OUHSC and the Oklahoma Medical Research Foundation (OMRF). Support for students is provided through competitive summer internships, participation in funded faculty research projects, and engagement in bioinformatics and genomics educational programs.

An evaluation component of OK-INBRE systematically tracks outcomes of the longstanding OK-INBRE summer program, using a validated statistical metrics system. Given the overall descriptive and observational nature of many of the available studies, we selected a study design and unique datasets that address the aforementioned limitations in the current literature. To our knowledge, the analysis presented here is the first study that fully incorporates a longitudinal dataset with a comparable control group that allows a direct comparison of outcomes, over a 15-year period, between students who were accepted and matriculated through a summer research program and a group of students who were demographically highly similar and had applied to but did not matriculate through the summer program. In addition to the longitudinal data, survey data to assess the learning experience had also been collected from 2015 to 2017, as a part of a national effort in which Oklahoma participated. Hence, the objective of this study is to evaluate comprehensively the effectiveness, impact, and outcomes of the OK-INBRE summer undergraduate research program using two data sources. Hypotheses related to the likelihood of OK-INBRE program participants earning graduate/medical degrees and pursuing a career in the STEM are tested.

METHODS

The OK-INBRE summer program

The OK-INBRE summer research program accepts applications from participating community colleges and undergraduate institutions in Oklahoma. Student applicants undergo a rigorous selection process by a statewide committee of 10 to 12 faculty from OK-INBRE partner institutions. An evaluation rubric is provided to the reviewers, which they may use as a guide in making their assessment of each applicant. The rubric includes the following criteria: academic standing, personal goal statement, overall GPA, the quantity and quality of STEM classes, letters of reference, prior research experience, and overall impression of the applicant. Reviewers provide a preliminary assessment of each application. At a subsequent review panel meeting, the applicants are individually discussed and a determination is made about acceptance into the INBRE summer program.

The program typically includes a minimum of 20 students from four-year undergraduate institutions and at least 15 students from community colleges throughout the state. Successful students are matched with mentors at research institutions and participate in an eight-week research project over the course of the summer. Program components include: (i) a one-week research “bootcamp” for those without prior research experience, (ii) a research project with a mentor based on research interests, and (iii) enrichment courses through the program covering topics such as graduate and professional programs, composition of a career goal statement, and preparation of a poster presentation. The types of research projects the students undertake cover a range of areas, including basic, translational, community-based, and population-based research.

To ensure continuous quality assurance and improvement of the program, OK-INBRE has been tracking summer research participants since program inception, and the evaluation has grown more robust and systematic over time. To make the evaluation plan more comprehensive, the OK-INBRE program began administering an external, multi-wave survey to participants in 2015 in order to obtain more granular information on participants’ perspectives regarding overall program experience and learning gains. Students completed questionnaires and exit surveys to complement the state-level tracking of academic progress and achievement (https://www.grinnell.edu/academics/areas/psychology/assessments/sure-iii-survey).

The OK-INBRE study has the approval of the OUHSC Institutional Review Board (IRB # 7434).

Data collection and compilation

Data for summer research program evaluation include both primary and secondary sources. The two sets of available data address different outcomes of program participation and thus render a more comprehensive “picture” of overall program effectiveness. Primary data came from participant evaluation of the summer research program, using the Grinnell College Survey of Undergraduate Research Experience (SURE; https://www.grinnell.edu/academics/areas/psychology/assessments/sure-iii-survey). Grinnell College developed and internally validated SURE to specifically assesses the research experience, learning, and future career directions of undergraduate students who have recently completed a summer research program (24). Students responded to the SURE survey at program completion. In addition to the Oklahoma data, Grinnell College collects student surveys from 336 institutions nationwide that participate in its summer student assessment. Therefore, the SURE data gave us the opportunity to examine responses of OK-INBRE students in comparison with those in a national cohort of students who participated in similar summer research programs. OK-INBRE student data from the Grinnell surveys are available for the years 2015, 2016, and 2017.

Secondary data were generated from the Oklahoma State Regents for Higher Education (OSRHE). By state mandate, the OSRHE tracks graduation, degree, and enrollment of students in all Oklahoma higher education institutions. Data on OK-INBRE applicants, including those who were not selected to participate, were extracted from the OSRHE data. This dataset included applicants to the OK-INBRE Summer Program from program inception in 2002 to 2017. A selected cohort of student applicants who had applied but did not participate in the program was designated as the control group to compare with those who did matriculate through the OK-INBRE summer program. The control group was composed of a purposive sample across a group of non-participants for a variety of reasons to ensure that demographics and academic performance were comparable with those of INBRE participants. In all, the control group included 100 students, of whom 25 were program alternates, 13 were offered a summer position but declined participation, and 62 were highly ranked but were not selected, over the same time-frame as the OK-INBRE students who matriculated.

Data analyses

Descriptive statistics on the OK-INBRE students who matriculated and the control group outlined above were compiled from OSRHE tracking data over the years from 2002 to 2017. Bivariate analyses, t-tests or chi-square statistics where appropriate, were conducted to examine differences between participants and controls in terms of academic progress and outcomes, as well as graduate/medical school attendance and pursuing a career in the sciences.

Descriptive statistics were also generated for the 2015, 2016, and 2017 SURE survey responses from both the Oklahoma and the national cohorts. Due to the unequal sample sizes between the OK-INBRE and national cohorts, only frequency statistics (e.g., number of students and percent) were generated and presented. Variables with Likert scale response categories were summarized using means, and frequencies were computed as proportions.

RESULTS

Since 2002, a total of 1,182 Oklahoma students have applied to the OK-INBRE summer research program and 569 have been selected for participation. Additionally, several students have been selected to return to the program for multiple summers. Given that students can apply and matriculate through the OK-INBRE program for multiple years, there were a total of 473 unique OK-INBRE summer students. Since 2005, the number of participants has exceeded 30 every year, the maximum being 51 in 2016.

Findings from longitudinal OSRHE data comparing participants and non-participants

Demographic profiles of the OK-INBRE sample from 2002 to 2017 included 57% female and 42% racial/ethnic minorities (14% African Americans, 14% Asians/Pacific Islanders, 8% American Indians/Native Americans, and 4% Latino/Hispanic). Among the participants, 69% were from four-year institutions and 31% were from community colleges. Comparing OK-INBRE participants and those in the control group, the sensitivity analysis revealed that there was no selection bias in gender (χ² = 0.33, p = 0.57) or between racial/ethnic groups (χ² = 0.009, p = 0.93). Moreover, the two groups did not differ significantly in academic performance, where the mean GPA of the participants was 3.72 and that of the control group was 3.70 (t = −0.80, p = 0.43), out of a possible 4.0. Furthermore, applicants in the control group who were not accepted into the program exhibited the highest mean GPA, at 3.76. There was a marginally significant difference in types of institution from which the participants were selected, with more participants from four-year institutions than community colleges (χ² = 3.53, p = 0.06), which would be expected since the NIH award provides 20 slots for students from four-year institutions and 15 for those from two-year colleges.

In the overall sample of both participants and controls, as shown in Table 1, 25% of the sample have completed a bachelor degree in the sciences, while 13% and 9% are enrolled in or have completed an MD/DO degree or a biomedical/science graduate program, respectively. Another 21% have enrolled in or completed a health-related professional program. Additionally, after earning their first degree, over 90% of the sample furthered their education in the sciences, with 59% of those participants enrolled in a science program, another 14% pursuing a degree in medicine, and 17% continuing in other health-related fields.

TABLE 1.

Academic outcomes.

Demographics	n	%*
Academic and Career Outcomes (n=573)
Completed Associates degree	17	3
Completed Bachelor of Science degree	143	25
Completed Bachelor in nonscience field	15	3
Enrolled in or complete MD/DO degree	72	13
Enrolled in undergraduate program	124	22
Enrolled in or completed biomedical/science graduate program	54	9
Enrolled in nonscience program	11	2
Enrolled in or completed health-related professional program	122	21
Unknown	15	3
Program Enrollment Post First Degree (n=269)
Science	158	59
Medicine	37	14
Other health	45	17
Humanities	7	3
Social science	6	2
General/undeclared	8	3
Other	8	3

^*Percent was rounded up to the whole number.

Sum of all percentages may slightly exceed 100.

Figure 1 illustrates academic progress or outcomes for OK-INBRE participants and students in the control group. The proportion of OK-INBRE participants and that of their counterparts in the control group who completed a bachelor of science degree [indicated as BS in Figure 1] is comparable (25% and 26%, respectively). At the time of the analysis, 23% of OK-INBRE participants and 18% of the control group were continuing their undergraduate education [UDG]. However, a far greater proportion of OK-INBRE summer program participants were enrolled in or had completed a MD/DO degree, at 14%, compared with 7% for the control group. The same observation can be made between those who were enrolled in or had completed biomedical science graduate programs [MS/PhD]: 11% for the OK-INBRE participants vs. only 4% for the control group. On the other hand, a higher proportion of students in the control group had enrolled in or completed a health-related professional [HPD] program: 28% compared with 20% for the OK-INBRE participants. These differences across the various degree categories were statistically significant between the two groups (df = 7, p = 0.0015).

FIGURE 1.

Comparison of academic progress and outcomes between INBRE participants and the control group. The graph shows the percent of students enrolled in the OK-INBRE Summer Research Program and in the control group who were progressing in or completed different degrees. The difference across degree categories and between INBRE and control groups was statistically significant at p=0.0015. INBRE = IDeA Network of Biomedical Research Excellence; AS/AA = completed associates degree; BS = completed bachelor of science; BA = completed other bachelor degree; MD/DO = enrolled in or completed MD/DO degree; MS/PHD = enrolled in or completed biomedical/science graduate degree; HPD = enrolled in or completed health-related professional degree; UDG = enrolled in undergraduate degree; NSD = enrolled in nonscience degree.

As shown in Figure 2, among the sub-sample of 269 participants who had enrolled in or completed their first postsecondary degree, about 30% of OK-INBRE participants enrolled in or completed graduate programs in the sciences, compared with 8% in the control group. Seven percent of OK-INBRE students were enrolled in or had graduated from medical school, which was the case for only 1.5% in the control group. Additionally, 7% of OK-INBRE summer program participants were enrolled in or had earned degrees in other health-related fields, compared with about 5% of those in the control group. The differences across degree programs and between INBRE and control groups were statistically significant (df = 2, p = 0.05).

FIGURE 2.

Advanced degree enrollment. The graph shows the percent of INBRE students and students in the control group who enrolled in advanced degrees in the sciences, medicine, or other health sciences among the subsample of 269 students who completed their first post-secondary degree. The difference across degrees and between the INBRE and control groups was statistically significant at p=0.05. INBRE = IDeA Network of Biomedical Research Excellence; MD/DO = enrolled in or completed MD/DO degree; MS/PHD = enrolled in or completed biomedical/science graduate degree.

Findings from the SURE survey from 2015 to 2017

Summary statistics of the SURE survey responses provide a mechanism to compare various characteristics of the learning continuum between the OK-INBRE summer students and those from the national cohort. Among the 2015 OK-INBRE students, there were more males than females and 36% identified themselves as racial/ethnic minorities (categories included are American Indian/Native American, African American/Black, Asian/Pacific Islander, Latino/Hispanic, two or more races, and other). The 2015 national cohort exhibited similar characteristics with the exception that there were more female participants. The 2016 and 2017 OK-INBRE cohorts included more women, at 57% both years, which is comparable with the national cohort. More than half of the participants in both cohorts reported prior research experience (Table 2).

TABLE 2.

Demographics of summer research program participants who responded to SURE learning assessment.

	2015		2016		2017		3-year Mean
	OK	All	OK	All	OK	All	OK	All
Number of respondents (n)	37	2,077	35	2,777	51	2,417	123	7,271
	%		%		%		%
Gender
Male	57	42	43	40	43	43	48	42
Female	43	58	57	60	57	57	52	58
Race/Ethnicity
American Indian/Native American	9	1	17	0.5	6	1	11	0.8
African American/Black	6	10	0	8	0	9	2	9
Asian/Pacific Islander	9	12	0	10	14	12	8	11
Latino/Hispanic	3	11	0	15	8	12	4	13
White	63	56	54	54	45	55	54	55
Two or more races	9	6	6	6	8	7	8	6
Other	0	2	9	2	10	3	6	2
Year in College
First	0	12	3	9	0	9	1	10
Second	41	21	40	18	26	17	36	19
Third	30	30	26	29	42	31	33	30
Fourth	30	25	29	25	26	31	28	27
Prior Research Experience	68	52	60	51	56	56	61	53

Table 3 captures the ratings of the learning experience among participants from both the OK-INBRE and national cohorts. Both reacted highly positively to their overall research experience, although the OK-INBRE participants ranked their experience slightly more positively than those in the national cohort. Among OK-INBRE students in 2015, 2016, and 2107, 76%, 77%, and 78% of those who participated, respectively, with a three-year mean of 77%, reported being “very satisfied with research experience” compared with 77%, 70%, and 70%, and a three-year mean of 72%, of the national cohort for the same time period. In terms of reactions about the experience with peer mentors and the mentoring experience as a whole, ratings reported by both cohorts are comparable. However, OK-INBRE students over the three-year period did indicate less satisfaction than the national cohort for two program components in particular: “seminars at which scientists discuss their research,” with a three-year mean difference of 7%, and “final presentation of work,” with a three-year mean difference of 4%. On the other hand, the three-year mean showed that 47% of the OK-INBRE students reported the “experience to be much better than expected,” compared with 41% of students nationwide. Similarly, the proportion of OK-INBRE students who reported satisfaction with safety seminars was 4% higher than that of the national cohort.

TABLE 3.

Participant ratings of summer research program experience^a.

	2015		2016		2017		3-year Mean
	OK	All	OK	All	OK	All	OK	All
Number of respondents (n)	37	2,077	51	2,417	35	2,777	123	7,271
	%		%		%		%
Satisfaction with program components (% positive)
Preparing an application or proposal	79	55	55	67	64	67	66	63
Seminars at which scientists discuss research	74	76	76	77	61	78	70	77
Seminars on safety	63	50	50	57	61	55	58	54
Instruction/discussion of ethics	57	60	60	62	63	62	60	61
Final presentation of work	84	77	77	87	80	87	80	84
Experience much better than expected	54	46	46	38	40	38	47	41
Primary supervisor above average or outstanding	83	84	84	83	80	83	82	83
Working with other students one of the best parts	53	33	33	41	38	42	41	37
Very satisfied with research experience	76	77	77	70	78	70	77	72
	Meanb		Meanb		Meanb		Meanb
Mentoring experience
Enjoyed the responsibility	4.40	4.75	4.75	4.24	4.80	4.33	4.65	4.44
Enjoyed teaching	4.40	4.75	4.75	4.25	4.80	4.43	4.65	4.48
Gained self-confidence as a researcher	4.60	4.25	4.25	4.43	4.60	4.38	4.48	4.35
Felt unprepared by supervisor	2.40	2.00	2.00	2.32	2.20	2.35	2.20	2.22
Improved oral communication skills	3.80	4.00	4.00	3.93	4.00	3.99	3.93	3.97
Given responsibility beyond experience	2.00	3.00	3.00	2.67	2.80	2.85	2.60	2.84
Felt pressure for others’ performance	3.80	2.75	2.75	3.35	4.00	3.21	3.52	3.10
Felt responsibility for the research	4.20	3.25	3.25	4.28	4.60	4.17	4.02	3.90
On his/her own too often	2.40	2.25	2.25	2.29	2.40	2.38	2.35	2.31
Role deepened understanding of research project	4.40	4.00	4.00	4.29	4.00	4.21	4.13	4.17
Role increase motivation to work on research	4.40	3.75	3.75	4.17	4.00	4.12	4.05	4.01
Summer research experiences more stressful	2.32	2.67	2.67	2.12	2.47	2.14	2.49	2.31
Research more interesting than course work	4.44	3.96	3.96	3.74	3.80	3.77	4.07	3.82

^aNot all students completed the survey without skipping a question and therefore the denominator for different questions may vary.

^bMean was calculated by averaging responses to a series of questions based on 5-point Likert scales, ranging from 1, strongly disagree to 5, strongly agree.

Regarding the mentoring experience, the three-year means showed that OK-INBRE participants rated their experience higher than did their national counterparts in enjoyment of responsibility, teaching, gaining self-confidence as researchers, ownership of the research, and finding research to be more interesting than course work. However, OK-INBRE participants reported feeling pressure for others’ performance and stress from the research experience (Table 3).

When assessing knowledge and learning gains from the SURE survey responses (Table 4), OK-INBRE students reported stronger agreement with various aspects of learning than their counterparts in the national cohort. Questionnaire items that showed the greatest differential in the rankings between the two cohorts across the three years include those that enhanced “clarification of career path,” “confidence in potential as a teacher,” and “improved self-confidence,” as well as those that improved understanding “how knowledge is constructed,” “that scientific assertions require evidence,” “science,” and “how scientists think.” In addition to the aforementioned items, significant differences in the rankings between the OK-INBRE and national cohorts were observed in items related to skills in “interpretation of results,” “science writing,” “ability to integrate theory and practice,” “ability to analyze data and other information,” “learning ethical conduct,” “learning laboratory techniques,” “learning to work independently,” and “becoming part of a learning community.” The three-year mean difference computed between the OK-INBRE and national cohorts showed that OK-INBRE students gave higher average rankings across all categories of the learning experience, including career orientation, confidence, and knowledge, skills, and understanding of science, than their national counterparts.

TABLE 4.

Participant ratings of learning gains from summer research program^a.

	2015		2016		2017		3-year Mean
	OK	All	OK	All	OK	All	OK	All
	Mean Likert Scaleb		Mean Likert Scaleb		Mean Likert Scaleb		Mean Likert Scaleb
Career orientation
Clarification of a career path	3.77	3.32	3.68	3.33	3.62	3.36	3.69	3.51
Confidence
Improved self-confidence	3.92	3.51	3.83	3.52	3.62	3.50	3.79	3.51
Confidence in potential as a teacher	3.49	3.19	3.72	3.21	3.44	3.19	3.55	3.20
Tolerance for obstacles in research process	3.94	3.84	3.98	3.87	3.81	3.85	3.91	3.85
Readiness for more demanding research	3.97	3.78	4.04	3.79	3.58	3.80	3.86	3.79
Knowledge, skills, and understanding of science
Understanding how knowledge is constructed	4.03	3.62	4.00	3.65	3.65	3.62	3.89	3.63
Understanding the research process	4.00	3.89	4.02	3.90	3.92	3.89	3.98	3.89
Ability to integrate theory and practice	3.94	3.61	4.02	3.65	3.77	3.62	3.91	3.63
Understanding how scientists work on real problems	3.97	3.82	4.26	3.85	3.92	3.86	4.05	3.84
Understanding that scientific assertions require evidence	3.86	3.59	4.15	3.61	3.65	3.60	3.87	3.60
Ability to analyze data and other information	3.86	3.72	4.04	3.73	3.77	3.73	3.89	3.73
Understanding science	4.00	3.58	4.04	3.60	3.69	3.60	3.91	3.59
Learning ethical conduct	3.66	3.28	3.62	3.26	3.46	3.34	3.58	3.29
Learning laboratory techniques	4.11	3.78	3.96	3.80	3.92	3.76	4.00	3.78
Ability to read and understand primary literature	3.67	3.58	3.89	3.56	3.46	3.56	3.67	3.57
Skill in giving an effective oral presentation	3.78	3.43	3.68	3.47	3.35	3.43	3.60	3.44
Skill in science writing	3.47	3.27	3.70	3.21	3.73	3.21	3.63	3.23
Understanding how scientists think	4.11	3.55	4.02	3.57	3.73	3.55	3.95	3.56
Skill in interpretation of results	3.83	3.69	4.00	3.68	3.69	3.67	3.84	3.68
Learning to work independently	3.83	3.69	4.17	3.71	3.73	3.72	3.91	3.71
Becoming part of a learning community	3.74	3.61	3.96	3.61	3.68	3.62	3.79	3.61

^aNot all students completed the survey without skipping a question, and therefore the denominator for different questions may vary.

^bMean was calculated by averaging responses to a series of questions based on 5-point Likert scales, ranging from 1, strongly disagree to 5, strongly agree.

As shown in Figure 3, when surveyed about their plans post graduation to pursue a PhD degree in a science field, a MD, or a joint MD-PhD degree, 79% of 2015 OK-INBRE participants indicated plans to do so, compared with 63% of the national cohort. The gap narrowed slightly between the two cohorts in 2016, with 74% of the OK-INBRE students indicating plans to pursue one of these degrees, compared with 65% in the national cohort. Similar differences between the OK-INBRE and national participants remained in 2017, where 70% of the OK-INBRE students declared plans to pursue one of the advanced degree, compared with 60% of the national cohort. Furthermore, OK-INBRE students demonstrated higher interest in pursuing medical or combined medical degrees (46% in 2015, 59% in 2016, and 31% in 2017), than the national cohort (30% in 2015, 28% in 2016, and 27% in 2017).

FIGURE 3.

Intention to pursue advanced degree. Comparison between the percent of OK-INBRE participants who indicated plans to pursue a PhD, MD, or joint MD/PHD degree and that of students in the national cohorts in 2015, 2016, and 2017. INBRE = IDeA Network of Biomedical Research Excellence.

DISCUSSION

Similar to findings from other studies that examined undergraduate research programs, our study showed positive gains in all aspects of learning and outcomes among program participants. Over a 15-year period, the OK-INBRE has trained 473 individual summer students. Our descriptive statistics showed that participants in the OK-INBRE program are either continuing in their respective degree programs after their participation or have completed their degrees. Many are also pursuing or have completed further education and have careers in the sciences or biomedical field. These results are comparable with two other published studies reporting on outcomes of similar IDeA-sponsored undergraduate research programs from North Dakota and Kansas. Similar to OK-INBRE participants who completed undergraduate degrees in Oklahoma, where 80% of them are pursing or have completed a graduate degree (MS or PhD) in the sciences, 20% are pursing or have completed a MD/DO degree, and another 19% are enrolled in or have completed a health sciences–related degree, 21%, 34%, and 22% of research program graduates in North Dakota are pursing or have completed a PhD degree, are pursing or have completed a MD/DO degree, and are enrolled in or have completed a health sciences-related degree, respectively (25). In Kansas, 37% of research program graduates are pursing or have completed a PhD degree, 19% are pursing or have completed a MD/DO degree, and 12% are enrolled or have completed a health sciences–related degree (26). It is worth noting that North Dakota and Kansas programs were only able to report descriptive statistics on degree enrollment and completion outcomes, while the Oklahoma program was able to apply statistical analyses to determine whether these differences across categories were significant (25, 26). The OK-INBRE findings are also consistent with prior publications indicating that experiences in a research program can improve participation and persistence, often by strengthening students’ views of themselves as scientists (23). The empowerment and confidence derived from their participation in research programs may increase or reinforce the commitment students have to completing their degrees and pursuing graduate education.

Examining findings over the three years during which OK-INBRE collected detailed data on different levels of the learning experience has demonstrated the OK-INBRE summer program’s impact on participants’ learning, attitude, and behavior toward future attainment of higher degrees or careers in the sciences, as well as actual outcomes. Overall, Oklahoma students were highly satisfied with their research experience, indicating that the program exceeded their expectations. They showed learning gains ranging from understanding the research process to thinking critically and presenting research findings effectively. In addition, 70% to 80% of the students between 2015 and 2017 indicated their postgraduate plans were to pursue a doctoral degree in science, medicine, or a joint degree (i.e., MD/PhD). With the exception of a few responses, the absolute figures computed for each question were higher than those computed for the national cohort. Although statistical significance cannot be tested due to unequal sample sizes, these findings support the value of OK-INBRE in contributing to the training and development of the next generation of biomedical researchers and healthcare professionals for Oklahoma. They also lend quantitative feedback to OK-INBRE stakeholders and leadership that the program components are appropriately designed and delivered as well as provide needed student training in research and positive outcomes in continuing education in the sciences and career path.

While OK-INBRE students were generally more satisfied with their summer research experience than the national cohort, the proportions of OK-INBRE students who reported satisfaction on two program components in particular, seminars in which guest scientists discuss research and the final presentation of students’ summer work, were slightly lower than those of the national average. These components are areas that are being closely examined by OK-INBRE leadership and mentors to help identify and develop strategies for improvement. Two remedies we are introducing are to allow summer student participants to select from all seminar series on campus to attend as this may allow students to identify seminars that may be the most relevant to their research interests. A second change we are implementing is to have students evaluate seminars to identify specific items that they like or dislike about certain seminar speakers. The combined strategies should allow us to improve these important aspects of the OK-INBRE summer program. Our findings also indicated that OK-INBRE students may need more guidance on their final presentation. By implementing a rubric for all presentation components for the students and mentors to follow, we hope to increase satisfaction and understanding in this area in future summers.

One unique and noteworthy feature of our evaluation was the availability of longitudinal data with a well-devised and appropriate control group. In addition to providing insights into how the program has changed over time and demonstrating program success, this dataset included 100 students in a control group made up of students who: (i) had applied to the OK-INBRE program but chose not to participate, (ii) were selected as alternates but did not participate, or (iii) had similar GPAs, course background, and strong recommendation letters but could not be offered an OK-INBRE summer program experience due to lack of funds. The purposive sampling of the control group aimed to ameliorate some confounding effects related to academic performance to detect possible differences due to program participation. The comparisons from the bivariate analysis showed significant differences between students who participated in the OK-INBRE summer program and those who did not in both enrollment and completion of graduate/medical degrees after their BA/BS degrees were conferred. While pursuing and completing a degree is driven by multiple factors, our results demonstrate gains in pursuing and completing advanced degrees in science and biomedical fields specifically among OK-INBRE participants compared with their counterparts in the control group. While the bivariate analysis cannot establish causal relationships between program participation and outcomes, the results did show that participation in the OK-INBRE summer research program resulted in a significant increase in matriculation to graduate and medical school programs. As the dataset continues to track OK-INBRE students over time, we will be able to achieve sufficient power in the sample size to better estimate the association between participation and outcomes using multivariate modeling.

The statistical approach leveraging both primary and secondary data as well as the unique control group used in our study could potentially be used as a template for evaluating other summer programs aimed at providing research opportunities to undergraduate students, making our evaluation design largely generalizable. The data elements collected for the OK-INBRE summer program provide a comprehensive and quantitative measure of program effectiveness. Complementing the OSRHE outcomes data, SURE surveys capturing different dimensions of learning directly from student participants across the country beginning in 2015 allowed us to observe trends in scientific competency and career orientation among OK-INBRE student participants in more depth. Analyzing both primary and secondary data sources enhanced the comprehensiveness of the evaluation and offered a more accurate picture of the student learning experience and its impact on their academic and career pursuits. The primary data from SURE provided information directly obtained from students about their experience, learning gains, and intention to further their education and career. The secondary data from OSRHE lend the opportunity to track independently, based on reporting from educational institutions, progress in education attainment as well as actual enrollment and completion of graduate and medical degrees.

While there are many strengths to leveraging both primary and secondary data as well as a control group, this evaluation is not without limitations. First, as is common with any evaluation efforts, tracking down students to complete the survey is challenging. The OK-INBRE has instituted efforts with a dedicated staff to follow up diligently with student respondents and encourage survey completion. Second, although we were able to compare the absolute numbers on various aspects of learning between OK-INBRE and nationwide samples, the cohorts were not sufficiently balanced to test whether differences between OK-INBRE students and the national cohort were statistically significant. While this may limit the extent of the study’s impact, it identifies an area that warrants further research. Third, although we were able to test associations between OK-INBRE participation and educational outcomes, the data we had did not permit us to assess causation between these two variables, and we recognize that many factors contribute to educational success. Lastly, while OK-INBRE did not show any selection bias in terms of demographics (e.g., gender, race/ethnicity) or academic institution and performance (e.g., type of institution, GPA) between participants and non-participants, we recognize that there could be a self-selection bias of the applicant pool as a whole, compared with the general student population in Oklahoma. Taking the initiative to apply to a summer research program, regardless of actual participation, these students are likely to have been quite highly motivated and determined. Therefore, the magnitude of program effectiveness among students who are already motivated toward a career in the sciences may be slightly tempered, possibly creating a ceiling effect. However, it is important to emphasize that, despite the motivation demonstrated by applying to the OK-INBRE program, there was a 100% and 175% increase in the number of students attending medical and graduate school, respectively, among students who matriculated through the OK-INBRE summer program compared with the control group. These findings suggest that components and contents of the OK-INBRE summer research program are providing at least some benefits that are desired for enhancing the pipeline of students applying to and attending graduate and medical schools in Oklahoma.

Undergraduate research experience programs can provide an important opportunity in which students enhance their critical thinking, learn about the scientific process, and develop the knowledge and values that will guide their future scientific and professional careers. The growth and support engendered from participation in research programs such as the one offered by OK-INBRE are critical for identifying and maintaining a talented pool of young scientists, as students benefit greatly from direct interactions with mentors who oversee their projects (27). More importantly, the comprehensive evaluation components that have been built into the OK-INBRE program to track program effectiveness at different stages of the learning continuum is forward-thinking, as this may be the first opportunity to assess a program by comparing participants with a control group with similar demographic and academic profiles. Like results reported by similar programs in Kansas and North Dakota (25, 26), findings from the analysis of both primary and secondary data available provide evidence that the OK-INBRE program serves as one of the valuable resources for the state of Oklahoma in exposing students to the science and medical fields and motivating them to further their education in these areas. These efforts may help to build a new generation of scientists and healthcare professionals for Oklahoma.