Collaborative Assessment Of Collective Reach And Impact Among INBRE Supported Summer Undergraduate Research Programs Across The United States

Abstract

The motivational outcome of undergraduate research experiences is an increasingly common component of STEM education practices. Student benefits associated with these experiences include increased interest and retention in STEM and/or research fields. Across the country, many institutional research activities in twenty-three states and Puerto Rico are supported through the National Institutes of Health’s Institutional Development Award (IDeA) Networks of Biomedical Research Excellence (INBRE) Program. INBREs are statewide collaborations of research intensive and primarily undergraduate institutions that are designed to support the biomedical research pipeline as well as faculty research. Most INBREs offer summer undergraduate research experiences to meet their program goals. While the structure and focus of these programs are tailored to state-specific needs, they typically include 10–15 week sessions and many emphasize participation from underrepresented student populations. In summer 2019, eleven INBREs collaborated to explore the collective reach and impact of their summer undergraduate research programs (SURPs). A common set of survey items were identified and added to pre- and/or post-program surveys. These items focused on the reach of the programs (e.g. demographics of participating students) and the impact of the programs on educational goals for students. In total, data from 461 students across 11 states were included in the project. One third of participating students were from underrepresented racial/ethnic groups; 28% were first-generation college students and 34% were Pell grant eligible. After the program, 72% of participants reported that they hoped to earn a doctoral-level degree. Our results suggest that INBRE-supported SURPs are successfully reaching underrepresented students and that INBRE-supported students widely anticipate pursuing graduate level study in STEM fields.

Introduction

Undergraduate Research Experience

Mentored research experiences are an increasingly common element of undergraduate science education (Krim et al., 2019). For students majoring in STEM fields, success in graduate school may be more strongly linked to the number of research experiences as an undergraduate (Vincent-Ruz, Grabowski, & Schunn, 2018; Weiner, 2014) than quantitative measures like GRE scores. Continuous active and authentic engagement in research experiences throughout the undergraduate career is important for overall, long-term professional success, beyond successful matriculation into a graduate degree program (Altman et al., 2019; Krim et al., 2019). Exercises in inquiry ultimately benefit all students, not just those applying for advanced degree programs (Madan & Teitge, 2013).

One complication with assessing student outcomes of undergraduate research programs is that applicants may already be highly motivated to succeed in STEM fields. Indeed, some studies indicate that undergraduate research experiences primarily reinforce STEM interest ignited during high school years or earlier (Linn, Palmer, Baranger, Gerard, & Stone, 2015; VanMeter-Adams, Frankenfeld, Bases, Espina, & Liotta, 2014). Whether or not potential undergraduate students have pre-existing STEM interest, the undergraduate research experience offers an opportunity to explore possible career options, refine post-undergraduate goals, and develop scientific professional skills (McSweeney et al., 2018). Additionally, students who participate in research experiences often benefit from focused mentoring (McSweeney et al., 2018; Zydney, Bennett, Shahid, & Bauer, 2002), develop a better understanding of the iterative and collaborative nature of current STEM research (Linn et al., 2015; Madan & Teitge, 2013), and have a deeper engagement with coursework after their summer research experience (Lopatto, 2007). Furthermore, students who have participated in a previous undergraduate research experience are more likely to continue or participate in future undergraduate research experiences (Thiry 2012).

Some have noted that the period in which students engage in mentored research experiences is irrelevant to positive outcomes and that the key element is the amount of time invested (Linn et al., 2015; Russell, Hancock, & McCullough, 2007). However, summer research experiences, in particular, may provide a well-structured entry point for authentic, immersive laboratory work and mentoring (Howell, Wahl, Ryan, Gandour-Edwards, & Green, 2019). The relatively short time frames of these experiences are often designed to expose students to the full arc of the research process, beginning with basic skill development and foundational scientific principles, through engagement in journal clubs and scientific decision-making, finally reaching a cumulative experience such as a poster or oral presentation.

Numerous reports have identified disparities among students from underrepresented groups from those who are awarded degrees in STEM related fields. A recent commentary highlighted the “enormous mismatch between the number of underrepresented minorities living in the United States and the number who work in STEM fields” and acknowledged that it is largely driven by a lack of retention, rather than a lack of interest in these fields among students from underrepresented groups (Committee on Underrepresented Groups, 2011). To support the retention of students from these underrepresented groups, many undergraduate summer research programs at research-intensive institutions have a goal to increase diversity in STEM fields (Collins et al., 2017).

IDeA program overview

Part of the National Institutes of Health (NIH), the Institutional Development Award (IDeA) program was formed in 1993 to facilitate the distribution of NIH funding to areas of the United States (U.S.) that had historically received little funding (National Institute of General Medical Sciences (NIGMS), 2020). As of October 2019, 23 states and Puerto Rico are eligible for, but not guaranteed, IDeA program awards from among five different mechanisms. Many IDeA states, but not all, are predominantly rural and vary significantly in their land size, population density, and demographics. Alaska and Montana, for example, are two of the physically largest states in the U.S. but are ranked in the lowest five for the population (U.S. Census Bureau, 2012). Several of the seven western IDeA states have significant transportation burdens, in terms of traveling between and among IDeA institutions, because of geographic barriers (like mountains in Wyoming and islands in Hawai’i) or isolation. In contrast, Rhode Island has the smallest landmass but the highest population density in the U.S.

The IDeA Networks of Biomedical Research Excellence (INBRE) are statewide networks consisting of a lead (research-intensive) institution and partner institutions (primarily undergraduate institutions (PUIs), community colleges, tribal colleges, and academic medical centers) that together develop and expand research capacity and increase access to research infrastructure and resources for faculty and undergraduate students. The networks build and strengthen the biomedical research capacity of each state by supporting faculty mentoring and research and providing outreach activities and research experiences to undergraduate students while acting as a pipeline to health and research careers (Chou, Hammon, & Akins, 2019).

Across the country, there is variation in the composition of INBRE networks both in terms of the number and types of institutions involved in each state. For example, Kansas has a network of ten institutions consisting of three graduate institutions and six PUIs – all of which are public institutions with one Indian Nations University. Delaware has six partner institutions including two hospital systems, and the Alaska network includes four University of Alaska campuses, the Alaska Native Tribal Health Consortium, and an Alaska Native nonprofit health care organization – the Southcentral Foundation. The Wyoming network has one university and seven community colleges. More information about the composition of INBRE networks is available on the NIH website (NIGMS, 2020).

INBRE Summer Undergraduate Research Programs

Differences in demographics and geographical areas mean that each participating state has a unique network best suited to their educational needs. However, many similarities exist across INBREs, making collaboration possible, especially in program components which overlap across states. One such overlap exists with summer undergraduate research programs (SURPs), which are part of nearly all INBREs. While there is diversity in the structure and format of INBRE-supported SURPs across the country (See Table 1 for some examples), all programs seek to support the biomedical research pipeline within IDeA states. All programs include professional development components, mentorship from a faculty and/or senior researcher, training on the responsible conduct of research and/or research ethics, and an opportunity for students to share their experience via poster and/or podium presentations. Most states limit participation to those who either attend school in the state or originate from the state.

Table 1.

Description of participating INBRE programs and evaluation tools, Summer 2019

State	Description of Summer Program Focus	Students	Survey tool
Alaska	15 week program for ≤10 students (half placed at the University of Alaska Anchorage and half placed at the University of Alaska Fairbanks).	9	Email survey with embedded Google form
Delaware	10 week program for students placed across 5 sites (larger universities, small college, hospitals). Emphasis on serving students from underrepresented groups.	76	Qualtrics
Hawai’i	10 week program for students placed across 9 campuses (University of Hawai’i system, community colleges, 2 private PUIs). Summer is part of the option for year-long experience (Summer, Fall, Spring) – students apply every session.	67	REDCap
Idaho	10 week program for 51 students placed at 9 sites around the slate including 3 state universities (multiple sites), 1 public PUI, 2 private PUIs and 1 research foundation	42	SurveyMonkey
Kansas	10 week program across 10 institutions. The summer session is part of a year-long research experience but is not required. Students may begin new in the summer or continue from a previous semester. All students are undergraduates t in network universities, except students at the lead institution, which does not have an undergraduate program, so students are recruited from die large metropolitan area from out of network universities.	43	REDCap
Kentucky	10-week immersive research experience for 20 students. Students are accepted from any non-research-intensive college or institution (public or private) throughout the state. Students accepted into the program are placed in research labs at either the University of Louisville or the University of Kentucky based on their interests.	16	REDCap
Maine	10 week program for students placed at INBRE partners (PUIs and research institutions) and other affiliated locations in Maine and the Northeast IDeA region.	15	SurveyMonkey
Nevada	10 week independent summer research experience open to students from any of the 8 INBRE partner institutions and other Western INBRE programs. A new 5 week summer research experience is open only to students from the PUIs and has greater presence and instruction from the mentor to serve as a preparatory stepping stone to the 10 week independent research experience.	22	SurveyMonkey
New Hampshire	10 week program at each of 8 partner schools (including community college system) and at Dartmouth College for students from partner schools and other IDeA states who apply and are accepted.	113	Qualtrics
Rhode Island	10 week program at each of 5 network institutions for students enrolled in one of the institutions, plus a 10 week program at University of Rhode Island or Brown University for any U.S. undergraduate student.	65	Qualtrics
Vermont	9 week program for students placed at sites around the state including one land grant university, one pharmacy school, a Veterans Affairs Medical Center, and an IDeA state HBCU.	8	Lime-Survey
	Total Students	476

In addition, most INBRE programs focus on supporting students from underrepresented populations and/or those from primarily undergraduate institutions (PUIs). This focus is implemented in different ways depending on local and program context. For example, in Delaware, intentional efforts are made during the recruitment and participant selection processes to diversify the program. In New Hampshire and Vermont, programs partner with institutions that serve underrepresented groups, such as HBCUs, in other IDeA states to increase the diversity of participants. In Hawai’i and Nevada, the structure of the network itself increases diversity of students since students from their partnering community college system and PUIs are eligible for the program, and those institutions primarily serve underrepresented racial/ethnic populations. The variety of approaches to increase diversity employed by INBRE programs illustrates the program’s overall common goals while respecting the individualized nature of each state.

In light of these common activities and goals, a cross-state evaluation project to demonstrate the collective impact of INBRE-supported SURPs across the country was undertaken. The evaluation questions guiding this project are:

1. What is the reach of the programs (e.g. demographics of participating students)?

2. What is the impact of the programs on participating students’ education goals?

Methods

Instrument Development

A team composed of evaluators and program administrators from several INBREs guided the data collection efforts for this project. The group identified a set of common core survey items to be included in pre-and post-program surveys for summer 2019. Survey item choices were guided by a) an analysis of our existing evaluation tools to identify overlapping content areas that most INBREs were already assessing, and b) our two areas of evaluation focus: Reach and Impact. In the end, a common core consisting of four items related to Reach and two items related to Impact were chosen. The Reach items are (1) race, (2) ethnicity (using Census categories), (3) whether the student is a first-generation college student, and (4) whether the student is Pell grant eligible (used as an indicator of being educationally disadvantaged due to low family income). Since a key aim of INBRE evaluation models is to assess the “impact on students…from PUIs, community colleges, and TCCUs as a pipeline into the graduate science programs within the state” (NIGMS, 2020), we chose the following Impact items: (1) What is the highest degree you eventually hope to earn? (closed-ended), and for those expecting Master’s, Ph.D., or other doctoral degrees: (2) Do you anticipate pursuing an advanced degree in a science, biomedical or health-related field? (definitely yes, probably yes, might or might not, probably not, definitely not).

Data Collection and Analysis

Following the selection of the survey items, participating states revised their pre-and/or post-program surveys to integrate some or all of the items for summer 2019. Most programs use online/electronic surveys (e.g. Qualtrics, SurveyMonkey, REDCap (Harris et al., 2009)) that are distributed to participating students via email (Table 1). Some programs integrated the demographic (Reach) items into their pre-program surveys or student applications. The Impact questions regarding education plans were added to both pre- and post-program surveys if possible. At the end of the summer, de-identified data from each state were combined into one database for analysis. First, descriptive analyses were conducted on the data set (e.g. frequencies for each item). To adjust for the demographics of the participating states, an expected value was calculated using Census data for 18–24 year olds from participating states, weighted for the sample size for each state (U.S. Census Bureau, 2019). Data were analyzed using SPSS for Mac version 26 (IBM).

Results

Measure of Reach

In total, survey data from 461 participating students across eleven states were included in the project. One-third of participants who completed surveys were students from underrepresented racial/ethnic groups; 15% were Asian, 8% were African American and 11% were Hispanic/Latino. Nearly 35% of participants reported being Pell grant eligible and 28% were first generation college students (Table 2).

Table 2.

Survey sample of participating INBRE states, Summer 2019

	Number	Percent	Expected Value* Percent
Race
White	294	66%	69%
Black or African American	35	8%	9%
Asian	67	15%	9%
American Indian and Alaska Native	10	2%	13%
Native Hawaiian and Other Pacific Islander	13	3%
Other	25	6%
Ethnicity
Hispanic or Latino	47	11%	14%
Not Hispanic or Latino	398	89%	86%
First-generation college student
Yes	124	28%	-
Pell grant eligible
Yes	92	34%	-
No	91	34%	-
Don’t know	86	32%	-

^*Expected value was calculated using Census data for 18–24 year olds, weighted for the sample size within each state. The 13% includes: American Indian and Alaska Native, Native Hawaiian and Other Pacific Islander, and Other.

Measure of Impact

At the end of the program, 72% of participants reported that they hope to earn a doctoral-level degree (including PhD, MD/DO, MD-PhD). Of those indicating plans to pursue a Master’s degree or higher, 85% are aiming for a science, biomedical, or health-related field.

Discussion

These results suggest that INBRE-supported SURPs are successfully reaching a diverse population of students, including underrepresented ethnic/racial groups, first generation college, and low-income students. The majority of participants plan to pursue a doctoral-level or Master’s degree, and most plan to pursue that degree in a science, biomedical, or health-related field. Many students participating in these programs may be predisposed to pursue higher education, which underscores the need for a control or comparison group to better show impacts. Work to develop and refine best practices for these types of studies is a goal for many INBRE programs. One recent report from the Oklahoma INBRE program indicates that involvement by undergraduates in INBRE research programs positively affects the enrollment of INBRE alumni in advanced degree programs (Chou, Hammon, & Akins, 2019). On the whole, however, IDeA states tend to have fewer opportunities for research because by definition they have fewer highly productive biomedical research institutions and academic medical centers than in non-IDeA states. The focus and drive for INBRE programs is to therefore provide research training opportunities to maximize the number of students who choose to stay in our states for those experiences.

Thus, INBRE programs have the potential to make a substantial impact on improving the diversity of the biomedical research workforce in the U.S., more specifically in IDeA states. While, for this paper, we report impacts related to graduate school plans, we recognize the importance of evaluating the college-to-career pathway (without advanced education) that many INBRE students choose. Future work from these authors will incorporate the full scope of student plans so that we can describe a more complete picture of the INBRE-supported biomedical research career pipeline.

The process we used for this project is worthy of sharing, and offers several widely applicable implications for evaluation collaboration among NIH programs and assessing their collective impact across states. First, the project shows the impact when similar STEM training programs collaborate to demonstrate collective reach and impact. The Collective Impact Model is commonly described in the evaluation literature as consisting of five conditions: common agenda, shared measurement, mutually reinforcing activities, continuous communication, and backbone support (HanleyBrown, Kania, & Kramer, 2012). This project demonstrated these conditions by (1) exploring INBRE funded programs as a whole (common agenda); (2) merging our individual evaluation tools to determine commonalities among programs (shared measurement); (3) working together to ensure everyone understood the goals and steps and were representative of the questions at hand (mutually reinforcing activities); (4) frequently communicating to address changes or discrepancies, keeping in mind our goal to share findings and information with the NIH and the public (continuous communication); and (5) benefiting from the continued encouragement of the NIH to promote IDeA program evaluation and disseminate findings. While each participating INBRE conducts its own evaluation process, the collective approach of this project allowed us to explore whether and how INBRE-funded programs, as a whole, impact students.

Second, one common goal of the INBRE program, as described by the NIH, is to build the future biomedical research workforce, and this project shows how the programs collectively contribute to that goal. Many participants in our summer programs hailed from diverse, underrepresented backgrounds. The NIH has recently affirmed its commitment to broadening diversity in biomedical research fields stating, “NIH encourages institutions to diversify their student and faculty populations to enhance the participation of individuals from groups that are underrepresented in the biomedical, clinical, behavioral and social science” (NIH, 2019). The results of this study suggest that INBRE-sponsored SURPs contribute to this goal, and additional resources could be focused on further describing the impact of these programs on students from underrepresented communities.

Third, while INBRE programs are similar, significant collaboration across states has been relatively rare. Staff across programs have different backgrounds and work in unique contexts depending on the institutions involved. Programmatically, small groups of states have embarked on collaborative research projects within the context of INBRE. For example, IDeA states in the northeast collaborate on a project focusing on sequencing and annotation of the little skate genome (Wang et al., 2012). Both the western and northeast regions have a coordinated SURP application process, by which students from any state in the region can apply to and potentially attend a SURP in any other state. Even so, the collaboration illustrated with this current project is novel in that it spans most of the IDeA regions in the country. To build from the success of this project, a group is working to identify additional programmatic areas that are common across states (e.g. faculty mentoring) and may use the same approach to align evaluation tools and approaches in the future.

Lastly, this model was driven by program-level needs and context. The evaluators in the group decided to work together, chose metrics together, and collaborated on dissemination efforts. These decisions reflected both our cross-state goals but also the goals and context of our specific programs. In contrast, other cross-site evaluation processes have been driven by funders and/or external evaluation consultants retained by the funder. For example, the NIH funded Clinical and Translational Science Award (CTSA) mechanism underwent external evaluation that was coordinated by an external evaluator (Trochim, Rubio, Thomas & Evaluation Key Function Committee of the CTSA Consortium, 2013). The approach taken for this project reflects a novel approach that allowed us to work together and share data while also maintaining our uniqueness and independence.

While this project has many strengths, several limitations should be acknowledged. As mentioned in the introduction, students often choose these programs because they have a pre-existing interest in graduate study in STEM fields. Thus, the impact data reported here may represent a confirmation of a previous interest rather than a change as a result of the program. We are planning additional evaluation efforts to separate the impact of our programs on these intentions, as described above. Ideally, these evaluation efforts should include longitudinal follow-up with students. Second, all data are self-reported, and some items were not included on some INBRE’s surveys for 2019. These limitations are not uncommon in survey research but should be acknowledged. Third, participating INBRE programs provided aggregate data to the primary author who combined the top-level counts (e.g., number of responses per question). Individual (student-level) responses were not merged into a single dataset, so some analyses and cross-tabulations were not possible. This means that certain results are not available, like the percentage of first-generation college students who are planning to get a PhD. We are also not able to explore changes from pre- to post-program that would provide additional insight into impact.

Table 3.

Survey results from participating INBRE states, Summer 2019

	Number	Percent
What is the highest degree you eventually hope to earn?
Associate’s (shown to community college students only)	2	1%
Bachelor’s	23	6%
Master’s	74	20%
PhD	123	33%
MD or DO**	90	24%
MD-PhD**	41	11%
Other doctoral degree (e.g. DDS, PharmD, DVM)	14	4%
Some other degree not listed here	8	2%
Total	375
(If Master’s, PhD, or Other doctoral degree: n=211) Do you anticipate pursuing an advanced degree in a science, biomedical, or health-related field?
Definitely yes**	143	68%
Probably yes**	43	20%
Might or Might Not	8	4%
Probably not	3	1%
Definitely not	2	1%
No response	12	6%
Total	211
** Number of students planning an advanced degree in a science, biomedical, or health-related field, including MD/DO and MD-PhD degrees:
	317	85%

Biography

Laura Lessard, PhD, MPH is an Assistant Professor of Behavioral Health and Nutrition at the University of Delaware and Director of Evaluation for Delaware INBRE. In her role, she oversees the internal and evaluation processes for Delaware INBRE programming and initiatives. She has over 15 years of experience as an evaluator. Her training includes an MPH in epidemiology and biostatistics from Tufts University and a PhD from Emory University’s Rollins School of Public Health.

Christine Smith is the Director of Courses, Conferences, and Evaluation at the MDI Biological Laboratory in Bar Harbor, Maine. She has served as the Evaluation Coordinator for the Maine INBRE Program since 2011, and in this role works closely with Maine INBRE leadership to develop and implement a comprehensive evaluation strategy and report outcomes. Additionally, she is the Research Training Coordinator for the Maine INBRE, managing the program’s short course research experiences for undergraduates and community college students. She received her BA from Bowdoin College, where she studied English and Biochemistry.

Sharon O’Connor is a Senior Research Scientist at the Center for Program Design & Evaluation at Dartmouth College. She has 30 years of experience in research and evaluation and leads the evaluation design and implementation for several federally supported grants. Her training includes an MBA from Boston University and MS from The Dartmouth Institute at the Geisel School of Medicine, Dartmouth College.

Sarah E. Velasquez, MAB, MS, MLS is the Director of the Communications Core for the Kansas INBRE at the University of Kansas Medical Center in Kansas City, Kansas. She has been with K-INBRE since 2010, where she evaluates programmatic and collaborative activities, enhances operational and programmatic communications, and highlights the role of K-INBRE in bioscience education and research.

Julie Benson is the Program Administrator for the Alaska INBRE program at the University of Alaska Fairbanks. Julie earned a Bachelor of Science in Radio-TV-Film and a Master of Liberal Arts from Texas Christian University. She has 15 years of experience in research administration.

Jessica Garfield is the Director of IDeA Research Administration at the University of Nevada, Reno School of Medicine where she works with the Nevada INBRE, a Cardiovascular COBRE, and the Mountain West Clinical and Translational Research Infrastructure Network. She has 15 years experience working with federal grants that support educationally disadvantaged students, and 10 years experience in NIH Research Administration. Jessica’s training includes a BA from the University of California, Davis, an MA from the University of Redlands, and she is the only Certified Research Administrator (CRA) at the University of Nevada, Reno School of Medicine.

Jane Onoye, PhD is an Associate Professor in the Department of Psychiatry at the University of Hawai’i John A. Burns School of Medicine. She serves as the Evaluation Director for the Hawai’i INBRE and has over 15 years of experience as a researcher and evaluator for federal and state grants related to biomedical and behavioral research and infrastructure.

Linda Liou is the Evaluation Director and Program Administrator for the Idaho INBRE Program. She earned her Bachelor of Arts in German and Music from Dickinson College (1979). After studying graduate-level bacteriology at the University of Idaho and working as a research associate in microbiology labs there for nearly 20 years, she transferred to the Idaho INBRE Program in 2008.