Recruiting Students into Science: Evaluating the Impact of the North Dakota IDeA Network of Biomedical Research Excellence

The primary goal of the North Dakota IDeA Network of Biomedical Research Excellence (NDINBRE) is recruitment of students into science careers. The National Institutes of Health (NIH) support the development of research through IDeA (Institutional Development Awards) to academic institutions in 23 states. One important component of NDINBRE is building the research capacity of the four baccalaureate institutions in this rural state. North Dakota is sparsely populated, with 9.3 persons per square mile, compared with the national average of 79.6 persons per square mile (U.S. Census Bureau 2010). Recruiting students into science from rural high schools is particularly challenging, given common difficulties among rural schools with recruitment and retention of science teachers and limited access to resources (Monk 2007; National Comprehensive Center for Teacher Quality n.d.).

Initially, the NDINBRE program supported infrastructure development, including funding for research facilities and relationships among scientists at the four public baccalaureate institutions and the two graduate-degree-granting institutions in the state. With improved science facilities at baccalaureate institutions, attention then turned to increasing faculty and student research productivity while recruiting students into science. NDINBRE undergraduate students have participated in a variety of research activities, such as collecting plants and animals in the field and conducting laboratory experiments in cell and molecular biology, biochemistry, organic chemistry, microscopy, and physiology.

Measuring the effectiveness of the NDINBRE program’s efforts to recruit students into science is necessary to provide feedback for further program development and to answer the key question: “What impact does a science-intensive research experience have on students’ educational and career paths?”

First, the method used to evaluate the program’s operations and outcomes is described. The evaluation tracked students after their participation in the program. The method and results for tracking and evaluating these students provide insight into their outcomes and the impact the program has had on their educational and career paths. The discussion presents implications for future evaluation based on a larger conceptual framework and makes recommendations for testing components of that framework.

Research on Career Decision Making

Previous research has focused on various aspects of students’ educational and career aspirations, ranging from identifying influential factors such as family influences and the impact of secondary education (Gloria and Robinson Kurpius 2001, 95; Jolly, Campbell, and Perlman 2004, 7–8) to differences based on sex and race (McWhirter 1997; Scott and Mallinckrodt 2005, 268). Aspirations are generally defined as a malleable psychological outcome or construct that is influenced by a variety of contextual factors (Wang and Staver, 2001, 313). Grounded in social cognitive career theory (Lent, Brown, and Hackett 1996), research on career development investigates the behaviors, self-efficacy beliefs, outcome expectations, and goals that influence career choices (Mau 2003).

Additionally, a number of previous studies have demonstrated the impact of a research or educational program on students’ academic and career outcomes (Bauer and Bennett 2003; Gonzalez-Espada, Wilson, and LaDue 2006). However, those studies rarely address recruitment of students from rural, frontier, and tribal areas into careers in science. Given the recommendations of previous research for tracking students over time, the lack of longitudinal studies, especially on students in rural areas, and the need to evaluate the effectiveness of the NDINBRE program, the current research aimed to locate students who had previously or were currently participating in the program, determine their current educational and career status and future intentions, and investigate the impact the program has had on their educational or career paths.

Methodology

This study, using a staggered prospective multiple-cohort design (Fienberg and Mason 1985), addresses the research question: What impact does a science-intensive research experience have on students’ educational and career paths? Current and former participants were asked to describe: (1) their experiences in the program, (2) their current educational and career status and aspirations, and (3) the program’s impact on their education and career.

The primary criterion for participation in INBRE is that a student be enrolled in a rural academic institution; all of the four baccalaureate institutions participating in NDINBRE are located in rural areas. Initially, a list was created of all students who had participated in the program since its inception in 2003, drawing from a variety of data sources: college faculty members’ student lists, conference participation lists, and financial records of the program’s payments to students. The list was revised further based on feedback from students and faculty. Faculty at each of the four baccalaureate institutions confirmed the list of students. At these four institutions, 180 students participated in the program; 74 (41 percent) provided in-depth information during the collection of data.

Instrumentation and Data Collection, Management, and Analysis

Tracking students

Longitudinal data collection is the most accurate way to report program outcomes and efficacy, especially when considering changes that may take place over a period of time. Maintaining accurate contact information is essential for longitudinal data collection, particularly when the sample includes individuals likely to relocate, such as young adults who often move to pursue educations and careers. Although many studies investigate the impact of an educational experience by following students over time (e.g. Pugnaire, Purwono, Zanetti, and Carlin 2004), information on the methods for finding and tracking students after high school is rarely articulated.

Over the past twenty years, a growing and rapidly changing body of literature has provided recommendations for increasing the effectiveness of telephone surveys as a data collection tool. Such literature has evolved as technologies, such as cell phones and caller ID, have increased, and authors now recommend multiple calls at varied times of the day over a relatively brief period of time. For example, in a review of the literature, Sangster (2003, 2–3) found that the best time to call was evenings and weekends, with only one weekday call during the day time in the first five attempts. More recently, Carley-Baxter, Triplett, Evans, et al. (2006) confirmed this conclusion for adults, but found that the best times to contact college students differed. Calling during the workweek, either day or night, was more successful in reaching college students than calling on the weekends.

The current study employed a variety of methods to identify and contact students, including faculty contacts, visits to colleges, attendance at workshops where students presented, phone, e-mail, mail, and social networking sites such as Facebook. A letter explaining the project and inviting participation was sent to all students for whom addresses were available. When possible, students were contacted via phone, as that is the desired method for completing surveys because of the accuracy and speed allowed. Inquiries of faculty members at the baccalaureate institutions and phone calls to the students’ permanent addresses, usually their parents’ homes were also successful methods for locating students. In some cases when accurate contact information (phone or email) was not available, Facebook or Internet searches were fruitful.

Data collection

Approval by the participating colleges’ institutional review boards (IRBs) was obtained prior to data collection. Students were invited to complete a survey collecting academic, employment, productivity (publications and presentations), demographic, and contact information. Students also reported their academic majors and degree information, if relevant, and their employment history. Students described their experience, for instance, who their mentor was, the type of compensation they received, and activities in which they participated. They were asked an open-ended question regarding the impact the program had on their academic and professional careers. The survey concluded with a request for recommendations for further development of the program. The same questions were asked whether the survey was completed by phone, e-mail, mail, or in person, although the format varied slightly according to the method of collection.

Students remembered and reported their research productivity inconsistently, that is, how many articles, presentations, and posters in which they had participated; therefore, an additional step was taken to obtain this information. All faculty members at the baccalaureate institutions were asked to create a list of products on which students were co-authors and/or for which NDINBRE resources were used.

Data management and analysis

Information from the surveys was recorded in an ACCESS database designed specifically for this study, which allowed easy data entry and confidential storage. Qualitative information was transferred into a Microsoft Word table for content analysis. First, phrases, the unit of analysis, were identified in each comment through an iterative process. Two researchers independently isolated the phrases, then compared and reached agreement about them. Because one comment could have contained several phrases, the total number of phrases is greater than the number of students who made comments.

Then, categories were developed through a similar iterative process, during which the two researchers developed the initial list of categories by coding a sample of comments independently. They compared results, revised the categories, and recoded independently until the final coding scheme emerged for each question. The categories in the final coding schemes were mutually exclusive; each phrase could only be coded in one category. The total number of phrases is reported for each question, frequencies and percentages are calculated by category, and comments for each category are provided, to explicate the meaning of the category.

Findings

To answer the question about the impact of a science-intensive research experience, students were asked to provide information about opportunities afforded them by NDINBRE, what their current educational and career status was, and their opinion about the impact of the program. They also were given the opportunity to make recommendations for changes in the program.

Opportunities

When asked to note opportunities available to them as a result of the program, students commented positively on their opportunities for research lab experience and general research experience (see Table 1). They also commented positively about their faculty mentors.

Table 1.

Opportunities Provided by the NDINBRE Program as Reported by Participating Students

Opportunity	N	%	Example Comments
Research Experience (lab)	33	3.0	I know way more about research than I would have had INBRE not been around.
Research Experience (general)	32	31.7	Gave me a chance to see research and I really liked it.
Faculty Mentors	15	14.9	Dr. XXX was EXCELLENT to us as students. Allowed the professors to be more available for research. Ended up getting more one-on-one, deeper level of attention.
Dissemination of Research	8	7.9	Got to travel to different meetings; travel and networking at meetings. I presented at those conferences, which was a big deal for my resume when I applied to graduate school.
Financial Assistance	7	6.9	It allowed me to earn money. Paid well.
Organization of Opportunity	5	5.0	It was well organized.
Collegiality with Fellow Students	1	1.0	I also enjoy working with fellow students on certain projects.
Total Responses	101	100

Although only a few students’ comments identified the opportunity to disseminate research as an important opportunity, network students and faculty members were prolific, creating 352 products, including posters, presentations, abstracts, and articles. The most common products were posters (see Table 2). Faculty and students presented at local, regional, and national venues, including the 100th Annual Meeting of the North Dakota Academy of Science, the 12th Annual American Society for Biochemistry and Molecular Biology Annual Meeting, the 1st Biennial National IDeA Symposium of Biomedical Research Excellence, the CUR Conference, the 2010 CUR Posters on the Hill, the 236th National Meeting of the American Chemical Society, American Association of Pharmaceutical Scientists, American Indian Consortium, the Society for Toxicology Annual Meeting, and the Louisiana State University Health Sciences Center Graduate Student Research Day.

Table 2.

Products Resulting from Students’ Research

Type of Product	N
Poster	216
Presentation	94
Abstract	17
Article	22
Grant	3
Total Products	352

Education and Career Status

Of the 64 students who had graduated from one of the four primarily undergraduate institutions, 47 were pursuing or had obtained an additional degree (see Table 3). Almost half (n = 21) were in health professions programs. Three had graduated from an advanced program; one was in a post-doctoral position, one worked as a medical illustrator, and one was a veterinarian. Academic institutions attended include private and public institutions in ND, Iowa, Louisiana, Arizona, Kansas, Minnesota, Arizona, Oregon, Washington, Wisconsin, Canada, and England.

Table 3.

Current Status of Graduates (Applying or Enrolled in Another Program)

Status	N	%	Type of Program
Applying to graduate program	2	3.1
Applying to health professions program	3	4.7	Medicine, physical therapy
In graduate program	18	28.1	Archeological conservation, biochemistry, biomedicine, chemistry, microbiology, molecular biology, psychology, public health, science
In health professions program	21	32.8	Chiropractic, dentistry, medicine, nursing, optometry, pharmacology, pharmacy, physical therapy, physician assistant, social work, veterinary science
In other professional program	1	1.6	Law
In undergraduate program	2	3.1	Science
Not pursuing a graduate degree	17	26.6
Total	64	100

Program Impact

The NDINBRE program had an impact on students’ employment and academic aspirations, as well as them personally (see Table 4). The three major themes, each had three sub-themes. Forty-two percent of the comments highlighted the influence that the program had had on employment, primarily career aspirations. Many students indicated that the research experience prepared them for graduate school. Students mentioned that they enjoyed the science laboratory experiences and appreciated the increased confidence they had gained, especially with presentations at conferences. Participation in the program also influenced students academically, through enriching their current experience and their future career choices. Finally, the experience positively impacted students’ knowledge, skills, self-confidence, and enjoyment of research.

Table 4.

The Positive Impacts of the Network Research Experiences

Themes	Subthemes	N	%	Examples of Comments
Career	Influenced career aspirations	26	23.9	Changed major to clinical lab science because of it. I was at a point where I picked a major when I didn’t think I could pick one.
	Current science-based employment	12	11.0	Job wise, the internship helped me get into a lab position that I am in now.
	Improved career opportunities	8	7.3	Opened up job opportunities, contacts.
Academic	Future academic aspirations and opportunities	20	18.4	It helped me gain experience and allowed me to get insight into what graduate school would be like in learning various techniques.
	Enhanced awareness	11	10.1	It showed me a different side of education that I hadn’t been exposed to before. I never knew about doing research under a professor.
	Enhanced academic experience	10	9.18	It was to actually get involved in research that was beyond a textbook setting, developing protocol, conducting experiments that haven’t been done.
Individual Impact	Increased skills	15	13.8	Learned a lot of applications and processes beyond class work.
	Confidence	5	4.6	Good because of knowledge and confidence.
	Enjoyment	2	1.8	I enjoy the research part of science more.
Total		109	100

Suggestions for Program Changes

Students made recommendations for program development. Most were about the need for increased resources (see Table 5). Other issues included better communications, increased recruitment efforts, the organization of the experience, the need to inform students more about the bigger picture, and increased opportunities, such as making research presentations. It is important to remember that student experiences occurred over a range of years, from the early 2000s to the present, and at several institutions. Therefore, some of the recommended changes may have already been implemented since the early years of the program.

Table 5.

Recommendations for Change

Themes	Subthemes	N	Examples of Comments
Resources	Lab	8	More money for lab I was working in.
	Student funding	2	Not enough grant money for students to be paid. Had to work, didn’t have time to participate.
Communication	Cross-institution collaboration	7	A little more intercommunication between principal investigators and the administrators of the program. Cooperation of the different research programs. You could compare each other’s research.
	More student input	1	More input from the students.
Recruitment		7	Wish more students from the reservation would participate in it.
Organization of experience		6	We went through all the stuff, but each of us was given a specific, different task so we weren’t told about the process. We just learned about the specific part. I didn’t know really what we were doing overall.
Flexibility in research topics		6	I would try to broaden and expand the topics.
Increased participation	More group work	1	Maybe more group stuff.
	More meetings & presentations	4	I would have liked to be able to present.
Duration		2	Try to make it extend over entire school year.
Nothing		31	I don’t think I would change anything.
Total		75

Discussion

Overall, students were pleased with the opportunities they had while participating in the NDINBRE program, especially the opportunity to learn research skills and to learn from faculty mentors. Students were very productive and were able to present at local, regional, and national conferences and publish articles. A high number of the participants who had graduated from their baccalaureate institution continued their education in a science or health-related field. The program’s positive impact on students’ confidence, their current academic success, future career aspirations, and employment was clear. The few suggestions for change centered on the need for even more resources; increased cross-institutional collaboration; recruitment of students, especially American Indian students; and the way in which their particular experience was organized. Taken together, these results support the positive impact of the program and the fulfillment of its goals. The recommendations for change provide guidance for examination of the current program to identify ways to further enhance students’ research experiences and the impact of the program.

Not all the former participants have been located. Although attempts will continue to be made to locate them, this will be an ongoing issue, as with all longitudinal studies. The use of new technologies, such as Facebook, should increase our capacity to achieve a high response rate. The initial methodology did not include a mechanism for determining what factors other than the NDINBRE program might have influenced students. Therefore, one cannot rule out the possibility that other factors besides this program might have influenced students’ career choices. The evaluation does not have a comparison group; given the circumstances of the program, neither a national or local control group is a possibility. However, as additional data are collected, the prospective cohort analyses of this research design will compensate for the lack of a control group.

Further Research

Students participating in the NDINBRE program will be surveyed annually. The number of methods used to locate each student will be recorded, to better determine the most successful tracking methods. Based on information gained from the process of data collection and student and faculty input, the next step will be to address the question, what factors, in addition to the program, might have influenced students’ academic and career paths. Some research suggests that factors in three areas (engagement, continuity, and capacity) are needed for student success (Jolly, Campbell, and Perlman 2004; Symonette 2008, 1). In this “trilogy model,” engagement refers to individual student factors such as interest; capacity reflects individual ability; and continuity reflects the impact of external institutions and opportunities. Future plans to combine this “trilogy model” with the bioecological approach (Bronfenbrenner and Ceci 1994), which considers factors at various levels of the social ecology, such as individual, family and societal factors, will allow a better delineation of the impact of the program on students. As the broader body of research regarding the impact of recruitment efforts grows, comparisons can be made across types of programs and regions regarding successful outcomes.

Biographies

Pat Conway, PhD, LCSW, is the evaluator for the North Dakota IDeA Network for Biomedical Research Excellence (NDINBRE). She conducts program evaluation of health-related programs. Conway is the editor of the Journal of Family Social Work.

Bridget L. Hanson, PhD, a 2010 graduate of the doctoral program in experimental psychology at the University of North Dakota, coordinated the evaluation of NDINBRE while a doctoral student. Her primary research interests involve the study of attributions in health and legal contexts.

Donald A. Sens, PhD, is the principal investigator of the National Center for Research Resources’ NDINBRE program and the National Institutes of Environmental Health Sciences’ STEER program.

Van Doze, PhD, is an associate professor and director of the NDINBRE-initiated summer undergraduate research programs at the University of North Dakota. He also is the principal investigator of an NSF-funded Research Experiences for Undergraduates Site program and serves on the editorial board of the Journal Advances in Physiology Education.

Karen Cisek, MS, currently is responsible for program development for NDINBRE. She assists with program design, logistics, execution, and evaluation. Prior to this position, she worked as the outreach coordinator for a neuroscience research program.

Thomas P. Gonnella, PhD. is a principal undergraduate research investigator in the NDINBRE program. His research group focuses on using fluorescence techniques to examine NADH-dependent enzyme systems.

Heidi Super, PhD, received her doctorate in molecular genetics and cell biology from the University of Chicago and completed a three-year position as a post doctoral research associate at the National Institutes of Health’s Rocky Mountain Labs in Hamilton, Montana.

Jennifer Ann Boeckel, MSW, currently is an evaluation specialist while she is completing a PhD in social work. Prior to returning to school she worked in the foster care arena in North Dakota and Minnesota.

Jennifer Wages, MSW, is a project coordinator in the Department of Psychology at Wake Forest University. She has worked as an evaluator for health-related projects since her graduation from the University of North Dakota.