An Approach to Improving Student Success in Science, Technology, Engineering, and Mathematics (STEM) Career Pathways

Abstract

In this article, we report on an 11-year study that explores approaches to improve student success in college by a five-week summer program in Mathematics and Language Arts for entering freshmen. To recruit students into the program, we invited students accepted at the university and listed as underrepresented and economically disadvantaged (Pell-eligible) by the Office of Institutional Research at California State University, Northridge. The program consisted of all-day Math and English enhancement in mixed ability groups. Results of this program examining Math and English performance at California State University, Northridge showed that students completing the summer programs during the 11-year study period had improved pass rates in Math and English at California State University, Northridge compared with students in a control group who did not participate in the summer program. The results show that intensive pre-college enhancement for entering freshmen can improve student success in college. Student graduation data from the early cohorts (2010, 2011, 2012) were obtained from Institutional Research. The summary results showed that students from the accepted/attending group had substantially increased GPAs and graduation rates, essentially closing the achievement gap. Increased interest in biomedical research careers was also developed by the program, as demonstrated by a five-fold number of summer enrichment participants entering the PhD, MARC (Minority Access to Research Careers) and RISE (Research Initiative for Scientific Enhancement) programs than students who did not attend summer enrichment.

Introduction

The US needs a more diverse biomedical workforce, one that is committed and prepared to address pervasive disparities in health outcomes and access to evidence-based health care within the African American, Native American, Hispanic and Pacific Islander communities.1-4

Improving preparation in mathematics among young scholars is one of the many initiatives that policy advisory groups and researchers5,6 have recommended to broaden participation of underrepresented minority students (URMs) in the Science, Technology, Engineering, and Mathematics (STEM) career pathways. Gatekeeper mathematics courses that undergraduates take at the beginning of their college training are important in determining who will be prepared to master the content of higher-level STEM course work and ultimately which students will persist and graduate with a STEM degree.7

The evidence points to disparities in performance between majority and URMs in gatekeeper math courses and this has been recognized as one cause of higher proportions of undergraduate URMs switching out of biomedical science majors before graduation. The root causes of the obstacles URM students face with respect to success in mathematics are many, including: inadequate pre-college preparation8; economic pressures resulting in long hours working off-campus, interfering with the ability to devote the necessary hours to study9; not having realistic expectations about the hours of study outside of the classroom that will be necessary to succeed, and preparation in proper study skills, or other tools for success10; psychological reactions such as stereotype threat that interfere with performance in high stakes testing situations11; or a lack of confidence in one’s efficacy to perform math tasks resulting from a lack of previous positive experience or external motivators that might lead one to think they would perform well.12

Whatever the sources of the challenges URM students face, we believe that they are offset by the enormous reservoir of strengths our students display: the motivation to work hard and succeed; a powerful curiosity to explore the natural world; a desire to use professional skills to give back to the community in which they grew up; and the desire to become role models who help the next generation of URM scholars.13

The MARC/RISE programs at California State University Northridge are designed to prepare a pool of underrepresented biomedical research students (URS) for advancement to graduate studies and careers in academia, government and the private sector. For the purposes of this report, we define URS as the ethnic/racial groups included in the URM designation as well as economically disadvantaged students—those eligible for Pell grant funding, who were included in the program eligibility criteria starting in 2009. With nearly 40,000 students, CSUN serves a large and extremely diverse group of undergraduate students in an urban/suburban setting. CSUN is an Hispanic-serving institution (HSI) and an Asian American and Native American Pacific Islander-serving institution (AANAPISI).

A university analysis of incoming CSUN freshman revealed an achievement gap, ie, greater need for remediation and lower proficiency levels in math and English for URMs compared with White/Non-Latino and Asian students (http://asd.calstate.edu/performance/remediation/17/Rem_Nor_Fall2017.htm).

Student success in subsequent college level work also revealed performance gaps (https://www.csun.edu/counts/current_undergraduate_students.php).

Because overall GPA is a key determinant for inclusion and participation in student development research programs at CSUN and other national programs, the Summer Math and Language Arts Enrichment Program (EP) grew out of conviction that with proper preparation, incoming URS freshmen could achieve success in math and English and then qualify to join our comprehensive biomedical research-oriented training program (MARC/RISE) or other special opportunities such as research internships or on-campus research opportunities.

Methods

Study Population: Participant Recruitment and Selection

From EP inception in 2006, the CSUN Office of Institutional Research (OIR) supplied contact information for potential candidates using these criteria: incoming freshman; URS, majoring in a basic biomedical science field; and, a high school GPA of 3.0 or above. After two years when the preliminary data showed improvement in first semester outcomes, the project was expanded to include Pell-eligible students in all subsequent years. They were supported by CSUN. Letters inviting program applications were sent to all qualified candidates, asking for a high school transcript, personal essay, and a letter of recommendation from a teacher or counselor. Applicants were accepted into EP based on high school science and math coursework and preparation, the strength of the recommendation and career interests focused on research careers rather than clinical care as stated in the personal essay.

Comparison Groups

The selection process allowed for four naturally occurring comparison groups: EP participants (the Treatment Group [TG]); Control 1, students accepted into the program but who never attended a session or who dropped out in the first days; Control 2, students who applied for the program but were not selected. One study reported here includes data for a third segment, Control 3—all other biomedical science students regardless of demographic background who entered CSUN as contemporaries of previously mentioned groups during the years 2010 to 2012.

Although the TG and Controls 1 and 2 were not randomly selected, we found them to be fairly equivalent in terms of incoming GPA, demographic background, and motivation. Control 1 is perhaps the best available control given they were initially accepted into the program. Many of the Control 2 students were turned down due to a preference to train for a health care pathway, the coursework for which still requires college-level coursework in mathematics.

Program Design

The program builds upon past programs that attempted to solve the conundrum of poor college preparation, particularly in math.14 The program consisted of five summer weeks of daily workshops, with three days devoted to math enhancement, one to English enhancement and one for field trips to science centers and marine environments. Students were dropped from EP by missing or being late to one session, which we believe is an essential ingredient for program effectiveness.

The program included pep talks introducing the benefits of research careers and PhD degrees in biomedical science. The field trips were designed to build friendships and a sense of belonging to a community of like-minded students. Students successfully completing the program received a certificate of recognition and $600-$700 payment.

Math Component

This component of the program focused on foundational high school math concepts, including algebra, math analysis, trigonometry, two-and three-dimensional geometry, and complex number properties. We attempted to stoke participants’ imagination through real-world applications of abstract mathematical concepts. Examples are geometry rules applied to engineering puzzles or probability concepts applied to gambling scenarios. Less emphasis was placed on mechanical execution of steps, and more on mathematical intuition. We demonstrated numerical “tricks” to add, multiple, divide or subtract large numbers without a calculator.

A diagnostic was administered the first day to establish baseline performance. Three 3-hour lectures per week were delivered, punctuated by hourly drills to reinforce learning. During the afternoon, tutors worked on assignments with smaller groups formed according to pace of learning and performance level. Homework was given along with simulated grades to maintain the commitment. When EP students needed remediation in foundational math concepts, we provided strategies to help them accelerate to the college level. Their roadblocks were diagnosed by observing them execute problem-solving steps outlined in class.

Language Arts Component

This part of the program built on academic reading and writing skills, while promoting intellectual curiosity. Students completed diagnostic exercises once per week to determine skill levels and instructional needs. Students were encouraged to read and take notes between sessions and to cultivate autonomous reading practices. Readings consisted of one assigned reading and one reading that each student chose based on their academic interests.

Sessions included short lectures, followed by practice exercises, focused on annotation, summarizing texts, paraphrasing ideas, citing evidence, synthesizing information, and audience awareness. Emphasis was placed upon development of the skills needed for writing. Afternoon sessions involved tutor-guided peer activities in mixed ability small groups that focused on sharing ideas and fostering community. Students discussed articles read outside of class, which was meant to facilitate critical thinking. Advanced students learned by assisting their peers, and less advanced students were encouraged to push themselves, thus providing a community that supported growth for all participants. Finally, the students completed timed writing scenarios, which applied previous lessons and knowledge to a genre-specific writing task. A one- to two-page article was read, followed by the drafting of a two- to three-page, hand written essay. We directed students to work on focus, clarity, development, organization, and synthesis of information when writing. The instructor gave students feedback in writing and orally. A multiple draft system for academic writing was implemented so that the students could learn to revise for cohesion and clarity once ideas were in place.

Data Collection Methodology

We report the results for four approaches to gathering evidence regarding the effectiveness of the EP. Study 1 was included as part of the original design of the program, while the research evolved over the years to include additional institutional and attitudinal data. All four studies were approved by the CSUN institutional review board and all procedures followed were in accordance with the ethical standards of the IRB and the Helsinki Declaration of 1975, as revised in 2000. Informed consent was obtained from all participants included in Studies 3 and 4. The CSUN IRB determined that it was not necessary to obtain informed consent from Study 1 and 2 participants because the analysis involved de-identified data. The external evaluator (Author 2) who conducted the analysis did not have access to data that would identify students on the data files.

Study 1

We tracked first semester mathematics and English course enrollments and outcomes for EP participants and Controls from EP inception in 2006 to 2016. At the conclusion of EP each year, the EP program coordinator provided student ID numbers to the CSUN OIR, indicating to which of the three groups each student belonged. In turn, OIR created a data file for the external evaluator containing the requested information, flagging group membership. No individually identifiable data were included in the file. Analyses were conducted using the most current version SPSS available that year. The 11 data files were combined for the purposes of the summative analysis. Relative risk statistics were calculated by entering study results into an on-line calculator (https://www.medcalc.org/calc/relative_risk.php)

Study 2

Using the same de-identified data procedures described above, OIR supplied grade point average (GPA) and enrollment/graduation status data for the 2010-2012 cohorts to the external evaluator on one file. The data were analyzed using the current version of SPSS at the time. The data for the three cohorts were analyzed in 2018, which gave the 2013 cohort a five-year period in which to complete a degree. The GPA data for a given student reflected either their final GPA at graduation or their then current GPA after the fall 2017 semester, if still enrolled.

Study 3

An on-line attitude survey measuring dimensions related to academic self-efficacy and math confidence was adapted from an established format.15 Starting in 2011, we sent emails inviting Treatment and Controls 1 and 2 students to complete the survey in the beginning week of EP program (pre) and again in the month following the conclusion of the program (post). The final survey database included program years 2011 to 2016, corresponding to the last six years of Study 1. The response rates for each group varied. Control 1 was dropped from the analysis because their pre-treatment response rate was quite low at 11%. Response rates for the two remaining groups were: Treatment, pre (45%); post (64%); Control 2, pre and post, 29%.

Due to IRB stipulations, we could not track the identities of respondents, so it was not possible to link a student’s pre- and post-surveys. Thus, each group might have had a somewhat different composition pre vs post. Students were incentivized using a $10 e-giftcard fulfilled through a second website linked to the anonymous survey.

The next step was to reduce the survey items down to composites using the reliability analysis function built into SPSS. For example, analysis revealed that 13 items related to math confidence (MathC) were highly inter-correlated, allowing us to compute a MathC composite score. Composite scores were computed for two other factors: research career outlook, eg, to what extent would a career involving hands-on research be rewarding; and academic self-efficacy, eg, to what extent were students confident in their academic abilities.

Study 4

Program staff gathered internal information regarding feedback from EP students and calculated numbers of EP and Control students who went on to become trainees in the CSUN MARC/RISE initiatives.

Results

Study 1

During the 11-year period, 90% of EP participants took a fall math course and 85% passed (Table 1). About half of the EP students took a math course geared to the college level, the rest were enrolled in a remedial math course. The results for EP participants compared favorably with both control groups. Relative risk (RR) statistics16 were calculated. Comparing Treatment to Control 1, RR (the chances of an EP student improving relative to a control) was 1.08 (P<.14). This represented a somewhat weak showing but the “Number Needed to Treat” (NNT) for an additional participant to pass a fall math course revealed the most telling information. Comparing Treatment vs Control 1 pass rates, the NNT was 15. In other words, for every 15 EP program participants, one additional passing grade in math was obtained. Extrapolating over the lifetime of the program, 20 additional students passed math. Given the low number of URS participating in research-based biomedical career pathways at CSUN, we consider this to be an administratively significant number. Looking at fall English course pass rates, the EP students did well, but not significantly better than controls.

Table 1. Fall Math and English course outcomes.

Treatment & comparison groups	Group size	Percent who attempt Fall Math	Percent who pass Fall Math (grade “d” or better)	Percent of Math courses at college level	Percent who pass Fall English (grade “d” or better)
Treatment group	308	90%	85%	53%	96%
Control 1	137	86%	78%	46%	93%
Control 2	214	85%	73%	34%	94%
Tests of statistical significance		Chi Sq=4.1, P<.13	Chi Sq=9.3, P<.01	Chi Sq=15.7, P<.001	Chi Sq=1.9, Not Significant

Study 2

The graduation and GPA analysis (Table 2) indicated that EP students graduated at higher rates than the three comparison groups and had maintained higher GPAs. We cannot conclusively claim that participation in EP was the cause for these positive results, but the results support the investment made in EP students to help prepare them for biomedical career pathways.

Table 2. Cumulative GPA and graduation rates, 2010 to 2012 Cohorts.

Group	n	GPA	6-yr graduation rate
Treatment group	89	2.85	68.5%
Control 1	24	2.59	62.5%
Control 2	49	2.54	49.0%
Control 3	2194	2.55	50.8%
TOTAL	2356	2.56	51.5%
Tests of statistical significance		ANOVA F=3.9; df between groups=3; df within groups=2332; P<.003	Chi Sq=12, P<.007

Study 3

While there was no pre-treatment difference in math confidence (MathC) between the EP group and Control 2, we found some evidence that MathC increased pre to post among EP students, but not among Control 2 students. However, the average MathC post-treatment ratings for both groups were within the same confidence interval range, indicating that the effect size was not great enough to conclude that participation in EP had made a noteworthy difference. None of the other scales registered significant increases comparing EP students relative to Control students. We believe it is important to report these results as an indication that more work is needed, which we discuss in the next section.

Study 4

In 2016, 20% of EP participants reported a possible interest in a biomedical PhD at the start of the program. At the end of five weeks, the percentage more than doubled to 48%. This gain is typical of what we have seen over the years. The number of EP participants into the MARC and RISE training programs is further evidence of success. Since inception, 21 of 308 EP participants (about 7%) entered MARC or RISE. As of this publication date, two are working toward biomedical PhDs, one completed medical school and one had completed a master’s in public health and was applying to earn a biomedical doctorate. Eight of the 451 Control 1 and Control 2 students (about 2%) also entered MARC or RISE.

Discussion

Given the need for URM students to enter and succeed in pathways leading to careers in the biomedical sciences, and the pervasive under-preparation these students experience at earlier stages of their educational development, it is critical that we as a nation continue to invest in interventions that that will help to ensure their academic success. College educators face a threefold challenge: obtaining the funding needed to launch and experiment with the necessary interventions; adopting research strategies that have the power to illuminate evidence-based best practices; and ultimately determining which program elements drive interventions that are effective in increasing math and language proficiency.

The research we present in this report illustrates partial, yet admittedly not complete solutions to this three-part challenge. We were successful in obtaining external and university funding to launch the EP initiative as well as the collaboration across departmental lines to involve faculty members from critical disciplines in the design and implementation of the program. We worked with an external expert in educational program evaluation to develop as rigorous a research approach as field conditions would permit in order to investigate whether the program was effective. Finally, the results we present here indicate that the program appears to accomplish our underlying objectives of helping students attempt college level math courses, pass their freshman math course, keep their GPAs reasonably high—at least in comparison with controls—and persist in college through to graduation. While there was weak support for the notion that program participation would yield psychological benefits in terms of confidence, we did not find the strong effects we had hoped for. Meanwhile, the program appeared to be promising in increasing interest among participants in joining undergraduate biomedical research training initiatives.

Implications for Future Research

If anything, the results strengthen our resolve to continue this work. We see ways in which our team and other practitioners and researchers could continue to refine this and similar efforts. 1) It would be beneficial to explore a program element aimed at generating psychological impacts such as confidence-building, self-efficacy and confronting the challenges some students face with phenomena such as stereotype threat. 2) Further integrating program participants into the campus community by helping them build a campus-based support network could help preserve the gains achieved during what is now a stand-alone summer entry program. 3) Finally, as resources allow, we see the need to adopt early warning detection systems during the freshman year so that we can intercept and assist students who otherwise might spiral into frustration and failure when taking math.

Study Limitations

The limitations of our current research also have implications for future research that our team might conduct, or others working with similar interventions. These are as follows: 1) The external evaluation focused on program outcomes rather than examining various elements of the intervention as part of the evaluation design. 2) A more formative approach to evaluation in which the evaluator works more closely with the instructors might yield insights into ways the day-to-day elements of the program could be strengthened. 3) It is unknown whether the absence of stronger findings supporting the effectiveness of the program in producing psychological benefit was because these elements were not emphasized in the program design or because we need to find measures that are more sensitive to the benefits that do occur. 4) As with most field studies, tighter experimental controls would give us more confidence in our ability to interpret the results. Further exploration may find opportunities for creating either a true randomized control trial if there were more qualified applicants than positions available or a stronger quasi-experimental design such as a regression discontinuity experiment.17 5) Finally, working with the National Student Clearinghouse would permit us to check for subsequent graduate school enrollments among the Treatment and Control groups.