The St. Jude STEM Clubs: An Afterschool STEM Club for Upper Elementary School Students in Memphis, TN

Abstract

Informal science, technology, engineering, and math (STEM) programs are important tools for broadening participation in STEM careers. The St. Jude STEM Club (SJSC) is a 10-week afterschool STEM club focused on real-world problems in pediatric cancer research and designed for students in the fifth grade. The SJSC is conducted in partnership with the Shelby County Schools (SCS), an urban school district that encompasses Memphis, TN and serves a disproportionate number of students from underrepresented backgrounds in science. In this report, we provide details on the club logistics, curriculum, pilot data and outcomes related to club impact on student attitudes towards science, and challenges and limitations of the program. Participants in the program reported significantly higher rates of STEM engagement, STEM identity, critical thinking, perseverance, and relationships with peers and adults compared to national normative data. This program description is intended to serve as a resource for other institutions wanting to use a similar strategy to broaden participation in STEM careers.

INTRODUCTION

In general, women and people of color, specifically Black/African American and Latino/Hispanic individuals, are underrepresented in science and engineering fields, including degrees awarded and careers (Beede et al., 2011; Hrabowski et al., 2011; U.S. News, 2015; NSF, 2019). For example, in 2016, women received around 20% of all degrees awarded in physical science (across all levels of degrees) and between 20–25% of degrees awarded in engineering. Additionally, 60% of all bachelor’s degrees in science and engineering fields in 2016 were awarded to White/Caucasian students, whereas 12% were awarded to Black/African American students, and 10% to Latino/Hispanic students (NSF, 2019). Only 4% of bachelor’s degrees in engineering were awarded to Black/African American students. These disparities become even more exaggerated at higher degree levels and in careers; about 70% of full-time scientists or engineers in 2016 were White/Caucasian, and about 8% were Black/African American and 8% Latino/Hispanic (NSF, 2019). These statistics indicate that equity in science and engineering education is still an unmet goal.

Research shows that accomplishment in science, technology, engineering, and math (STEM) careers is related to the frequency and number of STEM learning opportunities early in life (Wai et al., 2010). STEM learning occurs in multiple settings, including: the formal classroom, at home with families, in public libraries, out-of-school-time (OST) experiences such as afterschool clubs and summer camps (Afterschool Alliance, 2015), and even on vacation (Falk and Dierking, 2010). OST STEM programs are particularly important for broadening participation in STEM careers, helping to make STEM subjects more inclusive and creating sustained interest and participation in STEM disciplines (NRC, 2015; Afterschool Alliance, 2015). Research shows that OST STEM programs have the potential to increase STEM content knowledge (Bhattacharyya, 2011; Blanchard, 2015; Mouza, 2016; Newell, 2015; Tyler-Wood, 2012); foster STEM skills and attitudes (Barker et al., 2014); increase understanding and perceptions of science and provide opportunities and skills missing in formal education, such as job skills and use of scientific equipment, as well as increase self-concept and empowerment (Fadigan, 2005); and create environments where young people can engage in activities connected to their interests, be positioned as leaders, and increase their view of science (Gonsalves, 2014). Despite these findings, the need for OST STEM programs remains high, especially in areas of concentrated poverty where accessibility and affordability can often be a barrier to participation (Afterschool Alliance, 2016).

In this report, we describe the St. Jude STEM Club (SJSC), a 10-week afterschool STEM club focused on real-world problems in pediatric cancer research. We provide details on the club logistics, curriculum, pilot data and outcomes related to club impact on student attitudes towards science and challenges and limitations of the program. This program description is intended to serve as a resource for other institutions wanting to use a similar strategy to broaden participation in STEM careers.

PROGRAM DESCRIPTION

Overview.

The SJSC is an initiative of the St. Jude Comprehensive Cancer Center (SJCCC) Cancer Education and Outreach Program. The club aims to increase students’ science identities and critical thinking skills through hands-on projects and challenges related to real-world problems in pediatric cancer research.

School Partners.

The SJSC is conducted in partnership with the Shelby County Schools (SCS), an urban school district that encompasses Memphis, TN. SCS serves over 100,000 students in 206 k-12 schools (Tennessee Department of Education, 2018). The largest racial group in SCS is African American (76%), followed by Hispanic (14%), white (10%), other (4%), and Asian (2%) (some students may be listed in more than one group) (SCS, 2020). About 7% of students are classified as English language learners and 56% meet the qualifications for economically disadvantaged status (SCS, 2020). In total, the SJSC host clubs at 21 elementary schools that collectively reflect the overall demographics of SCS, but individually show variation, ranging from schools with 67% to 100% Black, Hispanic, or Native American Populations and from 23% to 86% students from economically disadvantaged backgrounds (Table 1). The Principal at each of the 21 school sites agrees to provide adequate space and technology for club meetings and identify a sponsor teacher.

Table 1.

Demographics of Fall 2019 Student Participants.

Gender Distribution
	School District (N = 106,377)	Partner Schools (N = 13,471)	SJSC Fall 2019 Cohort (N = 110)	Instructional Facilitators (N = 4)
Female	49%	49%	59%	75%
Male	51%	51%	35%	25%
Prefer not to answer	-	-	12%	-
Racial/Ethnic Distribution
	School District (N = 106,377)	Partner Schools (N = 13,471)	SJSC Fall 2019 (N = 91)	Instructional Facilitators (N = 4)
African American, Black	77%	67%	66%	50%
Latino or Hispanic	15%	22%	11%	-
White, Caucasian	7%	9%	5%	25%
Other	1%	2%	2%	25%
Prefer not to answer	-	-	16%	-

Sponsor Teachers.

There are no requirements for who can serve as a teacher sponsor other than being a teacher at the school. In many instances (79% of sponsors), the sponsor is a science or STEM teacher, but this is not required, as we did not want to limit our program to schools with a science teacher willing and/or able to host a club. The teacher sponsor is responsible for facilitating parent communication ef forts, assisting in classroom management during each meeting, and overseeing student dismissal. SJCCC compensates teacher sponsors for their time since afterschool activities are considered an addition to their regular work hours.

Instructional Facilitators.

Undergraduate and graduate STEM majors from local universities are selected to serve as instructional facilitators for the SJSC. The selection committee is intentional to ensure that the instructional facilitators reflect the racial demographics of SCS students. Each of the facilitators is assigned to four school-sites each semester, meeting with all four schools one day each week (Mon-Thurs) for ten weeks and agrees to commit to the entire program. Prior to program implementation, instructional facilitators receive a multi-day formal training on the curriculum (described in detail below).

Logistics.

Each school site hosts one club session each year in either the fall or the spring, with the exception of three schools that host one each semester to a different cohort of students due to increased student demand. Students are only allowed to attend one 10-week club session. The clubs meet once a week for ten weeks and are open to all fifth-grade students on a first-come-first-served basis, serving a maximum of 20 students each for a total potential reach of 480 students per year. Parents are asked to commit to their child(ren) attending the SJSC each week. On-going parent engagement efforts are made to prevent participant attrition, such as providing parents with a weekly newsletter to serve as a reminder for upcoming meeting dates and inform them of what their child(ren) is learning and obtaining parent feedback.

Each club session is designed to last 60 minutes and is conducted similarly (same weekly focus and activities) across all school sites. Participants are dismissed during regular school dismissal to the sponsor teacher who meets them at the designated location on-site at the school. After the session is over, participants are either picked-up by a parent/guardian or returned to their regular after care programming provided by the school.

Curriculum Design.

The curriculum for the club was developed by the SJCCC Cancer Education and Outreach team in collaboration with St. Jude scientists and local science educators and utilizes a combination of case-based (Herreid, 1994; Herreid, 2013) and project-based (Caprero et al., 2013) learning strategies. In addition, lessons are generally aligned to the Next Generation Science Standards (NGSS) (NGSS Lead States, 2013), incorporating the science and engineering practices and crosscutting concepts in addition to content. The Cultures of Thinking pedagogy (Ritchhart, 2015) and associated thinking routines (Ritchhart et al., 2011) are embedded within the curriculum as tools to make student thinking visible as they engage in the practices of science and engineering. See Table 2 for examples of think ing routines and how they support the practice of science and engineering within the SJSC curriculum.

Table 2.

Description of Select Thinking Routines used in the SJSC Curriculum and Their Connections to the SEPs.

Thinking Routine	Steps (Ritchhart, et al., 2011)	Corresponding SEPs (NRC, 2012)	Description	Use in the SJSC Curriculum
Think-Puzzle-Explore	What do you think you know about this topic? What questions or puzzles do you have on this topic? How might you explore the puzzles we have around this topic?	Constructing explanations and designing solutions to problems Asking questions (for science) and defining problems (for engineering) Planning and carrying out investigations	This routine illustrates how scientists activate prior knowledge to help construct initial explanations for their problem, use these initial explanations to ask targeted questions, and design investigations to either prove or answer their questions and either prove or disprove their initial explanation.	During the case study phase of the SJSC curriculum, students are introduced to Stacey, a young girl exhibiting signs and symptoms of osteosarcoma. Students use the Think-Puzzle-Explore routine to activate their prior knowledge of the human body, develop initial explanations for what is wrong with Stacey and brainstorm possible diagnostic tools for exploring Stacey’s health further.
Notice-Think-Wonder*	What do you notice? What do you think about that? What does it make you wonder?	Analyzing and interpreting data Constructing explanations and designing solutions to problems Asking questions (for science) and defining problems (for engineering)	This routine illustrates how scientists and engineers make careful observations by focusing students’ attention on the thinking generated by the data collected through their five senses. Through this intentional analysis of data, scientists begin to construct and/or refine initial explanations for their problem and enter into a new line of questioning that arise from the data.	This routine is used several times throughout the case study phase of the curriculum. Students use the Notice-Think-Wonder to compare Stacey’s X-ray, bone biopsy, and PET scan results to a normal, using the data to continually refine their initial explanation.
Claim-Support-Challenge	Make a claim about the topic, issue, or idea being explored. A claim is an explanation or interpretation of some aspect of what is being examined. Identify support for your claim. What things do you see, feel, or know that lends evidence to your claim? Raise a question related to your claim. What may make you doubt the claim? What seems left hanging? What isn’t fully explained? What further ideas or issues does your claim raise?	Engaging in arguments from evidence Using mathematics and computational thinking	This routine illustrates how scientists and engineers engage in arguments from evidence by focusing students’ attention on evidence to provide validity to a claim. It also encourages students to consider other possible explanations that can be derived from the evidence.	During the case study phase of the curriculum, students use the Claim-Support-Challenge routine to diagnose Stacey’s disease, using evidence they obtained through the analysis of the x-ray, bone biopsy, and PET scan.

Learning in the club unfolds over the course of ten weeks through two phases: the case study phase and the engineering design challenge phase. During the case study phase, student participants are encouraged to take on the role of scientist and physician to analyze a patient case related to osteosarcoma, a cancer of the bone, and explore the challenges associated with developing effective treatments for pediatric cancer patients (Figure 1). The challenges presented at the end of Phase 1 set the stage for the engineering design challenge in Phase 2. During the engineering design challenge, students take on the role of biomedical engineer and are given a design challenge to create a prosthetic hand that is able to pick-up a ping pong ball and place it into a cup (Figure 2). During the final session, students participate in a reflective exercise to consider how their experience in the club has shifted their thinking on who scientists are and what scientists do. Table 3 provides a detailed outline of individual lessons and learning objectives for each of the 10 sessions.

Figure 1.

Case Study Logic Model.

Figure 2.

Engineering Design Challenge Logic Model.

Table 3.

Outline of the St. Jude STEM Club Curriculum.

Session	Lesson/Activity	Description	Learning Objectives	SEPs	CCCs
1	Draw a Scientist	Draw and describe a picture of a scientist.	Discuss what science is and who can be scientists
2	Introduction to Stacey	Read through a patient story about a young girl named Stacey who begins to experience persistent pain in her knee.	Compare and contrast a normal vs. Stacey’s x-ray and biopsy Develop a claim, evidence to explain Stacey’s pain	Asking Questions and Defining Problems Engaging in Argument from Evidence	Patterns Cause and Effect
3	Stacey’s Diagnosis	Learn that Stacey has osteosarcoma, a cancer of the bone.	Compare and contrast PET scans Consider the challenges of treating cancer	Analyzing and Interpreting Data Constructing Explanations	Patterns
4–5	Prosthetic Design Challenge	Learn that many patients with osteosarcoma require prosthetics after receiving treatment and are challenged with designing a prosthetic hand.	Analyze the structure - function relationship for parts of the hand Consider possible materials for a prosthetic hand Design a prosthetic hand	Analyzing and Interpreting Data Planning and Carrying Out Investigations Defining Problems Designing Solutions	Structure Function Systems and System Models
6–9	Build Days	Using their design, build prosthetic hand.	Test and refine their prosthetic hand design Present their final product to the class	Designing Solutions Obtaining, Evaluating, and Communicating Information	Structure Function Systems and System Models
10	Student Reflections	Reflect upon their progress throughout the club.	Identify barriers to success Reflect on science/science identities	Defining Problems	Systems and System Models

Curriculum Training.

Curriculum training is facilitated by the SJCCC Cancer Education and Outreach team. Instructional facilitators are required to participate in a 2-day cur riculum training workshop. Workshops include an introduction to case-based teaching and learning and project-based learning strategies, an overview of materials management, and training on how to facilitate a classroom culture devoted to the science and engineering practices.

The training session begins by informing the trainees that the lesson will take them and their students through different scenarios resembling the real-world experiences of a pediatric cancer patient from the rise of initial symptoms, to the diagnosis, and finally to treatment. Next, trainees are introduced to the thinking routines utilized throughout the lesson. The training facilitator stresses that the goal of these thinking routines is to make the students’ thinking visible. Specifically, when students work in their groups on any specific thinking routine the facilitator should encourage visible thinking through questions to students such as “Why do you think that?” or “What makes you say that?” Trainees are informed that it is okay for students to generate incorrect answers when they first complete each thinking routine as long as they are working to make their thinking visible. Student misconceptions are often resolved or posed as future questions to be explored further in the larger group discussion led by the facilitator that follow each thinking routine. Once the trainees are familiarized with the pedagogical approach, the training facilitator leads the trainees through the lesson with the accompanied PowerPoint as if the trainees were the students.

Throughout program delivery, instructional facilitators attend weekly professional development sessions with the SJCCC Cancer Education and Outreach team on Fridays. Instructional facilitators share what happened at their sites during the week and any challenges they had, such as participants arriving late. Upon reflection, the group will then brainstorm ideas for the following week to address identified challenges. When necessary, the program leadership team works collaboratively with the teacher sponsor and school administration to determine the best solution.

FRAMEWORK FOR LEARNING

Overview.

The framework for learning in the SJSC is rooted in the constructivist learning theory. According to this theory, learning is an active process in which new learning builds upon prior knowledge and is guided by social interactions within the learning environment (Dewey, 1938; Vygotsky, 1978; Oliver, 2000). The role of the teacher in the constructivist classroom is to create a collaborative, student-centered learning environment (Oliver, 2000) using carefully scaffolded activities to facilitate learning (Copple and Bredekamp, 2009). In the SJSC, student thinking is scaffolded and made visible using thinking routines outlined in the Cultures of Thinking pedagogy (Ritchhart et al., 2011; Ritchhart, 2015). Through this lens, the SJSC is intentionally designed to promote the development of positive science identities and 21st century skills, specifically critical thinking, perseverance, and relationships with students and adults.

Promoting STEM Engagement through Narrative.

Previous research shows that the use of narrative is a valuable tool in science education, making abstract concepts more memorable (Norris et al., 2005; Browning and Hohenstein, 2015; Prins et al., 2017). During phase 1 of the program, students are introduced to Stacey, a young girl close in age to the program participants. Through the narrative, we learn small details about her life. She enjoys running on the cross-country team and has a stepfather who she turns to for support when she begins to experience pain in her knee. Her race and ethnicity are intentionally left out with the hopes that each individual student will use their imagination to “fill in the blank” with a race or ethnicity they find most relatable. The idea being that she could be their classmate or even a friend. These seemingly superfluous details are intended to draw upon the students’ sense of empathy, giving them motivation to persist through the story and develop an understanding of the science behind Stacey’s condition. Will she be ok? Will she survive? What will happen to Stacey?

Overlaying the thinking routines onto the narrative encourages students to engage more deeply with the content material than they might otherwise do, allowing students to move between reading, thinking, writing, and sharing more effectively. This same strategy is carried over into phase 2 of the curriculum in which students are challenged with designing a prosthetic hand. From Stacey’s story, students find a purpose for creating the prosthetic hand along with the necessary motivation to persist through the design challenge.

Promoting STEM Identity Development through Performance.

Frameworks of identity and scientific identity development are used in various ways to understand the experiences of students in science programs as well as to explore how students come to see themselves as a “science person,” which is viewed as a necessary step to pursuing science in future educational and career trajectories. This is particularly important for populations who have been underrepresent ed in scientific careers; for example, African American girls are likely to report a high level of interest and engagement in science during middle school, but are unlikely to want to pursue science as a career (Hanson, 2008). In education research, identity is generally conceptualized as having two principal components, performance and recognition (Gee, 2001). Science identity performance, providing authentic opportunities to use the appropriate tools, pose and research their own hypotheses, and become comfortable using the language of science, is an important part of developing a science identity (Lave and Wenger, 1992; Seymour et al., 2004; Hurtado et al., 2009; Carlone and Johnson, 2007). This idea is intentionally and explicitly woven throughout the SJSC curriculum.

The first session of the SJSC focuses on what scientists do and who scientists are. The session begins by asking students to draw an image of what they think of when they think of a scientist and describe what their scientist does on a typical day. Students then conduct a gallery walk to view all of the scientists produced by the class and look for similarities and differences across all of the scientists. Once the gallery walk is completed, the instructional facilitator guides the students through a conversation to identify and dispel misconceptions related to what the practice of science looks like and stereotypes about who can be a scientist. The end result of this conversation is designed to ensure students that anyone can be a scientist and that they, in fact are scientists. To reinforce this concept, each student is gifted a lab coat to wear throughout the SJSC and keep once the program is finished.

Through both the Case Study phase and the Engineering Design Challenge phase, instructional facilitators are trained to explicitly highlight moments when the students take on the role of a scientist to perform a task, being careful to name and identify the practices of science they are doing. For instance, when students are presented with an x-ray of a normal knee to compare to Stacey’s knee, the facilitator begins by stating, “A radiologist is the type of scientist who carefully examines x-ray images to look for abnormalities. Radiologists have to look with the careful eye of a scientist to make observations in patient x-rays and identify any differences that exist when compared to an x-ray from a healthy patient. Now, we are going to take on the role of the radiologist to see if there is anything that is different from the normal x-ray.” Students then work in small groups to conduct the Notice-Think-Wonder thinking routine (Ritchhart et al., 2011; Table 1) to identify the similarities and differences between the two x-rays and discuss their thinking and wonderings about Stacey’s x-ray before reporting back to the class.

By making thinking visible, students are able to develop their metacognition (Ritchhart et al., 2011) and, thereby, become aware of how the practices of science are implemented throughout the curriculum. For instance, the Notice-Think-Wonder routine described above encourages students to think about scientific observation as more than just what they sense (see, smell, taste, feel, hear) in the world, but also the thoughts and wonderings triggered by their senses. Students keep track of their thinking in their workbook and continually reflect back to previous thinking in order to build and refine their explanation for what’s wrong with Stacey’s health (Figure 3).

Figure 3.

Case Study Storyline.

Promoting Critical Thinking and Perseverance through Productive Failure.

The SJSC curriculum is intentionally designed for productive failure. Consistent with the constructivist theory of learning, productive failure is a learning strategy that presents a problem prior to the student’s having the formal knowledge necessary for solving the problem (Kapur, 2008). In this process, students are guided to first activate prior knowledge and apply it to the problem as they develop solutions or explanations before the facilitator reveals the canonical knowledge necessary to fully understand the problem.

During the Case Study phase of the SJSC curriculum, students are introduced to Stacey, a young girl presenting with the signs and symptoms of osteosarcoma. Students are not expected to know what osteosarcoma is, let alone be aware of the signs and symptoms of the disease. Rather, their lack of knowledge regarding pediatric cancers is leveraged at “the reveal” to explain why pediatricians often misdiagnose osteosarcoma. Instructional facilitators are trained to explain, “Like you, many pediatricians also misdiagnose osteosarcomas. This is because pediatric cancers, including osteosarcomas, are very rare and they often present with symptoms similar to more common problems, like growing pains in Stacey’s case. In fact, most pediatricians go their entire career and never have a patient with pediatric cancer.” In this way, students are likened to a pediatrician as people who both make mistakes and learn from failure.

Productive failure is also a key component of the engineering design process. During the engineering design challenge phase, students are required to enter into an iterative design cycle in which they design, build, try it out, and make it better (Figure 4). Instructional facilitators are trained to support students when their prosthetic fails to pick-up the ping pong ball and place it in the cup by encouraging them to reflect upon what worked, what didn’t work, and how they can improve their design, which students then reflect on in written format. This conveys the message that we learn as much or more through our failures than our successes.

Figure 4.

Engineering Design Cycle.

Promoting Relationships with Peers and Adults through Collaborative Learning.

The practice of science requires that scientists develop the skills necessary for collaborating with other scientists and communicating their science effectively. For this reason, the SJSC utilizes collaborative learning strategies to promote positive peer relationships (Meneski and Chi, 2019; Asterhan et al., 2014; Gijlers and de Jong 2013; Roseth, 2008). Group discussions are scaffolded using carefully chosen thinking routines to help activate students’ prior knowledge before entering into group discussion (Ritchhart et al., 2011; Ritchhart, 2015). These strategies are embedded throughout both the case-study phase and the engineering design phase of the program curriculum. Instructional facilitators are trained using the Cultures of Thinking pedagogy to create a learning environment in which individual and group thinking are valued, visible, and actively promoted (Ritchhart, 2015). In this approach, the teacher’s role becomes interactive, rooted in open with students. Thus, the goals of the constructivist learning environment to create a space where teachers and students share knowledge and authority over the learning process are actualized (Tam, 2000; Honebein, 1996).

DATA COLLECTION AND ANALYSIS

Overview.

The evaluation and impact of this program was conducted by an independent researcher in STEM education at the University of Memphis (co-author on this paper). Evaluation and impact was done using a mixed methods approach that included student retrospective self-change surveys and responses to open-ended questions. Study information was reviewed and approved by the University of Memphis Institutional Review Board.

Student Retrospective Surveys.

A retrospective post-then-pre design was chosen to avoid pretest sensitivity and response shift bias that results from pretest overestimation or underestimation (Howard, 1980; Rockwell and Kohn, 1989; Pratt et al, 2000; Lam and Bengo, 2003). On the final day of the program, student participants completed the Common Instrument Suite (CIS), a retrospective, self-change survey (Allen, 2019; Allen, 2017; Martinez et al. 2014). During the survey, participants were asked to intentionally consider how their answers to each prompt have changed as a result of participation in the program, on a scale of “Much Less Now” to “Much More Now.” The survey was used to determine the program’s impact on students’:

• STEM engagement (14 items; α = 0.91);

• STEM Identity (5 items; α = 0.88); and

• development of 21^st century skills, including critical thinking (5 items, α = 0.79), perseverance (4 items, α=0.85), relationships with peers (4 items, α = 0.74), and relationships with adults (4 items, α = 0.74).

Each item is scored on a scale of 1 to 5 with 1 representing students who responded “Much Less Now” and 5 representing students who responded “Much More Now.” The mean score across each domain was calculated by The PEAR Institute at Harvard Medical School and McLean Hospital and provided through a Qualtrics data portal along with national normative data. National normative data represents students across the nation who participated in various OST STEM activities and are assumed to self-selected into the programs, making them an ideal comparison group as students in the SJSC select to participate (Allen, 2019).

A modified two-sample t-test was used to compare the SJSC program data to national normative data. The modified two-sample t-test used the sample size, mean, and standard deviation of the St. Jude data and the sample size, mean, and theoretically largest possible standard deviation for the national data. Thus, any significant result from the two-sample t-test is truly significant because use of the actual standard deviation for the national data would give a larger t-statistic and smaller p-value.

Student Open-Ended Questions.

In order to contextualize the results of the CIS, we asked students to provide a written response to the following prompt: “Was your prosthetic successful? Why or why not?” These responses were qualitatively coded based on the CIS categories, specifically focusing on the development of the 21st Century skills of persistence, relationships, and critical thinking.

RESULTS

Overview.

In this report, we present pilot data on the impact of the program across each domain of the CIS survey along with qualitative data related to student perceptions of the program. The data presented here includes all participants; results by gender and race followed similar trends. A more detailed analysis of program outcomes will be presented in a later publication as we continue to expand the number of student participants, including a stratified analysis of the data by gender and race/ethnicity. Permission to use the CIS data was obtained from The PEAR Institute at Harvard Medical School and McLean Hospital. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of The PEAR Institute, Harvard Medical School, or McLean Hospital.

Participant Demographics and Attendance Rates.

During the Fall of 2019, the SJSC was conducted at 11 schools by 4 instructional facilitators. In total, 151 fifth-grade students participated in the clubs with an average weekly attendance rate of 67% across all sites with available attendance records. Individually, the school sites showed variation in participation and attendance, ranging from schools with 3 to 21 participants and 38% to 83% attendance rates (Table 4). Ninety-five percent of students participated in at least 7 out of 10 sessions. Demographics for each individual club matched that of the hosting school-site. See Table 1 for aggregate demographic data of SJSC participants and instructional facilitators.

Table 4.

Student Participation Numbers and Average Attendance Rate by School Site.

School Site	Total Participants	Average Attendance Rate
1*	21	-
2	12	78%
3	17	83%
4*	20	-
5*	13	-
6	20	76%
7*	21	-
8	3	38%
9	9	67%
10	9	56%
11	14	78%
Total	151	67%**

^*Attendance data is not available.

^**Limited to schools with attendance data.

Results from Common Instrument.

This section presents analysis of data across all club sites and compared to national normative data for STEM Engagement, STEM Identity, Critical Thinking, Perseverance, Relationships with Peers, and Relationships with Adults.

STEM Engagement.

The STEM Engagement scale on the CIS consists of 10 items that measure students’ interest and excitement in participating in STEM (e.g. I like to participate in science projects). Of the participants in the SJSC who completed this scale (N=110), 96% reported positive change in their overall STEM Engagement, 0% reported no change, and 4% reported negative change (Figure 5). From the analysis of the two-sample t-test, SJSC participants reported significantly higher rates of increased STEM Engagement compared to national normative data (p < .001) (Table 4).

Figure 5.

Percentage of Student Participants with Positive, No, and Negative Change Across the CIS Scales

STEM Identity.

The STEM Identity scale on the CIS consists of 7 items that measure students’ understanding of themselves as a person who can do STEM and be in STEM (e.g. I think of myself as a science person). Of the participants in the SJSC who completed this scale (N=105), 82% of students reported positive change in their overall STEM Identity, 7% reported no change, and 11% reported negative change (Figure 5). From the analysis of the two-sample t-test, SJSC participants reported significantly higher rates of increased STEM Identity compared to national normative data (p < .001) (Table 4).

Critical Thinking.

The Critical Thinking scale on the CIS consists of 5 items that measure students’ examination of information, exploration of ideas, and independent thought (e.g. I like to think of different ways to solve problems). Of the participants in the SJSC who completed this scale (N=109), 96% of students reported positive change in their overall critical thinking, 2% reported no change, and 2% reported negative change (Figure 5). From the analysis of the two-sample t-test, SJSC participants reported significantly higher rates of increased critical thinking compared to national normative data (p < .001) (Table 4).

Perseverance.

The Perseverance scale on the CIS consists of 4 items that measure students’ persistence in work and problem-solving despite obstacles (e.g. I keep working even if it takes longer than I thought it would). Of the participants in the SJSC who completed this scale (N=109), 94% of students reported positive change in their overall perseverance, 1% reported no change, and 5% reported negative change (Figure 5). From the analysis of the two-sample t-test, SJSC participants reported significantly higher rates of increased perseverance compared to national normative data (p < .001) (Table 4).

Relationships with Peers.

The Relationship with Peers scale on the CIS consists of 4 items that measure students’ positive and supportive connections with friends and classmates (e.g. I have friends who care about me). Of the participants in the SJSC who completed this scale (N=106), 83% of students reported positive change in their overall relationships with peers, 6% reported no change, and 10% reported negative change (Figure 5). From the analysis of the two-sample t-test, SJSC participants reported significantly higher rates of increased relationships with peers compared to national normative data (p < .001) (Table 4).

Relationship with Adults.

The Relationship with Adults scale on the CIS consists of 4 items that measure students’ positive connections and attitudes towards interactions with adults (e.g. There are adults who are interested in what I have to say). Of the participants in the SJSC who completed this scale (N=108), 88% of students reported positive change in their overall relationships with adults, 8% reported no change, and 5% reported negative change (Figure 5). From the analysis of the two-sample t-test, SJSC participants reported significantly higher rates of increased relationship with adults compared to national normative data (p < .001) (Table 4).

Results from Open-Ended Survey Question.

Results from the open-ended survey question were used to support the findings of the Common Instrument. Overall, most students reported that their prosthetic hand was successful (71 out of 110 responses). Forty-two of these responses explained that their hand was successful because it met the goals of the project: it was able to put a ping pong ball in a cup. However, a smaller number of responses indicating that the hand was successful reflected 21st Century Skills measured by the Common Instrument. Twenty students said their hand was successful because of the decisions made and processes followed by the group (critical thinking), for example, “It was successful because we noticed that our fingers were to big so we cut them as the size of our groups.” Six responses referred to the team working well together (collaboration/relationships with peers), for example, “The group prosthetic hand was successful because we all worked together.” Four responses indicated the team was persistent with their work (perseverance), for example, “Yes because we kept [trying] we never gave up.” Of the 26 students who did not feel their prosthetic hand was successful, almost all focused on logistics, such as not completing the task (“No because [it] could not pick up a pingpong ball”), not being able to finish construction (“no because I did not finish it”), or using the wrong materials (“no, the popsicle sticks [were] too thick”).

DISCUSSION

The results presented here suggest that the SJSC curriculum is well designed, supporting participants’ STEM Engagement and Identity by developing critical thinking and perseverance in participants, and encouraging peer and adult relationships, significantly more so than similar afterschool STEM programs conducted across the nation. This is particularly important given the demographics of the clubs (77% African American/Black or Hispanic/Latino; 59% Female) and the underrepresentation of women and people of color in science. It is important to build on these successes to increase the capacity and impact of the program as well as to better understand the mechanism of these outcomes.

While the use of productive failure was successful in promoting students’ critical thinking and perseverance, it may have had a negative impact on student peer relationships and, as a result, STEM Identities. Research shows that when students face micro-failures during collaboration, they engage in either beneficial (questioning, clarifying, explaining) or disadvantageous (arguing, ignoring, dominating) behaviors and that the choice of behavior correlates to learning outcomes (Lam, 2019). We speculate that students who engage in disadvantageous behaviors during micro-failures presented during the SJSC have the potential for damaging peer relationships, which may in turn damage their sense of belonging to the STEM community and ultimately their STEM Identity (Lave and Wenger, 1992; Gee, 2000). This may explain why we saw higher numbers of students indicate they had a negative change in relationships with peers and STEM Identities than any other scale. More research is needed to understand the relationship between disadvantageous behaviors, relationships with peers, and STEM Identities.

Alternatively, the SJSC experience in science may not match students’ experiences of science in the classroom. Research shows that this mismatch influences identity development (Zhai et al., 2013; Braund and Driver, 2005; Emvalotis and Koutsianou, 2017; Tan et al., 2015). Student participants in the program are self-selected and likely to have strong, positive STEM identities at the beginning of the program fostered, at least in part, by their success in the science classroom, which is unlikely to reflect authentic science. Elementary school teachers are often generalists with limited science content background, making it difficult for them to implement science and engineering practices (Shallcross and Spink, 2000; Nowicki et al., 2013) Furthermore, research shows that elementary teachers de-emphasize the practices of science based upon assumptions they hold about scientific practices students may or may not be able to successfully engage (Biggers et al., 2013). It may be that students whose STEM identities were fostered by success in unauthentic science experiences in the classroom became uncomfortable when exposed to authentic science practices rooted in productive failure, resulting in a cognitive dissonance that left them questioning their belonging in the STEM community. More research is needed to determine the extent to which this cognitive dissonance occurs as well as to identify strategies for partnering with schools and teachers to enhance the incorporation of science and engineering practices in their classroom pedagogy.

It is interesting to note that STEM Identity was slightly lower than STEM Engagement, suggesting that engagement is not sufficient for identity. This is consistent with previous research, particularly showing that students of color can have high interest and engagement in science, but not strong science identities (e.g., Hanson, 2008). It is possible to imagine that a student might find themselves interested and engaged in the story elements of the narrative without fully emerging themselves into the role of scientist as the curriculum design intended. Further iterations of the program should look for additional ways beyond performance to promote STEM Identities, such as opportunities for participants to be recognized by influential others (peers, teachers, parents, scientists). In addition, more research is needed to fully understand how the narrative approach might heighten the impact of the productive failure strategy on students’ critical thinking and perseverance and how this approach can be better leveraged to enhance STEM identity development.

The use of college students as facilitators may have allowed students to develop a new type of adult relationship, and also likely impacted the environment of the club, allowing students to separate from the traditional school day and keeping the club, which took place in a regular classroom at the school, from feeling like just another class. On the other hand, having teacher sponsors present maintained some structure, and facilitated logistical aspects of the club such as recruitment and communication with students and parents, while allowing students to interact with the teachers in new ways. It is unclear, however, whether or not these new adult relationships provided students with the recognition necessary for fostering positive STEM identities. In addition, this program did not explore the impact on the college students as facilitators of the program or the teacher sponsors of the program. The impact of participation on their STEM (or STEM teaching) identity is an important secondary outcome of this program.

Several areas have been identified as programmatic challenges or ongoing/future needs to continue supporting the program. First, finding college students available during the STEM Club timeframe, which is afterschool around 3:30 pm, can be difficult as many college students have classes at this time. Because the facilitators are generally STEM majors, they do not have much formal educational background and there is limited time available for facilitator training. In addition to basic pedagogical approaches already included in facilitator training, more information on educational topics such as collaborative learning and scaffolding/differentiation could help ensure all students are engaged in the club programming. Facilitators could also better support students when groups disagree about strategies or students are discouraged because their design doesn’t work.

The program is structured in a way so as to reduce barriers to participation, partnering with schools to host on-site and providing the club at no cost. Despite this, the SJSC failed to reach full capacity with several schools struggling to enroll only a handful of students. Many factors are likely to contribute to the variable participation rates observed across school sites, including teacher enthusiasm for the program, competing afterschool programming at the school, overall school size, and school culture. Research is needed to better understand how school structures impact science identity and interest in participating in afterschool STEM programming to see how we can partner with schools and teachers during the school day to effectively recruit more students and maximize club capacity.

Another programmatic next step is to improve communication with parents to increase the percentage of students completing the full 10 weeks of the program. Currently, a digital newsletter is sent weekly to families. In the future, sending home hard copies of the newsletter, and providing Spanish versions, or using text messaging services may help increase communication and potentially club attendance.

Table 5.

Comparison of the SJSC Data to National Normative Data.

CIS Scale	SJSC			National		Mean Difference	95% Confidence		P value
	N	Mean	S.D.	N	Mean		Lower	Upper
STEM Engagement	110	4.27	0.567	2100	3.84	0.43	0.29	0.57	< .001
STEM Identity	105	3.77	0.784	2055	3.30	0.47	0.29	0.64	< .001
Critical Thinking	109	4.34	0.60	9000	3.66	0.68	0.56	0.80	< .001
Perseverance	109	4.33	0.66	9000	3.66	0.67	0.54	0.80	< .001
Relationships with Peers	108	4.08	0.86	9000	3.59	0.49	0.32	0.65	< .001
Relationships with Adults	106	4.10	0.78	9000	3.43	0.67	0.51	0.82	< .001

ABBREVIATIONS

ALSAC

American Lebanese and Syrian Associated Charities

CIS

Common Instrument Suite

NGSS

Next Generation Science Standards

OST

Out-of-School-Time

SCS

Shelby County Schools

SJCCC

St. Jude Comprehensive Cancer Center

SJSC

St. Jude STEM Club

STEM

Science, Technology, Engineering, and Math