Success of Distance Learning During 2020 COVID-19 Restrictions: A Report from Five STEM Training Programs for Underrepresented High School and Undergraduate Learners

Abstract

In 2020, STEM training programs across the country were challenged to provide support to students during a nation-wide shutdown of research institutions in response to the COVID-19 pandemic. Five U.S. high school science internship programs funded by the Doris Duke Charitable Foundation, with a history of collaboration, developed innovative strategies for distance-learning (DL) opportunities during the pandemic. Forty under-represented high school and undergraduate students were paired with scientific mentors at one of the programs for a DL scientific internship. Summer training combined synchronous and asynchronous programming with research projects adapted for DL success. Ninety-five percent of students who participated were satisfied with the training experience, nearly identical to exit survey responses from 2019 when our programs were held in-person. More students were interested in pursuing a career in research at the end of the program and credited the DL experience with increasing interest in research careers. Some DL elements were ideal for underrepresented youth, including a more flexible schedule and elimination of cost and time for travel. While the lack of in-person instruction challenged our ability to create a strong student community, we found that preparation, communication, and flexibility were key elements to these successful DL programs. The increased emphasis on interpretation and analysis of data, rather than data collection, enhanced student learning. This manuscript highlights the changes made to our curricula, elements which were most successful, and recommends strategies for future distance-learning programming.

INTRODUCTION

In 2020, non-COVID-19 related scientific research was hampered or halted due to institutional practices to decrease the spread of the deadly COVID-19 virus. Most universities across the country were unable to host in-person internships due to social distancing restrictions related to the pandemic. These programmatic limitations came at a time when students were particularly vulnerable, both psychologically and academically (Le Vigouroux, 2021). As school instruction moved away from in-person interaction, students were restricted to learning in a completely virtual format for the first time. Not all youth were able to adapt successfully to online instruction. Unfortunately, underrepresented students were particularly vulnerable in this environment given the potential for limited resources for online learning, reliable internet, hardware availability, lack of quiet spaces within the home in which to learn, and the added responsibilities that came with economic stress and caring for other siblings (OECD, 2020).

Despite the challenges faced in the midst of the pandemic, the value of science training programs focused on underrepresented and disadvantaged (URM/DA) students cannot be minimized. For the purposes of this manuscript, we use the NIH definition of URM/DA, which includes individuals with racial or ethnic backgrounds underrepresented in the sciences, individuals with disabilities, or those who come from disadvantaged backgrounds (NIH, 2020). Research indicates that diverse groups outperform groups of homogenous high-ability problem solvers (Hong and Page, 2004), and research papers with an ethnically diverse set of coauthors are cited more frequently and on average are higher-impact (Freeman and Huang, 2015). Additionally, a diverse healthcare workforce expands health care access to minorities and underserved groups (Cohen et al., 2002). This is critical given the disparities in morbidity and mortality of many diseases across racial/ethnic lines and the well-earned distrust from minority groups to the healthcare system (Shoff and Tang, 2012; Kennedy et al., 2007). To overcome these disparities, broader participation is necessary in the biomedical and clinical workforce to build trust in all communities and accrue a more diverse group of patients into much needed clinical trials for developing and testing novel therapies.

Unfortunately, educational disparities often match those of health care with a lack of opportunities for those from URM/DA groups (National Research Council, 2012). There is a significant lack of URM/DA individuals in the current U.S. biomedical workforce. While 2019 census data demonstrates 13.4% and 18.5% of the population is Black and Hispanic respectively, those groups only represent 3.6% and 3.2% of medical school faculty (United States Census Bureau, 2019; AAMC, 2018).

Literature indicates that the majority of students who concentrate in STEM make that choice during high school, and that choice is related to a growing interest in science rather than enrollment or achievement (Norris and Agodoa, 2006). In response to the need for high school science training programs focused on URM/DA youth, the Doris Duke Charitable Foundation created the Clinical Research Continuum: High School to College (DDCF-CRC) program. The program funds eight institutions to provide authentic and mentored clinical research experiences to high school students and undergraduate alumni of the high school programs. These institutions include: Arthur Ashe Institute for Urban Health, Bradley University, Charles Drew University, Children’s Hospital Los Angeles, UCSF Benioff Children’s Hospital Oakland Children’s Hospital, Stanford University, University of Pittsburgh’s Hillman Academy, and University of Wisconsin. Studies demonstrate that augmenting the traditional educational experience through impactful research experimentation, tailored didactic training, professional skills development, dedicated mentor relationships, and a protected learning environment are all keys to increasing the number of URM/DA students in biomedical research and other STEM careers (Kobrak, 1992; Campbell, 1997; Maton et al., 2000; Nakamura and Shernoff, 2009; Kimand Sax, 2011; Noy and Ray, 2011; Toven-Lindsey et al., 2015). Additionally, success of youth internships is associated with building students’ self-confidence, science identity, research skills, professional development skills, and professional/scientific networks (Balster 2010; Toven-Lindsey et al 2015). DDCF has supported the named programs to employ these pedagogical practices for the past decade.

In this report, we summarize how five of the eight DDCF-CRC sites adapted to the imposed pandemic restrictions. Additionally, we discuss how the students and mentors responded to the change and provide student evaluations of the programs in comparison with prior in-person programming from 2018 and 2019. Based on these experiences, recommendations are proposed for future youth distance learning research training programs.

METHODS

Participating Institutions and Interactions.

Five of the eight U.S. science training sites funded by DDCF provided a distance learning curriculum in the summer of 2020 to both high school and undergraduate students. These sites included: The Arthur Ashe Institute for Urban Health’s Health Science Academy, NY (ASHE); the Children’s Hospital Oakland Research Institute Summer Student Research Program, Oakland, CA (CHORI); The Samuels Family Latino and African-American High School Internship Program at Children’s Hospital Los Angeles, CA (CHLA); Charles Drew University of Medicine and Science, Los Angeles CA (DREW) and the Hillman Academy, University of Pittsburgh, PA (PITT). The other three DDCF funded sites not included in this report had to defer their summer programming in 2020 due to institutional constraints and inability to identify enough mentors able translate a research experience into a virtual format in such a limited time.

Directors from the eight funded institutions meet annually to discuss program progress and to strategize for how best to reach and support the URM/DA students served. In between the annual meetings, directors correspond regularly to share ideas and curricular resources. In the beginning of 2020, faced with the challenges of running training programs in the midst of a pandemic, the directors held two virtual curricular planning calls to discuss DL strategies to employ. At the end of the 2020 summer programs, a survey was distributed to program directors and staff to qualitatively assess which aspects of the distance learning programs were most successful at engaging URM/DA students. Finally, four virtual meetings were held to discuss the DL programs, the results of the survey, and plan the manuscript.

The five sites that were able to adapt to a DL internship in 2020 are supported by both government and non-governmental agencies and/or foundations in addition to DDCF to provide a robust research training program for underrepresented youth (see acknowledgements). The five sites highlighted in this report provided science training opportunities to 91 URM/DA youth in 2020. However, the student survey data provided in this report is limited to the 40 students (30 high school and 10 undergrads) who were funded by the research training grant for high school and undergraduate students from the Doris Duke Charitable Foundation and who completed the DDCF-administered pre- and post- program survey evaluations [Grant Program: Clinical Research Continuum (CRC): High School to College].

Student Participants.

The DDCF CRC 4-year training grants provide resources for the support of eight to 15 students at each site annually; these numbers include both high school and undergraduate alumni trainees. Students qualified to participate in the DDCF funded CRC programs if they were U.S. citizens or permanent residents and considered underrepresented in biomedical sciences as defined by the NIH (NIH 2020). The criteria for URM/DA included: 1) those who self-identified as Black or African American, Hispanic or Latino/a, American Indian or Alaska Native, Native Hawaiian, or Pacific Islander, 2) individuals with disabilities as described in the Americans with Disabilities Act of 1990, or 3) those from a disadvantaged background as defined by NIH criteria (NIH, 2020). Each applicant’s academic experience and potential to pursue a career in biomedical research was considered based on information received from their application, transcript, resume and letters of reference. Each site had their own selection committee and used a wholistic approach to select qualified students who were both passionate about science and could benefit from the training program offered. The admissions criteria were identical in 2020 to previous years for all five sites. Additionally, each year, a sub-set of undergraduate students who previously participated in a CRC program as high school students were invited to return to build upon prior training. These students, typically two to five at each site, were selected from the cohort of DDCF CRC alumni at each institution. Undergraduate alumni were full participants in the research program but were also given leadership roles to aid the high school participants. Demographics for trainees supported at these five sites for 2018 to 2020 are provided in Table 1.

Table 1.

Combined Demographics of Students Enrolled in Five Summer STEM Research Training Programs between 2018 and 2020.

	2018	2019	2020
Number of Students	60	52	40
Male/Female, #
Male	17	20	12
Female	41	32	27
Another Identity	0	0	1
No response	2	0	0
Year in School Completed, #
9th	1	0	0
10th	9	4	0
11th	31	27	30
12th	4	7	0
Some Undergraduate	15	14	10
Hispanic, Latino, or Spanish origin, #
Yes	22	23	17
No	38	28	23
No response	0	1	0
Identifies as a racial/ethnic group underrepresented in STEM, #
Yes	25	16	21
No	20	17	9
Multiple identities/Did not respond	15	19	10
*Highest Educational Attainment of Parent, #
Some High School	9	6	7
High School Diploma	9	12	15
College Degree	11	9	6
Post-graduate Degree	21	13	9
No response/unknown	10	12	3

Note: These summary data were derived from the DDCF pre-program survey of students enrolled in the 2020 distance learning program.

^*Highest educational attainment refers to the highest level attained by any parent. “High school diploma” category may have some college. “College Degree” category may include associates degrees and bachelor’s degrees. “Post-Graduate Degree” category may include MS, PhD, and MD.

Due to national and institutional policies related to the COVID-19 pandemic, along with institutional restrictions regarding research with minors during the crisis, these five programs adjusted their curriculum from an intensive in-person research training program to a completely virtual program. Details regarding these distance learning curricula are provided in the results section.

Surveys.

Two separate sources, described below, were used to collect data for this report, 1) a funding agency administered pre/post survey completed by students about their training experiences and perceived benefits, and 2) a one-time survey directed at program directors and staff regarding the 2020 distance learning curriculum.

1). Student Survey.

The student surveys were conducted by the funding agency, DDCF, twice each year, once prior to the start of the programs (Pre), and again at the end of each summer program (Post). A link to the survey was distributed by program staff to students through email along with instructions for completion. The 22 question Survey Monkey questionnaire included questions regarding attitudes towards a career in research, research readiness, and scientific skill assessment. Five additional questions were added to the pre-survey in 2020 that focused on students’ comfort with distance learning, for a total of 27 questions in 2020. The exit survey included questions related to working with their mentor and overall program assessment. Except for the distance learning questions, the same entrance/exit surveys have been conducted each year. For the purposes of this report, student survey results from 2020 are presented along with the most recent program years (2018 and 2019). Year-to-year student survey responses were not compared statistically, given each cohort includes a unique selection of students and backgrounds. Information from these surveys are presented in Tables 1, 6 and Figure 1. Only data from the five programs with 2020 DL internships were included in the 2018 and 2019 data to maintain consistency across years.

Table 6.

Summary of Exit Student Survey Responses Regarding Research Interest and Quality of Programming from Five High School Training Programs Over 3 Years (2018 to 2020).

	2018 n=60	2019 n=52	2020 n=40
Attitudes Towards a Research Career
Has your experience in this program changed your level of interest in pursuing a research career?
I’m much more/somewhat more interested in research	70%	75%	98%
My interest in research has not changed	22%	19%	2%
I’m less interested/not at all interested in research	8%	6%	0%
Quality of Programming
My overall experience with this program was positive
Strongly agree or Agree	83%	96%	95%
Neutral	14%	4%	0%
Disagree or Strongly disagree	2%	0%	5%
I am satisfied with the quality of the mentoring I received during this program
Strongly agree or Agree	90%	88%	93%
Neutral	8%	10%	3%
Disagree or Strongly disagree	2%	2%	5%

Survey responses were originally a 5-point Likert scale (e.g. Strongly agree; Agree; Neutral; Disagree; Strongly disagree), but for this table, categories collapsed to 3-point Likert scale.

Figure 1.

Summary of Entrance and Exit Student Survey Responses Regarding Scientific Skill Proficiency from Five High School Training Programs over 3 years (2018 to 2020). Pre- (A, C, E) and post- (B, D, F) program student survey responses were collected using a 5-point Likert scale (e.g. No Experience, Little Experience, Some Experience, Much Experience, Extensive Experience). The survey was administered to all students. For visualization purposes, categories were collapsed to a 3-point scale: Little, Some, Much. Numbers of high school and undergraduate students who participated in the programs and provided responses to this survey included: 2018 (n=60), 2019 (n=52), 2020 (n=40).

2). Program Director and Staff Survey.

A Distance Learning Curriculum Survey was conducted at the end of the 2020 summer. The 30-question online survey was distributed to program directors and staff and focused on 2020 distance learning curricula details, best practices and successes, and challenges experienced. Information from this survey was summarized and presented in Tables 2–5.

Table 2.

Characteristics of Mentors Successful at Working with Youth in a Distance Learning Environment.

Character Trait	Description
Experienced	Previous experience with remote instruction
Flexible	Willing to change a project based on students’ capacity and experience
Patient	Ability to recognize when a student doesn’t understand
Organized	Plan for daily interaction with students and clear outline of research objectives
Motivated	Enthusiastic about research and excited to work with the student
Creative	Able to integrate fun into mentoring sessions to keep students engaged
Prepared	Plan for virtual research project prior to student’s arrival
Communicative	Solid plan for frequent communication

These data were provided by program staff through an online survey at the end of the 2020 Distance Learning Summer Program.

Table 5.

Examples of Activities for Creating Student Community in a Distance Learning Environment.

Emotional Support Activities	Group Activities
Daily meditation and mindfulness sessions Daily check-in with program faculty at start and end of each day Regular office hours with program leadership Workshops with social worker to gauge emotional temperature of group	Student group work to reduce isolation Journal club discussion groups Synchronous group fun activities: Dancercise, sharing about culture
Competitive Activities	Interactive Activities
Online competitive games: Science Jeopardy, Escape Room Challenges, Kahoot! Fun activities outside of research time: online board game nights, science movie nights, bingo	Online ice breakers at beginning of program Creation of consistent small groups for break out discussions, become students go-to community Break out room discussions lead by near-peer mentors: alumni, undergraduates and graduate students

These data were provided by program staff through an online survey at the end of the 2020 Distance Learning Summer Program.

IRB Considerations.

This evaluation, involving the use of the described surveys, was not considered research. Instead, it was determined to be non-clinical quality improvement by the UCSF Institutional Review Board. As such, the program evaluation was exempt from review by the Institutional Review Board process.

RESULTS

Program Overview.

Each of the five research internship sites are individually administered and developed their own unique curriculum. However, there are three training elements consistent across all sites: 1) selected students are assigned to a clinically relevant research project under the guidance of a mentor; 2) mentors are expected to meet with their students on a regular basis and ensure that students have the resources needed to conduct their research projects; and 3) additional enrichment activities are provided that help students develop the skills needed for clinical research, including but not limited to introduction to bioethics, protection of human subjects, exposure to scientific writing and experimental design. Additionally, all activities must adhere to institutional policies regarding minors in the workplace.

In 2020, program directors met virtually to discuss options for virtual research training and to identify shared resources. Creative solutions were necessary to rapidly pivot formerly in-person training to fully virtual programs. Each director chose a path forward for their individual program that was consistent with the overall goal of broadening participation through authentic research and mentorship. Qualitative analyses of post-program discussions among directors and through the director and staff surveys identified a few elements consistent among programs that were found to be useful in setting expectations in a DL setting. All programs observed that effective communication between students, mentors and administration officials was essential for easing stress during the unpredictable period. Mentors were given instructions to remain in regular contact with students and encouraged to make sessions interactive. During orientation at each site, curricular details were explained, expectations set regarding how much time was expected to be spent weekly on their research projects and enrichment activities, and guidelines for distance learning etiquette and success were shared.

The length of summer training program ranged by site from 5–8 weeks, with 20–40 hours of a combination of synchronous and asynchronous programming per week. This is an average of one week shorter than in pre-pandemic programming. Didactic synchronous training sessions ranged from 30–360 minutes, broken up into shorter presentations (30–45 mins). Based on student post-survey comments, these shorter virtual sessions were well received, particularly those which allowed for more time in interactive discussion. Mentors were asked to meet with students regularly, though the amount of time mentors spent with students discussing individual research projects ranged from 30–60 mins/day, 1–5 days per week. Only 38% of the 40 students met with their mentor 3–5 times per week. This was quite different when compared to in-person training years (2018, 2019) where 71% of students met with mentors 3–5 times per week. Program leadership also met with students each week, some as often as twice a day outside of synchronous training. Each program concluded summer training with a research symposium where students presented findings from their individual or group research projects.

Program Preparation.

All participating sites gathered information from students prior to program initiation including reliability of internet access, computer hardware, and access to video and audio. Of interest, though all sites asked about accessibility, only 15% (six of the 40) of the interns who participated in a program needed a computer. Each program also asked mentors about scheduling conflicts and ability to dedicate time to student training. Four of the five sites collected information on software resources and hours students could dedicate to training during COVID-19 including scheduling conflicts due to summer coursework, babysitting responsibilities, and/or work schedules. Each site required students to complete safety and ethical training prior to program initiation. Video connectivity issues were occasionally an issue for students, though audio connections were not a problem.

Applications used in the development of distance learning at the individual sites included: Google^™ Classroom and Suites, Microsoft Office^™, Discord^™, Moodle^™ and Blackboard^™ for program organization. Zoom^™ and Webex^™ were used for synchronous curriculum. Asynchronous activities enhanced the synchronous training through additional pre-recorded lectures (iBiology^™), flipped classrooms (Khan Academy^™), and lab simulation software (Labster^™).

Program Strategies.

Mentors who were particularly successful at transitioning to a virtual training environment included those with previous experience working in a virtual environment (Table 2). They integrated didactic training with games or quizzes (Kahoot^™) to enhance the learning experience, spent time prior to the summer planning their students’ research project and had a solid plan for regular communication with their student. Similar to in-person programming, students responded favorably to mentors who were patient, organized, enthusiastic about their research and excited to share with the mentee, and flexible enough to know when to revise project aims for success. Most students worked with scientific mentors on a 1:1 basis, though at two centers institutional policies regarding online communication with minors required a minimum ratio of 2:1.

Many traditional types of research projects were limited in the remote setting, e.g. medical information on patients previously used to investigate case series or retrospective clinical studies could not be shared with students who were off-site. Moreover, hands-on basic science projects were not feasible as students were not allowed in most laboratories. Mentors therefore revised their focus to include research projects that could be conducted in a completely virtual environment. Examples of successful projects included systematic literature reviews, clinical case series of de-identified patient data, data mining of publicly available datasets, image analysis, machine learning and computational projects such as those involving genomics, epidemiology, and modeling (Table 3). A variety of enrichment activities were added to synchronous curricula to enhance the training experience. Professional development workshops were provided, including resume preparation, college readiness, college essay writing, and interviewing skills. Presentations related to the college admission process and the changing landscape of admissions during the pandemic were included. Two sites included SAT preparation courses and financial planning for college workshops. Most notably, all sites included lectures or interactive discussions on career trajectories in the biomedical sciences (Table 4).

Table 3.

Successful Research Science Project Ideas for High School/Undergraduate Trainees in a Distance Learning Environment.

Research Category	Research Examples
Clinical	Processing and analysis of previously collected secondary clinical data sets Systematic literature reviews Case series of de-identified clinical data Online patient survey studies Program evaluations, quality improvement projects
Data Science	Data mining from publicly available datasets Computational projects (genomics, statistics, informatics, epidemiology, modeling, artificial intelligence, algorithm development) Machine learning
Wet-Laboratory	Analysis and interpretation of data sets previously generated within a laboratory Image Analysis Real-time wet-lab shadowing during lab procedure using a wearable camera

These data were provided by program staff through an online survey at the end of the 2020 Distance Learning Summer Program.

Table 4.

Examples of Specific Changes Made to In-Person Science Training Program Curriculum to Adapt to Distance Learning Format.

In-Person Curriculum	Distance Learning Modification
Research
Students worked on individual projects, and were typically assigned to work pairs: 1:1 student:mentor	At two of five sites, students were assigned to mentors/projects in teams of two in order to reduce isolation and increase interaction among the interns
Students worked entirely in-person either in wet laboratories or clinical research environments	Projects pivoted to focus on analysis and interpretation of data sets generated by the laboratory, on dry-lab studies (informatics, algorithm development) or computational projects (genomics, machine learning, epidemiology, modeling, etc.)
Wet laboratory experiences limited to students who were matched to basic science mentors only	Lab Simulation Software (e.g. Labster™, LabXchange™) modules allowed for all students to experience wet-lab work virtually.
Didactic Course Curricula
Provision of didactic lectures on basic biological principles	Shifted to a workshop/discussion format to maintain engagement. This was accompanied by a change in content focus towards the study of biology in the context of disease
Optional participation in journal clubs, non-program scientific lectures on campus, grand rounds etc.	Increased time spent on the “book club” segment, which was an interactive discussion of a popular science book about pandemics and virus research (Spillover, by David Quammen). The goal was to help students explain how biomedical research has an active role in improving human lives, leveraging their current interest in viruses to make a connection between what happens in a lab and their own lives
In-person provision of all lectures and research curriculum	Lectures were recorded for students who could not attend and broadcast via YouTube™ for those outside of the programs
Community Building
Students gathered in-person for lunches, small groups, and after-hours social events, game nights, movie watch parties, picnics over the course of the program	Social events were shifted to online trivia and game nights, Jeopardy™, Escape Rooms. Community dinners were orchestrated by ordering synchronized pizza delivery to students at home Care packages sent directly to students’ homes
In-person office hours	Created sub-groups of students lead by alumni for discussion breakout sessions; offered zoom office hours
Students’ primary connection was to their research mentor	More frequent/daily check-ins with students by the program leadership. The check-in meetings gave leadership the opportunity to provide students with additional support and to be informed of any challenges they may have encountered or any issues that needed to be addressed
CRC undergraduate alumni served as near-peer mentors to high school students on an as-needed basis	Provision of structured near-peer mentoring check-ins between undergraduate alumni with high school students 1:3 or 1:5 ratio
Enrichment Activities
College counseling, SAT prep provided through in-person workshops	All of these transitioned to online with minimal change required
Encouraged participation in lab group meetings and journal clubs (voluntary)	Required participation in journal club sessions lead by student undergraduate alumni

Note: These data were provided by program staff through an online survey at the end of the 2020 Distance Learning Summer Program.

Modifications for distance learning curriculum highlighted in this table were not necessarily instituted across all five program sites, though were found to be successful at the sites in which the changes were made.

Solutions and Overcoming Challenges of Distance Learning.

There were five significant challenges that were identified by program directors regarding distance learning. This included unfamiliarity of program leadership with distance learning technologies, identification of suitable mentors, ensuring all students were actively engaged, balancing the amount of time to spend in synchronous and asynchronous training, and of course, translating research experiences, particularly wet laboratory projects, to a virtual format.

Program directors who were unfamiliar with distance learning programming found creative ways to adapt to the new technology. Some program faculty found solutions by engaging with student alumni who were more capable and comfortable with the technology, for example students who had previous experience with Moodle^™, Discord^™, or Zoom^™ breakout rooms. These alumni were also called upon to interact as near peer advisors to students. This allowed program faculty to focus on enhancing the curricular content, and spend less time on the technology needed to deliver the content.

Transitioning to remote mentoring was a new experience for most of our mentors as well. All five sites found that frequent communication with students and mentors alike was critical and working with mentors prior to program initiation to identify feasible yet challenging virtual research projects was invaluable. One site used the ‘Entering Research’ curriculum from the National Research Mentoring Network (Balster et al., 2010) during orientation to help students and mentors navigate expectations and relationships.

One of the most notable challenges in distance learning was helping students form personal connections and building a sense of student community. Program administrators and staff were challenged with keeping students engaged and providing them with the social and emotional support needed. Each site developed their own strategies in combating these challenges (Table 5). Breakout rooms during synchronous activities were used extensively for small group discussions (Table 5). Students were encouraged to work in pairs to reduce isolation. Program staff checked in regularly with students and one program offered daily meditative sessions. A variety of fun social activities were also added to allow for students to engage with each other outside of their research activities; these included online competitive games (e.g. Escape Rooms, science Jeopardy^™, and Kahoot!^™), and various group activities (e.g., Dancercise^™, sharing about cultural backgrounds, and science movie nights). One program enriched the game nights by having pizza delivered to students’ homes.

Given the concurrent events such as social media’s circulation of images of police brutality against people of color and the socio-economic impact of COVID-19 pandemic, our students experienced additional stressors apart from the usual academic strain. On top of these external stressors, students faced economic stress related to parental job loss, additional responsibilities of providing care for siblings, and worry about an unknown future. Three of the sites provided additional social and emotional support to students in the form of mental health resources, group check-ins, mindfulness sessions, discussions to build empathy as well as discussions with social workers or other staff experienced in social justice, diversity, and inclusion.

Identifying the right balance between individual virtual research, asynchronous curricula, training group sessions and competition with non-program activities and responsibilities was another challenge faced by program directors. Students no longer spent most of their day with their mentors as communication between mentors and students was limited in the virtual environment. During in-person training years, the number of group sessions was limited to maximize time spent with their mentor either in the laboratory or clinic. As programs changed to our distance learning format, each program increased to having more frequent all-group Zoom^™ sessions. Students particularly appreciated synchronous sessions with hands-on programming and break out, small group discussions.

Finally, across all sites one of the biggest challenges was transitioning the wet laboratory, bench-based research experience to an online format. This was accomplished by recruiting mentors who could provide students with a project that was based either in analysis and interpretation of large data sets, images, etc. that the lab already had in hand (e.g., interpretation and mining of RNA-seq data, analyzing electroencephalogram traces), or in developing in silico approaches to research problems (e.g., training and testing an artificial intelligence algorithm to score clinical slides for parameters relevant to tumor diagnosis). This also required more active attention from mentors since students were not just in the lab every day. Some inventive mentors utilized Apple^™ Facetime technology to allow the student to shadow while performing real-time experiments in the laboratory. One program utilized the Labster^™ laboratory simulation software to allow all students to have some exposure to a virtual laboratory.

Student Survey Evaluations.

In total, 40 URM/DA students (30 high school, 10 undergraduate CRC alumni) participated in a DDCF funded program at one of five participating centers in 2020 (Table 1). These students all completed pre/post surveys administered by the DDCF. There were an additional 51 students who participated in training in parallel with the DDCF funded CRC students at all five centers (29 high school, 22 undergraduate students), but they were not surveyed by a common survey and are not included in this report. Fewer students participated in our combined programs in the summer 2020 compared to the previous year, a 23% decline in participation (Table 1). The reduced enrollment was primarily due to the loss of available mentors, who primarily are university faculty and trainees. Many labs were closed or forced to work remotely for parts of 2020, and so previously engaged mentors experiencing additional stressors did not have the bandwidth or resources to craft a research experience to an online format. The drop in numbers from some sites may have also been due to several student and administrative factors including but not limited to: 1) less interest in participation in completely virtual training program, 2) competing responsibilities for our students’ time due to the COVID-19 pandemic, and 3) additional economic stress the pandemic placed on families.

Students who participated and completed the entrance and exit surveys self-reported gains in scientific skill proficiency, specifically with regard to data analysis and interpretation as well as a better understanding the research process (Figure 1b,d,f). Moreover, nearly all students reported having a positive experience in distance learning: 95% either agreed or strongly agreed to having an overall experience with the program that was positive (Table 6). This was nearly identical to student exit survey responses from 2019 when our programs were held in-person.

Based on student survey responses to questions designed specifically for distance learning in 2020, 95% of students stated that they could learn from a variety of formats (e.g. lectures, videos, podcasts, and online discussion/conferencing). Moreover, from their previous experiences with online learning, 86% of students preferred online courses with weekly scheduled lessons. At the end of the program, all the students that were offered lectures with discussion boards/chat/break out rooms preferred that format for learning over curricula that did not have interactive discussion.

The primary goal of these training programs is to increase diversity in STEM by fostering URM/DA student curiosity in biomedical research. In 2020, 98% of students felt their experiences in the virtual training program was related to their increasing interest in pursuit of a research career (Table 6). Students who participated in the 2020 program were also highly satisfied with the quality of the mentoring provided through distance learning. These responses were from a group of students who appear to have had a high level of research experience upon entry to the program: 40–45% of students had extensive experience with analyzing, interpreting, and understanding the research process (Figure 1a,c,e).

The post survey student comments in the free text section revealed that despite efforts, students wanted more opportunities to build community among themselves and to get to know the other students through games, breakout rooms, social activities or just to chat. Students value connection with each other, perhaps even more so during times of enforced isolation.

Ten of the students, all undergraduates, who participated in the distance learning curriculum in 2020 were CRC alumni, meaning they participated in a CRC program previously as a high school student. The undergrads are full participants in their respective programs and as such their data was included in all tables and figures. A subgroup analysis was performed but is not reported as a table due to the small sample size. This is valuable information given these students could compare their experience with a previous in-person training curriculum. The overwhelming majority of these student alumni (9 of 10) either agreed or strongly agreed that their overall experience in the distance learning curriculum was positive. Two-thirds of this sub-group reported that the additional enrichment experiences provided (professional development, journal clubs, workshops) added to the success of the DL summer experience.

DISCUSSION

This past year of in-person instruction restrictions due to the COVID-19 pandemic presented a challenging learning environment for all programs attempting to support underrepresented youth in STEM education. Creative curricula were required as not all youth are able to adapt successfully to online instruction. Effective time management, basic technology skills, persistence, self-motivation, schedule flexibility for synchronous content as well as a quiet learning environment to focus are all necessary for success with distance learning. Unfortunately, underrepresented students are particularly vulnerable in this environment given the potential for limited resources for online learning, reliable internet, hardware availability, lack of quiet spaces within the home in which to learn and the added responsibilities that come with economic stress and caring for other siblings. Knowing these barriers existed and that there were few paid summer internship opportunities for high school students in 2020, five programs unfamiliar with distance learning curricula, adapted to provide paid internships and academic programming in a virtual setting.

Significant challenges exist in distance learning particularly for hands-on research training programs. There is no substitute for in-person bench research and no replacement for physician shadowing experiences. Closing ceremonies had to be reformatted; students could not participate in science field trips; social connections and bonding between students was stunted; and it was particularly challenging to participate in group projects. Moreover, program faculty struggled with the knowledge that as we prepared for a summer where all our curricula would be offered virtually, many of our students already had ‘Zoom^™ fatigue’ from their Spring classes. Despite all these difficulties, the advantages that the CRC program offered to our URM/DA students overshadowed these challenges. The increased flexibility of curriculum, removal of time and cost for travel to internship locations combined with creativity of the research projects allowed these students to focus on the research curriculum in a time of uncertainty. Moreover, program directors and staff were surprised by how smoothly students and mentors transitioned to distance learning and the comfort of students with the environment. As a result, student evaluations showed increased research skill development and they rated their experience with close to 100% approval.

There are several theories as to why students viewed the programming so favorably. First, there were few other opportunities for students during the COVID-19 lockdown, so students may have felt privileged by the offering, viewing the experience as a gift. It was a particularly isolating summer for most youth, with numerous emotional stressors. These programs offered a social connection with like-minded students as well as emotional support. Finally, as programs adapted to the new format, more attention was paid to group sessions and professional development, all of which may have had a positive influence on the students’ experience.

Moving forward, there are a few elements adapted for distance learning that program directors plan to incorporate into hybrid or in-person curriculum in the coming years (Table 7). It was recognized that the cost and time for travel to research sites placed unforeseen burden on many students. Therefore, programs will tailor their curriculum to offering regular virtual didactic sessions in addition to in-person programming. This has the added benefit of providing a broader range of scientific speakers due to increased speaker availability. Many lectures and events will continue to be recorded and uploaded to program sites and/or YouTube^™ for year-round use by students both in and outside of the programs. The increased emphasis on interpretation and analysis of data—rather than collection of samples and raw data—was an unexpected value-add for our students. We anticipated reduced enthusiasm for the research projects as there were no wet-lab experiences. To the contrary, our interns were very enthusiastic about their work. In retrospect, this makes sense, since drawing conclusions and framing results in the context of the literature can be one of the most exciting parts of research and can be more immediately rewarding than the actual data collection. We plan on emphasizing this more in future iterations of the program even after we are back on-site. Students also responded quite favorably to more opportunities for near peer mentoring, therefore most programs will be enhancing this aspect of their offerings. Finally, many of the software applications that were used to develop and organize the distance learning programming (e.g., Moodle^™, Discord^™, and Microsoft Teams^™) will continue to be used as they allowed for greater facilitation of programming and student discussions, regardless of physical location.

Table 7.

Advantages to Distance Learning and Adaptations to Consider when Returning to In-Person Programming.

Advantages to Distance Learning	Adaptations for Future Programming
Easy to record live lectures, easily access to students both inside and outside program	Record and upload virtual didactic lectures on program websites and/or YouTube channels for year-round use
Unique provision of scientific lectures from a variety of guest lectures (e.g., outside of local region)	Expand provision of guest lecture series, opportunities to include more diverse URM speakers and topics pertinent to the time (e.g. Vaccine development, gene therapy)
Increased creativity in type of research conducted	Increase emphasis on data interpretation and analysis vs. raw data collection for more rewarding experiences in short time interval- and increase students’ understanding of their topic
Schedule flexibility for students with added responsibilities at home	Continue to include some virtual training sessions to decrease cost and time for travel
Zero travel time allows for more time for programming	Increase frequency of all group meetings to aid in community development
Incorporation of remote, auxiliary peer and near-peer mentors	Expand programming to include more opportunities for remote near-peer mentors into student social network

These responses were provided by program staff through an online survey at the end of the 2020 Distance Learning Summer Program.

There are, of course, limitations to the conclusions that can be drawn from the pre-post student survey evaluations. Year-to-year results are not necessarily comparable, given each cohort offers a unique selection of students. One clear example of the possible year-to-year differences is the high number of 2020 students that self-reported prior experience analyzing data and interpreting scientific results on the pre-survey. Given this potential selection bias, we refrained from statistically comparing the year-to-year responses. Students who applied and were selected during COVID-19 could be unique, and perhaps more ready to adapt to a virtual learning environment compared to those who chose not to apply in 2020 or to defer. However, the selection criteria were unchanged in 2020 vs prior years.

Despite these limitations, it is evident that students who participated in the adapted distance learning curriculum increased their interest in research and that the virtual training program was related to their fresh perspective on biomedical research. Moreover, students had increased gains in research skill development including confidence in analyzing and interpreting data and a more complete understanding of the research process. This was in part due to the creativity of the scientific mentors in crafting unique research experiences, the ingenuity of program staff in offering an expanded curriculum as well as the inherent advantages to the distance learning environment for URM/DA students. As programs around the country plan to acclimatize to this new normal of hybrid learning, the suggestions offered here may prove useful in developing successful research training programs for URM/DA youth and likely will extend to programs working with students of any background.

ABBREVIATIONS

ASHE

The Arthur Ashe Institute for Urban Health’s Health Science Academy, NY

CHLA

The Samuels Family Latino and African-American High School Internship Program at the Children’s Hospital Los Angeles, CA

CHORI

Children’s Hospital Oakland Research Institute–Summer Student Research Program, Oakland, CA

CRC

Clinical Research Continuum

DDCF

Doris Duke Charitable Foundation

DREW

Project STRIDE: Charles R. Drew University of Medicine and Science, Los Angeles CA

PITT

Hillman Academy, University of Pittsburgh

STEM

Science Technology Engineering Math

URM/DA

Underrepresented Minority and Disadvantaged