Use of Science Lab Simulation During a Two-Week Virtual Biomedical Research Training Summer Camp for Underserved Minority Youth: A COVID-19 Adjustment

Abstract

The Maryland Action for Drug Discovery and Pharmaceutical Research (MADDPR) Program provides hands-on lab experience and mentoring to underserved minority high school students. With the inability to conduct an in-person STEM summer camp, the program transitioned to a virtual format in 2020. Thirty-three students and their PLTW teacher participated in live sessions using Blackboard Collaborate Ultra®. One highlight of the sessions was program faculty’s use of interactive simulation software such as science labs (Labster®), animal behavior (Sniffy the Virtual Rat®), and aseptic compounding (Virtual Interactive Clean Room®). Graduate student mentors worked with students in small virtual breakout sessions. Post-session survey data show that the majority of students felt comfortable participating in the simulation sessions. Students’ responses indicated that they enjoyed the virtual labs and appreciated the effort to implement the game-like lab simulation exercises. Remarkably, student ratings of the virtual sessions compared favorably and, in some cases, exceeded those from the same sessions conducted in-person in 2019. In post-camp surveys, 96% of the participants indicated an interest in pursuing careers in pharmacy/other health professions. Student and teacher comments also indicated that the virtual experience of the camp prepared both students and their teacher for the coming fall semester at school.

INTRODUCTION

A careful, deliberate approach to program planning, implementation, and evaluation in response to the COVID-19 pandemic enabled the University of Maryland Eastern Shore (UMES) Maryland Action for Drug Discovery and Pharmaceutical Research Program (MADDPR) to offer a valuable virtual experience to student participants. UMES’s experience offers important lessons about how best to reach students with STEM (Science, Technology, Engineering, and Mathematics)-focused programming both now and into the future.

UMES is an historically black college or university (HBCU) among the institutions of higher education in the United States established before the Civil Rights Act of 1964, and whose purpose was to serve the African-American community. UMES was founded as a land grant institution in 1886 on over 600 acres in rural Maryland. UMES School of Pharmacy is one of seven HBCUs with a pharmacy program and one of three pharmacy doctoral programs in the state of Maryland (University of Maryland Eastern Shore, n.d.).

Several lines of evidence point to the importance of hands-on laboratory science experiences in the learning processes of high school students. These experiences led to the development of critical thinking and problem-solving skills and, ultimately, stimulated students’ interest in pursuing education and careers in STEM (Alcéna-Stiner, 2020; Hernandez-Matias, 2020; Pender, 2010; Russell, 2007; VanMeter-Adams et al., 2017). Studies consistently show that students who have an interest in STEM are heavily influenced by extracurricular and hands-on laboratory experiences (Lemelson Foundation, 2010; Yonezawa et al., 2009).

Based on recent research, short-term STEM learning programs influenced participants’ perception of scientific competence and had a positive effect on students’ perception of scientific competence (Hernandez-Matias, 2020). Data from 149 high school students who performed hands-on research demonstrated that it sustained their interest in pursuing STEM-related academic degrees (VanMeter-Adams et al., 2017).

Findings from the MADDPR Program reaffirm these research results. MADDPR shows that a collaboration between high schools and universities to provide high-quality STEM experiences and mentorship to high school students can encourage their STEM interest and education and career choices.

This partnership between the UMES School of Pharmacy and high schools in Somerset County is funded by a Science Education Partnership Award (SEPA) through the National Institute of General Medical Sciences, an Institute of the National Institutes of Health. Somerset County is one of the most rural and underserved areas in Maryland. Nearly a quarter (24%) of county residents are in poverty per the most recent U.S. Census data, as compared to 11% of the U.S. population as a whole (U.S. Census Bureau, 2019). During the pandemic, a particular challenge for the region was broadband internet accessibility, in an area where more than a quarter (26%) of county residents lack the broadband service needed to participate in remote learning (U.S. Census Bureau, 2019). Minority and underserved youth residing in rural areas and coming from families of low socioeconomic status have few, if any, opportunities to experience the research methodology and tools used in drug discovery and biomedical research.

MADDPR’s main objectives are: (1) to deliver, evaluate, and disseminate a curriculum comprised of a series of interactive STEM modules providing hands-on training in research methodology in the areas of drug delivery and biomedical research to high school students, and (2) to implement a mentoring program for the high school students to provide support during and after curriculum delivery. MADDPR seeks to train minority youth in research methods of drug discovery and biomedical research and to motivate them to pursue higher education and careers in applied healthcare fields and biomedical research, with the support of near-peer mentors who are graduate students in the UMES School of Pharmacy.

The Principal Investigators (PIs) partnered with science teachers from Somerset County to develop an integrated curriculum that builds on Somerset County’s high school biomedical program, Project Lead the Way (PLTW). PLTW comprises four fundamental courses offered to sophomore, junior and senior high school students: 1) Principles of Biomedical Science, 2) Human Body Systems, 3) Medical Interventions, and 4) Biomedical Innovation. MADDPR’s two-year curriculum links the PLTW curriculum with hands-on training modules in drug discovery and biomedical research led by experienced pharmaceutical researchers using state of the art biomedical methodology. Over the course of the two-year program, participating students experience an array of hands-on modules, designed with PLTW teacher input. MADDPR participants also receive ongoing support from near-peer mentors who are graduate students in Pharmaceutical Sciences (Figure 1).

Figure 1.

A two-year curriculum for the MADDPR Program in relationship with the PLTW core coursework.

In its first year, MADDPR provided minority and underserved high school students with hands-on drug discovery and biomedical research training experiences that fostered students’ critical thinking skills, stimulated their interest in science, and encouraged positive attitudes towards science. During Year 1, 23 participating students (Cohort 1) experienced a week-long summer camp followed by three after-school sessions during the subsequent fall term. Throughout the first year, their near-peer mentors — UMES School of Pharmacy graduate students who were mostly minority students — offered guidance. Mentors helped to implement program activities and demonstrated, through their own experiences, the health and biomedical educational and career pathways. The program worked in tandem with the preexisting robust biomedical sciences core curriculum developed by PLTW at the high school. A key asset of the first-year program was the opportunity to be on the UMES campus, becoming familiar with UMES and its facilities, having the opportunity to work in the School of Pharmacy lab, using campus resources such as video cameras and 3-D printers, experiencing a model pharmacy and, most importantly perhaps, bonding with fellow participants.

Year 2 was expected to build upon the first year’s accomplishments, providing first year programming to a second cohort of students and expanding the program to incorporate the planned curriculum for the first cohort as they entered their second year. When expectations for an in-person program came to a halt, MADDPR committed itself to providing both its returning and new students with a rich virtual camp experience.

MADDPR’s pivot to an online experience for participants depended on strong collaborations with the Somerset County School District, School of Pharmacy faculty and graduate student mentors, the program’s training partner (4-H STEM), UMES administration and the UMES Department of Internet Technology. The robust rollout of the program during its first year (2019–2020) provided a strong foundation for a rapid refocus on virtual learning. The program committed to a rigorous evaluation, findings from which have guided the implementation of the program. Students responded well to the first-year hands-on programming and campus interactions. The program’s PIs worked rapidly and made a set of critical decisions, including the use of lab simulation software adapted for use with the high school participants, to facilitate the implementation of the virtual camp. These factors supported a virtual program experience that largely replicated the accomplishments of the first year.

Key factors that supported the early shift to virtual implementation included the ability to 1) partner with the PLTW high school teacher and school district; 2) reimagine the schedule of program activities; 3) use an existing UMES online platform; 4) utilize lab simulation software; 5) train those delivering the program in best practices in virtual learning; and 6) incorporate the current conditions of the pandemic in the program’s content. UMES utilized every available resource and offered students a rich summer experience that offered both tangible and intangible rewards.

During the COVID-19 pandemic, the processes that went into the successful transition from a laboratory based in-person camp to a virtual/online program are detailed below. Changes to the roles and responsibilities of PIs, program faculty, graduate student mentors, the PLTW teacher, the evaluation team, and administrators at both the University and School levels — as well as strategies employed by this collaborative team — laid the groundwork for the MADDPR’s successes discussed below. Challenges encountered by the program and others during this pandemic are also addressed.

METHODS

Program Planning and the Impact of COVID-19.

Following the successful first year of implementation, camp flyers and application forms were sent to the lead PLTW high school teacher who served as the coordinator for the camp application process. All but one of the 23 MADDPR Cohort 1 students planned to return for their second-year camp experience, the exception was a student who had moved out of the area. The PLTW teacher reported that the Cohort 1 students helped to recruit the second cohort as they enthusiastically described their own first-year experiences. In the summer of 2019, these students had the benefit of the hands-on lab and college experience on the UMES campus. The diversity of the lab experiences coupled with some fun activities were the main drivers for the apparent student enthusiasm for the program. By spring 2020, 25 new applicants (Cohort 2) and 22 returning students (Cohort 1) were planning to participate in MADDPR during Year 2.

On March 19, 2020, UMES suspended in-person learning in response to the statement from the University System of Maryland (USM) Chancellor encouraging USM universities to deliver instruction through distance-learning for the remainder of the semester (University System of Maryland, 2020). UMES immediately transitioned to online learning. Fortunately, most UMES School of Pharmacy faculty had been trained to use online learning tools to allow for continued instruction, despite campus closures due to inclement weather or other factors. Blackboard®, a learning/course management system, includes Collaborate Ultra®, an online tool that was used by most faculty. Collaborate Ultra® allows real-time video conferencing using voice over the internet audio, sharing of files and applications, and use of a virtual interactive whiteboard. It also features breakout rooms for small group discussions and chat-style messaging.

Planning for transitioning to an online MADDPR Program started almost immediately after the university halted in-person classes. Weekly meetings including the PIs, science teacher and the evaluation team facilitated planning and coordination of the virtual program. Planning also relied on the program’s logic model which included an emphasis on experiential learning and mentoring.

Critical Decisions Facilitating Successful Program Implementation.

In the wake of the pandemic, the PIs decided to move forward with offering a robust virtual program to Year 1 and Year 2 students registered for the 2020 program. This necessitated marshalling the necessary resources and implementing a set of adaptations to enable enrolled participants to engage in a now “virtual summer camp.” Critical decisions regarding implementation of the program included:

1. Making an early and firm decision to mount a virtual program, rather than delaying a decision or cancelling the 2020 program. The PIs decided immediately after the declaration of the pandemic by the World Health Organization and the cancellation of UMES in-person classes to forge ahead with the virtual program.

2. Fully partnering with the PLTW high school teacher. During the planning stage for Year 2, the high school teacher reached out to participating students to encourage and support their participation, assisted in ensuring that all participants had the necessary technology (borrowed as needed from the school district), obtained parental consents for online learning, and attended every session — frequently troubleshooting technical issues (PLTW Quotes 1 and 2). The teacher coordinated with participating students to ensure internet access, at times making it possible for some students to access the internet outside their own homes.

3. Compressing the program schedule. During the first year, engagement in program activities spanned a five-month period. Given the uncertainty of how the pandemic would unfold during the school year, the PIs, in consultation with the high school teacher, concluded they would incorporate planned fall sessions into an expanded two-week summer experience.

4. Using Blackboard® Collaborate Ultra®. Participating students used the same platform as the entire UMES campus community and were thus exposed to the classroom management and remote learning platform used on many college campuses. The UMES administration and Department of Internet Technology made it possible for participants to gain access to Blackboard® and Collaborate Ultra® during the camp’s implementation. The PIs and graduate student mentors assisted in orienting students to Collaborate Ultra®. Prior to the start of the program, the PIs tested all the systems during a dry run with all the camp participants, teacher and mentors to confirm student access to their UMES account credentials and oriented participants to key features in the platform that students would need to access.

5. Maximizing the use of lab simulation software. Student immersion in a variety of new lab experiences outside the normal classroom environment is a primary feature of the MADDPR Program. The American Chemical Society endorsed the use of lab simulation tools, stating “computer simulations that mimic laboratory procedures have the potential to be a useful supplement to student’s hands-on activities, but not a substitute for them” (American Chemical Society, n.d.). The PIs decided to use commercially available laboratory simulation software that allowed students to individually explore and control the various components of a lab experiment and observe findings that were determined by their intervention. The choice of lab simulation software depended on having learning objectives, topics, and functionality consistent with the MADDPR Program. Despite limited commercial availability of appropriate software, MADDPR was able to incorporate three interactive programs: over 200 virtual science labs with Labster® (University System of Maryland, n.d.), animal behavior experiments with Sniffy the Virtual Rat® (DiD Software Inc/Sniffy, n.d.), and an aseptic compounding simulation, the Virtual Interactive Clean Room® (Penguin Innovations, n.d.). These programs provided a close approximation of the types of lab experiences students would have had on-campus. UMES PharmD students had been using the Virtual Interactive Clean Room® which provides virtual training in aseptic techniques critical for safe compounding of sterile medications and safe practices related to parenteral drug therapy. Familiarity with this digital technology boosted the PIs’ confidence that MADDPR could adapt the program and design experiments that allowed high school students to experience the compounding of a sterile IV solutions.

6. Orienting faculty to use lab simulations. In addition to using the Virtual Interactive Clean Room®, program faculty reimagined additional sessions and how they might incorporate Labster®, with virtual labs covering a wide range of topics, in a modified format to assist MADDPR faculty in adapting their programs to an online environment. Access to Labster®, the virtual science lab simulations, was sponsored by the University System of Maryland. Faculty and session leaders were encouraged to use lab simulations if they were appropriate to their topic.

7. Training participating faculty and graduate student mentors in virtual learning best practices, including the use of breakout rooms. Faculty, staff, and mentors participated in a day-long 4-H STEM training offered by faculty of the UMES extension program. The session was consistent with the MADDPR Program’s commitment to yearly training by locally-based 4-H STEM educators who were familiar with the Somerset County student population and schools. Given the migration to virtual learning, the 4-H STEM training focused primarily on experiential learning and critical thinking, virtual teaching and learning, working with different learning styles, and virtual mentoring for high school students. The training offered insights that some faculty, in turn, were able to use during their college courses.

8. Incorporating COVID-19 content in the camp sessions. Faculty were encouraged, when appropriate, to incorporate the pandemic and related issues into their sessions. For example, during the planning of the summer camp, the antiviral drug, remdesivir IV infusion was in the approval process for treatment of COVID-19 hospitalized patients; it was incorporated as one of the planned aseptic compounding exercises.

9. Fully marshalling other available resources to provide assistance with program delivery. In addition to the PLTW teacher’s ongoing participation, the Somerset County Public Schools contributed to the program by providing administrative support and by lending computers to students who needed them to access the summer sessions. UMES administration and their Department of Internet Technology supported the credentialing of all program participants, non-UMES session facilitators, and the evaluation team to allow access to the Blackboard® Collaborate Ultra® platform. In addition, 4-H STEM, who were to be involved in annual faculty training for best practices in STEM education for high school aged students, incorporated and modeled virtual learning techniques during the pre-camp faculty training. The local hospital, which provided an in-person tour last year, created a virtual experience for the students.

10. Offering MADDPR students tangible rewards to celebrate their participation in and completion of the program. Before the program began, the collaborating teacher arranged for students/parents to pick up a home kit containing needed camp material: binder containing material for the camp, including a schedule of activities, guidelines for lab simulation activities and additional materials; a personalized lab coat and UMES branded notebooks, pens and USB drives; a college admission planning textbook; and other supplies provided by the program faculty specific to their session. Following both in-person and virtual program years, students also were given a $500 stipend upon completion of the program.

Program Implementation.

Despite the COVID-19 pandemic, the MADDPR Program was able to provide a two-week STEM virtual summer camp which closely approximated the original design. The program offered participants exposure to a variety of biomedical/pharmaceutical topics. The sessions planned for the virtual camp were adapted from plans for in-class sessions. In some instances, adaptations were made to accommodate virtual learning, such as switching a computer-aided design lab to analog methods to learn about the design process. Instructors used the COVID-19 pandemic as a learning opportunity and focused on its diagnosis and infection mechanisms where appropriate.

In addition to the program modules delivered by UMES faculty and staff, a planned in-person site visit to the local hospital was substituted with a virtual tour provided by medical staff at the facility. The virtual site visit allowed students to see the variety of ways in which pharmacy services are applied in a hospital setting.

Sessions for students commenced with an orientation to the two-week camp (Day 1) and closed with student capstone project presentations (Day 10). Eight additional sessions were presented to each cohort. Of the 16 topical sessions for both cohorts, half included the use of simulation software.

Sessions for First Year Students (Cohort 2).

During Year 2 of the program, 18 Cohort 2 students attended the following sessions:

Day 1: Orientation

Day 2: Drug Discovery and Development — Students were exposed to the overall program of drug discovery and development with examples of design, synthesis and structure-activity relationship of anticonvulsant agents to treat drug-resistant epilepsy.

Day 3: COVID-19 Diagnosis — The session was adapted from the prior year’s session on biomarkers to address the current context of the pandemic. Students explored and learned about the COVID-19 virus, testing, diagnosis and treating patients.

Day 4: Peninsula Regional Medical Center (PRMC) Virtual Visit — A fall field trip to the Pharmacy Division of PRMC was changed to a virtual visit organized by two senior hospital pharmacists at the medical center. Students were introduced to the role of hospital pharmacists and their wide range of responsibilities at the medical center. The two hospital pharmacists facilitated this session engaging the students as they observed hospital pharmacy operations.

Day 5: Virtual Sterile Compounding — Using the Virtual Interactive Clean Room® simulation, students were able to experience, in a virtual environment, how to garb personal protective equipment (PPE) and aseptically compound IV solutions in a laminar flow hood. One of the exercises was the compounding of a remdesivir intravenous sterile solution for the treatment of COVID-19 patients (Figure 2).

Figure 2.

Computer screen shot during the session where students were training on the correct garbing order of PPE before virtually proceeding to the clean room for aseptic compounding using the virtual interactive clean room® simulation software.

Day 6: Use of Design to Solve Medical Problems — originally planned as a session on Computer-Aided Drug Design (CADD), the session was reimagined to explore the use of design in medical administration, without the aid of the computer program. Students explored these concepts using materials that were included in the home kit they received prior to the start of the camp.

Day 7: Promoting Healthy Behaviors — Students learned to define carbohydrates, fats, and proteins and considered interpretation of food labels to identify the serving size, calories related to consuming carbohydrates and fats. They learned the health implications of overconsumption of calories, carbohydrates, proteins, and fats as well as the impact of physical activity on weight, cardiovascular health, and overall wellness.

Day 8: Opioid Abuse and Prevention — Students learned about opioid drugs, signs and symptoms of opioid overdose, and how to administer naloxone with several interactive exercises.

Day 9: Retail Pharmacy Operations — This session was adapted from the session at UMES’s Next-level Educational Simulation Training (NEST) center — a simulated mock-pharmacy area where participants can see the inter-working of a community pharmacy. Participants learned about the workflow process for a prescription. Participants also gained exposure on how pharmacists aid in patient counseling and preventive health.

Day 10: Capstone Project Presentation

Sessions for Second-Year Students (Cohort 1).

Fifteen Cohort 1 students returned to the MADDPR Program and completed the two-week virtual camp. Sessions included:

Day 1: Orientation

Day 2: Animal Behavior — The session utilized Sniffy the Virtual Rat® to explore the principles of classical and operant conditioning to guide animal behavior (Figure 3).

Figure 3.

Computer screen shot during the session where students were performing an animal behavior experiment using Sniffy the Virtual Rat® simulations.

Day 3: Understanding Viral Infections — The topic was focused on the epidemiology and virology of the COVID-19 pandemic and fundamentals of molecular biology and cell culture techniques. During break-out sessions, students used the Labster® software in a virology experiment (Figure 4).

Figure 4.

Computer screen shot in Collaborate Ultra® showing students and mentor viewing intestinal cells as they investigate the cause of a mysterious viral disease during their breakout session using Labster® microscopy simulations.

Day 4: Kinesiology and Physical Fitness —Students gained an understanding of health-related components of physical fitness. During break-out sessions, students used the Labster® software in an exercise physiology experiment.

Day 5: High Performance Liquid Chromatography-Mass Spectrometry (HPLC-MS) For Drug Analysis — Students applied a Labster® simulation to understand the concepts of liquid chromatography. They used a lab simulation exercise to determine the stability of a medication at different temperatures using HPLC. Students were also provided data and asked to determine caffeine levels in beverages using HPLC.

Day 6: Going After Breast Cancer — At the start of the session, students engaged in an interactive game through Kahoot® to test their knowledge about breast cancer (Kahoot!, 2021). They later used Labster® simulations to understand cancer pharmacology.

Day 7: Virtual Sterile Compounding — Using the Virtual Interactive Clean Room® simulation, students were able to experience, in a virtual environment, how to garb personal protective equipment (PPE) and aseptically compound IV solutions in a laminar flow hood. One of the exercises was the compounding of a remdesivir intravenous sterile solution for the treatment of COVID-19 patients.

Day 8: Antimicrobials — Students learned the concept of antimicrobial drug therapy, antibiotic resistance, and approaches to antibiotic discovery.

Day 9: Mechanisms in Prostate Cancer — Students explored cell culture of LNCaP prostate cancer cells and assessed changes in cell morphology resulting from cytokine-induced differentiation. Students used Labster® cell culture and microscopy simulations to perform exercises on how to harvest cells and make observations under the microscope.

Day 10: Capstone Project Presentations

Mentoring and Monitoring Student Participation.

Even in the virtual setting, near-peer mentoring continued as an integral part of the program. The three UMES graduate student mentors worked in small group breakout rooms during each program session, allowing participants to work in groups of five or six. Mentors also offered instructions and demonstrations during the lab simulations, gave education and career guidance, and served as positive role models.

The participating MADDPR faculty and the PIs circulated among the breakout groups to provide further support and guidance to the students. The Labster® programs included a dashboard that allows instructors to monitor the progress and performance of their students during the exercise (Figure 5). Typically, students responded to a set of questions in order to progress in each part of the lab experiment; the graduate student mentors were available to clarify and facilitate the student learning. The dashboard provided faculty with the percentage of questions completed in real time and allowed them to enter a particular breakout room where completion was lagging so they could assist students through the lab exercise.

Figure 5.

The Labster® dashboard gave program faculty the ability to monitor the progress of individual students in real time during laboratory simulation exercises.

The MADDPR Program, with a strong foundation from its first year of implementation, was compelled to quickly develop and implement a virtual program. The PIs, the high school teacher, presenting faculty, graduate student mentors, and students all joined forces to bring about a successful second year. One Cohort 2 student who was first introduced to the MADDPR Program in the virtual environment stated, “I would love to continue learning about the development of products and drugs that benefit the public. I really like the fact that someone just like me can create products and benefit the whole world for the better.”

RESULTS

Student and Mentor Program Participants.

The MADDPR Program enrolls participants from the two Somerset County, MD Public School District high schools. During the second year of the program (2020), the UMES MADDPR Program included 15 Cohort 1 returning science students, all of whom attend Somerset Public School District’s Washington High School and Academy in Princess Anne, MD. The high school serves a diverse rural community and is located close to UMES. In 2019, the first year of the program, a total of 23 biomedical-track high school students participated in the MADDPR Program; 65% of whom returned for their second year The COVID-19 pandemic may have had an effect on the students’ ability to return to the program for a second year. Many reported to their teacher that they were working at jobs that would prevent them from attending. The school district was able to lend the necessary technology to attend the virtual sessions, but some students were, nonetheless, challenged by connectivity issues.

Eighteen Cohort 2 students from PLTW entered their first program year, including 13 from Washington High School and Academy and five from Crisfield High School and Academy in Crisfield, MD. The students’ high school teacher, Somerset County School Districts’ PLTW Biomedical Sciences Instructor, attended all program sessions during the past two years of implementation.

Over the two years of the program’s implementation to date, three mentors worked with groups of five to eight students to facilitate their participation and understanding of the materials. These near-peer mentors were able to share their own education and career paths with participants.

Student attendance during the summer camp was strong. The summer camp comprised 10 days of programmed sessions, scheduled for three hours each day. All of the 15 participating Cohort 1 students attended at least eight of 10 sessions. Similarly, all of the 18 Cohort 2 students attended every session during the first week of the camp. There were some excused absences during the second week of camp and one student reported connectivity issues for one session because of a storm.

The pandemic resulted in severe disruptions for students and their families. Withdrawals from the program mostly comprised students who had committed to camp but were not able to attend because they had new employment obligations.

Demographic Characteristics of Participating Students.

Participating students were from Somerset County, Maryland, a rural and underserved county. The county’s median household income of $37,803 falls below the median household income of $62,843 for the United States as a whole (U.S. Census Bureau, 2019). Only 14% of high school graduates in this area successfully pursue higher education compared to 32% in the U.S. as a whole (U.S. Census Bureau, 2019). Per district policy, all students receive lunch at their school; the majority of students would be eligible for free or reduced lunch based on income levels.

The MADDPR Program has successfully recruited its target group of students. During the second program year:

• Fifty-two percent of students reported they were first generation students;

• Seventy-three percent of students were female, twenty-seven percent were male;

• The median age was 17 for both first-year and second-year students at the time of the camp. Students ranged in age from 15 to 19;

• Seventy percent were entering 12th grade; the remaining students (30%) were entering 11th grade;

• Seventy percent of students reported they were eligible for free or reduced lunch at their school;

• Nearly half of attendees were White (48%); 36% were Black or African American; 6% were two or more races; 9% “preferred not to answer”; and 12% of students were “Hispanic or Latin American.”

Faculty Participation.

Fourteen faculty across the UMES School of Pharmacy and Health Professions and Salisbury University in addition to two senior hospital pharmacists participated and delivered modules during the virtual summer camp.

Evaluation.

The UMES MADDPR SEPA program evaluation utilizes a mixed-method evaluation design that includes both quantitative and qualitative methods of gathering data (Leech and Onwuegbuzie, 2009; Maxwell, 2016). It is designed to meet both the formative and summative evaluation needs of UMES program administrators, participating faculty, participating high school teachers and other stakeholders involved with the program and to set the stage for further evaluation during future phases. The external evaluation team has been part of the planning process for the grant from its inception and continues to meet on a regular basis with the PIs and PLTW teacher. The annual evaluation has benefited from the review and input of the grant advisory committee, comprising subject area experts among university, school district, and community organizations. The present study was approved by the UMES Institutional Review Board as an exempt protocol.

The evaluation design includes a variety of methodologies and procedures. During the second year of the program, the evaluation activities concentrated on continuing to collect process and outcome evaluation data, as the mode of delivery shifted to an online platform. The COVID-19 pandemic necessitated several changes to the evaluation plan to accommodate online collection of survey data, virtual observations of sessions, and virtual focus groups. The second-year evaluation also included the design and implementation of a comparison group study. Sources of evaluation data included:

• Program records, materials, descriptive statistics and reports. The project staff maintains records about the program’s implementation.

• Virtual Summer Camp curriculum materials.

• UMES Faculty/Staff Peer Effectiveness Survey.

• MADDPR Summer Camp Student pre- and post-surveys.

• Post Summer Camp Graduate Student Survey.

• Post Summer Camp Faculty Follow-up Survey.

• Post MADDPR Program session “report card” surveys.

• Key Informant Interviews.

• Student Focus Group.

• Observations, informal reports and anecdotal materials.

Students Responded Positively to the Virtual MADDPR Summer Camp.

Students were presented with pre- and post-camp surveys at the beginning of camp and after all the sessions were completed. In addition, they responded to short after-session surveys to elicit feedback about each session, its content, and its usefulness to their education and career goals. The survey tools were developed jointly by UMES program planners and by the evaluation team.

Using a traditional scale of A, B, C, D, and F, high school student participants graded the camp highly and provided reasons for their ratings. All 13 Cohort 1 student respondents gave the camp an “A” (77%) or “B” (23%) for a “GPA” of 3.77. Among the 13 Cohort 2 respondents, 12 (92%) gave the MADDPR camp an “A” (77%) or “B” (15%) and the remaining respondent graded the camp a “C” (8%) for a “GPA” of 3.69. All of Cohort 1 and Cohort 2 students “strongly agreed” (85%) or “agreed” (15%) that it was worth their time and effort to participate in the MADDPR Camp. In addition, participants were asked to grade the activities using a traditional scale of A, B, C, D, and F for each session of the camp. All Cohort 1 sessions (for the second year of programming) received a “GPA” between 3.64 and 2.40. All sessions had a GPA of 3.07 or above with the exception of one session (GPA=2.40) which experienced technical difficulties. All Cohort 2 sessions received a “GPA” between 4.00 and 3.44. All of students gave the session on Retail Pharmacy Operations an “A.”

The virtual adaptation of the sessions offered during the second program year for Cohort 2 students compared favorably to those offered to Cohort 1 students during the first year when the MADDPR was an in-person camp in 2019 (Table 1). Five of the eight sessions received higher marks: Drug Discovery and Development, COVID-19 Diagnosis (Biomarkers), Use of Design to Solve Medical Problems, Opioid Abuse and Prevention, and Retail Pharmacy Operations. The virtual hospital tour and the session on healthy behaviors received slightly lower marks. Asked to name “the most valuable thing” they learned at the camp, students appreciated the information about pharmacy careers, the educational pathways to achieve their goals, and the opportunity to learn more about COVID-19. They also understood and appreciated the challenges involved in bringing the program to a virtual environment, in managing and helping them navigate technical challenges, and in providing valuable content (Student Quotes 1, 2, and 3).

Table 1.

MADDPR Student Assessment of Comparable In-person and Virtual MADDPR Sessions (n varies), 2019 and 2020. Five (5) of the eight sessions (indicated in bold) received higher marks; two received lower marks (indicated in italics). One virtual session was not offered during the first year.

Cohort 1 In-Person Session	Cohort 1 Session GPA	Cohort 2 Virtual Session	Cohort 2 Session GPA
Drug Discovery and Epilepsy	3.04	Drug Discovery and Development	3.65
Biomarker Activities	2.95	COVID-19 Diagnosis	3.76
PRMC Tour*	3.82	PRMC Virtual Visit	3.44
NA	-	Virtual Sterile Compounding*	3.44
3D Printing Activities (Computer Aided Design)	2.83	Use of Design to Solve Medical Problems	3.79
Promoting Healthy Behaviors*	3.90	Promoting Healthy Behaviors	3.75
Opioid Abuse and Prevention	3.39	Opioid Abuse and Prevention	3.73
Retail Pharmacy Operations	3.83	Retail Pharmacy Operations	4.00

Camp participants were asked to rate their level of agreement with a series of statements about the sessions, using a five-point Likert scale with 1 being “strongly disagree” and 5 being “strongly agree.” Among Cohort 1 and Cohort 2 students (n=26), all students “strongly agreed” (85%) or “agreed” (15%) that it was worth their time and effort to participate in the MADDPR Camp. The two cohorts responded similarly to most statements. One exception was a difference in their level of agreement about their graduate mentors’ “talking to them about their college experiences.” Based on the five-point rating scale, the average for Cohort 1 was 4.31 and for Cohort 2, the average was 3.92, perhaps reflecting the ongoing in-person contact that Cohort 1 had experienced with their mentors during the 2019 camp (Table 2).

Table 2.

Cohort 1 and Cohort 2 students responded to a set of statements about the MADDPR Program, Year 2. (n = 26)

Statement	Cohort 1 Average (n=13)	Cohort 2 Average (n=13)
It was worth my time and effort to participate in the MADDPR Camp.	4.85	4.85
I would recommend the MADDPR Camp to other students.	4.85	4.85
I was excited about the opportunity to participate in the MADDPR Camp.	4.77	4.46
I felt that I gained new skills as a result of attending the MADDPR Camp.	4.62	4.85
The MADDPR Camp was well organized.	4.62	4.69
My Graduate Student Mentor helped me feel welcome and comfortable at the MADDPR Camp.	4.54	4.38
My Graduate Student Mentor made me feel like he/she really wanted to see me reach my educational goals.	4.54	4.08
The MADDPR Camp was better than I expected it to be.	4.46	4.46
My Graduate Student Mentor talked about his/her experiences at college with me and the other members of my group.	4.31	3.92

Cohort 1 students had experienced the camp and fall sessions the prior year, before the COVID-19 pandemic necessitated virtual instruction. They were asked to describe how the virtual camp compared with the in-person camp they attended in 2019. Student responses indicated that they preferred the hands-on nature of the experience the prior year, but deeply appreciated the virtual experience and the work that was involved to make it possible (Student Quotes 4, 5, and 6).

Students' Education and Career Aspirations.

For the past two years of the program’s implementation, as part of the pre- and post-camp surveys, participating students were asked about their education and career aspirations. Matched pairs of responses have been analyzed among both cohorts.

Students Plan to Aim for Higher Education.

Students have been asked about their plans immediately following high school and their degree aspirations. Post-camp surveys in dicated that students were more uncertain about their educational plans and aspirations, however the context of the pandemic is an important consideration. In post-camp surveys, 68% of respondents stated they planned to attend a 2-year or 4-year college following high school. More than half of post-camp survey respondents stated they planned to achieve a masters (16%) or doctoral (40%) degree.

Career Interests Remained Focused on Biomedical and Health Fields.

In the same student surveys, respondents were also queried about their interest in pharmacy, other health professions in general, and nine potential fields in science or health. In post-camp surveys, all but one of the participating students reported they were “very” (28%) or “somewhat interested” (68%) in a career in pharmacy and all but one reported they were “very” (56%) or “somewhat interested” (40%) in a health profession. The early findings regarding education and career aspirations set the stage for future study as the program develops.

Faculty Valued the 4-H STEM Training and Camp.

Following the initial 4-H STEM training session on virtual learning, all respondents gave the training session as a whole an “A” (69%) or “B” (31%) for an overall GPA of 3.69 on a scale of 0 to 4. Following the completion of the camp, nine faculty (53%) of 17 who had facilitated sessions for the program and had participated in the 4-H STEM training in July 2020 completed a follow-up survey in September 2020 after presenting their sessions to the MADDPR Summer Camp participants. One-third (33%, or two respondents) noted they were “very satisfied” with their role as a session presenter for the MADDPR Summer Camp; the remaining four respondents (67%) said they were “somewhat satisfied.”

When asked if they were doing anything differently in their jobs because of the training they received from 4-H STEM, three respondents commented they better understood the needs of students in a virtual environment. As one faculty presenter said, “[I am] not letting the lack of response from students online overwhelm me [and am] creating short bursts of information for the students.” Overall, faculty presenters were positive in their responses to the program noting the quality of faculty presentations and that MADDPR is an “excellent program for the rural eastern shore.”

Graduate Student Mentors Rated the Camp Highly.

All three mentors responded to a follow-up survey in August 2020, the month following the completion of the camp. In surveys following the MADDPR Summer Camp, all three mentors stated it is “very important” to educate high school students about health science topics. Mentors were also asked to grade the sessions during the camp for each cohort. Sessions were highly rated, with most mentors offering an “A” or “B” for all but one session. Cohort 1 student session GPAs ranged from 3.67 to 3.33. Cohort 2 student session GPAs ranged from 3.67 to 3.00

Mentors were also asked to rate the clarity of the scientific information and the instructional activities on a scale of 5 (very clear) to 1 (very unclear). For Cohort 1 student sessions, mentors rated both the information and the activities at 5 or “very clear” (50%) or 4 (50%).15 For Cohort 2 student sessions, mentors rated both the information and the activities at 5 or “very clear” (67%) or 4 (33%). Mentors further reported that “almost all” (67%) or “some” (33%) students were interested in the material presented at camp.

All three mentors responded that the MADDPR sessions were “very valuable” overall for students. Likewise, all three mentors responded that the MADDPR sessions were “very valuable” academically for students. All mentors agreed the camp was well organized, concepts presented were appropriate for high school students, students demonstrated increased knowledge about pharmacy science after their camp experience and improved health science literacy, and — critically — that MADDPR was able to implement a robust virtual learning experience. The graduate students’ comments reflected both their enthusiasm about the virtual program’s ability to provide meaningful content to students and their renewed excitement about science and their decision to enter graduate studies. (Mentor Quotes 1–3).

DISCUSSION

During the summer of 2020, the COVID-19 pandemic forced several student/teacher enrichment camps across the country, including SEPA-sponsored programs, to switch to a virtual format from their original hands-on model (Cold Spring Harbor Laboratory, 2021; Fred Hutch, 2020; McQuillan, 2020; Mellieon-Williams et al, 2021; Tufts University School of Medicine Center for Science Education, 2020; University of Maryland Baltimore, 2020; University of Texas at Austin, 2020). Once the pandemic was declared and UMES cancelled in-person classes, the MADDPR Program made an early and firm decision to pivot to a virtual program rather than delaying or canceling the program.

The MADDPR Program, unlike other programs, offers a focus in drug discovery, pharmacy practice and health education while providing mentoring to underserved minority students in an HBCU setting. As a result, it was critical that the virtual program continue to maintain student engagement and foster student aspirations for college education. The results of MADDPR’s camp experience demonstrate that it was largely possible to achieve both of these primary goals. The assessment of the delivered program was made possible by adapting the evaluation plan to accommodate online collection of survey data (Survey Monkey®), virtual observations of sessions (Blackboard Collaborate Ultra®), and virtual focus groups (Zoom®). Several key elements allowed for a successful virtual transition. First and foremost, there was a strong collaboration of commitment and enthusiasm from students, teachers, instructing faculty and mentors to participate. The PLTW teacher and the returning Cohort 1 students, who largely had a positive experience from last year, served as ambassadors to engage and recruit Cohort 2 students. The virtual sessions were spread over a period of two weeks to accommodate individual participants’ work plans over the summer. Secondly, the virtual sessions that were offered remained true to national science standards. Several contemporary COVID-19 related topics that stimulated interest (Student Quote 1) also were incorporated. The selection of the topics and content were aligned with the students’ PLTW biomedical curriculum. When comparing some of the virtual sessions with in-person camp sessions held in 2019, results indicated that students had a comparable learning experience, with several sessions achieving even higher ratings (Table 1).

Encouraging anecdotal evidence in the literature indicates that both virtual and in-person activities positively impact students’ experiences (Mellieon-Williams et al., 2021). The use of simulations in the learning experience was largely successful due to instructing faculty’s familiarity with technology. Success also depended on ongoing IT support and pre-camp 4-H STEM training in virtual learning for presenters and mentors. A particularly valuable aspect of the virtual programs was the ability to monitor student engagement either through Blackboard Collaborate Ultra® or through some of these simulations like Labster® and Virtual Sterile Compounding®. Using features like live breakout sessions enabled faculty/mentors to interact with students anytime and to track their progress in real-time while performing a virtual experiment. Instructors were able to provide instant feedback and guidance. The students enjoyed the game-like features of the simulation programs offering them a “flight simulator” type experience.

The program focused on offering opportunities for the students to explore a variety of science topics and spark their interest in STEM careers, rather than being centered on measurements of student learning. The diversity of the curriculum topics each taught by a faculty expert in the field helped keep the students interested and engaged in the program. As an additional benefit, the virtual program boosted student confidence (PLTW Quote 1) and encouraged students to explore their education and career paths, with more Cohort 1 students indicating increased interest in life science and biomedical science and also plans to attend 4-year College and pursue advanced degrees.

Near Peer Mentoring.

The MADDPR Program, as part of its core objectives, also nurtured a viable near-peer mentoring support for the high school student participants in the virtual setting. Program data support the importance of University and high school collaborations in providing high quality mentorship to high school students (Hernandez-Matias, 2020). UMES pharmacy graduate students played an active role in facilitating curriculum delivery, practice simulation activities and share learning with their student mentees. The program provided adequate mentor support through pre-camp 4-H STEM training sessions focused on virtual mentoring and assigned small mentor-mentee groups (6:1) to promote interaction. In post-camp surveys, mentors acknowledged the importance of educating high school students about health science topics and corroborated student engagement and enthusiasm during the camp (Mentor Quotes 1, 2, and 3). The high school teacher reported that mentors were meaningfully engaged in the program (PLTW Quote 2). Both participating students and their mentors continued to value the relationship that was established. The MADDPR Program provided a unique professional development opportunity for the graduate student mentors by stretching them from their typical research and education experiences to teaching and mentoring younger students. Such experience would strongly complement their future academic or other professional careers.

Challenges.

There were some notable challenges in pivoting to an online programming. Several students lacked computer and internet access (PLTW Quote 3). Many of these issues were resolved prior to the start of the program with the help of the schools. In a few cases, the PLTW teacher addressed inadequate internet access by pairing students. Going forward, local and state initiatives to support internet accessibility will positively impact these concerns. A recent Maryland legislative action for example, has focused on expanding broadband for the rural communities in the Eastern Shore (Prensky, 2020).

Forging mentor-mentee relationships proved somewhat difficult in the online-only setting, particularly with new students. Because in-person student-to-student/mentor interactions were lacking, a critical aspect of nurturing peer relationships was missing. (Fields, 2009) (Gee, 2004) Program leaders have identified plans to address this by adding session(s) outside of instructional activities (e.g., informal virtual hangout sessions between mentor and mentees and meetings between new and returning students). Near-peer mentoring is challenging in a virtual environment. Recent literature suggests that a structured mentoring plan needs to be developed to adapt to the COVID-19 environment (Dean, 2021) and provide guidance on how to best build a mentoring community online (City University of New York, 2020).

Regrettably, students missed the opportunity to experience university campus life. Although virtual campus tours were offered, they lacked critical elements of campus life, such as group dining at lunch, campus walking tours, and exposure to the array of campus facilities.

Next Steps.

In 2021, the UMES MADDPR Program will explore a hybrid model, offering virtual and in-person experiences. MADDPR will continue its inclusion of two high schools with the goal to increase student participation and concurrently serve two cohorts of approximately 25 students each. The program also will include additional high school teachers to broaden the program’s reach. There are plans to adjust some module content to reflect contemporary topics of interest related to the COVID-19 pandemic (e.g., health disparity and equity and vaccine hesitancy).

Data collection for a long-term comparison study which will survey similarly-attributed nonparticipating students began in February 2021. The study and analysis will assess the effectiveness of the MADDPR Program in creating intended outcomes.

SUMMARY

Based on project data, teacher, mentor, and student feedback, and other evaluation findings, the UMES MADDPR Program is supporting education and career exploration in the biomedical sciences and health care fields among its high school participants. Intrinsic to the program model, the mentor relationships have developed as well. The MADDPR Program offers high school biomedical students robust campus-based science. The program is responsive to a critical educational void in rural Somerset County through a project that meshes with Next Generation Science Standards and draws on best practices. With its commitment to being a “learning organization,” the UMES SEPA project has the capacity to contribute to the literature on best practices in out-of-school-time high school science education.

As the COVID-19 pandemic continues to challenge the delivery of science education programs, key considerations to a successful online transition include, but are not limited to 1) ensuring adequate technology and hardware support; 2) mitigating the costs of transition; 3) providing access to simulation software; 4) ensuring familiarity with online learning through training of faculty and mentors; and 5) securing strong institutional support. MADDPR is also committed to fostering a learning community using social media to encourage student-student and student-mentor interactions among participants across different schools and programs. Moving forward it is possible to envision robust STEM experiences that are offered online. Elements of the virtual learning experience, like the lab simulations, can be incorporated into the program, even as it returns to the on-campus hands-on experience.

ABBREVIATIONS

IRB

Institutional Review Board

MADDPR

Maryland Action for Drug Discovery and Pharmaceutical Research

NIGMS

National Institutes of General Medical Sciences

NIH

National Institutes of Health

NSC

National Student Clearinghouse

PPE

Personal Protective Equipment

PI

Principal Investigator

PLTW

Project Lead the Way

SEPA

Science Education Partnership Award

STEM

Science Technology, Engineering, and Mathematics

UMES

University of Maryland Eastern Shore