Virtual Engagement in a Hybrid Community of Practice: A Descriptive Study on the Training and Integration of New Members into the Genomics Education Partnership during COVID-19

Abstract

This descriptive study examines the experiences of virtually-trained new members in a hybrid distributed community of practice (CoP) focused on undergraduate genomics education. We utilized a sequential explanatory mixed methods research design consisting of an engagement survey for all community members (n=124), followed by interviews with new members (n=15). Survey analysis identifies several areas in which new members do not differ from members with longer involvement, including in motivations for involvement, levels of engagement, satisfaction, and perceived benefits of community involvement. These findings indicate ways in which virtual training and integration was able to facilitate important community outcomes within a new, online context. Our interviews reveal important elements of training new CoP members, including onboarding, implementation, and community engagement opportunities, that successfully facilitated new members’ integration into the community and contributed to their meeting the aforementioned outcomes. The findings of this study provide useful lessons and structures for growing communities through virtual means.

Throughout the twenty-first century, calls for reform of undergraduate science, technology, engineering, and math (STEM) education have consistently echoed throughout academia, industry, and legislature (Henderson et al., 2012; Kezar et al., 2019; Seymour & Hunter, 2019). STEM reform literature has shown that faculty engagement with a community of practice (CoP) supports learning and pedagogy, leading to greater student engagement, retention, and graduation in STEM disciplines (Austin, 2011; Eddy et al., 2022; Fairweather, 2009; Kezar & Gehrke, 2015). STEM faculty CoPs have shown promise in improving individual teaching practices and the ability to drive institutional change (Gehrke & Kezar, 2017; 2019). Given the positive outcomes of CoPs, identifying opportunities to increase access to these communities can advance efforts for undergraduate STEM reform.

The Genomics Education Partnership (GEP) is a nationwide CoP that integrates active learning into the undergraduate curriculum through Course-based Undergraduate Research Experiences (CUREs) centered in bioinformatics and genomics. GEP research projects use bioinformatic tools to explore genome science (i.e., investigating structure, organization, and evolution of genes/genomes). The GEP functions as a distributed (i.e., multi-institutional) CoP, training faculty across the U.S. and Puerto Rico and then supporting their implementation of the GEP curriculum and research projects. Since its founding in 2006, GEP functioned as a distributed CoP that, except for the occasional email between individuals, relied solely on in-person gatherings to support its members; however, the COVID-19 pandemic necessitated fully remote community onboarding and engagement and prohibited the GEP from meeting in-person throughout March 2020-June 2022. Since June 2022, the GEP has functioned both remotely and in-person–as a distributed hybrid CoP–in which virtual engagement is coupled with in-person gatherings and events.

Recent research has highlighted the importance of a hybrid functionality for distributed STEM CoPs which can offer greater flexibility for community members (Kezar, Gehrke, & Bernstein-Sierra, 2017; 2018; Shadle, Liu, Lewis, & Minderhout, 2018). While the transition from fully in-person to strictly remote and, then quickly thereafter, a hybrid model of engagement posed challenges to the GEP, this served as an opportunity to better understand the experiences of new members as they enter an existing hybrid CoP. Lessons learned from the GEP can contribute to future CoPs seeking to grow their community with more accessible, remote options. This descriptive study aimed to assess the experiences of new members trained- and onboarded-virtually and identify methods that positively contributed to their integration within the community, by addressing the following research questions:

1. To what extent are virtually-trained new members participating in, and reporting benefits of, the GEP CoP, and how do these benefits differ from those experienced by in-person trained members?

2. Which aspects of the GEP facilitated the integration of virtually-trained new members into the CoP?

Literature Review and Project Background

This study was informed by the literature on CoPs and their role in STEM reform efforts. A CoP is a group of people interacting regularly to learn how to do something for which they share a common passion or concern (Allee, 2000; Kezar et al., 2018; Lave, 1988; Wenger, 1998; Wenger-Trayner & Wegner-Trayner, 2007). Kezar et al. (2018) identified community, domain, and practice as the three common factors defining all CoPs; as the name suggests, members in a CoP are a “community” of inter-connected individuals, with a common purpose or value (domain), that share ideas and resources (practice). CoPs are one of the most cited strategies in the STEM change literature, both as a guiding theoretical framework and a means for achieving change (Reinholz, White, & Andrews, 2021). This literature has largely focused on distributed communities that exist across institutions, although recent research has elaborated on the necessary design characteristics for effective community design and engagement in distributed hybrid CoPs, which rely on both in-person and virtual engagement (Donaldson, 2020; Eurby & Burns, 2012; Johnson, Jakopovic, & von Renesse, 2021; Kezar et al., 2017).

Design principles that successfully contributed to engaging CoP members include: 1) organic design which fosters evolution; 2) open communication lines between and among members; 3) varying levels of participation; 4) public and private community spaces; 5) focus on value of participation for sustaining the community; 6) fostering group processing and learner-directed opportunities; and 7) a predictable cycle of engagement and events within the community (Eddy et al., 2022; Kezar et al., 2017). Additional design considerations specifically for STEM reform arose from research into a subset of CoPs referred to as communities of transformation (CoT – Kezar et al., 2018). These communities share characteristics with CoPs but are also distinct in that they also a) have an “engaging, well-articulated, and clear philosophy” (Kezar et al., 2018, p. 843); b) enact that philosophy in the operations and engagement opportunities within the community; and c) focus on relationships formed within the community to support faculty in engaging the philosophy and approach to teaching on their own campuses. Outcomes associated with both CoP and CoT engagement focused on change in undergraduate STEM education are well-documented (see Eddy, 2022; Glaze-Crampes, 2020; Hill et al., 2019; Kezar & Gehrke, 2015; Miller & King, 2019).

Much of the research cited above focuses on distributed hybrid communities. These communities utilized a combination of in-person (e.g., training, annual meetings) and virtual (e.g., online newsletters, virtual meeting) engagement. The GEP largely functioned as a quasi-hybrid CoP for much of its existence prior to the COVID-19 pandemic, offering solely in-person trainings and an annual in-person faculty workshop, coupled with occasional digital communications and remote meetings of a few members working on a particular task (e.g., writing committee, curriculum revisions working group). Throughout 2006–2018, under a centralized leader, the GEP grew to a community of 125 faculty members from 108 institutions as a distributed CoP. With the announcement of retirement of the founder and longtime leader, the GEP began the transition to a more distributed community leadership structure that included plans to grow the community through a variety of strategies, and relied heavily on digital communication tools.

In 2019, the GEP was awarded two five-year grants–an Improving Undergraduate STEM Education (IUSE) grant from the National Science Foundation (NSF; Award # 1915544) and an Innovative Programs to Enhance Research Training (IPERT) grant from the National Institutes of Health (NIH; Project No. R25GM130517). A main focus of the NSF-IUSE grant was to develop a virtual model for training, mentoring, and supporting new GEP faculty while the NIH-IPERT grant was to increase faculty and student engagement in the GEP by leveraging regional clusters of participating GEP institutions (i.e., Regional Nodes). Prior to these grants, new members would officially join the GEP community after attending an in-person, multi-day training workshop. The three main objectives of new member training were to 1) introduce new members to the GEP research projects, bioinformatics tools, and research protocols; 2) familiarize new members with the GEP curriculum and CURE pedagogy; and 3) introduce new members to the GEP communication tools, community structure, and resources that support year-round work and CURE implementation. The paired approaches of the two grants gave GEP broader outreach and a unique opportunity to determine if a seemingly more economical virtual training strategy was actually sufficient to support implementation and persistence in the GEP by studying the differences in new member experiences between the existing in-person opportunities and new virtual training opportunities. However, onset of the COVID-19 pandemic forced the GEP to move all efforts for community growth and continued engagement entirely online, which necessitated the scaling up of virtual training and all member events to be offered solely online, rather than the intended hybrid model. Scaling the virtual training opportunities allowed for rapid growth of the community over the first two years of grant activity, which comprised 213 faculty from 181 institutions as of September 2021, and far out-paced initial community growth plans to reach 180 institutions over five years.

Recent literature has documented the ways in which disruptions of the COVID-19 pandemic affected higher education, including many examples of how the move toward online course delivery and professional development impacted STEM education (e.g., Erickson et al., 2021; Sikora, 2021; Speer, Lyon, & Johnson, 2021; Walsh et al., 2021), and efforts by its members to ensure successful delivery of the GEP curriculum in entirely remote contexts (Lopatto et al., 2023). While researchers have reported on the benefits and lessons learned from moving toward online CoP functioning as a result of COVID-19, these studies are limited by focusing largely within single institutional contexts (Bolisani et al., 2020; Haas et al., 2021; Yang, O’Reilly, Houghton, 2020).

The literature is replete with guidance for designing online learning experiences for adult learners. Hokanson et al. (2019) drew on this literature and developed a new structure for online synchronous workshops, providing professional development to STEM graduate students and postdocs, and identified valuable elements of effective workshop design— completing tangible assignments and structured reflection time develops a sense of community among participants. While these design characteristics can assist in developing individual workshops, the study does not extend beyond workshops to developing and integrating into an existing community.

Until recently, much of the literature related to online professional development and engagement has reported on experiences and outcomes of online training, not on comparing experiences and outcomes between virtual and in-person engagement (Rogers et al., 2022). More recent studies have reported on comparative outcomes between virtual and in-person professional development training and workshops in particular (Mallonee et al., 2017; Mullin et al., 2017; Rogers et al., 2022; Sol, Fuchs, & Mehl, 2016;). The findings of these studies highlight that similar outcomes can come through online and in-person delivery methods. Again, these studies tend to focus specifically on focused training, not on continued community engagement.

Our study seeks to fill the gaps highlighted above by describing the new member experience in an established distributed CoP that transitioned to fully remote engagement. We do so by first comparing the experiences of new virtually-trained members to those of more seasoned, in-person-trained members. We then elaborate on what allowed for successful integration into a pre-existing distributed community through solely virtual means. Our findings can inform communities seeking to either transition to greater virtual engagement, or to design solely virtual, new communities that could provide similar benefits to more traditional hybrid CoPs.

Methods

This study utilizes a sequential explanatory mixed methods design (Warfa, 2016), which is typical of CoP research that utilizes both quantitative and qualitative methods to examine trends and underlying mechanisms within communities (Fontaine & Millen, 2004). In line with an explanatory design, we began with a quantitative survey of all CoP members–focused on genomics research and teaching–that allowed us to examine trends in new member experiences compared to more seasoned members. Results of the quantitative survey partially informed interview protocol development for the qualitative portion of the study, in which we interviewed new community members to better understand the dynamics of their experiences while joining the community.

Data Collection and Participants

Our study community–the GEP–consists of faculty teaching all levels of undergraduate biology at community colleges, primarily undergraduate institutions, and research universities nationwide. Throughout 2006–2018, under the leadership of the founding Program Director–Dr. Sarah C.R. Elgin–the GEP grew by approximately 15–25 members each year by offering one or two in-person new member training workshops annually. After completing the training, given by Dr. Elgin and core staff, new members could access community resources, claim and submit scientific projects, and attend annual in-person GEP member workshops for additional professional development and community building. Upon the retirement of Dr. Elgin, the GEP community decided to transition to a distributed leadership model, offer virtual new member training, and organize most in-person activities around smaller, regionally-anchored clusters (i.e., regional nodes). This study is part of a larger NSF-funded IUSE project designed to examine the growth and evolution of the community as it transitions to a distributed leadership model, conducts virtual training, and supports the vast array of community members. Institutional Review Board approval was obtained from the Office of Research Compliance at The University of Alabama (EX-19-CM-118). Informed consent was procured, and participants were able to opt-in/out of individual survey and interview methods.

To answer the first research question, we developed a survey designed to measure the experience of all GEP community members (i.e., new and seasoned members) in Spring 2021. The survey was designed to capture important aspects of community involvement, including motivations for joining, engagement in GEP activities, and benefits associated with participation, in line with previous research on CoPs (Gehrke & Kezar, 2017; 2019). Between April and June 2021, GEP staff used the Qualtrics platform to generate the quantitative survey and automate email invitations and reminders. Invitations were sent to 208 active GEP faculty members, of which 124 completed the survey (60% response rate). Of the responding members, 42% identified as new, virtually-trained members with an average of 1.08 years (SD = 1.19) of GEP involvement, compared to 8.24 years (SD = 4.42) for members previously trained in-person (t(84.7) = 13.12, p < 0.001, Cohen’s d = 2.07); therefore, training modality served as a strong proxy for years of involvement within the community.1 The virtually-trained and in-person trained samples were similar based on participant demographics, including race (22% faculty of color vs. 24%, respectively), gender (65% female vs. 57%, respectively), identifying as first-generation college students (23% vs. 26%, respectively), and disclosing a disability (8% vs. 3%, respectively).

Upon analysis of the survey findings, and to answer the second research question, we developed interview protocols to better understand the experiences of virtually-trained members as they began to engage with the community. The semi-structured interview protocol was informed by the survey findings and asked participants questions related to their involvement with GEP, including how they learned about the CoP, their training and engagement experiences, and their views on the community at that point in their GEP membership. We split the sample of 52 virtually-trained members into four categories based on their survey responses: those who had utilized both the GEP curriculum and participated in the GEP CUREs (56%), those who utilized the curriculum but did not participate in the GEP-CUREs (19%), those who had yet to participate in either (12%), and those who had not consented to participate in interviews (12%). We randomized the order of participants in each of the first three groups and invited members to participate in interviews until we reached saturation. Most interviews were conducted between November 2021 and March 2022, lasted between 20 and 60 minutes, and were digitally recorded and transcribed. These interviews were conducted solely by two members of the research team; one member was from the external research community, while the other was a PI for one of the GEP grants and current member of the community. Analyses conducted of the interview data reveal similar experiences reported and themes uncovered between the two groups of interviewees, suggesting little bias in one of the researchers being a member of the community.

The interview sample consisted of 15 virtually-trained members of the GEP. At the time they completed the survey, the participants had between zero and two years of experience in the GEP (M = 0.87, SD = 0.52). The 15 interviewees were fairly representative of the larger sample of virtually-trained members, with 27% identifying as faculty of color, 53% identifying as female, 27% identifying as first generation, and 13% disclosing a disability.

Data Analysis

To understand the experiences of GEP involvement for virtually-trained members, we analyzed their survey responses in comparison to those provided by in-person-trained members. Descriptive statistics of the survey questions were evaluated to understand the experiences of virtually-trained members in six crucial areas: community participation, community knowledge, motivations for and benefits of involvement, plans for future involvement, and satisfaction with the community. Next, we compared responses of the two groups (virtually-trained and in-person trained) utilizing either independent sample t-tests or Chi-square. Based on prior research, we expected that the longer membership durations of in-person-trained members would contribute to higher levels of community knowledge and involvement and perceived benefits thereof (Gehrke & Kezar, 2019). Consequently, we were especially interested in areas where the new and seasoned groups did not statistically differ, which could indicate strengths in the introduction and integration of new virtually-trained members into the community.

Transcripts were reviewed by two members of the research team to identify both emerging themes for each question/content area of the interview protocol and overarching themes related to community involvement and integration. These two members were external consultants to the community, allowing for a more objective positionality in relating to and understanding the data. Interviews were coded utilizing a constant comparison approach (Creswell, 2013). Coding was largely an inductive process, by which codes emerged intrinsically from the data without a predetermined codebook.

Validity and Trustworthiness

An advisory committee, pre-validated survey items, and independent coding were implemented to ensure validity and trustworthiness. Both the survey and interview protocols were informed and reviewed by members of an internal GEP advisory committee, which included an external evaluator and an expert on education research methods. This advisory committee provided member checks of emerging findings to ensure findings were reflective of the community and member experiences. The survey utilized pre-validated items, which were informed by the CoP literature in previous studies (Gehrke & Kezar, 2017; 2019; Kezar et al., 2017). Finally, to ensure agreement on themes and resolve discrepancies, each transcript was independently coded by two researchers.

New members in the GEP are generally denoted as individuals who are recently trained and in their first year of implementation. Given the timing of the virtual training development and survey administration, we defined new members as those who were virtually-trained, which also correlated with the average years of experience of approximately one year. For the remainder of this paper, we use the terms virtually-trained/in-person-trained interchangeably with new members/seasoned members.

Findings

Our findings are split into two parts, one for each research question. We answer research question one by examining trends in responses to survey items for virtually-trained members and comparing them to those of in-person trained members in six areas: motivations for involvement, participation, community knowledge, satisfaction with the community, benefits of involvement, and plans for future involvement. The findings highlight areas in which new and seasoned members show similar responses, suggesting strengths of the virtual onboarding process and community integration. Below we provide key statistics calculated from these analyses, and all relevant statistical tests are included in Appendix A. Subsequently, we answer research question two through analyzing 15 interviews with virtually-trained members, highlighting aspects that supported new members’ virtual integration into the community.

Survey Findings

Motivation

Survey participants were asked to indicate their motivations for initially joining the GEP from a list of possible reasons, the response frequencies of which are presented in Figure 1. The top three reasons virtually-trained members gave for joining the GEP are desires to learn strategies for involving students in research (83%), improve teaching (67%), and learn a new curriculum (60%). These were also the top reasons given by previously in-person-trained members (82%, 72%, and 74%, respectively). The virtually- and in-person-trained groups only varied significantly in joining GEP in order to reinvigorate professionally (23% vs. 40%, respectively; X² (1, N = 124) = 4.0, p < 0.05), suggesting that despite joining under very different circumstances (i.e., virtually-trained joined during the COVID-19 pandemic) and modalities, newer virtually-trained members are seeking out the GEP for similar reasons to their more seasoned peers.

Figure 1.

Percentages of participants indicating reasons that motivated them to join the GEP. (a) Virtually- and in-person- trained members showed similar motivations for joining the GEP with one exception. In-person-trained members were more likely to cite a desire to reinvigo rate professionally compared to virtually-trained members according to Pearson Chi-square test of independence (p < 0.05).

Community Participation

Participants were then asked to indicate their frequency of participating in/attending GEP-specific events (e.g., summer workshops, committee meetings, regional node meetings) and engaging in GEP-related activities (e.g., presenting about the GEP, serving on a writing committee or co-authoring a GEP publication, mentoring students) since joining. Those who trained in-person exhibited significantly greater involvement in nearly all of these areas (see Appendix A), which is expected given their longer time spent with the community and more opportunities to engage. One area where we did not observe significant differences was in attending regional node meetings (X² (3, N = 123) = 2.68, p = 0.44). Regional nodes are a new initiative, beginning in 2019, within the GEP, in which members are assigned to a node based on the geographic region of their institution (e.g., New England, Southeast, Pacific Northwest). Fifty-six percent of virtually-trained members indicated they participated in their regional node at least once since joining, compared to 62% of in-person-trained members, with 33% of virtually-trained members indicating participating two or more times compared to 27% of in-person-trained members. Both groups sought out community participation through their nodes, despite one group having much more experience engaging with the community in person than the other. The lack of statistical difference in regional node participation between the two groups suggests a possible community structure that contributes to integrating new members into an existing community.

Perceptions of the Community

Participants were next asked to indicate their knowledge and perceptions of the GEP community, including their satisfaction with it. When asked to rate their knowledge of aspects of the GEP community (e.g., “I know how to become more involved in the GEP community,” “I know who to go to if I have a question about GEP,” and “I know how to become more involved in the leadership of GEP”), virtually-trained members and in-person-trained members varied significantly, with Cohen’s d effect sizes ranging from 0.32 to 0.99 in the direction of more seasoned members indicating a greater knowledge of the community. However, the two groups did not vary significantly in viewing the GEP as a welcoming community (t(122) = 0.17, p = 0.43), with virtually-trained members exhibiting very similar scores (M = 4.71, SD = 0.50) to in-person-trained members (M = 4.69, SD = 0.60). Despite having much less experience in the community and solely virtual engagement, new members still viewed the community as welcoming as their seasoned peers. New members are also satisfied with the community, exhibiting similar levels of satisfaction (M = 4.47, SD = 0.86) compared to their in-person-trained peers (M = 4.54, SD = 0.89; t(121) = 0.44, p = 0.33). Additionally, virtually-trained members are somewhat or extremely likely to recommend the community to a colleague (94%), similar to their in-person colleagues (96%, X² (2, N = 123) = 1.26, p = 0.53).

Satisfaction of new members also extends to the “products” of the GEP. Virtually-trained members are relatively satisfied with both GEP curriculum (M = 4.62, SD = 0.82) and research projects (M = 4.47, SD = 0.81) compared to their in-person trained peers (M = 4.50, SD = 0.79; and M = 4.56, SD = 0.80, respectively). These groups exhibit no statistical difference in their satisfaction with either the curriculum (t(122) = 0.79, p = 0.22) or the research projects (t(121) = 0.58, p = 0.28). They also exhibit the same likelihood of recommending GEP curriculum (98% for virtual vs. 100% for in-person; X² (2 N = 124) = 1.42, p = 0.49) and GEP research projects (98% for virtual vs. 100% for in-person; X2 (2, N = 123) = 2.42, p = 0.30) to a colleague.

Perceived Benefits of Involvement

The survey included 23 items assessing perceived benefits gained from their involvement with the GEP. Members trained in-person exhibited significantly higher levels of benefit from the community for 17 of the 23 benefits, with Cohen’s d effect sizes ranging from 0.37 to 1.74. However, new and seasoned members did not exhibit significant differences for six benefits, which are highlighted in Figure 2 (all benefits are listed in Appendix A). Despite having significantly fewer years of involvement with the GEP, new members are just as likely as seasoned members to report that their involvement led to changes in their teaching practice (Teaching), motivated them to be innovative in their practice (Innovation), allowed them to gain access to new curricular/pedagogical resources (Curriculum), connected them to a local (i.e., geographic proximity) network (Network), recharged/energized them in their work (Recharged), and lent them credibility for their approach to teaching (Credibility). These findings suggest that the mechanisms put into place by the GEP allowed new, virtually-trained members to gain benefits from involvement rather quickly, not only in their instructional practices but also in developing a community of support and giving them renewed energy for their work.

Figure 2.

Means + standard error for six benefits, in which virtually-trained and in-person-trained GEP members do not vary significantly according to independent t-tests. Measurement scale: 1 = Not at all; 3 = To some extent; 5 = To a great extent.

Future Involvement

The final set of survey questions asked participants to indicate their likelihood of future GEP involvement in the following areas (see Figure 3): attendance at annual or occasional GEP events (Events), teach with the GEP curriculum (Teach), engage in GEP scientific projects (Science), regularly attend a committee meeting (Committee) or a regional node meeting (Node), pursue a leadership role in the GEP (Leadership), contribute to an occasional GEP task (Contribute), or reduce participation in the GEP (Reduce). The only future involvement in which the virtually- and in-person-trained groups significantly differed was in the likelihood of regularly attending committee meetings (t(121) = 2.02, p < 0.05). Again, despite having much less experience with the community, new members by and large exhibit the same interest in being a part of the GEP community as their more seasoned peers, suggesting a positive and effective virtual integration into the community.

Figure 3.

Means + standard error for the likelihood of future involvement in the GEP. Measurement scale: 1 = Not at all; 2 = Somewhat likely; 3 = Very likely. (a) Virtually- and in-person- trained members showed similar likelihoods of future involvement with one exception. In-person-trained members are more likely to attend future committee meetings compared to virtually-trained members according to independent t-tests (p < 0.05).

Summary of Survey Findings

While many differences between virtually- and in-person-trained GEP members were evident and as expected, the areas in which both groups were similar revealed positive factors in which new members virtually integrated into the GEP community. Despite undergoing training and community integration during drastically different contexts (pre vs. during COVID-19), new and experienced GEP members by and large reported the same frequency of motivations for seeking out the CoP. Despite little experience, new members participated in regional nodes at the same rates as experienced members, indicating an avenue to welcome new members in virtual environments (more below). Additionally, we observed no differences in sense of and satisfaction with the community, despite new members having much less experience with it; the fact that this occurred solely from virtual engagement is positive. The early reported gains for new members in teaching and networking, coupled with intended future involvement, suggest value in virtual forms of new member integration. Findings from the next phase of our study suggest several reasons for why we observed these positive outcomes.

Interview Findings

Our interviews were intended to better understand the experience of new, virtually-trained GEP members to further elicit contextual and qualitative information to better elaborate on the quantitative findings. In this section we describe the following two themes that emerged revealing the factors that facilitated the integration of new members into the GEP community: 1) virtual training, leading to GEP curriculum implementation and 2) opportunities for virtually trained members to connect with the GEP community.

Virtual Training and Curriculum Implementation

In autumn 2018, the GEP began exploring virtual training (in addition to the long-standing in-person format) as an option for new members who are geographically isolated or cannot attend in-person. Training defaulted to fully virtual when the COVID-19 pandemic restricted in-person gatherings. As we mentioned in the introduction, the three main objectives of new member training were to 1) introduce new members to the GEP research projects, bioinformatics tools, and research protocols; 2) familiarize new members with the GEP curriculum and CURE pedagogy; and 3) introduce new members to the GEP communication tools, community structure, and resources that support year-round work and CURE implementation. We identified four sub-themes—virtual format, training content and organization, group discussions, and trainers— that were crucial to new member integration.

Virtual Format.

Virtually-trained members underlined convenience as an advantage of the virtual format, especially given that pandemic restrictions precluded in-person gatherings. Irrespective of pandemic status, most participants expressed preference for virtual training because it eliminates the need to travel, and better accommodates their work and personal schedules. One member shared:

I don’t know if I would be able to go somewhere in-person because of my family situation, my other obligations, I don’t know if I would have that dedicated time that I could do it in-person, but online worked beautifully.

However, from the interview data arose the perspective that “for somebody who has a background in genomics/bioinformatics, a virtual training may be enough.” This perspective implies that members who lacked genomics or bioinformatics backgrounds may need in-person training to reach mastery of GEP’s curriculum and protocols; however, despite this perspective, the members interviewed were able to complete virtual training, and many implemented the GEP curriculum in the 1st year after training.

Training Content and Organization.

Members generally agreed that the detailed in-class gene annotation presentations followed by hands-on homework contributed to their ability to implement GEP successfully. One member represented many by expressing appreciation for the detailed presentation of the materials:

I personally thought that all of the walkthroughs with screenshots were amazing… I had already specifically thought about bioinformatics analysis in the way that you write super thorough instructions for somebody else who’s never done it, and so I appreciated that approach to the material.

Members also had the opportunity to learn about three different scientific projects they could implement as a CURE, and discussed implementation strategies, which further contributed to their successful implementation.

Group Discussions.

Members viewed the scheduled group discussions, conducted as virtual “office hours” during which trainers could pose or answer questions with trainees, as an opportunity to learn from and support each other. During these check-ins, the groups spent most of the time discussing questions and challenges they faced related to the training homework and their implementation plans. Participants emphasized that group discussions reiterated important concepts and helped them process the GEP curriculum to retain information.

With the large volume of information from training to process and, for many, the short turnaround time before implementation, members also appreciated discussing implementation plans/approaches with their peers and receiving constructive feedback. They also heard stories about implementation strategies that seasoned GEP members have tested. One new member drew inspiration from the experiences of other seasoned implementers:

… the most valuable thing was actually listening to [what others] had to go through, the struggle before us, and kind of getting the confidence that it can be done in your way whatever works for you in your community… That gave me the confidence to at least try it.

Almost every participant alluded to how group discussions made training an even more welcoming space. One member summed up the general sentiment by sharing that group discussions:

…made it feel like there was permission for it not to always go really smoothly…you have another faculty member who’s like, “We tried this and it was a disaster,” and “This is what I learned from it,” and “This is how we can kind of move forward.”

Trainers and Staff.

In addition to group discussions, members highlighted interactions with trainers and staff as another valuable aspect of their training experience. Members praised the training team for being “very welcoming” and creating a “very safe environment to share ideas and share anything.” Members recalled that the training team allowed ample time for participants to ask questions and made all training materials (e.g., slide decks) available for members to review later. As one member described in detail:

One of the things that I really liked about our training was all three of the instructors left a lot of time for us to ask questions and really allowed us to kind of explore, because all of us were coming at it from a different…I mean, we had somebody who was a computer science kind of programming person who was thinking about doing this for bioinformatics. We have people who are working with seniors or even grad students with gene annotation, and I felt like they were really responsive to all of us and by opening up those forums where we could ask the questions that we needed for us, it helped everybody.

Implementation Experiences.

While the context of implementation varied widely based on the institution type (e.g., community college, public 4-year), their students’ class standing (i.e., lower- vs. upper- division courses), and departmental barriers (e.g., faculty shortages/teaching overload), nearly all virtually-trained new members reported success in implementing some aspect of the GEP curriculum. Some implemented just the inquiry-based instructional modules without the research component, while others used both the modules and GEP research projects in their courses or with students pursuing independent-study/mentored research experiences. The majority implemented the GEP curriculum as a CURE while embedding GEP modules as homework or as in-class introductions to group annotation projects. Some members, in addition to CUREs and modules, also mentored individual students in independent studies. For these members, some of their mentees would go on to serve as teaching assistants, usually for the same course, in later academic terms. Members who incorporated only GEP modules into their courses, either as in-class work or homework, cited two common reasons: 1) a wet-bench lab curriculum had already been established at their institution (specifically heading into post-COVID hybrid instruction) or 2) their demanding teaching/research schedules preclude exploration of other implementation strategies.

When asked to reflect on their experiences, members expressed satisfaction with the progress and outcomes of their implementation approaches given the circumstances (e.g., virtual engagement, COVID-19 pandemic). A recurring motif was the virtual learning environment itself. Some members noted the technical difficulties that arose – namely, unstable internet connection. Others pointed out that Zoom, even with its collaborative features like breakout rooms, was still only a substitute for the in-person learning that would naturally be more conducive to group work as well as student engagement. However, one member attributed a part of their overall success to their asynchronous implementation (aided by the virtual format), sharing that their students “could work on [annotation projects] when they want to, in the middle of the night if they needed to.” In sum, members generally considered their implementation experiences successful and felt optimistic about future implementations.

Support from GEP trainers and staff was crucial to post-training implementation. Interview participants attributed their implementation success largely to GEP office hours and virtual teaching assistants (former students from GEP classes who assist current students via virtual office hours), while they also recognized the support offered by the GEP staff. Lastly, members credited their implementation success to GEP online platforms. They accessed resources on one or more of the following: GEP website, project-specific Slack channels, and the Q&A forum on the community-wide Trello board.

Virtual Community Connection Opportunities

Faculty Workshop.

The signature event of the GEP community is an annual meeting for all GEP members (National GEP Faculty Workshop), providing members with the opportunities to celebrate achievements of the prior year, plan future work, network, share implementation strategies, and gain insights into GEP’s current state or progress toward its goals. The meeting itinerary also includes informal social time. When the national workshop transitioned from in-person to virtual, informal socialization was substituted with working groups conducted through Zoom breakout rooms.

Reflecting on their experiences, members generally described national workshops as “useful,” “informative,” and “interesting.” Interview participants reported the sessions (i.e., talks, presentations) and GEP community-building activities as equally beneficial. For the sessions, members emphasized implementation lightning talks (i.e., brief presentations on tested strategies), keynote speakers as well as other research presentations as most impactful. One member described their positive experience: “So, I loved it… I was exposed to different techniques that people would use; different ways people have implemented it, different things that are going on.”

Furthermore, members spoke about opportunities to connect with the GEP community. They found that working groups (i.e., virtual sessions facilitated by different GEP committees addressing specific projects or topics) act in part as “refreshers on the projects, where they are and where they’re going,” and reiterate the mission that there are “a lot of scientists out there who are really concerned about education and bringing these types of opportunities to students in a cost-effective way.” Members also viewed the talks/presentations and group discussions as opportunities to gain “a better sense of how to be involved” and further insights into the inner workings of GEP’s distributed leadership model.

While most interview participants found value in the virtual national workshop and expressed interest in attending again, many indicated strong preference for wanting to experience the in-person format. Some expressed preference for the hybrid format or thought it would serve as a back-up option if an in-person gathering becomes unfeasible while some would opt for the virtual-only format (though they did acknowledge the benefits of attending in-person). They contextualized this virtual-only preference by describing their challenges with having conflicting teaching/research and personal schedules. One member elaborated on their virtual-only preference:

I’m able to come because it’s virtual. I teach…I’m teaching four classes in the summer. There would not be a moment for me to get away if it wasn’t virtual. I’ll probably, unfortunately, miss chunks because I have to go teach.

While the virtual workshop provided opportunities for GEP members to still participate in their signature annual event and proved useful, it seems that virtually-trained members still felt that something was missing given what they had heard about previous in-person meetings from other GEP members.

Regional Nodes.

Compared to the annual national workshops, regional node meetings happen on a local scale and were originally envisioned as a cost-effective way to create smaller in-person communities and networking opportunities within GEP. However, during the pandemic, regional node meetings only took place virtually. Regional nodes are meant to keep members professionally and socially connected year-round. They also welcome student involvement by inviting students to give presentations on their current GEP work. Interview data showed that the frequency of meeting attendance varied among members; some members could only attend regional node meetings as their schedules permitted while others have attended as many times as their regional nodes had hosted meetings. The number of meetings that regional nodes hosted also varied. Some members recalled no announced meetings since they joined GEP whereas others noted their regional nodes held regular meetings anywhere from once per month to a few times per year. One member who belonged to a regional node with less activity “wished there were more engagement and more people [attending].”

For members who had the opportunity to participate in virtual regional node meetings, they frequently described them as “beneficial” and “helpful,” associating their participation with an increased sense of belonging to the general GEP community. During their interviews, members spoke about how regional node meetings sustain the community-building spirit that gained momentum at national workshops. Members appreciated that the smaller scale of regional nodes made it easier to build connections with other GEP members virtually and learn about different implementation strategies, as well as share research updates. One member found regional nodes less overwhelming than the national workshop because they could “see some familiar faces every time,” whereas at the national workshop it can be “hard to interact with people I don’t know” in large, virtual rooms. Additionally, some members recounted their experiences with not being able to maintain connections with formal mentors (usually assigned to them shortly after training). These members perceived regional nodes as a safe and inviting space that facilitated new mentorship connections for them.

Other Forms of GEP Connection Opportunities.

In addition to engaging with the community through the annual national workshop and regional nodes, members were asked to share their experiences with other forms of GEP community involvement. While a few noted no other involvement besides using GEP curricular resources and attending the national workshop, the majority of members shared various ways in which they engaged with the GEP community.

Typical activities and interactions members experienced were:

1. Mentoring relationships with more experienced GEP members or senior faculty within the community

2. Preparation of GEP-related posters and presentations for research conferences

3. Smaller-scale collaborations on curriculum development; either individually or pairing with other GEP members

GEP committee meetings are different in nature from another form of involvement for new members than those listed above. These have a built-in formal leadership structure and typically involve a larger group of people. Interview data revealed that members’ involvement ranged widely from listening in/not actively participating during meetings or attending meetings only when convenient, to attending regularly and/or being part of decision-making processes, to leading the committee. Specifically, members cited involvement with the publications or initiatives for one or more of these four committees: Assessment; Diversity, Equity, and Inclusion; Science and Information Technology; Professional Development and Mentoring. Participants who had participated in committees often learned of them from the national workshop, felt their structure was welcoming, and allowed them to get involved during early stages of their GEP membership.

Summary of Interview Findings

Our interview findings reveal that effective training and support for curriculum implementation, coupled with various opportunities for community involvement, facilitated community integration for new, virtually-trained members. While many of these mechanisms were designed for in-person or hybrid engagement, their structures in a virtual environment still allowed new members to learn about the curriculum/pedagogy and experience connections with other members, thus allowing them to become members of a community without the physical, in-person interactions that were so critical to the community pre-COVID.

Discussion

This descriptive study on the virtual integration of new members into an established STEM CoP provides valuable insight for both CoP leaders and researchers. By examining the reported experiences of new CoP members, we identified which involvement opportunities they utilized and benefited from within their first year. For example, involvement in small-scale regional node activities, in which new members connected with faculty at nearby institutions, contributed to an early sense of belonging and connection to the community. Further, we identified important outcomes, in which new members perceived the same levels of benefits as seasoned members despite having much less experience with the community. The balance of effective virtual training and implementation support, coupled with various opportunities to engage in the broader community, offer a roadmap of sorts for how to develop STEM CoPs.

The GEP’s experience with moving toward a virtual model of training and community engagement, while maintaining similarly positive outcomes for new members, provides critical lessons for communities considering a similar transition. Prior to the transition, GEP members almost exclusively engaged with the community in person. Our findings highlight how the training and community engagement aspects of CoP development can remain effective regardless of their modality. Signature events, community culture, and engagement with peers around implementation are significant design principles for CoPs (Gehrke & Kezar, 2017; 2019; Kezar et al., 2017), and our study suggests that these design aspects of CoPs are similarly as crucial for virtual engagement as they are for in-person/hybrid communities. Our findings show that virtual engagement could be a continued strategy for CoPs in a post-COVID world for new and pre-existing communities seeking more cost-effective and environmentally sustainable ways to foster community.

STEM CoP research indicates that peer-to-peer relationships and learning are essential for advancing outcomes of communities (Kezar et al., 2017; 2018), and their importance is reinforced by the ways in which the GEP restructured training as it moved toward a virtual format. Whereas the original training and onboarding was conducted by GEP leaders and staff in person, the move toward a distributed model of virtual training necessitated the expansion of the cadre of trainers to include more experienced members of the community. This facilitated connections among new and experienced GEP members that may not have developed until a later time in the pre-COVID era, reinforcing prior research on the value of including intentional time for engaging with others (Hokanson et al., 2019).

New members developed further connections in their regional nodes through virtual meetings, which provided access to the broader community much earlier in their GEP tenure compared to what could have occurred in previous years. As described above, new members were also provided with a formal mentor from the community, but we found little evidence in our interviews that these were as effective as the other connections new virtually-trained members made through their training cohort and regional nodes. It could be that more mentor-like relationships will form in the progression described by Kezar and colleagues (2017), moving from informal peer learning and brain-storming toward formal mentoring. Our initial findings suggest that new members benefit the most from simple access to opportunities to engage with others, specifically in the form of discussion spaces where they could share their own voices and experiences.

Our focus on the experiences of new GEP members spotlights an important advancement in this area of CoP research. Many studies of CoPs focus on general membership involvement and outcomes associated with community involvement (Eddy, 2022; Glaze-Crampes, 2020; Hill et al., 2019; Kezar & Gehrke, 2015; Miller & King, 2019). Gehrke & Kezar (2017; 2019) found that longer involvement in a community is naturally associated with greater perceived benefits, findings which are supported by our quantitative findings. However, despite their very short period of involvement, new members in our study expressed benefits related to learning new and innovative teaching approaches, accessing curriculum, networking with others, recharging, and gaining credibility for their approach to work. These could be areas of “early benefits” of community involvement that have not previously been highlighted in CoP research due to prior studies focusing on all community members rather than a subgroup of newly trained/joined. Further research into these and other benefits of initial community involvement can reveal important areas of focus for STEM CoPs to recruit faculty, grow networks, and retain members.

While this study focused on the experiences of new members, it also revealed key aspects of community structures that facilitated the move toward distributed and virtual engagement. Kezar & Gehrke (2017) presented a six-component sustainability model–assessment, leadership, feedback, professional staff, strategy, and financial–for CoPs that communities should consider as they move through stages of CoP development. By focusing on these six components at different times, CoPs can work to transition through the five stages of CoP development: potential, coalescing, maturing, stewardship, and transformation. At the point of leadership transition and expansion, the GEP was likely oscillating between the maturing and stewardship stages of community development, focusing on growing the community and balancing new ownership over the domain of the community with new leadership structures.

Based on our interview findings, three components of the Kezar & Gehrke (2017) sustainability model seemed to play an essential role in successfully bringing new members in through virtual structures. The first component was the distributed leadership model, in which more community members were now responsible for bringing their ideas to committees, regional nodes, and training interactions with new members. This allowed for not only an increase in the scale of the community, but to engage more members in the development of new members. The second component was the growth and development of a professional staff, which helped develop new policies, procedures, and supports to facilitate the structures needed for the increased complexity that came with virtual onboarding and engagement. The structures put in place to support new members by staff, including new communication tools and technologies to engage new members, were valuable to their connection to the community. Finally, the GEP developed clear strategies for growth and engagement that relied on these distributed and professional structures, an example of the “notion of strategy [tying] back to leadership and staffing” (Kezar & Gehrke, p. 343). Without the move toward a distributed model supported by a professional staff, the ability of GEP to enact these strategies could have been severely hampered. The support highlighted by participants above would not be possible without the distributed leadership and professional staff. Given this, our findings should be considered not only in light of the transition toward virtual engagement but also the ways in which they coincide with the transition to a distributed leadership model, the former of which occurred within two years of the latter.

One limitation of this study is the fact that we could only focus on the experiences of new members who were trained virtually. Prior to the COVID-19 pandemic, we intended to compare the experiences of virtually-trained members to those trained in person. While the lessons we learned about virtual engagement and onboarding are valuable, we were unable to discern how these strategies might have worked in comparison to more traditional in-person strategies. This leads us to areas for future research; as in-person gatherings have resumed, we will be able to examine differences of in-person- and virtual- training and engagement. Further, we will be able to better understand intermediate benefits for new members over time, and the unique contribution that virtual training and engagement made on their outcomes as they spend more time engaging with the community. One final area for future research is to examine whether the lessons learned from virtual engagement in the GEP can be utilized by new communities without the existing support and membership afforded GEP as an already established community. While we think these principles for virtual engagement are useful across all CoPs, new virtual communities should empirically interrogate their processes and outcomes in order to better understand this utility for new CoPs.

Conclusion

CoPs continue to form in response to calls for improvement of undergraduate STEM education. In order to improve their effectiveness, community leaders and scholars will need to understand ways to increase access while ensuring new members can benefit as other faculty did when the communities were smaller. The kinds of virtual engagement facilitated by the GEP provide helpful strategies for communities seeking to grow in more cost-effective and environmentally sustainable ways, while still ensuring a quality experience for new members. The early benefits that members gain through specific strategies of training and engagement suggest a strong foundation on which other communities can model their work.

Biographies

Sean Gehrke is the Executive Director of Educational Assessment and affiliate faculty in the College of Education at the University of Washington. His professional and scholarly work focuses on faculty engagement with STEM education and instructional reform efforts, outcomes-based assessment, evolving faculty roles in higher education, and college student development. Dr. Gehrke received his BA in Psychology from Lewis & Clark College, his M.Ed. in Counseling and Personnel Services from the University of Maryland College Park, and his Ph.D. in Urban Education Policy from the University of Southern California.

Anya L. Goodman, Ph.D. is a professor in the Department of Chemistry and Biochemistry at the California Polytechnic State University – San Luis Obispo. She earned her B.S. in Biology from Florida Atlantic University in 1996 and her Ph.D. from MIT in 2003. She joined GEP in 2006. She teaches courses in chemistry, biochemistry, biotechnology, and bioinformatics applications. Her research interests include course-based undergraduate research experiences in genomics and in biochemistry, work design of undergraduate research experiences, and faculty communities of practice.

LizAnne Ngo is a research scientist in the Office of Educational Assessment at the University of Washington. She shares with her team members the commitment to improving educational practice through the assessment and evaluation of educational programs within and outside the UW. LizAnne’s quantitative methodological interests center on longitudinal modeling and survey research. She earned her Ph.D. and Master’s in measurement and statistics and her B.S. in molecular biology from the University of Washington.

Catherine Reinke, Ph.D. is an associate professor in the Department of Biology at Linfield University. She earned her B.A. in Biology and English from Carleton College and her Ph.D. in Molecular Genetics and Cell Biology from the University of Chicago. Her research is focused on gene expression and, through collaborations with members of the Genomics Education Partnership. genome evolution in fruit flies and parasitoid wasps. She teaches courses in genetics, cell and molecular biology, and developmental biology.

Katie M. Sandlin is the Director of Curriculum for the Genomics Education Partnership (GEP; https://thegep.org/). Sandlin has been with the GEP since 2019. She received her BS and MS in Biology from The University of Alabama.

Laura K. Reed is a Professor of Biology at the University of Alabama. Dr. Reed received her BS in Biology (University of Oregon), her PhD in Ecology and Evolution (University of Arizona), was NIH-NRSA postdoctoral fellow (North Carolina State University), and joined the faculty at the University of Alabama in 2010. Her research focuses on the evolution of complex traits and effective science education methodology. She joined the GEP in 2011, taking on the role of Director in 2017. Under Dr. Reed, the GEP has more than doubled its faculty membership, enhanced its educational resources, and broadened its scientific scope.

Table A1.

GEP Engagement by Training Modality

	Never	Once	A few times (2–5)	Many times (> 5)
Attended new member training/welcome workshop as a trainer or TA
In-person	32%	56%	9%	3%
Virtual	40%	44%	2%	4%
Attended node meeting
In-person	38%	35%	21%	6%
Virtual	44%	23%	29%	4%
Led a presentation on GEP curriculum or research at a professional meeting
In-person	82%	10%	7%	1%
Virtual	96%	4%	0%	0%
Led or contributed to a micropublication
In-person	75%	6%	8%	11%
Virtual	87%	6%	4%	4%
Served as a lead author on a GEP education publication
In-person	93%	7%	0%	0%
Virtual	100%	0%	0%	0%
Attended national faculty meeting*
In-person	4%	13%	42%	42%
Virtual	79%	15%	6%	0%
Presented an/or led working group at national faculty meeting*
In-person	26%	25%	35%	14%
Virtual	96%	2%	2%	0%
Attended webinars (e.g., refreshers of national meeting, spring community updates)*
In-person	42%	25%	28%	6%
Virtual	80%	12%	8%	0%
Facilitated a node meeting*
In-person	69%	15%	15%	0%
Virtual	96%	4%	0%	0%
Attended GEP community gathering at another national professional meeting*
In-person	47%	6%	28%	19%
Virtual	90%	8%	0%	2%
Attended monthly committee meetings*
In-person	25%	4%	28%	43%
Virtual	81%	14%	4%	2%
Attended steering committee meetings*
In-person	72%	3%	7%	18%
Virtual	94%	4%	0%	2%
Presented GEP poster at a professional meeting as the lead presenter*
In-person	61%	28%	8%	3%
Virtual	96%	4%	0%	0%
Presented GEP poster at a professional meeting as a contributor*
In-person	41%	13%	27%	20%
Virtual	90%	6%	2%	2%
Contributed to a presentation on GEP curriculum or research at a professional meeting*
In-person	58%	15%	19%	7%
Virtual	96%	2%	2%	0%
Served on a writing committee for a GEP science publication*
In-person	86%	10%	4%	0%
Virtual	100%	0%	0%	0%
Contributed data as a co-author to a GEP science publication*
In-person	33%	17%	36%	14%
Virtual	56%	13%	23%	8%
Served on a writing committee for a GEP education publication*
In-person	70%	23%	6%	1%
Virtual	96%	4%	0%	0%
Co-authored with/mentored GEP student who were presenting at professional meetings*
In-person	51%	10%	25%	14%
Virtual	89%	8%	4%	0%
Served as a mentor to a GEP colleague*
In-person	49%	26%	25%	0%
Virtual	96%	2%	2%	0%

NOTES:

^*p < 0.05 based on Chi-square tests

Table A2:

Benefits of GEP Involvement by Training Modality

	In-Person-Trained (n=72) M (SD)	Virtually-Trained (n=52) M (SD)	Cohen’s d Effect Size
Allowed you to gain access to new curricular/pedagogical resources	4.56 (0.82)	4.65 (0.74)	−0.13
Connected you to a local (i.e., geographic proximity) network	3.29 (1.20)	3.44 (1.04)	−0.13
Led to changes in your teaching practice	3.90 (1.05)	3.75 (0.93)	0.15
Motivated to be innovative in your practice	4.00 (1.05)	3.79 (0.92)	0.21
Recharged/energized you in your work	3.92 (1.03)	3.65 (0.93)	0.27
Lent credibility to your approach toward teaching	3.90 (1.14)	3.60 (0.90)	0.29
Given you opportunities to pursue new grants/major projects	3.03 (1.31)	2.56 (1.26)	0.37*
Provided you with examples from which to model your work	4.08 (0.97)	3.69 (1.09)	0.38*
Connected you to people who share your professional interests	4.19 (0.95)	3.79 (0.92)	0.42*
Allowed you to have fun in a professional environment	4.10 (1.07)	3.65 (1.05)	0.42*
Lent credibility to your approach for professional work	3.80 (1.04)	3.36 (0.99)	0.44*
Contributed to your intellectual growth	4.39 (0.88)	4.00 (0.86)	0.45*
Connected you to people who share your personal interests	3.50 (1.14)	2.94 (1.20)	0.48*
Contributed to your understanding of the big picture of STEM reform	3.82 (1.05)	3.35 (0.89)	0.49*
Provided a safe place/support network outside of your institution	4.07 (0.98)	3.54 (1.15)	0.50*
Led to professional growth to improve your practice	4.15 (0.94)	3.69 (0.85)	0.51*
Fostered community for you that you don’t have at your institution	3.89 (1.13)	3.31 (1.06)	0.53*
Expanded your personal support network	3.56 (1.16)	2.90 (1.20)	0.55*
Provided opportunities to collaborate	3.97 (1.21)	3.25 (1.08)	0.62*
Provided support for career advancement	4.03 (1.10)	3.33 (1.13)	0.63*
Expanded your professional support network	4.26 (0.92)	3.62 (0.95)	0.70*
Led to publications for your students	3.17 (1.57)	1.87 (1.19)	0.92*
Led to publications for your CV	3.97 (1.42)	1.67 (1.15)	1.74*

NOTES: Measured on a 5-point scale (1 = Not at all, 3 = To some extent, 5 = To a great extent);

^*p < 0.05 based on independent t-tests