Abstract
Background
The COVID-19 pandemic has increased utilization of educational technology for surgical education. Our aim was to determine attitudes and behaviors of surgical education champions towards virtual educational platforms and learner engagement.
Methods
An electronic survey was distributed to all Association of Surgical Education members addressing i) methods of engagement in virtual learning ii) ways to improve engagement and iii) what influences engagement. Stratified analysis was used to evaluate differences in responses by age, gender, level of training and specialty.
Results
154 ASE members completed the survey (13% response rate). 88% respondents accessed virtual learning events at home. Most (87%) had joined a virtual learning event and then participated in another activity. 1 in 5 who did this did so “always” or “often”. Female respondents were more likely than males to join audio and then participate in another activity (62.3% v 37.7%, p = 0.04).
Conclusions
Virtual platforms do not automatically translate into increased learner engagement. Careful design of educational strategies is essential to increase and maintain learner engagement when utilizing virtual surgical education.
1. Introduction
The COVID-19 pandemic has led to swift and often unchartered transitions to provision of virtual surgical education.1, 2, 3 A rapid adaption by learners and educators alike has been required in order to ensure that this pedagogical shift is associated with effective learning. There are many obvious benefits to utilization of virtual educational platforms during this time, including adherence to social distancing requirements, educating learners in quarantine and isolation and ensuring that educational opportunities available to a greater number of learners.4 An excellent example is the American College of Surgeons Clinical Congress meeting in 2020. This meeting logged a record 30,000 registrants from more than 150 countries.5
Whilst the increased accessibility and, by virtue, increased attendance of virtual learning events is attractive, what is unclear is if this translates to either increased learner engagement or learning. Efforts have been made to study learner engagement during in-person learning events, but this has not been performed for the virtual platform.6 , 7 An additional challenge in the virtual platform is assessing the issue of “fake” engagement, and learner multi-tasking. The effects of multi-tasking may be especially pertinent for women, given the increased gender disparity observed during the COVID-19 pandemic with regard to increased childcare responsibilities and reduced proportional scholarly activity reported for women.8, 9, 10 Conducting medical education research in this area is important in order to avoid the tempting but unfounded assumption that increased number of learners leads to increased engagement or increased learning.
Attitudes and beliefs are important and powerful predictors of behavior.11 Studying these can help us determine barriers to engagement with virtual learning events, challenges in their delivery and construct and deliver strategies to mitigate these. The aim of this work was to determine the attitudes, beliefs and behaviors of surgical education champions with regard to virtual surgical education and learner engagement.
2. Methods
An electronic survey was distributed to all members of the Association of Surgical Education. These individuals were selected for the survey as they can be defined as surgical education champions and arguably best placed to offer insight into the subject matter. The survey questions were generated and revised during meetings of an ASE Educational Technology Committee workgroup and focused on the respondent perceptions and behaviors as learners in surgical educational events. Pilot testing of the survey was performed to optimize usability and response rates. Questions were further revised based on the results of pilot testing.
The questions pertained to respondent demographics, their engagement behaviors as learners during virtual surgical educational events, their preferences for engagement with virtual learning events and their perceptions of factors affecting virtual learning engagement. In addition, data were collected regarding the pattern of surgical event translation to the virtual platform at the respondent's institution. ASE members of all grades/professions were asked to complete the surveys for virtual educational events in which they were present in a learner role.
The distribution of the survey responses was reported as frequency and proportion. Stratified analysis was used to evaluate the difference in frequencies and proportions of the responses of questions 12, 13, 14 and 15 by strata of age, gender, level of training and specialty. The differences in the response proportions were determined by the Chi-square or Fisher's exact test as appropriate. All the analyses were performed on Stata version 17.0 (StataCorp LLC, College Station, TX, USA)
3. Results
The survey was sent to all physician (674), non-physician (183) and trainee members (346) of the ASE. A total of 154 (13%) ASE members completed the survey, of whom 46% were female, 62% were white and the majority (49%) were surgery attendings. Demographic details are shown in Table 1 . Most of the respondents (88%) accessed virtual learning events at home. Zoom was the most commonly used platform (92%) followed by Microsoft Teams (44%). A diverse range of learning and patient care events were translated into a virtual platform at respondent's institutions. Table 2 .
Table 1.
Responder demographics.
| Demographics | Number of respondents (% of total number) |
|---|---|
| Current practice | |
| Surgery attending | 75 (48.7) |
| Surgery fellow | 4 (2.6) |
| Surgery resident in clinical years | 10 (6.5) |
| Surgery resident in research | 7 (4.5) |
| Surgical educator, non clinical | 16 (10.4) |
| Educational administrator | 6 (3.9) |
| Medical student | 2 (1.3) |
| Allied healthcare professionals educator | 2 (1.3) |
| Specialty | |
| Acute Care Surgery | 28 (18.2) |
| Minimally Invasive Surgery | 25 (16.2) |
| Colorectal Surgery | 12 (7.8) |
| Cardiothoracic Surgery | 3 (1.9) |
| Pediatric Surgery | 8 (5.2) |
| Surgical Intensive Care | 17 (11.0) |
| Surgical Oncology | 17 (11.0) |
| Trauma Surgery | 21 (13.6) |
| Endocrine Surgery | 7 (4.5) |
| Other | 51 (33.1) |
| Gender | |
| Female | 70 (45.5) |
| Male | 49 (31.8) |
| Not disclosed | 35 (22.7) |
| Race | |
| White | 95 (61.7) |
| Black | 1 (0.6) |
| Asian | 17 (11.0) |
| Other | 1 (0.6) |
| Not disclosed | 40 (26.0) |
| Age | |
| 18–30 | 26 (16.9) |
| 31–40 | 39 (25.3) |
| 41–50 | 32 (20.8) |
| 51–60 | 18 (11.7) |
| 61–70 | 5 (3.2) |
| Not disclosed | 32 (20.8) |
Table 2.
Translation of in-person events onto the virtual platform (respondents select all that apply).
| Learning event | Number of respondents (%) |
|---|---|
| Journal Club | 89 (57.8) |
| Morbidity and Mortality meetings | 131 (85.1) |
| Grand rounds | 116 (75.3) |
| Didactic sessions (e.g. resident SCORE sessions) | 120 (77.9) |
| Regular departmental or societal meetings | 128 (83.1) |
| Multidisciplinary team meetings | 102 (66.2) |
| Other | 12 (7.8) |
Half of respondents preferred to engage using computer video as necessary (such as when talking). Table 3 . The format of the session (58%) and how much the participant planned to talk (51%) were the most common influences on modality of engagement (video versus audio). When respondents participated via audio only on their computer or cellphone, 63% felt less engaged with the session.
Table 3.
Preference for engagement by learners in virtual educational events (defined as all events in which the participant is attending with the primary goal of learning).
| Preference for engagement in virtual learning events | Number of respondents (%) |
|---|---|
| Computer with video on all the time | 45 (29.2) |
| Computer with video on as necessary (such as when you are talking) | 78 (50.6) |
| Computer with audio on only, no video | 9 (5.8) |
| Mobile device with video on all the time | 3 (1.9) |
| Mobile device with video on as necessary (such as when you are talking) | 4 (2.6) |
| Mobile device with audio on only, no video | 1 (0.6) |
| Other | 14 (9.1) |
More than half (60%) of respondents chose video engagement in order to form a connection with other participants Table 4 . If audio was preferred, 63% selected this as they had other activities to perform during the teaching time. The majority (87%) had previously joined a virtual learning event and then participated in another activity. One in 5 who did this did so “always” or “often” (Fig. 1 ). Female respondents were more likely than males to join audio and then participate in another activity (62.3% v 37.7%, p = 0.04).
Table 4.
Determinants of engagement modality for participants in virtual learning events (respondents select all that apply).
| Number of respondents (%) | |
|---|---|
| Why do you select video engagement? | |
| The instructor is on video | 48 (31.2) |
| The majority of the class is on video | 69 (44.8) |
| I feel like it is expected of me | 63 (40.9) |
| To show I am paying attention | 67 (43.5) |
| To form a connection with the instructor | 69 (44.8) |
| To form a connection with other participants | 92 (59.7) |
| Other | 9 (5.8) |
| Why do you select audio only engagement? | |
| I have other things to do during the session | 97 (63.0) |
| I have children/pets/other relatives at home and cannot guarantee privacy | 60 (39.0) |
| I am self-conscious | 18 (11.7) |
| I am concerned about my appearance | 42 (27.3) |
| I don't want my picture to be photographed and Tweeted | 13 (8.4) |
| Other | 28 (18.2) |
Fig. 1.
The frequency that participants join virtual learning events on audio and do not pay attention (% of total).
In terms of the influence of the “chat” function on learner engagement, 58% felt that this facilitated discussion and 38% felt it increased learner engagement. Nineteen percent of respondents felt that the chat function provided a distraction to the instructor and 16% felt it provided a distraction to the learner. Interestingly 46% respondents use the chat function to get their voice heard when there was a large number of learners in the group.
Most (86%) respondents felt that learner engagement in virtual educational sessions needs to be improved. One third felt that effective strategies to achieve this were mandatory video use and local “engagement champions”. Less popular strategies were virtual engagement tools such as eye tracking devices (5%) and observers to monitor learner behavior and study engagement (14%).
4. Discussion
The COVID-19 pandemic has led to increased utilization of virtual surgical education, however we have demonstrated that, in the experience of surgical education champions, participation in virtual learning events does not necessarily translate to learner engagement. Importantly, there is a female preponderance for multi-tasking during these virtual events.
Learner engagement in face-to-face teaching events has been studied using the STROBE tool using trained observers and an engagement rubric.6 , 7 Less is known about the efficacy of learner engagement in virtual learning events and no measure currently exists to assess this. There is a temptation from the position of surgical educators to be enthused about the increased number of learners in the session by nature of their accessibility. To date, however it is unclear whether the increased number of learners is associated with a concomitant increase in engagement. Indeed, studies have identified that in face-to-face teaching events students have reported faking this engagement for 23% of the class duration.12
Learner engagement is a complicated phenomenon encompassing both psychological and physical factors.13 Although the literature regarding learner engagement within medical education is limited, learner engagement in online continuing medical education has been associated improved patient outcomes.14 Engaging learners and being able to measure whether virtual learners are engaged with the learning event is crucial to determine whether learning is effective. The virtual platform brings with it a new dimension to this and the faked engagement observed in the in-person environment may be easier to achieve, e.g., audio only and mute options. Multi-tasking may also be easier in the virtual environment such as having alternative reading materials open on computer whilst in the virtual learning session. It is challenging for the educator to determine whether the learners are truly engaged in this context.
The many social implications of the COVID-19 pandemic include concern for the disproportionate effect on women in terms of caregiving for children and homeschooling responsibilities, and a reported significant reduction in hours worked and consequent reduction in academic productivity.15 , 16 In addition, women in the US spend 2 h more each day cleaning, cooking, taking care of children and doing other unpaid work compared to men per the Institute for Women's Policy Research and Oxfam report.17 This is reflected by our finding that 88% respondents attend learning events online at home and that female respondents were more likely than males to join audio and then participate in another activity (62.3% v 37.7%, p = 0.04). This is concerning as when multi-tasking, learners are less engaged with the primary educational focus and therefore teaching is less effective.18 It may be that female learners also multi-task more often in the in-person environment also, however the pandemic may have further widened the gap between female and male learners, already affected by childcare and homeschooling.
The issues we have identified with multitasking and perhaps “fake” engagement with virtual learning events was recognized by the majority of respondents who felt that engagement needed to be improved with virtual learning events. Respondents believed that engagement champions would be an effective potential solution. It is important to raise the awareness of issues surrounding suboptimal engagement with virtual learning but in addition, as surgical educators it is crucial to build active learning strategies into these virtual events. Common to both in-person and virtual learning events, the attention span of learners is short, around 10–15 min, and followed by frequent “lapses”19, hence active learning strategies i.e. those that are in concordance with constructivism and which are learner centered and in which the students are actively or experientially involved.20 Zoom offers polling tools that can be used to increase learner engagement and using the “flipped classroom” approach may also improve learner motivation. The latter is a blended learning strategy employed to stimulate learner engagement and active learning. The learner benefits from preparing with materials they access before the educational event.21 As learner engagement has been strongly associated with teaching effectiveness, it is key for educators to maximize engagement with the virtual platform.22
This work has limitations. Our response rate of 13% is lower than desirable, and increases the risk of unit non-response bias i.e. non-completion of the entire survey.23 Through survey design and administration we minimized the effects of coverage, sampling and measurement error. Specifically, by targeting ASE members we focused on those respondents most likely to reflect upon cause and effect of behaviors relating to the virtual learning environment, and to be invested in the results of this work. Careful design of survey questions through an iterative committee-based process grounded in Dillman's Tailored Design Method minimized measurement error.24 With regard to the high non-response rate there may be many reasons for this including the effect of the pandemic on survey fatigue, hours available for participation and burnout. Further, it has been demonstrated that lesser response rates are achieved with email based versus mail based surveys and this likely reflects the email volume medical professionals are exposed to25. Arguably this has been exacerbated during the COVID-19 pandemic, and given the proliferation in survey-based work during this global crisis, it is unlikely higher response rates are currently feasible. A further limitation is that the majority, 62% of the respondents, were white and therefore the responses may not be truly representative of the surgical education champion population as a whole. Also, all respondents are ASE members and the responses herein may not represent the views of surgeons or those interested in surgical education throughout the US. Despite this, it can be argued that ASE member surgical education champions may manifest the most engagement with surgical educational activities. It is likely then that rather than underestimating the problem, the true rates of disengagement with virtual learning events and multitasking are higher than those represented within this work. Finally, it is challenging for this work to provide a comparison of learner engagement between the virtual and in-person environment to determine whether engagement virtually is more, less or the same as with pre-pandemic educational events. Learner engagement during in-person events may be assessed with instruments such as the STROBE tool, however these are rarely utilized regularly outside of educational research work.26 , 27 Certainly, the majority of in-person surgical educational events will have no prospective assessment of learner engagement. There was no previously developed tool for the virtual environment to allow comparison until the recent publication of the VIEM (Virtual In-Class Engagement Measure). This may now allow contemporaneous comparison of engagement in virtual and in person learning events to occur.28
5. Conclusion
With the rapid transition to predominantly virtual surgical education during the COVID-19 pandemic it is essential to determine the effect of this on learner engagement, by extension, and learning. We have demonstrated that virtual platforms do not automatically translate into increased learner engagement. Careful design of educational strategies is essential to increase and maintain learner engagement when utilizing virtual surgical education.
Funding
Salary support for KEC was provided by the Washington University Surgical Oncology Training Grant (T32CA009621).
Disclosures/conflict of interest
None.
References
- 1.Calhoun K.E., Yale L.A., Whipple M.E., Allen S.M., Wood D.E., Tatum R.P. The impact of COVID-19 on medical student surgical education: implementing extreme pandemic response measures in a widely distributed surgical clerkship experience. Am J Surg. 2020 Jul;220(1):44–47. doi: 10.1016/j.amjsurg.2020.04.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Okland T.S., Pepper J.P., Valdez T.A. How do we teach surgical residents in the COVID-19 era? J Surg Educ. 2020 Sep-Oct;77(5):1005–1007. doi: 10.1016/j.jsurg.2020.05.030. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Coe T.M., Jogerst K.M., Sell N.M., et al. Practical techniques to adapt surgical resident education to the COVID-19 era. Ann Surg. 2020 Aug;272(2):e139–e141. doi: 10.1097/SLA.0000000000003993. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Dickinson K.J., Gronseth S.L. Application of universal design for learning (UDL) principles to surgical education during the COVID-19 pandemic. J Surg Educ. 2020 Sep-Oct;77(5):1008–1012. doi: 10.1016/j.jsurg.2020.06.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.American College of surgeons clinical congress news. 2020. https://www.acsccnews.org/a-successful-acs-clinical-congress-2020-comes-to-a-close/
- 6.Kelly P.A., Haidet P., Schneider V., Searle N., Seidel C.L., Richards B.F. A comparison of in-class learner engagement across lecture, problem-based learning, and team learning using the STROBE classroom observation tool. Teach Learn Med. 2005;17(2):112–118. doi: 10.1207/s15328015tlm1702_4. Spring. [DOI] [PubMed] [Google Scholar]
- 7.Alimoglu M.K., Sarac D.B., Alparslan D., Karakas A.A., Altintas L. An observation tool for instructor and student behaviors to measure in-class learner engagement: a validation study. Med Educ Online. 2014 Oct 10;19:24037. doi: 10.3402/meo.v19.24037. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Muric G., Lerman K., Ferrara E. Gender disparity in the authorship of biomedical research publications during the COVID-19 pandemic: retrospective observational study. J Med Internet Res. 2021 Apr 12;23(4) doi: 10.2196/25379. PMID: 33735097; PMCID: PMC8043146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Ipe T.S., Goel R., Howes L., Bakhtary S. The impact of COVID-19 on academic productivity by female physicians and researchers in transfusion medicine. Transfusion. 2021 Jun;61(6):1690–1693. doi: 10.1111/trf.16306. Epub 2021 Feb 10. PMID: 33527397; PMCID: PMC8013272. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Robinson L.J., Engelson B.J., Hayes S.N. Who is caring for health care workers' families amid COVID-19? Acad Med. 2021 Sep 1;96(9):1254–1258. doi: 10.1097/ACM.0000000000004022. PMID: 33635839; PMCID: PMC8378425. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Glasman L.R., Albarracín D. Forming attitudes that predict future behavior: a meta-analysis of the attitude-behavior relation. Psychol Bull. 2006 Sep;132(5):778–822. doi: 10.1037/0033-2909.132.5.778. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.B, Short J.L., Wolcott M.D., Singleton S., White P.J. Development of a self-report instrument for measuring in-class student engagement reveals that pretending to engage is a significant unrecognized problem. PLoS One. 2018 Oct 17;13(10) doi: 10.1371/journal.pone.0205828. Astin A. Student involvement: a developmental theory for higher education. N Engl J Med. 2012;366:1657. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Crenshaw K., Curry W., Salanitro A.H., et al. Is physician engagement with web-based CME associated with patient's baseline hemoglobin A1c levels? The Rural Diabetes Online Care Study. Acad Med. 2010;85:1511. doi: 10.1097/ACM.0b013e3181eac036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Krukowski RA, Jagsi R, Cardel MI. Academic productivity differences by gender ad child age in STEMM faculty during the COVID-19 pandemic. J Womens Health (in press). [DOI] [PMC free article] [PubMed]
- 15.Manzo LKC, Minello A. Mothers, childcare duties , and remote working under COVID-19 lockdown in Italy: cultivating communities of care. Dialogues Hum Geogr (online ahead of print).
- 16.Uncapher M.R., Lin L., Rosen L.D., et al. Media multitasking and cognitive, psychological, neural, and learning differences. Pediatrics. 2017 Nov;140(Suppl 2):S62–S66. doi: 10.1542/peds.2016-1758D. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.https://iwpr.org/wp-content/uploads/2020/01/IWPR-Providing-Unpaid-Household-and-Care-Work-in-the-United-States-Uncovering-Inequality.pdf
- 18.Edwards K.S., Shin M. Media multitasking and implicit learning. Atten Percept Psychophys. 2017 Jul;79(5):1535–1549. doi: 10.3758/s13414-017-1319-4. [DOI] [PubMed] [Google Scholar]
- 19.Bunce D.M., Flens E.A., Neiles K.Y. How long can students pay attention in class? A study of student attention decline using clickers. J Chem Educ. 2010;87(12):1438–1443. [Google Scholar]
- 20.Torralba K.D., Doo L. Active learning strategies to improve progression from knowledge to action. Rheum Dis Clin N Am. 2020 Feb;46(1):1–19. doi: 10.1016/j.rdc.2019.09.001. PMID: 31757278. [DOI] [PubMed] [Google Scholar]
- 21.Hew K.F., Lo C.K. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ. 2018;18:38. doi: 10.1186/s12909-018-1144-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Stephensen C.R., Bonnes S.L., Sawatsky A.P., et al. The relationship between learner engagement and teaching effectiveness: a novel assessment of student engagement in continuing medical education. BMC Med Educ. 2020;20:403. doi: 10.1186/s12909-020-02331-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Phillips A.W., Reddy S., Durning S.J. Improving response rates and evaluating nonresponse bias in surveys: AMEE Guide No. 102. Med Teach. 2016;38(3):217–228. doi: 10.3109/0142159X.2015.1105945. Epub 2015 Dec 9. PMID: 26648511. [DOI] [PubMed] [Google Scholar]
- 24.Dillman D.A., Smyth J.D., Christiam L.M. Wiley; Hoboken, NJ: 2014. Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method. [Google Scholar]
- 25.Edwards P.J., Roberts I., Clarke M.J., et al. Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009 Jul 8;(3) doi: 10.1002/14651858.MR000008.pub4.PMID:19588449. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.O'Malley K.J., Moran B.J., Haidet P., et al. Validation of an observation instrument for measuring student engagement in health professions settings. Eval Health Prof. 2003 Mar;26(1):86–103. doi: 10.1177/0163278702250093. [DOI] [PubMed] [Google Scholar]
- 27.Fuller K.A., Karunaratne N.S., Naidu S., et al. Development of a self-report instrument for measuring in-class student engagement reveals that pretending to engage is a significant unrecognized problem. PLoS One. 2018 Oct 17;13(10) doi: 10.1371/journal.pone.0205828. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Dickinson K.J., Caldwell K.E., Graviss E.A., et al. ASE Educational Technology Committee Assessing learner engagement with virtual educational events: development of the Virtual In-Class Engagement Measure (VIEM) Am J Surg. 2021 Sep 28 doi: 10.1016/j.amjsurg.2021.09.021. S0002-9610(21)00547-X. Epub ahead of print. PMID: 34602277. [DOI] [PubMed] [Google Scholar]