Abstract
Introduction:
The University of California, San Francisco Institute for Global Orthopaedics and Traumatology Surgical Management and Reconstructive Training (SMART) course has instructed orthopaedic surgeons from low-resource countries on soft-tissue reconstruction. Before the COVID-19 pandemic, the course was conducted in-person; however, it was transitioned to a virtual format during the pandemic. The aim of this study was to determine participant preferences regarding a virtual or in-person SMART course format.
Methods:
Survey data were collected via e-mail after each SMART course using RedCap or Qualtrics. Statistical analyses were conducted using Stata.
Results:
There were 247 survey respondents from 44 countries representing all world regions, with Africa (125, 51%) the most represented. Of those who attended both an in-person and virtual course, most (82%) preferred the in-person format. In addition, all measured course outcomes were significantly better for participants attending the in-person course. The most common reason for not attending an in-person course was the cost of travel (38, 51%).
Discussion:
This study demonstrated a preference toward in-person learning for the SMART course. In addition, those surgeons participating in the in-person course endorsed increased positive outcomes from the course. Increased emphasis should be placed on in-person surgical skills training for low-resource surgeons.
The Institute for Global Orthopaedics and Traumatology (IGOT), founded in 2006 by orthopaedic surgeons from the University of California, San Francisco (UCSF), builds international academic partnerships to find long-term solutions to the high burden of musculoskeletal disease in low- and-middle income countries (LMICs).1,2 Through the core pillars of education, research, leadership, and advocacy, IGOT has facilitated international knowledge exchange, conducted multiple research projects including randomized controlled trials, and organized international continuing education courses, with the most prominent being the Surgical Management and Reconstructive Training (SMART) course.1,3,4
Since its inception in 2014, the SMART course has focused on teaching orthopaedic surgeons from LMICs on how to manage soft-tissue reconstruction in low-resource settings, where plastic surgeons are often not available.4 Currently, the SMART course covers a variety of additional topics across the orthopaedic surgery subspecialties, with an emphasis on musculoskeletal trauma. The SMART course, and other similar short-term surgical education courses, have demonstrated efficacy for increasing surgical skills and increasing the adoption of new procedures in LMICs.5,6
Before the start of the coronavirus (COVID-19) pandemic, the SMART course was conducted in-person at UCSF as well as in Mexico, Nepal, and Tanzania. The in-person courses were usually held over 2 days and included a mixture of didactic lectures, case discussions, and cadaveric dissection sessions to teach surgical techniques and approaches.4 However, during the pandemic, in-person courses were paused, and in an effort to continue with education efforts, the course was moved online to a live webinar format. The live webinar format consisted of similar didactic sections and recorded laboratory sections demonstrating various techniques. After each in-person course and live webinar, quality improvement surveys were conducted to assess the effectiveness and satisfaction with the material presented. The aim of the current study was to compare participant experiences and determine participant preferences for either the virtual or in-person SMART course format.
Methods
Study Design
Data were collected as a part of ongoing quality improvement efforts for the SMART course. After each in-person course and webinar, IGOT staff emailed participants a survey to assess their satisfaction with the course and identify opportunities for improvement. With the evolution of the COVID-19 pandemic and the greater likelihood of resuming in-person courses, e-mail addresses from all previous in-person and virtual courses were compiled and sent a survey assessing preferences regarding the in-person versus virtual course format.
The survey included questions about demographics, past course attendance, preferences regarding each course format, and subjective accomplishment of various course outcomes. Surveys were designed by the study authors for the purposes of this study and were not validated. The surveys comprised a variety of question types, including multiple choice, five-point Likert scales, multiple answer selection, and free-text responses designed to assess a participant's feelings regarding their experiences with in-person and/or virtual SMART courses they had participated in. The survey was distributed in both English and Spanish to all participants to account for the regional specific languages.
Data Management and Statistical Analysis
All surveys were conducted and stored on Qualtrics and RedCap, which are secure web-based applications.7,8 Data from both language versions were aggregated and cleaned. Descriptive statistics were used to compare participant experiences and preferences. Student t-tests were used to compare the means of all course outcomes between the in-person course and the virtual course. The Bonferonni correction was used to account for multiple comparisons with significance set at P < 0.0056. All statistical analysis was conducted with StataSE 17.9
Ethical Approval
The protocol for this study was approved by the University of California, San Francisco Institutional Review Board. The study was determined to be exempt under the protocol number 21-35660.
Results
There were 1515 e-mail addresses compiled from prior SMART courses over the 12-year period, with 247 (16%) participants, primarily from courses conducted since 2017, responding to the course format survey. Respondents represented 44 countries from all world bank regions, with Africa (125, 51%) the most represented, followed by South Asia (34, 14%), Latin America (27, 11%), East Asia (26, 11%), the Middle East (12, 5%), North America (5, 2%), and Europe (4, 2%). Tanzania (32, 13%), Nepal (28, 11%), and Nigeria (22, 9%) were the most represented countries. Most survey respondents were orthopaedic surgeons from teaching tertiary/referral hospitals with more than 5 years of surgical experience, as can be seen in Table 1.
Table 1.
Demographics
| Survey Respondents | n = 247 |
| Survey language | |
| English | 236 |
| Spanish | 11 |
| Training level | |
| Surgeon | 200 |
| Trainee | 40 |
| No response | 7 |
| Specialty | |
| Orthopaedic surgeon | 180 |
| General surgeon | 11 |
| Other | 4 |
| No response | 53 |
| Years of practice | |
| 1–5 years | 76 |
| 6–10 years | 46 |
| >10 years | 73 |
| No response | 52 |
| Hospital type | |
| Tertiary/referral hospital | 163 |
| Secondary/regional hospital | 54 |
| Primary/district hospital | 21 |
| No response | 9 |
| Teaching hospital | |
| Yes | 191 |
| No | 47 |
| No response | 9 |
| In-person course attendeda | n = 155 |
| San Francisco | 83 |
| Tanzania | 56 |
| Nepal | 18 |
| Mexico | 6 |
| Other | 8 |
| Virtual course attendeda | n = 138 |
| Open fracture management | 81 |
| Pelvic fractures | 50 |
| Limb deformities | 29 |
| Tibial plateau fractures | 45 |
| Advanced flaps | 52 |
| Pelvic ring injuries | 50 |
Respondents may have attended multiple courses.
Course characteristics and preferences were recorded, as can be seen in Table 2. Of those who attended an in-person course (155, 63%), the majority attended either the San Francisco (83, 54%) and/or Tanzania (56, 36%) SMART courses. More than half of the participants (59%) attended the last 2 SMART course available in-person (2018 to 2019), with 35 (23%) being repeat in-person course participants. There were 75 participants who listed reasons for having not attended an in-person course. The most common reasons for not attending an in-person course were the cost of the course/travel (38, 51%) and the COVID-19 pandemic (32, 43%). Of those who selected that they were not aware of an in-person course (34, 45%), there were 24 (71%) who stated that they would attend the course, but 10 respondents (29%) stated that they would not. Overall, there were 215 (87%) of the total survey participants who either agreed or strongly agreed that in-person courses should resume.
Table 2.
SMART Course Characteristics and Preferences
| In-Person SMART Course | Virtual SMART Course | ||
| Why not attend in-person Course?a | n = 75 | Why not attend virtual Course?a | n = 87 |
| Was not aware, but would attend | 24 | Was not aware, but would attend | 39 |
| Was not aware, but would NOT attend | 10 | Was not aware, but would NOT attend | 5 |
| COVID-19 | 32 | COVID-19 | 0 |
| Difficulty accessing | 8 | Difficulty accessing | 12 |
| Time of course | 5 | Time of course | 26 |
| Length of course | 1 | Length of course | 1 |
| Structure of course | 0 | Structure of course | 1 |
| Content of course | 0 | Content of course | 1 |
| Ability to focus | 0 | Ability to focus | 6 |
| Level of participation | 1 | Level of participation | 5 |
| Ability to network | 7 | Ability to network | 10 |
| Travel cost | 38 | Travel cost | 0 |
| Other | 7 | Other | 11 |
| Why prefer in-person Course?a | n = 49 | Why prefer virtual Course?a | n = 6 |
| Structure | 29 | Structure | 2 |
| Content | 23 | Content | 3 |
| Ability to focus | 43 | Ability to focus | 3 |
| Level of participation | 39 | Level of participation | 4 |
| Ease of access | 14 | Ease of access | 6 |
| Ability to network | 23 | Ability to network | 2 |
| Travel | 9 | Travel | 3 |
| Cost | 1 | Cost | 4 |
| Other | 5 | Other | 0 |
Respondents could select multiple answers.
There were 138 participants (56%) who attended a virtual course, with the course on open fracture management being the highest attended (81, 59%). There were 87 participants who listed reasons for having not attended a virtual course. The most common reasons for not attending a virtual course were the timing of the course (26, 30%), an inability to access the webinar because of lack of Internet or a computer (12, 14%), an inability to focus and/or participate during the webinar (11, 13%), and an inability to network with other surgeons in the virtual format (10, 12%). Of those who selected that they were not aware of an in-person course (44, 51%), there were 39 (89%) who stated that they would attend the course and 5 participants (11%) stated that they would not.
Of the 60 participants who had attended both versions of the course, there were significantly more respondents who preferred the in-person SMART course format (49, 82%) over the virtual format (6, 10%). Of those who preferred the in-person format, the most common reasons listed were an increased ability to focus (43, 88%), participate (39, 80%), and the structure of the course (29, 59%). Notably, the content of the course and the ability to network with other surgeons were also listed by 23 surgeons each (47%). Of the six participants who preferred the virtual format, all of them listed the ease of accessing the content as a reason for their preference. The ability to participate and the cost-effectiveness of a virtual course were also listed by four participants. There were an equal number of participants who preferred the virtual course live versus recorded.
Outcomes from both the in-person and virtual course were recorded, as shown in Table 3. Of those who attended the respective courses, 58 participants (37%) and 25 participants (18%) listed outcomes from the in-person and virtual SMART courses, respectively. Most participants selected “strongly agree” or “agree” for all SMART course outcomes. More participants selected “strongly agree” in the in-person course than the virtual course. In addition, there were nine participants who selected “disagree” regarding outcomes to the virtual course while no participants marked “disagree” for the in-person course. Overall, there was a statistically significant difference between all outcomes favoring the in-person over the virtual SMART course.
Table 3.
Mean Outcomesa of In-Person Course (IPC, n = 58) Versus Virtual (Web, n = 25) Course.
| Outcome Measured | IPC | Web | P |
| Gained new knowledge | 4.60 | 4.08 | 0.0001b |
| Gained new skill | 4.66 | 3.60 | <0.0001b |
| Course was organized | 4.48 | 4.00 | 0.0014b |
| Course pace was appropriate | 4.35 | 3.68 | <0.0001b |
| Course was relevant | 4.55 | 4.20 | 0.0112b |
| Participant able to focus | 4.55 | 3.60 | <0.0001b |
| Participant able to participate | 4.40 | 3.44 | <0.0001b |
| Participant used gained skills | 4.60 | 3.58 | <0.0001b |
| Participant has changed practice | 4.48 | 3.76 | <0.0001b |
Outcomes were ranked from “strongly disagree” (1) to “strongly agree” (5).
Statistically significant.
Discussion
This study identified participant preferences regarding the IGOT SMART course in an in-person versus a virtual format. Of the 247 global survey respondents, most (87%) agreed that in-person courses should resume. Of those who attended both an in-person and virtual course, most (82%) preferred the in-person format. In addition, mean outcomes were markedly better for participants attending the in-person course.
The value of virtual versus in-person training has been a major debate throughout the COVID-19 pandemic. During the pandemic, most educational experiences have transitioned at least temporarily to the virtual format out of necessity, and much research has been conducted regarding virtual surgical education. Many studies have shown positive attitudes toward a virtual format.10-13 As was shown in this study, participants in these studies look positively on the ease of access and availability of the virtual format. In addition, the cost of the training course and/or travel is often a significant barrier in LMIC settings, as evidenced by this study, and can be potentially overcome through the implementation of a virtual course. Travel restrictions and COVID-19 protocols have made the value of virtual training even more apparent in recent years. However, most virtual courses focus on lecture-based materials, nontechnical content, and/or trainee-level education. Although virtual reality has been shown as an effective teaching method for technical skill development by multiple randomized-control trials, this type of virtual training is often not accessible to LMIC surgeons and is not tailored toward group participation.14-16
Despite the benefits of virtual learning, most of the study participants prefer the in-person training format. The stated drawbacks of virtual courses demonstrated in this study, including internet access, scheduling conflicts, and decreased ability to focus, are consistent with other studies.11,12 Notably, many of the studies focusing on surgical education were conducted in high-resource settings, so the drawbacks and barriers to the virtual format are likely exacerbated in LMIC settings. For example, as of 2020, only 49% of Tanzanians, the largest cohort of participants in this study, had access to the Internet.17
One major barrier presented in this study was the lack of awareness surrounding the availability of the SMART course. Although emails were consistently distributed to the participants about the courses, relatively few were aware of them. This may further demonstrate the limited reliability of electronic communication in LMIC settings. In addition, it has been shown that the most-worked trainees often have more negative feelings toward virtual training courses, which possibly translates to the overwhelming workloads of surgeons in LMIC settings.12
This study demonstrated superior outcomes when participating in the in-person course relative to the virtual courses. This is likely due to the cadaver-based surgical skills training sessions available in the in-person setting that were not available in the virtual setting. There have been relatively few studies analyzing the effect of in-person versus virtual surgical skills training. Some studies have looked at performance in medical education and/or technical skills, but with varying results.18,19
There are several limitations to this study. This study was a survey-based investigation with relatively low statistical power given the sample size. In addition, multiple participants only attended one of the course formats, which could bias their preferences. Additional research is needed to further investigate in-person versus virtual surgical training and outcomes.
Conclusion
This study demonstrates that there is a preference toward in-person learning for participants of the IGOT SMART course. In addition, those surgeons participating in the in-person course endorse increased positive outcomes from the course. Increased emphasis should be placed on in-person surgical skills training for LMIC surgeons. In addition, increased efforts should be made to advertise and fund these opportunities.
Footnotes
Author contributions: Flores: Conceptualization, Methodology, Analysis, Writing. Brown: Conceptualization, Methodology, Analysis, Writing. MacKechnie: Conceptualization, Methodology, Writing. O’Marr: Analysis, Writing. Rodarte: Analysis, Writing. Shearer: Conceptualization, Methodology, Writing.
None of the following authors or any immediate family member has received anything of value from or has stock or stock options held in a commercial company or institution related directly or indirectly to the subject of this article: Mr. Flores, Mr. Brown, Ms. MacKechnie, Mr. O’Marr, Ms. Rodarte, and Dr. Shearer.
Contributor Information
Michael J. Flores, Email: michael.flores@yale.edu.
Kelsey E. Brown, Email: kelsey_brown@brown.edu.
Madeline C. MacKechnie, Email: madeline.mackechnie@ucsf.edu.
Patricia Rodarte, Email: patricia_rodarte@brown.edu.
David W. Shearer, Email: david.shearer@ucsf.edu.
References
- 1.Conway DJ, Coughlin R, Caldwell A, Shearer D: The Institute for global orthopedics and Traumatology: A model for academic collaboration in orthopedic surgery. Front Public Health 2017;5:146. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Roser M, Ritchie H: Burden of disease. Our World Data. Accessed March 3, 2022 https://ourworldindata.org/burden-of-disease. [Google Scholar]
- 3.Ibrahim J, Liu M, Yusi K, et al. : Conducting a randomized controlled trial in Tanzania: Institute for global orthopaedics and Traumatology and the muhimbili orthopaedic Institute. J Orthop Trauma 2018;32suppl 7:S47-S51. [DOI] [PubMed] [Google Scholar]
- 4.Wu HH, Patel KR, Caldwell AM, Coughlin RR, Hansen SL, Carey JN: Surgical management and reconstruction training (SMART) course for international orthopedic surgeons. Ann Glob Health 2016;82:652-658. [DOI] [PubMed] [Google Scholar]
- 5.Challa S, Conway D, Wu HH, et al. : Can a 2-day course teach orthopaedic surgeons rotational flap procedures? An evaluation of data from the Nepal SMART course over 2 years. J Orthop Trauma 2018;32suppl 7:S38-S42. [DOI] [PubMed] [Google Scholar]
- 6.Wang YT, Meheš MM, Naseem HR, et al. : Assessing the impact of short-term surgical education on practice: A retrospective study of the introduction of mesh for inguinal hernia repair in sub-saharan Africa. Hernia 2014;18:549-556. [DOI] [PubMed] [Google Scholar]
- 7.Harris PA, Taylor R, Minor BL, et al. : The REDCap consortium: Building an international community of software platform partners. J Biomed Inform 2019;95:103208. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG: Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377-381. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.StataCorp: Stata Statistical Software: Release 17. College Station, TX, StataCorp LLC, 2021. [Google Scholar]
- 10.Ottesen TD, Montoya RL, Ogunleye TD, et al. : Implementation and impact evaluation of a virtual orthopaedic continuing medical education conference in a low-resource country. J Surg Educ 2021;78:1629-1636. [DOI] [PubMed] [Google Scholar]
- 11.Bartoletta JJ, Hinchcliff K, Rhee P: Learner preferences and perceptions of virtual hand surgery education during the COVID-19 pandemic. J Hand Surg 2023;48:405.e1-405.e8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Nagaraj MB, Weis HB, Weis JJ, et al. : The impact of COVID-19 on surgical education. J Surg Res 2021;267:366-373. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Sambhwani S, Alshahwani A, Andritsos L, Sheikh N: Maintaining postgraduate healthcare education during COVID-19: Does a virtual format allow effective engagement and personal development?. Cureus 2021;13:e19066. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Zaid MB, Dilallo M, Shau D, Ward DT, Barry JJ: Virtual reality as a learning tool for trainees in unicompartmental knee arthroplasty: A randomized controlled trial. J Am Acad Orthop Surg 2022;30:84-90. [DOI] [PubMed] [Google Scholar]
- 15.Walbron P, Common H, Thomazeau H, et al. : Virtual reality simulator improves the acquisition of basic arthroscopy skills in first-year orthopedic surgery residents. Orthopaedics Traumatol Surg Res OTSR 2020;106:717-724. [DOI] [PubMed] [Google Scholar]
- 16.Lohre R, Bois AJ, Pollock JW, et al. : Effectiveness of immersive virtual reality on orthopedic surgical skills and knowledge acquisition among senior surgical residents: A randomized clinical trial. JAMA Netw Open 2020;3:e2031217. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Tanzania: Number of internet users 2015-2020. Statista. Available at:. https://www.statista.com/statistics/1225895/number-of-internet-users-in-tanzania/Accessed February 22, 2022. [Google Scholar]
- 18.Soni NJ, Boyd JS, Mints G, et al. : Comparison of in-person versus tele-ultrasound point-of-care ultrasound training during the COVID-19 pandemic. Ultrasound J 2021;13:39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Kim S, Willett LR, Pan WJ, Afran J, Walker JA, Shea JA: Impact of required versus self-directed use of virtual patient cases on clerkship performance: A mixed-methods study. Acad Med J Assoc Am Med Colleges 2018;93:742-749. [DOI] [PubMed] [Google Scholar]