See one, teach yourself one, do one: Barriers and opportunities in self-administered training and assessment for global surgical education

Abstract

Objective

We aimed to determine the most important perceived barriers to the implementation of self-administered training and assessment in surgical education according to subject matter experts. With these findings, design thinking was used to explore possible interventions and develop a theory of change for overcoming identified barriers. Specifically, implementation was focused on expanding the surgical skills of associate clinicians (ACs) in low-to-middle-income countries (LMICs).

Methods

A qualitative study with 10 field experts representing surgeons, educators, and engineers from the US, South America, and East and West Africa was conducted. Interviewees were selected through purposeful snowball sampling until thematic saturation. Semi-structured interviews were conducted over video conference or in-person. Open-ended responses were synthesized, coded, and used to identify key barriers for scaling simulation-based learning and self-administered training and assessment in low-resource settings.

Results

We identified four major barriers to widespread implementation of self-administered training and assessment: demonstration of the safety and quality of surgical care provided after self-administered training; validation of the principle of self-administered training and assessment; translation of simulation skills to surgical knowledge; and integration into existing task shifting and task sharing legal landscapes.

Discussion

Increasing surgical capacity in LMICs is an urgent need that could be expanded with carefully developed self-administered training and assessment for ACs. The implementation process will be variable depending on local culture and regulations but is dependent on an international community of local champions to first produce a common body of evidence supporting the technology's utility and then to generate local excitement for its integration into existing systems.

Highlights

• •Self-administered educational tools show potential in scaling surgical capacity.
• •This technology's reach is limited by minimal evidence proving safety and efficacy.
• •Implementing self-administered tools will require an incremental cultural shift.
• •This shift needs local leaders who understand their specific sociolegal context.

Introduction

The global shortage in the surgical workforce hinders access to safe and timely surgical care for millions of people worldwide [1]. Surgical workforce density is one of the most important correlates to overall procedure volume and peripartum maternal survival, with rural areas and low-to-middle-income countries (LMICs) especially affected by the shortage of surgical, obstetric, and anesthetic providers [[1], [2], [3], [4], [5]].

Expansion of the surgical workforce is limited by long training durations and, in low-resource areas, the small number of local faculty. The well-known adage “see one, do one, teach one” is descriptive of the current process of surgical training: trainees first observe a particular procedure; perform it under supervision; then guide a fellow learner through. However, this model perpetuates existing geographic inequity because access to training depends on existing surgeon densities.

Associate Clinicians (ACs), as defined by the World Health Organization, are “professional clinicians with basic competencies to diagnose and manage common medical, maternal, child health, and surgical conditions… [who] are generally trained for 3 to 4 years [of] post-secondary education in established higher education institutions.” [6] In low-resource areas, task shifting or task sharing of surgical procedures to these ACs can be very common. In some countries, as many as 90 % of obstetric surgeries, 38.5 % of general surgery procedures, and 43 % of non-obstetric laparotomies are performed by both non-surgeons and non-physicians [7]. The Surgical Education Learners Forum (SELF) aims to provide ACs, medical physicians, and surgeons in low-resource areas the opportunity to increase their knowledge and skills during and even after they have completed their formalized training through self-administered training and assessment, a validated educational system with the potential to safely expand surgical access in their region.

This initiative was developed in response to the Intuitive Foundation's 2020 Global Surgical Training Challenge (GSTC), with the goal of democratizing and decentralizing surgical education through widely available, open-access educational modules combining medical knowledge, psychomotor skill simulation, and rigorous, validated self-administered assessment. Such assessment is paired with explicit guidance for determining areas of improvement and eventual mastery.

SELF publishes online modules on the open-source platform, Appropedia [8]. Each module has three primary elements: (1) Knowledge acquisition (2) Simulation training, and (3) Self-administered assessment. Assessment is carried out through pre-and post-learning quizzes, self-verification checklists, and peer-to-peer feedback guided by validated checklists. One module has even included artificial intelligence-driven assessment of video-recorded simulated procedures. Each module contains instructions for constructing a physical simulator using low-cost and locally available materials, allowing implementation at scale in low-resource settings.

The aim of this study is to identify barriers to scaling the SELF initiative in order to develop strategies for expanding its reach in target communities.

Methods

A convenience sample of semi-structured interviews was generated with a diverse range of global surgical education experts, who were selected by purposeful snowball sampling through the authors' institutional contacts. The questions used to guide conversations with interviewed subjects are included in Table 1. Following initial interviews, each interviewee was asked to recommend others. Recruitment continued until thematic saturation was achieved. Overall, the interviewees represented a diversity of geographic experience and domain expertise. Of the ten experts, nine were primarily surgeons, including four who represented medical institutions based in East and West Africa. Seven of the surgeons have active projects in low-resource settings, spanning Africa, South America, and Asia. Two of the interviewees represented the GSTC, with engineering or design backgrounds. Six of the interviewees were surgeon educators who either led educational programs at their institutions or served on the boards of medical licensing organizations in their regions.

Table 1.

Sample Interview Guide. A sample of the questions used to guide interviews with subject matter experts.

Summary of interview guide
Interview guide section	Sample questions
General Questions about SELF	What are your first impressions of SELF? What is exciting? What is concerning? Who do you think could benefit from training like this in a low-resource context? When you are working abroad, who do you wish you had more of? Who do you wish was better skilled? Would this be a good way to train that person? What can be achieved with physical simulation? What cannot?
Scaling SELF	What do you see as the key challenges for SELF to be rolled out to medical officers (regulatory, clinical evidence, business model, distribution, supply chain, etc.)? Do you see any challenges in scaling (e.g. regional difference, content scale)? What are the potential ways to address these challenges?
Domain specific questions for Educators	What are your main concerns about self-assessment of surgical skills? How is readiness to perform new skills assessed now? What works well? What doesn't? What do you think must be assessed before you'd trust someone or to perform a new skill? What parts of this can be done independently? Would any of the following means of validating a medical officer's new skill before practicing on patients make self-training feasible? a) having a certified doctor watch the first one? b) having a certified AI to approve? c) course endorsement with reputable organizations, e.g. Stanford Humanitarian Surgery Course?
Domain specific questions for Technologists	Given your experience in AI in self-assessment, in what ways can AI make self-assessment more possible? What is still not possible? Who would be excited about a technology like this? Who should be leading the charge? How do we get the word out about this technology?
Domain specific questions for Surgeons	Is this something that you can see yourself and other surgeons using to continue to learn new skills? Why or why not? How many more patients would your hospital be able to serve in a week if medical officers could use SELF to learn to do more operations? What would you be able to do with the extra time if medical officers were able to use SELF to learn to do more operations?

Over video call or in-person, interviewees were asked to share open-ended responses to questions about the advantages and disadvantages of simulation-based learning and self-administered training and assessment, both for general surgical education and specifically for training ACs in low-resource areas. A thematic analysis of recorded open-ended responses was conducted. The qualitative data were coded and categorized into major barriers to scale. Identification of these barriers formed the basis for a design thinking focused discussion among the authorship team.

Results

Major identified barriers

Analysis of the semi-structured interviews revealed four major barriers. Representative thematic comments are shown in Table 2.

Table 2.

Key Ideas From Interviews. Anonymized, representative ideas taken directly from interviews and categorized by coded barrier. Direct quotes are delineated by quotation marks.

Representative thematic comments from interviews
Barrier	Representative interviewee responses
Demonstration of safety and quality of surgical care	“The goal is to collect so much data that people cannot say no.” “You can get people to train using and getting better with simulators, but it is unclear how suitable the person can operate in the real world.” We would need to gather data to show there is no or little need for the attending surgeon to stand behind and intervene in those practicing in simulators.
Validation of the principle of self-assessment	“Ultimately, SELF has to be part of a larger protocol. There is no way to get around in-person monitoring.” “Self assessment is the hard part. When we are there in-person, we are giving feedback every second. We are coaching them through. The problem with self-assessment is that you want to train a lot of people, but you don't have enough manpower to provide that level of feedback.” “In training to look for competence, we look for confidence, economy of motion, ability to go fast or slow down, and ability to quickly identify struggles. These are all hard to quantify.”
Translation of simulation skills into surgical knowledge	The biggest issue is scalability: not on the availability product, but the assessment meeting the needs for the learner to achieve the full goal of the procedure. “It is a hard paradigm to sell. Do the peers learning through simulators know how to assess and do the right thing during surgery?”
Integration into an existing legal landscape	“Credentialing is a very contentious issue in global health. It is key to make sure people are safe. We will have the best luck [with credentialing] if it is integrated into an existing program.” “The barriers would be legal in Kenya, which heavily borrowed the British model with heavy regulations and hierarchies. It would be difficult to penetrate the legal system in Kenya, unless there is a change in law”

Barriers

• 1.Demonstration of the safety and quality of surgical care provided after self-administered training

The most important and frequently discussed barrier for the SELF platform was the need to demonstrate positive clinical outcomes for patients treated by self-trained and assessed practitioners. Several subjects expressed that proving patient safety alone would not be sufficient. Instead, according to interview participants, this training modality had the additional burden of demonstrating non-inferior clinical outcomes when compared to traditionally trained surgeons.

• 2.Validation of the principle of self-administered training and assessment

The efficacy of self-administered assessment in surgical education is not yet well established in medical literature, leading to widespread hesitancy to allow learners to train autonomously without traditional supervisory measures and clinical assessments (e.g., exam scores, certifications, faculty feedback). Concern related to this barrier had two parts. First, subjects were unsure if learners could assess their own strengths and weaknesses well enough to make independent progress. Second, there was uncertainty surrounding whether learners could objectively assess their readiness to apply their skills in an actual procedure without external expert evaluation and approval.

• 3.Translation of simulation skills to surgical knowledge

Effective educational programs and simulators must guide novice learners in building not just technical skills, but also the clinical acumen to discern when a given procedure is indicated and how a patient must be managed pre- and post-operatively. Users must also be well-prepares to recognize and manage both common and potentially fatal intraoperative complications. Subjects raised concerns that learners may not be able to adequately learn these elements and demonstrate appropriate clinical judgment after training in an abbreviated, simulated environment. It was thought to be critical to validate the modules in their ability to achieve these aims since these modules are not simply for refining an existing technical skill, but instead for expanding knowledge and skills for managing a new condition not currently being treated by the practitioner.

• 4.Integration into an existing task shifting and task sharing legal landscape

There was also concern about the regulatory permissiveness and local acceptance of task shifting and task sharing. This is governed largely by medical boards and government ministers and varies both between and within countries. Many countries explicitly regulate the practice and may not allow ACs to provide surgical care, even in low-resource areas of great need or in the face of substantial evidence of its effectiveness or utility. Beyond the legality of task shifting and sharing, several subjects cited visceral and personal objections to the practice. Even in countries where it is legal and, to varying degrees, encouraged, subjects demonstrated that the personal opinions of surgeons at a given institution may limit its actual use.

Discussion

Scaling surgical capacity is an urgent, global need that may be addressed by simulation-based learning and self-administered training and assessment. Currently, there is no objective, international, and universally accepted method of evaluating surgical competency. Self-administered training and assessment is uniquely positioned to complement the current system of evaluation with a necessarily objective means for demonstrating operative readiness at all levels of training. Additionally, as a more pressing and short-term goal, these modules may aid in decentralizing and scaling surigical education through technology tailored to low-resource settings, where the need to bolster surgical capacity is greatest. This study identified four major barriers to making SELF technology more widely available globally, especially for associate clinicians (ACs). This is a critical first step towards developing plans for refining and delivering this technology at scale.

Beyond SELF, the four barriers mentioned above reveal important considerations for any initiative seeking to promote self-administered training and assessment. Although the first three barriers focus on different nuances of a complete self-assessment system, together they reflect the concern most frequently cited by subjects: a current lack of evidence. To uphold patient safety, the burden of proof for any new intervention is necessarily high. Rigorously designed studies conducted in target low-resource areas must demonstrate that patient outcomes are improved as compared to the current standard of care, which is often no or an extremely delayed procedure. This is not the same standard of non-inferiority to a traditionally trained surgeon that some subjects personally believed necessary, but it is an important initial demonstration of feasibility. From there, iterative improvements in the technology may allow the standard of care to approach that of a traditionally trained surgeon. Secondly, studies must prove that self-trained practitioners can successfully and safely perform their first procedures with minimal to no expert supervision. Quantifying the lack of expert intervention in a trainee's first few operations is key to demonstrating that mastery can be achieved by self-administered tools. Unequivocal evidence supporting these two hypotheses is necessary for proof-of-concept and will create the foundation for wider implementation of this technology. This will pave the way for studies demonstrating learner competency in real clinical scenarios with increasingly distanced expert intervention (e.g. from in-person expert to remote expert via video conference to possibly no expert presence at all).

After generating and disseminating said evidence, a more detailed conversation about how to integrate self-administered assessment into existing educational systems can begin. The specifics of these conversations will vary by practice setting, but designing modules in a way that generates local buy-in will be paramount, given the diversity seen among subjects' practice settings in terms of local laws and beliefs surrounding both task sharing/shifting and self-administered assessment. The current surgical workforce must not only be convinced by evidence of this technology's utility; it must be excited about its potential for both increasing surgical access and improving the working conditions of the existing cadre. There are several potential ways to create this excitement. One example would be sponsoring several surgeon groups a year to create modules specific to a particular procedure and practice locale such that current surgeons become personally invested in this new training modality. Further, studies that demonstrate the benefits of self-administered assessment for current surgical educators in hours of travel saved, number of additional manuscripts published, or number of additional grants awarded as compared to surgeons who spend time supervising learners traditionally could also help generate buy-in from members of the surgical workforce interested in academic pursuits beyond teaching who currently spend more time supervising trainees than they would ideally.

The enumeration of these barriers may make self-administered assessment seem unachievable in the short-term. However, current global surgery literature demonstrates that the paradigm shift has already begun, given that many surgeries are already performed by non-surgical healthcare professionals in diverse geographic areas in LMICs [7]. SELF as a technology recognizes this reality borne of necessity in many parts of the world, and, in response, seeks to provide validated, standardized tools and methodology to guide and scale these existing training practices. Although studies on SELF are still in their early stages, the principle of using self-training to teach skills to surgical learners has been demonstrated. For example, a study of surgical residents at the University of Texas Southwestern Medical Center proved that residents were able to achieve proficiency in laparoscopy skills with self-training [9]. Furthermore, a study of 20 surgical residents at Rush University Medical Center demonstrated that a two-week simulation training led to a statistically significant decrease in the need for faculty direction [10]. Beyond teaching procedural skills, the construct validity of simulation in surgical decision-making has also been established [11,12].

Although generating evidence in the medical literature is foundational, it need not be the first or only way to promote self-administered training and assessment. For example, in Nigeria, a country where task shifting is already an established and accepted practice, The Tibial Fracture Fixation team, a finalist in the GSTC, is an engine for institutionalization of the process, advocating for integration of their self-administered modules into their ACs' formal curriculum. Simultaneously, this team is conducting research to generate the evidence that will help promote acceptance and excitement internationally. On the other hand, educational leaders in other countries are interested in building the process into existing undergraduate and graduate medical training while they await further evidence of the technology's utility in training independent practitioners without direct supervision in their current scope of practice, such as ACs. In countries like these, integration into the existing educational system may be the most effective means of generating the acceptance and excitement necessary to consider the application of self-administered training to clinicians who have already completed formal training.

Overall, these barriers indicate that increasing access to surgical training, and thus, surgical care will require intentionally and incrementally guiding a global conversation on how surgical education is best achieved, especially considering the current shortage of providers and inequities in surgical densities. The logistical challenges to scaling a technology such as SELF, including access to broadband networks and affordability of even low-cost materials cannot be ignored. However, this investigation has underscored the importance of the unique sociocultural challenges of an innovation that challenges a well-established educational paradigm. Clearly identifying the barriers to implementation as this study has done gives persons and organizations who are interested in furthering these efforts the knowledge with which to develop solutions suited to their varied local contexts.

Limitations

While interviews with multiple global surgery and surgical education experts were conducted, the perspectives of prospective SELF learners and potential patients were not acquired. Learners could provide more insights into perceived barriers to and motivations for learning through SELF. The patient's perspective must also be explored in future research to understand how to navigate potential barriers created by hesitation to receive care from SELF-trained practitioners.

Lastly, only a handful of specific local contexts were fully explored. Institutional barriers, such as task shifting limitations, vary greatly between and within countries. For example, though task shifting is prohibited in Kenya, it is both legal and increasingly practiced in Nigeria. The need for SELF is also stratified by rural vs. urban dimensions. Hospitals in city centers tend to be better-equipped and staffed and may see less need for SELF. Insights from more practitioners in different settings would provide a more comprehensive idea of SELF's potential.

Conclusions

Our study investigated the barriers to scaling self-administered training and assessment, a nascent technology in surgical education for training ACs in surgical skills and decision-making to increase the number of surgical providers in low-resource areas. Interviews of subject matter experts revealed four key barriers to scale: demonstration of the safety and quality of surgical care provided after self-administered training, validation of the principle of self-administered training and assessment, translation of simulation skills to surgical knowledge, and integration into existing task shifting and task sharing legal landscapes. Addressing these barriers will require targeted and context-specific interventions to create local buy-in and, most importantly, generate evidence of this technology's utility. This study provides insight into the most important considerations in implementing and scaling self-administered training and assessment, which will be key for guiding proponents of self-administered training and assessment in creating the cultural change necessary for its widespread use.

Ethical approval

This study did not involve patients or clinical samples and was therefore not subject to ethics committee review.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

CRediT authorship contribution statement

Olubunmi A. Fariyike: Conceptualization, Data curation, Investigation, Methodology, Project administration, Visualization, Writing – original draft, Writing – review & editing. Jacqueline Yao: Conceptualization, Investigation, Methodology, Writing – original draft. Mehdi Baqri: Conceptualization, Investigation, Methodology, Writing – original draft. Peggy Liao: Conceptualization, Data curation, Formal analysis, Investigation, Methodology. Catherine Mohr: Supervision, Conceptualization, Writing – review & editing. George Korir: Conceptualization, Supervision. Taseer Feroze Din: Conceptualization, Supervision, Writing – review & editing. Adam L. Kushner: Writing – review & editing. Sherry M. Wren: Writing – review & editing, Supervision.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Sherry M. Wren reports a relationship with The Intuitive Foundation that includes: consulting or advisory. Adam L. Kushner reports a relationship with The Intuitive Foundation that includes: consulting or advisory. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.