Development of Nested Entrustable Professional Activities: A Delphi Consensus Study in General Surgery

Visual Abstract.

ABSTRACT

Background

Entrustable professional activities (EPAs) are widely used in competency-based medical education; however, most EPAs reflect skills expected later in training, limiting their ability to capture incremental progression among mid-level residents. This lack of granularity obscures meaningful development toward autonomy, undermines resident motivation, and delays identification of struggling learners. We describe a framework of nested EPAs in surgery—defined as smaller, discrete, professional tasks embedded within core EPAs—to address gaps in assessing mid-level trainee progression. This model can be adapted across medical specialties facing similar assessment limitations.

Objective

To describe the development of nested EPAs using surgery as a reproducible model for other specialties.

Methods

In 2024, we created descriptions for nested EPAs using a systematic process. We developed 160 narratives across 10 nested EPAs, organized into 4 performance domains: anatomy, surgical technique, potential errors, and operative steps, each with 4 levels of increasing autonomy (10×4×4). Second, we performed a Delphi consensus study to finalize the nested EPAs and associated narratives (80% agreement for consensus). The Delphi panel included 15 surgical experts (5 women and 10 men) from 8 surgical subspecialties, affiliated with 1 general surgery residency program. The focus was on intraoperative nested EPAs.

Results

In the first round, 136 of 160 items (85%) reached consensus, with 17 updated based on feedback and included in the second round along with 24 items that initially did not meet consensus. All 41 revised items (100%) achieved consensus in the second round.

Conclusions

We demonstrated a model for developing nested EPAs.

Introduction

Graduate medical education is embracing competency-based education (CBE), with entrustable professional activities (EPAs) as the most common competency assessment framework to capture in-the-moment entrustment decisions. In surgery, the American Board of Surgery (ABS) mandates assessment of 18 core EPAs for general surgery residency graduation in the United States.¹ The ABS EPA pilot demonstrated feasibility, but showed that postgraduate years (PGYs) 2 to 4 out of 5 rarely achieve complete entrustment, with lower average entrustment-supervision scores in the intraoperative phase of care.^2-4 This reflects the expected skill set of mid-level residents; however, it may limit residents’ understanding of their progression toward entrustment, discourage them from seeking EPA assessments, and hinder program leadership in identifying struggling residents.⁴

In undergraduate medical education, the EPA framework has evolved through the development of nested EPAs,⁵ “small units of professional practice meeting the EPA definition, which are part of a larger EPA to be entrusted later in training.”⁶ Nested EPAs provide focused feedback for junior trainees on smaller tasks that must be practiced and mastered before being integrated into broader EPAs.^5-7 The application of nested EPAs in graduate medical education, however, remains largely unexplored. This granular assessment may sustain resident motivation by highlighting incremental progress that core EPAs or Accreditation Council for Graduate Medical Education (ACGME) Milestones may not capture.

To improve the assessment of mid-level residents, we propose intraoperative nested surgical EPAs. We defined intraoperative nested surgical EPAs as those standalone tasks that occur within more complex procedures. Nested surgical EPAs act like building blocks within complex procedures, allowing mid-level residents to gain full entrustment first on smaller foundational tasks that are less complex and therefore more easily achievable than the core EPAs. For example, “open entry into the abdomen” is a discrete task common across operations and requires both technical and cognitive competence, such as knowledge of anatomy and operative steps. By identifying tasks that enable mid-level residents to reach full entrustment, we can gain a clearer understanding of the progression of entrustment.

Novel EPA development, which would apply to nested EPAs, includes a data-driven selection of EPAs followed by an iterative process such as a Delphi consensus approach.⁸ Studies on novel EPA development in surgery are scarce beyond the 18 core EPAs, and the concept of nested EPAs remains largely unexplored across surgical and other medical specialties.^2,3,8,9 Furthermore, while EPA completion is required for graduation, requirements for number, thresholds, and distribution remain undefined, making this an opportune time to pursue novel EPA development. Therefore, we developed nested surgical EPAs to provide a more granular intraoperative assessment of mid-level surgical residents, enhancing problem identification and offering clarity on resident progression of autonomy. Though our nested EPAs are specific to surgery, we provide a model for developing novel EPAs such as nested EPAs in graduate medical education.

KEY POINTS

What Is Known

Existing entrustable professional activities (EPAs) often lack the granularity needed to track meaningful intraoperative progression among mid-level residents, limiting feedback and early identification of struggling learners.

What Is New

This study introduces and provides preliminary validity evidence for a structured set of intraoperative nested EPAs, developed through a systematic process and Delphi consensus, to capture smaller, progressive steps toward autonomy.

Bottom Line

Nested EPAs offer residency programs the potential for a more precise, developmentally sensitive tool for assessing growth; this model could be adapted by other specialties to enhance competency-based evaluation.

Methods

Study Design

We employed a data-driven approach for candidate nested surgical EPA selection by identifying common foundational tasks that occur within the core EPAs and other, more complex procedures, followed by a systematic iterative process to draft the narrative descriptions necessary for EPAs.^6,10 We then finalized the nested surgical EPAs and associated narratives following a modified Delphi consensus study.^11,12 We chose a Delphi approach to achieve consensus on the nested surgical EPAs given its common use in novel EPA development.^11-15 We completed all phases of the study in 2024.

Item Development

Items are defined as the nested surgical EPAs, along with their definitions and narratives, which were developed and then refined through the Delphi approach.

Nested Surgical EPA Identification:

To identify potential surgical tasks that could be used as nested surgical EPAs, we convened an education leadership team comprised of 2 education research scientists, 2 associate program directors of general surgery, the vice chair of education in surgery, and 5 surgical residents. The surgical residents were all senior residents (1 PGY-4, 3 PGY-5s, and 1 PGY-6) participating in education research fellowships, and 2 held master’s degrees in education. This education leadership team is distinct from the Delphi participants, with no overlap between the 2 groups. First, the team collectively defined nested surgical EPAs to ensure clear understandings of the concept.¹⁶ The team defined nested surgical EPAs as “…standalone surgical tasks that commonly occur as a part of a more complex procedure and should occur in multiple different types of procedures. The nested EPA should be a task that a mid-level trainee could achieve complete entrustment on prior to gaining entrustment on the entire EPA. The purpose of a nested EPA is to improve the assessment of junior residents by identifying a task that junior residents could achieve complete entrustment on.”

Using EPA data from our institution, we identified 22 potential nested surgical EPAs by cataloging common operative tasks appearing in both core EPAs and non-core procedures (procedures assessed with the same entrustment supervision scale but not included among the core EPAs). These non-core procedures frequently occur in general surgery training and are increasingly relevant as residents progress, thus supporting their relevance for nested EPAs.¹⁷ We tallied their occurrences across core and non-core procedures to determine which candidate nested EPA tasks were most frequent in general surgery training (Figure 1, Box 1). From this list, we selected nested EPAs that were distinct from both the core EPAs and each other. For example, although lysis of adhesions was frequent, it was excluded because it is part of other potential nested EPAs (eg, abdominal entry) and already covered in the core EPA for small bowel obstruction. We also confirmed that the selected nested surgical EPAs all fit within procedures listed in the ACGME defined category minimum requirements for general surgery.¹⁸ Finally, we aimed for the nested surgical EPAs to represent activities that a resident could achieve autonomy in by the end of their third clinical year. We selected the third year of clinical residency as it represents a critical point halfway through a 5-year residency, as residents start assuming chief responsibilities and are often expected to take on portions of cases with less direct supervision.

Figure 1.

Frequency of Candidate Nested EPAs Occurring in Core EPAs and “Other” Procedures

Abbreviations: EPA: entrustable professional activity; MIS, minimally invasive surgery.

Note: The frequencies of candidate nested surgical EPAs within core EPAs and “other procedures” are presented in Box 1. The selected nested surgical EPAs are shown in green. Tasks shown in blue were not selected because they are already included in other EPAs and/or embedded within the selected nested EPAs. Tasks not selected due to low frequency are shown in red.

Box 1. Core EPAs and Non-Core Procedures Used for Mapping Out Frequency of Potential Nested EPAs

• Abdominal hernia – MIS

• Abdominal hernia – open

• Acute abdomen

• Anorectal disease

• Appendicitis – MIS

• Breast disease

• Colon disease

• Gallbladder disease – MIS

• Gallbladder disease – open

• Inguinal hernia

• Inguinal hernia – MIS

• Small bowel obstruction/lysis of adhesions

• Soft tissue infection

• Cutaneous and subcutaneous neoplasms

• Dialysis access

• Thyroid procedures

• Parathyroid procedures

• Non-core procedures

  • Liver resection

  • Pancreas resection

  • Gastrectomy

  • Limb amputation

  • Rectopexy – MIS

  • Ileostomy takedown

  • Colostomy takedown

  • Abdominal fistula takedown

  • Splenectomy

  • Gastrostomy or jejunostomy tube placement

  • Paraesophageal hernia repair/Nissen – MIS

  • Lung resection – MIS

Abbreviations: EPA, entrustable professional activity; MIS, minimally invasive surgery. Note: A list of the core EPAs as well as the most common procedures with EPAs completed that are not a part of the core EPAs. This list was used to identify the frequency of which the candidate nested EPAs occurred across these procedures and pathologies.

Narrative Development:

Narrative items are the written descriptions of the expected behaviors associated with each level of performance. For each nested surgical EPA, we identified 4 domains to assess trainee performance: knowledge of anatomy, surgical technique, recognition of potential errors, and steps of the operation.¹⁹ The 4 domains used to assess resident performance were developed based on the narratives of the ABS core EPAs, the ACGME Milestones, and the expertise of the general surgery program leadership team. Each domain contained 4 levels of increasing autonomy, analogous to the 4 levels on the entrustment-supervision scale.^10,20 This approach divides the narratives into smaller, focused domains, enabling more specific and actionable feedback while enhancing user-friendliness by avoiding a single, lengthy block of text. The narratives underwent a systematic iterative process until agreement was established. The process began with 2 authors (A.M., C.G.) drafting the narratives, using language and formatting like the core EPAs and following EPA creation guidelines.^1,16 The education leadership team and surgical faculty with EPA expertise reviewed the narratives to assess clarity, relevance, and appropriateness. Based on their feedback, we revised and refined the items. This process was completed 3 times, until agreement was reached. This process generated 160 narrative items (10 EPAs × 4 domains × 4 levels). The Delphi consensus study consisted of this list of nested surgical EPAs, along with their definitions and narrative items.

Delphi Study

Expert Identification and Recruitment:

We identified experts as faculty members who most frequently operated with residents and received a score greater than a 3.25 out of 4 on a 6-item form that residents use to rate faculty. This criterion was chosen to identify a faculty cohort with extensive experience working with residents who demonstrated proficiency in key categories of effectiveness as an educator. Although our experts were affiliated with a single general surgery program, they represented diverse institutions, surgical specialties, and years in practice.

Delphi Process:

Prior to starting the study, we determined that 80% consensus would be the standard for an item to be included.^12,13 After experts expressed an interest in participating, we conducted the first round of the process with face-to-face guidance to facilitate survey completion, given the number of items. Experts responded in a yes/no format if residents could reach complete entrustment by the end of the third clinical year of residency. For the definitions and narratives, experts dichotomously responded with agree/disagree. If participants did not agree with an item, we asked for their suggestions to improve the item. We then asked the same experts to participate in a second round online. The second round included items that participants had previously agreed upon as a reminder, but they were asked to provide feedback only on the revised items. Faculty entered responses in a Qualtrics form (Qualtrics LLC). We maintained anonymity between participants throughout the process to prevent bias from peers. Participants did not convene at any point to discuss the development of the nested surgical EPAs. We disseminated results to all participants at the end of the second round.

Analysis:

After experts participated in the first round, we measured the response rate and percentage of agreement. We required a minimum of 12 out of the 15 participants to reach consensus. If there was less than 80% consensus, we updated the items using participant feedback and included them on the second round of the Delphi. If an item reached 80% consensus, but could be improved based on participant feedback, it was revised and also included in the second round of the Delphi. Each item was analyzed independently, allowing a single level from 1 domain to be considered in isolation during round 2. If we did not reach consensus after 2 rounds, a third round would be completed. This process is summarized in Figure 2.

Figure 2.

Description of Expert Identification and Delphi Process

Our institutional review board (IRB) determined this study to be exempt from review (IRB 23-39766).

Results

Out of 16 experts who expressed interest in participating before starting the process, 15 (94%) subsequently completed both rounds. These 15 experts were 5 women and 10 men. Experts included a practicing general surgeon (1) and general surgery subspecialists: vascular surgery (1), trauma and acute care surgery (3), endocrine surgery (1), pediatric surgery (1), colorectal surgery (5), minimally invasive surgery (2), and surgical oncology (1). The study participants represented 4 institutions: a large academic center, a level one trauma center, a major health maintenance organization hospital, and a private community hospital. Experts had a median of 15 years in practice (range 0.5-35). We included 1 faculty member with a high average evaluation score, who was in their first 6 months of independent practice and had fewer average cases, to provide an additional perspective. Experts received an average score of 3.84 (range 3.34 to 4) on resident-completed faculty evaluations and had an average of 161 cases with a resident over the prior academic year (2023-2024).

We presented 10 nested surgical EPAs in the first round of the Delphi (Box 1). The Table includes a sample of one of the finalized nested surgical EPAs to demonstrate the information requested for expert evaluation. Each nested surgical EPA was listed with a corresponding definition to ensure a clear understanding of the task. After the first round of the Delphi, 8 of the 10 (80%) nested surgical EPAs passed with experts agreeing residents could reach complete entrustment on the task before the end of the third clinical year. “Exposure and ligation of a major vessel” and “mobilization of stomach and entry of lesser sac” did not pass. Based on participant feedback, we revised the titles and definitions of these nested surgical EPAs. The titles were updated to “exposure and control of a major vessel” and “entry of lesser sac.” Both nested surgical EPAs reached 80% consensus in the second round of the Delphi (final list of nested surgical EPAs displayed in Box 2).

Table.

Nested EPA: Open Abdominal Closure

Definition	Primary fascial closure, either simple or running, with multilayer superficial closure. Able to select appropriate suture and needle size. Reapproximates abdominal wall layers with appropriately sized bites on fascia and equal travel between sides. Includes decision-making, positioning, and placement of abdominal drains.
Level	Knowledge of Anatomy	Surgical Technique	Steps of Operation	Recognition of Potential Errors
1 Limited participation: Demonstrates understanding of information and has very basic skills	Demonstrates basic understanding of layers of abdominal wall	Handles instruments safely but tentatively Follows intraoperative directions Displays coordinated hand movements for simple maneuvers under direct instruction, but is inefficient with basic skills	Requires prompting to articulate principles of fascial closure	Able to evaluate basic complications (eg, pulled through suture, large empty subcutaneous space) Requires assistance to recognize some signs of complex complications (eg, hollow viscus injury, muscle perforator bleeding)
2 Direct supervision: Demonstrates understanding of the steps of the operation but requires direction through principles and does not know the nuances of a basic case	Understands layers of abdominal wall, including changes expected above/below arcuate line Able to describe different types of closure (eg, simple, running)	Performs basic closure (eg, suturing, tissue exposure) skills with guidance Actively retracts and assists with basic portions of the operation Requires moderate guidance to ensure appropriate fascial bite sizes Able to close superficial layers but requires occasional guidance on depth/quantity of superficial layer closure	Performs basic closure (eg, suturing, tissue exposure) skills without guidance Demonstrates economy of motion Independently ensures appropriate fascial bite sizes and distance traveled Independently closes superficial layers without guidance Able to identify fascia on re-operative/scarred field, but requires some guidance to ensure appropriate fascial bites	Able to independently manage basic complications (eg, pulled through suture, gaps between sutures, large empty subcutaneous space) Requires assistance to manage complex complications (eg, hollow viscus injury, muscle perforator bleeding) Articulates techniques to avoid hollow viscous injury during fascial closure
3 Indirect supervision: Can do a basic operation but will not recognize abnormalities and does not understand the nuances of an advanced case	Understands all layers of abdominal wall Able to perform fascial closure using evidence-based data Requires some guidance with closure in patients with previously violated or weakened fascial planes	Performs basic closure (eg, suturing, tissue exposure) skills without guidance Demonstrates economy of motion Independently ensures appropriate fascial bite sizes and distance traveled Independently closes superficial layers without guidance Able to identify fascia on re-operative/scarred field, but requires some guidance to ensure appropriate fascial bites	Able to perform fascial and superficial layer closure with minimal guidance in straightforward case Has basic knowledge of mesh closure techniques, but unable to determine need without prompting Appropriately chooses needle and suture type	Able to independently manage both basic complications (eg, pulled through suture, gaps between sutures, large empty subcutaneous space) and complex complications (eg, hollow viscus injury, muscle perforator bleeding) with some guidance Demonstrates techniques to avoid hollow viscus injury during fascial closure
4 Practice ready: Can manage more complex patient presentations and operations and take care of most cases	Understands all layers of abdominal wall Independently identifies abnormal anatomy and is able to pivot operative plan as needed to accommodate for unexpected findings Able to walk a junior resident through abdominal wall closure in a straightforward case	Independently performs setup for efficient closure Identifies intraoperative challenges (scarred/damaged fascia) that requires careful technique Adapts tissue handling to avoid additional tissue damage	Independently performs fascial and superficial layer closure in both straightforward case and complicated case (eg, redo laparotomy, fascial closure under tension or after dehiscence, in patients with poor wound healing, ie, immunosuppression) Devises/implements plan when deviation from the initial operative plan is required Independently determines need for adjuncts such as mesh	Independently able to manage both straightforward and complex surgical complications Takes appropriate precautions to minimize common complications associated with fascial/superficial abdominal wall closure Knows when to leave skin open or perform delayed closure

Abbreviation: EPA, entrustable professional activity.

Note: Sample nested EPA and narratives with 4 levels of progression of autonomy across the resident performance domains (Knowledge of Anatomy, Surgical Technique, Steps of Operation, Recognition of Potential Errors).

Box 2. Final List of Nested EPAs

• Bowel anastomosis

• Creation of ostomy

• Exposure and ligation of a major vessel

• Minimally invasive entry to abdomen

• Mobilization of left colon

• Mobilization of stomach and entry of lesser sac

• Mobilization of right colon

• Open abdominal closure

• Open entry to abdomen

• Ostomy takedown

Abbreviation: EPA, entrustable professional activity. Note: The 10 nested EPAs that reached 80% consensus after 2 rounds of the Delphi.

All experts reviewed the 160 narrative items. In the first round, the group reached consensus on 136 of 160 decisions (85%). The online supplementary data Table presents the first round Delphi results, where green denotes items that reached consensus, yellow represents items that were revised despite achieving consensus, and red indicates items that failed to reach consensus.

The first round left 24 items to be considered in the second round. Additionally, the authors chose to update 17 of the items that reached consensus on the first round with expert feedback to provide additional clarity and specificity on the narrative item. Experts reviewed these 41 items in the second round of the Delphi, and all items (100%) reached consensus. The online supplementary data contains the final list of the 10 nested surgical EPAs with their corresponding definitions and narratives.

Discussion

We identified and defined intraoperative nested surgical EPAs through a data-driven and systematic iterative process, followed by a modified Delphi approach to reach consensus on 10 intraoperative nested surgical EPAs and their associated narratives. This model can be followed by other disciplines to identify tasks meriting entrustment earlier than other more complex tasks, allowing assessment data to help inform a resident of their progress.

The results of our study will provide faculty with a tool to provide feedback on relevant surgical tasks. Feedback is essential for residency training, but residents often feel it is inadequate.^21-23 EPAs can improve formative feedback, especially when used in tandem with a structured debrief.²⁴ Nested EPAs create an opportunity for targeted feedback on more granular tasks, which can be mastered earlier in training. The feedback on nested EPAs can be used to recognize increased autonomy and decreased supervision, and to give meaning to progressional steps, both to sustain learner motivation and to attest and document progression. Nested EPAs offer an opportunity for early problem identification through this granular assessment, potentially highlighting performance issues that might be overlooked with core EPAs alone.

Across procedural disciplines, studies support breaking down procedures into discrete tasks to improve surgical education and procedural learning.^25-27 We used a similar case breakdown, with the emphasis on assessment of these discrete tasks. Additionally, we selected tasks that incorporate both technical and cognitive components and align with the defining principles of EPAs as discrete, observable, and entrustable units of practice.^1,5-7,20,28 We recognize that breaking down cases into specific tasks could reduce feasibility and risk faculty disengagement, thus reducing completion rates.²⁹ However, the nested surgical EPAs are not intended to replace or complicate the existing EPA framework, but to enhance it by providing flexibility and granularity where needed. Faculty burden should remain unchanged, as they are already required to complete EPAs for resident graduation and would continue completing a single assessment per trainee per operation, whether for a core EPA, non-core EPA, or nested EPA, using the same platform and workflow. Nested EPAs, like core EPAs, are not intended to replace ACGME Milestones but rather to provide additional ad hoc assessment data that can support summative evaluations.

When considering nested EPAs for graduate medical education, there are a few considerations. Nested EPAs are a developmental concept and should lose their meaning as a trainee progresses, as something that was useful in the past but no longer relevant as the core EPA is more relevant for the advanced trainee.^5-7 Therefore, completion of training should focus on completion of core EPAs. Nested EPAs could be useful in situations involving multiple trainees since a junior learner may perform only some foundational tasks while a senior resident performs the core EPA. It would be more appropriate to assess the junior learner on a nested EPA rather than the procedure in its entirety. Additionally, the role of nested EPAs should be considered in relation to ACGME Milestones. While both frameworks are developmental, their purposes differ. Milestones represent specialty-specific markers of competence achieved over time and help track longitudinal progress along a training trajectory.³⁰ In contrast, nested EPAs focus on real-time entrustment decisions for discrete clinical tasks, which are more actionable and intuitive than the Milestones. For example, a task like “opening the abdomen” may be entrusted with indirect supervision early in training, with real-world implications for autonomy and patient care. Milestones do not have a focus on a decision to entrust trainees with task. The reason to define nested EPAs is to operationalize a stepwise and safe increase in autonomy during training.³¹

There are limitations to this study. First, development of the nested surgical EPAs was informed by analysis of non-core procedures from a single general surgery residency program; however, the resulting nested EPAs align with the nationally required core EPAs and therefore remain broadly applicable across programs. Institutions with different case mixes may identify additional tasks appropriate for inclusion. Second, experts are all from the same city in the United States, with all institutions affiliated with the same general surgery residency program. We chose these sites since we thought it was critical to be able to conduct the first round in person due to the numerous ratings and the overall detailed nature involved in defining nested EPAs. We believe that the diversity of disciplines and sites helps to minimize the bias of a single general surgery training program. However, we do acknowledge that including very different geographical (eg, non-urban) regions or countries could yield different opinions on suitable nested EPAs. Finally, we proposed a model for nested EPA development specific to general surgery, making our work most easily translatable to other procedural specialties. Future studies should examine how integrating nested EPAs into routine assessment influences faculty engagement, resident development, and program-level decision-making.

Conclusions

We used a modified Delphi approach to develop intraoperative nested surgical EPAs to improve the assessment of mid-level surgical residents and provide a model adaptable to other specialties.

Author Notes

Funding: The authors received a local institutional grant for $25,000 (UCSF Innovations Funding) which largely supported the salary of the medical student research assistant.

Conflict of interest: Drs Murillo and Brian participate in the Intuitive-UCSF Simulation-Based Surgical Education Research Fellowship. Dr Gomes participates in a research fellowship through Johnson & Johnson.

Editor’s Note

The online supplementary data contains the results of the first round of the Delphi study for 10 candidate nested surgical EPAs and the final list of the 10 nested surgical EPAs with their corresponding definitions and narratives.