American Board of Orthopaedic Surgery’s Initiatives Toward Competency-Based Education

Abstract

The American Board of Orthopaedic Surgery (ABOS) is the national organization charged with defining education standards for graduate medical education in orthopaedic surgery. The purpose of this article is to describe initiatives taken by the ABOS to develop assessments of competency of residents to document their progress toward the independent practice of orthopaedic surgery and provide feedback for improved performance during training. These initiatives are called the ABOS Knowledge, Skills, and Behavior Program. Web-based assessment tools have been developed and validated to measure competence. These assessments guide resident progress through residency education and better define the competency level by the end of training. The background and rationale for these initiatives and how they serve as steps toward competency-based education in orthopaedic residency education in the United States will be reviewed with a vision of a hybrid of time and competency-based orthopaedic residency education that will remain 5 years in length, with residents assessed using standardized tools.

Introduction

One mission of the American Board of Orthopaedic Surgery (ABOS) is to maintain and define educational criteria for graduates of accredited orthopaedic programs to sit for Part I and Part II examinations as part of board certification. To fulfill this mission, the ABOS has made steps toward competency-based education through the development of the ABOS Knowledge, Skills, and Behavior (ABOS KSB) Program. The ABOS has partnered with national educational organizations, including the Accreditation Council for Graduate Medical Education (ACGME), the American Orthopaedic Association/Council of Orthopaedic Residency Directors (AOA/CORD), the Association of Residency Coordinators in Orthopaedic Surgery (ARCOS), and the American Academy of Orthopaedic Surgeons (AAOS).

Historical Perspective

Modern surgical education is largely credited to have started with William Stewart Halsted. Halsted developed educational programs and trained residents using a pyramidal system. Competency was personally assessed by Halsted, and he determined the endpoint of residency training¹.

Edward Churchill, a general surgeon in Boston in the 1950s, is credited with changing surgical training from a pyramidal apprenticeship system to a time-based system. Since then, surgical training has had a predefined duration, at the end of which independent practice begins².

Competency-based education with competency as an endpoint, rather than time, was trialed at the University of Toronto³. Residents need different amounts of time to acquire important skills; thus, it is rational that competency should be the endpoint of training rather than time. Despite this logic, there have been obstacles to implementing a competency-based education system, including residents fulfilling service needs, expense, lack of a curriculum, faculty time and effort, and inadequate assessments to support a judgment of competency. Other international initiatives toward orthopaedic competency-based education have been made in both the United Kingdom and Australia^4,5. The entire Canadian Graduate Medical Education system, surgical and nonsurgical, has committed to competency-based training goals, described in a publication by the Royal College of Physicians and Surgeons of Canada entitled “Competence by Design”⁵ with full engagement by 2021.

In the United States, the ACGME has introduced multiple initiatives since 2003 toward defining and incorporating competency into the accreditation of residency training programs (Table I). Specific to competency-based education, in 2008, the ACGME defined 6 core competencies and provided a toolbox for programs to assess these competencies (Table II). The 6 core competencies have been widely accepted and are now integral to all U.S. graduate medical education (GME) programs. With the advent of the Next Accreditation System, the ACGME introduced the Milestones Program to provide uniform benchmarks across all U.S. GME. Unfortunately, in orthopaedic surgery, although required, the Milestones have not been uniformly embraced⁶. This negative experience provided evidence that to be widely accepted assessments must add value to an educational program and has helped drive the Milestones 2.0 Project described below.

TABLE I.

ACGME Initiatives for Accreditation Standards

Year	ACGME Initiatives for Accreditation Standards
2003	80-Hour Work Week
2007	Case Log System
2008	Core Competencies
2011	Work-Hour Restriction
2013	Surgical Case Minimums
2013	Milestones Project
2013	Next Accreditation System
2017	Clinical Experience, Education Hours, and Resident Well-Being

ACGME = Accreditation Council for Graduate Medical Education.

TABLE II.

2008 ACGME Core Competencies

1. Patient care

2. Medical knowledge

3. Practice-based learning

4. Systems-based practice

5. Professionalism

6. Interpersonal skills and communication

ACGME = Accreditation Council for Graduate Medical Education.

ABOS Overall Vision

Currently, to be eligible for the ABOS Part I Board Examination in Orthopaedic Surgery and to practice independently, a resident must complete 5 years in an ACGME-accredited orthopaedic residency program, including time requirements of 46 weeks of adult orthopaedics, 46 weeks of fractures/trauma, 23 weeks of pediatric orthopaedics, and 23 weeks of basic and/or clinical specialties, in addition to specific rotation requirements during the PGY1 year⁷. The Program Director (PD) must attest to the resident's competence at the end of the training time in order for the resident to be eligible for the ABOS Part I Examination taken in July after successfully graduating from an ACGME Residency Program. There are no specific guidelines defining resident competence for independent practice, and there is no defined detailed curriculum describing what they must accomplish; these determinations are at the discretion of the PD of each program.

To improve this situation, in 2015, the ABOS developed a vision for steps toward competency-based education by developing the KSB project. The first part of the vision was that residents would be actively involved in their progress toward becoming a competent surgeon. Second, to determine whether competency was achieved, standardized assessment tools would be required. The ABOS planned to engage the residents and develop competency assessments of resident performance in 3 core areas: knowledge, skills, and professional behavior.

Knowledge

The Part I ABOS Examination is a secure test with a passing standard that is taken after residency training is successfully completed. This timing has disadvantages because the resident has not been assessed with a standardized test for medical knowledge while in the training environment. Presently, the PD signs off on a graduating resident's competence without standardized testing of knowledge⁸.

During training, the only nationally accepted assessment of residents' medical knowledge is the AAOS Orthopaedic In-Training Examination (OITE). The OITE is a comprehensive, multiple-choice, computer-based examination designed to facilitate knowledge assessment in established principles and conventional procedures and treatment modalities in orthopaedic surgery using an OITE blueprint. As a protected examination given annually since 1963, the OITE relies on approximately 275 new questions annually as its questions are released back to the residents for educational purposes; this has the disadvantage of precluding the use of validated, standardized questions. However, this test is designed primarily as an educational tool⁹. Unfortunately, the correlation of the OITE with the Part I Board Examination is only moderate¹⁰.

National Collaboration

The ABOS KSB Program has a goal of establishing a knowledge standard that will be administered during residency training. In March 2019, a Knowledge Task Force was convened with representatives from the AOA/CORD, ABOS, ACGME, and AAOS (Table III) to discuss tactics to improve standardized knowledge assessment during residency. The group agreed to the plan shown in Table IV, which continues to be implemented, including agreement to use a common blueprint for the 2 examinations and linking of the 2 examinations through common questions given on both examinations.

TABLE III.

Resident Knowledge Assessment Task Force Meeting Attendees March 12, 2019

Orthopaedic Organization	Task Force Attendees
American Board of Orthopaedic Surgery	April Armstrong, MD, Michael Bednar, MD, Jim Carpenter, MD, John Harrast, MS, Larry Marsh, MD, David Martin, MD, Terry Peabody, MD, and Ann Van Heest, MD
American Academy of Orthopaedic Surgery	David Halsey, MD, Norman Otsuka, MD, Paul Tornetta, MD, Courtney Dunker, Elizabeth Durham, Margie Heisler, Peter Shores, Jonathan Sprague, Connie Thorsen, and Ann Troise
Accreditation Council of Graduate Medical Education	Pam Derstine, PhD, and Dawn LaPorte, MD
American Orthopaedic Association/Council of Orthopaedic Residency Directors	Matthew Beal, MD, Monica Kogan, MD, Joshua Patt, MD, and Jodene Brown
American Board of Medical Specialties	David Swanson, PhD

TABLE IV.

Collaboration Agreement on Resident Knowledge Assessment: March 2019

Goals	Actions
Collaboration on the development of a Knowledge Blueprint to be used as the basis for both ABOS Part I examination and OITE content	Completed January 2020
Linking of the ABOS Part I examination and OITE through	(1) 50-60 OITE questions to be reviewed by ABOS question writing process and incorporated as nonscore items on Part I examination (July) and OITE (November) (2) Withholding of the 50-60 linking questions from release each year to establish a pool of standardized equator questions (3) Psychometric linking of the 2 examinations based on the use of the equator questions

ABOS = American Board of Orthopaedic Surgery, and OITE = Orthopaedic In-Training Examination.

In 2020, the AAOS OITE and the ABOS Part I Examination were successfully linked by delivering a standard group of questions on both examination forms. These questions were not released after the examination. This allowed a psychometric evaluation that led to the development of a standard on the AAOS OITE that could be roughly correlated with a passing performance on the ABOS Part I Examination. The ABOS also revised its blueprint in conjunction with the AAOS and appropriate subspecialty societies; this blueprint is used for both the ABOS Part I Examination and the AAOS OITE development starting in 2020¹¹. These developments should theoretically give the PDs valuable tools to follow resident knowledge acquisition.

Surgical Skills Assessment Tool Development

Most of residency training occurs in the operating room learning technical skills and outside of the operating room developing patient care and medical decision-making skills. There are no standardized, widely used assessments of these skills. These learning environments provide an opportunity for real-time workplace assessments of patient care skills.

The ABOS KSB Program began its efforts in competency-based education by developing and piloting surgical skills assessment tools during resident education. The goal was to provide a framework for documenting surgical competency using real-time practice-based assessments by faculty of core orthopaedic procedures for each resident before graduation. Smartphone-optimized web-based technology was used to enhance user acceptance of the program¹².

The ABOS developed 2 surgical skills assessment tools for comparison. A group of 16 program directors (AOA/CORD) collaborated with the ABOS to develop a formative tool evaluating 8 facets of surgical management based on the work of Hamstra in Ottawa (O-tool)¹³. The O-score includes an evaluation of 8 steps of the surgical procedure using a 5-point scale, as outlined in Table V. The P-score is a summative evaluation using a 5-level assessment based on the work of Zwischenberger^14,15, as outlined in Table VI. To compare these 2 tools, the ABOS and AOA/CORD collaborated in a study from January to June 2017, comparing the O-tool and P-tool web-based evaluation assessments as completed by faculty¹². Two hundred ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using the 2 web-based evaluation tools in a cross-over design. One thousand one hundred fifty Ottawa (O-score) evaluations, a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, a single-question summative evaluation. The findings demonstrated construct validity among entry-level, intermediate-level, and advanced-level residents. The survey results indicated that greater than 70% of the faculty felt confident that this surgical skills assessment allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training¹². Both the O and P tools were thus combined into the ABOS Surgical Skills Assessment Tool.

TABLE V.

O-Score Assessment (2-Part Table)

Domain	Description
Preprocedure plan	Gathers/assesses required information to reach diagnosis and determine the correct procedure required
Case preparation	Patient correctly prepared and positioned, understands approach and required instruments, and prepared to deal with probable complications
Knowledge of specific procedural steps	Understands steps of the procedure, potential risks, and means to avoid/overcome them
Technical performance	Efficiently performs steps, avoiding pitfalls and respecting soft tissues
Visuospatial skills	3D spatial orientation and able to position instruments/hardware where intended
Postprocedure plan	Appropriate complete postprocedure plan
Efficiency and flow	Obvious planned course of the procedure with economy of movement and flow
Communication	Professional and effective communication/utilization of staff

Level	Description
1—“I had to do”	Requires complete hands-on guidance, did not do, or was not given the opportunity to do
2—“I had to talk them through”	Able to perform tasks but requires constant direction
3—“I had to prompt them from time to time”	Demonstrates some independence but requires intermittent direction
4—“I needed to be in the room just in case”	Independence but unaware of risks and still requires supervision for safe practice
5—“I did not need to be there”	Complete independence, understands risks and performs safely, practice ready

Scale for O-score used for each domain.

TABLE VI.

P-Score Assessment Tool

Level	Description
Novice	Attending surgeon provides maximum assistance Demonstrates knowledge of anatomy Demonstrates basic operative skills (e.g., incision, excision, and wound closure) Positions the patient First assists and observes
Low intermediate	Attending surgeon provides significant assistance and direction Performs the approach with minimal assistance Identifies anatomical landmarks Identifies most of the critical steps Demonstrates proficiency with the component technical skills Demonstrates an increasing ability to perform different key parts of the operation with attending assistance Demonstrates room setup and equipment management
High intermediate	Attending provides modest assistance and direction Capable of performing the approach Familiar with anatomic landmarks Identifies and is capable of performing almost all the critical steps Demonstrates proficiency with the component technical skills Attending assistance required for the most challenging portions of the procedure
Supervision-only competent	Attending surgeon not required to provide active assistance Knows steps and transitions easily Able to direct and assist a more junior resident Aware of environment and can manage patient safety and coordinate the operative team Capable of performing the procedure in practice independently
Advanced expertise-proficient	Performance matches that of the advanced surgeon Capable of performing complex procedures independently Capable of independent management of intraoperative complications

The ABOS envisioned the use of the Surgical Skills Assessment Tool for an expanded number of surgical procedures. To develop this vision, a Resident Surgical Skills Assessment Task Force Meeting was held on April 13, 2019, with representation from the ABOS, AOA/CORD, AAOS, and ACGME (Table VII). This task force used multiple sources of information regarding surgical case frequencies^16,17 to establish a list of core procedures (Table VIII) that all orthopaedic residents should be able to competently perform by the time of residency graduation and a list of subspecialty procedures (Table IX) that all orthopaedic residents should have required participation during residency training.

TABLE VII.

Resident Surgical Skills Assessment Task Force Attendees Meeting April 3, 2019

Orthopaedic/Surgical Organization	Task Force Attendees
American Board of Orthopaedic Surgery	April Armstrong, MD, Michael Bednar, MD, Jim Carpenter, MD, Kevin Garvin, MD, John Harrast, MS, Shep Hurwitz, MD, Larry Marsh, MD, David Martin, MD, Terry Peabody, MD, Scott Porter, MD, Charles Saltzman MD, Ann Van Heest, MD
American Orthopaedic Association/Council of Orthopaedic Residency Directors	Lauren Geaney, MD, Trent Guthrie, MD, Matthew Karam, MD, Joshua Patt, MD
Accreditation Council of Graduate Medical Education	Pam Derstine, PhD, Dawn LaPorte, MD
American Academy of Orthopaedic Surgery	Kristy Weber, MD
Society for Improving Medical Professional Learning	Brian George, MD

TABLE VIII.

Tier 1, Core Procedures That All Graduating Residents Should Document Competency: List of 35 Procedures

Orthopaedic Subspecialty	Tier 1: Core Procedures
Adult reconstruction	1. Primary total hip arthroplasty 2. Primary total knee arthroplasty
Foot and ankle	1. Tibial shaft fx 2. Unimalleolar/bimalleolar (syndesmosis) fx
Hand	1. Carpal tunnel release 2. Trigger finger release 3. Radius/ulna shaft fx 4. Distal radius fx 5. Carpal/metacarpal/phalangeal fx/dislocation
Oncology	1. Prophylactic fixation of impending pathologic femur fx 2. Below-knee amputation
Pediatrics	1. Ankle/distal tibia fx
Shoulder and elbow	1. Simple shoulder arthroscopy (e.g., debridement and subacromial decompression) 2. Distal humerus fx/elbow dislocation 3. Radius/ulna shaft fx
Spine	1. Spine exposure
Sports medicine	1. Quadriceps/patellar/Achilles tendon repair 2. Arthroscopic meniscectomy 3. Meniscal repair 4. ACL reconstruction
Trauma Upper Lower General	1. Distal humerus fx/elbow dislocation 2. Radius/ulna shaft fx 3. Distal radius fx 4. Carpal/metacarpal/phalangeal fx/dislocation 5. Femoral neck fx 6. Intertrochanteric fx 7. Femoral shaft fx 8. Distal femoral fx 9. Tibial plateau fx 10. Tibial shaft fx 11. Unimalleolar/bimalleolar (syndesmosis) fx 12. Fasciotomy 13. Unilateral external fixation 14. Debridement of open fx 15. Hardware removal

TABLE IX.

Tier 2, Subspecialty Procedures That All Graduating Residents Should Document Exposure: List of 53 Procedures

Orthopaedic Subspecialty	Tier 2: Subspecialty Procedures
Adult reconstruction	1. Revision total hip arthroplasty 2. Revision total knee arthroplasty
Foot and ankle	1. Ankle arthroscopy 2. Ankle/subtalar/midfoot/metatarsal phalangeal of great toe fusion 3. Ankle/foot ligament/tendon repair/reconstruction 4. Bunion correction 5. Tibial pilon fx 6. Hindfoot fx 7. Midfoot fx/dislocation 8. Fx nonunion/malunion
Hand	1. Wrist/hand bone procedure (e.g., fusion and excision) 2. Wrist/hand soft-tissue repair/reconstruction 3. Proximal radius/ulna fx
Oncology	1. Biopsy of bone lesion 2. Biopsy of soft-tissue lesion 3. Benign bone tumor curettage and grafting 4. Benign soft-tissue resection
Pediatrics	1. Epiphysiodesis 2. Pelvic/femoral osteotomy 3. Hip arthrotomy/drainage 4. Hip/knee/ankle tendon lengthening 5. Slipped capital femoral epiphysis 6. Supracondylar humerus fx
Shoulder and elbow	1. Rotator cuff repair 2. Advanced shoulder arthroscopy (e.g., tarsorrhaphy and labral repair) 3. Total/reverse/hemi shoulder arthroplasty 4. Distal biceps tendon repair 5. UCL repair/reconstruction 6. Ulnar nerve decompression 7. Clavicle fx 8. Proximal humerus fx 9. Humeral shaft fx 10. Proximal radius/ulna fx
Spine	1. Single‐level spine fusion 2. Multilevel spine fusion 3. Spinal decompression/laminectomy 4. Discectomy 5. Sugar tongs/halo application
Sports medicine	1. Rotator cuff repair 2. UCL repair/reconstruction 3. Hip arthroscopy 4. LCL/MCL/PCL repair/reconstruction

LCL = lateral collateral ligament, MCL = medial collateral ligament, PCL = posterior cruciate ligament, and UCL = ulnar collateral ligament.

Because orthopaedic patient care competency includes both surgical skills and nonoperative patient care, the Orthopaedic Nonoperative Evaluation Tool (ONE Tool) has been developed and is planned to be implemented to match the Milestones 2.0 Patient Care Evaluation and Management of Adult and Pediatric Patient Milestones. The ONE Tool evaluates the management skills of a resident for adult and pediatric patients in the outpatient, perioperative, and emergency settings and can be used as part of the Milestones 2.0 biannual evaluations. The same electronic platform is used for the ONE Tool as for the other ABOS assessment tools, and it is still in its development phase.

The Role of Surgical Simulation

The ABOS also recognizes that surgical simulation is an important key to acquiring basic surgical skills and should have a role in achieving competency. In 2013, the ABOS promoted surgical skills training by requiring PGY 1 skills laboratory training with an associated curriculum and assessments. The American Board of Surgery (ABS) uses the Fundamentals of Laparoscopic Surgery¹⁸ surgical skills simulation test as a requirement for board certification. To investigate the options for similar simulation testing in orthopaedic surgery, the ABOS convened a Surgical Simulation Workshop in August 2016 with multiple invited guests, including members of the ABS. This workshop identified that orthopaedic surgery would benefit from low-cost simulations with proficiency benchmarks for common orthopaedic surgical skills. To this end, the ABOS collaborated with the Orthopaedic Research and Education Foundation in 2018 and 2019 to award 3 research grants for the development of suitable simulations for basic orthopaedic skills and to demonstrate their value in benchmarking skills competency, with results scheduled to be realized in 2022. The results of these research projects will provide important ongoing information because the future role of surgical simulation in residency education continues to be evaluated.

Professional Behavior Assessment Tool Development

In 2018, the ABOS embarked on the development of the ABOS Behavior Assessment Tool using the same online platform for securely requesting assessments as was used for Surgical Skills Assessments. The ABOS designed and implemented an assessment tool incorporating assessment of 5 domains of professional behavior (Table X). For each of the 5 professional behavior domains, evaluators are asked to rate the residents by the scale, strongly disagree (1), disagree (2), neutral (3), agree (4), or strongly agree (5). Through collaboration between the ABOS and AOA/CORD, 18 residency programs piloted the use of the behavioral tool. Residents requested assessments from faculty at the end of their clinical rotations, and a 360-degree request was performed near the end of the academic year. For external validation, program directors rated resident professionalism before the trial for comparison to the behavioral tool results. The results were published in 2020¹⁹ describing 9,892 evaluations, completed for 449 different residents by 1,012 evaluators. 97.7% of all evaluations were scored level 4 or 5 (agree and strongly agree) across all the 5 domains. 2.4% of all evaluations scored level 3 or below reflecting poorer performance and an opportunity for improvement. Evaluators found the ABOS Behavior Tool easy to use (96%) and that it was an effective tool to assess resident professional behavior (81%). The ABOS Behavior Tool was externally validated and was able to identify that about 3% of the residents are considered low performers or “outliers,” who could be counseled regarding unprofessional behavior while in residency training. The 5-domain construct makes it an effective actionable tool that can be used to help develop performance improvement plans for residents¹⁹.

TABLE X.

Description of the ABOS Behavioral Tool

Professional Domain	Descriptors
1. The resident adheres to ethical principles	Demonstrates honesty and integrity (i.e., worthy of the trust bestowed on us by the patients and the publics good faith, reports and analyzes medical errors, maintains confidentiality, understands their scope of practice with appropriate use of knowledge and skills, and trustworthy) Exhibits ethical behavior in professional code of conduct (i.e., student recognizes that being an orthopaedic surgeon is a “way of life” that serves the patient and community, advocates in the best interest of the patient, goes “above and beyond,” they “do the right thing,” respects diverse patient populations, including but not limited to diversity in sex, age, culture, race, religion, disabilities, and sexual orientation)
2. The resident communicates effectively with patients and with people who are important to those patients	Shows compassion/empathy (i.e., collaborates with the patient and enhances the relationship) Demonstrates communication and listening skills (i.e., attentive, shows patience, respects patient autonomy and empowers them to make informed decisions, and manages communication challenges with patients and families) Shows respect for patient needs (i.e., respects patients' viewpoints and considers his/her opinions when determining healthcare decisions, regards the patient as a unique individual, treats the patient in the context of his/her family and social environment, and takes time to educate the patient and his/her family)
3. The resident effectively interacts with other people working within the health system	Shows ability to work with faculty, peers, and medical students (i.e., shows respect, supports faculty mission to provide quality patient care, works collaboratively, can work with a team and cares for other members of the team, able to resolve conflicts effectively, adapts to change, and creates effective personal interactions) Students' level of composure (i.e., ability to handle difficult situations with ease, has good coping strategies, and manages stress well) Students' identity formation (i.e., ability to “fit in” with their role as a student learner, shows maturity in their specific role as a student physician learner and socialized to the medical environment)
4. The resident is reliable	Work ethic (i.e., shows interest and available, protects patients' interests, driven, willingness to conduct patient care without prompting, and committed to maintaining quality of care) Punctuality (i.e., shows to clinic, OR, conferences, and call cases on time) Level of responsibility/accountability (i.e., ability of the resident to answer for his/her conduct, timely completion of medical records or other required tasks, acknowledges his/her limitations, strives for excellence, show pride in his/her actions and thoroughness, and level of confidence that a task will be performed)
5. The resident is committed to autonomous maintenance and continuous improvement of competence in self, others, and systems	Students' ability to self-assess (i.e., the resident recognizes his/her limits; ability to self-reflect and hold themselves accountable; commits to life-long learning; identifies strengths, deficiencies, and limits in one's knowledge and expertise; and personal responsibility to maintain emotional, physical, and mental health) Students' receptiveness to critique (i.e., the resident responds to feedback by accepting criticism, looks at self objectively, and changes his/her actions)

ABOS = American Board of Orthopaedic Surgery, and OR = operating room.

Collaboration with ACGME Milestones 2.0

To provide a more unified system of resident competency assessment, the ABOS has had representation on the ACGME Milestones 2.0 development group. Assessments are needed for both certification of the individual resident (ABOS) and accreditation of the residency program (ACGME). The 2 organizations are working to collaborate so that the ABOS Surgical Skills, ONE Tool, and Professional Behavior assessments can be used to meet both requirements. For example, the 5 domains of the ABOS Behavior assessment can be used not only as part of the ABOS assessment of the individual resident's performance but also as part of the Milestones requirements listed in Figure 1. Furthermore, in Milestones 2.0^20,21, Patient Care Operative Management Milestones can be assessed in the ABOS Tier 1 Core Procedures (Table VIII) and the ABOS Tier 2 Subspecialty Procedures (Table IX). The dual reporting usage of both the Behavior and Surgical Skills Assessment Tools is designed to minimize the burden of reporting.

Fig. 1.

Mapping of the ABOS Behavior Tool results as an assessment for the Milestones 2.0 requirements. ABOS = American Board of Orthopaedic Surgery.

Future Plans

As of 2022, the ABOS has enrolled 46 residency programs in the ABOS KSB Program, including both allopathic and osteopathic programs from varied geographic regions and varied sizes^12,19 with rollout to additional residency programs planned. The use of nationally required assessments will lead to national benchmarks to determine standardized acceptable levels of competence for knowledge, patient care skills, and professional behaviors. The vision is that completion of residency would include not only a time-based requirement of 60 months but also a competency-based requirement for knowledge, patient care skills, surgical skills, and professional behavior. By setting the bar of expectation for knowledge, skills, and behaviors during residency training, a framework is provided to guide educational goals. For example, one resident may score extremely high on the OITE (Knowledge) but have very poor surgical technique (Skills). Another resident may be highly liked with excellent communication skills and patient rapport (Behavior) but lack significantly on his/her medical comprehension (Knowledge). Finally, a resident may have incredible “surgical hands” (Skills) but exhibit poor bedside manners (Behavior). In each of these scenarios, strengths and weaknesses of resident performance can be assessed and addressed, during residency training. Identifying strengths and weaknesses in resident performance can help focus the education program while the resident is in the resident program learning environment. In addition, it will be possible to set passing standards that will be necessary for residents to achieve before sitting for the ABOS Part I Examination, leading to important steps toward competency-based education. Communication between residency PDs and fellowship directors, using a nationally standardized ABOS KSB format, may also enhance the focus during fellowship training and, ultimately, the transition to independent practice.

There are many challenges to institute the ABOS KSB project nationally. The collaborations that have begun between multiple national orthopaedic and education organizations will need to continue and progress. Big data will need to be collected and analyzed. Passing standards cannot be set without this type of rigor. Training of the orthopaedic faculty will be necessary to validate that the assessments are performed according to set standards. Programs must avoid increasing the assessment burden on residents and faculty by collaborating with ACGME and orthopaedic residency programs to prevent multiple parallel systems. Providing a web-based system that remains easy to access, simple to complete, and meaningful to the resident and faculty as the trainee progresses through the education program will be essential. Currently, most faculty fill out assessments related to resident performance. These assessments can be collated using data security systems within the secured ABOS national database with defined rules for access and housed similarly to other educational and performance assessments. The ABOS will keep the financial burden low by underwriting the costs of the ABOS KSB assessment system and an ABOS Resident Dashboard.

Conclusions

The ABOS has set a goal to improve U.S. Orthopaedic Resident Education by enhanced assessment of competency. Achieving this goal requires identification of what residents must achieve to begin the independent practice of orthopaedic surgery in 3 areas: knowledge, skills, and professional behavior. The ABOS, in collaboration with the AAOS and AOA/CORD, has already made important steps to make this a reality. We envision a hybrid of time and competency-based orthopaedic residency education that will remain 5 years in length, with residents being assessed using tools that have nationally accepted standards. These assessments will guide their progress through residency education and better define that they have achieved a required competence level by the end of their training. The ABOS has achieved some steps toward this vision, with more work ahead.