Ten ways to get a grip on designing and implementing a competency-based medical education training program

Abstract

Background

Globally there is a move to adopt competency-based medical education (CBME) at all levels of the medical training system. Implementation of a complex intervention such as CBME represents a marked paradigm shift involving multiple stakeholders.

Methods

This article aims to share tips, based on review of the available literature and the authors’ experiences, that may help educators implementing CBME to more easily navigate this major undertaking and avoid “black ice” pitfalls that educators may encounter.

Results

Careful planning prior to, during and post implementation will help programs transition successfully to CBME. Involvement of key stakeholders, such as trainees, teaching faculty, residency training committee members, and the program administrator, prior to and throughout implementation of CBME is critical. Careful and selective choice of key design elements including Entrustable Professional Activities, assessments and appropriate use of direct observation will enhance successful uptake of CBME. Pilot testing may help engage faculty and learners and identify logistical issues that may hinder implementation. Academic advisors, use of curriculum maps, and identifying and leveraging local resources may help facilitate implementation. Planned evaluation of CBME is important to ensure choices made during the design and implementation of CBME result in the desired outcomes.

Conclusion

Although the transition to CBME is challenging, successful implementation can be facilitated by careful design and strategic planning.

Introduction

Medical training programs worldwide are transitioning to competency-based medical education (CBME).^1,2 The key principles of CBME include explicit delineation of expected knowledge and skills, focusing on observable outcomes, de-emphasizing time-based training, and ensuring more frequent formative assessments.³ Key components common to CBME include Entrustable Professional Activities (EPAs) and milestones (Table 1). An Entrustable Professional Activity (EPA) is a unit of professional work, consisting of knowledge, skills and attitudes (competencies) that can be entrusted to a learner for unsupervised execution.⁴ Milestones represent developmental steps that mark progression in a competency which learners must demonstrate as they progress through training.^1,5 Regular review of trainees’ progress during training is monitored and adjudicated by a competence committee to determine whether competency within each EPA has been achieved.

Table 1.

Definitions and examples of key terms in Competency-Based Medical Education

Term	Definition	Example
Entrustable Professional Activity	Unit of professional work Consists of competencies that can be entrusted to a learner for unsupervised practice	Perform a consultation Deliver serious news to a patient
Competency	Knowledge, skills and attitudes	Take a patient history
Milestone	Developmental steps that mark progression in a competency	Describe the elements of a history ⬇ Gather relevant information from a patient encounter ⬇ Actively listen and respond to patient cues ⬇ Use patient-centred interviewing skills to gather a relevant patient history

^*Modified from the Royal College of Physicians and Surgeons CanMEDS Milestones guide

CBME has been implemented in many countries and educational systems. In Canada, CBME has been implemented by the Royal College of Physicians and Surgeons of Canada in the form of Competence-by-Design (CBD), in which trainees progress through four stages of training (Transition to Discipline, Foundations, Core, Transition to Practice).⁶ Each stage of training includes specific EPAs, which trainees must achieve prior to progressing to the next stage. In Canada, all training programs in a specialty transition to CBD simultaneously and have a commonly agreed upon set of EPAs and milestones. Currently, there have been four cohorts (28 of 39 specialties) which have transitioned to CBD as of July 2020.

Implementation of a complex intervention such as CBME represents a marked paradigm shift involving multiple programs and stakeholders, including trainees, teaching faculty, residency training program committee members, and program administrators. Although the key concepts and elements of CBME have been defined, national specialty committees and individual programs are given the responsibility of determining how best to adapt and implement CBME in their specific environments and residency programs. The road to implementation can be bumpy and educators must be prepared to navigate the visible “potholes” as well as unexpected “black ice.” Common “black ice” issues that educators may encounter include: skepticism regarding need for change and potential benefits of CBME to trainee education; overly ambitious requirements for direct observation and complex assessment plans; limitations in resources and time for implementation of change; and feasibility of integration into daily clinic flow. Careful preparation enables educators to avoid and manage unexpected challenges and help facilitate a smoother journey to the destination.

As members of the medical oncology national committee, our group of medical oncology residency program directors and educators were early adopters of CBME in Canada, one of the first non-surgical specialties and just the second cohort in Canada. Through a series of workshops, the group developed subspecialty specific EPAs and milestones and worked to get a grip on designing and implementing CBME. In this article, we provide ten ways to get a grip on designing and implementing a CBME training program, sharing our collective experiences, successes and challenges to help other specialties avoid common patches of “black ice” as they transition to CBME.

No Research and Ethics Board approval was required.

Pre-implementation/design phase of CBME

Recommendations 1-4

1. Encourage broad representation and processes for early and ongoing feedback

Successful implementation of CBME requires the identification and engagement of stakeholders early in the process and should involve faculty, learners, and relevant members from related disciplines. Early engagement should educate stakeholders about CBME and the rationale for change.^{7, 8} Input should be sought to ensure their concerns are heard and that the final CBME product resonates with stakeholders. Educators should recognize that buy-in is an ongoing process rather than a one-time check-box in the process of change management. It is an integral part of the implementation process that evolves as change is internalized by the users and stakeholders. Ongoing efforts to engage stakeholders is important and buy-in can be said to have been reached when the point of ownership is reached; often times this is when the introduced change takes on a form that is relevant for the stakeholder’s context.

2. Start with the key concepts in the design process

There are many potential approaches for developing a CBME framework. Our group of program directors and educators started by selecting competencies that we felt were essential for oncology trainees to acquire. EPAs were subsequently defined and developed and competencies were matched to specific EPAs. This resulted in a large number of competencies mapped to each EPA, which were too numerous to feasibly assess and required further selection. This also resulted in a number of “orphan” competencies which were not encompassed or assessed within any of the EPAs.

Based on our experience, we recommend developing EPAs first by determining the most significant tasks of the specialty that reflect professional practice. This ensures that the EPAs included will resonate with clinicians and learners. Furthermore, key competencies for the execution of each EPA can then be identified allowing selection of only those most critical to the EPA to be assessed.

3. Design for success - Develop an assessment plan that is realistic, practical and achievable

Development of the assessment plan should be done early during the design process to ensure it is integrated with other elements of CBME design and implementation. When building an assessment plan, multiple components need to be considered. These components include:

1. Selection of what should be assessed;

2. Choice of appropriate assessment tools;

3. Identification of the minimum acceptable number of observations (including settings, assessors) necessary for entrustment and promotion;

4. A system for capturing assessment elements that is easy to use in the workplace;

5. An assessment platform that collates data in a manner that is helpful to multiple stakeholders including learners and decision makers.

The assessment plan should provide multiple data points to inform entrustment and promotion decisions.⁷ Evidence regarding the optimal number of assessments and data points is currently lacking⁹ and therefore pragmatic approaches are recommended. Careful consideration should be given to “numbers needed” to ensure sufficient data for decision making, feasibility, sustainability and to minimize both assessment fatigue and administrative burden. A community of practice of clinicians involved in medical education may be helpful in gathering consensus.

Assessment tools which have worked in similar contexts and with known reliability, feasibility and validity should be considered.^10,11 Many of the current assessment tools are anchored using a scale based on “expectations” (below to above expectations) which have poor validity and reliability.^12,13 These assessments often provide little discrimination between learners and yield little specific and actionable feedback needed to guide learner development. Assessments should therefore incorporate entrustability scales, which in contrast, are aligned with real-world judgements made by supervisors about when learners are trusted to act without supervision, and can be more meaningful to assessors and learners.¹⁴ Work associated with documentation and tracking of assessments should be kept to a minimum and not detract from learning.

4. Choose where direct observation is needed

Direct observation plays a key role in assessment and feedback in CBME. Observing learners provides valuable information to support formative assessment (feedback for learning) and coaching for professional development.¹⁵ Careful selection of which aspects of learner performance need to be observed is important. In addition consideration of what faculty members can realistically observe in a given period of time is critical to facilitate uptake.¹⁶ Competencies that are essential to the profession should be identified and flagged for direct observation.¹⁷ While our group initially planned to include direct observation for the majority of EPAs, after identifying the core skill common to these EPAs (communication) we chose to limit direct observation to those EPAs that were the most critical and easily observed. This was important in convincing teaching faculty to perform direct observation, a challenging task, given it is not currently engrained in the culture of medical education.

Implementation of CBME

Recommendations 5-10

5. Run a pilot pre-implementation to engage learners and faculty

Field testing can serve several important functions during the implementation of CBME. It can crystallize concepts that have been introduced during faculty development, but may remain abstract and difficult to understand until faculty and learners carry out an assessment. It can also provide insight into practical elements of implementation. Identifying issues related to an electronic assessment platform, the impact on clinical care delivery flow, or to the functioning of the competence committee will allow them to be addressed prior to full scale implementation.¹⁸

6. Engage faculty and learners to help champion CBME

Engaging stakeholders to help implement CBME is important. Identifying faculty and learners who are already engaged can help facilitate communication about CBME, amplify support for implementation and help engage colleagues.⁸ Support from the department chair can also promote participation from other faculty members. Encouraging input from those who have not yet bought into CBME is important to stimulate dialogue and allow change leaders to address concerns.

As programs may experience similar challenges, lessons learned should be shared within institutions to help other programs at various stages of implementation tackle challenges more effectively. This can be facilitated by CBME faculty leads for each academic institution. Programs implementing nationally should establish a mechanism for regular feedback though teleconferences or web-based meetings or boards.

7. Develop and use a curricular map to ensure alignment between EPAs and rotations

Curriculum maps are visual representations of the curriculum and include educational strategies, course content, learning outcomes, educational experiences, assessment, and the individual learner’s personal timetable and progress (Figure 1 and 2).¹⁹ Curricular maps ensure all elements of the curriculum are covered and that rotations align with the required elements for each stage of training.^20,21 For example, an EPA requiring inpatient care can be correctly mapped to a rotation where residents are most likely to execute this type of task.

Figure 1.

“Year at a glance” curriculum map

Figure 2.

Direct curriculum mapping

8. Utilize Academic Advisors to aid in trainee learning and navigation through CBME

Academic Advisors (AA) can play a key role in CBME and are faculty members assigned to guide the learner through the educational program and ensure learners complete EPAs and other required training activities as expected.²² AAs should be involved with regularly reviewing the learner’s assessment portfolio, helping develop individualized learning plans, and preparing for competence committee meetings. AAs also provide coaching and support and engage in deliberate academic mentoring.²³ Mentors on the other hand provide trainees guidance about their career.²² Typically, the AA is a member of the teaching faculty with a demonstrated interest in learner education, a clear understanding of role and responsibility of AA, and familiarity with program, training opportunities and requirements.²² In small faculties they may also serve as a clinical supervisor for some rotations. AAs, unlike mentors, are typically formally assigned by the program and liaise with the program director.

9. Leverage unique opportunities and local resources for assessment

Existing clinical rotations and training experiences within the local institutions can be used to harness unique opportunities for assessment and feedback. For example, learner longitudinal clinics, which are often less busy, can be leveraged as opportunities for direct observation and allow assessment of progression through the stages of training. Objective structured clinical examinations (OSCE) and simulation can be helpful in assessing competencies that occur relatively infrequently in day-to-day clinical practice.²⁴

10. Plan for evaluation of the design and implementation process

Continuous evaluation of a new educational model is important to monitor the downstream intended and unintended consequences of decisions made during the design and implementation process, some of which are made through consensus and for pragmatic reasons. This facilitates an iterative process that allows for changes to be incorporated in response to the real world experience. In nationally coordinated implementation of CBME, planning allows for a cohesive approach to evaluation. Identifying common areas of interest for evaluation and research can allow for rich data collection and enhance the reliability of decision making based on results of the evaluation process.

Conclusion

CBME represents a major shift in the educational paradigm. Implementation of CBME will vary widely depending on the context of the specific training program, training specialty, hospital, university and country. However, the lessons learned and imparted by our group are not limited to medical oncology and may prove valuable to other programs undergoing this transition. Careful design of key foundational components of CBME including EPAs and assessment plans, are critical to the success of the program. In addition, such a broad undertaking requires the engagement of stakeholders not only within the program but also across the university and potentially nationally as well. Thoughtful and coordinated evaluation is important to monitor for and respond to intended and unintended consequences of decisions made during the transition to CBME.

The suggestions outlined are meant to help programs avoid the unexpected “black ice” issues that can arise during the implementation of CBME which will hopefully lead to a less treacherous transition.

Authorship

The authors write on behalf of the Royal College of Physicians and Surgeons of Canada medical oncology Competency By Design evaluation subcommittee

Conflicts of interest

Dr. Tina Hsu has received honorarium from Apobiologix, Celgene, Ipsen, Eisai and Genomic Health. All other co-authors report no conflicts of interest.

Funding

None

Prior presentations

This work has been previously presented at International Association for Medical Education (AMEE) 2018 and the World Competency Based Medical Education Summit 2018.