Development, Implementation, and Meta-Evaluation of a National Approach to Programmatic Assessment in Canadian Family Medicine Residency Training

Abstract

The growing international adoption of competency-based medical education has created a desire for descriptions of innovative assessment approaches that generate appropriate and sufficient information to allow for informed, defensible decisions about learner progress. In this article, the authors provide an overview of the development and implementation of the approach to programmatic assessment in postgraduate family medicine training programs in Canada, called Continuous Reflective Assessment for Training (CRAFT). CRAFT is a principles-guided, high-level approach to workplace-based assessment that was intentionally designed to be adaptable to local contexts, including size of program, resources available, and structural enablers and barriers. CRAFT has been implemented in all 17 Canadian family medicine residency programs, with each program taking advantage of the high-level nature of the CRAFT guidelines to create bespoke assessment processes and tools appropriate for their local contexts. Similarities and differences in CRAFT implementation between 5 different family medicine residency training programs, representing both English- and French-language programs from both Western and Eastern Canada, are described. Despite the intentional flexibility of the CRAFT guidelines, notable similarities in assessment processes and procedures across the 5 programs were seen. A meta-evaluation of findings from programs that have published evaluation information supports the value of CRAFT as an effective approach to programmatic assessment. While CRAFT is currently in place in family medicine residency programs in Canada, given its adaptability to different contexts as well as promising evaluation data, the CRAFT approach shows promise for application in other training environments.

As competency-based medical education (CBME) becomes more widely adopted for health professions education training around the world, there is a growing need for examples of implemented CBME curricula and assessment programs. Specifically, there is a desire for descriptions of innovative approaches to assessment that are generating appropriate and sufficient information to allow programs to make informed, defensible decisions about learner progress. Meeting this need is crucial: to justify the CBME paradigm shift away from traditional approaches to training, it is necessary to show that CBME programs are doing a better job of teaching and assessing the competence of learners compared with previous approaches. ^1–4

In this article, we describe the development and implementation of CBME in family medicine postgraduate training in Canada, with an emphasis on the approach to programmatic assessment. We present a meta-evaluation of findings that support the approach taken in Canadian family medicine training, as well as offer some insights into lessons learned.

Context

In 2010, the College of Family Physicians of Canada (CFPC) introduced CBME to family medicine residency training with the Triple C competency-based curriculum (Triple C). ⁵ Triple C establishes the curriculum and assessment standards for all family medicine postgraduate programs in Canada and is the basis for family medicine specialty–specific accreditation requirements. ⁶ Postgraduate (residency) training in Canada is a 24-month program that follows completion of an undergraduate medical degree (e.g., MD). See Supplemental Digital Appendix 1, at http://links.lww.com/ACADMED/B281, for a brief description of both family medicine training in Canada and the Triple C curriculum.

The assessment of competence as guided by Triple C is outlined in the Continuous Reflective Assessment for Training (CRAFT) document, ⁷ which presents high-level guidelines for competency-based teaching and assessment. In the next sections, we present details of how CRAFT was developed, what it looks like, and how it was implemented.

Development of CRAFT

The CRAFT approach to programmatic assessment is grounded in published evidence from experts in learning and assessment. ⁷ The developers of CRAFT took a constructivist approach to designing a system of assessment that employs nonstandardized tools alongside defined approaches, with a focus on workplace-based assessment. In developing CRAFT, the working group was guided by the following beliefs: the need to consider the context of the assessment, ⁸ recognition of the subjective nature of assessors, ⁹ and the importance of ensuring that multiple observers contribute to assessment because there is value in incorporating a variety of valid viewpoints ¹⁰ into workplace-based assessment. ^11–13 Along with assessment of learning, there is the explicit incorporation of the principles of assessment for learning, ¹⁴ guided self-assessment, ¹⁵ and assessment as learning. ¹⁶ While assessment for learning has become widely recognized as a necessary consideration in designing CBME programs, ¹¹ CRAFT was developed to incorporate assessment as learning through the explicit requirement for reflection (the R in CRAFT) as part of developing accurate self-assessment skills in learners. Coaching learners to engage in guided self-assessment ^17–19 begins early in postgraduate medical training, and the CRAFT process intentionally engages learners in regular active reflection of both low-stakes (formative) and high-stakes (summative) assessment information.

The goals of CRAFT are twofold:

• Delineate for training programs the requirements of a context-appropriate learning and assessment framework that will ensure that rigorous and defensible assessment leads to graduating competent physicians; and

• Instill in those physicians the skills that they will need to maintain and enhance their competence while in practice, through fostering and promoting self-assessment skills and adaptive approaches to identifying and remedying gaps in knowledge or skills and safely applying that new knowledge to patient care.

The end result is an approach to programmatic assessment in residency training that incorporates multiple assessment elements to allow for informed decision making about each learner’s progress in demonstrating competence. The CRAFT model provides a high-level approach to combining learning and assessment activities in an intentional way to design a programmatic assessment system that can be adapted to local contexts, including size of program, resources available, and structural enablers and barriers. The CRAFT framework aligns with the 5 dimensions of programmatic assessment described in the model proposed by van der Vleuten and colleagues in 2012. ¹³

What Is CRAFT?

The CRAFT model encompasses both the “what” and the “how” of programmatic assessment in a learner-centered approach. The “what” of CRAFT clearly defines what should be assessed in the 2 years of family medicine residency training, across the CanMEDS-FM roles and the CFPC’s essential skill dimensions. ²⁰ The competencies to be assessed are captured in the assessment objectives for family medicine. ^21,22 The “how” of CRAFT has 2 components: the structure of the learning and assessment environment and a high-level model of programmatic assessment that delineates the activities and processes that must occur in each residency program. A graphic depiction of the CRAFT model of continuous and recursive programmatic assessment is shown in Figure 1. While all residency programs must design programmatic assessment frameworks that align with the CRAFT model, there is considerable flexibility to allow each program to include specific approaches and tools, assign roles, and delineate processes that are appropriate for their local context, including program structure, location, and resources.

Figure 1.

Graphic depiction of the recursive flow of the CRAFT programmatic assessment model. The main phases of the programmatic assessment model are in the middle of the circle, while the activities and tasks associated with each phase are in the outside arrows. The activities and tasks in the outside arrows happen collaboratively between the learner and the teachers who work with the learner. All decisions from the Decide phase are informed by the Assess phase, and feed into the next stage of the Learn phase, in a continuously iterative cycle of programmatic assessment. Abbreviation: CRAFT, Continuous Reflective Assessment for Training.

The first aspect of the “how” of CRAFT requires that the learning and assessment environment is structured to ensure continuity of supervision for every learner. Each incoming learner is matched to a continuous advisor, similar to the structure of master’s or doctoral graduate programs. As a result, there is continuity of supervision to monitor development and demonstration of competence over time, as the advisor has access to all assessment information about the learner from beginning to end of residency training. Whether the advisor has a direct assessor role or is a nonassessing mentor varies between training programs, according to local context and structure. The continuous advisor acts as a coach for the learner, guiding the learner along the trajectory of development of competence. Learners use their own assessment data collected across multiple contexts and from multiple observers to reflect on their progress, strengths, and gaps. The learner’s reflections are discussed with the advisor, who offers suggestions and guidance as the learner engages in this guided self-assessment process and helps the learner to course correct when their interpretation of their own progress does not align with the assessment evidence.

The bespoke aspect of CRAFT (i.e., a high-level national guideline that allows for a programmatic assessment design that is suitably tailored to a specific local program context) is best understood by considering a key resource of CRAFT, the in-training assessment (ITA) template ⁷ (Table 1). For each step or task in the ITA template, a description of each activity is presented, and the role of the learner is usually described. However, the ITA template is intended to offer high-level guidelines, rather than to be a prescriptive set of rules; as a result, the ITA template specifies program-specific elements to allow for local decisions about what assessment roles will look like. It should be noted, however, that accreditation standards require that any local, program-specific assessment elements must align with the overall CRAFT guidelines and must adhere to the phases of the CRAFT model of programmatic assessment. Further, graduation from residency training should not occur until a learner has successfully demonstrated competence in all required competencies. At that point, the program director attests that the learner has completed residency training and may write the certification examination (a national requirement).

Table 1.

The In-Training Assessment Template^a Aligned With the Phases and Elements of the CRAFT Model

As shown in Figure 1, there are 3 main phases to CRAFT (learn, assess, decide), and each phase has specific elements such as observe, adjust, and adapt. In Table 1, we show how the phases of the CRAFT framework align with the tasks and steps of the ITA template.

One innocuous word encapsulates a key component of CRAFT: the “adapt” element of the decide component (Figure 1). In this context, adapt refers specifically to adjustments to training for a learner, based on assessment evidence as the learner progresses toward competence. Most often, it is assumed to refer to modifications to a learner’s training program as a result of gaps identified (i.e., need for a period of training focused on addressing gaps informally, or a formal remediation contract). However, adapting is also used to modify a learner’s training program when the learner demonstrates expected competence before the end of a scheduled rotation or even before the end of the 24 months of family medicine residency training. In these cases, the learner’s program is modified to allow the learner to focus on getting extended or enhanced clinical experience in areas aligned with their career goals, or to master the clinical competencies beyond the level of competence that the program expects of learners for graduation.

The rationale for adjusting training for highly competent learners rather than allowing for early completion of training is simple: family medicine residency training in Canada is only 24 months long, arguably the shortest postgraduate training program in the world. Becoming a competent physician requires more than demonstrating competence in specific areas at specific intervals; rather, competence must be demonstrated repeatedly and consistently over time and across different contexts. Further, the time that learners spend in postgraduate training allows them to gain experience in a safe environment before they begin unsupervised practice, and contributes to their identity formation as physicians. The flexibility of the adapt component of CRAFT allows for learners to experience individualized training to meet their needs, either through enhanced learning experiences within the minimum time of training (24 months) or through modifications to training to address gaps (including the potential for extension of training beyond 24 months). The one exception for shortening training is the case of transfer learners (learners who transfer out of another specialty residency program and into a family medicine residency program). Transfer learners may undertake a family medicine residency of less than 24 months if they have already demonstrated expected competence in one or more areas relevant to family medicine, and the program thus decides that they do not need to complete the associated rotational learning experiences.

Implementation

Implementation of CRAFT occurred between 2010 and 2016. The year of implementation varied across the 17 family medicine residency programs in Canada and depended on local resources and contexts. Each program took advantage of the high-level nature of CRAFT and the ITA guidelines to adapt processes and tools that were appropriate for local contexts. ^23–27 An overview of the similarities and differences in CRAFT implementation between 5 different family medicine residency training programs can be seen in Chart 1; detailed information is available in Supplemental Digital Appendix 2, at http://links.lww.com/ACADMED/B281. These 5 programs represent both English- and French-language programs from Western and Eastern Canada.

What is notable about CRAFT implementation across the 5 programs shown is how many similarities there are in processes and procedures. This is despite the intentional flexibility of the CRAFT guidelines, and suggests that allowing programs to adapt processes to suit their local context may result in more uniformity than might be expected. There are 2 major areas where local differences are seen: in the overall structure and nomenclature of local administrative oversight of each program (found in the Description column of Supplemental Digital Appendix 2, at http://links.lww.com/ACADMED/B281), and in the assessment tools and competence frameworks used by each program (found in the first step of the ITA step/tasks).

The remarkable success and speed of implementation of CRAFT in all family medicine training programs across 17 schools and in 2 official languages was likely facilitated by the flexibility inherent in the CRAFT guidelines. This likely reduced barriers to implementation that can result when program leaders and educators feel that change is being imposed on them, rather than feeling like they have autonomy to make change at their own pace and in a way that works for them and their program.

The other aspect of implementation of CRAFT that contributed to its success was the approach taken by the CFPC. CRAFT was introduced gradually through workshops at conferences and meetings with assessment directors and program directors across the country. These workshops and meetings allowed early adopters to share their successes and failures in venues that allowed for frank discussion of lessons learned. The lessons of early adopters were also openly discussed at the CFPC working groups that contributed to the development of CRAFT, and changes were incorporated into the ongoing development of the CRAFT guidelines. This incremental and iterative approach to implementation of a national guideline meant that by the time CRAFT was formally introduced, many programs had shared local innovations, and other programs had adapted those innovations for their own contexts. While there were, and continue to be, problems encountered by programs as they fully transform their programs to be competency-based, ²³ openly sharing ideas and solutions between residency programs eased the transition significantly.

Chart 1.

Overview of Similarities and Differences in CRAFT Implementation for 5 Different Family Medicine Residency Training Programs in Canada, Using the In-Training Assessment Tasks as Illustration^a

Meta-Evaluation of CRAFT

Evaluation data for CRAFT come from multiple sources, making a meta-evaluation ²⁸ an appropriate approach. Using a theory-led, outcomes-focused program evaluation framework, our meta-evaluation focused on the assumptions of CRAFT using 2 sources of data: an institutional audit of assessment information and processes in 4 programs that were early adopters of CRAFT (Table 2), and published evaluation data (Appendix 1). In discussing published evaluation findings, it is important to note that in many publications, authors tend to use the local name for their assessment framework (e.g., Portfolio Assessment Support System ²⁶ and Competency-Based Achievement System [CBAS] ²⁷) rather than the CRAFT guidelines language, or they refer to the Triple C.

Table 2.

Institutional Audit Evaluation Data From 4 Canadian Family Medicine Residency Programs^a

The published evaluation data for the meta-evaluation came from an existing repository of 13 publications (including a white paper and a conference presentation) related to Triple C or CRAFT that the author group created. One author (S.R.) then searched MEDLINE, EMBASE, and PsycInfo with a time filter of 2010–2022 for other publications, using the keywords CBME, competency-based, assessment, family medicine, and Canada. Inclusion criteria were articles about Triple C and/or CRAFT in Canadian family medicine residency programs that included evaluation data; exclusion criteria were reviews, program descriptions without evaluation data, and articles about other specialties or undergraduate learners. The same author (S.R.) scanned the titles and abstracts of the 35 results and identified 2 more publications for inclusion that were not in the original repository. The data for the institutional audit were provided by program faculty who were or are directly involved in assessment in each program, and who are also co-authors of this article (S.R., K.L., K.S., T.v.d.G., M.P.). The data were analyzed using qualitative content analysis to summarize the existing evaluation findings and align those findings with 3 key assumptions of CRAFT. This project was deemed exempt from need for ethics review based on the ARECCI screening tool (A pRoject Ethics Community Consensus Initiative).

Assumption 1

Learners are matched to a continuous faculty advisor, giving each learner a dedicated advisor to act as a competency coach. This advisor helps guide the learner to monitor their learning and progress over time, and to support the learner with coaching and guided self-assessment.

Evaluation evidence to support assumption 1 included 4 publications reporting evidence that supported the value of each learner having a continuous advisor to coach and support the learner, including improved skills in reflection and self-assessment, ^29,30 significant improvements in addressing identified behavioral and/or learning concerns, ²⁷ and resident self-reports of increased confidence in identifying personal learning needs. ³¹ Multiple publications reported associations between the advisor–resident relationship and the identification of residents in difficulty. ^23,24,27,31 At a national level, Oandasan and colleagues reported on findings from a longitudinal self-report survey conducted with learners in family medicine training programs in Canada. In comparing responses from learners at the start of training and at the end of training, a significant difference was seen in learners’ self-reported ability to identify their own learning needs, ³² and residents’ self-reported feelings of preparedness for practice were significantly associated with agreement that the resident had “an identified person guiding my development.” ³³ Finally, indirect evidence for assumption 1 was provided in a publication in which the authors associated self-reported mastery mindset with the coaching and guided self-assessment experienced by a cohort of residents who graduated from a CBAS residency program. ³⁴

Assumption 2

Assessment data (especially formative) should come from multiple observers across multiple contexts to give an accurate picture of the strengths and gaps of each learner. As much as possible, assessment data should be based on direct observation in the clinical workplace and should be transparently communicated with the learner through feedback and documentation to facilitate and support learning.

Evidence for assumption 2 addressed the value of assessment data from multiple observers and the increase in direct observation and opportunities for feedback. Ross and colleagues reported significant increases in early detection and remediation of learners in difficulty when comparing data from before and after implementation of CBAS. ²⁷ Other publications also included evaluation evidence for increased identification of residents who were struggling. ^23,31,35 This finding has also been anecdotally observed (but not formally evaluated) in other programs across Canada (Table 2). Multiple authors reported evidence for increases in feedback to support learning. ^{23–26,35–37} Griffiths and colleagues found that regular feedback had become part of the culture of assessment in their program, ²⁹ a finding also reported by Ross and colleagues. ²⁵ In a longitudinal national survey of graduating residents from 2015 to 2019, over 85% of residents in each cohort self-reported agreement with the statement “I had many informal opportunities for feedback.” ^32,33 However, it should be noted that in all of the publications cited here except the longitudinal survey, ^32,33 a need for more faculty development was also reported.

Assumption 3

The CRAFT guidelines are intentionally written to be high level, rather than specific and concrete. This is so that programs can design and implement programmatic assessment systems that align with the guidelines, but use tools and processes that are specifically appropriate for each individual program’s resources and context.

Schultz and colleagues reported on specific facilitating factors that aided the change management needed for successful implementation of their local iteration of CRAFT. ^26,31 Two studies that examined data about Triple C implementation included specific data regarding CRAFT. In the evaluation worked example by Hamza and colleagues, the authors identified the high-level guidelines for curriculum and assessment as facilitators of successful implementation, partly by contributing to individual programs’ sense of autonomy and ownership. ³⁸ A multi-institution realist study by Ellaway and colleagues also found that the high-level approach rather than proscriptive directions facilitated uptake by allowing programs to adapt and adopt processes and tools for their local contexts. ²³ However, Ellaway and colleagues did identify that the wide variation among programs may be problematic.

An overview of specific evaluation data from 4 early-adopter programs (Table 2) shows since implementation of CRAFT, there is consistency across programs in the numbers of progress reviews: now, all programs require a summative progress review meeting and completion of a progress review report every 4 months. Before implementation of CRAFT, progress reviews occurred less frequently in all programs. Further, implementation of CRAFT also resulted in the introduction of learning plans in all programs. In looking at the amount and type of assessment data collected about each learner over the 2 years of training, some minor variation is seen. This is because the CRAFT guidelines do not dictate specific numbers for different types of assessments. Rather, programs have the ability to adjust numbers of assessments dependent on each learner’s trajectory in the development of competence. Learners who consistently demonstrate competence do not necessarily need to be assessed (formatively or summatively) as frequently as do learners who are encountering difficulty. This allowance for learner-centered adjustments to assessment is a result of the emphasis on learning that is embedded in the development of CRAFT.

While the findings from our meta-evaluation support the belief that CRAFT is having a beneficial effect on family medicine residency training, more findings are needed from large-scale studies across multiple programs, as well as long-term outcomes of graduates in practice.

Limitations of the meta-evaluation

A limitation of this meta-evaluation is that while individual programs carry out program evaluation, very few of those program evaluation findings are published. The findings from these projects have been shared among family medicine program directors and assessment directors at the CFPC’s Family Medicine Forum conference, including at the assessment directors’ meeting held during the Family Medicine Forum, but minimal findings have been disseminated more broadly. A further limitation is that few programs are able to compare outcomes using pre- versus postimplementation data for CRAFT, primarily due to how long CRAFT has been in place (over 10 years in many programs). While some authors have been able to conduct and publish comparison studies (see Ross and colleagues ²⁷), many programs either do not retain assessment data going back more than a decade, or do not have local capacity or expertise to conduct secondary data analysis of archived assessment data.

CRAFT in Other Contexts

While the CRAFT guidelines are intended to inform design of programmatic assessment in family medicine residency training, the key assumptions of CRAFT are based in educational psychology theory and learning principles, making them transferrable to multiple contexts. Further, CRAFT is not tied to a specific competency framework, nor does it dictate specific tools, so it can be used in any specialty or health care profession and within any educational context. For example, CRAFT could guide programmatic assessment in undergraduate medical education. Students entering medical school would be assigned to a continuous advisor for the duration of medical school. This continuous advisor would form a relationship or educational alliance ³⁹ with the student, and would have access to all assessment data about the student. The continuous advisor and the student would have regularly scheduled meetings to discuss the student’s progress, and the advisor would support and guide the student in reflection and guided self-assessment and be responsible for contributing an informed judgment of the student’s progress. Assessment information in medical school could expand beyond traditional summative assessments and examinations to include multiple formative assessments based on direct observation that would facilitate feedback conversations, as was seen by Griffiths and colleagues 3 years after their institution’s implementation of CRAFT. ²⁹ While this adoption of CRAFT would certainly take work, it would be a transformative way to approach undergraduate medical education.

Similarly, CRAFT could be used in almost any specialty—with the same caveat as above about the work required to introduce this model of programmatic assessment. The easy part would be identifying tools to use for assessment: many specialty programs use entrustable professional activities for assessment, which are already in use in some Canadian family medicine residency programs within the CRAFT model. ³⁵ The most challenging part of implementation would be revising established institutional structures to increase direct observation of learners, and introducing regular formative assessment with accompanying feedback conversations.

Concluding Remarks

CRAFT offers a framework for programs to design learning environments and assessment processes that are intended to support the development of skills needed for lifelong learning—especially self-assessment and self-monitoring of the safe application of new skills and knowledge. Further evaluation data are needed to build upon existing findings.

While CRAFT is currently in place in family medicine residency programs in Canada, this model can be adapted for application in other training environments. The high-level guidelines of CRAFT and the ITA template allow for adaptation to local contexts and resources in settings beyond family medicine residency training in Canada.

Acknowledgments:

The authors wish to thank Tatjana Lozanovska for her support in writing this article.

Supplementary Material

acm-98-188-s001.pdf ^{(260.3KB, pdf)}

Appendix 1 Publications Sharing Evaluation Data About CRAFT Implementation in Family Medicine Residency Programs in Canada Used for the Meta-Evaluation of CRAFT