Setting and Problem
Faculty physicians need to be able to use a variety of tools to complete reliable and valid assessments of learners on the Accreditation Council for Graduate Medical Education (ACGME) competencies. However, many faculty and program directors lack a background in education or assessment, or they lack knowledge of the benefits and drawbacks of different types of assessment instruments. Guided by learning theory, which indicates that learner engagement is a critical component of behavioral change, we sought to actively engage faculty in becoming familiar with a variety of assessment methods, along with their appropriateness for evaluating learners. We adapted a highly interactive faculty development game, “From Madness to Methods,” published in MedEdPORTAL, to accomplish this goal.
Intervention
We performed a literature review of assessment methods in higher education and medicine in order to develop a comprehensive listing of tools. The faculty development game, “Assessment and the ACGME Competencies,” is structured using 2 card decks. One deck is composed of 13 common assessment tools; each lists a description of 1 tool and a detailed list of advantages and disadvantages of that assessment method. The second deck consists of the 6 ACGME competencies (figure; card sets available at https://catalyst.uw.edu/workspace/krpatton/52872), with the cards featuring detailed descriptions of specific behaviors defining that competency.
Figure.
Example of Assessment and Competency Cards
Participants are divided into small groups of 4 to 8 players at each table, and each participant is randomly dealt 6 to 8 assessment tool cards (some cards may be duplicates, in which case it would be up to the player to decide which to use during a turn). One competency card is selected from the second deck and turned up on the table to be shown to the group. In turn, each participant in the group chooses to present 1 of the assessment tools from his or her hand, and strives to argue why that specific tool is the best way to assess that particular competency.
A facilitator describes the rules of the game and circulates among the participants to encourage interaction and discussion. He or she provides reinforcement of the concept that multiple tools are often valuable for both the learner and for the assessment; there is no true “one best method.” Short, focused debates are encouraged by the facilitator. A “winner” for the competency is selected by the entire small group. This is meant to engender an environment of collegial competition. Participants are allowed to attempt to persuade the group of the superiority of a particular assessment tool; the goal is to encourage engagement with the competencies and assessment tools. The format is repeated to review the desired number of competencies. Each competency may take from 10 to 15 minutes to complete, depending on the size of the group. At the end of the small group session, all players reconvene to discuss assessment pearls generated from individual groups.
Outcomes to Date
Participants have reported that the exercise is enjoyable, highly interactive, and useful for evaluating a variety of tools, particularly when considering the intent and outcome of a specific assessment tool. The exercise took 35 minutes to evaluate 3 competencies. The 24 participants were attendees of a faculty development session on assessment and represented a wide range of medical specialties. Participants completed a “1-minute written feedback” survey, and 75% independently found the exercise “useful” or “very useful.” Other themes in the collected feedback included “useful for thinking about improving teaching”; “useful for improving choice of assessment tools”; “helped to analyze the effectiveness of particular assessment tools”; “good for allowing application of the assessment tools to “real life”; and “I enjoyed the wisdom of colleagues.” The game is highly modifiable, as other measurable, specialty-specific behaviors can be easily substituted for a particular competency, and additional assessment outcomes can be used.
The Assessment and the ACGME Competencies game provided an engaging and interactive way to provide comparisons of multiple assessment tools in the context of the ACGME competency outcomes in a short period of faculty development time.