Launching the Next Steps to Improve Hospice and Palliative Medicine Fellow Performance Assessment: A Look Back to the Initial Toolkit of Assessment Methods

Abstract

Education leaders in hospice and palliative medicine (HPM) have long acknowledged the challenge of fellow performance assessment and the need for HPM-specific fellow assessment tools. In 2010, and in alignment with the Accreditation Council for Graduate Medical Education's (ACGME's) directive toward competency-based medical education, the national HPM Competencies Workgroup curated a set of assessment tools, the HPM Toolkit of Assessment Methods. The Toolkit has been a resource for HPM fellowship directors in evolving practical, multifaceted fellow assessment strategies. Now, as American Academy of Hospice and Palliative Medicine plans for a national workgroup in 2020 to define current HPM fellow assessment methods and to propose strategies to strengthen and standardize future assessment, the Toolkit provides a strong base from which to launch. However, the field learned important lessons from the 2010 Workgroup about the consensus process, gaps in areas of assessment, opportunities to address gaps with new or adapted tools, and limitations in implementing the Toolkit over time in terms of tracking, accessibility, and dissemination. This article describes the development of the Toolkit, including recommended tools and methods for assessment within each ACGME competency domain, and links the lessons learned to recommendations for the 2020 workgroup to consider in creating the next HPM assessment strategy and toolkit. Effective implementation will be crucial in supporting fellows to reach independent practice, which will further strengthen the field and workforce to provide the highest quality patient and family-centered care in serious illness. This will require an inspired, committed effort from the HPM community, which we enthusiastically anticipate.

Key Message

This article highlights lessons learned from the national development and consensus process that led to the 2010 Toolkit of Assessment Methods for hospice and palliative medicine fellow competency-based assessment. Key recommendations are presented as the process to define an updated HPM assessment strategy for fellows begins in 2020.

Background

Graduate medical education (GME) in the United States is guided by a competency-based framework that strives to achieve measurable, observable trainee outcomes.¹ The Accreditation Council for Graduate Medical Education (ACGME) Next Accreditation System¹ includes two key components to accreditation: reporting milestones² and Clinical Competency Committees,³ specifically designed to monitor and iteratively improve educational outcomes both at the trainee level and the programmatic level. Despite these efforts, practical, meaningful assessment of individual trainees remains a crucial challenge.4, 5, 6, 7, 8 The Clinical Competency Committee structure encourages robust, semi-annual discussions of individual trainee performance across the six ACGME core competencies of patient care, medical knowledge, practice-based learning and improvement (PBLI), interpersonal and communication skills (ICS), professionalism, and systems-based practice (SBP) using the developmental framework of reporting milestones.³ However, identifying assessment tools that provide actionable data across specialty-specific competencies to inform milestones reporting is an ongoing challenge.4, 5, 6, 7, 8, 9, 10, 11, 12

Hospice and palliative medicine (HPM) educators and fellowship training programs have been working for well over a decade to meet this challenge of identifying assessment tools that provide meaningful specialty-specific competency data over the concise 12-month timeframe of an HPM fellowship. HPM has been an official subspecialty of the American Board of Medical Specialties since 2006 with ACGME oversight and accreditation of HPM fellowships since 2008 (Table 1 ). Since that time, the American Academy of Hospice and Palliative Medicine (AAHPM) has supported multiple national workgroups, led by educators in the field, tasked with creating fellowship standards, including consensus competencies for HPM fellowships13, 14, 15, 16 and, more recently, developmental milestones^{17 , 18} with additional, complementary education guidelines19, 20, 21, 22 that align with ACGME recommendations (Table 1). In 2010, the national HPM Competencies Workgroup curated a set of assessment tools and created the HPM Toolkit of Assessment Methods ²³ with instructions for use. The Toolkit was to provide all HPM fellowship programs with a robust set of instruments to assess fellows across competency domains, with the overall goal of creating a well-developed, meaningful, and practical path of performance assessment.

Table 1.

Hospice and Palliative Medicine (HPM) Fellowship Development Timeline and History

Year	Event	Significance for the Field of HPM
1989	Cleveland Clinic Palliative Medicine Fellowship28	The first fellowship in the field is created with a single fellow position
2000	Survey of US Palliative Medicine Fellowships29	This first inventory effort documents 20 existing fellowships with 39 fellow positions
2002	Initial Voluntary Standards for Palliative Medicine Training30	A workgroup of palliative medicine fellowship leaders develop these standards to be used for the early accreditation of palliative medicine fellowships
2003	Palliative Medicine Review Committee31	American Board of Hospice and Palliative Medicine and American Academy of Hospice and Palliative Medicine (AAHPM) cosponsor this initial national accrediting committee for hospice and palliative medicine fellowships prior to the transition to Accreditation Council for Graduate Medical Education (ACGME) oversight
2006	HPM becomes a medical specialty of the American Board of Medical Specialties	HPM becomes eligible to create ACGME fellowship pathway
2007	HPM Competencies13,14	Workgroup develops core competencies with measureable outcomes for fellowship training, which inform new ACGME HPM fellowship program requirements
2008	HPM fellowship programs begin ACGME accreditation process	Existing prior and new HPM fellowship programs submit materials for ACGME accreditation under the initial ACGME HPM Fellowship Program Standards which incorporate content from the HPM Competencies
2010	HPM Toolkit of Assessment Methods23	Workgroup defines a set of 18 assessment tools to measure fellow performance
2013	HPM Clinical Competency Committees3	HPM fellowships create these scheduled reviews of individual fellow performance in alignment with new ACGME requirements
2014	HPM Pediatric Competencies15,16	Workgroup develops pediatric core competencies for fellowship training
2014	HPM Reporting Milestones (developmental milestones)	HPM, with other internal medicine subspecialties, defines shared, general developmental framework for fellow assessment in 23 sub-competencies
2015	HPM Entrustable Professional Activities19,20	Workgroup identifies 17 observable, measurable physician tasks that define independent HPM practice for fellows
2018	HPM Curricular Milestones21,22	Workgroup defines 22 teaching elements for a proposed fellowship curriculum
2019	HPM specific Reporting Milestones (replace 2014 Milestones)17,18	Workgroup defines a developmental framework for fellow assessment in 20 HPM sub-competencies
2020	HPM fellowships continue steady growth32	AAHPM data document an increase from 105 to 151 programs during 2013-2020 with fellow positions increasing from 303 to 465

All Workgroups 2007-present were supported by AAHPM, and all workgroup members were educators in the field of HPM.

Since its creation, the Toolkit has been a resource for fellowship directors to evolve practical, multifaceted assessment strategies for HPM fellows working toward independent practice and graduation. However, much has advanced in technology, innovation, and the field while key limitations of the Toolkit need to be addressed. Now, as AAHPM plans for a national workgroup in 2020 to examine current assessment of HPM fellows and propose strategies to strengthen and standardize future assessment, the methods, tools, and lessons learned from the Toolkit of Assessment Methods provide a strong base from which to launch. In this article, we describe the development of the HPM Toolkit of Assessment Methods, the individual assessment tools with recommendations for use, and Workgroup consensus recommendations for assessing fellows within each ACGME competency domain. We also outline key gaps, lessons learned, and recommendations for the 2020 workgroup to consider in creating the next HPM assessment strategy and toolkit.

Methods of Toolkit Development

The HPM Competencies Workgroup developed the Toolkit of Assessment Methods through a formal review and consensus process during the final phase of the multiyear Competencies Project. The nine final-phase Workgroup members included educational leaders in the HPM field, current and past HPM fellowship directors, and junior HPM faculty on a clinician-educator path. Candidate assessment tools were identified from within and outside the HPM field. The Workgroup divided into small groups to identify appropriate tools for each competency domain, reviewing known but unpublished tools, tools from the literature, and those from the ACGME Outcome Project²⁴ and other related evaluation websites. In addition, AAHPM sent an email request to HPM educators inviting them to submit appropriate tools in use at their sites. In selecting final instruments for inclusion in the Toolkit, we were guided by the characteristics of a good instrument, adapted from Epstein⁹ and the ACGME Outcome Project²⁴ (Table 2 ), as well as additional criteria the Workgroup defined for HPM-specific instrument selection (Table 2). In determining criteria for inclusion, the Workgroup prioritized practical assessment approaches that respected the realities of limited education resources and allowed for simultaneous assessment across multiple competency domains and in various patient care settings.

Table 2.

Criteria for Selecting Hospice and Palliative Medicine (HPM)-specific Tools for the Toolkit of Assessment Methods

I. Characteristics of a Good Instrument (Adapted from Epstein9) 1.Reliability: The measurement is accurate and reproducible. 2.Validity: It measures what it is meant to measure in a given HPM setting (face validity and external validity). 3.Low cost/feasibility: It requires a reasonable amount of time or effort for faculty, trainee, institution, and so on. 4.Acceptability: Trainees, faculty, and academic community find it palatable and do not resist its use. 5.Potential Impact on Future Learning and Practice: It promotes learning and improved practice in itself. 6.Objectivity: It reduces the impact of subjective judgment. 7.Provides valuable information: It garners new and useful data.

II. Additional criteria informing HPM-specific final instrument selection for HPM Competencies Project Workgroup 1.Ability to serve multiple purposes (can be used for evaluation of more than a single Accreditation Council for Graduate Medical Education competency domain) 2.Alignment with HPM Competencies, version 2.314 3.Past experience with the instrument 4.Ability to assess interdisciplinary team role and relationships 5.Flexibility for different settings and contexts (i.e., time of year, home hospice vs inpatient care) 6.Ability to be used by members of different disciplines to assess fellows 7.Element being evaluated is frequent enough to enable evaluation in a short rotation

Results

The HPM Competencies Workgroup identified 64 assessment tools across the six ACGME domains for initial review. Most tools aligned poorly with the selection criteria outlined previously and in Table 2, and very few instruments were specific to HPM. In addition, multiple ACGME competency domains and key HPM subcompetencies were not evaluable with the identified tools. Moreover, while many of the identified tools were in use at one or more fellowship sites, very few had been validated or published in the literature. After our initial review of all instruments, we identified, by consensus, the 18 best tools for inclusion in the Toolkit (Table 3 ) and the two best HPM assessment methods for each ACGME competency domain (Table 4 ). Because of the lack of alignment between the HPM Competencies and available tools, the Workgroup elected to both adapt existing tools and create new ones to better fit HPM fellowship assessment needs. Of the final 18 tools, eight were adapted from their original versions, and 10 were newly created (Table 3). See the Appendix for specific tools.

Table 3.

Eighteen Tools in the HPM Toolkit of Assessment Methods by ACGME Competency Domain: An Overview

Recommended Tool from Toolkit	Tool Overview and Purpose	Recommended Use	Other Details
Patient and family care
Patient and Family Care—Attending Physician Assessment (part of PFC 3-Tool Bundle)	•Attending assessment of a fellow's communication skills and management of commonly seen symptoms, using a Likert-scale format •Requires direct observation of fellow performance	• PFC 3-Tool Bundle consists of 3 complementary tools o Patient and Family Care—Attending Physician Assessment o Patient and Family Care—Chart Review o Patient and Family Care—Fellow Self-Assessment •The 3 tools assess similar skills through 3 different means (attending report, fellow self-assessment, or chart review) •Each tool can be used individually or along with the 2 other tools in the PFC 3-Tool Bundle •Used together, the PFC 3-Tool Bundle can give multidimensional view of fellow (e.g., by assessing the accuracy of fellow self-assessment) • PFC 3-Tool Bundle assesses 8 subcompetencies in total •Recommended for use over the course of the academic year, selecting 2 of the 8 subcompetencies per rotation, so that all 8 are assessed by the end of the year	•All tools in PFC 3-Tool Bundle were created by Workgroup and derived from the Master Assessment Table •Not piloted prior to release •No psychometric testing
Patient and Family Care—Chart Review (part of PFC 3-Tool Bundle)	•Chart review tool for assessment of several PFC subcompetencies •Assesses documentation of patient care encounter using yes-or-no chart abstraction format •Chart abstraction can be performed by fellow, peer or faculty
Patient and Family Care—Fellow Self-Assessment (part of PFC 3-Tool Bundle)	•Fellow self-assessment tool, using a Likert-scale format •Assesses broadly across subcompetencies within PFC
Chart Abstraction Checklist—Pain Assessment	•Used to evaluate the fellow's documentation of a pain history, relevant physical exam, and assessment and management plan	•Chart review tool, in a yes/no or partial completion format • Chart review tools can be used in three ways. o Fellow as self-assessment o Peer assessment o Assessment by attending or interdisciplinary team member •Optimal use requires that the evaluator provides direct feedback to the fellow	•Adapted by Workgroup •Piloted in academic settings •No psychometric testing
Chart Abstraction Checklist—Psychosocial-Spiritual Assessment	• Used to evaluate the fellow's documentation of aspects of the psychosocial and spiritual assessment, including o Psychiatric and social history o Health habits o Spirituality and spiritual history o Assessment and plan
Practice-based learning and improvement
Faculty Evaluation of Fellow—Checklist PBLI	•Simple tool with 9-point Likert scale assessing the PBLI subcompetencies •For faculty to assess the fellow's ability to accept and use feedback constructively, to apply self-reflection in clinical practice, to practice evidence-based medicine, to show mastery of basic teaching strategies, and to participate in practice improvement and patient safely measures	• Faculty and Team Evaluation Checklists can be used for global PBLI assessment on an intermittent basis, typically after three months and three months prior to fellowship end. May also be used for end-of-year summative evaluation. •The form can be completed by faculty and/or team members or may also be used as a self-assessment checklist by the fellow, allowing for a multidimensional evaluation of PBLI competency. •Optimal use requires that the written assessment is accompanied by more detailed in-person feedback •Designed for the inpatient setting but may be adaptable to other settings	•Created by Workgroup and derived from the Master Assessment Table •Not piloted prior to release •No psychometric testing
Team Evaluation of Fellow—Checklist PBLI	•Identical to the Faculty Evaluation of Fellow: Check List above, allows members of the interdisciplinary team to evaluate the PBLI skills
Small Group Teaching Checklist	•Allows faculty members to assess fellows' small group teaching skills, including both preparation and actual teaching behaviors •Addresses multiple sub-competencies related to a fellow's teaching skills but not other areas of PBLI	•For faculty members to assess a fellow when directly observing a fellow-led education session •Optimal use requires the faculty member to provide verbal feedback, based on the tool, to the fellow directly after the teaching activity •For use across settings, wherever teaching occurs.	•Adapted by Workgroup •Piloted in academic settings •No psychometric testing
Interpersonal and Communication Skills
Communication Skills Evaluation	•Provides broad-based assessment of 12 subcompetencies, using a 9-point Likert scale •Attends to subcompetencies without attention component skills required to complete a specific communication task (e.g., it requests a single rating for skill in leading a family conference rather than assessing individual skills) •Does not assess details of communication in different clinical scenarios or with interdisciplinary team	•Can be used for global assessment on an intermittent basis (e.g., quarterly or biannually) • Appropriate for use by many different evaluators: o Attending physicians o Interdisciplinary team members o Peers o Fellows, as a means of self-assessment •Lends itself to a 360° approach, excluding the patient and family. •Designed for the inpatient setting but may be adaptable to other settings	•The original version of this tool was created by David Weissman, MD, Medical College of Wisconsin •It was adapted to this version by Bob Arnold, MD, University of Pittsburgh, Palliative Care Fellow Communication Skills Evaluation) •Renamed by the Workgroup for ease of use in the Toolkit •This tool has been widely used in the HPM field but is not validated
The SECURE Framework—Palliative Care (SECURE-PC)	•Provides opportunity for detailed assessment of 30 observable communication skills used during a communication encounter in a “yes/no” or “not applicable” format •The SEGUE acronym (Set the Stage, Elicit Information, Give Information, Understand the Patient's Perspective, End the Encounter) follows the flow of a medical encounter from beginning to end •To adapt the tool for use in HPM, the Workgroup added “Respond to emotions”	•For attending evaluation of a fellow during a directly observed communication encounter with the goal of assessing specific, observable communication skills •Use monthly or quarterly to track a fellow's communication skills on a longitudinal basis • Other uses of the SECURE-PC: o As a teaching tool, outlining component skills at various stages of a clinical encounter o For learner-identified goals, in which the learner picks a specific skill to be observed (e.g., “assesses patient's and family's desire for information and how information should be shared.“) o In this way, the learner could work through several skills in the SECURE-PC over multiple clinical encounters •Does not address interdisciplinary team communication or communication with colleagues	•The Workgroup adapted and renamed the SECURE-PC for use in HPM with permission from Gregory Makoul, PhD, creator of the original SEGUE Framework 25 •The psychometric properties of the original SEGUE are well established25 •The SECURE-PC was not empirically tested prior to release in the Toolkit
Professionalism
Assessment of Professionalism in Palliative Care	•Global assessment in a yes/no checklist format derived from the Master Assessment Table •Assesses 18 targeted sub-competencies of Professionalism, as well as related sub-competencies from other core competencies (e.g., “eagerness to teach” from PBLI)	•Used for comprehensive assessment on an intermittent basis (e.g., quarterly or semi-annually •Can be completed by attending physician, team, peers, and for fellow self-assessment. •Lends itself to a 360° approach, excluding the patient and family •Adaptable to a variety of settings	•Created by Workgroup and derived from the Master Assessment Table •Not piloted prior to release •No psychometric testing
Reflective Journaling Self-Care Exercise	•Focuses on fellows' tools, skills and resources for self-care •Encourages reflection through writing on selected self-care sub-competencies	•Recommended for use at two points in the year, early and late •Fellow completes this independently and reflects on it with a faculty member or with peers •As a guided reflection with peers, it can help fellows learn from one another and support each other •The tool is not setting specific, and similar tools could easily be created specific to a setting or fellowship program.	•Adapted by Workgroup •Piloted in one academic setting prior to release in Toolkit •No psychometric testing
Systems-based practice
Faculty Evaluation of Fellow: Checklist – SBP	•Both of these tools are derived from the Master Assessment Table •They are identical except that one is designed for attending assessment and the other for interdisciplinary team assessment of the fellow • They use a simple rating scale to assess core SBP sub-competencies, allowing for assessment of a fellow's: o awareness of and responsiveness to the larger context and system of health care o ability to effectively utilize system resources to provide high-value care	•Used for global assessment on an intermittent basis, typically after three months and three months before the end of fellowship •Can also be used for an end-of-year summative evaluation • Assessments can be completed by o Faculty o Interdisciplinary team members o Fellow, as a form of self-assessment •Optimal use requires that the written assessment is accompanied by more detailed in-person feedback •Adaptable to inpatient or outpatient settings	•Created by Workgroup and derived from the Master Assessment Table •Not piloted prior to release •No psychometric testing
Team Evaluation of Fellow: Checklist – SBP
Multi-Domain Tools
360° Evaluation	•Allows multiple members of the interdisciplinary team to assess a fellow's professionalism, humanism, patient care, and teamwork •Tailored specifically to work in hospice and palliative care and interdisciplinary teams	•Recommended primarily as a formative measure to aid fellows' development (e.g., can be completed every 3-six months) •Can be used as a summative instrument at end of fellowship	•Adapted by Workgroup •Piloted in academic settings •No psychometric testing
Academic Portfolio	•Consists of both an educational portfolio and a professional development portfolio •Provides a means of documenting a fellow's scholarly work and professional activities in order to demonstrate growth over time	•Designed for use throughout fellowship to document academic progress •Can be used by fellow or by supervising faculty • Used for a variety of purposes: o Fellow identification of specific goals o Self-directed learning and self-evaluation o Documenting outcomes o Fostering mentor- and peer-supported growth	•Adapted by Workgroup •Piloted in academic settings •No psychometric testing
Master Assessment Table (MAT)	•The MAT is both an evaluation tool and a reference document for HPM fellowship faculty creating an evaluation program •As a reference document, it represents the workgroup consensus opinion as to the most important sub-competencies to be assessed in each ACGME competency domain and the best evaluation methods to assess them •As an evaluation tool, it provides a framework for constructing unique checklists for site specific evaluation •Provides flexibility in the breadth and depth that can be covered for one or multiple ACGME competencies or sub-competencies	• As an evaluation tool, can be used to create new site-specific checklists based the desired means of assessment and the sub-competencies on is targeting for assessment o For example, one could create a unique attending physician checklist to evaluate fellow interpersonal and communication skills by selecting a subset of the sub-competencies listed in that competency for yes/no or scored responses •Checklists created from the table can range from short and focused too long and comprehensive •The MAT can be adapted to meet fellowship specific needs for different settings where other evaluation tools do not exist	•Created by Workgroup from the HPM Competencies, version 2.314 •Not piloted prior to release •No psychometric testing •Some of the other tools in the Toolkit created by the Workgroup were derived from the MAT
Palliative Medicine Structured Portfolio	•Designed to address HPM content for the PBLI and SBP sub-competencies •Allows one to compile a comprehensive picture of a fellow's work and progress over time •Focuses on SBP and PBLI competencies directly •Its flexible structure allows fellows and faculty to choose additional ACGME competencies and sub-competencies to target and in what depth and breadth •The portfolio has multiple components: evaluative, reflective and scholarly	•Fellows work with this tool and a faculty mentor to build a personalized portfolio over the course of the fellowship year •Different portfolio components can be completed over time frame most appropriate to the task, such as during a rotation or block of time or longitudinally •Depending on the component chosen, a specific setting or time point may be easier to use (e.g., the fellow might prepare to present at journal club in a particular rotation setting) •Other evaluation tools may be added to deepen a portfolio	•Created by Workgroup •Not piloted prior to release •No psychometric testing

Table 4.

Workgroup Recommendations by ACGME Competency Domain: Assessment Methods and 14 Tools

ACGME Competency Domain	Suggested Assessment Methodsa	Recommended Tools
Patient and Family Care (PFC)	1)Attending Physician Assessment of Fellow 2)Chart Review	• PFC 3-Tool Bundle i. Patient and Family Care—Attending Physician Assessment ii. Patient and Family Care—Fellow Self-Assessment iii. Patient and Family Care—Chart Review •Chart Abstraction Checklist— Pain Assessment •Chart Abstraction Checklist— Psychosocial-Spiritual Assessment
Medical Knowledge (MK)	1)Multiple Choice Exam 2)Attending Physician Assessment of Fellow	•No specific tools were recommended for MK in the Toolkit • Workgroup recommended creation of: o Attending Physician Assessment of Fellow (e.g. written evaluation by clinical rotation) o Creation of a subspecialty in-service examination (multiple-choice format)
Practice-Based Learning and Improvement (PBLI)	1)Attending Physician Assessment of Fellow 2)Team Assessment of Fellow	•Faculty Evaluation of Fellow: Checklist PBLI •Team Evaluation of Fellow: Checklist PBLI •Small Group Teaching Checklist
Interpersonal and Communication Skills (ICS)	1)Attending Physician Assessment of Fellow 2)Team/Peer Assessment of Fellow or Fellow Self-Assessment	•Communication Skills Evaluation •SECURE Framework–Palliative Care (SECURE-PC)
Professionalism	1)Attending Physician Assessment of Fellow 2)Team, Peer or Patient/Family Assessment of Fellow	•Assessment of Professionalism in Palliative Care •Reflective Journaling Self-Care Exercise
Systems-Based Practice (SBP)	1)Attending Physician Assessment of Fellow 2)Team Assessment of Fellow	•Faculty Evaluation of Fellow: Checklist SBP •Team Evaluation of Fellow: Checklist SBP

^aThere are many excellent assessment methods for each ACGME competency domain. The two listed here per ACGME competency domain are those selected by HPM Competencies Workgroup consensus as best practice for that competency domain.

The HPM Master Assessment Table (MAT) is one of the tools created by the Workgroup, and it is designed to serve both as a reference document for fellowships and as a multidomain assessment tool. The MAT was derived directly from the previously published HPM Competencies^{13 , 14} by Workgroup consensus, and it identifies the most essential and representative HPM subcompetencies within each ACGME competency domain and targets them for evaluation. The MAT also includes recommended assessment methods for each targeted subcompetency. Each targeted subcompetency included in the MAT was felt to constitute an exemplar or sentinel skill, the mastery of which was likely to reflect broader mastery of the entire competency. The MAT thus served as a condensed guide for creating an HPM fellowship evaluation plan and as a resource for constructing new checklist tools to meet fellowship-specific needs. Of the other nine newly created tools, eight are checklists derived from the MAT.

The Toolkit results for each of the 6 ACGME competency domains are summarized in the following paragraphs. Please refer to Table 3 for an overview of each of the 18 tools, including each tool's purpose and recommended use. Table 4 outlines the primary assessment methods and tools recommended by the Workgroup to assess trainees within each ACGME competency domain. The full downloadable Toolkit, including the detailed introductory document, remains available for open access use on the AAHPM website (http://aahpm.org/fellowships/competencies), and all tools are included in the Appendix in their downloadable forms. For the 2010 Toolkit, eight tools had been piloted in at least one academic setting. None of the tools had been psychometrically validated; to our knowledge, this remains true to present. Finally, the four multi-domain tools together cover HPM subcompetencies from all the ACGME domains.

Patient and Family Care

The HPM Competencies Workgroup renamed this competency domain, “Patient and Family Care”, to acknowledge a broader, supportive care focus inclusive of all loved ones for the seriously ill patient. Five tools were endorsed for this domain, three created and two adapted (Table 3), which together provide a multifaceted view of important global palliative care concepts and specific skills in the Patient and Family Care (PFC) competency. They do not, however, comprehensively assess all PFC skills. These instruments are designed for use in the inpatient setting, and it may be difficult to translate their use with ease to community settings, such as nursing homes or clinic. In addition, these tools do not address subcompetency areas more specific to the interdisciplinary team and working with multiple teams and professionals. Some of the Multi-Domain Tools, 360° Evaluation, Palliative Medicine Structured Portfolio, and the MAT, address PFC subcompetencies as well.

The Workgroup recommended Attending Physician Assessment of Fellow and Chart Review as the preferred assessment method for PFC.

HPM Medical Knowledge

The Workgroup identified a lack of robust tools for evaluating the HPM medical knowledge competency, a problem that persists to the present. Most tools designed to assess HPM medical knowledge, including those reviewed by the Workgroup, are global and nonspecific, using a checklist or a scale to represent a supervising physician's impression of a fellow's broad knowledge. Although these can be useful as a means of documenting general impressions, they are too vague to be a useful means of assessing HPM-specific medical knowledge. As a consequence, no existing tools were endorsed by the Workgroup for this domain. However, two Multi-Domain Tools include Medical Knowledge subcompetencies: the Palliative Medicine Structured Portfolio and the MAT.

The Workgroup recommended a Multiple-Choice Exam and Attending Physician Assessment of Fellow as the preferred assessment methods for medical knowledge.

Practice-Based Learning and Improvement

The initial review of instruments for PBLI yielded a few examples of portfolios, chart abstraction tools, and practice improvement modules that poorly addressed the targeted HPM subcompetencies. Thus, the Workgroup created three tools for this domain (Table 3). In addition, the Multi-Domain Tools, including 360° Evaluation, Palliative Medicine Structured Portfolio, Academic Portfolio, and the MAT, cover targeted PBLI subcompetencies, as do the Chart Abstraction Checklists from the PFC domain.

The Workgroup recommended Attending Physician Assessment of Fellow and Team Assessment of Fellow as the preferred assessment methods for PBLI.

Interpersonal and Communication Skills

The Workgroup identified two tools for the ICS competency, both adapted and one renamed with permission from the SEGUE instrument.²⁵ The two tools are complementary to one another, with one serving as an intermittent global assessment and the other providing more specific assessment of directly observable skills in a patient-physician encounter. However, subcompetency areas more specific to the interdisciplinary team, specific communication scenarios (e.g., delivering difficult news, conducting goals-of-care conversations, and discussing artificial nutrition and hydration), and communication with colleagues are not well addressed. Multi-Domain Tools that include ICS subcompetencies are the 360° Evaluation and MAT.

The Workgroup recommended Attending Assessment of Fellow, Team/Peer Assessment of Fellow, and Fellow Self-Assessment as the preferred assessment methods for ICS.

Professionalism

Initial review of instruments for professionalism yielded a dearth of useful tools specifically addressing targeted subcompetencies. Consequently, the Workgroup adapted one tool and created a second tool. Additional aspects of professionalism are measured in Multi-Domain Tools such as the MAT and the 360° evaluation.

The Workgroup recommended Attending Assessment of Fellow and Team, Peer, or Patient/Family Assessment of Fellow as the preferred assessment methods for the professionalism competency.

Systems-Based Practice

Initial review of tools available for this competency suggested that portfolios and attending physician assessments are promising methods. However, specific tools that matched the methods, breadth, and depth of the targeted SBP subcompetencies for HPM were lacking. Consequently the Workgroup created two tools for SBP (Table 3). The Multi-Domain Tools that include SBP subcompetencies are Palliative Medicine Structured Portfolio, Academic Portfolio, and the MAT.

The Workgroup identified Attending Physician Assessment of Fellow and Team Assessment of Fellow as the two preferred assessment methods for SBP.

Multi-Domain Tools including the Master Assessment Table

Each of the 4 Multi-Domain Tools—360° Evaluation, Academic Portfolio, Palliative Medicine Structured Portfolio, and MAT—span at least three ACGME competency domains and represent varied assessment methods. In addition, the MAT was constructed to help fellowships focus on specific subcompetencies to target for evaluation and the most appropriate assessment methods. It can also be used as a dynamic framework for creating needed tools.

These evaluation instruments add significantly to the potential assessment options for HPM fellowships. They also address some of the important characteristics for HPM-specific evaluation as identified by the HPM Competencies Workgroup (Table 2). For example, the 360° Evaluation easily incorporates the emphasis on interdisciplinary team member perspectives.

Discussion

The final phase of the HPM Competencies Workgroup defined 18 tools for the 2010 HPM Toolkit of Assessment Methods. This was the first such review for a new subspecialty that was rapidly transitioning to ACGME fellowship accreditation and expanding with many new fellowship programs. The Toolkit provided a starting point for fellowships to begin formulating ACGME competency-based evaluation plans. Creating a forum in which such tools could be shared with the field at large was one of the most exciting aspects of the publication of the initial Toolkit. AAHPM allowed open access for downloading tools from the website and supported the creation of a 2010 webinar for AAHPM membership to learn how best to use the Toolkit.

The 2010 Workgroup process and creation of the Toolkit taught us many lessons about tools, strategies, and challenges in assessing HPM fellows. First, our consensus process was limited in multiple ways. The Workgroup was a small group of dedicated HPM educators who stepped forward together to address the urgent need for the field to advance competency-based assessment. The AAHPM community was much smaller and even less diverse at that time. A formal needs assessment was not performed given the limited data available from fellowship programs, the evidence base, national resources to identify tools, and published expert guidance.

Second, the Workgroup identified glaring gaps where tools simply did not exist or where HPM subcompetencies were not covered to the necessary level of specificity. The most pronounced deficit was in the medical knowledge domain where no tools specific to HPM content were identified. Certain subcompetencies, such as those pertaining to interdisciplinary and interprofessional team interactions and specific communication scenarios (e.g., discussing prognosis and pronouncing death), did not have defined methods of assessment. We also identified a dearth of tools that included patient and family member assessment of fellow professionalism and communication skills.

Thus, as a third lesson from the Workgroup process, we identified the need to create new tools and adapt existing tools to address deficits in tool type and in the evaluation of specific subcompetencies. As such, we developed multiple new tools for the Toolkit and adapted most of the others. The Workgroup focused on tool type in recommending the creation of an objective structured clinical examination for the field as well as expansion of 360° tools, including some focused on specific encounter types, such as family meetings or management of actively dying patients. Because many of the 18 tools reflect a bias toward the inpatient hospital setting, we suggested new or adapted tools to focus on assessment in the ambulatory, home-based, facility-based, and hospice settings, all critical to HPM training and the palliative scope of practice. In addition, the Workgroup recommended that the field develop an annual in-service knowledge examination and an attending physician assessment of fellow knowledge. Although individual fellowships may have developed these tools locally, neither of these tools has been introduced for the field at large as of 2020. By contrast, some communication assessment tool innovations by fellowship programs have been shared nationally. For example, in the direct observation realm, two publications highlight newly created tools for the assessment of HPM fellow family meeting skills.^{26 , 27}

Finally, dissemination and implementation of the 2010 Toolkit was limited by tracking and accessibility. The initial online posting format did not support centralized tracking of use and feedback. Accessibility, beyond the initial release of tools, was limited to single tool downloads to paper or online documents from a website. Moreover, faculty training for implementation was based on written instructions for each tool and one introductory webinar. As fellowship directors have passed along the baton of leadership over time, transmission of wisdom and original source of approach in assessment has been program dependent and variable. Standardization through a centralized process or platform could improve accessibility and support faculty training materials.

As the ACGME framework continues to evolve a decade later, and HPM fellowship structures and standards with it, our history and progress with the 2010 Toolkit position us well to modernize and refine the HPM assessment strategy for fellows. Combining this with our key lessons learned, we suggest five recommendations to the field with the aim of supporting and informing the next phase of HPM assessment work.

Five Recommendations

First, we recommend forming a Workgroup whose membership builds on a broader representation of the field and thus capitalizes on the increasing diversity of the now much larger and growing group of HPM fellowship directors and educators. Such an effort would prioritize an engaging, inclusive, and transparent national consensus process and result in a more effective and representative product.

Second, we recommend performing a formal needs assessment, building on the prior lessons learned to define current trends in recent tool and assessment method use aligned with ACGME competency domains. Such a needs assessment would create an inventory of current tools in use—older, new, and adapted—as well as defining existing deficits and anticipated forthcoming requirements in fellow assessment. Which instruments are most widely, easily, or often used? Which practice settings or skills lack assessment tools? These data will be more robust in 2020, informed by further experience and published data in trainee assessment in competency-based medical education.

The resulting opportunity leads to our third recommendation: Address known deficits in a deliberate and organized fashion while proactively identifying and anticipating emerging needs, including assessment challenges arising as new technology expands. COVID-19 has accelerated the shift toward telehealth opportunities, and HPM can stretch creatively to define fellow assessment in patient/family/colleague/learner encounters with different combinations of telephone and video interactions in synchronous and asynchronous formats. For example, how can we best assess a fellow teaching learners via Zoom or leading a family meeting with a patient on video and family members on speaker phone? In addition, it makes sense to integrate new, timely approaches to telehealth encounters into the assessment paradigm, including consideration of virtual approaches to assessment itself.

Our fourth recommendation presents perhaps the greatest challenge and opportunity for the 2020 workgroup: Create a robust, centralized platform to standardize dissemination, implementation, and feedback for newly defined assessment strategies and tools. HPM fellowship directors would benefit greatly from a nimble use of technology that creates flexible, accessible packaging of a full spectrum of practical assessment tools. Such a centralized virtual platform would also ideally provide evolving guidance on how to update and train clinician educators to use these tools and other standardized approaches to assess fellows, while also supporting management and tracking of fellow and faculty performance outcome data. Thus, the final and related recommendation is for the 2020 Workgroup to engage psychometric expertise to determine how to measure fellow assessment outcomes most meaningfully and to design the dissemination strategy and platform in a manner that supports formal validation studies of selected tools.

Conclusion

The 2010 HPM Toolkit of Assessment Methods was a first effort to comprehensively gather and evaluate assessment tools for HPM fellows, with a goal of supporting competency-based medical education and a shift to hands-on, real-time assessment of fellow performance. Through this work, the HPM field learned important lessons about the consensus process, gaps in areas of assessment, and limitations in implementing the Toolkit over time in terms of tracking, accessibility, and dissemination. We also had the opportunity to create new tools and adapt existing ones to address deficit areas.

A new consensus process focused on HPM fellow assessment tools and strategies will begin in 2020 and can build on these lessons learned as well as the wisdom and experience from the field over the last decade. Ideally, the next HPM assessment strategy will prioritize a broader, more diverse consensus process; a formal needs assessment; strategies to address current gaps and anticipated future needs; a robust centralized platform to drive implementation and support faculty training; and engagement of psychometricians to optimize tools and fellow performance outcomes. Effective implementation will be crucial in supporting fellows to reach independent practice, which will further strengthen the field and workforce to provide the highest quality patient and family-centered care in serious illness. This will require an inspired, committed effort from the HPM community, which we enthusiastically anticipate with the formation of the 2020 AAHPM Workgroup on HPM Assessment.