Clinical Competency Committees and Assessment: Changing the Conversation in Graduate Medical Education

The Accreditation Council for Graduate Medical Education's (ACGME) new accreditation system has introduced a significantly different world of assessment with its language of milestones and centralized oversight of trainee performance by Clinical Competency Committees (CCCs). Just as the shift to competency-based education and assessment required a culture shift for graduate medical education programs across the United States,¹ so, too, does the shift to using CCCs to determine trainee progression on milestones. In this article, we offer our perspectives on the role of the CCC and discuss challenges and opportunities for graduate medical education programs as they enter this new world of assessment.

It's Not Your Same Old Residency Evaluation Committee

CCCs, as outlined by the ACGME, will not function in the same way that many of the old residency evaluation committees did. Simply identifying failing residents or fellows and discussing remediation plans will not be enough. Performance appraisals of progress will need to be conducted for each trainee—including high performers—on each milestone. Professional judgment, contributed by individual CCC members, will be critical to the integrity of the review process and the final determination of whether trainees are ready for unsupervised practice. Over time, these in-depth reviews will undoubtedly make CCCs privy to knowledge about the quality and quantity of assessments and the absence of curricular elements as they consider assessment evidence for each milestone. Thus, CCCs will become aware of gaps within their program's current assessment approach and the curriculum as a whole. Residency and fellowship programs will need to clarify the role of their CCC² and specify what authority it will have in promoting needed change relative to assessment practices, faculty development,³ and curricular changes.

A Systems Approach

The CCC members may be helped by viewing assessment as a system,⁴ rather than as independent assessment events. Utilizing a systems thinking lens^5–8 will prompt CCC members to examine the purpose and function of assessment processes, the amount and quality of evidence, and feedback loops within their programs. We believe a systems approach to assessment will result in useful and defendable performance appraisals.

Relationships with Stakeholders

All systems include multiple stakeholders. A residency program's assessment system will include, but may not be limited to, the following stakeholders: residency program leaders, the CCC, chief residents, assessors (faculty and others), faculty development providers, trainees, the program evaluation committee, curriculum developers, and ultimately the public. For optimal program improvement, feedback loops^6,9,10 between the CCC and other stakeholders may need to be examined and strengthened (figure). While department-wide faculty development is not part of the CCC's mission, feedback from the CCC to the program director may include recommendations for enhanced faculty development efforts for those engaged in assessment. For instance, if CCC members discover that faculty on the same rotation appear to be using different criteria to assess the same group of trainees (a reliability issue), feedback to the program director will be critical for enhancing the performance appraisal process.

FIGURE .

Example of a Feedback Loop

Abbreviation: CCC, Clinical Competency Committee.

Rethinking Assessment

Milestones Are Not Assessments

Milestones are developmental markers on the path to independent practice.¹¹ Milestones offer specific performance criteria, which were lacking within prior ACGME competency language.¹¹ The goal within competency frameworks is to provide continuous formative feedback to learners to promote optimal performance and support assessment for learning.¹² The use of milestones can theoretically allow individual trainees to identify areas for improvement at an earlier stage. In the past, this was not always possible when using competencies and subcompetencies to guide assessment. In addition, curricular and assessment shortcomings within training programs as a whole may be identified, such as assessments that measure constructs or skills too broadly. Programs will need to identify assessment tools that answer specific questions. Yet, even the perfect tool will not guarantee valid scores if faculty differ in how they use these tools. Faculty development will be critical in ensuring that all assessors understand the what, how, and why of assessment. The milestone review process thus has the potential to promote change at the program level as well.

What You May Need Is Evidence—Not Another Assessment Tool

As residency and fellowship programs are confronted with ACGME mandates, they may rush to create new assessment tools to measure the milestones, give their faculty copies of the entire set of milestones to assess trainees, or add milestone language to the single evaluation form used at the end of each rotation. We argue that rather than focusing exclusively on assessment tools, CCCs will need to identify a wide range of evidence—from multiple sources and contexts—to ensure the validity of performance appraisals.¹³ Evidence can take many forms,¹⁴ including artifacts (eg, published articles/abstracts, awards for best oral presentation, case logs, and committee membership lists); data from assessment tools (eg, in-training examination scores, faculty assessments, 360-degree evaluations, and patient surveys); and even chart audits, quality improvement project notes, or simulation training activities. We recommend that CCCs use a template that includes specific criteria to allow for a standardized approach to the analysis and synthesis of evidence during trainee reviews. The criteria to be included in a CCC review template will be influenced by the values and mission of individual training programs (table).

TABLE .

Impact of Program Culture or Mission on How Evidence is Considered by the Clinical Competency Committee (CCC)

CCC members will be called on to synthesize and interpret information in order to make promotion recommendations. These judgments are not simply a summing up of multiple assessments. Rather, they require careful weighing of evidence against discrete outcomes. The discovery of discrepant data or information that contradicts other evidence will need to be examined and investigated. Dealing with this requires professional judgment.

Do You Need a Culture Change?

Promoting a Culture of Continuous Improvement

When engaging in the milestone review process, CCC members may be confronted with evidence that their program is overly reliant on summative (evaluative, infrequent) assessments. To successfully implement a culture that promotes learning and performance improvement, the meaning and value placed on assessment—especially high-stakes assessment—may need to undergo a dramatic change. More emphasis will need to be placed on formative assessment (informal, frequent) throughout training programs^12,15 to ensure that assessment functions to support learning. Feedback, based on focused observations and specific criteria, is necessary to ensure continuous performance improvement.^3,15,16 This may be a radical change for programs that emphasize summative assessments, including the ubiquitous end-of-rotation evaluations, rather than provide continuous feedback to learners.

A Shift to Learner Accountability

For programs that overwhelmingly use teacher-centered assessment, a culture shift to promote learner accountability will be necessary. Residents and fellows will need to know what they should be learning and should be expected to contribute to their own learning.¹² They will need to understand the range of evidence (from assessment tool data to publications) that will be used to make judgments about their performance. They will also need to understand the role feedback can play in learning, performance improvement, and their own self-assessments.^16,17 We recommend that trainees take an active role in the milestone assessment process. For some competency domains, a portfolio model, where trainees actively collect evidence and reflect on experiences,^18–22 may be an attractive approach.

Conclusion

The milestone project is an opportunity for implementing a new assessment culture within graduate medical education programs, one that embraces authentic, work-based assessment and uses both assessment of learning (summative) and assessment for learning (formative) to support continuous performance improvement. Faculty may benefit from adopting a systems approach to assessment to promote open communication among all stakeholders. Programs that embrace this culture change will undoubtedly reap the benefits of promoting trainee progression through the milestones, as envisioned by the ACGME.