Introduction
In the Next Accreditation System (NAS), resident performance of educational Milestones will be 1 indicator of the educational effectiveness of residency programs. The Physical Medicine and Rehabilitation (PMR) Milestones were developed to identify the critical knowledge, skills, and attitudes residents need to become independently practicing physiatrists. During the process of developing the Milestones, the working group also considered how the Milestones could be assessed, and initiated a process for national constituent involvement in the development of PMR–specific assessment methods.
Milestone Development History
The PMR Milestone Working Group (Box) was selected by the Residency Review Committee (RRC) for PMR and the American Board of PMR (ABPMR). The working group included 9 faculty physicians who were current or former members of the RRC, directors of the ABPMR, residency program directors, and 1 RRC resident representative. In addition, an advisory panel of senior leaders from the field provided feedback and critical assessment of the Milestones via multiple teleconferences during the 2-year process. The working group benefited greatly from the work already done by the Phase 1 specialties, the Accreditation Council for Graduate Medical Education (ACGME) expert panel, and by the expertise of the ACGME staff.
Box Members of the Milestone Development Group
Working Group
William L. Bockenek, MD, Carolinas Medical Center, Chair
Anthony Chiodo, MD, University of Michigan Health System
Caroline Fischer, MBA, Accreditation Council for Graduate Medical Education (ACGME)
Gerard Francisco, MD, University of Texas Medical School at Houston
Anna Gaines, MD, Tri Rivers Surgical Associates
Susan Garstang, MD, Rutgers New Jersey Medical School
Michelle Gittler, MD, University of Chicago Medicine
Wendy Helkowski, MD, University of Pittsburgh
Mary McMahon, MD, Cincinnati Children's Hospital Medical Center
James Sliwa, MD, Rehabilitation Institute of Chicago
Susan R. Swing, PhD, ACGME
Carol Vandernakker-Albanese, MD, University of California-Davis Health System
Advisory Group
Timothy Brigham, MDiv, PhD, ACGME
Gary Clark, MD, MetroHealth System
Karen Kowalske, MD, University of Texas Southwestern Medical Center
Teresa L. Massagli, MD, University of Washington
William Micheo, MD, Instituto San Pablo, Puerto Rico
Michael O'Dell, MD, New York Presbyterian Hospital
Sunil Sabharwal, MD, VA Boston Healthcare System
Barry Smith, MD, Baylor College of Medicine
Kathryn Stolp, MD, Mayo Clinic
Gregory Worsowicz, MD, University of Missouri
The working group began by reviewing the literature on competencies and Milestones, the Phase 1 specialty Milestones in development, the accreditation standards for PMR, and the blueprint for the ABPMR initial certification examination. Using this information and member expertise, the working group rigorously debated and then selected patient care tasks and patient conditions to serve as a framework for the Patient Care and Medical Knowledge Milestones. The working group also reviewed the ACGME's multidisciplinary expert panel's literature-based and experience-based model set of Milestones for Professionalism, Systems-Based Practice, Practice-Based Learning and Improvement, and Interpersonal and Communication Skills. The PMR Working Group maintained the basic structure of the expert panel's Milestones but significantly modified them to be specific and relevant to PMR.
During the 2-year development period, Milestone updates were provided and input from the field was sought at meetings of the American Academy of Physical Medicine and Rehabilitation, the Association of Academic Physiatrists, the ABPMR, and via 3 webinars directed toward PMR residency program directors and department chairs.
The working group conducted a pilot test of the Milestones in the spring of 2012, convening their own clinical competency committees to rate residents at each level of training and followed that up by seeking structured feedback from all program directors. From these activities, it became clear that PMR residency programs had varied curricula that would not allow the Milestone levels to be time-dependent. As an example, some programs have rotations teaching electrodiagnostic procedures throughout the residency, whereas others cluster such rotations in 1 block of time. These findings validated the decision to write the Milestones as a progression in skills without time-tied achievement expectations (with the exception of Level 4 as the graduation target). The Level 1 Milestones also raised concerns. Some PMR programs are 4 years in duration, and include the internship year, whereas others begin 3 years of training at the postgraduate-2 level. For the 4-year programs, some of the Level 1 Milestones are not assessable, and the Milestone rating scale was modified to include a rating of “Has Not Yet Achieved Level 1.”
Over the course of multiple meetings and ongoing work, the working group drafted, revised, and refined the Milestones using feedback from the field and the results of the working group's pilot test. The most significant revision was the creation of a single set of Medical Knowledge Milestones for reporting to the ACGME.
General Features of the Specialty Milestones
PMR programs will report semiannual Milestone evaluations on 19 subcompetencies (or Milestone groupings). These include 7 Patient Care subcompetencies, 1 Medical Knowledge, 3 Systems-Based Practice, 3 Practice-Based Learning and Improvement, 3 Professionalism, and 2 Interpersonal and Communication Skills subcompetencies. The Patient Care Milestones describe important physiatric patient care tasks. The single set of Medical Knowledge Milestones rate application and synthesis of physiatric knowledge in a developmental fashion. The original 9 Medical Knowledge Milestone subcompetencies, which related to specific disorders commonly cared for by physiatrists, were retained as an addendum for program use in curriculum development. Milestones in the other 4 general competencies are organized around subcompetencies from the ACGME's Common Program Requirements or the expert panel's work.
Envisioned Practical Use in Evaluating Residents
Most PMR residency programs use a global evaluation constructed as Likert scales (eg, 1–9) modeled after the ABPMR annual summary for evaluation of residents on their clinical rotations. As constructed, the scale suffers from lack of differentiation of the levels 2–8 and from rating multiple skills or attitudes within each item. The Milestones should improve competency assessment by making the expected behaviors for progression in skills and readiness to enter practice more explicit for both residents and faculty. This will allow programs to more closely monitor the progress of individual residents and implement focused remediation, if needed. Concurrently, the ABPMR is modifying the annual summary to incorporate use of the Milestones in rating resident competence, as well as readiness to enter practice without supervision, and for admission to Part 1 of the initial certification examination.
Recommendations for Competency Committee Composition and Functioning
Formation of a Clinical Competency Committee (CCC) is a new expectation for programs, and as such, requires support from the department leadership for protected time to conduct its work. The CCC should be appointed by the program director and include a minimum of 3 faculty members. Faculty should be from the major training sites and should be representative enough that there is familiarity with each resident. PMR residents do work closely in teams with nonphysician members of the health care team, so programs may want to consider inclusion of nurses or therapists on the CCC. The program director often has the most detailed knowledge of resident performance, but has to balance the role of advocate and evaluator. The work of the CCC is to fairly and thoroughly evaluate residents semiannually, and advise the program director regarding progress, promotion, remediation, or dismissal.
The PMR Milestones are not training level–specific, and residents may encounter experiences in different years of training, based on the program structure. Thus each CCC will need to determine if residents in their programs have achieved the expected level of competence to progress to the next level of training or to independent practice, or if remediation or dismissal is in order. Information obtained from the CCC should be forwarded to the program director and mentors to incorporate into the feedback provided during the resident semiannual evaluation.
Establishing Milestone Validity, Utility, and Practicality
In July 2012, members of the working group were asked to pilot use of the Milestones in their own programs. Programs evaluated at least 1 resident at each level of training. The feedback was generally positive about the clarity of behaviors expected to be addressed for each competency. Milestone ratings trended upward by year of training, with a fair amount of overlap by year. Larger programs noted more concerns about the amount of time to complete the evaluation process than smaller programs, in which all members of the CCC were familiar with each resident's performance. It also became apparent that the most commonly used assessment tools were not adequate to directly assess the Milestones, and that added faculty development programs would be needed to train faculty to accurately observe and rate behaviors. The most commonly used evaluations in PMR residencies are global evaluations, multisource evaluations, direct observation, checklists, and structured oral examinations. Most of these tools would require inference to use in Milestone assessment.
The PMR national program directors' workshop in the spring of 2013 focused on the evidence basis of evaluation tools to help assess the competencies. Members of the program director council were recruited to trial evaluation tools, and to tailor them to the Milestones to help CCCs in making Milestone assessments. Six groups of program directors trialed various tools in their programs and were asked to determine if raters could distinguish among the Milestones, if the Milestones represent a progression in skills, if the tools were feasible and practical for raters and CCCs, and the type of faculty development needed to support reliable use of the tools. Results will be presented at the spring 2014 program director council, with a goal of having all programs ready to assess the Milestones as we enter the NAS in July 2014.
Conclusion
The PMR Milestones will be used to provide the RRC and ACGME with evidence of the educational effectiveness of PMR training programs. They also will be valuable in assessing the progression of individual residents to identify residents who may be in need for added education or remediation, and to assure the public that graduating physiatrists have achieved the competence to prepare them for unsupervised practice.