EVALUATION, FEEDBACK AND REMEDIATION IN ANESTHESIOLOGY RESIDENCY TRAINING: A SURVEY OF 124 UNITED STATES PROGRAMS

Abstract

Purpose

It was the purpose of this study to provide a review of evaluation, feedback and remediation methods in U.S. residency programs during 1995-96. The information gathered is to serve as a framework for discussions within and amongst programs regarding ways to enhance their current processes of evaluation, feedback and remediation, and to serve a baseline for future assessments.

Methods

A three-page survey was sent to the program director of each of the 145 anesthesiology programs listed in the ACGME/NRMP Directory. Information about the resident evaluation process (including techniques of gathering information, frequency of evaluations, faculty compliance, and modes of offering feedback), departmental clinical competence committee, probation and remediation policies for problem residents, and the use of formal examinations during residency was sought.

Results

There was an 86.1% response rate. Frequency of evaluation of residents ranged from daily through quarterly: evaluations used both narrative comments and rating scales in 89% of institutions. Faculty compliance in the evaluation process was greater than 75% in 45.1% of programs. Only 25 programs offered formal training about resident evaluation to their faculty. Clinical competence committee meetings average five times annually and 95% of committees are chaired by someone other than the Department Chairperson. 27% of them have resident members. 67.7% of programs have a written policy regarding problem residents, while 82.2% report having a formal probation policy. 48.3% programs use standardized tests to provide feedback and guidance to their residents.

Conclusions

There is a tremendous variety of techniques and methodologies employed among anesthesiology residency programs with regard to evaluation, feedback, and remediation, within the framework of the ACGME guidelines. Providing faculty training in assessment of, and offering feedback to, residents is one area in which many programs can begin to strengthen their current procedures.

INTRODUCTION

Over the past several years, residency training programs in anesthesiology have undergone revolutionary changes, including alterations in the size of programs, increasing diversity of educational backgrounds of trainees, and revisions to program requirements from the Residency Review Committee for Anesthesiology (RRC) of the Accreditation Council for Graduate Medical Education (ACGME). Individual program directors are in the process of developing new strategies and approaches to evaluating and offering feedback to both successful and problem residents, in response to these new internal and external demands.

The purpose of this study was to gather information regarding the state of evaluation, feedback and remediation in the anesthesiology residency training programs in the U.S., during the 1995-96 academic year.

Methods

A three-page survey, developed by the ad hoc Committee for Resident Evaluation of the Society for Education in Anesthesia was mailed to the program director of each of the anesthesiology programs in the United States which was listed in the ACGME/NRMP directory. The survey (Appendix A) requested statistics for the academic year 1995-96. Data about the program size were requested. Information about the resident evaluation process, (including techniques of gathering information, frequency of evaluations, faculty compliance, and modes of offering feedback), departmental clinical competence committees (CCC), probation and remediation policies for problem residents and the use of formal examinations during residency was sought.

RESULTS

Demographic data: After two mailings of the 145 surveys distributed, 124 surveys of anesthesiology residency training programs were returned partially or totally completed by the directors of the programs. This represents a response rate of 86.1%. After two mailings of the 145 surveys distributed, 124 surveys of anesthesiology residency training programs were returned partially or totally completed by the directors of the programs. This represents a response rate of 86.1%.

The average number of residents per class was 7 CA-1s (range 0-23), 9 CA-2s (range 1-30), 11 CA-3s (range 2-35) and 4 CA-4s (range 0-30). 33 programs had 3 or fewer CA-1 residents including 6, which reported zero. The average total resident class size was 25 if CA1’s, CA2’s and CA3’s were included. With CA4’s included it was 29. There were 20 programs in which the total number of CA1-3 residents was fewer than 9. The size of the CA-1 class compared to the CA-3 class is shown in Table 1.

91 programs were university based, 24 university-affiliated and 7 non-affiliated. The average number of attending physicians in each academic department was 43. For the 107 respondents to the item requesting resident-faculty rations, the mean was .74, with the maximum being 2.0. The distribution of ratios appears in Figure 1.

Resident evaluation: Multiple methods are employed among training programs to document resident evaluation. Only narrative evaluations were used by 4 programs (3.2%), a rating scale solely by 9 (7.3%) and a combination of narrative evaluation and rating scale by 110 (89.4%). The frequency of evaluation of residents appears in Multiple methods are employed among training programs to document resident evaluation. Only narrative evaluations were used by 4 programs (3.2%), a rating scale solely by 9 (7.3%) and a combination of narrative evaluation and rating scale by 110 (89.4%). The frequency of evaluation of residents appears in Figure 2.

Data collection was by written form in 114 programs (92%); by computer terminal entry in 18 (14.5%) and by other methods, including scannable bubble sheets, in 5 (4%). Several programs reported use of both written form and computer entry. Narrative comments were a required part of the evaluation process in 93 (75.0%) programs. The comments were required for unsatisfactory ratings in 84 (90.3%) programs and for outstanding ratings in 35 (28.2%).

89% of responding programs use an evaluation system that comprises both rating scales and narrative comments. 67.5% include the American Board of Anesthesiologists clinical competence categories (essential attributes, acquired character skills, knowledge, judgment and clinical skills) as part of their departmental rating system, 26.5% used several of the categories while 6% used none of them.

Faculty compliance in completing evaluations appears in Figure 3. This compliance is monitored in 69.35% programs. The most common consequence of noncompliance was receiving repeated requests from an administrator for completion. Significant consequences for noncompliance, including extra weekend call, decreased bonuses or adverse effect on appointment or promotion, were reported by 3 programs. Only 25 programs reported that faculty development included formal training about resident evaluation.

Review of evaluations with the resident occurred biannually in 40% and quarterly in 39% of programs. Eleven programs report a system of monthly or more frequent review of evaluations. One department reports that feedback sessions are always available, but occur only at the resident’s request. Multiple programs note that more frequent feedback is given if performance problems are identified. Feedback is given verbally during conferences with faculty advisors who are members of the CCC, the CCC chair, department chair, or the vice-chair for education in 92% of institutions, and during 7.1% of those sessions a hard-copy summary evaluation is reviewed. 6.6% of programs send written summary evaluations exclusively to their “non problematic” residents and the remaining programs offer access to evaluations by the resident for their own viewing. 34.6% of programs identify the faculty members to their specific evaluative comments.

Resident feedback sessions are documented by an advisor summary in 83% of programs, and 32% of those programs require a resident signature on the summary. Three programs place a letter documenting the feedback session in the resident’s file and send a copy to the resident, while one institution reports that they tape their feedback sessions and have them transcribed. Several institutions use the minutes of the CCC meeting as their source of documentation of feedback.

Clinical competence committees: The average size of the CCC is 8 members, and approximately 24% of academic department faculty are CCC members. Committee meetings average 5 times each year. In 42.6% of responding institutions the Department Chairperson is a member of the CCC. 95.0% of CCCs are chaired by someone other than the Department Chairperson. One or more of the following appoints members of CCC: Department Chair, CCC Chair, Coordinator of Resident Training or Chair/Vice Chair for Education. Twelve departments have all of the subspecialty division chiefs serve on the CCC, 6 have the entire faculty as their CCC and many choose members who have expressed an interest and are involved in resident teaching.

27.0% of CCCs have resident members. Resident members are permitted to vote in 15.6 % of CCCs or 57.7% of those CCCs that have resident representation.

CCC decisions are made by consensus in 54.2% of programs, majority vote in 39.2%, and two-thirds majority vote in 5%. CCC decisions are final in 37.1% of departments, while in 49.2% programs the Department Chair and in 13.7% programs academic faculty can overrule CCC decisions.

A secondary function of the CCC is the selection of the chief resident. This occurs in 5.7% of programs, while committee members offer input into the selection in 27.4% of programs.

The problem resident: 82.2% of programs report having a formal probation policy, although it exists in written form in only 67.7%. 61.3% routinely provide this policy to their residents. Several institutions are governed by university personnel policies that prohibit formal use of the term “probation,” but do have a period of close observation and scrutiny. Two institutions mentioned the involvement of the Dean of the Medical School and the Dean for Graduate Medical Education in their probation process. Only 53.3% of institutions inform the entire anesthesia faculty of a resident’s probationary status. 70% offer a formal appeals process for the resident who has been placed on probation.

Probation involves the creation of a written plan with specific goals delineated, a time course for them to be met and the consequences of failure, in the majority of the responding institutions. During this period remedial assignments or study plans may be given, one-on-one preceptors assigned, the resident’s senior or “moonlighting” privileges may be revoked, extra time may be spent on specific clinical rotations or an entire year be repeated. Probation is overseen by one or several of the groups indicated in Table 2.

The duration of probation ranges from a 10 day suspension through an indefinite period, which must be completed prior to the final 6 months of training. 18.5% of programs have a 3-6 month, 44.1%, a 6 month and 12.7%, a one year limit on the remediation period. 6.7% of programs place no time constraint on the duration of remediation. During the probation period 78% of programs evaluate a resident more frequently than their non-probation residents. The frequency of evaluation of these residents by faculty appears in Figure 4.

Examinations in resident education: The frequency with which various examinations are administered is seen in Table 3. 76.7% of programs used the results of standardized tests to provide feedback and give guidance to their residents. In 25.0% of programs there were specific consequences (including extension of education, remediation, unsatisfactory evaluation on the ABA clinical competence report, loss of privileges, probation or termination) for poor performances. Rewards for superior performance include consideration for chief resident, a travel allowance or a stipend for book purchases. Six programs use the examination primarily for curriculum development and modification. The frequency with which various examinations are administered is seen in Table 3. 76.7% of programs used the results of standardized tests to provide feedback and give guidance to their residents. In 25.0% of programs there were specific consequences (including extension of education, remediation, unsatisfactory evaluation on the ABA clinical competence report, loss of privileges, probation or termination) for poor performances. Rewards for superior performance include consideration for chief resident, a travel allowance or a stipend for book purchases. Six programs use the examination primarily for curriculum development and modification.

DISCUSSION

It is widely known that the number of residents entering anesthesiology training programs has sharply declined. The decrease was generally greater in university programs as compared to university-affiliated or non-affiliated programs which challenges the traditional notion that university programs are having greater success in resident recruitment, and therefore less vulnerable in the current climate. There were 9 programs that had fewer than the ACGME minimum requirement of 9 total residents, and are, accordingly, in jeopardy of losing their accreditation. Extrapolating from the CA-1 class sizes reported, 20 programs had three or fewer members, and may be in a similar position in two years, if this trend continues.

The resident-faculty ratio ⩽1 reported by greater than 80% of programs suggests opportunities for improved educational experiences, with close faculty-resident interaction, and should strengthen the educational program by permitting resident assignments to be made on the basis of educational value, rather than service needs.

A strong system of resident evaluation is important to assure the progress of trainees, to help identify and remedy problems, and to assure the quality of program graduates. Schueneman et al., in a longitudinal, 15-year multifactorial, repeated-measures designed study, looked at the accuracy of rotation evaluation forms that assessed the usefulness of clinical performance ratings in predicting competence in a general surgery residency. They determined that evaluations can provide a quantitative basis in the documentation of overall proficiency, in particular the identification of superior and inferior residents, when the population is relatively homogeneous, and they offer the suggestion that accuracy could be increased by simply instructing the faculty on the expected use and importance of their evaluation form. Therefore it is surprising that so few of the responding programs provide their attending staff formal education in how to evaluate residents and offer constructive feedback.

ACGME program requirements for resident evaluation are specific as to their objectives, but offer the individual institutions flexibility in achieving them, as evidenced by the data. CCCs are critical bodies in the majority of U.S. anesthesiology residency training programs to ensure that RRC requirements for resident evaluation are met, and frequently exceeded.

Probation and remediation practices vary widely among the residency training programs surveyed. Surprisingly, many programs consider probation a means toward termination and only a few program directors noted that therapeutic interventions and alternative forms of remediation drawn from outside their own department were made available to the problem resident.

Knowledge assessment is an important part of judging the progress of residents and evaluating their clinical competence. The ability of residents to be successful in the American Board of Anesthesiologists certification examination has importance for both the individual resident as well as the residency program, as the success of program graduates in achieving board certification is one of the factors considered by the RRC in making decisions regarding program accreditation. All programs responding to the survey use some method of formal examination as part of resident education, but the majority does not have firm guidelines for acceptable performance. This suggests that subjective evaluation by the faculty is the principal technique employed to determine the adequacy of the residents’ knowledge base. Correlation of subjective evaluation by faculty with residents’ performance on standardized examinations deserves further study.

One shortcoming of our survey is that it failed to elicit any use of objective structured clinical examinations (OSCEs) in anesthesiology residency training. OSCEs use real or simulated patients in a multistation format that evaluates a variety of clinical skills and attitudes as well as cognitive objectives. With increasing availability of human patient simulators, educators will need to define their value as tools for resident evaluation.

We reviewed the state of evaluation, feedback and remediation in U.S. anesthesiology residency training programs and found a tremendous diversity of techniques and methodologies employed. The most glaring observation is that while a many programs report a that majority of faculty participate in evaluation, feedback and remediation processes, only a small percentage offer formal training in these areas. Designing and implementing faculty development curricula to address these issues may be one way in which departments can improve upon their current procedures. Considering the transformation residency training is currently undergoing, it will worthwhile to revisit these issues in the future.