Are Commonly Used Resident Measurements Associated with Procedural Skills in Internal Medicine Residency Training?

Abstract

Background

Acquisition of competence in performing a variety of procedures is essential during Internal Medicine (IM) residency training.

Purposes

Determine the rate of procedural complications by IM residents; determine whether there was a correlation between having 1 or more complications and institutional procedural certification status or attending ratings of resident procedural skill competence on the American Board of Internal Medicine (ABIM) monthly evaluation form (ABIM-MEF). Assess if an association exists between procedural complications and in-training examination and ABIM board certification scores.

Methods

We retrospectively reviewed all procedure log sheets, procedural certification status, ABIM-MEF procedural skills ratings, in-training exam and certifying examination (ABIM-CE) scores from the period 1990–1999 for IM residency program graduates from a training program.

Results

Among 69 graduates, 2,212 monthly procedure log sheets and 2,475 ABIM-MEFs were reviewed. The overall complication rate was 2.3/1,000 procedures (95% CI: 1.4–3.1/1,000 procedure). With the exception of procedural certification status as judged by institutional faculty, there was no association between our resident measurements and procedural complications.

Conclusions

Our findings support the need for a resident procedural competence certification system based on direct observation. Our data support the ABIM’s action to remove resident procedural competence from the monthly ABIM-MEF ratings.

One important task for Internal Medicine programs is to insure that graduating residents have acquired competence in core Internal Medicine procedures. To be to be eligible to sit for the American Board of Internal Medicine (ABIM) certifying examination, residents must be certified by their program director to be competent to perform 6 core procedures: arterial blood gas, arthrocentesis, thoracentesis, central line placement, paracentesis, and lumbar puncture. Procedural competence is an important skill as procedural complications are a potential source of morbidity and mortality in hospitalized and ambulatory patients.

Determining competence is a high stake but a difficult assessment. Like many domains in clinical medicine, procedural competence has both cognitive and technical aspects. A resident needs to know the indications, potential complications, and be able to appropriately discuss this with patients or their families or both (cognitive skill); residents must also become proficient in the different methods and skills needed to properly perform the procedure (technical skill). Until being certified as capable of performing a procedure independently, residents commonly keep a record of procedures performed in a logbook. Often, no record is kept of the outcome of such procedures (other than in the patient charts).

While a variety of modes for documenting procedures performed by residents have been implemented, no study has examined the association between the number of times a procedure is performed (technical skill), resident confidence or assessment of cognitive skill in their ability to perform a given procedure and complications from that procedure in IM house staff. Likewise, studies have not clearly established if an association exists between procedural complications and commonly used resident assessment tools in IM residency training. Because procedural complications may result from errors in these cognitive or technical aspects, or both, we hypothesized that having 1 or more major complications in a procedure may serve as a proxy for the lack of competence acquisition (an outcome measurement). Resident procedural complications are readily tracked by most program directors and, if an association exists, could lead to important program change.

The purpose of our study was to determine the rate of complications in medical procedures performed by residents, and whether current commonly used method for procedural competence certification predicts having 1 or more complications while performing a procedure during residency training. We also assessed the association of having 1 or more complications in a procedure with commonly used cognitive product measurements—in-training examination and ABIM board certification percentile scores. We investigated this association with cognitive measurements for 2 reasons. Performing a procedure has cognitive aspects that may correlate with these commonly used measurements of fund of knowledge—i.e., complications could be because of insufficient knowledge of anatomy, procedural risks or alternatives, or both. Also, if no correlation was found, this would support the need for direct observation of trainees to assess procedural competence.

Method

Study Setting

Wright-Patterson Medical Center (WPMC) is an Air Force Regional Medical Center that provides primary and tertiary care to military beneficiaries of all ages. WPMC has board-certified faculty in IM and all of the IM subspecialties. During the study period, WPMC was the primary teaching hospital for a categorical Internal Medicine program with 6 to 8 residents per postgraduate year (PGY).

There was 1 primary service attending for each of the 3–5 ward teams. The primary service attending followed their patients on both the floor and the ICU (i.e., open ICU during the study period). The primary service attending reviewed all charts everyday and was aware of all procedures done and major complications. The primary service attending was responsible for signing the procedure log.

Study Population

The 1990–1999 residency program graduates were used as the study group. All residents who completed their 3 years of residency training at WPMC were included. No other exclusionary criteria were used.

Study Measurements

Our process measures included the attending physicians’ monthly rating of the resident’s procedural skills on the ABIM-MEF and monthly procedure log sheets. In previous studies, we demonstrated the reliability and validity of the ABIM monthly evaluation form¹ and the feasibility and validity of this procedure log system.²

During the study period, we used a modified version of the ABIM monthly evaluation form (MEF), in a format recommended by the ABIM. The procedural skills rating question from our modified ABIM-MEF rated resident performance on a 1–9 scale (1 = frequent errors or disregard for patient risk or comfort, 5 = average quality, 9 = excellent quality, model of proficiency). The ABIM-MEF is a global evaluation form.

We collected data on the 6 core ABIM required procedures. Data obtained from the procedure log sheets included: total number of times each procedure was performed and number of months into residency that each resident was deemed competent to perform each required ABIM procedure.

At our institution, residents were required to record whether there had been any complications on their procedural log sheet. This data collection was nonpunitive. Procedure complications were determined by reviewing resident self-reported procedure log sheets. Two of the authors (SD and LC) independently categorized the resident-reported complications as being major or minor. Major complications were defined as requiring further procedural intervention or medical therapy or both. An example of a major complication was a pneumothorax after a thoracentesis. An example of a minor complication would be mild site tenderness requiring no medication or intervention after an arthrocentesis. The charts of all the reported major complications were reviewed by the authors to determine the accuracy of characterization in the logs; a random sample of reported minor complications was also reviewed to ensure accuracy.

Additional correlates of procedural complications for this study included PGY-2 in-training examination (ITE) percentile scores, PGY-3 ITE percentile scores, and ABIM certification exam percentile scores (ABIM-CE). PGY-1 residents in the program did not take the in-training examination during the study period.

Procedural Certification System

Residents at WPMC are required to perform procedures under the direct supervision of an attending physician or a certified resident until they are “certified” as competent in the procedure.

We stratified time to procedural competence as “prompt certification”(a resident who is certified 1 or more standard deviations (in months) below the mean for the cohort in that procedure), “delayed certification” (certified 1 or more standard deviations, in months, after the mean for the cohort in that procedure), and “average certification” (the remaining cohort).

Certification in procedures is obtained through a “competency validation” protocol. The resident must perform the procedure in the presence of a credentialed attending physician and fulfill these criteria¹: The resident must verbalize knowledge of the indications, contraindications, potential complications, and different routes for the procedure, and appropriately document the procedure in the medical record. The resident must also display the necessary technical skills in the procedure and confidence, the latter being demonstrated through actual performance of the procedure and approaching an attending physician for a competency validation. Residents approach faculty members when they feel that they have acquired the requisite knowledge and technical skills in a procedure. This may occur after performing a given procedure less than 3 times or after greater than 30 times. With rare exception, the faculty member who grants competency validation is not aware of the number of times that the resident had previously performed the procedure.

If the resident displays the necessary knowledge, skill, and confidence in a procedure, the attending physician signs the procedure log in a special area, validating that the resident is competent to perform the procedure (e.g., competency validation). The program director then certifies the resident to perform the procedure without direct supervision based on the attending physician’s competency validation and review of procedure documentation.¹

Attending physicians are required to review and cosign each procedure log to help ensure accuracy of resident reporting. The attending physician signs this log at the end of the month, which also facilitates reporting of delayed complications (i.e., complications occurring several hours to days or more after the procedure). A resident cannot turn in a procedure log to the program director without an attending physician’s signature.

Data Analysis

Inferences were made at the 0.05 level of significance, using two-tailed tests. Spearman Rho was used for assessing associations between our variables and complication rates. All calculations were done using Stata (Version 7.0, College Station, Tex.). This study was determined to be IRB exempt.

Results

There were 71 graduates during this 10-year study period. Two were excluded because they did their internships at different hospitals, leaving 69 graduates for the current study. All residents who started residency training during this time period completed their Internal Medicine residency training in 3 years (i.e., no decelerated residents).

Study Measurements: Descriptive Data

On average, each resident performed and reported 245 procedures during training (range = 91–342). There were 2,212 monthly procedure log sheets¹ and 2,475 ABIM-MEFs during this 10-year study period.² All procedure log sheets were cosigned by an attending physician. The average number of procedure log sheets per resident was 32.7 (range = 15–36) and the average number of ABIM-MEFs per resident was 26.2 (range 24–39). Ninety percent of ABIM-MEF scores were between 7 and 9 (range 1–9). On rare occasions (<1% of ABIM-MEFs), an attending physician rated a resident’s procedural skills on the ABIM-MEF without documenting that they observed the resident performing a procedure (i.e., no procedures listed on the resident’s signed procedure log sheet).

There was a low overall resident self-reported complication rate for all procedures. Using the resident as the unit of analysis, the overall rate of complication was 2.3/1,000 procedures (95% CI: 1.4–3.1/1,000 procedures) and there was a wide range in the length of time to procedural certification (1–23 months). Using the individual procedure as the unit of analysis (i.e., more than 1 resident often performed the procedure), the complication rate was 6.2/1,000 procedures. Over the 10-year study period, there were 38 major complications reported (Table 1): 42 residents (61%) reported zero, 15 (22%) had 1, 8 (12%) had two, 3 (4%) had 3, and 1 (1%) had 4 major complications during the study period. For 6 major complications, 2 residents reported the same complication. Major complications listed on procedure logs were accurately reported and randomly selected minor complications—this was confirmed by chart review by two of the authors (LJ and SD).

Table 1.

Resident Reported Procedural Complications, Wright–Patterson Medical Center, 1990–1999

Procedure	No. of times performed	No. of complicationsa	Complication ratea
ABG	388	0	0
A-line	1,424	1	0.7/1,000
Arthrocentesis	415	0	0
Central Line	1,932	22	11/1,000
Lumbar Puncture	821	0	0
Paracentesis	415	0	0
Thoracentesis	760	15	20/1,000
TOTAL	6,155	38	6.2/1,000

^a Major complication/s.

The 69 residents sat for the in-training examination (ITE) during PGY-2 and PGY-3. The median ITE percentile score was 85 in PGY-2 (range: 22–99th percentile) and 84 in PGY-3 (range: 25–99th percentile).

Study Measurements: Associations

There was no association between having 1 or more procedure complications in any of the required ABIM procedures and ABIM-MEF procedural skill ratings (all p > 0.05; r < 0.20). Likewise, there was no association between having 1 or more procedural complications, in any required ABIM procedure, and written examination performance (ITE and ABIM certifying examination; all p > 0.05; r < 0.20).

There was also no association between ABIM-MEF procedural skills ratings and ITE-2 decile score (p > 0.05; r = 0.04), ITE-3 decile score (p > 0.05; r = 0.05) or ABIM-CE decile score (p > 0.05; r = 0.02). Further, there was no association between total number of times each required procedure was performed during residency, total number of all procedures performed during residency, number of months into residency to be deemed competent to perform each procedure, prompt certification in any procedure, or delayed certification in any procedure and having 1 or more procedural complications (all p > 0.05, r values < 0.20).

On average, complications occurred 5.3 months before competence certification (95% CI: 0.6 to 9.9 months before certification). Of all the complications, 70% occurred before the resident became certified in a procedure. For the remaining 30% of complications that occurred after being certified in a procedure, having a complication was associated with the number of additional times that a procedure was performed—the greater the number of times that a procedure was performed after certification, the more likely that a complication occurred (r = 0.48).

Discussion

Procedure training, certification, and outcome measurements for certification have long been an area of uncertainty in IM training. Methods for measuring procedural skills in Internal Medicine residency training have not been rigorously studied, nor has the relationship between procedural skills and other resident performance measurements been clearly defined. In this study, we found low complication rates for ABIM-required procedures, similar to previous reports. We also found that certified residents were significantly less likely to report a future complication in that procedure (both lower number of complications and lower complication rates) during residency. This finding is reassuring—it appears that competence-based certification is a valid method for attaining independence in core ABIM procedures. There was also no protection from having a complication in 1 procedure by being certified as competent in another, suggesting internal reliability of our procedural competence certifying method.

The lack of relationship between how quickly residents became certified and complication rates suggests that resident self-assessment of preparedness correlates with proficiency in a given procedure (manifest by future complications in a procedure). This is also supported by the finding that prompt or delayed certification status in any procedure did not correlate with future complications—our data suggest that resident confidence in performing procedures unsupervised, when given clear and specific performance criteria, is associated with improved procedural outcomes (fewer complications).

There was no association between other commonly used resident measurements and complications, including ABIM-MEF ratings of procedural competence or scores on the ITE-2, ITE-3, or ABIM certifying examination (ABIM-CE). This important finding provides data to support the ABIM and ACGME’s movement to remove procedural skill ratings from the new ABIM-MEF that incorporates the core competencies. This may be because of the complex relationship between factors that influence complications (i.e., supervision, experience) and issues that influence resident ratings (i.e., rater skills, amount of observation). ITE and ABIM-CE scores reflect primarily cognitive competencies. Whereas it may be possible to assess whether or not an individual has the cognitive understanding of the indications and potential complications of procedures with a written test, it appears that actual complications relate more to performance of the procedure rather than cognitive understanding of the procedure. Further, on some occasions, attending physicians rated a resident’s procedural skills without direct observation of their skills (i.e., no procedures for the month on cosigned log sheet, yet they filled in a procedural skills score on the ABIM-MEF).

The lack of association between ITE and ABIM-CE scores with having 1 or more complications in an ABIM-required procedure supports the need for separate tools for assessing procedural competence. Likewise, the significant association between certification status and complications (number and rate) underscores the need for direct observation to properly monitor resident procedure performance before certification.

Our reported complication rate is consistent with nationally reported rates for thoracentesis (25/1,000; reference³) and immediate complications from central line insertion (15/1,000; reference⁴). Of note, central line insertion included internal jugular, subclavian, and femoral vein insertion; the latter may have contributed to our lower complication rate.

Our study was limited by several factors. First, one of our underlying assumptions was a relationship between competency and having 1 or more procedural complications. Not only does this assumption have face validity, it makes sense. In this study, we did find a relationship between competence certification and lack of complications, suggesting that this assumption is partially valid. It is not surprising that the relationship we found was only modest, as complications can happen even in the most expert of hands. An ideal study looking at the relationship between competence and procedural complications would have included a rigid, gold standard assessment of each resident’s competence. However, our study was “real-world.” The measures we report, attending ratings of competency, resident procedural logs, ABIM-MEFs and ITEs, are commonly used in US residency programs. Second, our data was collected from a single institution in a single specialty and may not be generalizible to other residency programs and/or specialties. The duration of our study period, the completeness of data, and our prior published description of tools used in this study strengthen our findings. Third, we included no “longer term” procedural outcome measurements. Fourth, our findings were limited by the low number of major complications, and complications were self-reported. However, the resident’s supervising attending physician for the month is required to proofread and sign the procedure log. Attending physicians corrected procedures and reported complications on over 5% of procedure logs. Also, our complication rate was consistent with national published rates, and random chart review confirmed the accuracy of complications on procedural log sheets. Additionally, most of the procedures studied generally have a low complication rate, and a lack of competence in a procedure may not be reflected by having 1 or more complications. Complications are so rare and underreported that they may be a poor marker for lack of competence unless there are repeated complications of the same type of procedure. For example, a previous study of resident skill in performing flexible sigmoidoscopy revealed that limited experience can be associated with poor performance, although there were no complications.⁵ Fifth, because of the low rate of complications, we were underpowered to show the correlations between complication rates and ABIM MEF ratings, ABIM board scores or in-service training examinations to be statistically significant. For all of these, the correlations we found were quite low, with Pearson’s rho for each at less than 0.05. Even if we had the power to show such weak associations to be statistically significant, the clinical significance would be questionable. Sixth, we did not explore the potential impact of simulation to teach procedural skills. This was explored in a recent study;⁶ the impact for IM procedure training is unknown. Seventh, our study was limited by range restriction of our resident measurements. Whereas it is possible that associations may have been found if range restriction was not present, the extremely weak correlation we found suggests that these rating methods are poor predictors of resident procedural complication rates. There is literature to support poor correlation between resident confidence and competency. Our system of competency certification has shown the opposite, which may, in part, be because of resident and faculty understanding of the components of competency validation—this is outlined to both residents and faculty in departmental meetings throughout the year. We also acknowledge that 1 competency certification observation has limited reliability. We supplement this observation with procedure note review and discussion with the resident regarding alternate routes, indications, potential complications. Our finding of a lower complication rate after certification is reassuring; providing some support regarding the reliability and validity of our system. Furthermore, we have previously demonstrated that our residents received their competency certification for each required procedure within 1 SD of the corresponding ABIM recommendation.² Finally, we randomly reviewed a sample of the charts of minor complications; a complete review would allow for description of minor complications.

Our data support the need for a resident procedural competence certification system based on direct observation. Our data also provide preliminary support for moving from a system of procedural competence assessment, used at some residency training programs, that is solely based on attaining a given number of attempts in a procedure to a system based on resident perception of procedural proficiency with clear and specific observed performance criteria. In our system, residents, whether obtaining certification early or late, had significantly lower rates of complication after certification than before.

For future studies of procedures, outcomes could include whether the procedure was successfully performed, whether a complication occurred, and whether patient management or outcomes were effected by the procedure. To date these data have not been comprehensively reported for any Internal Medicine (IM) residency program. Future research could try to tease out what is the “best” method for appraising procedural competence. Our study suggests that 1 possible model could be competency validation.

Our data support the ABIM’s movement to remove resident procedural competence from the monthly ABIM-MEF ratings. The lack of correlation between ABIM-MEF ratings of resident procedural competence and resident procedural complications suggests the lack of validity of this measure for rating procedural skill. This may reflect lack of sufficient direct observation in any given month by supervising physicians or the possibility that the independent system of rating and certifying procedural competence did not cross over to ABIM-MEF scores.

Our findings suggest that commonly used cognitive measurements in Internal Medicine are not a proxy for poor procedural performance, as measured by complications, and that procedural proficiency needs to be assessed by direct observation.