Creation of an Innovative Inpatient Medical Procedure Service and a Method to Evaluate House Staff Competency

Abstract

INTRODUCTION

Training residents in medical procedures is an area of growing interest. Studies demonstrate that internal medicine residents are inadequately trained to perform common medical procedures, and program directors report residents do not master these essential skills. The American Board of Internal Medicine requires substantiation of competence in procedure skills for all internal medicine residents; however, for most procedures, standards of competence do not exist.

OBJECTIVE

1) Create a new and standardized approach to teaching, performing, and evaluating inpatient medical procedures; 2) Determine the number of procedures required until trainees develop competence, by assessing both clinical knowledge and psychomotor skills; 3) Improve patient safety.

DESIGN

A Medical Procedure Service (MPS), consisting of select faculty who are experts at common inpatient procedures, was established to supervise residents performing medical procedures. Faculty monitor residents’ psychomotor performance, while clinical knowledge is taught through a complementary, comprehensive curriculum. After the completion of each procedure, the trainee and supervising faculty member independently complete online questionnaires.

RESULTS

During this pilot program, 246 procedures were supervised, with a pooled major complication rate of 3.7%. 123 thoracenteses were supervised, with a pneumothorax rate of 3.3%; this compares favorably with a pooled analysis of the literature. 87% of surveyed house staff felt the procedure service helped in their education of medical procedures.

CONCLUSIONS

The “see one, do one, teach one” model of procedure education is dangerously inadequate. Through the development of a Medical Procedure Service, and an associated procedure curriculum and a mechanism of evaluation, we hope to reduce the rate of complications and errors related to medical procedures and to determine at what point competency is achieved for these procedures.

Procedural mishaps are a significant source of morbidity and mortality among hospitalized patients. In the Medical Practice Study, procedural complications were second only to medication errors as a cause of adverse events.^1,2 This problem is particularly concerning because internal medicine residents report being inadequately trained to perform common medical procedures.^3,4 The American Board of Internal Medicine (ABIM) requires substantiation of competence in procedure skills for all internal medicine residents; however, program directors report that graduating residents do not master essential procedure skills.^5,6 In a survey of 383 internal medicine program directors, only 67% believed their residents were proficient in central venous catheter placement.⁶

For most internal medicine procedures, standards of competence do not exist. As a minimum level of proficiency, the ABIM currently recommends that medical residents perform a specified number of directly supervised, successfully completed procedures. A body of experts predetermined these standards by consensus, rather than on the basis of clinical data. In fact, one study found that these standards may underestimate the number of procedures necessary to achieve competency. In this survey of 232 university-based internal medicine residents, 28% did not feel comfortable performing central venous line placement, despite performing on average 3 times the minimum number recommended by the ABIM.³

At many academic medical centers, house staff are expected to perform most medical procedures and to teach medical students these techniques. However, procedural instruction is poorly standardized because the most widely used method is the apprenticeship model of “see one, do one, teach one.” Neither trainee performance during procedures nor quality of instruction is formally assessed in most residency programs.⁷ In addition, due to conflicting clinical responsibilities, most programs lack mechanisms for qualified faculty to teach, observe, and track procedural experience. Furthermore, academic general internists infrequently perform medical procedures and report a lack of confidence in teaching these procedures.⁸ In light of these training deficiencies, the lack of standards in assessing procedural competence comes as no surprise.

In order to address these problems, we developed a comprehensive and novel Medical Procedure Service (MPS) that seeks to 1) create a new and standardized approach to teaching, performing, and evaluating inpatient medical procedures; 2) determine the number of procedures required until trainees develop competence, by assessing both clinical knowledge and psychomotor skills; 3) improve patient safety.

Description

The MPS was established at Beth Israel Deaconess Medical Center (BIDMC) in 2002. The MPS consists of select faculty who are experts at common inpatient procedures. These faculty supervise residents performing medical procedures in a variety of settings, including the medical wards and intensive care units. The MPS focuses on the complete procedure, including obtaining consent, ensuring familiarity with the relevant anatomy, teaching the psychomotor skills required for the specific procedure, and identifying and managing complications. Central venous lines and thoracenteses are performed under the direct supervision of MPS faculty who are also interventional and critical care pulmonary attendings, and utilize a portable ultrasound device whenever indicated.⁹ Two interventional pulmonary faculty members supervise the vast majority of these procedures. Lumbar punctures and paracenteses are performed under the guidance of hospitalist physicians who have extensive training and experience with these procedures. Each month one hospitalist is designated to cover medical consults and medical procedures. In addition, BIDMC has a critical care intensivist on site 24 hours a day, and this faculty member supervises urgent medical procedures during nights and weekends. Pulmonary and hospitalist faculty cover the procedure service as just one aspect of their clinical responsibilities. With 4.7 procedures per week, each taking an average of 45 minutes, we estimate that MPS coverage requires 0.1 faculty full-time equivalent. The billing revenue generated from supervised procedures (388 relative value units in our pilot study) defers some of the cost of having an intensivist on site overnight; previously, when these procedures were not supervised, an important source of clinical revenue was lost.

When an intern on a medical service needs to perform a medical procedure, they contact the MPS faculty through a unique pager number to arrange the time and place for the procedure. During the actual procedure, the intern caring for the patient remains the primary operator of the procedure unless a more experienced clinician is required. If the more senior resident involved in the patient's care feels comfortable teaching the intern how to perform the procedure, then the MPS faculty member provides guidance for both the intern performing the procedure and the supervising resident.

While MPS faculty will monitor residents’ psychomotor performance, clinical knowledge will be taught by following a complementary, comprehensive curriculum. For each procedure, a web-based, multimedia program outlines standard indications, contraindications, benefits, and risks associated with each procedure. Still images with overlying graphical elements illustrate anatomical landmarks, and short digital video segments demonstrate dynamic actions when needed. Pertinent references can be accessed by hyperlinks from the website.

Before performing each procedure for the first time, residents must review these curricular materials and complete a short online quiz assessing their comprehension of the proper procedure technique.

After the completion of each procedure, the trainee and supervising faculty member independently complete online questionnaires. Faculty evaluate the resident's understanding of anatomy, familiarity with the procedure kit, utilization of sterile technique, and procedural skill; they also document when faculty intervention was required to properly identify anatomic landmarks, to redirect or modify technique, or to otherwise complete the procedure (Table 1). The supervisor records the number of needle passes, time required to complete the procedure, and any immediate complications that occur. Residents assess their own level of confidence with the procedure, independently track whether they were able to successfully complete the procedure, and report any complications. Our results will also include a formal chart review to determine whether complications developed subsequent to the submission of the online procedure survey.

Table 1.

Procedure Evaluation Categories

Faculty Evaluation Categories

• Anatomy

• Equipment

• Sterile technique

• Procedural skill

• Needle passes

• Time to complete procedure

• Averted errors

• Adverse events

Resident Evaluation Categories

• Needle passes

• Adverse events

• Comfort level with

^ Indications/contraindications

^ Anatomy

^ Sterile technique

^ Managing complications

^ Performing procedure with and without supervision

The project was reviewed and approved by the Committee on Clinical Investigations, the Institutional Review Board, and the Privacy Board at BIDMC. A pilot of the MPS began in July 2002. The completed curricula and evaluation forms were introduced in July 2003. The procedure service has now expanded to 24-hour coverage; we anticipate supervising >90% of inpatient procedures performed by medical residents.

Evaluation

We piloted the MPS prior to the completion of all curricular and evaluation materials. House staff and faculty have received the program enthusiastically. During the first 12 months of the pilot program (July 2002 until June 2003), 246 procedures were supervised, with a pooled major complication rate (defined as pneumothorax, traumatic organ injury, or significant bleeding) of 3.7%, 95% confidence interval of 1.7% to 6.8% (Table 2). One hundred twenty-three thoracenteses were supervised by the procedure service, with a pneumothorax rate of 3.3%; of the 4 patients who developed a pneumothorax, none required placement of a chest tube. This complication rate compares favorably with a pooled analysis of the literature, which demonstrates a pneumothorax rate of 10.6% when performed by supervised house staff.^10–19

Table 2.

Complication Rate by Procedure

Type of Procedure	Total Performed	Total Complications	Complication Rate, %	Serious Complications*	Serious Complication Rate, %
Thoracentesis	123	6†	4.9	5	4.1
Central venous catheter	57	4‡	7.0	2	3.5
Paracentesis	37	4§	10.8	1	2.7
Lumbar puncture	29	1||	3.4	1	3.4
Total	246	15	6.1	9	3.7

^*Major complication defined as pneumothorax, hemorrhage, or traumatic organ injury.

^†Four pneumothoraces (major complication), none of which required a chest tube placement; 1 hemothorax requiring a chest tube (major); 1 hematoma not requiring a blood transfusion (minor).

^‡One hematoma in a central line site requiring a blood transfusion (major); 1 hemorrhagic complication requiring a blood transfusion (major); 1 cannulation of an improper vein (minor); 1 fractured central venous line cap (minor).

^§One hemorrhagic complication requiring a blood transfusion (major); 2 absence of peritoneal fluid in abdomen after procedure was initiated (minor); 1 catheter became kinked and the procedure was aborted (minor).

^||One venous blood return during cerebral spinal fluid sampling requiring a blood transfusion (major).

Using a preliminary evaluation form, faculty rated the residents as either satisfactory or needing improvement in the categories described earlier. Familiarity with anatomy and positioning of the patient were satisfactory in 96% and 98% of supervised procedures, respectively. Correct sterile technique was observed in 99%. Familiarity with the equipment was satisfactory in 93% of procedures.

The number of needle passes required to successfully complete the procedure was also measured. Sixty-one percent of the procedures were completed with 1 pass, and in 86% the procedure was completed in 1 or 2 passes. For central line placement and thoracentesis, increasing numbers of needle passes are associated with a statistically significant increase in complication rates.^12,14,20 In the 15 procedures with complications, operators used a higher number of needle passes (mean, 2.1 passes) than in the 220 procedures without complications (mean, 1.6 passes); however, this difference was not statistically significant (P = .10, Wilcoxon rank sum test).

Although the complication rate is the primary endpoint, we have tracked and will continue to track errors in technique as a secondary endpoint. We will identify errors that were averted (or “intercepted”) by faculty supervision. For example, a trainee who directs a large bore needle at an inappropriate angle would be alerted by the supervising faculty member, possibly averting a complication. This secondary endpoint is a marker of trainee proficiency, as well as a sensitive marker for errors. For instance, a resident who performs a procedure without warnings from the supervisor is more likely to be viewed as fully competent in that procedure.

At the end of our pilot, the chief medical residents surveyed the medical house staff. Of the 82 total responses (59% response rate), 87% felt that the procedure service helped in their education of medical procedures; 90% believed the MPS helped prevent complications, and 83% believed the service was either time neutral or saved time.

Conclusion

Despite its time-honored tradition, the “see one, do one, teach one” model of procedure education is dangerously inadequate. A standardized program of teaching and evaluation allows more direct faculty observation and feedback, establishes a method to determine when competency is achieved, and reduces medical errors and complications.

Through the development of a Medical Procedure Service at a major urban teaching hospital, and an associated procedure curriculum and a mechanism of evaluation, we hope to reduce the rate of complications and errors related to medical procedures and to determine at what point competency is achieved for these procedures. While our pilot has demonstrated a low rate of thoracentesis complications, this finding requires replication and further study. Because of ethical and practical concerns of withholding important curricular developments or the participation of supervising faculty, we felt that a randomized trial was not justified.

Interventions such as the one described here also help residents to meet Accreditation Council of Graduate Medical Education (ACGME) core competency requirements. However, clear standards by which one can define competence at medical procedures do not exist. We anticipate that our data will allow us to understand the process by which trainees attain competence and determine which parameters are most useful in defining competency. By tracking parameters such as number of needle passes, adverse event rates, averted errors, self-reported confidence levels, time to completion of a procedure, and faculty assessment of knowledge, skill, and technique, we hope to determine the number of procedures a trainee must perform in order to master internal medicine procedures. This type of information will help to establish national standards of training.