Assessing Competency in Thoracentesis: Lights, Camera, Action!

Pulmonary and critical care medicine fellows are expected to achieve competence in certain invasive procedures (1). A national survey of pulmonary and critical care medicine and critical care medicine fellowship programs reported a high level of confidence in determining fellows’ competence in performing thoracentesis. Assessment methods varied, including 35.8% of respondents setting a minimum requirement of 1–10 thoracenteses (2). However, numerical procedural experience does not necessarily translate into competence (3), resulting in a wide range of set minimum thresholds. Simulation training to develop mastery of skills is recommended before clinical practice (4), but competence during simulation is not evidence of true performance-based skills in patients (5). In addition, feedback from direct observations can also be variable. Therefore, there is a need for more objective, standardized means of assessment on actual patients.

In this issue of ATS Scholar, Singas and colleagues attempt to address this gap by describing a more objective means of assessing fellows’ competence in performing thoracentesis on actual patients based on offline scoring of procedural video recordings (6). The prospective study was performed at a single center. Eight first-year fellows underwent a 2-hour training program on thoracentesis, which included didactics, review of a thoracentesis checklist created via an iterative process, watching a procedural video, and engaging in deliberate practice using an ultrasonography capable task trainer. Fellows were tested for competence once they indicated self-perceived mastery of the procedure and completed at least five thoracenteses under direct supervision by a faculty member. In addition, a faculty member had to confirm readiness to undergo testing. Then, fellows performed a thoracentesis on a patient while wearing a head-mounted video camera, which recorded the procedure. Competency was achieved if all 30 items on the thoracentesis checklist were completed. Scoring was done by the supervising attending based on direct observation and by two separate faculty members who reviewed the recording.

Seven fellows successfully completed all checklist items; one fellow passed after further training and testing. There was 95.6% agreement among the three scorers. Fellows also completed a knowledge test before training and at the time of the competency testing. Both mean knowledge scores and confidence scores had significantly increased, 6.75–9.13 (P = 0.03) and 36.88–48.13 (P < 0.001), respectively. Thus, the authors concluded that their study supports the use of video-based testing for assessment of competence in performing thoracentesis.

There are several strengths to this study. First, the video-based testing provided an objective means to assess performance-based competency while minimizing recall bias. The two video scorers assessed all the videos and could readily identify errors based on the checklist. Second, reviewing the video provided valuable feedback for the fellows. This has the potential for fellows to structure their learning around individual needs, such as specific parts of the procedure they find challenging (7). In this case, it was the manipulation of the three-way stopcock. Finally, it led to changes to optimize procedural performance in sterile technique.

The study is not without its limitations. First, the study was performed at a single center with a small number of participants and no control group. Thus, the methods and results may not be generalizable. Second, their thoracentesis checklist had not been previously validated. Some of the items overlap with a 23-item thoracentesis checklist developed previously based on a modified Delphi technique (8) and a 14-item pleural safety checklist (9). There remains a great need for the development of validated assessments. Third, video review did not capture all elements of the checklist, such as the consent process and review of laboratory values. Incomplete video capture of procedural aspects has previously been described, such as portions of wire handling and drape handling during central venous catheterization video review (10). Thus, it does not obviate the need for direct in-person supervision for complete assessment of competency. Fourth, there is potential for the fellow to modify their behavior because of awareness of being observed and recorded, known as the Hawthorne effect. Finally, faculty were not blinded to the fellow performing the procedure, which may have introduced bias.

To consider implications of the study findings, one needs to address the feasibility of using video recordings in training and education. Potential risks to patients’ rights have been examined and organized into six categories: informed consent policies, informed consent procedures, recorded medical errors, secondary use of recordings, collateral patient information, and public trust issues (11). An additional concern may be the medicolegal aspects in the event of a complication (12). The authors note that their hospital consent form included consent for videotaping for training and education purposes. The videotaping was also explicitly discussed with the patient with video records stored appropriately. After scoring, the video record was permanently erased. Review of the recordings is also a time-intensive process and requires a substantial commitment from faculty. Despite these challenges, there are many benefits to video review systems. Video training has been shown to improve resident knowledge, operative performance, and participant satisfaction (13). It can also be used to assess the effectiveness in the application of simulation training to patient scenarios (14).

Is it time for “Lights, camera, action”? The authors have made an important contribution in assessing the feasibility of video recordings for performance-based competency. It was generally well received and low cost, did not interfere with the procedure, and had many benefits to education and quality improvement. However, the ability to implement video-recording programs will depend on local regulations and hospital policies, including appropriate storage of data.