Entrusting the Process: Assessment of Critical Care Ultrasound Skills in Fellowship Trainees

Point-of-care ultrasound (POCUS) has established broad clinical utility as both a diagnostic and procedural tool in multiple specialties (1), and given its value in the management of critical illness (2), a category of specific POCUS applications has become known as critical care ultrasound (CCUS). Accordingly, the Accreditation Council for Graduate Medical Education (ACGME) requires pulmonary and critical care medicine (PCCM) fellows to demonstrate knowledge in “imaging techniques commonly employed in the evaluation of patients with pulmonary disease or critical illness, including the use of ultrasound” (3). However, the ACGME does not outline any specific guidelines on content, assessment metrics, or thresholds for competency. Efforts to more clearly define content are seen by the European Society of Intensive Care Medicine (4) and Canadian Critical Care Society (5). Even in these, however, it is unclear what metrics should be used to assess competence in the defined domains. An example of this challenge is reflected in the evolution of the emergency medicine ACGME milestone structure, which previously used a combination of numerical and knowledge assessments for the relevant entrustable professional activity (EPA) milestone. For example, an early version of the emergency medicine milestones required trainees to perform a minimum of 150 focused ultrasound examinations, but the most recent version simply includes “interprets results of diagnostic testing including point-of-care ultrasound” (6, 7). Although there is general agreement regarding the importance of competency in CCUS for practicing critical care clinicians, the specifics of training and assessment of CCUS practitioners are heterogeneous and continue to evolve.

In this issue of ATS Scholar, Israel and colleagues describe the validity and reliability of an EPA-based tool to assess competence in CCUS for PCCM fellows (8). The study examined content, response process, reliability, and relation to other variables as four sources of validity evidence. The authors convened a panel of eight experts from multiple institutions to define a list of core EPAs in CCUS for PCCM fellows. The final list included seven diagnostic and four procedural EPAs rated on a modified 5-point Ottawa entrustability scale. The tool was piloted over a 6-month time period at a single institution. Fellows and internal medicine residents were asked to voluntarily perform self-assessments of their skills, and supervising attendings used the tool to evaluate the trainees’ performance.

A total of 54 assessments were recorded by 23 unique trainees and 13 unique attendings. Assessments were recorded for three of the diagnostic and all four of the procedural EPAs defined in the study. Content validity was established via expert consensus, and response process validity was demonstrated by low variance and high agreement between evaluator and trainee self-assessment scores. Reliability was established using generalizability theory, with a high true variance in entrustment scores attributable to the trainee. Because the procedural EPAs were infrequently assessed, the authors could not draw conclusions regarding the validity evidence for these EPAs.

As the authors note, this is the first study of its kind to link CCUS competency assessment to an EPA-based entrustment scale where observations are directly linked to clinical decision making and/or procedural performance. It adds to a small but growing body of literature regarding competency assessment in CCUS and is a much-needed step in determining the best approach to helping fellows develop and refine this important skill set. The tool’s greatest strength is the manner in which it was developed, integrating prior frameworks set forth by professional societies with a consensus agreed upon by a group of multiinstitutional experts. The methods used to generate and assess the validity of the EPAs for this tool were rigorous and, as conceived, provide a tool that is intuitive to use and aligns with other commonly used assessment tools in PCCM fellowship. Moreover, this methodology potentially serves as a model for refining EPAs within a given domain and testing the validity. In this particular case, the low level of variability between evaluator scoring supports the methodology, and although not necessary for the EPA to be useful, the demonstrated consistency in scoring by attending evaluators and self-assessment by fellows is noteworthy.

Despite the thoughtful approach to its design, the study was small, with sufficient evidence for only 3 of the initially designated 11 EPAs, including the evaluation of 1) patients with shortness of breath/respiratory failure, 2) hypotension/shock, and 3) pleural effusion. Accordingly, it may be more appropriate to describe this as a pilot of a methodology to assess the individual EPAs validated in the study rather than as an approach to the appraisal of trainee competency in CCUS as a whole. Notably, four of the defined diagnostic EPAs (assessment of intraabdominal free fluid, deep venous thrombosis, cardiac arrest, and acute kidney injury) were not performed by any trainees in the study. With the small numbers of assessments in a single center, it is impossible to know if this was due to lack of participant comfort or low frequency of use of these applications. If the latter, it raises the question of whether some of the agreed-upon EPAs may or may not be core to achieving general competency in CCUS.

Generally speaking, the variability in number of assessments performed across tasks and the subsequent lack of sufficient validity evidence for 8 of the designated 11 EPAs makes clear that this study is not sufficient to define a generalizable, comprehensive tool to deem someone holistically competent in CCUS. Indeed, the question should be raised whether the goal should be to develop such a tool when a more accurate determination of competency might be better defined on the basis of the individual CCUS skill. For example, we do not say that a trainee has to be competent in paracentesis to perform insertion of a central venous catheter. Perhaps the same educational philosophy should be extrapolated to CCUS, wherein a trainee can be designated competent in a discrete ultrasound skill, such as the use of CCUS to evaluate a patient in shock, rather than across all potential clinical applications. This approach is implicitly supported by the methodology of the study wherein some faculty were able to supervise certain CCUS EPAs but not others. This strategy would potentially define the components that could build a bridge between the overlapping components of the different applications of POCUS within different clinical specialties.

On the basis of this and prior authors’ work in developing various CCUS assessment tools, it seems the best approach to competency determination may be a hybrid combination of existing tools that can be applied in the appropriate setting or a shift to conceptualizing competency in discrete CCUS skills rather than ubiquitous application across all of critical care. Use of POCUS as part of critical care practice is becoming increasingly ubiquitous, given its importance for rapid assistance with clinical decision making, with resultant expectations for competency of PCCM graduates. The disconnect between this expectation and means to best assess skill performance remains a challenge for the critical care education community, especially given the continuing limitation in qualified faculty. Although this study is an important step in the path toward a validated CCUS assessment tool and potentially provides the methodology for defining the course of the path itself, more work remains to develop an optimal shared mental model regarding expectations for competency in CCUS before an appropriate assessment tool can be meaningfully developed.