Pragmatic assessment of resident performed cardiac point of care ultrasound using a validated scoring metric

Abstract

Background

Cardiac point of care ultrasound (POCUS) is increasingly utilized by trainees across various specialties and can rapidly assess the cardiac status of unstable patients. However, the quality of trainee performed cardiac examinations has been reported only in controlled studies. In this study we aimed to assess the quality of all cardiac POCUS performed by internal medicine trainees at a major academic medical center over a three-year period.

Methods

256 studies were included and were graded by experts blinded to postgraduate year (PGY) using a previously validated scoring metric.

Results

We found significant improvement in overall quality of resident performed cardiac POCUS from PGY 1 to 3 (10.8, 10.2, 13.2, p = 0.012). Assessment of left ventricular function was possible in 82% of studies and increased from PGY 1 to PGY 3 (77%, 76%, 88%, p = 0.025). Similar trends were seen in the assessment of the pericardial space (67%, 71%, 84%, p = 0.012). Images sufficient for right ventricular and volume status assessment were less commonly found (65%, 60%, 75% and 60%, 49%, 57%, respectively).

Conclusions

This study provides a real world experience of the level of diagnostic accuracy that can be expected from IM trainees with minimal hands-on supervision.

1. Introduction

Cardiac point of care ultrasound (POCUS) is increasingly used by trainees in internal medicine (IM).[1] It can rapidly assess cardiac function in patients with dyspnea or hemodynamic instability.[2] Major societies now offer guidelines for the proper indication and performance of cardiac POCUS.[3] There exists widespread interest in integrating POCUS into residency training, however there are limited studies that go beyond assessment of confidence to an assessment of the quality of exams performed by trainees, and none do so outside of the controlled environment of a study.[4], [5]

2. Methods

As part of an educational initiative, infrastructure was added to allow wireless archival for POCUS studies in a single hospital of a large academic medical center.[6] All uploaded cardiac POCUS performed between 1/2017 and 12/2019 with at least one acquisition of a cardiac structure and an identified author were included. Each examination was randomly interpreted by one of five board certified cardiologists (blinded to other information). The examinations were scored using the previously validated RACE scoring system.[7] 15 identical studies were interpreted by all five graders to assess inter-rater differences. Statistical analysis was performed using STATA v14.2. Relationships between scores and PGY level were determined through univariate linear regression. Interobserver scores were generated using Cohen’s kappa with multiple graders. This study was approved by the IRB.

3. Results

A total of 553 studies were scored by the graders, of which 256 were included in the final analysis. 297 studies were excluded because they were performed by a fellow or year of the performing trainee was undeterminable. 52 POCUS studies were performed by PGY 1, 77 by PGY 2, 127 by PGY 3, and 3 by PGY 4 trainees. The numbers of studies increased over time, with 18 during 2017, 51 during 2018, and 198 during 2019 (p < 0.001). Overall, the mean total score out of 25 increased from PGY 1 to PGY 3 (10.8 to 13.2, p =0.012) (Table 1).

Table 1.

Evaluator scores for difference components by PGY (Mean and SD, or proportion).

	PGY 1	PGY2	PGY3	P value
PLAX (0–5)	2.8 (1.7)	2.9 (1.8)	3.4 (1.7)	0.039
PSAX (0–5)	2.3 (2.0)	2.2 (2.0)	2.8 (2.0)	0.042
Apical (0–5)	1.7 (1.8)	1.9 (1.9)	2.3 (1.8)	0.026
Subcostal (0–5)	1.7 (2.0)	1.1 (1.8)	2.0 (2.0)	0.158
IVC (0–5)	2.3 (2.1)	2.1 (2.1)	2.7 (2.3)	0.287
Total score (0–25)	10.8 (6.4)	10.2 (6.5)	13.2 (6.8)	0.012
LV (0/1)	(40/52) 77%	(59/77) 77%	(112/127) 88%	0.025
RV (0/1)	(34/52) 65%	(46/77) 60%	(95/127) 75%	0.186
Volume status (0/1)	(31/52) 60%	(38/77) 49%	(73/127) 57%	0.910
Pericardium (0/1)	(35/52) 67%	(55/77) 71%	(107/127) 84%	0.012

PLAX (Parasternal long axis), PSAX (Parasternal short axis), IVC (inferior vena cava), LV (left ventricle), RV (right ventricle), PGY (post graduate year)

Assessment of left ventricle (LV) function was possible in 82% of studies and increased from PGY 1 to PGY 3 (77%, 76%, 88%, p = 0.025). Similar trends were seen in the assessment of the pericardial space 77% (67%, 71%, 84%, p = 0.01). In contrast, only 68% of exams had sufficient views of the right ventricle (RV) for evaluation, with no trend by PGY. Similarly, only 55% of exams had sufficient views of the inferior vena cava (IVC) for right atrial pressure assessment, with no trend with PGY. Combined Kappa for the 15 exams that were interpreted by all readers was poor at 0.21.

4. Discussion

We report an analysis of 259 IM trainee performed cardiac POCUS exams at a major academic medical center over a nearly three-year time span. There was a significant increase in utilization over time. Diagnostic quality imaging was present in most examinations. There was improvement in total score from PGY 1 to PGY 3. This study provides a real world experience of the level of diagnostic accuracy that can be expected from IM trainees with minimal hands-on supervision.

POCUS is a valuable resource, but one concern is the quality of data that can be acquired by trainees. Studies like this clarify the potential of trainees to acquire interpretable images. It is noteworthy that over 80% of relatively untrained PGY3 residents obtained satisfactory images to assess LV function and the pericardial space, two key pieces of information in the evaluation of unstable cardiac patients. For more technically challenging structures, including the RV and IVC, trainee performance was lower.

There are several limitations to this study. Firstly, uploading of examinations was not mandated. An additional limitation is that, the educational curriculum was a convenience intervention delivered primarily during clinical cardiology rotations so not every resident would be guaranteed any or similar exposure during their first two years. There was a chance that the exposure was time dependent which could explain the proficiency of the PGY3 residents relative to their peers as they would be more likely to have had one or multiple training sessions. An additional limitation is that 37% of the reviewed scans did not contain sufficient information to identify the PGY year of the performing trainee, and were thus excluded from analysis stratified by PGY year. A secondary analysis comparing these excluded scans to those which included PGY year showed that in general these scans were of lower quality, perhaps consistent with lower trainee confidence. Finally, our interobserver variability was low and suggested that the RACE score did not perform as reliably with our expert graders, as in prior studies. This suggests that in future work including that instrument, additional effort should be placed in training and validating the performance of graders.

Multiple prior publications have demonstrated that trainee scan confidence increases with training, but there are limited descriptions of scan quality once the intervention ends. We report 3 years of cardiac POCUS at a major academic medical center. Cardiac POCUS use increased over time, and POCUS quality improved in several key domains during medical training. Further work is needed to assess trainee interpretation of these performed studies, and whether there is downstream benefit to patient care. Subsequent studies should ensure that image quality, and not just trainee confidence, are considered as outcome measures.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.