Clinical Integration of Point‐of‐care Ultrasound by Emergency Medicine Residents: A Single‐center Mixed‐methods Study

Abstract

Background

Point‐of‐care ultrasound (POCUS) competence consists of image acquisition, image interpretation, and clinical integration. Limited data exist on POCUS usage patterns and clinical integration by emergency medicine (EM) residents. We sought to determine actual POCUS usage and clinical integration patterns by EM residents and to explore residents' perspectives on POCUS clinical integration.

Methods

We conducted an explanatory sequential mixed‐methods study at a 4‐year EM residency program. In phase 1, EM ultrasound (US) attendings observed PGY‐4 EM residents' clinical integration of POCUS in real time while on shift in the emergency department (ED). EM US attendings evaluated residents on their intent to perform POCUS, actual POCUS usage, and competence per patient encounter. We used logistic regression to analyze these parameters. In phase 2, we conducted semi‐structured interviews with the observed PGY‐4 residents regarding POCUS usage and clinical integration in the ED. We analyzed qualitative data for themes.

Results

Emergency medicine US attendings observed 10 PGY‐4 EM residents during 254 high‐acuity patient encounters from December 2018 to March 2019. EM US attendings considered POCUS indicated for 26% (66/254) of patients, possibly indicated for 12% (30/254) and not indicated for 62% (158/254). Of the 66 patients for whom EM US attendings considered POCUS indicated, PGY‐4s intended to perform POCUS for patient management 61% (40/66) of the time. PGY‐4s subsequently incorporated POCUS into patient management 73% (48/66) of the time. EM US attendings considered PGY‐4s entrustable to perform POCUS independently 81% (206/254) of the time. We did not find a statistically significant association between shift volume, shift type, or POCUS application, and resident intent to perform POCUS nor competence. Interviews identified three factors that influence PGY‐4's POCUS clinical integration: motivations to use POCUS, barriers to utilization, and POCUS educational methods.

Conclusions

This mixed‐methods study identified a significant gap in POCUS utilization and clinical integration by PGY‐4 EM residents for clinically indicated cases identified by EM US attendings. As clinical integration is a cornerstone of POCUS competence, it is important to ensure that EM resident POCUS curricula emphasize training on clinical utilization and indications for POCUS while on shift in the ED.

Since 2001, the Accreditation Council for Graduate Medical Education has recommended that graduating emergency medicine (EM) residents demonstrate competence in point‐of‐care ultrasound (POCUS). ¹ Since then, the Society for Academic Emergency Medicine (SAEM), the American College of Emergency Physicians (ACEP), and the Council of Emergency Medicine Residency Directors (CORD) have emphasized the importance of requiring POCUS competence for EM residents. ² , ³ The Society of Clinical Ultrasound Fellowships (SCUF) defines POCUS competence as “clustered skills of medical knowledge and technical aptitude to employ clinical ultrasound for optimized patient care and clinical outcomes.” ³ Most POCUS experts agree that POCUS competence consists of three distinct domains: image acquisition, image interpretation, and clinical integration. Accurate clinical integration builds on image acquisition and interpretation skills. Clinical integration includes both understanding when (and when not) to perform POCUS and how to integrate the findings into medical decision making. ² , ³ , ⁴ , ⁵ , ⁶ While SAEM, ACEP, and CORD have published recommendations for EM resident POCUS education and competency assessment, actual POCUS educational practices in the United States still vary widely. ³ , ⁷ , ⁸

Literature regarding best practices for education and assessment of resident image acquisition and image interpretation is well developed. ² , ³ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² Image acquisition and interpretation skills can be efficiently and accurately taught and assessed through asynchronous lectures, task simulators, scan shifts, case log review, multiple‐choice exams, image review, standardized direct observational tools (SDOTs) and observed structured clinical exams (OSCEs). ³ , ⁷ , ¹⁰ , ¹³ , ¹⁴

However, there is a dearth of literature describing best practices for education and assessment of resident clinical integration skills. The PC12 subcompetency in the EM Milestones document suggests that a graduating resident should “consistently utilize and integrate appropriate ultrasound applications into clinical management.” ¹ , ¹⁵ SAEM, ACEP, and CORD recommend longitudinal POCUS education during EM residency. ² These recommendations serve as a general benchmark but are difficult to implement as there is no specific nor standardized approach. ³ , ¹⁶ , ¹⁷

Competency‐based medical education suggests that trainees will develop image acquisition, image interpretation, and clinical integration competence at different rates throughout training and practice. ⁴ , ⁵ , ⁶ Attaining expertise in one domain does not predict expertise in another domain. ¹⁸ Furthermore, competence in image acquisition and interpretation is a necessary building block to develop accurate clinical integration. ¹⁹ Thus, a complete POCUS competency assessment should ideally be multifaceted and should collect data on resident image acquisition, image interpretation, and clinical integration skills. ² , ³ , ⁴ , ⁶ The tools currently suggested for assessment of clinical integration skills, SDOTs and OSCEs, lack important real‐life variability that complicates actual POCUS usage and integration. ² , ³ , ⁸ , ¹⁰ To our knowledge, no research has been performed regarding in situ observation of EM residents' POCUS utilization and clinical integration. In this study, we sought: 1) to assess PGY‐4 EM resident competence to use POCUS to diagnose and manage patients while on shift in the emergency department (ED) and 2) to explore resident perceptions on POCUS usage and clinical integration in the ED.

METHODS

Study Design

We conducted an explanatory sequential mixed‐methods study at an academic PGY1–4 EM residency program. We believed that a mixed‐methods approach would add depth to our conclusions and minimize the inherent limitations of using either quantitative or qualitative methods alone. ²⁰ , ²¹ A mixed‐methods design achieves this by broadening and triangulating findings discovered from both components. This is particularly important when studying a new topic in a complex and natural environment, as we did. ²¹ An explanatory sequential design involves a two‐phase methodology. Quantitative data are initially collected and analyzed. Quantitative results then inform purposeful selection of participants for the qualitative phase, which is used to shed light on the quantitative results. ²⁰ An explanatory sequential design was ideal for our project. We primarily gathered observational data regarding resident POCUS usage and then explored their opinions in order to understand the patterns of usage discovered in the quantitative strand. As such, our study was conducted in two phases: 1) real‐time observation of EM residents' use of POCUS while working in the ED and 2) qualitative assessment of these residents' perspectives on POCUS utilization and integration.

Study Setting

Our EM residency is a PGY1–4 academic program. Residency class size varies from 13 to 15 residents per year. Our ED sees an annual volume of 113,000 patients. Our hospital trains one to three EM ultrasound (US) fellows per year who work clinically as attendings in the ED. We conducted the study in the acute area of our ED. Triage to Acute, the highest‐acuity area, is determined by the EM triage nurse. The majority of patients triaged to Acute are Emergency Severity Index (ESI) level 1 to 3 with a high proportion being ESI level 1 and 2. ²² There are three 8‐hour Acute EM attending shifts per day. During each shift, the PGY‐4 and EM attending work jointly to evaluate and manage all patients and supervise junior providers (EM residents, EM physician assistants, and non‐EM rotating residents).

The US curriculum at our institution is currently multifaceted; however, at the time of this study, the PGY‐1 US rotation was the core of our POCUS curriculum. Residents have no clinical responsibilities during this 4‐week rotation. They attend twice‐weekly US conference with didactics and scan review. Residents additionally spend approximately 18 hours per week scanning patients in the ED with direct bedside supervision by an US faculty member or fellow. As PGY‐3s, residents complete an OSCE designed to assess their ability to acquire images, interpret images, and incorporate them clinically in all US applications. Any deficiencies are noted and retested in the PGY‐4 year. EM US faculty or fellows deliver US‐themed didactics approximately six times per year at EM resident conference. Residents are required to complete at least 300 scans by the end of residency.

Inclusion and Recruitment

All PGY‐4 residents who worked in the acute area of our ED during the study period were eligible for inclusion in the study. Prior to the study period, we notified residents via e‐mail that US faculty and fellows would be observing their usage of POCUS while on shift in the ED.

Quantitative Study

We investigated the following using a questionnaire:

• ‐Indications for POCUS as determined by EM US attendings;
• ‐Patterns of POCUS usage and clinical integration by PGY‐4 EM residents during clinical shifts;
• ‐Competence of PGY‐4 residents using POCUS to diagnose and manage critically ill patients.

Survey Development

To assess these objectives, we developed a survey instrument (see Data Supplement S1, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wiley.com/doi/10.1002/aet2.10463/full). We revised this survey after discussions with US faculty and medical education experts to ensure content validity. Four EM US attendings and two research associates (RAs) piloted the survey in the ED for 2 weeks prior to the study period. All four EM US attendings and the two RAs met to analyze and discuss pilot survey response inconsistences. We then revised the survey to agree upon content and format. It was specified that for POCUS to be “indicated,” the performance needed to be a critical diagnostic or management action for the patient. We included a three‐point behaviorally anchored global competence assessment in the survey. This was based on the International Federation for Emergency Medicine (IFEM) framework for POCUS assessment. ⁴ We adapted this framework to specify competence in critically ill patients. We used the terminology entrustable, pre‐entrustable, and not entrustable with a priori agreed‐upon definitions. We agreed that a resident's competence would be assessed per patient encounter. As POCUS competence naturally builds from basic knowledge to image acquisition and interpretation then clinical integration, we were not able to separate clinical integration competence from the other domains. ¹⁹ As per the 2016 revised POCUS milestones, we specified that for a resident to be entrustable they should “consistently utilize and integrate appropriate ultrasound applications into clinical management” without assistance. ¹⁵ We intended the survey development process and the data collection process detailed below to strengthen validity by providing content and response process evidence. ²³

Figure 1.

POCUS applications deemed clinically indicated in critically ill emergency patients. Assessments were determined by EM US attendings. More than one application could be selected per patient encounter. abdominal fluid = abdominal portion of the focused assessment with sonography in trauma examination; DVT = deep venous thrombosis; IVC = inferior vena cava; MSK = musculoskeletal; POCUS = point‐of‐care ultrasound; PVR = postvoid residual; US = ultrasound.

Data Collection and Analysis

Four EM US attendings performed observations while working clinical shifts in the acute area of the ED from December 2018 to March 2019. Two attendings were US faculty members, both of whom completed emergency US fellowships and have at least 10 years of clinical experience in EM. Two were fellows; one was a second‐year EM US fellow and the other was a first‐year EM US fellow.

The two RAs who were present during the survey discussions administered the survey to the faculty and fellows while on shift in an effort to maintain response consistency. RAs were typically present in the ED for other research studies and worked closely with attendings, so their presence was not unusual. We collected data on each patient who presented to the acute area of the ED during each shift. As per usual, the PGY‐4 presented each patient to the EM US attending, including their diagnostic, treatment, and management plan. The EM US attending then separately evaluated the patient. The RA then asked the EM US attending if POCUS was indicated, possibly indicated, or not indicated. For each case, the RA asked the attending if the PGY‐4 had planned to perform POCUS (presented it as part of their initial plan), planned to perform POCUS only after prompting, or had no plan to perform POCUS for the patient's diagnosis or management. All EM US attendings agreed a priori that if the resident did not mention POCUS in the initial plan, and it was indicated, they would prompt by saying “What about ultrasound?”. The RA additionally tracked whether or not the POCUS was performed and by whom. As is natural for patients requiring active resuscitation, the EM US attending and the resident evaluated the patient simultaneously. It is typical in our residency for the PGY‐4 resident to guide the resuscitation and the attending to observe unless it is critical to intervene. During these cases, if the PGY‐4 resident did not use POCUS during the resuscitation, nor guide a junior provider to use POCUS, the EM US attending assumed that the PGY‐4 did not intend to use POCUS. We determined a priori that we would debrief post‐resuscitation with the resident to confirm these assumptions and clarify intent.

We collected and managed study data using the REDCap electronic data capture tool hosted at Partners Health System (www.projectredcap.org). We exported data to Microsoft Excel Version 16.23 (2019) and Stata MP 15.1 (StataCorp 2017). We report descriptive statistics as ratios and percentages. We analyzed quantitative data using logistic regression for the dependent variables POCUS indication, resident intent to perform POCUS, and competence, controlling for shift volume, shift time, and attending. We used chi‐square analysis to determine if there was a statistically significant difference between the frequency of each attending's assessments for each dependent variable. We intended the chi‐square analysis to augment internal structure validity for our construct. Individual resident performance was controlled using random slopes for each resident.

Qualitative Study

We conducted a qualitative arm to add depth to the analysis and to attempt to explain patterns discovered during quantitative analysis. ²⁰ We followed a pragmatic worldview, wherein individual researchers have the freedom of choice to use all research approaches available in order to provide the best understanding of the research problem at hand. ²⁰ We used a generic qualitative approach as our study design did not fit specific qualitative methodologies. ²⁰ We invited all residents to interview. We scheduled six residents for interviews who were felt to be an equal representation of the cohort of 10 residents including all possible demographics and viewpoints.

Data Collection

An US fellow with training in qualitative methods conducted six semi‐structured approximately 20‐minute interviews with the PGY‐4s. The US fellow had worked clinically with the PGY‐4s for 2 years at the time of the interviews. We started the interview with open‐ended questions about the PGY‐4's relationship with POCUS during shift and then adapted the interview accordingly to explore the results discovered in the quantitative arm. The interviews were recorded using an encrypted iPhone XS and its proprietary voice memo application (Apple Inc., 2019). After recording, the US fellow deidentified the interviews and transcribed them using the transcription software TranscribeMe! (TranscribeMe Inc., Version 3.1.1). The US fellow reviewed the transcripts for accuracy.

Data Analysis

Two EM US fellows, one of whom conducted the interviews, reviewed the transcripts in depth to identify segments of data that pertained to specific concepts or topics (codes). Both fellows performed all steps of qualitative data analysis. They performed open coding first to obtain as many codes as possible that aligned with the study questions. After completing open coding, the investigators attempted to group codes into coding categories. Repeating categories formed the basis of thematic analysis and key themes were identified. We performed data collection and analysis concurrently to inform future data collection. We modified interview questions for clarification or further exploration of opinions. To maximize confirmability, we met three times to review themes and reflect on limiting bias. The entire US research group additionally met to perform peer debriefing on themes. As there is no literature on this subject, we were not able to find evidence that confirmed or disconfirmed our findings. We performed member checking via e‐mail to ensure that themes identified by the investigators accurately reflected participant narratives. We e‐mailed the qualitative results to all six residents who were interviewed. We asked them to review and report back as to whether or not the data accurately reflected their opinions. We received feedback from four of six residents that the narrative included an accurate reflection of their views. We did not hear back from the rest (2/6).

The interviewer was not in a position of power over participating residents because the interviews were conducted in the final few weeks of the final month of residency and fellowship. At that point, the fellow had no further evaluation nor supervisory role with the residents. The interviewer ensured confidentiality, used open‐ended questions to allow for a variety of opinions including those that contradicted the majority opinion, and explored data to identify themes that were unexpected or contradicted the norm. The study was deemed exempt by the institutional review board.

RESULTS

Quantitative

Emergency medicine US attendings observed 10 PGY‐4 EM residents from December 2018 to March 2019. These observations occurred during the 15 consecutively scheduled shifts for the four EM US attendings in the acute area. There were 254 patient encounters during the study period. Shift volume ranged from eight to 28 patients per 8‐hour shift.

Indications for POCUS

Point‐of‐care US was considered indicated by the EM attending for 26% of patients. Figure 1 highlights application type selected for the clinically indicated cases. (Multiple applications could be selected as indicated for each patient encounter.)

Resident Intent to Use POCUS

When POCUS was indicated, residents mentioned POCUS in their plan without prompting 61% of the time. (Table 1).

Table 1.

Resident Competence Assessment, POCUS Utilization and Clinical Integration Patterns in Critically Ill Patients

POCUS Indicated?*	Yes	Maybe	No	Total
	26% (66)	12% (30)	62% (158)	254
Resident intent†
Yes, without prompting	61% (40/66)	3% (1/30)		16% (41/254)
Yes, with prompting	17% (11/66)	3% (1/30)	1% (1/158)	5% (13/254)
No	23% (15/66)	93% (28/30)	99% (156/158)	78% (199/254)
Left blank‡			1	1
Competence§
Entrustable, can perform independently	68% (45/66)	57% (17/30)	91% (144/158)	81% (206/254)
Pre‐entrustable, requires some assistance	24% (16/66)	37% (11/30)		11% (27/254)
Not entrustable, cannot perform without complete assistance	6% (4/66)			2% (4/254)
Left blank‡	1	2	14	17
Was POCUS performed?
Yes	29% (19/66)	0% (0/30)		20% (19/96)
Yes, but not by PGY‐4||	61% (40/66)	27% (8/30)		50% (48/96)
No	11% (7/66)	73% (22/30)		30% (29/96)
Was POCUS incorporated into patient care?
Yes	73% (48/66)
Required some prompting	2% (1/66)
No	12% (8/66)
Left blank‡	9

POCUS = point‐of‐care ultrasound; US = ultrasound.

^*As assessed by EM US attending (χ² = 10.1229, p = 0.120).

^†As assessed by EM US attending (χ²⁼7.8292, p = 0.05).

^‡Blank responses were not filled out by attending on shift for unknown reasons.

^§As assessed by EM US attending (χ² = 39, p = 0.00).

^||This included junior residents or attending.

Assessment of Competence

The PGY‐4 was considered entrustable to use POCUS to diagnose and manage critically ill patients in the ED in 81% of patient encounters. (Table 1).

Resident Use and Incorporation of POCUS

When POCUS was indicated, POCUS was actually performed 89% of the time (by PGY‐4 or other provider) and subsequently incorporated into PGY‐4's patient management 74% of the time (Table 1). We determined “incorporation into practice” by asking if the resident used POCUS findings to explicitly change or confirm management or disposition decisions, which we ascertained from the case discussion with the PGY‐4.

Regression Analysis

POCUS Indication

Chi‐square analysis did not detect a difference between individual attending assessment of indication (Table 1). We did not detect an association between attending assessment of indication and shift volume. Attendings were less likely to say POCUS was indicated than not indicated for the 3 pm to 11 pm shift versus the 7 am–3 pm shift (odds ratio [OR] = 0.40, 95% confidence interval [CI] = 0.17 to 0.97, p < 0.05).

Resident Intent to Use POCUS

There was a difference detected between individual attending assessment of resident intent to use POCUS (χ² = 7.8292, p = 0.05; Table 1). However, when using logistic regression to control for resident, shift volume, and shift time, this difference was not detected. The only exception was that attending 4's assessment of POCUS indication aligned with the residents’ assessment of indication significantly more than attending 1 (OR = 0.11, 95% CI = 0.013 to 0.96, p < 0.05)). We did not find a statistically significant association between shift volume, shift time, or application indicated, and resident intent to use POCUS.

Competence

There was a difference detected among attending assessment of competence (χ² = 39, p < 0.01; Table 1). However, when using logistic regression to control for resident, shift volume, and shift type this difference was not detected. We did not find a statistically significant association between competence and shift volume, shift type, attending, or application indicated.

Qualitative

We conducted six resident interviews during May 2019. Three PGY‐4s were female and three were male. Upon graduation, three entered community practice and three entered academic fellowships (one of these was an US fellowship). We felt that this was representative of the cohort of the 10 residents who were observed, because four of 10 (40%) went into community practice. Three residents were above the median for considering POCUS without prompting and three were below the median.

Based on analysis of interview transcripts, we grouped factors that influence POCUS clinical integration by PGY‐4s under the following three categories:

1. Motivations to use POCUS;

2. Barriers to utilization;

3. How they learn clinical integration of POCUS

Under each category, we describe key themes and define the themes backed by representative quotes from PGY‐4s.

Motivations to Use POCUS

Under this category, we include themes that relate to perceived indications for residents’ use of POCUS during Acute ED shifts.

Immediately Rule In or Out diagnosis

Participants stated that POCUS helped with time management because of its ability to rapidly rule in or out diagnoses at the bedside.

I don't have the luxury of going back in the room or re‐visiting the [patients'] data … when I'm managing 25 sick patients … it [US] has become part of my pattern recognition.

For Sick Patients With Specific Chief Complaints

Residents emphasize that POCUS is most powerful as a diagnostic tool when the patient is “sick” (abnormal vital signs and/or clinically appearing unwell). They additionally emphasize that they most frequently use POCUS for sick patients with cardiopulmonary chief complaints and/or undifferentiated hypotension.

Anytime that EMS brings [a patient] in on BIPAP I get the US in the room. I find the difference between diuretics/nitro and albuterol in someone who is already panicking is pretty important.

Risk Stratify Patients to Guide Further Treatment/Management

Residents explain that POCUS can be useful for patients who do not have a clear diagnosis from initial history and physical examination. They use POCUS to risk stratify these patients to guide imaging, medication, or disposition decisions.

The diagnosis was kind of up in the air and we ended up sending the patient, intubated, to the CT scanner where they decompensated. Maybe, if I'd seen that big dilated right ventricle, I would've thought twice about intubating that person.

One resident notably stood out from the rest when they remarked they use US “All the time. US is benign. There is no downside to looking. Never have I found an US worked against me.”

Barriers to POCUS Utilization

Under this category, we include themes that relate to perceived barriers to residents' use of POCUS during acute ED shifts.

Lack of Evidence for Clinical Utility

PGY‐4s emphasize only using POCUS when there is literature to support its utility. They specifically mention the lower predictive value of POCUS for patients who clinically appear well with stable vitals and the lack of evidence supporting POCUS usage in stable patients with abdominal trauma, and for volume status assessment.

I try not to FAST young stable people because that is a misapplication of the research and we shouldn't do it and I think we actually (CT) scan too many young people as a result.

Perception That POCUS Would Slow Down Other Processes

PGY‐4s mention needing to appropriately allocate resources, especially time, when patient volume is high and that doing POCUS often takes time way from other important tasks.

… it's a trade‐off. US … would take time, may not … change what I want to do … I already know I'm going to admit this patient … [US] would slow down … other … processes … like reviewing … with the juniors or making sure that people get beds earlier or finishing notes.

Lack of Confidence in Clinical Integration Skills

PGY‐4s mention that they do not use POCUS due to a lack of confidence in their own clinical integration skills. In fact, most are still unsure if they are over or underutilizing POCUS and express a desire to hone this skill.

I have to ask myself, “am I using it too much and perhaps not in the right context?” And then will I be identifying things I was never looking for that might lead me own a diagnostic pathway that was sort of completely erroneous?

How PGY‐4s Learn Clinical Integration

Under this category, we include themes that relate to perceived factors that contribute to how PGY‐4s learn POCUS clinical integration skills.

“On‐the‐job” Development of Personal Practice Preferences

Participants explain that they develop their US practice style through personal struggles with time and resource management, and self‐reflection while on shift.

No one ever taught me. I was doing a lot of unnecessary ultrasounds to begin with for fun, and I kept being like “What is it changing? … If it's not going to change my clinical management then I should not be wasting my time on something useless and try to be more efficient with what I spend my time on.

Formal Teaching Focuses on Image Acquisition and Interpretation Rather Than Clinical Integration

Residents mention that time during the intern US rotation is spent learning image acquisition and interpretation skills rather than clinical integration skills, which they mention are distinctly different skills.

When we do our US rotation, we go around and we US … people that are separate from clinical care, and so I think the crux is really integrating it. It's one thing to learn how to do the scans … but then to try to integrate … I think is the hard thing.

Learn by Example from US Attendings, Not Through Explicit Discussion

Residents note that they mostly learn POCUS clinical integration from watching US attendings incorporate US into their patient workup and management. Residents note that there is not an explicit discussion of POCUS test characteristics nor indications for use.

It's a real cultural difference … when you have [an attending] who not only is encouraging residents to do ultrasound but is confident in interpreting the results … You can be in a dry spell … working with faculty who aren't as facile with US who aren't prompting you …, and then you get an attending who … [is] prompting you … Then that kind of clicks in our brain.

The minority mention that some younger faculty and critical care faculty additionally model clinical integration of POCUS. In addition, two residents mention their own self‐study outside of shift guiding their clinical integration education.

DISCUSSION

To our knowledge, our study is the first to publish actual data regarding EM resident POCUS usage and clinical integration while on shift. In a survey of 539 EM residency graduates from 18 different residency programs, residents expressed that the only reliable method of POCUS competency assessment was through in situ observation and that other methods (case log review, quality assurance, SDOTs, OSCEs) were all “poorly representative” of their skills. ¹³ Clinical reasoning experts additionally emphasize the importance of real‐time workplace‐based assessment of trainees as the workplace is the “core of medical competence.” ²⁴

At our institution, all residents are considered competent to perform POCUS by the end of their PGY‐3 contingent on successful completion of at least 300 expert‐reviewed scans and passing an US OSCE. All of the observed PGY‐4s in our study met these criteria. While clinical integration is naturally addressed during POCUS education, there is no deliberate curriculum for clinical integration at our institution. Formal assessment of clinical integration is not currently part of our resident POCUS evaluation. Therefore, little has been known about actual POCUS clinical integration by residents, their accuracy of clinical integration, nor the efficacy of our current curriculum.

After observing our residents, we identified an important gap between when EM US attendings and PGY‐4 residents consider POCUS indicated. We additionally observed that PGY‐4s performed POCUS and integrated it into clinical management less often than we thought was indicated. PGY‐4s mentioned POCUS in their initial plan 61% of the time when POCUS was deemed indicated by EM US attendings. Our on‐shift observations demonstrated that PGY‐4s were competent to perform POCUS independently (in critically ill patients) only 81% of the time. These findings have encouraged us to reconsider our POCUS curriculum and to be more deliberate about teaching and assessing POCUS clinical integration to our residents.

While we can make no conclusions about other residency programs, we hope that this information may also inspire other POCUS educators to deliberately consider their clinical integration curriculum. This deliberate consideration is especially important in light of the fact that we still lack best practice guidelines for POCUS clinical integration curriculum and assessment. ² , ³ , ⁴ , ¹⁵ One Canadian residency program has developed a POCUS curriculum rooted in the theory of deliberate practice, including reflection and feedback during clinical shifts. ²⁵ Their published data only reports image acquisition and interpretation skills, not on‐shift clinical integration skills. Given the importance of in situ observation for competency assessment, ³ , ²⁴ we hope that our findings further inspire the development of a deliberate practice clinical integration curriculum in the United States and a rigorous on‐shift competency assessment tool for clinical integration skills.

As learners’ perspectives are critical for developing effective instructional design, we attempted to explore what guides PGY‐4s’ current understanding of POCUS clinical integration through our qualitative phase. ²⁶ Our study is the first qualitative description of EM residents’ perceptions on POCUS usage and clinical integration. We identified key factors that contribute to PGY‐4 EM resident POCUS clinical usage and the manner in which PGY‐4s learn how to clinically integrate POCUS at our program. Our qualitative data helps us to understand what guides residents' POCUS clinical utilization and integration so that we can more effectively facilitate the development and elaboration of frameworks that are more attuned to reality. ¹⁸ While our findings may not be generalizable to other programs, we believe that this information may resonate with other POCUS educators and hope that it will encourage deliberate consideration of residents’ perspectives.

Finally, our study is the first to identify actual on‐shift indications and practice patterns for POCUS usage by a small group of EM US experts. This is important information that will be incorporated into our future clinical integration curriculum. Furthermore, it could be of interest to other POCUS educators or clinicians who use POCUS to diagnose and manage critically ill patients to learn of the actual practice patterns of our small group of POCUS enthusiasts.

LIMITATIONS

This study has many limitations. Our findings are most applicable to our residency program. Generalizability may be limited to only 4‐year academic programs with an US fellowship. Aside from its single‐center nature and limited generalizability, on‐shift assessment has inherent limitations and is subject to many biases and questionable reliability. ³ , ¹⁷ Given that attendings were documenting their interpretation of resident intent to perform POCUS, we do not know the residents’ actual thoughts. We believed that asking residents their opinions would have resulted in significant bias via the Hawthorne effect and would not have been feasible given the significant time and resource demands while on shift.

It is noted that only EM US attendings determined whether POCUS was clinically indicated and not all EM attendings. We felt that only EM US faculty members and fellows reliably possessed the requisite knowledge and experience to assess POCUS clinical integration skills. We acknowledge that there may be a bias toward a perception that US is indicated as EM US faculty are likely to incorporate US into clinical decision making more often than EM faculty without specific US training. We attempted to strengthen validity through methods mentioned, but did not perform a traditional inter‐rater reliability for internal structure. We felt that it would not be feasible to review all 254 charts to calculate a kappa for inter‐rater reliability, especially as this would lack the important real‐life variables of shift volume and patient flow that often impact the decision to perform POCUS or not. We attempted to maximize response process and internal structure through the use of RAs. We understand that most programs do not have access to RAs, which further limits generalizability and reproducibility of our conclusions. We additionally did not assess consequences of our testing nor provide a significant amount of evidence in the form of relation to other variables. ²³

While we attempted to maximize methodologic rigor in the qualitative arm, we did not interview all residents and therefore may have missed other opinions. Finally, interviews were performed by one of the EM US fellows, instead of an independent third party, which could have led to bias in the interview responses.

CONCLUSIONS

In this mixed‐methods evaluation of residents’ perceptions and actual point‐of‐care ultrasound performance while on shift, we identified a significant utilization and integration gap by our PGY‐4 emergency medicine residents for clinically indicated cases identified by emergency medicine ultrasound attendings. We believe this study helps to further our understanding of how to teach the critical skill of point‐of‐care ultrasound clinical integration and how to assess emergency medicine residents' point‐of‐care ultrasound competence. We believe that this study highlights the importance of deliberate and proper training in point‐of‐care ultrasound clinical integration. We additionally provide important and novel perspectives from emergency medicine residents regarding point‐of‐care ultrasound clinical utilization and integration.

This information will be used to guide curricular improvement locally at our institution. We hope that it inspires other point‐of‐care ultrasound educators to assess their residents in situ in the ED and to deliberately consider their clinical integration educational and assessment methods.

Supporting information

Data Supplement S1. Attending survey.

AEM Education and Training 2020;4:212–222