Accuracy of focused cardiac ultrasound interpretation among emergency and critical care medicine residents in Ethiopia: A multi-center cross-sectional study

Abstract

Introduction

Focused cardiac ultrasound (FoCUS) has emerged as a valuable tool in emergency and critical care medicine, allowing for rapid assessment of cardiac function and structure at the bedside. This rapid diagnostic technique holds particular promise in resource-limited settings like Ethiopia, where access to standard echocardiography may be limited and delayed. However, the accuracy of FoCUS interpretation is highly dependent on the operator's skills and expertise. To inform the design of effective interventions, the study aimed to assess the accuracy of FoCUS interpretation and associated factors among senior Emergency Medicine residents at two large referral teaching hospitals in Ethiopia.

Methods

A cross-sectional study was conducted from October to December 2023 among 80 residents at Tikur Anbessa Specialized Hospital and St. Paul's Hospital Millennium Medical College. To assess diagnostic accuracy, 15 pre-selected cardiac ultrasound videos (normal and pathological cases) were selected from American College of Emergency Physicians website and the PoCUS Atlas, and accurate interpretation was defined as correctly answering at least 12 out of 15 readings. A binary logistic regression model was fitted to identify significant factors at the 5% level of significance, where significant results were interpreted using adjusted odds ratio (AOR) with 95% confidence interval (CI).

Result

The overall accuracy in interpreting FoCUS findings was 47.5% (95% CI: 38.8–60.0%), with highest for collapsing Inferior Vena Cava (91.3%) and standstill (90.0%), and lowest for Regional Wall Motion Abnormality of Left Ventricle (46.3%). Residents who received training (AOR=4.14, 95%CI:1.32–13.04, p = 0.015), perceived themselves as skilled (AOR=4.81, 95%CI=1.06–21.82, p = 0.042), and felt confident in acquiring and interpretation (AOR=3.16, 95%CI=1.01–9.82, p = 0.047) demonstrated significantly higher accuracy.

Conclusion

The study identified a low overall accuracy in FoCUS interpretation, with accuracy improving with training and better perceived skill and confidence. Training programs with simulation, continuous education, and mentorship are crucial to enhance these critical skills.

African relevance

• •Limited access to specialists and equipment for standard echocardiography in Africa makes FoCUS a more feasible alternative for urgent cardiac emergencies.
• •Training Emergency Physicians (EPs) in FoCUS can significantly enhance emergency cardiac care in Africa where EPs are more readily available compared to cardiologists.
• •There is a lack of integration of FoCUS training programs into the curriculum for EPs.

  Assessing FoCUS competency in African setting helps to inform the design of effective training programs for future emergency physicians.

• •Implementing FoCUS training for EPs has the potential to improve patient survival and decision-making during cardiac emergencies in African settings.

Introduction

Focused Cardiac Ultrasound (FoCUS), a streamlined echocardiography application performed by physicians, is rapidly gaining popularity, particularly in Emergency Departments (EDs) [1,2]. While conventional echocardiography is crucial for many time-sensitive cardiovascular emergencies, it may not be widely available or readily accessible, leading to significant delays [3]. To address this, it is recommended to use FoCUS in EDs and critical care units to expedite decision-making and improve patient survival [[4], [5], [6]].

With advancements in technology and increasing expertise, FoCUS is becoming the standard approach in both EDs and Intensive Care Units (ICUs) [2,[7], [8], [9], [10]]. Despite the life-saving potential of FoCUS and its established role in emergency settings, many low- and middle-income countries (LMICs) struggle to provide essential echocardiographic examinations for critically ill patients. This is due, in part, to a limited number of cardiologists, cardiac sonographers, and well-trained Emergency Physicians (EPs) [11].

Given that FoCUS is a critical skill for EPs, who are more readily available in the EDs, teaching them to perform FoCUS has the potential to significantly improve care during urgent or pressing cardiac emergencies. Studies have shown that with proper training and technological facilities, non-cardiologists working in emergency settings can acquire FoCUS competency on par with cardiologists [[12], [13], [14]]. As a result, it is recommended that Emergency and Critical Care Medicine (ECCM) residency programs provide structured training in ER ultrasonography (15). The American College of Emergency Physicians (ACEP) Council of Directors Policy Statement advises that emergency medicine residents undergo 40 hours of training, which includes completing 150 ultrasonography tests, 25 to 50 of which are focused on cardiac investigations [15]. While the utility of this training is presumably greatest in LMICs, it has not been widely implemented in these regions [1,[16], [17], [18]]. Studies also show that there is a significant performance gap in the ability of emergency physicians to provide accurate and timely diagnosis of cardiovascular conditions in emergencies [17,[19], [20], [21], [22]].

In Ethiopia, efforts are underway to launch FoCUS training programs and integrate them into the curriculum for ECCM residents but it is at its early stage [23]. While no study has yet assessed the competency of FoCUS interpretation among ECCM residents in Ethiopia, studies on Point-of-Care Ultrasound (POCUS) and Electrocardiogram (ECG) skills of residents in similar study settings revealed that only about one-third claimed to be proficient in performing POCUS and none of them could accurately interpret ECG findings. The studies identified a lack of adequate equipment, training opportunities, and a defined curriculum as significant factors for the skill gap [24,25]. Therefore, to inform the design of effective interventions in this setting, the study aimed to investigate the diagnostic accuracy of FoCUS interpretation and associated factors among ECCM senior residents at two large referral teaching hospitals in Ethiopia.

Methods

Study Setting and Design

A cross-sectional study was conducted from October to December 2023 at two large tertiary referral and teaching hospitals in Ethiopia: Tikur Anbessa Specialized Hospital (TASH) and St. Paul's Hospital Millennium Medical College (SPHMMC). These two hospitals are pioneers among the seven hospitals in the country that provide a three-year emergency and critical care medicine (ECCM) specialty training program [26]. In addition to their regular EM residency training, the institutions offer a separate training program for residents in EM PoCUS, which also includes FoCUS. The program is primarily provided by the Toronto Addis Ababa Academic Collaboration in Emergency Medicine (TAAAC-EM), established in 2010 as an educational partnership between Addis Ababa University and the University of Toronto. The training materials and scanning techniques are based on the Canadian Emergency Department Echo (EDE) program. The trainings typically last one month and are offered six times a year. The program is delivered by a team comprised of faculty and residents from the University of Toronto. It has been a self-sustainable curriculum since February 2020 [26].

The conduct and findings of the study is reported following the STARD (Standards for Reporting Diagnostic Accuracy Studies) guideline for reporting diagnostic accuracy studies [27].

Population and Sample Size

All senior ECCM residents, second-year residents who had completed the first six months of their training and all third-year residents, who were attending their residency training at the two hospitals from October to December 2023, were included in the study. A total of 91 senior residents were identified as eligible for the study, 44 from TASH and 47 from SPHMMC.

Operational Definitions

Accuracy in interpreting FoCUS: Diagnostic accuracy (overall agreement) in FoCUS interpretation was defined as correctly interpreting at least 80% of the total FoCUS readings shared. That means, a cut-off point of 12 correct readings out of the 15 FoCUS readings was applied [28].

Skill in using FoCUS: Proficiency in patient positioning, probe manipulation, image acquisition, and identification of relevant cardiac structures and abnormalities which encompasses the ability to apply appropriate ultrasound techniques and protocols for specific clinical indications and accurately perform measurements and calculations related to cardiac function [29].

Level of confidence in acquiring and interpreting FoCUS images: The level of confidence in the process of skillfully obtaining clear and accurate ultrasound images of the heart and analyzing them to identify cardiac structures, functions, and abnormalities [30].

Data Collection Procedures and Quality Assurance

Data was collected using a self-administered, pre-tested, standardized electronic tool containing basic information about the residents' sociodemographic characteristics, FoCUS training, skills, confidence, experience, and diagnostic accuracy assessment. To assess diagnostic accuracy, the residents were asked, in an open-ended format, to interpret the findings from 15 cardiac ultrasound videos that encompassed a diverse range of normal cases and pathologies focusing on typical life-threatening situations in the emergency room were chosen from the official website of American College of Emergency Physicians (ACEP) and The PoCUS Atlas (TPA) with permission [31,32]. The ACEP website states that this web-based ultrasound guide aims to expose more emergency physicians to this valuable diagnostic tool. The cardiac section, located under the basic section, provides a series of standardized images and videos of normal views and various pathologies with clear visibility. The PoCUS Atlas is an open-source, collaborative ultrasound education platform. This unique platform allows users to create, share, and curate free ultrasound educational content contributed by global educators. The sites host a plethora of videos produced by reputable and qualified experts in the field of POCUS, including members of the leadership and editorial teams.

From these websites, 15 videos were selected for this study based primarily on proper visibility of normal and pathological findings, ensuring inclusivity of a wide range of variations using the Canadian EDE program as a guide. Accordingly, the selected examinations included a normal subxiphoid view, normal Apical-4-chamber (A4C) view, normal Parasternal Long Axis (PLAX) view, normal Parasternal Short Axis (PSAX) view, normal Inferior Vena Cava (IVC) views, IVC with respiratory collapse, large IVC with minimal respiratory variability, Right Ventricle (RV) thrombus on A4C view, a subxiphoid video of an effusion of the pericardium with tamponade physiology, PLAX view of hyperdynamic systolic function, PLAX view of reduced ejection fraction, A4C view of RV dilation and septal bowing, Regional Wall Motion Abnormality (RWMA) of Left Ventricle (LV), McConnell's sign, and cardiac standstill. After verifying the completeness and consistency of each entry, the data was exported to SPSS software version 25.0 for data management and analysis.

Statistical Analysis

To summarize participant characteristics, frequency and proportion were used for categorical variables. For numeric variables, median with interquartile range (IQR) was used after testing for normality of data distribution with the Shapiro-Wilk test, where a p-value < 0.05 indicated skewed data.

The diagnostic accuracy of each of the 15 FoCUS readings was measured as the percentage of agreement with the gold standard reference cardiac ultrasound readings and presented using a bar chart. The overall diagnostic accuracy of FoCUS interpretation was calculated as the percentage of agreement with the gold standard, which was defined as correctly answering at least 12 out of 15 readings from all assessments.

To identify factors associated with the accuracy of FoCUS interpretation, a multivariable binary logistic regression model was fitted at the 5% level of significance, including variables identified as significant on univariate analysis at the 25% level of significance, as well as those found to be scientifically relevant. Significant results (p-value < 0.05) were interpreted using adjusted odds ratio (AOR) with 95% CI. The adequacy of the final model was evaluated using the Hosmer and Lemeshow test, which revealed that the data fit the model adequately with a p-value of 0.605.

The study was conducted after receiving ethical approval from the Ethics Review committee of the Department of Emergency Medicine under Addis Ababa University, College of Health Sciences Institutional Review Board (Ref. No. EM/552/2015). The study was carried out in accordance with relevant guidelines and regulations. Written informed consent was obtained from all participants. To safeguard confidentiality, data collection tools remained anonymous by omitting participants names and identifiers, and access to the collected data was strictly limited to the research team.

Results

Participant Characteristics, FoCUS Training, and Utilization

Of the 91 eligible senior residents, 80 provided a complete answer to the shared questionnaire and were included in the final analysis, including 41 from TASH and 39 from SPHMMC. The median age of the participants was 30.0 years (IQR= 29.0–31.0 years), and 56 were males. Additionally, 45 were in their final year of residency.

A total of 44 residents completed an online or in-person training on FoCUS interpretation during their residency program, with all of them claiming to have attended all sessions. Of these, 42 were third-year residents. Furthermore, 27 took the training in their second year, and the remaining 17 took it in their first year. There was no training offered during the third year of residency. Additionally, 43 out of the 44 trainees who completed the course had taken it in the year preceding the data collection. While none of the participants considered the training adequate, 32 said it was insufficient, and the remaining 12 thought it might be enough.

Regarding their self-perceived ability to use ultrasound in the ED, 41 participants strongly agreed and 25 agreed that they possessed the necessary skills. Thirteen participants remained neutral, and only one disagreed. Most participants reported using FoCUS for four or more cases during the day, while for less than four cases at night.

Seventy-seven residents correctly answered that five fundamental views in adult cardiac ultrasonography could be acquired successfully. Regarding their level of confidence in acquiring and interpreting proper FoCUS images, 11 residents reported feeling very confident, 19 somewhat confident, 15 neutral, 25 somewhat unconfident, and 10 very unconfident.

Forty residents indicated that they frequently sought assistance with interpreting FoCUS scans, with 16 seeking help "always" and 24 seeking help "often." The most frequent requests for assistance were directed towards senior residents (73) and ECCM consultants (70), followed by peers (62), cardiologists (47), and radiologists [22]. (Table 1)

Table 1.

Participant characteristics, FoCUS training, and utilization of ECCM senior residents at two large referral hospitals in Ethiopia, October to December 2023 (n = 80).

Variable	Frequency	Variable	Frequency
Age category (Median, IQR)	30.0 (29.0–31.0)	Number of patients FoCUS applied on for one duty in nighttime
Gender		1	10
Male	56	2	35
Female	24	3	28
Year of residency		4	6
2nd year	35	>4	1
3rd year	45	Number of fundamental views that can be acquired from FoCUS
Took FoCUS training		4	3
No	36	5	77
Yes	44	Confidence in acquiring and interpreting FoCUS
Time of FoCUS training		Very confident	11
1st year	17	Somewhat confident	19
2nd year	27	Neutral	15
Was the FoCUS training adequate?		Somewhat not-confident	25
No	32	Very not-confident	10
Maybe	12	Frequency of seeking assistance for FoCUS Interpretation
Have skills to use FoCUS		Always	16
Strongly agree	41	Often	24
Agree	25	Sometimes	20
Neutral	13	Rarely	20
Disagree	1	Never	0
Strongly disagree	0	Source of assistance
Number of patients FoCUS applied on for one duty in daytime		Senior residents	73
1	4	Consultants	70
2	6	Radiologists	23
3	10	Cardiologists	47
4	17	Peers	62
>4	35

FoCUS - focused cardiac ultrasound, ECCM -Emergency Medicine and Critical Care Medicine.

Accuracy of FoCUS interpretation

Based on the analysis of the 15 FoCUS findings, 38 residents correctly identified 12 or more readings, resulting in an overall diagnostic accuracy of 47.5% (95% CI: 38.8–60.0%). Accuracy was highest for collapsing IVC (91.3%) and standstill (90.0%), and lowest for RWMA of LV (46.3%). (Fig. 1)

Fig. 1.

Diagnostic accuracy of the 15 FoCUS readings among ECCM senior residents at two large referral hospitals in Ethiopia, October to December 2023 (n = 80).

Factors associated with accuracy of FoCUS interpretation

After adjusting for perceived skills in using FoCUS, confidence in acquiring and interpreting FoCUS images, and frequency of seeking help with interpretations, the odds of delivering an accurate FoCUS reading was four times higher for residents who received FoCUS training during their residency program compared to those who did not take the training (AOR = 4.14, 95% CI: 1.32–13.04, p = 0.015).

Furthermore, after adjusting for other variables, the odds of delivering an accurate FoCUS reading among residents who perceived themselves as having good skills in using FoCUS and those who reported to have confidence in acquiring and interpreting FoCUS were 4.81 (AOR=4.81, 95% CI=1.06–21.82, p = 0.042) and 3.16 (AOR=3.16, 95% CI=1.01–9.82, p = 0.047) times higher than those who did not perceive to have good skills and lacked confidence, respectively. (Table 2)

Table 2.

Factors associated with accuracy of FoCUS interpretation among ECCM senior residents at two large referral hospitals in Ethiopia, October to December 2023 (n = 80).

Variables	Accurate Diagnosis		COR (95% CI)	AOR (95% CI)	p-value
	Yes	No
Took FoCUS training
No	12	24	1	1
Yes	26	18	2.89 (1.15–7.23)	4.14 (1.32–13.04)	0.015*
Have skills to use FoCUS
Neutral/Disagree	3	11	1	1
Agree	35	31	4.14 (1.06–16.21)	4.81 (1.06–21.82)	0.042*
Confidence in interpreting FoCUS
Neutral/Not-confident	20	30	1	1
Confident	18	12	2.25 (0.89–5.67)	3.16 (1.01–9.82)	0.047*
Frequency of seeking assistance for FoCUS Interpretation
Sometimes/rarely	13	27	1	1
Always/ often	25	15	3.46 (1.38–8.69)	2.12 (0.76–5.93)	0.151

COR= Crude Odds Ratio, AOR= Adjusted Odds Ratio,.

^⁎= statistically significant.

Discussion

This study aimed to assess the accuracy of FoCUS interpretation and associated factors among 80 ECCM senior residents from two large referral hospitals in Ethiopia. The overall accuracy in interpreting FoCUS findings was 47.5% (95% CI: 38.8–60.0%). This is a low level of overall accuracy where picking findings in the emergency setting is critical to make result-based intervention to improve patient outcome. However, this accuracy is significantly higher compared to another study conducted in the same setting in 2021 which evaluated Electrocardiogram (ECG) interpretation accuracy and found that none of the participants could interpret the findings correctly [25]. This difference likely stems from the two studies using different imaging modalities, which require distinct interpretation skills. Furthermore, advancements in technology, better access to devices, and improved training opportunities specifically focused on FoCUS may have also contributed to the improved performance in this study.

The study revealed that from the specific findings, accuracy was highest for collapsing IVC (91.3%) and standstill (90.0%), and lowest for RWMA of LV (46.3%) showing large variability across specific findings. The variation in accuracy between findings could occur due to factors like complexity of specific finding and the frequency of encounter of that specific finding in the emergency setting. Low accuracy of RWMA diagnosis was reported in another study conducted in Kenya. The lower accuracy to detect and interpret RWMA could be due to the lack of proper phased array probes, which are instrumental in assessing cardiac function [11].

Significant factors that affect FoCUS interpretation accuracy were found to be taking FoCUS training, perceived skill in using FoCUS, and perceived confidence in acquiring and interpreting FoCUS findings. Taking FoCUS training during a residency program was associated with a fourfold increase in the odds of delivering an accurate FoCUS reading. Despite the fact that almost all who participated (43/44) had taken the training in the year preceding data collection, and none of them felt it was adequate, being trained was associated with a significant increase in diagnostic accuracy. This could be explained by the fact that almost all who took the training (42/44) were in their third year of residency, indicating that their longer academic exposure and practical experience as senior residents, in addition to the FoCUS training they received, likely contributed to the higher accuracy. Studies that assessed the diagnostic accuracy of POCUS and ECG among senior residents in the same study setting also found that third-year residents had a significantly higher accuracy rate, showing that seniority is related with improved overall diagnostic skill [25].

Furthermore, residents who believe they have the necessary skills and feel confident in their ability to use and interpret FoCUS findings were found to have nearly fivefold and threefold increased odds of providing an accurate interpretation compared to those who lacked these qualities, respectively. This could be because residents who perceive themselves as having good skills and confidence might be more engaged and motivated during their training and practice, resulting in better understanding and performance and, ultimately, higher accuracy.

Being the only study conducted in two of the largest teaching hospitals in the country, the findings provide a valuable basis for designing targeted interventions. However, some limitations should be considered when interpreting the results. The FoCUS readings came from a standard online source without additional clinical data, potentially impacting residents' judgment and accuracy. The study did not assess the specific effects of FoCUS training based on factors like depth, modality, and timing, limiting our understanding of its impact on resident performance. Despite the objective assessment method, observer bias could have inflated the accuracy rate. Furthermore, the study assessed competency solely through interpretation accuracy. A comprehensive assessment should include objective evaluations of image acquisition, technical skill, knowledge application in practical settings, and effective decision-making in routine clinical practice, as these all demonstrably affect residents' FoCUS skills [[33], [34], [35]]. Finally, the study included residents from two tertiary-level referral hospitals, where patient flow dynamics and educational environments could differ from other teaching hospitals, limiting the generalizability of the findings to such settings.

Conclusion

The study identified a low overall accuracy in FoCUS interpretation among senior ECCM residents at two large referral teaching hospitals in Ethiopia. Residents who received FoCUS training, perceived themselves as skilled, and felt confident in interpretation demonstrated significantly higher accuracy. These findings underscore the critical need to improve ECCM residents' skills through structured training programs that include simulated practice with real-world scenarios, promote continuous education and skill maintenance, provide mentorship and supervision, including remote consultation with FoCUS-trained specialists, and periodic assessments to ensure skill retention. Furthermore, there is a need to evaluate and potentially revise the existing curriculum to ensure it equips residents with the necessary skills and knowledge. To further improve diagnostic accuracy and, ultimately, patient outcomes, future longitudinal research should investigate how training program design elements influence program effectiveness. Additionally, competency assessment should encompass multiple domains, and studies should periodically track residents' diagnostic accuracy as they receive continuous training and feedback.

Dissemination of results

Results from this study was shared with staff members at the data collection site at the department thesis work presentation.

Authors’ contribution

Authors contributed as follows to the conception or design of the work; the acquisition, analysis, or interpretation of data for the work; and drafting the work or revising it critically for important intellectual content: NMM, TBA, KGA and TWL 16% each, NAM, YWH, FHM 7% each, YAB, MT, BT 5% each. All authors approved the version to be published and agreed to be accountable for all aspects of work.

Consent for publication

Not applicable

Availability of data and materials

All relevant data are available upon reasonable request from the corresponding author.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.