Pocket-Sized Ultrasound as an Aid to Physical Diagnosis for Internal Medicine Residents: A Randomized Trial

Abstract

BACKGROUND

Proficiency and self-confidence in the physical examination is poor among internal medicine residents and interest in ultrasound technology has expanded.

OBJECTIVE

We aimed to determine whether a pocket-sized ultrasound improves the diagnostic accuracy and confidence of residents after a 3-h training session and 1 month of independent practice.

DESIGN

This was a randomized parallel group controlled trial.

PARTICIPANTS

Forty internal medicine residents in a single program at an academic medical center participated in the study.

INTERVENTION

Three hours of training on use of pocket-sized ultrasound was followed by 1 month of independent practice.

MAIN MEASURES

The primary outcome was a comparison of the diagnostic accuracy of a physical exam alone versus a physical examination augmented with a pocket-sized ultrasound. Other outcomes included confidence in exam findings and a survey of attitudes towards the physical exam and the role of ultrasound.

KEY RESULTS

Residents in the intervention group using a pocket-sized ultrasound correctly identified an average of 7.6 of the 17 abnormal findings (accuracy rate of 44.9 %). Those in the control group correctly identified an average of 6.4 abnormal findings (accuracy rate of 37.6 %, p = 0.11). Residents in the intervention group identified on average 15.9 findings as abnormal when no abnormality existed (false positive rate of 16.8 %). Those in the control group incorrectly identified an average of 15.5 positive findings (false positive rate of 16.3 %). There was no difference between groups regarding self-assessed confidence in physical examination. Residents in the intervention group identified 6.1 of 13 abnormal cardiac findings versus the control group’s 4.5 of 13, an accuracy rate of 47.0 % versus 34.6 % (p = 0.023).

CONCLUSIONS

The diagnostic ability of internal medicine residents did not significantly improve with use of a pocket-sized ultrasound device after a 3-h training session and 1 month of independent practice.

Trial Registration: clinicaltrials.gov: number NCT01948076; URL http://clinicaltrials.gov/ct2/show/study/NCT01948076?term=ultrasound+physical+exam&rank=2

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-014-3086-4) contains supplementary material, which is available to authorized users.

INTRODUCTION

The physical examination is an essential component of medical care. However, proficiency and self-confidence in the physical exam are poor across all levels of training, and physical exam skills do not appear to improve over the course of residency.^1–3 Even when performed by experienced clinicians, the physical examination often lacks accuracy and can lead to missed diagnoses.^1,3,4

Interest has expanded in the use of ultrasound to improve the accuracy of the physical examination. Advances in technology have allowed for miniaturization and affordability of ultrasounds, making individual ownership and daily use possible. Pocket-sized ultrasound devices are powerful tools that have been found to perform similarly to traditional high-end systems when used by experienced clinicians.^5–13

Internal medicine residents desire ultrasound training, but self-reported competence is low.¹⁴ Several studies have shown that after brief training, residents can become competent in performing and interpreting basic ultrasound examinations. However, most studies focused on a limited number of findings or very specific settings such as the ICU, and have involved small numbers of residents.^15–19 Others have shown that even after extensive practice, residents’ competence in performing and interpreting sonograms is poor.²⁰ We hypothesized that brief training with and access to a portable ultrasound would increase the diagnostic accuracy of the cardiac, abdominal and pulmonary examinations in the outpatient setting among medical residents.

METHODS

Design Overview

We performed a randomized, controlled trial of 40 internal medicine (IM) residents at an academic medical center. Twenty participants received a 3-h training session on use of pocket-sized ultrasound and were given the device for 1 month of individual practice. All participants participated in an assessment session involving the examination of ten patient-subjects. The intervention group examined each of the patient-subjects first with a traditional physical exam, and then using a pocket-sized ultrasound. The control group performed only a physical exam on each of the patient-subjects. All participants rated their confidence in their diagnoses and completed a survey of attitudes towards the physical exam and use of ultrasound. The enrollment was set at 40 residents to power the study to be able to detect a 30 % difference in diagnostic accuracy between the two groups.

Setting and Participants

Resident Subject Enrollment

All residents in the Brigham and Women’s Hospital (BWH) IM residency program were invited to participate in the study in October 2012. Residents with prior formal ultrasound training were excluded. The first 40 residents who responded to the email advertisement were enrolled in the study and provided verbal consent to participate. Thirty-nine attended the assessment session. One participant in the intervention group dropped out due to a family emergency. Demographics are provided in Table 1A.

Table 1.

Study Participants

A. Resident Subjects
	Physical Exam	Ultrasound
Number of residents	20	19
Post-graduate year (PGY) Level
1	6	4
2	11	4
3	2	10
4	1	1
Mean Age	29.3	29.5
Female	8	10
On inpatient clinical rotation	15	10
Career intent
Primary Care	5	5
Hospitalist	1	2
Fellowship	14	12
Cardiology	5	6
B. Patient Subjects
Age	Sex	BMI	Physical Exam Finding
58	Male	25	Normal
35	Female	36	Hepatomegaly, Paralyzed left hemidiaphram
77	Male	25	Normal
58	Male	30	Splenomegaly, Ascites
68	Female	38	Tricuspid regurgitation, Mitral regurgitation, Hypertrophic obstructive cardiomyopathy
64	Male	28	Aortic regurgitation
60	Male	24	Mitral regurgitation
41	Male	29	Left ventricular enlargement
69	Male	29	Left ventricular enlargement, Aortic regurgitation, Mitral regurgitation
48	Male	23	Tricuspid regurgitation, Mitral regurgitation, Aortic regurgitation, Mitral valve prolapse

Patient Subject Enrollment

Fourteen patients were recruited from outpatient practices at BWH from October to November 2012 and ten were selected to participate in the assessment activity based on their examination findings and availability to attend a day-long assessment session. Demographics and exam findings can be seen in Table 1B.

The Partners Health Care System institutional review board approved the study. All subjects provided informed consent and were compensated.

Pocket-Sized Ultrasound Device

The pocket-sized ultrasound device used was the V-Scan (General Electric). The device weighs 390 g, is 135 mm × 73 mm × 28 mm, and the probe is 120 mm × 33 mm × 26 mm. It has a 3.5 in. display and two-dimensional (2D) imaging and color Doppler blood flow capabilities. The bandwidth of the phased-array probe is 1.7 to 3.8 MHz.

Randomization and Curricular Training

The 40 participating residents were randomized by blind selection in an alternating pattern. The 20 residents randomized to the intervention group received a training session consisting of 1.5 h of didactics followed by 1.5 h of hands-on training in groups of two to three residents. Cardiac ultrasound was taught by an attending cardiologist and chest and abdominal ultrasound were taught by two attending radiologists. The intervention group was provided an individual pocket-sized ultrasound for use in daily work for 4 weeks. The control group did not receive an ultrasound or training.

Assessment Session

One month after the ultrasound training, study participants attended an assessment. Gold standard physical findings for the ten patient-subjects were defined as the ultrasound exam by an attending radiologist (abdominal and lung exams) or cardiologist (cardiac exams). After the gold standards were established, the ten patient-subjects were examined by each of the resident study subjects. Residents were directed to perform a 12-item cardiac examination or 10-item abdominal and pulmonary examination. Upon entering each patient’s room, residents were given an assessment form where they marked the presence and absence of prespecified findings and their confidence in each finding on a scale of 1–5 (see supplemental appendix online).

Control group participants had 5 min with each patient to perform a focused physical examination and mark their findings on the assessment form. Intervention group participants had 10 min with each patient, during which time they first performed a physical examination and recorded their findings, and then subsequently were given a pocket-sized ultrasound to re-examine the patient, recording their findings on a new assessment form. The patient-subjects also underwent physical examination by a senior IM physician at our institution.

OUTCOME MEASURES AND STATISTICAL ANALYSIS (TABLE 2)

Table 2.

Explanation of Outcomes and Comparisons

Grey Shading denotes statistical significance

Primary Outcome

The primary outcome was diagnostic accuracy, and was compared between the intervention group, after having performed both physical exam and ultrasound exam, and the control group. We used two metrics for comparison: (1) a diagnostic score for correctly identifying physical abnormalities that were present (“present score”), and (2) a diagnostic score for correctly identifying the absence of physical abnormalities (“absent score”).

The ten patient participants had a total of 17 physical exam abnormalities and 95 normal exam findings (Table 1B). To compute the “present score”, each resident was given a point for correctly identifying the presence of each of the 17 physical exam abnormalities, for a total possible “present” score of 17. The “absent score” was calculated for both the cardiac and chest/abdomen exams by summing the total number of findings correctly identified as “absent” (for example, one point was awarded for correctly marking the box on the assessment form for the absence of aortic stenosis in a patient who did not have aortic stenosis). We compared the “present scores” and “absent scores” of the two groups using the Wilcoxon Rank Sum Test. The false positive rate was calculated by using the number of findings incorrectly identified as “abnormal” divided by the total number of findings (95). An additional analysis was the diagnostic accuracy of the senior IM physician, calculated by comparing their physical exam findings with ultrasound gold standards.

Secondary Outcomes

Intervention Group Improvement with Ultrasound

Within the intervention group, we compared diagnostic accuracy using the physical examination with diagnostic accuracy using the ultrasound examination. Comparisons of the “present score” and “absent score” were done using the Wilcoxon Signed-Rank Test, while the comparison for each of the 17 items that make up the “present score” was done using the McNemar’s Test.

Diagnostic Confidence

We asked study participants to rate their confidence for each exam maneuver on a five-point Likert scale indicating how certain they were that they had correctly identified the finding as present or absent. We compared diagnostic confidence of the control and intervention groups. We also assessed any change in confidence in the intervention group from physical exam to ultrasound exam.

Confidence-weighted diagnostic ability

A confidence-weighted diagnostic score was calculated by multiplying the number of correctly identified abnormal and normal findings by the confidence in that particular finding as reported on a 1–5 scale. Thus, a correctly identified finding was awarded a higher point value if the resident was more confident about the finding. Diagnostic confidence and confidence-weighted diagnostic accuracy were compared both between the control and intervention groups as well as within the intervention group using the Wilcoxon Rank Sum Test.

Resident Survey

On the day of the assessment session, residents in both the control and intervention groups completed a brief survey that assessed attitudes towards the physical exam and use of ultrasound. Those in the intervention group were asked additional questions about their use of the hand-held ultrasound over the prior 4 weeks and the adequacy of their training.

RESULTS

Primary Outcomes

The ten patient participants had a total of 17 physical exam abnormalities (Table 1B). Residents in the intervention group correctly identified an average of 7.6 of the 17 findings for an accuracy rate of 44.9 % Those in the control group correctly identified an average of 6.4 findings for an accuracy rate of 37.6 %, which was not statistically different (p value = 0.11, Wilcoxon Rank Sum Test) (Table 3). The senior IM physicians correctly identified nine out of 17 findings (52.9 %) using physical examination alone.

Table 3.

Comparison of Primary Outcome (Intervention Group Ultrasound Exam Versus Control Group Physical Exam)

Identification of normal and abnormal physical exam findings: “Present Score” and “Absent Score”
	All findings (Cardiac/Chest/Abdomen)		Cardiac findings only		Chest/Abdomen findings
	Present Score (%)	Absent Score (False Positive Rate*)	Present Score (%)	Absent Score (False Positive Rate*)	Present Score (%)	Absent Score (False Positive Rate*)
	Abnormal findings correctly identified of 17 possible	Normal findings correctly identified of 95 possible	Abnormal findings correctly identified of 13 possible	Normal findings correctly identified of 59 possible	Abnormal findings correctly identified of 4 possible	Normal findings correctly identified of 36 possible
Intervention Group Ultrasound Exam	7.6 (44.9 %)	79.1 (16.8 %)	6.1 (47.0 %)	49.4 (16.3 %)	1.5 (38.2 %)	29.7 (17.5 %)
Physical Exam Group Physical Exam	6.4 (37.6 %)	79.6 (16.3 %)	4.5 (34.6)	51.3 (13.1 %)	1.9 (47.5 %)	28.3 (21.4 %)
p-value from Wilcoxon Rank Sum test	0.11	0.28	0.023	0.96	0.34	0.016

*False positive rate is 100 % minus the percentage of normal findings correctly identified as normal

For the 13 cardiac exam abnormal findings, the diagnostic accuracy of those in the intervention group was higher than in the control group (47 % versus 34.6 %, p value = 0.023, Wilcoxon rank sum test) (Table 3). There was no difference between the two groups with respect to identification of abnormal abdominal and pulmonary findings.

We also recorded the diagnostic accuracy of the two groups to correctly confirm the absence of abnormalities (i.e., identifying absence of splenomegaly in a patient with a normal spleen). There were 95 physical findings across the ten patient-subjects that were confirmed as normal by the gold standard ultrasound examination. Residents in the intervention group identified an average of 15.9 findings as abnormal when no abnormality was present (false positive rate of 16.8 %), while those in the control group incorrectly identified an average of 15.5 normal findings (false positive rate of 16.3 %). There was no statistical difference between the rates at which residents in the two groups identified normal findings (p = 0.28). The senior IM physician identified ten abnormal findings when no abnormality was present (false positive rate of 10.5 %).

Secondary Outcomes

Intervention Group Improvement with Ultrasound

Within the intervention group, we found significant improvement in diagnostic accuracy for abnormal findings when comparing the resident’s initial physical exam with the subsequent ultrasound exam (39.3 % vs. 44.9 %, p value = 0.043, Table 4). No difference in the false positive rate between physical exam and ultrasound exam within the intervention group (16.6 % vs. 16.8 %, p value = 0.81) was found.

Table 4.

Comparison of Secondary Outcome (Intervention Group Physical Exam Versus Intervention Group Ultrasound Exam)

Identification of normal and abnormal physical exam findings: “Present Score” and “Absent Score”
	All findings (Cardiac/Chest/Abdomen)		Cardiac findings		Chest/Abdomen findings
	Present Score (%)	Absent Score (False Positive Rate*)	Present Score (%)	Absent Score (False Positive Rate*)	Present Score (%)	Absent Score (False Positive Rate*)
	Abnormal findings correctly identified of 17 possible	Normal findings correctly identified of 95 possible	Abnormal findings correctly identified of 13 possible	Normal findings correctly identified of 59 possible	Abnormal findings correctly identified of 4 possible	Normal findings correctly identified of 36 possible
Intervention Group Ultrasound Exam	7.6 (44.9 %)	79.1 (16.8 %)	6.1 (47.0 %)	49.4 (16.3 %)	1.5 (38.2 %)	29.7 (17.5 %)
Intervention Group Physical Exam	6.7 (39.3 %)	793 (16.6)	5.0 (38.5 %)	50.3 (14.8 %)	1.7 (42.1 %)	29.0 (19.4 %)
p value from Wilcoxon Signed Rank test	0.043	0.81	0.0084	0.51	0.61	0.19

*False positive rate is 100 % minus the percentage of normal findings correctly identified as normal

Confidence in Diagnostic Ability

There was no difference in diagnostic confidence between the control group and the intervention group (p = 0.51). However, we did find a significant improvement in diagnostic confidence within the intervention group when comparing physical exam confidence to ultrasound exam confidence for both the abnormal findings (p = 0.0006) and for the normal findings (p = 0.0002).

Confidence-Weighted Diagnostic Ability

When correctly identified abnormal findings were weighted by confidence, such that a correctly identified finding was awarded more points if the resident was confident in the finding, there was no difference found overall for either abnormal findings (p = 0.13) or normal findings (p = 0.33). However, when residents in the ultrasound group correctly identified an abnormal cardiac finding, they were significantly more confident in that finding than the physical exam group (p = 0.0069).

Residents in the intervention group were also much more likely to be confident in their correct finding when using their ultrasounds as compared to their physical exams. This was true for both the identification of abnormal findings (p = <0.0001) and for the identification of normal findings (p = 0.016).

Survey Results

Our survey of participants found that 92 % agreed that ultrasound training should be a required part of IM residency, 77 % agreed that, “if I had a portable ultrasound it would often contribute to management decisions”, and 69 % agreed that “if I had a portable ultrasound I would use it frequently.” Eight percent of study participants agreed that “portable ultrasound will reduce the usefulness of the physical exam.”

However, of participants in the intervention group, only 5 % used their ultrasound daily and 63 % used their ultrasound less than once a week. Only 47 % of participants felt that they received adequate ultrasound training. Only 32 % of those in the intervention group reported an increased ability to diagnose patients with ultrasound after completing this study (Table 5).

Table 5.

Survey Results

A. Percentage of study participants (n = 39) who agree with the following statements:
Ultrasound training should be a required part of internal medicine residency	94 %
I received adequate physical exam training	54 %
Physical exam often contributes to management decisions	74 %
Portable ultrasound will reduce the usefulness of the physical exam	8 %
If I had a portable ultrasound it would often contribute to management decisions	77 %
If I had a portable ultrasound I would use it frequently	69 %
B. Survey of intervention group (n = 19):
Was the ultrasound device sufficiently portable?	79 %
Did technical problems interfere?	5 %
Would you recommend the device?	89 %
Was training adequate?	47 %
How often did you use your ultrasound?
Less than once a week	63 %
Once a week	26 %
Once daily	5 %
I received adequate training in acquisition of ultrasound images	32 %
During the last four weeks my confidence in ability to diagnose with ultrasound increased	32 %

DISCUSSION

This study showed no overall improvement in diagnostic accuracy among residents using physical examination supplemented with hand-held ultrasound after a 3-h training session as compared with residents relying on the physical examination alone.

When data were stratified by organ system, we found a statistically significant improvement in correct identification of abnormal cardiac findings with use of the hand-held ultrasound in the intervention group when compared to the control group. However, there was no improvement in the ability to correctly identify normal findings, leading to similar false positive rates between the two groups.

We did observe a statistically significant improvement in both diagnostic accuracy and diagnostic confidence for residents in the intervention group who examined patient-subjects using a traditional physical exam and then subsequently performed a focused ultrasound examination on the same patient. However, these outcomes weren’t statistically different when compared with the diagnostic accuracy and confidence of residents in the control group who performed the physical exam without the aid of a hand-held ultrasound.

Ultrasound is a modality with high sensitivity for detecting abnormalities, some of which may not have true clinical significance. By choosing the ultrasound exam by attending cardiologist and radiologists as our gold standard, we may have included findings that were minimally abnormal and could not be detected by physical exam. However, re-analysis of our data using only abnormalities that were present on both the senior physician’s physical examination as well as the gold-standard ultrasound exam demonstrated no significant difference between our control and intervention groups. The low diagnostic accuracy of study participants may be due in part to the fact that study participants performed examinations without the benefit of any clinical history, with limited time, and in a standard exam room (i.e. not optimally positioned for ultrasound scanning). Although the diagnostic performance was poor, it is consistent with previous literature.¹ Furthermore, our senior physicians identified 53 % of abnormal physical findings, which was better than the performance of the residents in our study.

Analysis of resident confidence shows that when residents have access to a hand-held ultrasound device, they feel more confident in their exam findings when using the device as compared with the physical exam alone. However, we also found that there was no difference in diagnostic confidence between residents using a hand-held ultrasound compared with a control group using only the physical exam.

Our study was powered to detect a 30 % difference in diagnostic ability between residents using a hand-held ultrasound and those using the physical exam. There may be a benefit to hand-held ultrasound that is smaller than 30 % and was not detected due to our small sample size. Another limitation is that despite randomization, there was asymmetric distribution of postgraduate year-levels in our intervention and control groups.

Additionally, our survey results showed that those in our intervention group used their ultrasounds infrequently and felt they received inadequate training. However, the vast majority did feel that ultrasound training should be required during residency training.

Although this study didn’t demonstrate an improvement in physical exam accuracy with the use of the portable ultrasound, we feel further investigation is warranted. Several factors may have contributed to the negative findings in this study, including the short training time and the multiple organ systems studied. This combination may not have allowed learners sufficient time to become proficient. Empiric data suggest that basic proficiency in echocardiography can be achieved after 40 h of didactic training and 40 supervised scans.^15,21 The American College of Cardiology recommends 6 months of training and the performance of 150 exams with interpretation of 300 exams for proficiency to perform independent echocardiography,²² and the American Society of Echocardiography on use of hand-carried ultrasound issued similar recommendations.²³ However, studies have shown that shorter duration of training may be sufficient for developing proficiency in select ultrasound findings.^17,24 For example, Bailey et al. showed that primary care residents could correctly identify abdominal aortic aneurysms after a 2-h training,²⁵ and a Duke study found that a focused 3-h training was sufficient for residents to achieve basic proficiency in the detection of four common cardiac abnormalities.²⁶ More recently, Panoulas et al. showed that a 2-h didactic training was sufficient to teach cardiac ultrasound skills to a small group of medical trainees.¹⁹ These studies differed from ours in that we attempted to have residents become adept in a larger set of findings, rather than focusing on a single finding or organ system. At our institution, experience with beginning cardiology fellows has shown that one or more targeted training sessions focusing on specific findings, and sequentially expanding the set of physical abnormalities under study can be effective. A similar graded training approach for a single organ system may be effective with IM residents and may warrant further investigation.

Given the time constraints of residency training today, finding teaching time is challenging.²⁷ Even with two dates for the 3-h training session, we had difficulty recruiting residents to participate in this study. Additionally, usage of hand-held ultrasounds was low during the practice period, perhaps because many participants were on elective rotations during the study period or because residents on ward rotations may not have had time to incorporate ultrasound into their busy clinical routine. Other studies have also shown that even when residents are highly motivated, time constraints limit residents’ ability to perform ultrasound studies.¹⁵ These observations may support future investigation of ultrasound training at an earlier stage of medical training. This earlier start would provide a longer period of time to develop expertise and efficiency with the portable ultrasound prior to starting a busy residency with competing time demands.

CONCLUSIONS

In summary, we did not demonstrate improved diagnostic accuracy over traditional physical exam among medicine residents using pocket-sized ultrasound after a brief 3-h training session. Pocket-sized ultrasound may have the potential to be a valuable aid to enhancing the physical exam, yet the findings of this study suggest that physicians with hand-held ultrasounds cannot expect to improve their diagnostic skills over the period studied. Longer or more intensive training may be required for internists in order to take advantage of the diagnostic potential of hand-held ultrasound.