Table 3.

Review table detailing study characteristics and relative rating of methodology and validity

Origin	Study design	Demographics	Instrument	Results	Conclusions	Limitations	Study quality
BuFstam et al.6 Malaysia 2014	Type: bedside echocardiography Setting: ED Teaching: Three‐hour web‐based didactic module, 3 h practical module Assessment: Trainee images compared with cardiologist views	9 EM trainees with reported low baseline knowledge; series of 100 consecutive stable patients	Ax: Technical and interpretive skills Measures: ‐Kappa coefficient for agreement	Variable correlation depending on cardiac views Agreement: Quantitative and qualitative LV functional estimates: 93% Pericardial effusion: 98% IVC diameter assessment: 64.2%	Able to interpret with reliable accuracy, feasibility of web‐based learning	Observer effect noted; no control group; low participant numbers; unclear patient selection	Biases affecting result; subjective diagnostic questions Rated 3/10
Caronia et al.8 USA 2014	Type: Focused vascular sonography Setting: ICU Teaching: 2‐h didactic and practical module Assessment: Trainee images compared with ultrasound technician views	19 IM trainees; 143 studies on series of 75 consecutive patients with clinical features suggesting lower extremity DVT awaiting formal radiology	Ax: Ability to achieve diagnosis of DVT Measures: ‐Kappa coefficient for agreement	Substantial agreement for DVT diagnosis Above knee DVT: sn 97%; sp 63% Common femoral and popliteal DVT: sn 86%; sp 97% Agreement: Kappa: 0.70 No isolated superficial thromboses identified by residents Time from formal ultrasound to report: 14.7 h	Residents can be trained to perform three‐step U/S examination; more subtle pathology not elucidated	Low number of DVT positive studies; Low number of participants; trainees not blinded to clinical context	Simplistic diagnostic question; underpowered Rated 3/10
Caronia et al.7 USA 2013	Type: Renal sonography Setting: Inpatients – ICU, general ward, intermediate care Teaching: Five‐hour didactic module, three supervised examinations Assessment: Trainee images compared with formal radiology report	17 IM trainees with no prior training in stated U/S modality; 125 studies on convenience series of 66 patients	Ax: Rule out renal obstruction and identify sonographic findings of CKD Measures: ‐Sn and Sp ‐Kappa coefficient for agreement	Variable agreement depending on renal pathology Hydronephrosis: sn 94%; sp 93% Renal atrophy: sn 100%; sp 83% Echogenicity: sn 40%; sp 98% Renal cysts: sn 60%; sp 96%	Demonstration that basic skill can be taught	Selection/interest bias – trainees undertaking elective in critical care sonography; substantial variation in number of studies performed by trainees; Not powered to gauge proficiency; image acquisition skills not reported	Small study; blinded; simplistic employment of U/S   Rated 4/10
Chalumeau‐Lemoine et al.9 France 2009	Type: ICU U/S Setting: ICU Teaching: 8.5‐h didactic teaching, staggered practical sessions Assessment: Trainee images compared with radiologist study	8 ICU trainees; 129 clinical questions on convenience series of 77 patients; Patients included in series at discretion of intensivist	Ax: Rule in/out pathology Measures ‐Kappa coefficient for agreement ‐Student's t‐test for time to imaging	High agreement for immediately clinical relevant questions Most common clinical questions: presence of pleural effusion, presence of obstructive uropathy; signs of chronic renal insufficiency Agreement: Overall: 84.4% (kappa 0.66) Questions with a potential therapeutic implication (retrospective): 95% (kappa 0.86) Time to imaging Trainees vs. radiologists (37 ± 39mins vs. 296 ± 487mins, P = 0.004)	Limited general u/s interpretation can be taught after brief training; need to better define learning curve and potential patient benefits	Selection bias – inclusion at discretion of intensivist; broad range of U/S uses; competency in ultrasound assessed only by diagnostic accuracy; low number of studies for each question subset	Competency only assessed by diagnostic accuracy; underpowered Rated 3/10
Gaspari et al.10 USA 2009	Type: RUQ U/S Setting: ED Teaching: 9‐h didactic teaching; at least two practical sessions Assessment: Trainee images recorded and reviewed by expert	37 EM trainees and 7 attendings (stratified); 352 patient series	Ax: Identification of gallbladder pathology including image quality, presence of required images Measures ‐Kappa coefficient for agreement	Interpretive and technical error rates decreasing with experience Agreement: Overall: kappa 0.917 ‐Number of poor‐quality ultrasounds decreasing after average of 7 scans ‐Performance of >25 (point of ‘credentialed’) ultrasound scans: increased agreement ‐Average number of scans before decrease in poor‐quality images: seven scans	Agreement varied by experience level; supports performance of 25 U/S before clinical competency	Variable informal experience base; non‐specific patient enrolment; no comparison to images attained by experts	Very specific focus on a very select group of patients limiting transferability of results Rated 5/10
Gulati et al.11 USA 2015	Type: MSK U/S Setting: Inpatients Teaching: web‐based learning, 2‐h didactic; unquantified practical session Assessment: OSCE	15 IM residents; 3‐month unstructured practical experience	Ax: Presence of knee effusion with assessment of theory, image acquisition and interpretation Measures: ‐Student's t‐test for pre‐/post‐intervention test scores ‐Descriptive statistics	Significant improvement in proficiency post‐intervention Statistically significant increase in performance across all OSCE subcomponents Trainee survey: Experience reported as ‘very valuable’ Requested more training examinations	Can be feasibly incorporated but institution and educator dependent	Low number participants and focus; variability postulated to relate to individual learning curves; use of Likert scales to gauge trainee impressions; study did not produce expected numbers of positive and negative exams	High degree of trainee variability in experience over study period; simplistic non‐acute modality of U/S Rated 4/10
Jacoby et al.12 USA 2007	Type: focused vascular sonography (two‐point DVT examination) Setting: ED Teaching: 90‐min mixed didactic and practical session, individual instruction not specified Assessment: Trainee images compared to blinded vascular technician	Six EM trainees; series of 121 symptomatic extremities	Ax: Presence of DVT in ‘femoral’ and ‘popliteal’ sties Measures: ‐sn and sp ‐Descriptive statistics	High degree of sn and sp Residents identified 8/9 positive cases of acute DVT within target area Overall prevalence of DVT within target area: 7% Overall sn 89% sp 97%	Limited examination with considerable but not perfect sensitivity	Small sample size, no follow‐up on patient treatment or outcome, not consecutive patients presenting to ED, chronic DVT/partial occlusion counted as ‘no’; low prevalence of positive findings	Diagnostic accuracy simplistic and underpowered for transferability of findings Rated 3/10
Jang et al.13 USA 2004	Type: compression U/S for proximal lower extremity DVT Setting: ED Teaching: One‐hour didactic teaching with demonstration, no practical component Assessment: Trainee provisional diagnosis vs. subsequently established DVT	Eight EM trainees; non‐consecutive series of 72 patients	Ax: presence/absence of proximal DVT Measures: ‐sn and sp	High rates of DVT detection 63/72 patients at 1 year follow‐up Overall sn 100% and sp 91.8%	High detection rates compromised by selection bias	Non‐consecutive series within less technically challenging patients included; optional participation – ‘US interest bias’; low number of study participants	Select patient group limiting external validity Rated 4/10
Jones et al.14 USA 2003	Type: bedside echocardiography Setting: ED Teaching: 6‐h didactic course; one‐hour practical component Assessment: Performance on post‐intervention written and practical test	30 EM residents with non‐cardiac ultrasound experience	Ax: achieving specified goals and views; U/S theory Measures ‐Paired t‐test	Statistically significant increase in test scores post‐intervention 9/30 residents excluded as did not attend all instructional material Written score: pre 54% vs. post 76% (P < 0.005) Practical score: pre 56% vs. post 94% (P < 0.005) Significant increase in scores on pre‐/post‐theory and practical	Increasing scores indicating that minimal competency can be achieved in short time frame	Test not formally validated; practical element performed on healthy volunteers; no control; high dropout rate given voluntary nature of study; no period of knowledge consolidation	High non‐completion rate affecting results implications of study limited by short time frame Rated 5/10
Kimura et al.15 USA 2012	Type: bedside echocardiography Setting: ICU, cardiology rooms Teaching: curriculum consisting of 10 supervised examinations with 12 h of lectures and 12 h of bedside tuition over each year of study Assessment: Standardised CLUE – clinical examination performed at end of each clinical year	41 IM trainees assessed over 3‐year residency programme	Ax: Diagnostic and technical skills; knowledge assessment Measures: ‐descriptive statistics ‐inferential statistics ‐Likert survey	High rate of competency 8/41 trainees failed to achieve >80% competency in final CLUE assessment No correlation between CLUE pass rate and general academic performance, chief resident selection, gender Questionnaire finding self‐reported change in clinical behaviour	U/S competency not linked to academic performance; feasibility in terms of costs when in‐hospital resources used	Resident experience and performance subjective; convenience series of studies; skillset development reliant on informal ad‐hoc learning	Designed to rigorously test uptake and application of new skills; clear objective assessment Rated 7/10
MacVane et al.16 USA 2012	Type: TVUS for intra‐uterine pregnancy Setting: ED Teaching: one‐hour didactic lecture with theoretical competency examination; 10 observed TVUS attempts Assessment: Immediate diagnostic assessment with images videotaped and reviewed by expert	22 EM trainees; 75 TVUS performed; Residents with established no prior experience in TVUS	Ax: presence/absence of intrauterine pregnancy Measures: ‐Kappa coefficient for agreement	High degree of correlation observed Correlation: Overall: 91.1% By PGY: PGY‐1 – 100%; PGY‐2 – 92.1%; PGY‐3 – 93.3%	Concordance related to PGY; PGY‐1 outperformance suggesting level of experience no limitation to U/S training; supports early education in residency	Unquantified previous general ultrasound experience; small number in study; convenience sample; assumption that supervising clinician did not assist trainee	Key biases discussed; technical modality of general U/S Rated 6/10
Poulsen et al.17 Denmark 2015	Type: abdominal U/S Setting: ED Teaching: 8‐h didactic lectures and practical sessions Assessment: Images saved and results correlated with definitive diagnosis post‐formal imaging/investigation	Three general/intern trainees; series of 45 patients	Ax: diagnosis of acute abdomen Measures ‐Descriptive statistics	High proportion of images obtained clinically relevant 21/45 patients proceeded to specified definitive investigation and therefore included 14/21 patients had diagnostic agreement Qualitative image assessment: Useful images on all abdominal organs for all patients except large bowel and pancreas	Junior trainees can correctly identify pathology and abdominal structures after short intervention	Participating intern potentially also the treating physician and not blinded to clinical presentation; education intervention details not provided (course not devised for project)	Limited transferability of results; very low number of participants and scan rate Rated 3/10
Schnobrich et al.18 USA 2013	Type: general U/S Setting: no specific Teaching: 30 h of training over five‐day course – online, didactic and practical elements Assessment: Competency assessment pre‐/post‐teaching intervention	32 IM trainees	Ax: theoretical test; practical assessment on simulators and test subject Measures ‐Descriptive statistics ‐Paired t‐test ‐Likert survey	Demonstrated increase in capability 29/32 trainees completed pre‐test and 25/32 completed post‐test Test scores: ‐Group 1: pre‐test A score 36% post‐test B score 73% (P < 0.001) ‐Group 2: pre‐test B score 43% post‐test A score: 75% (P < 0.001) Survey results: ‐pre/post: increased perceived skill in image acquisition, interpretation and clinical application (all P < 0.001) ‐Usefulness and relevancy rated 4.6/5; practicability of U/S rated 4.5/5	Successful in preparing interns to perform supervised scans; suggests that change in resident behaviour observed	Broad use of ultrasound assessed; no intervening clinical exposure for duration of study period; low number of participants	Results in keeping with expected improvement after intervention; uncertain clinical relevance for trainees Rated 5/10
Tolsgaar et al.19 Denmark 2015	Type: TVUS Setting: Gynaecology clinics Teaching: participants randomised to intervention group receiving competency‐based simulation training in addition to clinical training – one‐hour didactic lecture followed by supervised practical sessions Assessment: Competency assessment involving expert review of recorded images 2 months post‐intervention	33 O+G trainees completing on average 58 and 63 scans in intervention and control groups respectively	Ax: image acquisition, interpretation, documentation and subsequent clinical decision making Measures ‐Descriptive statistics ‐Inferential statistics	Simulation‐intervention group achieving higher assessment scores 26/33 randomised residents completing study requirements and analysed Assessment ‐Mean performance scores intervention group vs. control group (59.1% vs. 37.6%, P < 0.001) ‐Achievement of pre‐determined pass rate intervention group vs. control group (85.7% vs. 8.3%, P < 0.001)	Simulation led to a substantial improvement in clinical performance at 2 months	Degree of patient variability in both consolidation phase and assessment; external factors not assessed ‐ monetary, time, transferability; pass/fail scoring on final assessment; unclear level of clinical supervision during study period for each participant	Simulators and performance indicators validated in literature; poor external validity to other ultrasound modalities Rated 7/10
Unluer20 Turkey 2010	Type: Abdominal US Setting: ED Teaching: 3‐h didactic and 3‐h practical training Assessment: Trainee diagnosis compared with definitive investigation or patient follow‐up	4 EM trainees; series of 174 patients	Ax: presence/absence of SBO Measures ‐Descriptive statistics ‐sn and sp	High degree of accuracy demonstrated Presence of SBO ‐Sn: 97.7% ‐Sp: 92.7% ‐PPV: 93.3% ‐NPV: 97.4% ‐LR: 13.4 84/90 true positives; 76/78 true negatives No significant difference between bedside ultrasonography and radiographer performed study	Trainees consistently able to diagnose SBO on U/S	Clinical status of patient known to trainee; small number of trainee participants	Results limited by key biases and low number Rated 3/10

U/S, Ultrasound; LV, left ventricle; ICU, intensive care unit; ED, emergency department; IM, internal medicine/physician; DVT, deep vein thrombosis; sp, specificity; sn, sensitivity; PPV, positive predictive value; NPV, negative predictive value; LR, likelihood ratio; Ax, assessment; OSCE, objective structured clinical examination; MSK, musculoskeletal; CLUE, cardiovascular limited ultrasound examination; TVUS, transvaginal ultrasound; PGY, postgraduate year; SBO, small bowel obstruction.