Table 2.

Psychometric Properties of Study Tool Compared With Other Available EBM Competence Tests

Test property	Measure to be used	Acceptable results	RESET tool	ACE tool	Berlin (two different sets)	Fresno
Content validity	Expert opinion	Test covers EBM topics	All items were scored at least “important” by > 50% of experts	Acceptable	Not reported	“revisions based on experts’ suggestions”
Inter-rater reliability	Inter-rater correlation	Expected to be high (> 0.6)	0.9	N/A	N/A	0.76–0.98
Internal reliability	Cronbach’s alpha [23] Item-total correlation	0.6–0.7 = questionable 0.7–0.8 = acceptable 0.8–0.9 = good > 0.9 = excellent	0.6	0.69	0.75, 0.82	0.88
		> 0.30 (per Fresno)	0.37	0.14 to 0.20, apart from three items (0.03, 0.04, and 0.06)		0.47 to 0.75
Item difficulty	Percentage of candidates who correctly answer each item	Wide range of results allows test to be used in expert and novice groups	7–86% (trainees), 30–100% (experts)	36–84%	Not reported	24 to 73%
Item discrimination	Item discrimination index (ranges from − 1.0 to 1.0)	All items should be positively indexed, > = 0.2 is considered acceptable	0.17–0.65	0.37–0.84	Not reported	0.41–0.86
Construct validity	Mean scores of experts and novices compared by t test % passing for expert and novice groups compared by χ2 test		48-point test: novice 29.4, expert 35.4 (p < 0.001)	15-point test: novice 8.6, intermediate 9.5, advanced 10.4 (p < 0.001)	ANOVA: 4.2 control, 6.3 course participants, 11.9 expert (p < 0.001)_	212-point test: novice 95.6, expert 147.5 (p < 0.001)

N/A, not assessed