Table 2.

Comparison of SBA and VSA questions and scores

	SBA	VSA	SBA–VSA difference and statistical significance
Question facility* Mean (SD), range	0.61 (0.20), 0.16–0.95	0.40 (0.21), 0.02–0.85	0.21 (0.19), −0.32 to 0.65 Paired t-test, t=7.89, p<0.001
Positive cue rate (question level) Mean (SD), range (%)	42.7 (21.3), 3.9–85.7		One-sample t-test (Null hypothesis≤20%) t=7.53, p<0.001
Internal consistency (Cronbach’s alpha)	0.693	0.731	−0.038 F 1416,1416=1.262, p<0.001
Questions required for an alpha of 0.8	89	74	15
Cohen pass mark†	28/50	18/50	N/A
Pass rate using Cohen pass mark (%)	71.2	66.3	Kappa=0.59 z=22.2, p<0.001
Question discrimination Median (IQR), range	0.184 (0.135–0.220), 0.003–0.287	0.192 (0.121–0.259), −0.006 to 0.395	−0.004 (−0.083 to 0.034), −0.296 to 0.225 Wilcoxon test, z=−1.36, p=0.175

*Facility: proportion of students answering correctly.

†Calculated as 60% of the score of the 95th percentile student and assuming scores due to guessing of 20% for the SBA and 0% for the VSA.

N/A, not applicable; SBA, single-best-answer; VSA, very-short-answer.