Predictive validity of selection for entry into postgraduate training in general practice

The recent paper by Patterson et al¹ reports the encouraging and considerable predictive validity of MCQ (CPS ‘clinical problem solving’) and situational judgment test (SJT) selection tests for performance on the two MRCGP formal examination components.

The MCQ and the SJT correlate with the Applied Knowledge Test (AKT) at 0.85 and 0.69, respectively (range-corrected). From data provided in the paper, together the two selection tests can be seen to predict 74–75% of AKT score variance. The two tests also correlate with the old-style Clinical Skills Assessment (CSA: 0.55, 0.57, respectively, range-corrected). Together they predict 37–38% of CSA Score variance. But the subsequent Selection Centre (SC), then a three station OSCE and correlating with the CSA at 0.41 (range corrected), is reported as only explaining an additional 2% of CSA score variance.

Given that a set of computer-delivered multi-choice tests will cost £100–200 per candidate and that the true cost of a selection centre will be at least £1000, the latter seems to provide relatively very poor value, especially with about 5000 candidates being shortlisted for selection in 2012.² Perhaps the £5 million-plus could be better used in supporting poorly-performing trainees?

Also, those of us who are responsible for devising OSCE assessments would be grateful to learn how one with three stations, single marked, can be devised such that its reliability (Cronbach’s α) is 0.87, when considerably longer similar assessments give far lower reliability estimates; for example, MRCGP CSA (13 stations) 0.77,³ and iMRCS Part B (8 station subtests) 0.68–0.72 and (10 station subtests) 0.76–0.78.⁴