Selecting Tomorrow’s Doctors

Application to medical school is a competitive process. In 2009 there were 27,429 applications for approximately 8000 places in the UK, up 13.7 per cent on the previous year¹. The cost of training a medical student is circa £200,000 but the cost of selecting the wrong applicant can be even greater².

Traditionally medical schools have relied on performance in knowledge-based examinations for selection and although these distinguish academically able applicants, it is often the failure to develop non-cognitive competencies such as motivation and/or empathy and ability to communicate that lead to problems for doctors in their professional lives. If we accept that we want our doctors to have the non-cognitive skills to relay information to us as patients and also to have the cognitive skills to consider management/prognosis with us then it follows that we should accept our applicants to medical schools on both their cognitive and non-cognitive abilities. There is now worldwide support for this approach both in the UK from the Medical Schools Council and the GMC and in the US from the Accreditation Council for Graduate Medical Education³.

Widening participation so that medical students are representative of the population they serve has assumed importance politically. In 2008 the four lower socioeconomic classes accounted for only 15 per cent of medical students in the UK⁴. Some schools seek to actively redress this by selecting on aptitude rather than achievement, the latter being related to socioeconomic factors and type of school attended.

Given the highly competitive nature of selection which can be secretive and varies between Universities, the Schwartz report into fair admission to Higher Education has recommended the following five principles and two guidelines: 1) the selection process should be both transparent and should be published and available online; 2) selection should consider both achievement and potential; 3) selection methods should be reliable, valid and informed by best practice; 4) the predictive validity of selection methods should be monitored; 5) staff should receive training in selection processes; 6) there should be feedback to unsuccessful candidates; 7) barriers to selection should be minimised e.g. disability considered post selection⁵.

Given that attrition rates in medical school are low it can be argued the selection exercise is of even higher importance. Instruments used for selection include personal statements, academic references, tests of previous academic performance, aptitude tests, personality tests, random selection and interviews. Not surprisingly given the variety of instruments there is considerable controversy surrounding their most effective use.

Methods used to select Medical students

Personal Statements and Academic References

Ferguson et al followed up the 1995 cohort at Nottingham Medical School over a five year period. They used manifest coding to categorise applicant’s personal statements and academic references6. They found that information on the academic reference did not predict academic performance whereas there was a correlation between content matter in the personal statement and aspects of clinical performance. On the other hand in a recent paper which defined selector practice from Bart’s. and the London where personal statements are used to screen for interview, the authors commented that the study confirmed the subjective nature and low reliability of this process⁷. Personal statements are subject to plagiarism and UCAS has claimed that up to five per-cent of personal statements amongst eight hundred applicants to Medicine contained material borrowed from three online example statements. They have recently introduced copycat software to address this⁸.

In my experience of examining academic references over the last three years I have yet to find an unfavourable one.

Tests of previous academic performance

A-level results have been found to be a consistent predictor of academic performance in medical school in the UK⁶. At McMaster, grade point average has been found to be a predictor of academic performance and clinical performance in their graduate entry programme. Unfortunately A-levels/GCSEs have become less useful because of grade inflation, as the vast majority of applicants achieve 3A grades. Over the past 20 years the proportion of A-grades has risen from 9-27 per-cent⁹. For this reason the A star grade which is based on performance at A2 and is awarded to approximately the top ten per-cent across all A-level subjects has been introduced (70 per-cent of the 2010 cohort accepted on the basis of A-level performance to QUB Medical school had at least one A star and one student had five). Females perform better in both GCSEs and A-Levels than males¹⁰. It has also been argued that A-levels are biased and that grades can be affected by type of school attended. Widening participation was a priority of the Department of Health under its proposal in 2004 to increase the number of training places in the UK for medicine¹¹. For graduate entry the use of grade point average is fraught with difficulty because some degrees are seen to be easier than others and there is variability of grades awarded between different universities¹². Our own findings relating to the 2008 entry cohort to Medicine at Queens University Belfast (QUB) show that the best predictor in both first and second year examinations is GCSE performance (Cronbachs alpha 0.8).

McManus et al in a longitudinal study of students entering Westminster Medical school have shown that A-level grades have long term predictive value for undergraduate and postgraduate careers including time taken to obtain membership. The authors argue that A-level assesses achievement and that past achievement affects future achievement¹³. James has shown that for 1986-90 entrants to the undergraduate course at Nottingham medical school achieving a high grade in A-level Chemistry predicted success at BMedSci and a high grade in A-level Biology predicted success in BMBS¹⁴.

Aptitude Tests

These tests are designed to assess the applicant’s aptitude for medicine. They include the Graduate Medical Schools Admission test (GAMSAT) used both in Australia and for graduate entry in the Republic of Ireland, the Medical Colleges Admission test used in the US (MCAT), the Biomedical Admissions test (BMAT) and UK Clinical Aptitude test (UKCAT) used in the UK. The rationale behind these tests is that they should be free from bias, assess ability rather than achievement and would help distinguish between candidates scoring at the GCSE/A level ceiling. BMAT, GAMSAT and MSAT are in various formats but essentially consider written communication, critical reasoning and problem solving. They have a scientific component. The UKCAT consists of four cognitive subtests namely verbal reasoning, quantitative ability, abstract reasoning and decision analysis (all attributes felt to be important in medicine). It was first administered as an online test in 2006 by the UKCAT Consortium comprising 26/31 of the UK Medical schools by its agent Pearson Vue in testing centres throughout the world. Each cognitive test has a scale score ranging from 300-900 with a mean set to 600 using the 2006 reference cohort. Universities and candidates receive both subtest scores and the total test score which ranges from 1200-3600. A fifth test, currently administered for research purposes only, is a behavioural test intended to measure a number of non-cognitive competencies to include empathy, robustness and integrity: features associated with good doctors. Reliability scores of the cognitive subtests range from moderate to high. Scores showed a negative correlation with age; males performed better than females and the ethnic grouping black/British were the lowest performing group. The UKCAT Consortium claims it is a fair and reliable test¹⁵.

There have been several small studies on the predictive validity of UKCAT. One from Dundee/Aberdeen and one from Nottingham show no predictive validity whereas one from Newcastle does^16-18. Our own findings at QUB for our 2008 entry cohort show no relationship between UKCAT score (both total score and subtest scores) and performance in years 1 and 2 of the course. This may change in the later part of the course where there is more emphasis on clinical reasoning and problem solving. All these studies suffer from small numbers and problems with generalizability. The potential for the UKCAT project is that it now has 4 cohorts of applicants who have taken the test accompanied by progression data from member Medical schools. This is an enormous achievement with considerable potential and it is hoped that as a result the predictive validity of the test will be determined.

There is mixed evidence internationally for the predictive validity of aptitude tests. McManus followed a Westminster cohort of medical students for 20 years. Whilst A-levels were predictive of outcome a standard IQ test was not¹³. Tests such as MCAT in the US have shown predictive validity. Emery has shown predictive validity for preclinical examinations with BMAT but only includes one medical school and numbers are small¹⁹. Concerns have been raised about UKCAT. A recent study from Scotland of 1st year medical students has raised concerns about the cost and fairness of the test , the use of the data in selection processes by medical schools and the influence of test preparation20. UKCAT state that 6.5 per-cent of their applicants receive bursaries and that their website offers two timed practice tests which address issues around candidates needing to pay commercial organisations for test preparation¹⁵. A recent study of A- level and UKCAT performance in applicants applying to UK Medical and Dental schools in 2006 found that UKCAT moderately correlated with A-level and that the total score did provide a useful proxy for A-levels in the selection process. It did show a bias toward males and social class 1 applicants21. The definitive verdict on how much UKCAT gives added value to the selection process should be available in the next few years and is keenly anticipated.

Personality testing

Ideally doctors should be safe practitioners who manifest considerable job satisfaction. We know that introverted, neurotic doctors can burn out although conversely the same traits can be associated with safe practitioners²². There is no consensus of opinion on the personality best suited for the practising physician. Doherty and Nugent have reviewed seven longitudinal studies which examined student’s scores on valid personality tests and compared these with outcome measures around performance and stress. The studies come from UK, Belgium, US and Norway. Four of the studies looked at personality factors and academic success, one considered personality and clinical competence and two looked at relationship between personality and stress. The authors concluded that not only does conscientiousness predict long term success in medical training but also vulnerability to stress if it is accompanied by high levels of neuroticism and low levels of extraversion²³. These findings are in keeping with job performance findings in other professions²⁴.

Powis claims that the inclusion of personality tests in selection to Australian Medical schools significantly adds to the ability to predict candidates who will perform well in the course²⁵. UKCAT includes psychological tests within its battery of non-cognitive tests which are currently being administered for research purposes only and are not currently being used in the selection process.

Random Selection

Dutch Medical schools select sixty per-cent of their intake using predetermined criteria and the remainder by national ballot. Applicants can apply three times. Dutch students are in favour of this as they argue that it redresses the power of the Medical schools²⁶. In 2003 Queen Mary’s London, previously referred to as Bart’s. and the London introduced a ballot for entry into its graduate entry programme. The negative publicity created by the ballot led to its withdrawal.

Interviews

Traditional interviews are used as part of the selection process in most medical schools in UK, US and Australia. The process usually comprises two or three interviewers using either unstructured or semi-structured questions. Medical students, NHS staff, the public and academics have all been used as interviewers. Traditional interviews are subject to bias in terms of gender, age, race, halo effects, hawk/dove interviewer effects, tendency to place weighting on unfavourable information, effect of similarity/dissimilarity of interviewer to interviewee etc²⁷. In addition the interview process can be affected by the impact of the previous candidate and the effects of pre-interview information about the applicant. While the face validity of interviews is strong Kreiter has shown that they have a low to moderate reliability and has called into question the fairness of the interview in the selection process²⁸. Harasym et al in their paper on reliability and validity of interviewers judgments using simulated candidates have shown that interviewer accuracy was only moderate (56 per-cent) and questioned the validity and reliability of two person interviews²⁹. The context of the interview can also affect outcome i.e. the individual’s ability to problem solve can be affected given differing scenarios. Even though examiner reliability can be improved by training and the use of semi-structured questions a single interview in a specific context may not provide a true assessment of the applicant’s ability. Similar arguments led to the development of the Objective Structured Clinical Examination which tests clinical skills using a multiple sample approach. Eva and his colleagues from McMaster coined the term Admissions OSCE or Multiple Mini Interview(MMI) in response to concerns about the reliability of interviews³⁰. This consists of multiple stations each with a different assessor, set in different contexts (non-clinical) designed to test predetermined non-cognitive competencies held to be important for a medical career. The fresh start with each station dilutes hawk/dove assessors and allows independent assessment in multiple situations. His original paper tested for the domains of critical thinking, ethical decision making, communication skills and knowledge of the healthcare system. A reliability coefficient (Cronbach’s Alpha) of 0.65 was reported which is much higher than that associated with traditional interviews. Axelson and Kreiter have shown that increasing the number of interview stations added more to reliability than increasing the number of interviewers per station³¹. Reiter et al have looked at violations of test security. They describe several studies where MMI station stems were randomly allocated to some groups of applicants. This did not influence applicants’ performance ranking³². Rosenfield in another study from McMaster concluded that although MMIs require greater preparatory effort and the need for an assessment centre they require fewer person hours and have cost advantages over traditional interviews³³. Humphrey et al used MMIs to recruit paediatric SHOs in Warwickshire. Both candidates and interviewers agreed that the process was a fair and acceptable tool for selection in UK specialist training³⁴. Eva has shown that for Canadian students the correlation between admissions MMI and the number of stations passed in the OSCE component of the Canadian Qualifying Examination required for licensure was r=0.43(p<0.05). It was also a better predictor when compared with other selection instruments used in McMaster³⁵.

Graduate –V-Undergraduate Entry to Medical School

In North America medical school programmes are almost exclusively 4 year graduate entry programmes (GEP) whereas in Australia and the UK there is a mixture of GEPs and undergraduate programmes to which graduates can be admitted. James in his retrospective study of predictors of success in the undergraduate course in Nottingham Medical school 1970-95 showed that mature or graduate entrants were more successful in obtaining a first class BMed.Sci. degree but were less successful in passing BMBS¹⁴. A study comparing the academic performance of graduate and undergraduate entry medical students completing the same preclinical curriculum at the University of Melbourne showed that graduate students performed better in all four bioscience assessments and also on early clinical skill assessments³⁶. This study reflects our own findings at QUB for the 2008 entry cohort in that although our numbers of graduates are small over the first 2 years of the course, the graduates performed better in all tests and significantly so in yr1 Bioscience exams and across all yr 1 and 2 Objective Structured Clinical Exams. (Table1)

Table 1.

Graduate v Undergraduate performance in Examinations for Y1 and2 of the Medical Course QUB (2008 entry)

Group Statistics
	Graduate	N	Mean	Std. deviation	Std. Error Mean	P value
Bioscience exams Y1	no	220	68.51	7.12	0.48	0.03
	yes	15	72.58	5.33	1.38
Bioscience exams Y2	no	219	70.89	8.16	0.55	0.18
	yes	15	73.79	7.24	1.87
Three Clinical Exams(OSCE) Y1 and 2	no	219	74.5	7.42	0.50	0.03
	yes	15	78.82	6.95	1.80

Arguments for graduate entry include higher motivation, widening diversity, faster production of doctors and proven ability in an academic tertiary environment. School leavers on the other hand are close to the academic ceiling and have good study skills. Although there is evidence that graduate entry widens diversity there is no evidence that graduates make better doctors³⁷.

In March 2007 the QUB/DHSSPS Strategic group met to discuss graduate entry to Medicine at QUB³⁸. A separate graduate programme was rejected because of cost and the sustainability of two courses competing with each other side by side for placements/resources. It was felt that instead there should be multiple entry points into the Medical course and a strategy was developed to double the percentage of graduates from 6 per-cent in 2007 to 12 per-cent. It was felt that this mix of students some of whom had already completed a degree programme would bring diversity to the student experience given the mature approach of graduates and their different life experiences when compared to school leavers

Selection for Medicine (2012 entry) at QUB

There is considerable variation in the selection tools used by Medical schools in the UK. Most schools interview but criteria for shortlisting for interview vary from evidence of previous academic ability, performance in aptitude tests, predicted performance at A-level, scoring of personal statements or a combination of the above³⁹.

Relaying bad news to a cancer patient requires both communication skills and empathy along with cognitive knowledge of the management options and prognosis. The desired endpoint is not a bookworm or a butterfly but a well rounded doctor who exhibits both cognitive and non-cognitive competencies³⁵. This concept along with best evidence on selection from the literature and our own research findings at QUB on the predictive validity of our selection tools has fashioned a change in our processes for 2012 entry.

Alongside this has been our strategy to internationalise the school and to encourage graduate entry to medicine. Our 2010 entry cohort includes 17.6 per-cent graduate entry, 16 per-cent from GB and ROI and 5 per-cent are international students. Fifty four per-cent of the 2010 intake were female. 20.8 per-cent of our 2009 entry cohort were from socio-economic groups 4 and 5; higher than most other schools⁴. The Department of Health (N.I.) has recently increased our international numbers from 12 to 26 for 2011 entry in keeping with the proportion of international students in other UK medical schools. We are currently actively recruiting both students and staff from South East Asia and North America.

For 2011 entry we had 850 plus applicants competing for 236 EU and 26 international places. There are a number of entry pathways into the Medical school and these include Y14, post A- level and graduate entry.

It has been agreed by both the School and University that for 2012 entry there will be a two stage Admissions process. In keeping with the best practice this process is transparent, has been published , is available on line⁴⁰ and the predictive validity and reliability of our selection instruments is monitored in keeping with Schwartz guidance⁵. The first stage will recognise past academic achievement in keeping with evidence from the literature along with recognition of the importance of aptitude tests (UKCAT). We do not exclude applicants on the basis of aptitude tests alone. In stage one which is cognitive, applicants will be scored and ranked as follows

For Y14 applicants the best 9 GCSEs will be considered with 4 marks for an A star and 3 for an A. Maximum 36 points

For graduates holding a 2:1 Honours degree or better (or predicted to achieve same) and who hold 3Bs at first attempt(ABB from 2013 entry) in the specific subject requirements at A level-36 points will be allocated.

For post A -level applicants who already have 3As at A-Level and an A at AS Level 36 points will be allocated.

For ROI applicants the best 9 junior cert intermediate are considered with 4marks for an A and 2 marks for a B. The maximum mark is 36 points

All applicants will take UKCAT in the year of entry and their overall score will attract up to six points The distribution of total UKCAT scores for our 2011 entry cohort is shown in Table 2. This score will be added to their knowledge based score and all applicants ranked. The top circa 500 applicants will then be considered under stage 2 of the selection process which will consist of a nine station multiple mini interview to determine non-cognitive performance. Multiple Mini Interviews are used to test non-cognitive competence in keeping with best evidence available from the literature. The applicant’s personal statement is considered within this process. The non-cognitive competencies which are tested have been determined using a Delphi technique by both the public and Faculty and have been published⁴⁰

Table 2.

Ukcat banding scores of eu applicants applying for medicine at qub in 2011

Band Score	Scoring Range	Medicine Banding Total 2011 APPLICATION	%
0	1200 - 1299	6	1
	1300 - 1399
	1400 - 1499
	1500 - 1599
	1600 - 1699
	1700 - 1799
	1800 - 1899
1	1900 - 1999	22	3
	2000 - 2099
2	2100 - 2199	87	11
	2200 - 2299
3	2300 - 2399	198	24
	2400 - 2499
4	2500 - 2599	259	33
	2600 - 2699
5	2700 - 2799	160	20
	2800 - 2899
6	2900 - 2999	63	8
	3000 - 3099
	3100 - 3199
	3200 - 3299
	3300 - 3399
	3400 - 3499
	3500 - 3600
	Total	795	100

Final decisions about whether or not to make an offer will be made on the basis of interview ranking alone (i.e. Stage 2 results) and not in combination with other factors. During the 2011 entry process, approximately 200 applicants took our MMI. Each station lasted for 5 minutes and the examination was blueprinted to test for motivation, communication, empathy, problem solving, integrity and ethical reasoning. One third of our stations involved role-players and the others were semi-structured interviews. Prior to the interviews all assessors were trained and all participated in a standardisation process on the day of the assessment. For our 2011 MMIs the Cronbach’s alpha was 0.56. and there was a Gaussian distribution of marks from 30-85 per-cent for candidates. The MMIs were standard set using the borderline regression method and offers were made to applicants who reached the cut score as determined by the panel of assessors. The school is currently actively recruiting assessors and these positions are open to both clinical academic, non-clinical academic and NHS staff. The MMI process is a considerable challenge for us and will require 120 days of assessor time per annum.

While some of our Admission instruments will favour certain groups we try to achieve an overall balance and are currently monitoring all of our selection tools to ensure equality.

Further details regarding the admissions process for Medicine at QUB along with video clips of MMIs are available on www.qub.ac.uk/schools/mdbs/medical/Prospectivestudents.

We feel we now have a selection process which meets GMC recommendations in that it is transparent, objective, uses a variety of selection tools which are constantly monitored in keeping with best practice and also considers both cognitive and non cognitive factors. We hope this approach will widen participation compared to the previous approach which was largely cognitive and relied on selection using very narrow parameters. Our approach to selection also has an advantage in that NHS colleagues, the wider public and academic staff will all have an input into selecting tomorrow’s doctors.

Conflict of Interest

The author is a Director on the UKCAT Board and a member of the UKCAT Development and Research groups