Can we improve on how we select medical students?

Getting the right policy for admission to medical school is a balancing act: be fair to society by choosing people with the potential to be good doctors; and be fair to the applicants—that diverse group of people who for many reasons want to set out on the long road to a medical career. Selection is not an exact science but we must use what evidence we have to ensure that we do our best by all concerned. There is widespread agreement that we should select future doctors on wider criteria than scores of academic success^1,2, though in practice many medical schools have valued pre-admission academic scores at the expense of other considerations³. There are recognized drawbacks to the use of school exam performance even as a measure of intellectual competence. One study has shown that a major causal determinant of A level results is social class, independent of ability⁴, and some would-be medical students elect to focus on sciences for their school leaving exams because very high marks are more easily achieved in the physical sciences than in the humanities⁵. The conviction that only exam results give valid and reliable data has been trenchantly dismissed as a ‘seductive but fallacious’ belief in the precision of quantitative tests⁶. We are reminded that all selective instruments depend on subjective judgments and each must be accountable to the rules of reason, fairness and public scrutiny⁷. However, if we decide to consider non-cognitive criteria, a legitimate concern is that the many specialties of medicine need diverse skills and they must not be too narrow. We also want to be reassured, if we include noncognitive characteristics, that we can assess them reliably and that such evaluation can predict personal character over years of practice.

While we need to maintain diversity of skills and personality, there are some characteristics which we demand in any doctor. Enough intellectual ability to do the job, plus honesty, integrity and conscientiousness, must be at the heart of good practice⁸. Helpfulness and willingness to cooperate come close behind⁸, while patients give high priority to interpersonal skills and empathy². The personal welfare of the profession is another consideration⁹. Doctors are more vulnerable than comparable professional groups to alcoholism, drug abuse and suicide^10,11. Burnout is well recognized, and has a high cost for the individual, for colleagues and for the quality of service that patients get¹². One answer may be better support for psychologically vulnerable doctors^12,13 (together with improved working conditions for all doctors), but perhaps we should try to evaluate ability to deal with stress right from the start.

ARE PERSONALITY CHARACTERISTICS STABLE OVER ADULT LIFE?

If we seek to identify the personal characteristics we want in a medical student, can we have any confidence that they tell us anything about future personality or adjustment? Studies that assessed medical undergraduates and followed them up for between 15-30 years^12,14,15 indicate that doctors who are psychologically well in middle age had good self esteem as students, had an open, flexible approach to life, enjoyed a warm relationship with their parents, and had little anxiety and depression and low anger under stress. In contrast, doctors vulnerable to later substance abuse, to suicide and to burnout in middle age had significantly poorer measures of psychological health as undergraduates. Other long-term studies of stability of personality characteristics have shown that personality traits exhibit high test-retest correlations over intervals of 6 to 45 years^16,17,18,19. These findings signify a substantial continuity of personality disposition in adulthood, suggesting a stable tendency to be either happy or unhappy, well or poorly adjusted.

WHAT FACTORS GENERALLY PREDICT FUTURE JOB PERFORMANCE?

There is relevant research both within medicine and outside it. Useful information comes from industry, where serious money has gone into finding out what makes a good professional²⁰. They measure outcome in hard cash and find that the most productive people are about 40% better than average, while the least are 40% worse than average²¹. Is this too different from medicine to be relevant? Look around: we all know who gets the work done and keeps up to date, and who slips through life doing the minimum. These are not the only criteria for a decent doctor but they matter. There is consistent evidence that, for work involving complex tasks, the best predictor of effectiveness is some measure of mental ability or IQ, and the higher you go up the professional scale the more IQ matters. At the highest managerial level it accounts for almost 70% of performance variability²². So in demanding evidence of high IQ (even in the form of exam results) we have got something right. Predictability can be improved by including some measure of other factors. Further factors consistently found to add to prediction of performance are integrity and conscientiousness: these do not correlate with IQ²³. No additional predictability comes from the number or nature of outside interests; years of education adds little to predictive validity; and the number of courses a person has been on is of no value (so much for how we measure ‘continuing professional development’). Previous job performance adds to prediction for those already in the profession, but adds nothing at entry. Some of these results are counter-intuitive: this is because IQ overlaps with other things. So a quick learner will have good performance in a previous job which will correlate so highly with IQ that it adds little to predictive validity²⁰.

WHAT FACTORS PREDICT ACADEMIC FAILURE IN MEDICINE?

The first thing that strikes anyone exploring the work on predictors in medicine is that we are obsessed with exam results: by far the largest number of papers examines predictors of passing exams. This may be justified because of the economic and personal waste of losing students who begin a medical degree but fail to complete, with loss from schools that select at entry, both in the UK and elsewhere, generally reported between 8% and 10%^24,25,26,27. However, most studies assess ‘failure’ in broad terms to include all students who re-take an examination, as well as those who are excluded from the course, so predictors should be treated with caution. Although virtually all students are high academic achievers at school, from the top 0.4%⁸ to the top 10%²⁹, school and medical exam scores do correlate, with contribution to variability reported between 16%²⁹ and 58%³⁰. Some UK studies show that certain science A levels predict exam success, variously putting biology, chemistry or physics in prime place^31,32,33, and research from outside the UK reports associations between performance in physical sciences and in medical exams^34,35,36. Generally this association falls later in the course, with no difference to longer term success or failure^37,38,39,40.

Non-academic factors also predict exam success or failure. Some researchers report that older students are more likely to fail exams^36,38,41, but others have not found this⁴². Several US studies found higher failure rates among women and ethnic minority students, although most eventually graduate^36,38,41, and one school reported that students admitted through affirmative action were as likely to graduate as those admitted by use of traditional criteria⁴³. Proficiency in English is important for students for whom English is not their first language^44,45, and in the US, reading skills of disadvantaged minority students have been shown to predict academic success⁴⁶. Non-cognitive factors are stronger predictors for women and ethnic minority students than for white men in the US. For women, interview ratings and previous relevant experience were more predictive than previous exam scores⁴⁷, while for ethnic minority students, locus of control and ability to self-evaluate were predictors^48,49. One US study showed that different cognitive and non-cognitive factors correlate with academic success in different schools, so different cultures and teaching styles influence outcome⁵⁰.

It has been argued that we cannot reduce loss further⁵¹, because some failure is inevitable and we cannot avoid a few students' wanting to change career. However, two medical schools have shown that careful selection and good support can have a positive impact. In Newcastle, New South Wales, for five years 50% of students were selected on academic marks alone but underwent a lengthy structured interview which was not used for selection. As a result, some students were admitted with very low interview scores. The remaining 50% were selected from a wider band of academic performance but scored high in interview. Analysis after ten years showed a significant correlation between low interview score and later drop-out but no correlation between academic score at entry and drop-out. Reasons for dropping out were academic failure or a variety of personal reasons, including lack of motivation for study or for medicine²⁸. Another example of low drop-out comes from McMaster University in Ontario, which also invests heavily in selection and in addition offers ‘remediation’ for students having academic difficulty. In one five-year period in a class of 100 students, only one student was excluded because of academic failure, 3 changed careers, while 8% had remedial help⁵².

WHAT PREDICTS GOOD CLINICAL PERFORMANCE?

Investigators looking for early predictors of what makes a good clinician generally use reports from clinical clerkships and from the house officer or intern year. However, we should note that drop-out will mean that some unsatisfactory students will have left before the house officer year. Clinical performance is not generally predicted by pre-entry academic scores^{1,35,53,54,55,56,57}: the one report of correlation between matriculation scores and clinical performance noted that matriculation scores included 50% contribution from school teacher assessment⁵⁸. Neither age nor gender predicts clinical performance, nor does previous study of physical sciences, but there is evidence that previous study of English and humanities correlates with better clinical performance^5,34,59. There are some reports of association between clinical performance and admission interviews^55,56,60,61, although others reveal no correlation^54,58. In a school that carefully evaluates applicants, empathy and motivation to be a doctor were found particularly important in predicting both clinical and academic success⁶².

WHAT ARE THE MOST RELIABLE PROCEDURES TO ASSESS PREDICTORS OF FUTURE PERFORMANCE?

If we can agree that there are certain characteristics that we want to select in prospective doctors, what is the best way of doing this? Research shows that, if we want to add usefully to a measure of intellectual ability in predicting later job performance, our best instrument is the structured interview. While an unstructured interview adds about 8% to prediction of subsequent performance, the structured interview adds around 24%⁶³. Psychometric tests to measure desirable personal characteristics do predict future performance, but their validity may be compromised if they are used as a selection tool: the desired answer is not usually difficult to identify, and applicants who lack integrity are the most likely to manipulate the results⁶⁴. However, some schools have applied psychometric tests at the point of entry rather than using them to select, and have found correlation between these tests and scores given in interview^65,66. This suggests that a well conducted interview may give similar information and that, if constructed to assess desired characteristics such as conscientiousness or helpfulness, it will give a reasonably reliable evaluation²⁰.

Character references from a previous employer or tutor have potential to add to prediction. However, legislative changes in the US in the 1980s meant that an employer giving an adverse report could be sued by the employee: as a result, the predictive validity of personal references in the US has fallen to almost zero²⁰. The reliability of UCAS references in the UK may be similarly threatened. The motivation of the referee is uncertain: some tutors may feel their first loyalty to their student, others may feel compromised by recent data protection legislation that removes the confidentiality of previous years. One medical school in New Zealand has adapted the traditional reference system by writing to head teachers with specific questions, and requesting a rating of the candidate's qualities against the level the head teacher believes to be desirable in a doctor. The long-term predictive validity of this method has not been published, but the school believes it provides valid information and correlates well with other non-cognitive indices (and not at all with academic scores)³⁵. Some schools, particularly those which do a lot of small-group work in the course, use an assessment of performance in small groups as a ‘live’ way to assess interpersonal skills^29,52. Evaluation of students in this setting correlates highly with interview scores, and is reported to predict both problem-solving ability and group interaction⁵².

WHAT CONSTITUTES CURRENT BEST PRACTICE?

In summary, the evidence is that we need to select students with good intellectual ability and that examinations, despite limitations, have some validity. For some candidates—e.g. older applicants, or those from disadvantaged social backgrounds—we may want to look for reliable measures of intellectual ability other than the traditional A levels. We seek individuals who are conscientious and have integrity, who are empathic and motivated to become doctors, and who are psychologically robust enough to enjoy a successful medical career. Some medical schools, mainly outside the UK, have already recognized best practice and have put great care and resource into their selection procedures, with well-planned structured interviews, focused reports from schools and evaluation of interpersonal behaviour. As detailed above, there is evidence that this investment is worthwhile in terms of the suitability of students selected, and economically in terms of student loss during the course.

AND WHERE ARE WE IN THE UK?

The greatest single barrier to a more careful selection process in the UK is the amount of resource that each school has to invest. At present, would-be medical students apply to up to four medical schools. All but four of the UK's present twenty-four medical schools interview about 500 to 1000 applicants for their five or six year MB BS courses. Many interviews are still unstructured, and not all schools require their interviewers to be trained. It is unusual for the interview to be more than 15 or 20 minutes, and while brief interviews may be reliable⁶⁷ the validity of a 15-minute interview is doubtful⁶⁸. The fact that many candidates are interviewed four times underlines the wastefulness of our present national procedure, but the cost to individual schools to improve radically would be prohibitive. Our present system does not offer society the best practice available: at present we almost certainly turn away people who would make good doctors and accept some who will be mediocre or poor. We could probably reduce loss from the medical course, and so save money and save personal distress among those who were allowed to make an unwise choice. We could also be more just to applicants, and begin the process of education by showing that we are very serious about the kind of personal qualities that we want in a doctor.

The Civil Service, the Armed Forces, and many business corporations have had selection boards for many years: the Civil Service believe these to be money well spent, and industry has gone further and demonstrated their cost effectiveness²⁰. Those medical schools which invest heavily in their selection procedures admit that it is not cheap: on the other hand, it is not cheap to lose students unnecessarily or to employ a poorly motivated or unhappy doctor. There is a strong argument for pooling resources so that applicants get one good assessment instead of four poor ones. This does not preclude medical schools' maintaining individuality and some degree of choice, and candidates will continue to visit schools and attend open days. However, it is time that UK medical schools got together to collaborate in setting up a first-class selection process that is fair to society and fair to all those people who hope to be the doctors of tomorrow.