Is the admission test for a course in medicine a good predictor of academic performance? A case−control experience at the school of medicine of Turin

Abstract

Objectives

The usefulness of university admission tests to medical schools has been discussed in recent years. In the academic year 2014–15 in Italy, several students who failed the admission test appealed to the regional administrative court (‘Tribunale Amministrativo Regionale’—TAR) requesting to be included, despite their test results, and all were admitted to their respective courses. The existence of this population of students generated a control group, in order to evaluate the predictive capacity of the admission test. The aim of the present work is to discuss the ability of university admission tests to predict subsequent academic success.

Setting and participants

The study involved 683 students who enrolled onto the first year of the degree course in medicine in the academic year 2014–15 at the University of Turin (Molinette and San Luigi Gonzaga colleges). The students were separated into two categories: those who passed the admission test (n1=531) and those who did not pass the admission test but won their appeal in the TAR (n2=152).

Outcomes

The validity of the admission test was analysed using specificity, sensitivity, positive and negative likelihood ratios (LH+, LH−), receiver operating characteristic (ROC) curves, area under the ROC curve (AUC), and relative (95% CI).

Results

The results showed that the admission test appeared to be a good tool for predicting the academic performances in the first year of the course (AUC=0.70, 95% CI 0.64 to 0.76). Moreover, some subject areas seemed to have a greater discriminating capacity than others. In general, students who obtained a high score in scientific questions were more likely to obtain the required standards during the first year (LH+ 1.22, 95% CI 1.14 to 1.25).

Conclusions

Based on a consistent statistical approach, our study seems to confirm the ability of the admission test to predict academic success in the first year at the school of medicine of Turin.

Strengths and limitations of this study.

• This is the first study based on a consistent statistical approach and conducted in an Italian university to evaluate the reliability of the Italian university admission test.

• The presence of the TAR (‘Tribunale Amministrativo Regionale’) control group improved the reliability of the results.

• Comparability of the groups: the registration year, lessons, programmes, teachers and classrooms were the same for both regular and TAR students rendering the comparison of the two groups highly valid.

• Further investigations will be required to analyse the two cohorts over a longer period of time.

• The predictive capacity of the test was only studied here in relation to academic performance in the basic courses; a relationship with clinical skills was not considered in this work.

Introduction

In recent years, the growing demand for higher qualifications has created both economical and technical problems. Indeed, as discussed by Reibnegger and colleagues,¹ the number of classrooms, laboratories, infrastructures, technical staff and teachers has had to increase contemporaneously in order to manage large numbers of students without compromising teaching quality. Moreover, the level of youth unemployment has raised important questions about the number and quality of graduates with respect to job opportunities, suggesting the need for more stringent selection procedures.

The usefulness of university admission tests, in particular for degree courses in medicine, has been widely discussed in recent years, both in Europe^1–3 and in other continents.⁴ The USA was the first to use admission tests in student selection procedures (eg, the Moss test in 1928) and different versions have been created over the years. The most recent test, the MCAT (Medical College Admission Test), was created in 2007 and is now used by almost all the colleges in North America.² In regard to European countries, no standards have been formulated to date. The European Union has provided general advice only regarding the quality of education that points towards progressive standardisation. Each country is thus allowed to take personalised actions: the English (UKCAT³—UK Clinical Application Test) and Austrian¹ tests were created in 2006, the Irish test² (HPAT—Health Professions Admission Test) was formulated in 2009, while France has never introduced an admission test, preferring the strategy of simply barring students who do not make the grade at the end of their first year from progressing in their course.⁵ Other approaches have also been considered, including a totally open access to courses (eg, as applied in Austria until 2002)¹ and random selection (eg, in Holland until 1999).⁶ In Italy, an admission test was proposed in 1987 by Zecchino (the Minister for Public Education) and finally introduced in 1999 as a law (264/99). The required skills, and how to test them, change from country to country (table 1).

Table 1.

Comparison of admission tests in different countries

	Italian admission test	HPAT	UKCAT	UMAT	MCAT
Country	Italy	Ireland	England	Australia	USA
Type of test	MCQ	MCQ	MCQ	MCQ	MCQ
Subsection of test	General culture Logic Biology Chemistry Mathematics and physics	Logical reasoning and problem solving Interpersonal understanding Non-verbal reasoning	Verbal reasoning Quantitative reasoning Abstract reasoning Decision analysis	Logical reasoning and problem solving Understanding people Non-verbal reasoning	Logical reasoning and problem solving
Duration	1 hour 40 min	2 hour 30 min	2 hours	2 hours 45 min	4 hours 30 min
Reference	Decreto Ministeriale n. 9867	Kelly et al 2	Sartania et al 3	Kelly et al 2	Prideaux et al 4

HPAT, health professions admission test; MCAT, medical college admission test; MCQ, multiple choice question; UKCAT, UK clinical application test; UMAT, undergraduate medicine and health science admission test.

A good review of the test types can be found in Prideaux et al.⁴ In brief, the tests can be divided into cognitive, non-cognitive, and written tests and interviews. Moreover, in some cases (USA, UK, Australia, Canada, New Zealand and South Africa), considerable importance is given to ethnic minorities and disadvantaged groups in order to facilitate their admittance. The general trend is to use written cognitive tests, while almost all avoid interviews⁴ because they are less predictive and more time- and money-consuming. In Italy, the test is administered nationwide for the public universities, prepared by the Ministry for Education, Universities and Research (Ministero dell’Istruzione, dell’Università e della Ricerca—MIUR). The test is written and comprises 60 multiple choice questions (MCQs) to be answered within a limited time⁷ (see table 1 for details). Moreover, access for disadvantaged groups and non-Italian people is guaranteed by the reservation of places for these categories.

In Italy, a debate is ongoing about the present situation and the possibility to improve the admission test. Investigations into how the results of the admission tests can predict academic success are thus required. Indeed, many European studies highlight a tight relationship between admission test results and academic performance,^{3 8} personal skills² and level of motivation.¹ However, other authors (eg, Yates et al ⁹) have reported discordant results regarding the same tests. The large majority of these studies have concerned situations outside of Italy.^{6 10–12} Thus, literature on the admission test results in Italy is very scarce and not specific to medicine (see Lancia et al ¹³ regarding nurses and Mannella¹⁴ for veterinary medicine). The present statistical study (although involving a single university) therefore makes an important contribution to the discussion on the usefulness of admission tests, both for Italian and non-Italian readers.

Another consideration that should be made regards the selection bias of the majority of past studies—that is, the lack of adequate control groups. Indeed, only the results of students who had passed the admission test could be considered (as those failing the test were not admitted to university), and comparisons could only be made against previous cohorts (ie, the students who entered the university without being tested)¹; however, in this case no information about the scores of the comparison group is present.

In Italy, in the academic year 2014–15, many of the students who failed the admission test appealed to the regional administrative court (Tribunale Amministrativo Regionale—TAR) on the account of supposed irregularities that occurred during the examination. The court accepted the appeal of recursive students on the basis of a ‘supposed infringement of anonymity principle’ and granted them admittance onto their respective courses. This ‘extraordinary’ situation generated a ‘control group’ of students (with lower test scores) useful for evaluating the predictability of the admission test.

Objectives

The aim of the present work was to evaluate whether a university admission test for a degree course in medicine can predict subsequent academic success in medical students, taking advantage of the particular 2014–15 cohort of students from the University of Turin

Methods

Setting

In 2016 a working group with the aim of assessing the predictive effectiveness of an admission test was organised by the Italian institution composed of presidents of the Italian degree course in medicine (named ‘Permanent Conference of Presidents of Degree course in Medicine’ which allows homogeneity and coordination of the schools of medicine). Preliminary results of the group’s activities (relative to just some Italian colleges), which will be published in the Permanent Conference’s Journal, highlight the need for changes to be made to different colleges, which do not always propose comparable curricula and whose courses and examinations are organised differently. Within this framework, the two constituent medical colleges of the University of Turin (Molinette and San Luigi Gonzaga) have long been developing this line of research on their students.

Database

The study was approved by the Degree Course Council for the School in Medicine of the University of Turin (Molinette and San Luigi Gonzaga colleges) and by the Students’ Committee. Approval by an ethical board is not explicitly required in Italy when the analysis of retrospective data is carried out, especially when data do not deal with disease conditions or the use of pharmaceutical products. In order to meet the requirements of the Helsinki Declaration, the analyses were performed on an anonymised database (without sensitive data) provided directly by the medical schools.

The present study involved 683 students who enrolled onto the first year of the degree course in medicine in the academic year 2014–15 at the University of Turin (Molinette and San Luigi Gonzaga colleges). The students were divided into two categories: those who passed the admission test (Regular, n1=531) and those who did not pass the admission test but won their appeal in the TAR (TAR, n2=152).

All students were monitored until the end of the first year’s last exam session (January 2015 to May 2016).

Statistical methods

The following data are presented as means, standard deviations (SD), median and 95% confidence intervals (95% CI) for the two investigated student groups (Regular and TAR): admission test score; secondary school final grade; number of university exam credits (CFU) acquired; the average first-year exam grade.

Considering the number of CFU accumulated in the observation period as the principal end-point measure, students were classified into the following categories defining two different reference standard (RS):

RS1

• Students who acquired half, or more than half, of the required credits at the end of the first year (P1)

• Students who acquired less than half of the required credits at the end of the first year (N1)

RS2

• Students who acquired all the CFU required for the first year (P2)

• Students who did not acquire all the CFU in the first year (N2)

In order to evaluate the predictability of the admission test, the achievement of RS1 and RS2 (independent of the student categories Regular and TAR) was evaluated using receiver operating characteristic (ROC) curves.^15–17 Indeed, the ROC curve illustrates the ability of the admission test to discriminate true positive cases (sensitivity) from false positive (1-specificity) cases. If the test has high predictive capacity the curve grows rapidly; this shape should produce a large area under the ROC curve (AUC) reported with relative 95% CI. An AUC value >0.5 and close to 1 indicates a good level of predictability of the test.

In order to analyse the weight of each of the subject areas on the predictive capacity of the test, the ROC curve, the AUC and relative 95% CI were calculated for total score and for the individual sub-areas of the test.

Finally, we evaluated the ‘goodness’ of the cut-off score used at the University of Turin (33.9) for discriminating between admitted and non-admitted students. The analysis was based on sensitivity (Se), specificity (Sp), positive likelihood ratios (LH+), negative likelihood ratios (LH−), and relative 95% CI values.^{15 18 19}

Results

Table 2 shows the number, mean secondary school final grade, CFU, and mean first year exam grade of the students enrolled in the first year according to group (Regular and TAR), whereas table 3 presents the scores achieved in the admission tests (total and per subject area). The results show that students comprising the Regular group obtained higher test scores and more CFU at the end of the first year than TAR students.

Table 2.

Number of students, secondary school final grade, university exam credits (CFU) and mean first year exam grade in each student category (Regular vs TAR) attending the two universities of Turin

		Total			Molinette			San Luigi
		Secondary school final grade (max 100)	CFU acquired (max 51)	Mean first year exam grade (max 30 L)	Secondary school final grade (max 100)	CFU acquired (max 51)	Mean first year exam grade (max 30 L)	Secondary school final grade (max 100)	CFU acquired (max 51)	Mean first year exam grade (max 30 L)
Regular	Mean (SD)	88.2 (10.2)	36 (16.0)	26.5 (2.1)	88.5 (10.3)	37.3 (17.2)	26.6 (2.0)	87.9 (9.8)	35.1 (12.3)	26.3 (2.4)
	Median	89	40	26.7	90	44	26.9	89	37	26.5
	N	531	531	492	402	402	368	129	129	124
TAR	Mean (SD)	79.1 (10.6)	27.4 (18.9)	24.5 (2.3)	80.7 (10.4)	28.2 (20.1)	24.9 (2.3)	77.5 (11.0)	26.6 (15.1)	23.8 (2.2)
	Median	78	27	25	80	30	25	77.5	26.5	24
	N	152	152	123	124	124	100	28	28	23
	Mean difference (95% CI)	8.4 (6.7 to 10.2)	9.1 (6.3 to 11.9)	1.9 (1.5 to 2.3)	7.8 (5.7 to 9.9)	9.1 (5.5 to 12.3)	1.8 (1.3 to 2.2)	10.4 (6.2 to 14.5)	8.4 (3.2 to 13.7)	2.5 (1.4 to 3.6)

CFU, university exam credits; TAR, tribunale amministrativo regionale.

Table 3.

Average score achieved in the admission test per student category (total and per subject area)

		Biology	Chemistry	General culture	Physics and mathematics	Logic	Total
Regular	19.5 (4.8)	8.9 (3.2)	6.2 (2.9)	0.7 (1.1)	6 (3.1)	19.5 (4.8)	41.2 (6.2)
	Median	8.9	6.3	0	5.6	19.8	39.9
	N	531	531	531	531	531	531
TAR	Mean (SD)	5.1 (3.6)	2.8 (2.7)	0.3 (1.0)	2.3 (2.4)	13.3 (4.3)	23.9 (6.7)
	Median	5.5	2.5	0	1.8	13.4	25
	N	152	152	152	152	152	152
	Mean difference (95% CI)	3.8 (from 3.2 to 4.4)	3.4 (from 2.9 to 3.8)	0.4 (from 0.1 to 0.6)	3.7 (from 3.1 to 4.2)	6.2 (from 5.3 to 7.1)	17.3 (from 16.2 to 18.5)

TAR, tribunale amministrativo regionale.

Based on the ROC curve analysis, the admission test appears to be good at predicting RS1 achievement (AUC=0.67, 95% CI 0.63 to 0.71; table 4). The specific analysis performed for different sub-areas of the test show that some have a greater discriminating capacity than others; those with greater discriminating capacity are: biology (AUC=0.61, 95% CI 0.57 to 0.66); physics/mathematics (AUC=0.63, 95% CI 0.59 to 0.68); and chemistry (AUC=0.65, 95% CI 0.61 to 0.69) (table 4).

Table 4.

Area under receiver operating characteristic curve (AUC) and relative 95% CI for total and per subject area scores

	RS1 AUC (95% CI)	RS2 AUC (95% CI)
Total	0.67 (0.63 to 0.71)	0.70 (0.64 to 0.76)
Biology	0.61 (0.57 to 0.66)	0.62 (0.56 to 0.68)
Chemistry	0.65 (0.61 to 0.69)	0.63 (0.57 to 0.69)
General culture	0.55 (0.50 to 0.59)	0.57 (0.51 to 0.64)
Physics and mathematics	0.63 (0.59 to 0.68)	0.63 (0.57 to 0.69)
Logic	0.54 (0.50 to 0.60)	0.59 (0.53 to 0.65)

Once again, the admission test score appears to be predictive of RS2 achievement (AUC=0.70, 95% CI 0.64 to 0.76) (table 4). Also in this case, the same scientific sub-areas have a greater discriminating capacity than the others: biology (AUC=0.62, 95% CI 0.56 to 0.68), physics/mathematics (AUC=0.63, 95% CI 0.57 to 0.69) and chemistry (AUC=0.63, 95% CI 0.57 to 0.69) (table 4).

Specific analysis of the ROC curve numerical values shows a value between 30 and 35 to be a good cut-off score to use as a selection criterion for student admittance onto the course, in accordance with the institutional one used by the Turin colleges (33.9 in the year 2014–15). In order to understand the reliability of the test, the real classification used at the University of Turin—that is, Regular and TAR students (based on a cut-off score equal to 33.9)—was evaluated in relation to reference standard RS1 and RS2 (table 5).

Table 5.

Predictive capacity of the admission test on the number of CFU acquired

	RS1		RS2
	P1*	N1†	P2‡	N2§
Regular	308	223	101	430
TAR	52	100	10	142
Total	360	323	111	572
Se (95% CI)	0.86 (0.82 to 0.89)		0.91 (0.84 to 0.95)
Sp (95% CI)	0.31 (0.26 to 0.36)		0.25 (0.21 to 0.28)
LH+ (95% CI)	1.23 (1.15 to 1.32)		1.22 (1.14 to 1.29)
LH− (95% CI)	0.46 (0.22 to 0.54)		0.36 (0.23 to 0.56)

*P1, students who acquired half or more of the required credits at the end of the first year.

†N1, students who acquired less than half of the required credits at the end of the first year.

‡P2, students who acquired all the CFU required for the first year.

§N2, students who did not acquire all the CFU in the first year.

CFU, university exam credits; LH−, negative likelihood ratio; LH+, positive likelihood ratio; RS, reference standard; Se, sensitivity; Sp, specificity.

For both RS1 and RS2, sensitivity is high (Se=0.86, 95% CI 0.82 to 0.89 for RS1; Se=0.91, 95% CI 0.84 to 0.95 for RS2), whereas specificity is low (Sp=0.31, 95% CI 0.26 to 0.36 for RS1; Sp=0.25, 95% CI 0.21 to 0.28 for RS2). Analogously, positive likelihood ratios are both higher than 1 (LH+ 1.23, 95% CI 1.15 to 1.32 for RS1; LH+ 1.22, 95% CI 1.14 to 1.25 for RS2), whereas negative likelihood ratios are both less than 1 (LH− 0.46, 95% CI 0.22 to 0.54 for RS1; LH− 0.36, 95% CI 0.23 to 0.56 for RS2).

The high values of sensitivity indicate that the majority of students who achieve the RSs are from the Regular group, while a low specificity indicates that a number of ‘Regular’ students are also among those who did not achieve the two RSs (table 5). This interpretation is also supported by the good value of LH+, which also indicates that being ‘Regular’ is a ‘protective factor’ for achieving both RS1 and RS2.

For our study, however, it is essential to highlight the low value of LH−, which indicates that relatively few TAR students achieved both RSs (in particular RS2). This indicates that a low score in the test also predicts a low probability of achieving both RSs.

Discussion

The admission of a significant number of students to the degree course in medicine at the University of Turin who did not pass the admission test in the year 2014–15 gave us, for the first time, the opportunity to compare the academic results of two student groups in the same context. A close correlation between academic success and test results was found; in particular, students who got a high score in scientific questions were more likely to achieve the requested standards during the first year. Thus, although improvements could still be made to the admission test, it seems to be a good tool for identifying those students who are more likely to perform well during the first year of the course.

Other studies have analysed the performance of students who passed an admission test versus open access students (see, for example, Reibnegger et al ¹), but the comparison could only be made between students of different years. In our case, registration year, lessons, programmes, teachers and classroom characteristics were the same for both Regular and TAR students. For the purposes of our study, it is important to point out that 80% of the TAR students achieved a score of 20 to 31 points in the test (the minimum score for the admission to Turin’s School of Medicine was 33.9). This substantiates the TAR group as a good reference.

The Italian test is comparable to other tests used in international contexts, at least in relation to question type and exam duration (table 1), and for this reason our results may also be of interest outside Italy.

It is important to highlight the similar results found in terms of usefulness and predictability. For example, Sartania et al ³ underlined a clear correlation between ‘total science score’ (our biology, chemistry and physics/mathematic scores) and ‘education performance’ (our RS2, although in Sartania et al ³ a longer observational period was considered).

Most of the previous studies^1–3 confirmed that admission tests are able to predict the academic results in the first year. Nevertheless, no generalised predictability is assessed in these studies, because of the variety of the evaluation periods and reference standards. For example, Sartania et al ³ evaluated the overall career of the students, while Reibnegger et al ¹ and Kelly et al ² and our study considered the results of the first or second years. As far as the reference standards are concerned, Reibnegger et al ¹ investigated the dropout rates (lower in the students passing the admission test), whereas Kelly et al ² were interested in the prediction criteria for clinical and communication skills.

The strengths of this study are the presence of a valid control cohort (TAR) and the possibility of specific analysis per admission test subject area. Although several debates are ongoing in Italy regarding which specific subject areas are most useful for discriminating between potential medical students, our study shows that the results for questions on biology, chemistry and physics/mathematics in the current admission test present the best predictability.

Several limitations of our study should, however, be taken into consideration. First, we only considered students admitted to the course of medicine in Turin and not a wider Italian cohort. Second, this work constitutes an initial explorative analysis, limited to the first year of the medical course.

It is worth pointing out that the admission test is predictive of academic success, but not necessarily the ability to practise as a physician. This aspect has been previously stressed by a number of authors,^20–22 but no definitive conclusion has been reached. In general, however, we can say that a single admission test is unable to predict ability to practise as a physician. Important information could be obtained by following our two cohorts over a longer period of time. Indeed, this study is ongoing in order monitor the two cohorts throughout the complete academic path, re-evaluating their results also with respect to subjects (eg, clinically oriented courses) different from those considered in the admission test.

Conclusions

University admission test scores are able to predict subsequent academic success in the first year of the degree course in medicine; the test is therefore useful for both students and medical schools. Indeed, it discourages students who do not pass the test from enrolling in the course, driving them towards alternative courses, and saving them both time and money. With this selection procedure in place, universities are able to manage a lower number of more motivated students with higher probabilities of obtaining success. This allows a more efficient use of infrastructural and personnel resources. However, the discriminatory capacity of the admission test could be improved by replicating the analysis presented at the end of the fourth and sixth years, investigating the relationship between admission test results and clinical skills.