Analysis of predictors of success in the MRCP (UK) PACES examination in candidates attending a revision course

Abstract

Objective

To determine factors that predict success of candidates taking a revision course in preparation for the MRCP (UK) PACES (practical assessment of clinical examination skills) examination.

Design

A questionnaire survey of candidates attending a PACES revision course. Results were correlated with subsequent pass lists published by the Colleges of Physicians

Setting and subjects

Candidates attending courses in June and October 2002. In total, 523 candidates completed questionnaires, evenly balanced between UK and overseas graduates.

Results

Of 483 candidates who took the examination immediately after the course, 219 (45.3%) passed. UK graduates were more likely to pass (67.0%) than overseas graduates (26.2%) (p = 0.003, odds ratio 5.72). For UK graduates, pass rates were higher for white candidates (73%) than for ethnic minorities (56%) (p = 0.012, OR 2.15) and for those who passed at the first attempt in the MRCP (UK) part 2 written paper (p = 0.003, OR 2.90). For overseas graduates, those who had been qualified for less than eight years were more likely to pass (p = 0.001, OR 2.78). More overseas (45.7%) than UK (30.8%) graduates were confident that they would pass, but confidence did not predict success.

Conclusion

Among candidates taking a revision course, UK graduates are more likely to pass the PACES examination than non‐UK graduates. Ethnic minority UK graduates seem to have a significantly poorer success rate, although this requires confirmation in an independent sample. If confirmed, these differences merit further investigation to assess whether they reflect genuine differences in ability.

The Royal Colleges of Physicians of Edinburgh, Glasgow, and London share a common membership examination, the MRCP (UK). Doctors wishing to complete training in a medical specialty in the UK must pass this two part examination. The first part (MRCP (UK) part 1) comprises a written multiple choice paper, while the second part (MRCP (UK) part 2) consists of both a written and a clinical examination.

The structure of this clinical examination was changed in June 2001 to the PACES (practical assessment of clinical examination skills) format.¹ Candidates are examined at five clinical stations. The stations comprise: (1) respiratory and abdominal (16 points); (2) history taking (8 points); (3) cardiovascular and neurological examination (16 points); (4) communication skills (8 points); and (5) skin, locomotor, endocrine and ophthalmic examination (8 points). The mark required to pass is 42 points.

MRCP (UK) has an international reputation and attracts many candidates who have trained and work outside the UK. It is important that the examination process be free of bias. To date however, the Colleges of Physicians have not published data on the demographic distribution of those candidates, either successful or unsuccessful, who sit the examination. A number of factors may affect the probability of passing this examination. One is the availability and quality of postgraduate medical education. Others such as sex, ethnic origin, and the country of undergraduate medical education, are not amenable to change. This study used a questionnaire to obtain demographic data regarding candidates preparing to sit the PACES examination and to investigate factors that may predict a candidate's success.

Methods

The candidates were all attending a four day revision course (PasTest) during the month before sitting the PACES exam. Courses are filled in chronological order of applications received without any candidate selection, until they reach a specified maximum number.

Candidates (n = 534) attending PasTest clinical courses in preparation for the PACES examination in June and October 2002 were asked to complete a questionnaire (available on line http://www.postgradmedj.com/supplemental). It included questions about the age, sex, ethnic origin, country of first medical qualification, year of qualifying, first language, postgraduate medical training, and previous examination experience. It also sought to assess each candidate's confidence in their chances of passing the PACES examination. Six candidates attended a course in both June and October. Their data from the second course were excluded from further analysis.

Four hundred and eighty three candidates indicated their intention to sit the subsequent PACES examination and completed the questionnaire correctly. Those candidates that were not planning to sit the subsequent PACES examination (n = 38) either planned to sit the PACES examination at a later date or to sit the Irish MRCP examination. Two questionnaires were not completed satisfactorily. Pass lists published in Clinical Medicine, were consulted to determine each candidate's success or otherwise. Each candidate's name was checked against the pass list, ensuring that variations in spelling or in the order of the candidate's names did not lead to them being incorrectly classified as having failed. Pass lists in Clinical Medicine contain the surname, first name, and university of medical graduation. By checking both the first name and surname of each candidate against the surname published in the pass list, and cross checking with the university of graduation, we are confident we were able to ensure correct assignment in 100% of cases. Pass lists for the October 2002 were not published in Clinical Medicine, but the College of Physicians emailed us a full list. Statistical analysis was performed using SPSS (version 7.5). Factors that might predict a candidate's success were analysed by χ² and logistic regression analysis.

Data are presented separately for graduates of United Kingdom and of overseas medical schools because of an a priori hypothesis that the prevalence of factors that might predict examination success would vary between these groups. It was also an a priori hypothesis that ethnicity might affect likelihood of passing among graduates of UK medical schools. In addition, data were collected for those candidates who had previously failed the PACES regarding their marks at each of the five stations.

Ethical approval

Ethical approval was not required. No patients were involved and submission of a completed questionnaire was voluntary. A verbal explanation of the study was announced at the start of each course. One of the authors (RB) explained that full names were required because the questionnaire data would be cross checked with the published pass list at a later date, and that any data published by us would be made anonymous. Eleven candidates declined to complete the questionnaire.

Results

The candidates

Of the 534 candidates attending the relevant clinical courses, 523 (98%) returned questionnaires. Table 1 summarises details of the 483 who completed the questionnaire correctly and sat the PACES examination. Table 2 shows the stations failed by candidates who had taken the PACES before unsuccessfully.

Table 1 Characteristics of PasTest revision candidates sitting PACES examination.

Number of candidates	All candidates	UK graduates	Overseas graduates
Total (%)	483	227 (47.0)	256 (53.0)
June	237	119	118
October	246	108	138
Male (%)	278 (57.6)	92 (40.5)*	186 (72.7)*
Female (%)	205 (42.4)	135 (59.5)*	70 (27.3)*
Passed (%)	219 (45.3)	152 (67.0)*	67 (26.2)*
English first language (%)	262 (54.2)	202 (90.0)*	60 (23.4)*
Previous BSc degree (%)	158 (32.7)	119 (52.4)*	39 (15.2)*
Previous PACES course (%)	206 (42.7)	105 (46.3)	101 (39.4)
Age (years) mean (SD)	30.6 (4.8)	28.1 (2.8)	32.8 (5.2)
Year qualification (SD)	1996 (4.1)	1998 (1.2)	1994 (4.7)
Ethnicity (%)
White	193 (40.0)	148 (65.2)	45 (17.5)
African	33 (6.8)	1 (0.5)	32 (12.5)
Afro‐Caribbean	5 (1.0)	4 (1.8)	1 (0.4)
Asian	161 (33.3)	53 (23.3)	108 (42.2)
Oriental	54 (11.2)	16 (7.0)	38 (14.8)
Other	37 (7.7)	5 (2.2)	32 (12.5)
Previous experience	Months of experience: mean (SD)
UK teaching hospital	6.9 (9.9)	10.7 (10.8)	3.6 (7.7)
UK district hospital	10.2 (11.3)	13.7 (11.3)	7.2 (10.4)
Overseas	18.0 (37.0)	1.1 (3.9)	32.9 (45.8)
Total	35.2 (33.2)	25.5 (9.6)	43.7 (43.0)

*Significant difference between UK and overseas graduates (p<0.05). p Values not shown for differences in ethnic origin or overseas experience, as these were clearly different between UK and overseas candidates.

Table 2 Stations failed by candidates in their previous PACES examinations.

	Chest examination	Abdominal examination	History taking	Cardiovascular examination	Neurological examination	Communication skills	Mixed* examination
All candidates	46	59	44	66	54	39	50
% (n = 122)	37.7	48.4	36.1	54.1	44.3	32.0	41.0
UK graduates	17	24	17	29	22	12	19
% (n = 50)	34.0	48.0	34.0	58.0	44.0	24.0	38.0
Overseas graduates	29	35	27	37	32	27	31
% (n = 72)	40.3	48.6	37.5	51.4	44.4	37.5	43.1

*Mixed examination = combined dermatology, endocrine, locomotor, and ophthalmology examination.

Overall 278 of 483 candidates were male (57.6%) and 205 female (42.4%). There were more female (135) than male (92) candidates among the UK graduates, while the converse was true among overseas graduates. The difference in the sex distribution between UK and overseas graduates was significant (p<0.0001). While there was a trend towards higher pass rates for female UK and overseas graduates than for male graduates, neither achieved significance (70% female compared with 62% male UK graduates, p = 0.196; 34% female compared with 23% male overseas graduates, p = 0.07). Hence, sex alone is unlikely to account for the difference in pass rates between UK and overseas cohorts.

UK graduates were more likely to be under 30 years of age than their overseas counterparts (p<0.0001) who had, on average, graduated from medical school four years earlier.

UK graduates were more likely to speak English as a first language (89%) than overseas graduates (23.4%). They were also more likely to have a BSc or similar further degree (p<0.0001).

In total, 248 candidates (51.5%) had passed the MRCP (UK) part 1 examination at the first attempt and there was no significant difference between UK (55.1%) and overseas (48.0%) graduates in this respect. More UK trained candidates had passed the MRCP (UK) part 2 written examination at the first attempt compared with overseas trained candidates (81.9% and 65.1%, p<0.0001).

Success in passing the PACES examination

Of 483 candidates, 219 (45.3%) passed the subsequent PACES examination. The following factors were analysed, both by univariate (χ² test for categorical variables) and logistic regression analysis, for their ability to predict a pass: age, sex, ethnic origin, use of English as a first language, medical school (UK compared with overseas), possession of a higher degree (BSc or similar), having been on a previous PACES course, having experience of working in UK teaching or district general hospitals and success at the first attempt in previous MRCP (UK) examinations (part 1 and part 2 written). Table 3 shows predictors identified as independently influencing success in the subsequent PACES examination.

Table 3 Logistic regression predictors identified as independently influencing success in the subsequent PACES examination.

Predictor	Wald χ2 value	Degrees of freedom	p Value	Odds ratio of passing	95% Confidence intervals
All candidates (n = 483)
Passed part 2 written paper first time	9.27	1	0.002	3.64	2.31 to 5.73
Graduated from UK medical school	8.67	1	0.003	5.72	3.86 to 4.87
White ethnic origin	5.17	1	0.023	2.04	1.42 to 2.94
UK graduates (n = 227)
Passed part 2 written paper first time	8.59	1	0.003	2.90	1.45 to 5.80
White ethnic origin	6.37	1	0.012	2.15	1.21 to 3.81
Overseas graduates (n = 256)
Number of years qualified	6.06	1	0.014	2.78	1.48 to 5.22

In the group of 483 candidates both univariate and regression analysis showed a significant difference in the likelihood of passing the PACES examination between graduates from UK medical schools (67.0%) and those who had qualified overseas (26.2%) (p = 0.003). The two other factors that predicted success in the PACES examination were being white (compared with ethnic minority groups; p = 0.023) and success at the first attempt in the MRCP (UK) part 2 written paper (p = 0.002).

UK and overseas graduates

As the medical school from which candidates had graduated influenced factors such as ethnicity and use of English as the first language, all variables were analysed again for UK graduates (n = 227) and overseas graduates (n = 256).

Among UK graduates the only factors to increase the probability of passing PACES were being white (73% pass rate compared with candidates from ethnic minority groups, whose pass rate was 56%) (p = 0.012) and success at the first attempt in the MRCP (UK) part 2 written paper (71.5% pass rate compared with candidates who had sat the part 2 written more than once, whose pass rate was 46.3% (p = 0.003). While ethnicity predicted pass rates in the PACES examination, it did not affect the likelihood of passing on the first attempt in the MCQ based part 1 MRCP (UK) (white = 55.4% compared with ethnic minority = 54.4%). Although there was a trend to lower pass rates for ethnic minorities in the part 2 written paper, this was not significant (85.1 compared with 75.9%; p = 0.088).

Among graduates of non‐UK medical schools the only significant factor was date of first medical qualification. In the χ² analysis candidates who had been qualified for less than eight years were more likely to pass (p = 0.001) and this was confirmed in the logistic regression analysis, in which the number of years as a candidate had qualified was treated as a continuous variable (p = 0.014).

Candidate confidence

To establish how confident candidates were the questionnaire also asked them to rate their chances of passing each section (history taking, communication skills, and physical examination) of the PACES examination on a scale of 1 to 4, similar to the scale used by examiners. Those candidates who scored themselves 3 for all three sections were deemed to be confident of passing the PACES examination. A greater proportion of the overseas graduates (45.7%) were confident that they would pass the examination than graduates from UK medical schools (30.8%) (p = 0.001). This means that nearly twice as many overseas candidates thought that they would pass than actually did (pass rate 26.2%), in contrast with UK graduates of whom at least two candidates passed for each one who was confident of passing (pass rate 67.0%).

Interestingly the pass rate among the confident candidates was not significantly higher than among their less confident colleagues. Among UK graduates a slightly higher proportion of confident candidates passed (72.9%) compared with those who were not confident (64.3%), but this difference was not significant (χ²; p = 0.21). Among overseas graduates 26.8% of confident candidates passed compared with 25.7% of those who were not confident (χ²; p = 0.86).

Causes of failure in previous PACES examinations

To determine which part of the PACES examination candidates were most likely to fail we asked them to recall the marks that they had obtained in each part of their previous, failed, PACES examinations (table 2). Candidates were most likely to have failed (scored 2 or less in) the cardiovascular examination station (54%) and least likely to have failed communication skills station (32%).

In these previous PACES examinations, there was a trend towards overseas graduates being more likely to have failed the communication skills station than UK graduates, but this was not significant after correction for the number of variables measured (p = 0.031 uncorrected, p = 0.22 after Bonferroni correction).

Discussion

We have shown that the overall pass rate of unselected doctors taking the PasTest PACES course in the subsequent PACES examination was 45%. There are no figures published by the Colleges of Physicians, but a report for all UK medical colleges suggested an overall pass rate of 44%–79% for UK graduates, and 28%–67% for overseas graduate.² While the Colleges of Physicians have been unable to achieve a response rate to its ethnicity questionnaire of more than 35%,² we were able to achieve a response rate >98%. We found an overall pass rate for UK doctors of 67%, compared with only 26% for overseas graduates. For UK medical graduates, a pass was predicted by success at the first attempt in the MRCP (UK) part 2 written examination, and being white. Interestingly, ethnic minority UK trained doctors had success similar to that of white UK trained doctors in terms of number of attempts at the MCQ based MRCP (UK) part 1 examination, and in the written part of part 2, and it was only in the face to face examination that they were significantly less successful. For overseas trained doctors, being qualified for over eight years reduced the chances of success significantly. Candidates were not good at predicting their own success. Importantly, other factors such as sex, possession of a BSc, and number of attempts at the part 1 examination showed no effect on success in the PACES examination. Working predominantly at a teaching hospital conferred no increase in the chance of passing the examination.

The important limitation to the generalisability of our findings is that the study population was drawn from only one commercially operated course, and was not a random sample from the whole population taking the examination. Recruitment was on a first come first served basis, but it is possible that less able candidates are more likely to attend a revision course, and this might be a source of bias. However, our finding of no difference in success rates at the first attempt during MRCP (UK) part I for UK and overseas trained candidates suggests that the standard of the groups was, at least initially, equivalent.

We found a lower pass rate for ethnic minority than white UK trainees in the PACES examination. The pass rate for the Membership of the Royal College of General Practitioners (MRCGP) was the same for UK graduates whether of white or Asian origin, although Asian doctors trained overseas faired significantly worse.³ For Membership of the Royal College of Psychiatrists (MRCPsych), younger age when taking the examination and having trained at a British or Irish medical school predicted success, in keeping with our findings. However, that study did not attempt to examine the influence of candidate ethnic origin on success rates.⁴ At undergraduate level, one study showed a twofold lower final examination pass rate for UK born ethnic minority students than white students, although this was true for MCQ examinations too, indicating that racial discrimination was unlikely to be the cause.⁵ In the USA, a study of undergraduate performance in objective standardised clinical examinations (OSCE) in obstetrics and gynaecology showed no effect of ethnicity on overall scores or interpersonal skills scores.⁶

Our data need confirmation in an independent sample. Nevertheless, one interpretation of these preliminary data might be that they show evidence of racial discrimination by MRCP (UK) examiners against ethnic minority doctors. In this context, studies showing that ethnic minority students fare less well than white students in entry to UK medical schools,⁷ in final examinations,⁸ in job applications,⁹ in disciplinary action by the General Medical Council,¹⁰ and in the granting of merit awards¹¹,¹² may suggest a background of institutional racism. However, the Colleges provide training in racial and cultural awareness for examiners, and have tried to monitor the ethnic origin of candidates to help them identify any discriminatory practice. Moreover, the PACES format, with 10 different examiners assessing each candidate independently, is structurally much less open to bias by a small number of examiners than the format it replaced. Racism is an easy accusation to make yet difficult to disprove, and there may be other reasons for differences in pass rates.

What other explanations could there be for a reduced pass rate by ethnic minority doctors of UK origin? One perspective comes from a study of the effect of ethnicity on performance in a final OSCE at the University of London.¹³ Ethnic minority students scored lower than white students in communication skills, although no overt discrimination was found in video recordings. Male ethnic minority students displayed various moves to distance themselves from patients, and were given low grades by both examiners and simulated patients. In some instances, white examiners had a “textbook notion” of a patient centred consultation that did not accord with ethnic minority patients' views. There are “institutional norms” regarding what counts as good consultation that change over time. The authors hypothesised that students from ethnic minorities may be more likely to live with their family and so be less exposed to the social talk around medicine that occurs in institutional life. They would thus be disadvantaged, but not through any direct influence of racism.

The study by Roberts et al at postgraduate level is also relevant.¹⁴ This investigated the MRCGP oral examination for its fairness, using linguistic analysis. The study elucidated three types of talk used by examiners during oral examinations, known as personal experience discourse, professional discourse, and institutional discourse. They noted that examiners and candidates would shift from one discourse frame to another, and that such subtle changes were more difficult to follow by candidates from ethnic minority and non‐British backgrounds. They recommended that examiners have training about their role in producing hybrid discourses in the oral examination. They also recommended that examination boards develop and publish examples of oral questions, together with examples of different candidates' answers and examiner's comments on these. Such action by the Colleges of Physicians may help to reduce the gap in success rates noted in this paper.

The questionnaire used in this study is available on line (http://www.postgradmedj/supplemental).