Annual Review of Competence Progression (ARCP) performance of doctors who passed Professional and Linguistic Assessments Board (PLAB) tests compared with UK medical graduates: national data linkage study

Abstract

Objective To determine whether use of the Professional and Linguistic Assessments Board (PLAB) examination system used to grant registration for international medical graduates results in equivalent postgraduate medical performance, as evaluated at Annual Review of Competence Progression (ARCP), between UK based doctors who qualified overseas and those who obtained their primary medical qualification from UK universities.

Design Observational study linking ARCP outcome data from the UK deaneries with PLAB test performance and demographic data held by the UK General Medical Council (GMC).

Setting Doctors in postgraduate training for a medical specialty or general practice in the UK and doctors obtaining GMC registration via the PLAB system.

Participants 53 436 UK based trainee doctors with at least one competency related ARCP outcome reported during the study period, of whom 42 017 were UK medical graduates and 11 419 were international medical graduates who were registered following a pass from the PLAB route.

Main outcome measure Probability of obtaining a poorer versus a more satisfactory category of outcome at ARCP following successful registration as a doctor in the UK.

Results International medical graduates were more likely to obtain a less satisfactory outcome at ARCP compared with UK graduates. This finding persisted even after adjustment for the potential influence of sex, age, years of UK based practice, and ethnicity and exclusion of outcomes associated with postgraduate examination failure (odds ratio 1.63, 95% confidence interval 1.30 to 2.06). However, international medical graduates who scored in the highest twelfth at part 1 of the PLAB (at least 32 points above the pass mark) had ARCP outcomes that did not differ significantly from those of UK graduates.

Conclusions These findings suggest that the PLAB test used for registration of international medical graduates is not generally equivalent to the requirements for UK graduates. The differences in postgraduate performance, as captured at ARCP, following the two routes to registration might be levelled out by raising the standards of English language competency required as well as the pass marks for the two parts of the PLAB test. An alternative might be to introduce a different testing system.

Introduction

The healthcare workforce is becoming increasingly international and globalised.¹ In the United Kingdom in 2012, 37% of the doctors registered with the national medical regulator, the General Medical Council (GMC), had qualified in other countries; 27% had obtained their medical degree from outside the European Economic Area (EEA).² Before their names are placed on the GMC’s List of Registered Medical Practitioners, allowing the doctor to legally practise medicine in the UK, all doctors must fulfil the criteria laid out in the 1983 Medical Act.³ For international medical graduates, several routes to such registration exist (box 1).

Box 1 Requirements for GMC registration for international medical graduates

• Most international medical graduates register with the GMC via the Professional and Linguistic Assessments Board (PLAB) test

• To sit the PLAB, candidates must hold a medical degree from a recognised institution,⁵ provide evidence of competency in English, and have undergone at least 12 months of postgraduate clinical experience

• If an approved UK health provider offers sponsorship to an overseas doctor with at least three years’ postgraduate training the GMC can issue a certificate of sponsorship,⁶ allowing the doctor to practise for a limited time

• International medical graduates may apply for a Certificate of Eligibility for Specialist Registration (CESR),⁷ which would allow them to practise in the UK at consultant level. This involves submitting a portfolio of evidence of competence to the relevant medical royal college for evaluation

• Similarly, overseas doctors can apply for a Certificate of Eligibility for General Practice Registration (CEGPR),⁸ allowing them to work as general practitioners in the UK. This involves providing a portfolio of evidence to the Royal College of General Practitioners for evaluation, as well as obtaining two structured referees’ reports

To take the Professional and Linguistic Assessments Board (PLAB) test (box 2), international medical graduates must provide evidence of language competency in English. This usually means having passed an International English Language Test System (IELTS) assessment with a score of band seven (good user) or above, before applying for PLAB part 1. The IELTS test is in four parts—listening, speaking, reading, and writing—and can be taken as many times as desired.⁴ Each part is graded between band one (non-user) and band nine (expert user).

Box 2 Professional and Linguistic Assessments Board (PLAB) test

• The PLAB test is in two parts, which must be passed sequentially and may be retaken as many times as desired

Part 1

• Part 1 of the PLAB test assesses the medical knowledge of candidates as relevant to the UK. It is a three hour examination with 200 multiple choice questions for which the candidate has to select the single best answer. The test covers the following areas: applying knowledge and experience to practise, clinical care, assessment, and clinical management

• The pass mark is set by a variant of the Angoff method, whereby experts decide the minimum scores that would be acceptable for the test items¹¹

Part 2

• Part 2 of the test must be passed within three years of passing part 1

• It consists of 14 objective structured clinical examination (OSCE) stations. Each station consists of a five minute clinical scenario in which candidates are observed by a single examiner and scored on their use of skills and knowledge

• The skills assessed are clinical examination, practical skills, communication skills, and history taking

• The pass mark for part 2 is set via the borderline group scoring method.¹² This involves weighting the scores for the OSCE stations. Weightings are decided according to expert opinion of the score expected of a “minimally competent” candidate and are informed by scores obtained by previous cohorts

The PLAB test is designed to ensure that overseas doctors demonstrate clinical competencies that would be equivalent to those that a UK graduate would be expected to have achieved by the end of foundation year one. This is a basic initial training year immediately following medical graduation, which consists of clinical placements focused on medicine and surgery, although attachments in other specialties such as psychiatry and emergency medicine are increasingly being offered. Once foundation year one has been completed, foundation year two posts can be applied for; these continue basic clinical training, although with an expectation of additional autonomy and responsibilities. Thus, a successful PLAB graduate is able to apply for foundation year two posts, in competition with both UK and EEA medical graduates.

All doctors in UK postgraduate training are assessed regularly. Trainees who started before 1 August 2007 are covered by the Record of In-Training Assessment (RITA).⁹ Trainees who started on or after 1 August 2007 are assessed by an Annual Review of Competence Progression (ARCP).¹⁰

An ARCP (or previously RITA) panel is composed of at least three senior doctors working in the relevant specialty. One member must be a postgraduate dean or training programme director, and at least two must be drawn from a list that includes a royal college representative, an educational supervisor, and other holders of roles related to postgraduate medical training. There should also be a representative of the doctor’s employer.¹⁰ The assessment is generally conducted annually but may be carried out more frequently if there are concerns or other circumstances, such as insufficient evidence presented, that require this. The panel considers evidence, presented by the trainee and his or her trainer at that time, of the trainee’s progression. The panel then decides whether the trainee is competent in the required skills and has the necessary knowledge to progress to the next level of training. Such evidence may include passing a postgraduate examination required for membership of the relevant royal college. The royal colleges set the curriculum and the competencies that would be expected to be achieved at each level of training in their specialty. The timing of these examinations varies across specialties, but in most instances one would be expected to undergo a defined period of approved UK postgraduate training before being permitted to sit an examination. There are some exceptions—for instance, one can sit the Membership of the Royal College of Physicians (MRCP) examination, part 1, after 12 months of non-UK medical practice. Failure to pass royal college examinations can sometimes, but not invariably, impede progress in training, depending on the specialty and examination. For example, to enter into higher specialist training in psychiatry (the last three years of postgraduate training leading to eligibility for employment at consultant level), one must have passed all three parts of the relevant membership examination. At ARCP, the trainee would present a portfolio of structured observed tasks and procedures that have been assessed by the trainer and other practitioners involved in the doctor’s training.

Table 1 outlines the possible outcomes at ARCP and RITA.¹³ If a doctor has not developed a specific competency, but is generally deemed to be progressing adequately, the panel may recommend that some targeted training is provided without an extension to the training period. If more serious deficiencies in competencies are perceived, the panel may recommend that the usual training period is extended. Following this, if satisfactory progress is still not made, the doctor may have to leave the training programme. “Out of programme” experience does not include periods of maternity leave.

Table 1.

 List of possible outcomes awarded to medical trainees at Annual Review of Competence Progression and Record of In-Training Assessment

Outcome	Description
Annual Review of Competence Progression
1	Satisfactory progress; competences achieved as expected
2	May progress but requires specific/targeted training to achieve certain competences
3	Has not achieved competences required to progress; up to 12 months’ (6 months in Core and GP) additional training required
4	Released from training programme with or without specified competences
5	Incomplete evidence provided
6	Recommendation for completion of training having gained all required competences
7	Fixed term specialty outcome—specific competences achieved
7.1	Satisfactory progress in or completion of Locum Appointment for Training (LAT)/Fixed Term Speciality Training Appointments (FTSTA) placement
7.2	Development of specific competences required; additional training time not required—LAT/FTSTA placement
7.3	Inadequate progress by trainee; additional training time required—LAT/FTSTA placement
7.4	Incomplete evidence presented—LAT/FTSTA placement
8	Out of programme experience; time may or may not count towards training
Record of In-Training Assessment
C	Satisfactory progress; competences achieved as expected
D	May progress but requires specific/targeted training to achieve certain competences
E	Has not achieved competences required to progress; up to 12 months’ (6 months in Core and GP) additional training required
F	Out of programme experience; time may or may not count towards training—usually refers to period of research or specialist training undertaken, either in UK or abroad
G	Recommendation for completion of training having gained all required competencies

Recently, there has been a significant focus on the clinical performance of overseas doctors. A study reported that, compared with white UK graduates, non-white international medical graduates were almost 15 times more likely to fail the clinical skills assessment component of the Membership of the Royal College of General Practitioners (MRCGP) examination, even after adjustment for sex, age, and performance on the applied knowledge test of the assessment.¹⁴ In addition to concerns relating to postgraduate examination performance, evidence suggests that, compared with UK graduates, international medical graduates are more likely to be referred or censured for concerns relating to fitness to practise, even after control for the potential influence of enquiry characteristics (source, type, and content), sex, time since qualification, and specialty (general practice versus non-general practice).¹⁵

The GMC commissioned the analysis reported here as part of a review of the PLAB system, and it builds on a recent review of the test.^{16 17} A key focus was “to examine whether international medical graduates granted full registration following . . . the PLAB test are more or less likely than other . . . doctors to experience difficulties in medical practice.” This study was intended to complement a separate strand of research examining differential pass rates at the MRCP and MRCGP examination.¹⁸ However, by using ARCP rating as the outcome of interest, we were able to evaluate the performance of doctors in postgraduate training across all medical specialties, not just medicine and general practice for which the MRCP and MRCGP examinations are respectively taken. Moreover, assessment at ARCP takes into account a wider range of factors than postgraduate examination passes. We anticipated that the results would shed light on the magnitude of any existing differences in performance and elicit some of the potential mediating and moderating influences for any disparities observed. We hoped that the findings would also provide indications of whether the PLAB test could be modified to increase its effectiveness in encouraging equivalence between international medical graduates and UK graduates.

Methods

Our aims were to compare ARCP outcomes between international medical graduates who had passed the PLAB and UK graduates, both before and after controlling for the effects of potential confounding factors, such as age, sex, and duration of UK based experience. The technical appendix gives a more detailed description of the data collection and classification, model building, missing data patterns, and data imputation, along with a number of supplementary results.

Data sources

Data on ARCP outcomes from the UK deaneries (administrative regions responsible for postgraduate medical training at the time^{19 20}) for 2010-12 came from the GMC. We converted outcomes relating to the RITA into the equivalent ARCP outcomes where applicable. In total, 125 208 ARCP outcomes relating to educational progress and performance were available for 53 436 doctors who were either UK graduates or international medical graduates who had passed the PLAB test. Figure 1 shows the flow of study data. The IELTS scores (graded 7.0 to 9.0) were available for 25 896 of international medical PLAB graduates. PLAB test data were available for all 27 726 international medical graduate candidates who passed the examination from July 2000 to the end of 2010. As pass marks vary between diets (sittings), we converted the raw PLAB scores to the scores relative to the pass mark for that sitting. Multiple resits of postgraduate medical examinations may be associated with disproportionately poor performance.²¹ Therefore, we explored the potential association between the number of PLAB sittings and later ARCP performance.

Fig 1 Flow of data through study

The research team had access to a dichotomised version of ethnicity (white/non-white) obtained via the GMC National Training Survey. Additional demographic data such as sex, world region where the primary medical qualification was obtained, date of birth, and date of first registration (or provisional registration if present) came from the List of Registered Medical Practitioners.²²

Model building

For the purposes of our analyses, we used random effects ordinal logistic regression to estimate the odds of receiving a less satisfactory (rather than a more satisfactory) ARCP outcome depending on the world region of qualification (international medical graduate versus UK graduate). For international medical graduates, we were also able to assess how IELTS and PLAB scores were associated with ARCP outcomes. We also conducted analyses controlling for the effects of several potential confounding variables such as age, sex, IELTS, years of UK based experience, and ethnicity. Individual doctors had each undertaken varying numbers of ARCPs during the study period. Therefore, we used a multilevel model whereby ARCP outcomes were conceptualised as nested within individual doctors.

We conducted “simulation” of a change in pass mark by dividing PLAB graduates into twelfths on the basis of their examination performances at pass. We used twelfths rather than tenths to provide a slightly finer grained picture. We then compared these subgroups of PLAB graduates with UK graduates.

Results

Statistical significance should be assumed to be at the P<0.001 level, unless otherwise stated.

Descriptive statistics

Table 2 shows the demographic characteristics of the doctors and degree of missing data. Compared with the UK graduates, international medical graduates were older and more likely to be male, to have had less time practising in the UK, to be of non-white ethnicity, and to have undergone more ARCPs in the study period. This could have indicated either longer training or a tendency to have more frequent assessments. As can be seen in table 2, few data were missing with the exception of ethnicity, which was missing in just under a quarter of cases. The sample was drawn from all medical specialties. We grouped these into 12 specialties largely on the basis of the UK royal colleges associated with the training scheme.²³

Table 2.

 Demographic characteristics of 53 436 doctors with Annual Review of Competence Progression (ARCP) outcomes available. Values are numbers (percentages) unless stated otherwise

Characteristic	UK graduates (n=42 017)	International medical graduates (n=11 419)	Proportion missing
Male sex (%)	18 666 (44)	6877 (60)	0
Mean (SD) age, years	31.61 (4.1)	35.89 (4.1)	7/53 436 (0.000)
Mean (SD) years of UK experience	6.69 (3.5)	6.12 (1.9)	0
Non-white ethnicity	9381/32 458 (29)	8387/8894 (94)	12 084/53 436 (23)
Median (interquartile range) No of ARCPs taken	2 (1-3)	2 (2-3)	0

All intergroup differences are significant at P<0.01 level. P values for intergroup testing were derived from χ² test in case of sex, ethnicity, and number of ARCPs taken. Kruskall-Wallis test was used to compare mean age and years of UK based experience at ARCP between groups.

ARCP outcomes

Figure 2 depicts the proportion of the five main categories of ARCP outcome in the two groups of doctors. In total, 11 419 international medical graduates who registered via the PLAB test underwent 28 189 ARCPs (excluding “out of programme experience”). We had data on 42 017 UK graduates undergoing 97 019 ARCPs. As can be seen, international medical graduates are underrepresented compared with UK graduates in the categories representing adequate progression (that is, ARCP outcomes 1 and 6 and RITA C and G) and overrepresented in the other, less satisfactory, categories (for example, ARCP outcome 3: “extended training required”). Figure 3 shows the ARCP outcomes when those associated with reported examination failure are excluded. As expected, the proportion of less satisfactory ARCP outcomes falls for both groups, although intergroup differences remain. We also observed that UK graduates were significantly more likely than international medical graduates to report having “out of programme experience.” Of 54 122 doctors with any ARCP results recorded, only 530 (5%) of the 11 418 international medical PLAB graduates were recorded as having this as an ARCP outcome category. This contrasts with UK graduates, of whom 3582 (9%) of 42 017 doctors had this outcome recorded at least once.

Fig 2  Annual Review of Competence Progression (ARCP) outcomes (n=125 208) for doctors in sample (n=53 436). International medical graduates registered via Professional and Linguistic Assessments Board examination (PLAB IMG) and UK graduates (UKG) shown separately

Fig 3  Annual Review of Competence Progression (ARCP) outcomes in sample (n=119 367) not associated with postgraduate examination failure in sample of doctors (n=52 878). International medical graduates registered via Professional and Linguistic Assessments Board examination (PLAB IMG) and UK graduates (UKG) shown separately

Univariable analysis

For the purposes of the analyses, we treated ARCP outcome as an ordinal (ordered categorical) variable and coded it as follows: 1=satisfactory progression; 2=insufficient evidence presented; 3=targeted training required (but training time not extended); 4=extended training time required/left programme.

To estimate the unadjusted (raw) odds of obtaining a less satisfactory ARCP outcome according to the predictor variables (world region of qualification, IELTS and PLAB scores) we conducted a series of ordinal logistic regressions. International medical graduates who registered via the PLAB had more than twice the odds of obtaining a less satisfactory ARCP outcome compared with UK graduates (odds ratio 2.46, 95% confidence interval 2.35 to 2.58). Increasing age (odds ratio 1.04, 1.03 to 1.04), male sex (1.42, 1.37 to 1.48), and non-white ethnicity (1.98, 1.88 to 2.08) were also associated with significantly increased odds of obtaining a less satisfactory outcome at ARCP. In contrast, UK based experience increased the odds of a more satisfactory ARCP rating by 10% per year (odds ratio 1.10, 1.09 to 1.11). (As in logistic regression, obtaining the odds ratios for the reverse outcome (in this case a more satisfactory ARCP outcome) is achieved by dividing 1 by the original odds ratios.)

Table 3 depicts the proportion of international medical graduates in the sample, according to medical specialty, along with the proportion of ARCPs that were reported as being satisfactory (that is, ARCP outcome 1 or 6, or the RITA equivalent). We also obtained unadjusted odds ratios for each medical specialty group, both with and without inclusion of ARCP outcomes associated with postgraduate examination failure. Figure 4 shows the results, which suggest that in most specialties international medical graduates have a higher odds of obtaining less satisfactory ARCP outcomes compared with UK graduates. Perhaps the two most striking features in figure 4 are the very high odds ratios of being in a less satisfactory ARCP category if you are an international medical graduate in general practice or psychiatry and the reduction in these disparities for these two specialities once ARCP outcomes related to examination failure are excluded.

Table 3.

 Proportion of international medical graduates registering via PLAB test in each speciality group along with proportions of Annual Reviews of Competence Progression (ARCP) in which outcome indicated satisfactory progress. Values are numbers (percentages)

Specialty	Proportion of international medical graduates	Proportion satisfactory ARCP outcomes
Anaesthetics and intensive care	708/5371 (13.2)	10 131/12 680 (79.9)
Emergency medicine	258/1723 (15.0)	2482 /3422 (72.5)
General practice	3327/14 564 (22.8)	23 287/28 997 (80.3)
Medicine	2663/13 907 (19.2)	26 894/37 292 (72.1)
Obstetrics and gynaecology	861/2456 (35.1)	4232/6008 (70.4)
Ophthalmology	88/758 (11.6)	1443/1819 (79.3)
Paediatrics	1128/4146 (27.2)	7444 /10 210 (72.9)
Pathology and clinical sciences	237/1058 (22.4)	1890/2385 (79.3)
Psychiatry	1738/3886 (44.7)	6128/8535 (71.8)
Public health	24/288 (8.3)	529/605 (87.4)
Radiology	264/1412 (18.7)	2766/3220 (85.9)
Surgery	885/7402 (12.0)	13 653/17 613 (77.5)

Fig 4 Results from random effects ordinal regression models for obtaining less satisfactory Annual Review of Competence Progression (ARCP) outcome as international medical Professional and Linguistic Assessments Board medical graduate (versus UK graduate) by medical specialty. Estimates depicted in grey are from analyses in which ARCP outcomes associated with royal medical college membership examination failure were excluded

On univariate analysis, both IELTS and PLAB scores (relative to pass mark at first sitting) were significantly predictive of more satisfactory ARCP outcomes. For IELTS, each point above 7.0 scored by the doctor being assessed increased the odds of a more satisfactory ARCP outcome by 69% (odds ratio 1.69, 1.55 to 1.85). For PLAB part 1, each point gained above the pass mark increased the odds of being in a more satisfactory category by 2% (odds ratio 1.02, 1.02 to 1.02). For PLAB part 2, this value was approximately 8% (odds ratio 1.08, 1.08 to 1.09).

We also “simulated” the effect of raising the entry level mark for both IELTS (the main route to PLAB) and the pass scores for the PLAB. Figure 5 depicts the unadjusted odds ratios for obtaining a less satisfactory ARCP outcome according to world region of qualification. As seen in figure 5, international medical graduates scoring the minimum 7.0 on IELTS required for PLAB examination entry are more likely to receive a less satisfactory ARCP outcome than are UK graduates. However, the disparity between international medical graduates and UK graduates decreases with increase in IELTS scores; PLAB graduates with IELTS scores of 9.0 have only a 50% higher odds of receiving a less satisfactory ARCP outcome compared with UK graduates.

Fig 5 Unadjusted (raw) odds ratios for risk that Annual Review of Competence Progression (ARCP) outcome falls into less satisfactory category (international medical graduate registered via Professional and Linguistic Assessments Board versus UK graduate) for different simulated International English Language Test System (IELTS) “pass” score thresholds. Odds ratios and associated standard errors were derived from a series of multilevel ordinal logistic regression models

Figures 6 and 7 show that as the PLAB scores (at pass) for international medical graduates increase the discrepancy in performance at ARCP between this group and UK graduates decreases. This difference reaches statistically non-significant levels only for a PLAB part 1 score that would place a candidate in the top twelfth of PLAB graduates. This is equivalent to around 32 points above the current pass mark (that is, 1.8 standard deviations above the mean for the present sample of PLAB candidates). According to figure 7, no realistic pass mark could be set for PLAB part 2 that would ensure overall equivalent ARCP performance between international medical graduates and UK graduates.

Fig 6 Unadjusted (raw) odds ratios for risk that Annual Review of Competence Progression (ARCP) outcome falls into less satisfactory category (international medical graduate registered via Professional and Linguistic Assessments Board (PLAB) versus UK graduate) for different simulated PLAB part 1 pass marks. Odds ratios and associated standard errors were derived from a series of multilevel ordinal logistic regression models. Twelfths are used to rank international medical graduates’ performances on part 1 of the examination, with those in the first group having a score at least 32 points above pass mark and those in the 12th group being in the lowest scoring twelfth (score of at least 0 above pass) of PLAB graduates

Fig 7 Unadjusted (raw) odds ratios for risk that Annual Review of Competence Progression (ARCP) outcome falls into less satisfactory category (international medical graduate registered via Professional and Linguistic Assessments Board (PLAB) versus UK graduate) for different simulated PLAB part 2 pass marks. Odds ratios and associated standard errors were derived from a series of multilevel ordinal logistic regression models. Twelfths are used to rank international medical graduates’ performances on part 2 of the examination, with those in the first group having a score of at least 12.1 points above pass mark and those in the 12th group being in lowest scoring twelfth of PLAB graduates

Multivariable analysis

To explore the extent to which PLAB status and performance were independently predictive of ARCP outcome, we developed a series of ordinal logistic regression models controlling for the potential effects of demographic variables. Missing values for ethnicity were multiply imputed.²⁴ For these analyses, we excluded ARCP outcomes associated with examination failure. By doing so, we hoped to elicit some of the mediators between region of qualification and postgraduate performance beyond those related to examination pass rates.

As can be seen in table 4, even after we controlled for the effects of sex, mean age at ARCP, and years of UK based experience, and allowing for significant interactions, international medical graduates are more likely to receive a less satisfactory ARCP outcome than are UK graduates. Age interacted with UK based experience, indicating that in older doctors UK based experience was less strongly predictive of a more satisfactory ARCP outcome. A significant interaction also existed between PLAB graduate status and ethnicity, which suggested that being of non-white ethnicity was not as predictive of a less satisfactory ARCP outcome in international medical graduates, compared with UK graduates. Lastly, an interaction existed between age and ethnicity, which suggested that in older doctors the association between non-white ethnicity and obtaining a less satisfactory ARCP outcome was accentuated.

Table 4.

 Random effects ordinal logistic regression model for obtaining less satisfactory Annual Review of Competence Progression (ARCP) outcome for international medical graduates (IMG) versus UK graduates, adjusted for sex, age, years of UK based experience, and ethnicity*

Predictor	Odds ratio (95% CI)
Male sex	1.29 (1.23 to 1.34)
Age	1.00† (0.98 to 1.01)
UK experience	0.71 (0.68 to 0.74)
Non-white ethnicity	0.72‡ (0.49 to 1.06)
Age/UK experience interaction	1.01 (1.00 to 1.01)
IMG status/non-white ethnicity interaction	0.69 (0.54 to 0.89)
Age/non-white ethnicity interaction	1.02 (1.01 to 1.03 )
IMG status	1.63 (1.30 to 2.06)

*Data from 119 352 ARCPs undergone by 52 871 doctors were used, with outcomes associated with examination failure excluded. All odds ratios are significant at P<0.01 level unless otherwise indicated.

†P=0.39.

‡P=0.1.

PLAB performance

PLAB scores at both parts of the examination (at first attempt) significantly predicted ARCP performance, even after we controlled for the potential effects of demographic factors and IELTS scores. Ethnicity was not included as a covariate in these models, as it was not a significant predictor of ARCP outcome in international medical graduates according to the results of a univariate analysis. Table 5 shows the results. We noted one significant interaction between the score obtained at PLAB part 1 and years of UK based experience. This suggests that candidates with higher scores at PLAB part 1 and who gained more UK based clinical experience had a particular advantage when later undergoing an ARCP. As can be seen in table 5, PLAB part 1 score had no effect on ARCP outcome except through the interaction with UK based experience. We may therefore speculate that the benefits of obtaining a higher PLAB part 1 score, which is knowledge based, may generally only be realised at ARCP in the presence of appropriate clinical exposure and training.

Table 5.

 Random effects ordinal logistic regression model for obtaining less satisfactory Annual Review of Competence Progression (ARCP) outcome (n=24 474) as predicted by Professional and Linguistic Assessments Board (PLAB) examination scores at first attempt in international medical graduates (n=10 678)*

Predictor	Odds ratio (95% CI)
PLAB part 1
Male sex	1.24 (1.14 to 1.36)
Age	1.04 (1.03 to 1.05 )
UK experience	0.91 (0.88 to 0.93)
Overall IELTS score	0.77 (0.70 to 0.84)
PLAB part 1/experience interaction	0.99 (0.99 to 0.99)
PLAB part 1 score	1.00† (0.99 to 1.01)
PLAB part 2
Male sex	1.16 (1.07 to 1.27)
Age	1.04 (1.03 to 1.05)
UK experience	0.89 (0.87 to 0.92)
Overall IELTS score	0.80 (0.72 to 0.87)
PLAB part 2 score	0.95 (0.95 to 0.96)

IELTS=International English Language Test System.

*ARCP outcomes associated with examination failure were excluded. All odds ratios are significant at P<0.01 level unless otherwise indicated.

†P=0.69.

We also explored the effect of re-sitting the PLAB on future ARCP performance. As can be seen in table 6, compared with passing the PLAB parts after multiple attempts, passing at first sittings was independently and significantly predictive of a more satisfactory outcome at ARCP. For PLAB part 1, this effect seems to level off at three attempts (that is, two resits), whereas for part 2 these effects level off at two attempts (one resit).

Table 6.

 Random effects ordinal logistic regression model estimating effect of number of attempts at Professional and Linguistic Assessments Board (PLAB)examination, parts 1 and 2, on obtaining less satisfactory Annual Review of Competence Progression (ARCP) outcomes (n=24 474) in international medical graduates (n=10 678)*

Predictor	Odds ratio (95% CI)
Part 1 of PLAB test
Male sex	1.24 (1.14 to 1.35)
Age	1.04 (1.03 to 1.05)
UK experience	0.89 (0.87 to .92)
Overall IELTS score	0.73 (0.67 to 0.80)
Two attempts at PLAB part 1	1.30 (1.16 to 1.45)
Three attempts at PLAB part 1	1.69 (1.41 to 2.02)
Four or more attempts at PLAB part 1	1.60 (1.29 to 1.98)
Part 2 of PLAB test
Male sex	1.21 (1.11 to 1.32)
Age	1.04 (1.03 to 1.05)
UK experience	0.89 (0.86 to 0.91)
Overall IELTS score	0.73 (0.67 to 0.80)
Two attempts at PLAB part 2	1.34 (1.20 to 1.50)
Three or more attempts at PLAB part 2	1.49† (1.14 to 1.94)

IELTS=International English Language Test System.

*Passing the exam at first sitting was used as base (comparator) category. ARCP outcomes associated with exam failure were excluded. All odds ratios are significant at P<0.001 level unless otherwise indicated.

†P=0.003.

Discussion

This is the first study to report on overall postgraduate performance of UK doctors in training, as evaluated at ARCP. We observed considerable differences in ARCP outcomes between UK graduates and international medical graduates (who registered via the PLAB system). Some, but certainly not all, of the differences in ARCP performance may be explained by disproportionately high rates of membership examination failure, especially in psychiatry and general practice. The observed intergroup differences also seem to be partly mediated by sex, age, and years in UK practice.

In terms of evidence relating to the validity of the PLAB examination, raising the pass mark for both parts of the test may reduce the disparity in performance observed between international medical graduates registering via the test and UK graduates. We also observed that linguistic performance is also strongly predictive of later ARCP performance. Moreover, superior performance on both parts of the PLAB test is independently and significantly predictive of better ARCP outcomes. These benefits seem to be partly mediated by interactions with other factors, such as length of exposure to UK practice. Time spent working in the UK is likely to help international medical graduates to become more confident in relation to the cultural aspects of healthcare, as well as enhancing clinical competencies.

Comparison with other studies and possible interpretations

Our findings are highly concordant with those of the accompanying report, studying the performance at MRCGP and MRCP of international medical graduates and UK graduates,¹⁸ in that only the highest scoring PLAB candidates approached or reached equivalent levels of performance to UK graduates. This suggests that the problem of non-equivalence is present across most (although not all) medical specialties and is not fully explained by intergroup differences in postgraduate examination performance in medical specialties or general practice. These related studies provide strong evidence that the PLAB examination is, in its current form, unable to ensure equivalence between these two groups of doctors.

After adjustment for rates of membership examination failure, the effect of being an international medical graduate changed most in general practice and psychiatry but relatively little in other specialty groups. Both these specialties have postgraduate examinations that have previously been shown to have had relatively high failure rates in international medical graduates compared with UK graduates.^{14 25} The possible causes of these differential pass rates have been intensively debated over the past year or so, and these discussions are likely to continue in a legal arena. Both specialties contain above average proportions of international medical graduates (psychiatry is the specialty group with the highest proportion). This will allow any intergroup distinctions present to surface, as any analysis will be adequately powered to show such differences with a high level of confidence. Secondly, both specialties place a high degree of emphasis on communication. This is likely to be true not just in the pure linguistic sense but also in a cultural and subcultural context. For example, in psychiatry, a certain amount of judgment must be exercised to differentiate culturally (or subculturally) normal from pathological ideation. This situation, which may require more advanced communication skills, is likely to place international medical graduates at an overall disadvantage. Thirdly, both general practice and psychiatry are “shortage specialties,” in which recruitment and retention of medical staff is recognised as challenging: a recent report highlighted that there are, on average, just 1.8 and 1.3 applicants to core training jobs in psychiatry and general practice, respectively, compared with 9.6 in the most competitive speciality of neurosurgery.²⁶ In practice, this may mean that PLAB candidates who passed with lower scores may be unable to secure a post in their preferred specialism but successfully apply for psychiatric or general practice training. This is likely to lead to higher rates of examination failure and less satisfactory outcomes at ARCP.

As suggested by Esmail and Roberts,¹⁴ racial bias in such examinations cannot be ruled out entirely, especially given that non-white UK graduates were noted to have performed more poorly on the clinical skills assessment component, even allowing for achievement on the knowledge based portion of the examination. This observation is consistent with the findings of an earlier systematic review of ethnicity and medical academic performance.²⁷ However, the evidence for candidate-examiner interactions relating to ethnic status in such examinations is fairly weak.^{28 29} It is likely that more complex reasons underlie these disparities, including the “streaming” of more able candidates (both UK graduates and international medical graduates) into better deaneries and training posts, as well as more subtle interactions between cultural factors and examination ratings in observed clinical examinations and exercises, the latter of which contribute to ARCP outcomes. Our observation that international medical graduates were significantly less likely than UK graduates to have “out of programme experience” would be consistent with differences in the quality of training. Such a difference may reflect relatively reduced opportunities for involvement of international medical graduates in clinical research or their overrepresentation in specialties in which such training opportunities are less freely available.

Strengths and potential limitations

This analysis used a national sample that, in theory, included all doctors who had passed both parts of the PLAB during the timeframe of the study. The international medical graduates who passed the PLAB after 2010 but underwent ARCPs from 2010 to 2012 would not have been included in this analysis. This is likely to have been a relatively small proportion of the international medical graduates who registered via the PLAB test and who underwent ARCPs during the study timeframe and is unlikely to have significantly affected the validity of our findings. Moreover, data on demographic factors such as sex, age, and years since UK registration were relatively complete. Data on ethnicity were less complete, although this was addressed through multiple imputation. The use of multiple imputation permitted the inclusion of observations in the models, including ethnicity, that would otherwise have been excluded due to missing data. This increases study power and potentially reduces the risk of bias while preserving the standard errors for the model estimates.³⁰ As ethnicity could be predicted from the other variables in the complete data, we were able to use a plausible model to impute the missing values. However, multiple imputation is based on the “missing at random” assumption (that is, that the absent values are likewise related to the observed variables). Although the findings of our sensitivity analysis tend to suggest that ethnicity was missing at random (see technical appendix), we had no way of robustly testing this assumption. This represents the primary limitation of multiple imputation in such circumstances.³¹ Moreover, little doubt exists that the analysis would have been more informative if a greater degree of detail were available for both reported ethnicity and specific country of qualification. In this case, the dichotomisation of ethnicity into non-white/white, to preserve anonymity,³² meant that few conclusions could be drawn regarding the potential effect of this variable on ARCP performance; generally, this factor served as a proxy for international medical graduates status. Thus the validity of drawing comparisons regarding the effect of this ethnic status across both international medical graduates and UK graduates, in whom the meaning of the construct may be very different, is questionable. Further work should focus on finer grained analysis in which graduates from different countries are compared. Such analyses should discriminate between international medical graduates who were taught in English and those taught in other languages, given our findings in relation to IELTS scores. One should also bear in mind that the ARCP data were obtained from doctors who were further on in their training than foundation year two. Therefore, we could speculate that intervening factors could have affected the disparity in performance between PLAB graduates and UK graduates. For example, international medical graduates may have experienced a poorer quality of training.

In comparison with royal college examinations, little is known about the reliability of the ARCP process. There may be instances of outliers, in terms of ARCP outcomes, in specific specialties within certain deaneries.³³ Moreover, there may have been some inconsistency in the reporting of the reason for an unsatisfactory ARCP outcome (such as examination failure) in 2010 (Smith D. Reliability of ARCP data collection by the GMC, 2010-2012. Personal communication to P A Tiffin, 2014). Nevertheless, compared with examinations, ARCPs may be a closer, although still imperfect, proxy for clinical practice, as they consider a relatively wide range of evidence provided by a trainee. However, like clinical examinations, they cannot be considered free from cultural influences and opportunities for assessor bias, which would seem to be a priority area for future research. The extent to which international medical graduates are represented as ARCP panel members or what effect, if any, this may have on the outcomes of assessments is currently unknown. Certainly, monitoring the world region of qualification as well as the ethnicity of postgraduate assessors and examiners may be important in research relating to potential bias or discrimination.

Although we were able to analyse the data according to specialty group, the low numbers of trainee doctors (and especially international medical graduates) in some of these (such as ophthalmology and public health) precluded adequately powered subgroup analysis in some cases. Moreover, the role that examinations play in ARCP outcome differs according to specialty; some have examinations earlier on in training, whereas others have “exit” fellowship examinations that allow entry to the specialist register. However, we hoped that variation across specialties would “wash out” so that an overall, average, picture of the performance of international medical graduates registering via the PLAB relative to UK graduates at ARCP could still be obtained. No data on deanery were available to the research team (this may have identified trainees in smaller specialties). However, deanery location has previously been shown to have a minimal effect on overall postgraduate performance; a recent report cited the intraclass correlation for deanery on MRCGP pass rates as generally less than 0.001.¹⁴

Understanding whether particular components of the PLAB examination, such as history taking, were especially predictive of ARCP performance would have been desirable. Unfortunately, this was not possible as PLAB subtest scores are not standardised between diets. Also, our analysis included only international medical graduates who obtained registration via the PLAB examination, so these findings cannot be applied to those who obtained GMC registration via alternative routes.

Conclusions and implications for policy

Although several demographic variables are predictive of ARCP performance, the main purpose of the PLAB examination is to ensure equivalence between international medical graduates and UK graduates at the end of foundation year one. However, these findings will contribute to continued discussions about the role that ethnicity plays in the assessment of UK doctors in training. Certainly, further detailed research and investigation is required before any assumption of systematic racial bias is made. The British Medical Association has recommended, following a survey of medical royal colleges, that data collection on the demographic diversity of their postgraduate examination candidates should be improved; only eight of 18 colleges reported analysing pass rates in relation to characteristics such as sex, ethnicity, disability, religious faith, and sexual orientation.³⁴

Patients need reassurance that doctors qualifying from overseas are as competent and professional as UK graduates, irrespective of any underlying demographic differences. Language proficiency is considered to be an important component of a physician’s ability and could be considered a necessary, if not sufficient, condition for clinical competence.³⁵ Nevertheless, the effect of IELTS scores on the disparity between international medical graduates’ and UK graduates’ ARCP performance was striking. The IELTS often uses examples of English that are quite technical in nature, as well as including role play in the speaking component. The test may thus be evaluating constructs relating to cognitive ability that are, to some extent, distinct from pure linguistic performance. This may partly explain the relatively strong ability of IELTS scores to predict ARCP performance in international medical graduates. Improving overseas doctors’ confidence in culturally sensitive communication may be effective in improving consultation skills; one Scottish pilot of such a training programme reported encouraging findings.³⁶ Certainly, our results suggest that increasing the IELTS score (or equivalent) at which international medical graduates are permitted to take the PLAB examination will reduce the disparity in ARCP performance compared with UK graduates. There may also be ways in which health services can support overseas doctors in adapting to the culture of the UK.³⁷

The pass mark for both parts of the PLAB would clearly need to be set much higher to substantially reduce the differences in performance observed between international medical graduates and UK graduates. In line with the findings from the parallel study,¹⁸ our results suggest that the pass mark would have to be raised considerably (by at least 30 points for part 1 of the PLAB) to substantially move towards equivalence; so much so that very few of the current PLAB candidates would have passed. This does not mean that current PLAB graduates would not succeed at an examination with a higher pass mark, but they would need to raise their performance to do so. An alternative would be to introduce a different examination. Our findings also indicate that limiting the number of times the PLAB examinations can be taken is likely to result in some closing of the performance gap between international medical graduates and UK graduates. Together, such changes would create severe workforce planning challenges for the NHS, which has traditionally relied on international medical graduates, especially in the less popular specialties such as psychiatry. It is also interesting to note that in Canada international medical graduates must pass both parts of the Medical Council of Canada examination—the Medical Council of Canada Qualifying Examination (MCCQE). The pass mark is identical for international medical graduates and Canadian medical graduates.³⁸ Despite this, as in the UK, international medical graduates later show higher failure rates at postgraduate examination.³⁹ Thus, it may be erroneous to conclude that if only the pass mark of the PLAB were set high enough, international medical graduates in the UK would show identical performance at later assessments. Even marked variation between graduates of different UK medical schools in performance at royal college membership examinations has been previously reported.⁴⁰ As such, achieving early “equivalence” between UK graduates and international medical graduates may be unfeasible and partly depend on exposure to UK practice and culture. Rather, medical regulators must decide on the levels of clinical skill and professionalism required for UK practice and implement policy accordingly.¹⁶ In the meantime, we would recommend that the IELTS (or equivalent) score required to take the PLAB is increased and that pass marks for both parts of the PLAB test are raised, with the number of resits permitted restricted. These steps are likely to result in reduced disparities in postgraduate performance between PLAB graduates and UK graduates.

What is already known on this topic

• Compared with UK medical graduates, international medical graduates are more likely to fail postgraduate royal college examinations

• International medical graduates are also more likely to be referred to the General Medical Council for concerns relating to fitness to practise

• However, the extent to which the Professional and Linguistic Assessments Board (PLAB) test ensures that international medical graduates show equivalent performance to UK graduates who have completed foundation year one is unclear

What this study adds

• The PLAB examination system does not result in equivalent performance between international medical graduates and UK graduates, as evaluated later at Annual Review of Competence Progression (ARCP)

• Performance at both the International English Language Test System and the two parts of the PLAB are significantly predictive of ARCP outcome in international medical graduates

• Some, but not all, of the differences in ARCP performance may be explained by disproportionately high rates of membership examination failure, especially in psychiatry and general practice

Cite this as: BMJ 2014;348:g2622