Table 2.
Predictors of success and failure in certification exams
| Author/ year/ sample size/country/RoB | Predictor | OR (95% CI) | P Value | Outcome | Baseline risk for outcome | Predictor/absolute success or failure change |
|---|---|---|---|---|---|---|
| Age | ||||||
|
Grierson, 2017 [15] N = 469 Canada |
Age 10 years decrement | 1.68 (1.15, 2.44) | 0.007 | Success on Simulated Office Orals component of the Certification in The College of Family Physicians examination | 91.5% | Each 10 years decrement in age increase success rate by 3.3% (95%CI: 1–4.8% more) |
|
Grierson, 2017 [15] N = 469 Canada |
1.86 (1.33, 2.60) | 0.001 | Success on Short Answer Management Problems component of the Certification in The College of Family Physicians examination | 85.5% | Each 10 years decrement in age increase success rate by 6.1% (95%CI: 3.2–8.4% more) | |
|
Grierson, 2017 [15] N = 469 Canada |
1.76 (1.32, 2.33) | 0.001 | Success on composite Certification in The College of Family Physicians examination | 79.3% | Each 10 years decrement in age increase success rate by 7.8% (95%CI: 4.2–10.6% more) | |
|
Grierson, 2017 [15] N = 431 Canada |
1.54 (1.08, 2.18) | 0.02 | Success on Royal College of Physicians and Surgeons of Canada certification examination | 84.4% | Each 10 years decrement in age increase success rate by 4.9% (95%CI: 1–7.8% more) | |
|
Menzies, 2015 [42] N = 2,056 UK |
Age 10 years decrement | 0.71 (0.53, 0.97) | 0.029 | Membership of the Royal College of Paediatrics and Child Health (MRCPCH) part 1B examination outcome | 41% | Each 10 years decrement in age decrease success rate by 8% (95%CI: 0.7% less to 14.1% less) |
|
Tiffin, 2014 [39] N = 52,871 doctors UK |
Age 10 years decrement | 1 (0.86, 1.16) | 0.39 |
Less satisfactory Annual Review of Competence Progression (ARCP) outcome for international medical graduates (IMG) versus UK graduates (Ordinal outcome) |
NR | |
|
Sturesson, 2020 [28] N = 564, Sweden |
NR | NR | Failing Licensing Exam | Older candidates were more likely to fail | ||
|
Haukilahti, 2012 [44] N = 383 Finland |
Age < 35 vs. ≥ 35 | 1.02 (0.44, 2.38) | 0.96 | Success in licensing examinations of IMG doctors in Finland | 51.3% | IMG doctors < 35 vs. ≥ 35 was 0.5% more (20% less to 20.2% more) likely to pass the licensing exams |
| Sex: Female vs. Male | ||||||
|
Grierson, 2017 [15] N = 469 Canada |
Female vs. Male | 1.96 (1.15, 3.33) | < 0.01 | Success on Short Answer Management Problems component of the Certification in The College of Family Physicians examination | 85.5% | Female were more likely to be successful 6.5% more (95%CI: 1.7 to 9.7% more) |
|
Menzies, 2015 [42] N = 2,056 UK |
Female vs. Male | 0.95 (0.77, 1.18) | 0.63 | Membership of the Royal College of Paediatrics and Child Health (MRCPCH) part 1B examination outcome | 41% | Female were less likely to be successful 1.2% less (95%CI: 6.1 less to 4% more) |
|
Haukilahti, 2012 [44] N = 383, Finland |
Female vs. Male | 1.80 (1.02, 3.16) | 0.041 | Success in licensing examinations of IMG doctors in Finland | 51.3% | Female were more likely to pass the exam 14.2% more (95%CI: 0.5–25.6% more) |
|
Shaw, 2014 [41] N = 116 UK |
Female vs. Male | 3.33 (1.01, 11.11) | 0.048 |
Passing Clinical skill assessment In trainees who completed ST3 year General Practice Speciality in UK |
68% | Female were more likely to pass the exam 19.6% more (95%CI: 0.2–28% more) |
|
Tiffin, 2014 [39] N = 52,871 doctors UK |
Female vs. Male | 1.29 (1.23, 1.34) | NR | Obtaining more satisfactory Annual Review of Competence Progression (ARCP) outcome (outcome is ordinary variable) to assess all doctors in UK postgraduate training | Females were more likely to obtain more satisfactory Annual Review of Competence Progression (ARCP) | |
|
Mathews, 2017 [31] N = 876 Canada |
Sex (Female vs. Male) | 1.67 (1.20, 2.33) | 0.002 | Obtain Speciality Designation in family medicine |
Female IMG were more likely obtain speciality designation. 8% (95%CI: 3%,11%) |
|
| Fluency in English | ||||||
|
Grierson, 2017 [15] N = 469 Canada |
Fluency in English | 2.14 (1.25, 3.65) | < 0.01 | Success on Short Answer Management Problems component of the Certification in The College of Family Physicians examination | 85.5% | Fluent candidates were 7.16% (95%CI: 2.55–10.06%) more likely to be successful. |
|
Grierson, 2017 [15] N = 469, Canada |
Fluency in English | 1.89 (1.18, 3.0) | 0.008 | Success on composite Certification in The College of Family Physicians examination | 79.3% |
Fluent candidates were more likely to be successful. 8.6% more (95%CI: 2.6%, 12.7% more) |
|
Grierson, 2017 [15] N = 431 Canada |
Fluency in English | 1.87 (1.04, 3.38) | 0.04 | Success on Royal College of Physicians and Surgeons of Canada certification examination | 84.4% |
Fluent candidates were more likely to be successful. 6.6% more (95%CI: 0.5%, 10.4% more) |
|
Patterson, 2018 [1] N = 1,874 UK |
English fluency 1 score IELTS test |
Coefficient (95%CI) Linear regression 4.21 (3.18, 5.25) | < 0.001 | Clinical Skills Assessment (CSA) component of Membership of the Royal College of General Practitioners (MRCGP) | One score increase in IELTS (ranging 1–9) was associated with 4 (95%CI: 3–5) score increase in CSA | |
| Race and Ethnicity | ||||||
|
Bessant, 2006 [43] N = 483 UK |
White ethnic origin | 2.04 (1.42, 2.94) | 0.023 | success of candidates taking a revision course in preparation for the MRCP (UK) PACES (practical assessment of clinical examination skills) examination. | 45.3% |
White ethnicity graduates were more likely pass PACES 17.5% (95%CI: 8.7%, 25.6%) |
|
Tiffin, 2014 [39] N = 52,871 doctors UK |
White vs. Non-white ethnicity | 1.39 (0.94, 2.04) | NR | Obtaining less satisfactory Annual Review of Competence Progression (ARCP) outcome (outcome is ordinary variable) to assess all doctors in UK postgraduate training | NA (outcome is an ordinary variable) | |
| Nationality | ||||||
|
Haukilahti, 2012 [44] N = 383, Finland |
Nationality Other vs. Russian | 1.35 (0.81, 2.27) | 0.247 | Success in licensing examinations of IMG doctors in Finland | 51.3% | 7% (95%CI: -5%,19%) |
|
Schabort, 2014 [14] CFPC, N = 69; RCPSC, N = 85 Canada |
Country of birth (Other vs. Canada) | 1.54 (0.49, 4.87) | 0.46 | Success in Certification examination–College of Family Physicians of Canada, simulated office oral (SOO) and short-answer management problem (SAMP) | 71% | 8% (95%CI: -16%, 21%) |
|
Schabort, 2014 [14] CFPC, N = 69; RCPSC, N = 85 Canada |
Country of birth (Other vs. Canada) | 1.53 (0.50, 4.73) | 0.46 | Success on first try of Royal College of Physicians and Surgeons of Canada (RCPSC) examination. | 78% | 6% (95%CI: -14%, 16%) |
| Time from Graduation/ Graduation Recency | ||||||
|
Haukilahti, 2012 [44] N = 383 Finland |
Time from graduation < 10vs ≥ 10 | 1.20 (0.52, 2.86) | 0.658 | Success in licensing examinations of IMG doctors in Finland | 51.3% | 5% (95%CI: -15%, 24%) |
|
Schabort, 2014 [14] CFPC, N = 69; RCPSC, N = 85 Canada |
Years since graduation | 1.05 (0.87, 1.25) | 0.63 | Success in Certification examination–College of Family Physicians of Canada, simulated office oral (SOO) and short-answer management problem (SAMP) | 71% | 1% (95%CI: -3%, 4.4%) |
| Place of Qualification/Graduation | ||||||
|
Menzies, 2015 [42] N = 2,056 UK |
Place of qualification (UK graduates vs. IMG) | 3.17 (2.41, 4.17) | < 0.001 | Membership of the Royal College of Paediatrics and Child Health (MRCPCH) part 1B examination outcome | 41% |
UK graduates were more likely to pass MRCPCH exam 28% more (95%CI: 22–33% more) |
|
Bessant, 2006 [43] N = 483 UK |
Graduated from UK medical school. N = 483 |
4.87(3.86, 5.72) | 0.003 | Success of candidates taking a revision course in preparation for the MRCP (UK) PACES (practical assessment of clinical examination skills) examination | 45% |
Graduates from UK vs. overseas medical schools were more likely pass PACES 35% more (95%CI: 31–37% more) |
|
UK graduates with White ethnic origin N = 227 |
2.15 (1.21, 3.81) | 0.012 |
White vs. non-white ethnic UK graduates were more likely pass PACES 19% (95%CI: 5%, 31%) |
|||
|
Shaw, 2014 [41] N = 116 UK |
European vs. non-European university | 21.3 (5.6, 91.3) | < 0.001 |
Passing Clinical skill assessment in trainees who completed ST3 year in UK |
50% | Candidates graduated from European universities were 45.5% more (95%CI: 34.9 to 48.9% more) likely to pass CSA |
| Previous Experience | ||||||
|
Schabort, 2014 [14] CFPC, N = 69; Canada |
Previous Internship | 2.34 (0.61, 8.92) | 0.22 | Success in Certification examination–College of Family Physicians of Canada, simulated office oral (SOO) and short-answer management problem (SAMP) | 71% | Previous internship associated with 14% more (95%CI: 11% less to 25% more) success |
| Previous residency | 0.92 (0.29, 2.89) | 0.88 | Previous residency associated with 2% less (95%CI: 29% less to 17% more) success | |||
| Professional experience | 0.39 (0.098, 1.54) | 0.18 | Previous residency associated with 22% less (95%CI: 52% less to 8% more) success | |||
| Research experience | 1.18 (0.40, 3.54) | 0.76 | Previous residency associated with 3% more (95%CI: 22% less to 19% more) success | |||
|
Schabort, 2014 [14] RCPSC, N = 85 Canada |
Previous Internship | 4.09 (1.24, 13.5) | 0.02 | Success on first try of Royal College of Physicians and Surgeons of Canada (RCPSC) examination. | 78% |
IMG residents who previously completed internship were more likely to pass RCPSC exam on first try 16% (95%CI: 3%, 20%) |
| Previous residency | 1.48 (0.46, 4.76) | 0.51 | 6% (95%CI: -15%, 16%) | |||
| Professional experience | 0.49 (0.14, 1.66) | 0.25 | -15% (95%CI: -45%, 7%) | |||
| Research experience | 0.67 (0.20, 2.26) | 0.52 | -8% (95%CI: -37%, 11%) | |||
|
Sturesson, 2020 [28] N = 564 Sweden |
Participating in Complementary Program for physician (CPP) | NR | NR | Failing licensing exam | NR | Participants in CPP were less likely failing licensing exam |
|
Schabort, 2014 [14] CFPC, N = 69; Canada |
Human Development Index value (HDI) | 0.49 (0.002, 97.7) | 0.79 | Success in Certification examination–College of Family Physicians of Canada, simulated office oral (SOO) and short-answer management problem (SAMP) | 71% | -43% (95%CI: -78%, 21%) |
|
Schabort, 2014 [14] CFPC, N = 69; RCPSC, N = 85 Canada |
Human Development Index value (HDI) | 0.15 (0.001, 37.9) | 0.50 | Success on first try of Royal College of Physicians and Surgeons of Canada (RCPSC) examination | 78% | -43% (95%CI: -78%, 21%) |
|
Grierson, 2017 [15] N = 431 Canada |
Human Development Index value (HDI) | 41.75 (4.29, 406.05) | 0.001 | 84% |
Candidates from countries with higher HDI are more likely to succeed. 15% (95%CI: 11%, 16%) |
|
| Previous exam and qualifications | ||||||
| USA | ||||||
|
Peterson, 2020 [33] N = 3,255 USA |
USMLE Step1 | 0.99 (0.99, 1.01) | NR | Failing to obtain American board of family medicine (ABFM) certificate by IMGs | 8% failed | Each higher score in USMLE step 1 reduce probability of failing by 0.07% (95%CI: 0.07%less to 0.07% more) |
| USMLE Step 2 Communication and interpersonal skills | 1.00 (0.98, 1.02) | NR | Each higher score in USMLE step 2 do not change probability of failing (95%CI: 0.15%less to 0.15% more) | |||
|
USMLE Step 2 Spoken English proficiency |
1.04 (1.02, 1.06) | NR | Each higher score in USMLE step 2 Spoken English proficiency increase probability of probability of failing by 0.3% (95%CI: 0.15–0.44% more) | |||
|
USMLE Step 2 Integrated clinical encounter |
0.99 (0.97, 1.01) | NR | Each higher score in USMLE step 2 Integrated clinical encounter reduce probability of failing by 0.07% (95%CI: 0.22%less to 0.07% more) | |||
| USMLE Step 2 CK | 0.99 (0.98, 0.99) | NR | Each higher score in USMLE step 2 CK reduce probability of failing by 0.07% (95%CI: 0.07–0.15% less | |||
| USMLE Step 3 | 0.99 (0.98, 1.00) | NR | Each higher score in USMLE step 3 reduce probability of failing by 0.07% (95%CI: 0.00–0.15% less | |||
| ABFM in-training examination (ITE) PGY-1 | 0.99 (0.99, 0.99) | NR | IMGs who got higher score on ITE PGY1 were less likely to fail in ABFM certificate. -0.07% | |||
| ABFM ITE PGY-2 | 0.99 (0.99, 0.99) | NR | IMGs who got higher score on ITE PGY2 were less likely to fail in ABFM certificate. -0.07% | |||
| ABFM ITE PGY-3 | 0.99 (0.99, 0.99) | NR | IMGs who got higher score on ITE PGY3 were less likely to fail in ABFM certificate. -0.07% | |||
| Exam in UK | ||||||
|
Menzies, 2015 [42] N = 2,056 UK |
Part 1 A concurrently undertaken | 1.21 (0.81, 1.83) | 0.36 | Membership of the Royal College of Paediatrics and Child Health (MRCPCH) part 1B examination outcome | 41% | 4.7% more (5% less to 15% more) |
|
Patterson, 2018 [1] N = 1,874 UK |
Clinical Problem-Solving Test CPST | Coefficient (95%CI) Linear regression 0.05 (0.04, 0.07) | < 0.001 | Clinical Skills Assessment (CSA) component of Membership of the Royal College of General Practitioners (MRCGP) | NR | One score increase in CPST was associated with 0.05 (95%CI: 0.04, 0.07) score increase in CSA |
| Situational Judgment Test SJT | Coefficient (95%CI) Linear regression 0.07 (0.05, 0.09) | < 0.001 |
IMG with higher SJT score were more likely pass the CSA One score increase in SJT was associated with 0.07 (95%CI: 0.05, 0.09) score increase in CSA |
|||
|
Bessant, 2006 [43] N = 483 UK |
Passed part 2 written paper first time in all candidates. | 3.64 (2.31, 5.73) | 0.002 | success of candidates taking a revision course in preparation for the MRCP (UK) PACES (practical assessment of clinical examination skills) examination | 45% |
Candidates who passed part 2 written paper first time were more likely to pass PACES 30% (95%CI: 20%, 37%) |
| Exam in Canada | ||||||
|
Mathews, 2017 [31] N = 876 Canada |
Had skill assessment (Yes vs. No) | NR | NS | Obtain Speciality Designation in family medicine | 78.1% | |
| Years between MD and PGME ≥ 6 years vs. ≤ 5 years | NR | NS | ||||
Green: Low risk of bias, Yellow: moderate risk of bias, Red: high risk of bias
NS = Not-significant, NR = Not reported, Bold font = Significant;