Predicting international medical graduate success on college certification examinations

Abstract

Objective

To determine predictors of international medical graduate (IMG) success in accordance with the priorities highlighted by the Thomson and Cohl judicial report on IMG selection.

Design

Retrospective assessment using regression analyses to compare the information available at the time of resident selection with those trainees’ national certification examination outcomes.

Setting

McMaster University in Hamilton, Ont.

Participants

McMaster University IMG residents who completed the program between 2005 and 2011.

Main outcome measures

Associations between IMG professional experience or demographic characteristics and examination outcomes.

Results

The analyses revealed that country of study and performance on the Medical Council of Canada Evaluating Examination are among the predictors of performance on the College of Family Physicians of Canada and the Royal College of Physicians and Surgeons of Canada certification examinations. Of interest, the analyses also suggest discipline-specific relationships between previous professional experience and examination success.

Conclusion

This work presents a useful technique for further improving our understanding of the performance of IMGs on certification examinations in North America, encourages similar interinstitutional analyses, and provides a foundation for the development of tools to assist with IMG education.

More international medical graduates (IMGs) than ever are being admitted into Canadian residency programs. Unfortunately, the rates of failure on college certification examinations are substantially higher for IMGs than for their counterparts who graduate from North American medical schools. In Canada, for instance, retrospective review of the Royal College of Physicians and Surgeons of Canada (RCPSC) and College of Family Physicians of Canada (CFPC) examination outcomes has identified that IMGs have substantial difficulty achieving certification. For example, in 2007, only 66% of IMG residents were successful on the CFPC examination, compared with 90% of their Canadian medical graduate counterparts. The overall success rate for IMGs on the CFPC Certification examination rose to 74% in 2008 but dropped to 64% and 51% in 2009 and 2010, respectively. The problem is equally troubling among IMG residents in other specialties: between 2005 and 2010, only 75% of IMG residents passed the RCPSC examination, while 96% of Canadian medical graduate residents demonstrated the required performance for certification.^1–4 This pattern of performance is particularly troubling as the number of IMGs admitted to Canadian residency programs continues to grow in response to a country-wide rural physician shortage.⁵

A number of different hypotheses have been proposed to explain the depressed IMG certification success rates. Some of these reflect common demographic characteristics of the population, such as difficulties with English⁶ and the general older age of IMG residents,⁷ while others focus on the features of the foreign medical education experience,^8–10 the length of time elapsed since medical school graduation,¹¹ and the potential effect of previous clinical experiences.¹² Some have even proposed ideas about the effects of cultural beliefs about gender roles and physician-patient communication.^13,14 Walsh and colleagues offer a review of factors and hypotheses.¹ However, to date, no comprehensive effort has been made to uncover which factors underpin IMG certification examination performance.

An independent review of access to postgraduate programs by IMGs in Ontario was conducted and submitted to the Ontario Ministry of Health and Long-Term Care and the Council of Ontario Universities in 2011.¹⁵ This review highlighted very clearly the importance of discerning these factors:

The faculties of medicine, supported by the Ontario government, should identify research priorities to increase the evidence base for selection decisions and outcomes, including … [p]redictors for success in residency and beyond, including the best ways to weigh and measure those factors .… We recommend that support for research on predictors of success and ways to improve certification exam results, using the Ontario experience, should be a priority.¹⁵

In the present study, representatives of the McMaster University postgraduate medicine programs in Hamilton, Ont, act on these recommendations, and we take an explained variance approach to determining the value of the information available at the time of residency admission in predicting IMG performance on the certification examinations. To do so, linear and logistic regression analyses were performed using the information submitted to the McMaster University postgraduate residency programs through the Canadian Resident Matching Service (CaRMS) between 2005 and 2011 in order to determine which characteristics predict IMG residents’ certification examination performance and to what extent. An IMG applicant’s CaRMS package provides details about his or her medical school experience, the country in which he or she studied, his or her professional activities abroad, and usually his or her marks on the Medical Council of Canada Evaluating Examination (MCCEE), among other things. Because of the incredible complexity of the data set and the wide variation in previous speculation about these factors, we did not perform these analyses as a test of any one specific hypothesis. Rather, our intentions are 2-fold. First, we anticipate that the results of these analyses will improve our understanding of the IMG certification performance phenomenon in general. Second, we hope that this work will establish the feasibility of this methodologic endeavour and encourage more comprehensive (ie, interinstitutional) analyses.

METHODS

Regression models were developed in order to determine which factors from among those available at the time of residency selection predict the performance of McMaster University IMG residents on the CFPC and RCPSC certification examinations. In order to facilitate model development, 2 separate data sets were obtained (CFPC, n = 69; RCPSC, n = 85). Three dependent variables were selected for models based on the CFPC data set: score on the simulated office orals (SOOs) portion of the examination (continuous variable), score on the short-answer management problems (SAMPs) portion (continuous variable), and pass or fail on the SOOs and SAMPs combined (dichotomous variable). Because the assessment rubric that generated the continuous variables for the SOOs and SAMPs was changed by the CFPC in 2007, these values were transformed and presented as z scores. The dependent variable for models based on the RCPSC data set was pass or fail on the first attempt of the RCPSC certification examination (dichotomous variable). Multivariable linear and logistic regression models were employed when the dependent variables were continuous and dichotomous, respectively.

Independent variables for each model were organized according to whether they related to professional experience or demographic characteristics. When factors related to professional experience were used to predict the dependent variables, the following were included as independent variables in the regression model: previously completed internship (yes or no), previous residency (yes or no), previous professional physician experience (yes or no), and research experience (yes or no). Age (years), first language other than English (yes or no), time since graduation from medical school (years), and country of medical training (continuous) were selected as independent variables in the models for demographic characteristics. Country of birth (Canadian or other) was also included in the models for demographic characteristics because a large contingent of Canadians study medicine abroad and are required to compete for IMG residency positions when they return to Canada. Each country was assigned a value according to the Human Development Index (HDI),¹⁶ thereby transforming the associated country of medical training into a continuous variable. The HDI assigned to each country was based on the 2011 rating. Where independent variables were dichotomous, the “no” scenario was used to define the reference category in each regression model.

In addition, a Pearson correlation matrix was produced in order to determine the magnitude of association between each of the variables in the data set. Again, this was done for both the CFPC and RCSPC data sets. Of primary interest was the correlation between the MCCEE scores and the certification examination scores. All data used in this study were collected from among McMaster University IMG residents who attempted either certification examination between 2005 and 2011. Analyses were performed using IBM SPSS, version 20, for Mac OS. A P value of less than .05 was considered significant; however, because the purpose of this study was exploratory, consideration was given to effects approaching P < .1. All data were collected and analyzed in accordance with the guidelines set out by the Hamilton Integrated Research Ethics Board, which provided ethics approval.

RESULTS

College of Family Physicians of Canada examination

Analyses of the CFPC database yielded 6 regression models (3 outcome measures for each of 2 independent variable themes). When examining the determinants of scores on the SOOs, professional experience (β = −5.07, P = .04), HDI value (β = 17.8, P = .046), and a first language other than English (β = 4.39, P = .05) were significant after adjustment for various factors (Table 1). Linear regression model parameters related to all factors included in both the professional experience (R² = 0.07) and demographic (R² = 0.15) models for the SOOs are presented in Table 1.

Table 1.

Multivariable linear regression model for performance on the CFPC Certification examination: Dependent variable is score on the SOOs.

INDEPENDENT VARIABLE	β	95% CI	P VALUE
Professional experience
• Internship	3.00	−1.80 to 7.83	.22
• Previous residency	1.44	−2.84 to 5.71	.51
• Professional experience	−5.07	−9.95 to −0.19	.04
• Research experience	−2.01	−6.11 to 2.08	.33
Demographic characteristics
• Country of birth	2.01	−1.91 to 5.92	.31
• HDI value	17.8	0.37 to 35.3	.046
• Age	−0.04	−0.58 to 0.50	.89
• Years since graduation	−0.24	−0.77 to 0.30	.38
• First language	4.39	−0.08 to 8.86	.05

CFPC–College of Family Physicians of Canada, HDI—Human

Development Index, SOO—simulated office oral.

Professional experience was also a predictor of SAMPs scores (β = −4.39, P = .045) after adjustment for various factors. Country of birth was associated with scores on the SAMPs, with those born outside Canada showing modestly better performance (β = 3.37, P = .049). Predictor variables for SAMPs, along with their corresponding P values, are presented in Table 2. The R²′ values for the professional and demographic models of SAMPs were 0.07 and 0.19, respectively.

Table 2.

Multivariable linear regression model for performance on the CFPC Certification examination: Dependent variable is score on the SAMPs.

INDEPENDENT VARIABLE	β	95% CI	P VALUE
Professional experience
• Internship	2.09	−2.16 to 6.34	.33
• Previous residency	−1.04	−4.81 to 2.72	.58
• Professional experience	−4.39	−8.69 to −0.09	.045
• Research experience	−1.17	−4.77 to 2.44	.52
Demographic characteristics
• Country of birth	3.37	0.01 to 6.74	.049
• HDI value	−4.95	−19.95 to 10.1	.51
• Age	−0.20	−0.66 to 0.27	.40
• Years since graduation	−0.20	−0.66 to 0.27	.39
• First language	1.85	−1.99 to 5.70	.34

CFPC–College of Family Physicians of Canada, HDI—Human

Development Index, SAMP—short-answer management problem.

Forty-nine of the 69 IMGs in the CFPC database passed both the SOOs and SAMPs examinations (71%). Logistic regression models developed to predict success on both examinations yielded no significant factors from among those related to professional experience or demographic characteristics. The odds ratios (ORs) and corresponding P values for factors included in each model for determining success on both examinations are presented in Table 3. Hosmer-Lemeshow test results for both models were non-significant (P > .6), demonstrating a good fit of the data. However, the classification tables showed approximately 75% concordance between model predictions and actual observation, with most of the “failures” on both examinations classified as “successes.”

Table 3.

Logistic regression model for performance on the CFPC Certification examination: Dependent variable is success on both SOOs and SAMPs.

INDEPENDENT VARIABLE	OR	95% CI	P VALUE
Professional experience
• Internship	2.34	0.61 to 8.92	.22
• Previous residency	0.92	0.29 to 2.89	.88
• Professional experience	0.39	0.098 to 1.54	.18
• Research experience	1.18	0.396 to 3.54	.76
Demographic characteristics
• Country of birth	1.54	0.49 to 4.87	.46
• HDI value	0.49	0.002 to 97.7	.79
• Age	0.86	0.71 to 1.04	.12
• Years since graduation	1.05	0.87 to 1.25	.63
• First language	1.07	0.29 to 3.95	.92

CFPC–College of Family Physicians of Canada, HDI—Human

Development Index, OR—odds ratio, SAMP—short-answer management problem, SOO—simulated office oral.

The Pearson correlation matrix showed a moderate positive correlation between MCCEE and SAMPs scores (R = 0.48, P < .001). Likewise, there was a modest, although significant, positive correlation between MCCEE scores and success on both SOOs and SAMPs (R = 0.32, P = .018).

Royal College of Physicians and Surgeons of Canada examination

Two additional logistic regression models were developed using data from the RCPSC data set, such that predictors of success on the RCPSC examination could be determined. Of the 85 IMGs in the data set, 66 passed the examination on the first attempt (78%). After adjustment for factors related to professional experience, individuals who had participated in a previous internship had increased odds of passing the RCPSC examination when compared with those who had not participated in such an internship (OR = 4.09, P = .02). Regression models yielded no other significant factors. The ORs and corresponding P values for factors included in both the professional experience and demographic models are presented in Table 4. Hosmer-Lemeshow test results for both models were non-significant (P > .99), again demonstrating a good fit of the data. However, these results should be interpreted with caution, as the classification tables for both models showed a substantial misclassification among “failures” (80% overall concordance). Finally, there was a modest and significant correlation between MCCEE scores and success on the RCPSC examination (R = 0.264, P = .027).

Table 4.

Logistic regression model for performance on the RCPSC certification examination: Dependent variable is success on first try of RCPSC examination.

INDEPENDENT VARIABLE	OR	95% CI	P VALUE
Professional experience
• Internship	4.09	1.24 to 13.5	.02
• Previous residency	1.48	0.46 to 4.76	.51
• Professional experience	0.49	0.14 to 1.66	.25
• Research experience	0.67	0.20 to 2.26	.52
Demographics
• Country of birth	1.53	0.50 to 4.73	.46
• HDI value	0.15	0.001 to 37.9	.50
• Age	0.84	0.67 to1.05	.13
• Years since graduation	1.16	0.91 to 1.48	.23
• First language	0.66	0.21 to 2.12	.49

HDI—Human Development Index, OR—odds ratio, RCPSC—Royal College of Physicians and Surgeons of Canada.

DISCUSSION

This study aimed to identify which factors among the many that are collected and evaluated through the CaRMS application process might serve as the best predictors for IMG resident performance on the CFPC and RCPSC certification examinations. Our hope in performing these analyses was that we might come to better understand the phenomenon of depressed IMG certification examination performance in North America, while also providing a methodologic framework for conducting similar but more comprehensive analyses across multiple institutions. Our results highlight first language, birthplace, country of medical school training, and previous professional experience as among the most important factors for predicting IMG success. However, it is important to note that the effects described for first language and birthplace are counter to what might be intuitively expected. Specifically, the analyses revealed that CFPC IMGs who spoke English as a first language performed worse on the SOOs than those who did not speak English as a first language and, similarly, IMG Canadians who studied abroad performed worse on the CFPC SAMPs than their foreign-born counterparts did. Of note, these results contradict previous research (eg, van Zanten and Boulet, 2008⁶), which has argued that English as a first language and Canadian birth are positive predictive factors. It is important to recognize that the predicted effect of these factors on the test scores (4.39 points and 3.37 points, for first language and birthplace, respectively) in our study is likely small (the SOOs and SAMPs have assessment rubrics that exceed 600 total points). However, it appears that being a native English speaker or a Canadian who studies abroad provides minimal if any advantage with respect to the Certification examination.

The data set also revealed a notable but non-significant effect of age on IMG certification success. For the CFPC (OR = 0.86, P = .12) and RCPSC (OR = 0.84, P = .13) cohorts, with each year of age older and after adjustment for the other demographic variables included in the models, IMGs were 14% and 16% less likely to be successful, respectively. Unfortunately, this analysis does not reveal whether this effect is a function of age-related declines in performance, age-associated factors (eg, family, finances), or some combination of both. Indeed, some have suggested that older IMG residents develop work habits during their medical training that are harder to adapt to the demands and expectations of the North American medical system.⁷

Yet, perhaps the most interesting finding surrounds the discipline-specific effect of having clinical experience before commencing the Canadian residency program. In particular, professional experience was a negative predictor for CFPC IMG performance on the SAMPs, while an internship was a positive predictor of IMG performance on the RCPSC certification examination. This pattern of results might be indicative of the number of IMGs who are trained in specialist medicine in their home countries who take on family physician roles in Canada or a cultural aspect to family practice that is not present in the specialist disciplines that affects the way foreign-trained medical graduates perform in the clinical environment. That the HDI score associated with the country of medical school training is a significant predictor of SOOs performance seems to support the latter position, as the countries that are most culturally similar to Canada rate the highest on this measure.

Limitations

Our study is limited by the fact that we were only able to examine the outcomes for IMG residents at a single university. This meant that the overall number of IMGs was fairly small, and we cannot rule out the possibility of bias related to the particular university where the study was conducted. We are hopeful that future work will replicate this study at other universities and in inter-institutional contexts.

Conclusion

This work represents a critical initiative in developing an evidence-based platform for determining more specifically which of the many proposed factors are most useful for explaining IMG examination performance. In the process, McMaster University embraced the recommendations of the Thomson and Cohl report in “supporting research and pilot projects to promote successful IMG selection, training, and assessment processes.”¹⁵ Moving forward, this work will benefit from collaborative, replicative efforts across Canadian residency programs. Doing so will result in a stronger, more comprehensive data set that is reflective of the IMG population continent-wide, which will provide opportunities for rigorous hypothesis testing of the proposed mechanisms. Ultimately, this work will improve understanding of the IMG certification success phenomenon, help residency programs identify at-risk residents, and underpin the development of specific educational and remedial interventions.

EDITOR’S KEY POINTS

• More international medical graduates (IMGs) than ever are being admitted into Canadian residency programs. Unfortunately, the rates of failure on college certification examinations are substantially higher for IMGs than for their counterparts who graduate from North American medical schools.

• This study aimed to determine predictors of IMG success in order, ultimately, to improve understanding of what leads to IMG certification success, to help residency programs identify at-risk residents, and to underpin the development of specific educational and remedial interventions. The authors found that first language, birthplace, country of medical training, and previous professional experience were among the most important factors for predicting IMG success.

• Of interest, the effects described for first language and birthplace were counter to what might be expected and to what previous studies have reported. Specifically, IMGs who spoke English as a first language and IMG Canadians who studied abroad performed worse on elements of the certification examinations than their counterparts did, although the effects were small. Further, the effects of previous professional experience were discipline specific, with it being a negative predictor for family medicine IMG success and a positive predictor for Royal College certification success.

POINTS DE REPÈRE DU RÉDACTEUR

• De plus en plus de médecins diplômés à l’étranger (MDE) sont admis dans les programmes de résidence canadiens. Malheureusement, leur taux d’échec aux examens de certification des collèges est considérablement plus élevé que celui des diplômés des facultés de médecine nord-américaines.

• Cette étude voulait identifier les prédicteurs de réussite pour les MDE, et ce, pour éventuellement mieux comprendre les raisons de leur réussite aux examens de certification, pour aider les programmes de résidence à identifier les résidents à risque et pour mettre en place des interventions spécifiques de nature éducative et corrective. Les auteurs ont observé que la langue maternelle, le lieu de naissance, le pays de la formation médicale et l’expérience professionnelle antérieure comptaient parmi les principaux prédicteurs de réussite de ces MDE aux examens.

• Il est intéressant de noter que dans le cas de la langue maternelle et du lieu de naissance, les effets étaient contraires à ce qu’on aurait croire et à ce que les études antérieures ont rapporté. Plus spécifiquement, les MDE dont la langue maternelle était l’anglais et les MDE canadiens qui avaient étudié à l’étranger performaient moins bien aux items des examens de certification que leurs contreparties, quoique la différence soit faible. De plus, les effets de l’expérience professionnelle antérieure dépendaient de la discipline; en effet, l’expérience constituait un prédicteur de réussite négatif pour les médecins de famille formés à l’étranger, mais elle était un prédicteur de succès positif dans le cas de l’examen de certification du Collège royal.