How do IMGs compare with Canadian medical school graduates in a family practice residency program?

ABSTRACT

OBJECTIVE

To compare international medical graduates (IMGs) with Canadian medical school graduates in a family practice residency program.

DESIGN

Analysis of the results of the in-training evaluation reports (ITERs) and the Certification in Family Medicine (CCFP) examination results for 2 cohorts of IMGs and Canadian-trained graduates between the years 2006 and 2008.

SETTING

St Paul’s Hospital (SPH) in Vancouver, BC, a training site of the University of British Columbia (UBC) Family Practice Residency Program.

PARTICIPANTS

In-training evaluation reports were examined for 12 first-year and 9 second-year Canadian-trained residents at the SPH site, and 12 first-year and 12 second-year IMG residents at the IMG site at SPH; CCFP examination results were reviewed for all UBC family practice residents who took the May 2008 examination and disclosed their results.

MAIN OUTCOME MEASURES

Pass or fail rates on the CCFP examination; proportions of evaluations in each group of residents given each of the following designations: exceeds expectations, meets expectations, or needs improvement. The May 2008 CCFP examination results were reviewed.

RESULTS

Compared with the second-year IMGs, the second-year SPH Canadian-trained residents had a greater proportion of exceeds expectations designations than the IMGs. For the first-year residents, both the SPH Canadian graduates and IMGs had similar results in all 3 categories. Combining the results of the 2 cohorts, the Canadian-trained residents had 310 (99%) ITERs that were designated as either exceeds expectations or meets expectations, and only 3 (1%) ITERs were in the needs improvement category. The IMG results were 362 (97.6%) ITERs in the exceeds expectations or meets expectations categories; 9 (2%) were in the needs improvement category. Statistically these are not significant differences. Seven of the 12 (58%) IMG candidates passed the CCFP examination compared with 59 of 62 (95%) of the UBC family practice residents.

CONCLUSION

The IMG residents compared favourably with their Canadian-trained colleagues when comparing ITERs but not in passing the CCFP examination. Further research is needed to elucidate these results.

For many years international medical graduates (IMGs) have made up approximately 25% of the physician work force in Canada. In British Columbia (BC), 22% of postgraduate physicians have been trained abroad.^1–3

There is evidence in the literature that IMGs do not do as well as Canadian-trained residents in their residencies or on their final examinations; and there is a perception among program directors that IMGs do not perform as well.^4–7

In 2005, the BC provincial government increased its funding for residency training for IMGs from 6 to 18 positions per year, with 12 positions in the specialty of family medicine and 6 in other specialties.⁸ A provincially appointed IMG task force recommended that IMG family practice residents be trained at a newly created site at St Paul’s Hospital (SPH) in Vancouver, BC, with the first cohort starting in July 2006. St Paul’s Hospital has had an existing site for family practice residency training since 1994, and the attending clinicians have had extensive experience of family practice teaching as well as assessing IMGs since 1992.⁹ The existing SPH family practice residency site continued to recruit residents through the Canadian Residency Matching Service and matched only medical students from Canadian (or US) medical schools in the first iteration. The IMG residents were ranked through Canadian Residency Matching Service only if they had completed the BC IMG clinical assessment program.

The 2 residency programs (ie, SPH site and IMG site) would exist side by side and share many of the same teaching resources and some of the same rotations. Most IMGs in Canada are simply integrated into existing residency programs and their particular needs as IMGs are not examined in any depth.

The faculty members that were appointed to oversee the new IMG site had experience in teaching Canadian-trained graduates as well as long-standing exposure to and a specific interest in teaching IMGs.¹⁰ The anticipation was that by creating a specific training site for IMGs, the challenges associated with the various backgrounds and training of IMGs could be addressed—even though the 2 programs would be integrated educationally to a certain extent. In particular, the residents at the IMG site were given more extensive exposure to ethical, cultural, and behavioural medicine issues than their Canadian counterparts were. More time was spent analyzing doctor-patient relationships and communication issues.

To our knowledge (University of British Columbia [UBC] Family Practice Residency), this is the first time in North America that a training site has been created specifically for IMGs, with the added advantage of working alongside Canadian-trained residents. We believed that this IMG site provided a unique opportunity to monitor the educational progress of the IMGs in this setting by comparing them with their Canadian-trained colleagues.

METHODS

Every family practice resident in BC is continuously assessed by means of a Web-based evaluation, which is based on the 4 principles of family medicine. Preceptors complete the evaluation at the end of every rotation; a rotation can last for as little as 2 weeks for electives to as long as 8 weeks for most of the core rotations. A review was conducted of all the in-training evaluation reports (ITERs) for both the IMGs and the Canadian-trained residents, as well as the results of the Certification in Family Medicine (CCFP) examination, which is taken by our residents in May of the second year of the 2-year residency. Ethics approval was obtained from the UBC Providence Health Care Research Ethics Board.

The residents from the IMG and SPH sites were asked for their written consent for their ITERs to be collated and reviewed by the author with the individual evaluations being blinded before review. Consent was requested by the program coordinators, and any identifying details in the evaluations were removed.

Each ITER was graded initially as either pass or fail. Additionally, each evaluation was given one of the following designations: exceeds expectations, meets expectations, or needs improvement. The ITERs also contained the evaluators’ written comments but these were not seen by the author.

The CCFP examination results were obtained from the central office of the UBC Family Practice Residency Program with the residents’ permission.

RESULTS

All of the residents at the 2 sites gave written permission to have their ITERs reviewed. Residents who had gaps in their training for more than 3 months (all were on maternity leave) were excluded from analysis. Three second-year residents and 1 first-year resident from the SPH site (SPH Canadian graduates) were excluded, and no residents from the IMG site were excluded. Participants included 12 first-year and 9 second-year residents from the SPH site (SPH Canadian graduates), and 12 first-year and 12 second-year IMG residents from the IMG site. The ITERs from the cohorts between 2006 and 2008 (second-year residents) and between 2007 and 2008 (first-year residents) were analyzed to see if the SPH Canadian graduates and the IMGs showed significant differences. Results are presented in Tables 1 and 2.

Table 1.

Results of ITERs of first- and second-year SPH Canadian-trained residents and IMGs in the UBC Family Practice Residency Program, from the cohorts between 2007 and 2008 (first-year residents) and between 2006 and 2008 (second-year residents)

	NO. OF EVALUATIONS RECEIVED BY FIRST-YEAR RESIDENTS		NO. OF EVALUATIONS RECEIVED BY SECOND-YEAR RESIDENTS
ITER CATEGORIES	SPH CANADIAN GRADUATES N = 125,* N (%)	IMGs N = 127,* N (%)	SPH CANADIAN GRADUATES N = 188,* N (%)	IMGs N = 244,* N (%)
Exceeds expectations	53 (42)	49 (39)	106 (56)	72 (30)
Meets expectations	69 (55)	76 (60)	82 (44)	165 (67)
Needs improvement	3 (2)	2 (2)	0 (0)	7 (3)

ITER—in-training evaluation report, IMG—international medical graduate, SPH—St Paul’s Hospital.

^*N refers to the total number of evaluations received by that group; each resident received an evaluation at the end of the rotation.

Table 2.

Overall results of the ITERs: Comparison of SPH Canadian graduates with IMGs.

ITER CATEGORIES	NO. OF EVALUATIONS RECEIVED BY SPH CANADIAN GRADUATES N = 313,* N (%)	NO. OF EVALUATIONS RECEIVED BY IMGs N = 371,* N (%)
Exceeds expectations and meets expectations	310 (99)	362 (98)
Needs improvement	3 (1)	9 (2)

ITER—in-training evaluation report, IMG—international medical graduate, SPH—St Paul’s Hospital.

^*N refers to the total number of evaluations received by that group; each resident received an evaluation at the end of the rotation.

In the 2 first-year resident groups (ie, SPH Canadian graduates and IMGs), SPH Canadian graduates and IMGs have very similar ratios in the categories exceeds expectations, meets expectations, and needs improvement. In particular, Fisher exact tests (2-tailed) indicated that the rate of evaluations in the needs improvement category versus the other 2 categories was comparable in the 2 groups, P = .68. It remains to be seen if this is mirrored in the second year of their residencies.

A different pattern occurred between the second-year SPH Canadian graduates and IMGs, with a wider gap between the 2 groups: SPH Canadian graduate residents consistently had a larger ratio of the exceeds expectations designation. Remarkably, no second-year Canadian-trained resident had a single needs improvement designation in the whole 2 years. The rate of the needs improvement designation was significantly higher for the IMGs, P = .02 (2-tailed Fisher exact test).

However, when looking at the total number of needs improvement designations of the first- and second-year residents’ rotations, there is little difference between the 2 groups, with very few needs improvement designations (9 for IMGs and 3 for SPH Canadian graduates) being seen as weak in a rotation yet still meriting a pass. This difference was within the variation of results expected by chance, P = .21 (2-tailed Fisher exact test). Out of 684 evaluations given, only 1 was rated as a fail. Of the SPH Canadian graduate residents, 3 (1%) evaluations were given a needs improvement rating compared with 9 (2%) evaluations of the IMGs.

There were, however, considerable differences between IMGs and the Canadian graduates across all of the UBC family practice sites in the CCFP examination results. Of the 12 IMGs who completed the examination in May 2008, 7 passed. The percentage of passes in this group (58%) is in keeping with the historical national pass rate for IMGs (Brailovski C, oral communication, September 2008). In comparison, of the 62 Canadian-trained UBC Family Practice residents who revealed their results to the program, 59 passed—for a pass rate of 95%.

DISCUSSION

There could be any number of explanations for these results, especially drawing comparisons between a well established family practice residency site (ie, SPH site) and a fledgling site (ie, IMG site) unlike any other in the country. The average age of the IMG residents was 40 years and that of the SPH residents was younger than 30 years. Most of the IMGs also had family responsibilities, with children ranging from newborns to university students. English was a second language to all 24 IMGs.¹⁰ There are also some considerable differences between the 2 sites, especially in the second year of residency, with the IMG site having a more traditional rotation-based schedule compared with the SPH site’s horizontal schedule.

It could be argued that the ITERs are not a good method of assessing readiness for practice. Although ITERs are extensively used in Canadian residency programs as well as in other parts of the world, there continues to be criticism in the literature as to their reliability in forecasting the competence of medical residents^11,12; however, ITERs continue to be widely used and are constantly reviewed to improve their ability to predict performance.^13,14 Program directors express the need for ongoing assessments during residency training in addition to a final examination mandated by the Royal College of Physicians and Surgeons of Canada or the College of Family Physicians of Canada.¹⁵ If, however, the written comments of the evaluators had been assessed, it is possible that more differences might have been found rather than only relying on the limited categories reviewed.

The outcome for IMGs in the CCFP examination is disappointing, especially considering that very knowledgeable faculty members, many of whom have served as examiners or on the examination committee at the College of Family Physicians of Canada, worked very diligently with the IMG candidates to thoroughly familiarize them with the examination requirements. Specifically, each resident had the opportunity to learn about how simulated office orals are constructed and marked, and he or she was given at least 3 practice simulated office orals before sitting the CCFP examination.

This result is at odds with the perception that the IMG residents performed very adequately during their residency training and compared favourably with their Canadian-trained colleagues.

Limitations

The numbers in this analysis are relatively small and time alone will tell if the results in future will be different. Analyses at other locations where larger numbers are incorporated will be needed before definitive conclusions can be drawn.

This analysis does appear to support the conclusion that even when IMGs train in a dedicated site with highly experienced teachers, they still struggle to match their Canadian counterparts in passing their final examinations. One explanation is that, in some way, IMGs are disadvantaged by the format of the CCFP examination. It could be cultural; it could be the subtle nuances of the English language; it could be the more direct approach that some IMGs have in eliciting medical information; or it could even be that some examiners lack experience with IMGs.^16,17

The contrast between the results of the examination and the results of the ITERs might lead to the opinion that perhaps more weight should be given to the ITERs in determining a resident’s fitness for practice rather than relying solely on a terminal examination.

Conclusion

This analysis reveals that IMGs continue to have difficulties with passing the CCFP examination even when they have been carefully selected and have obtained residency training in a dedicated site. The results of the ITERs, however, seem to indicate that IMGs are seen by their teachers to be competent physicians who are ready for practice after 2 years of family practice residency training.

EDITOR’S KEY POINTS

• There is evidence in the literature that international medical graduates (IMGs) do not do as well as Canadian-trained residents in their residencies or on their final examinations; and there is a perception among program directors that IMGs do not perform as well.

• Most IMGs in Canada are simply integrated into existing residency programs, and their particular needs as IMGs are not examined in any depth.

• A specific training site for IMGs was created in order to address the challenges associated with the different backgrounds and training of IMGs.

• This analysis shows that IMGs continue to have difficulties with passing the Certification in Family Medicine examination even when they have been carefully selected and have obtained residency training in a dedicated site; however, results of in-training evaluation reports indicate IMGs are seen by their teachers to be competent physicians.

POINTS DE REPÈRE DU RÉDACTEUR

• Les données de la littérature indiquent que les diplômés internationaux en médecine (DIM) ne réussissent pas aussi bien que les résidents formés au Canada dans leur résidence et leurs examens finaux, une opinion que partagent les directeurs de ces programmes.

• Au Canada, la plupart des DIM sont simplement intégrés dans des programmes de résidence existants, sans que leurs besoins particuliers soient véritablement examinés.

• Un site de formation spécifique aux DIM a été créé afin de répondre aux défis liés aux origines et formations particulières de ces résidents.

• Notre analyse montre que les DIM continuent d’éprouver des difficultés pour obtenir leur certification en médecine familiale même s’ils ont été soigneusement sélectionnés et s’ils ont fait leur résidence dans un site créé pour eux; toutefois, les professeurs qui les évaluent en cours de formation considèrent qu’il sont des médecins compétents.