A critical analysis of The Royal College of Physicians and Surgeons of Canada examination experience

Abstract

The Royal College of Physicians and Surgeons of Canada (RCPSC) plays a leading role in specialty and subspecialty post graduate medical education (PGME) in Canada. As the RCPSC accredits PGME programs, these programs are structured to meet the RCPSC Competence by Design model and their CanMEDS roles. RCPSC Certification is required by Medical Regulatory Agencies (MRAs) across Canada as a condition of entry to independent practice. The RCPSC relies heavily on the use of high-stakes subject examinations as a key component of its Certification process. Recently, questions have been raised regarding the usefulness of such high-stakes examinations. If such examinations are to be fair and equitable, they must be designed and implemented in accordance with best practices for educational testing and the processes for implementation and grading must be transparent and fair. This paper reviews the recent literature on high-stakes examinations and best practices in examination construction, references the findings of a survey of RCPSC examination experiences conducted by the Society for Canadians Studying Medicine Abroad exploring the perception of respondents, and raises concerns regarding RCPSC examinations related to validity, reliability, and fairness. The paper concludes by recommending closer scrutiny of RCPSC examination processes by interested stakeholders and by provincial MRA's who delegate entry to practice decisions to the RCPSC.

Introduction

The Royal College of Physicians and Surgeons of Canada (RCPSC) was established by an Act of Canadian parliament in June 1929 to oversee post graduate medical education.¹ Over the course of the past 95 years, the RCPSC has established a prominent and highly influential role in Canadian medical education and entry to practice. The RCPSC influences the delivery of post graduate medical education for every specialty except family medicine (which is overseen by the College of Family Physicians of Canada). Through their Competence by Design and CanMEDS Framework,² and through their accreditation of all post graduate medical education programs in universities across Canada³ the RCPSC directs post graduate specialty training in Canada. Almost every provincial medical regulatory agency (MRA) in Canada requires RCPSC Certification as a condition of full Registration as a specialist. While the RCPSC is not legislatively empowered to make entry to practice decisions, most provincial MRAs have delegated their legislative authority to make decisions about competence for entry to practice in a specialty to the RCPSC as a third-party assessor. As a result of this delegation of authority, the RCPSC has assumed a leading role in determining entry to practice to medicine in Canada, with RCPSC examinations and Certification representing the final pre-requisite to application for full licensing by a provincial MRA. Given the significant role in entry to practice that the RCPSC has come to play, it is important to critically examine their Certification processes, and particularly their use of specialty and subspecialty examinations, to ensure that their examination processes are transparent, objective, impartial and fair, as required by fairness legislation in most provinces.

Standards for psychological and educational testing

The RCPSC Certification process does not exist in a vacuum. While the RCPSC is an independent not for profit Canadian corporation with the mandate of overseeing post graduate medical education, it exists within a legislative and academic environment which demands certain standards of practice for its Certification processes. An example of one academic standard is that established by the Standards for Educational and Psychological Testing (SEPT) established jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education.⁴

The SEPT are intended to provide guidelines for best practices in developing and evaluating educational tests and examinations and ensuring that educational tests and their interpretation of test scores is valid for the test’s intended uses. The SEPT provide guidelines for establishing validity and reliability of educational testing. Validity refers to the extent to which evidence and theory support that test scores are consistent with their intended uses. Reliability refers to “the consistency of scores across replications of a testing procedure, regardless of how this consistency is estimated or reported.”^4(p33)

According to the SEPT, validity of a test may be evaluated in several ways. One is content validity, or “an analysis of the relationship between the content of a test and the construct it is intended to measure.”^4(p14) Another method is convergent and divergent validity which involves evidence of the extent to which test scores relate to, or differ from, other measures intended to address the same or different constructs. Reliability is typically assessed through replications of the testing procedure to determine whether test results are consistent over time. This may involve retesting of the same individual over time, determining whether two parts of a test measuring the same criteria are consistent with each other (often referred to as split-half reliability), and other psychometric measures of consistency of test results.

In addition to concerns about validity and reliability, the SEPT addresses the issue of fairness in testing. Tavakol and colleagues in their paper on ensuring fairness in assessment in health professions education emphasize the important role the SEPT plays in ensuring that potential biases in test development and administration are minimized, and that tests are fair for all intended groups regardless of examinee characteristics.⁵ According to the SEPT, fairness includes elements such as fairness in treatment during the testing process, fairness as a lack of measurement bias, fairness in access to the constructs as measured without the test taker being biased by personal characteristics such as age, disability, gender, race, ethnicity or language, which ensures that the test measures only what it is intended to measure, and not factors that are irrelevant. Fairness also includes ensuring the validity of individual test score interpretations for the intended uses. The SEPT note that when drawing inferences about an examinee’s performance, skills and abilities, it is important to consider how the examinee’s individual characteristic such as ethnicity may interact with the design and implementation of the testing situation. They caution that, “Individuals who differ culturally and linguistically from the majority of the test takers are at risk for inaccurate score interpretations.”^5(p53) This consideration is particularly relevant when assessing the competence of international medical graduates (IMGs).

The Medical Council of Canada Qualifying Examination Part 1 as a “gold standard”

The Medical Council of Canada Qualifying Examination 1 (MCCQE1) provides an excellent example of the above SEPT standards in practice, and some might argue that it represents a “gold standard” in assessment of medical competency in Canada. The MCCQE1 is a standardized examination that “assesses the critical medical knowledge and clinical decision-making ability of a candidate at a level expected of a medical student who is completing their medical degree in Canada.”⁶ The MCCQE1 is required to be completed by all IMG applicants to the Canadian Residency Matching Service (CaRMS) for residency training, and is part of the requirements for the Medical Council of Canada Licentiate credential required by many provincial MRAs for Registration of all applicants including Canadian Medical Graduates (CMGs).

The Medical Council of Canada (MCC) regularly publishes a document entitled “Annual Technical Report.”⁷ This technical report transparently describes the MCCQE1 examination construction and outcomes with sections that include:

• Exam development based on an exam blueprint

• Psychometric properties of the exam including:

  • Standard setting
  • Validity
  • Reliability

• Exam security

• Quality assurance

• Examination Results including a breakdown of cohorts into Canadian Medical Graduates (CMGs) and International Medical Graduates (IMGs), and first time and repeat test takers

• Results of a post examination candidate survey

The MCCQE Part 1 Annual Technical Report practices correspond closely with the guidelines provided by the SEPT. The MCCQE1 appears well researched with clear transparent reporting of validity, reliability and psychometric properties of the exam. It provides feedback from candidates through a post-examination survey and transparent reporting of results. This transparent reporting creates confidence in the fairness of the MCCQE1 examination process which is also a key requirement of the SEPT. Finally, it is worth noting that the MCC provides examination candidates with a variety of exam preparation resources,⁸ including practice tests, and that there are also commercially available question banks which are a valuable study aid in preparing for the examination.⁹

How does the RCPSC examination process compare to the SEPT and the MCC?

A search of the RCPSC website using search terms such as “validity” and “reliability” reveals little information regarding validity and reliability of their various specialty exams. One document on the RCPSC website that relates to the development of multiple-choice questions (MCQs) for RCPSC exams is a 2004 publication that offers guidance to committee members in developing MCQs.¹⁰ This document describes the process for developing a well-constructed MCQ and mentions the importance of reliable and valid MCQs, but offers little information regarding how to assess and document that MCQ exams are valid and reliable. There is also a document on the RCPSC website most recently edited in 2018 entitled “Roles and Responsibilities of Examination Boards and Specialty Committees with Regards to Examinations.”¹¹ This document describes a variety of reports available to these committees including: a psychometric examination report, a multiple-choice question report, and a post examination survey report. This document contains strict limitations on the extent to which information from these reports may be shared, even within the RCPSC committees, and there appears to be no public sharing of these reports permitted, which reflects a concerning and apparently intentional lack of transparency compared with the recommendations of the SEPT and the practices of the MCC.

A search of the academic literature in Google Scholar using the search terms such as “validity,” “reliability,” “RCPSC,” and “examinations” also returned no studies researching the validity or reliability of RCPSC exams. This dearth of research may be a result of RCPSC confidentiality strictures and reflects a troubling lack of transparency regarding RCPSC exam processes. Regarding exam reporting, the only score reporting found on the RCPSC website was a page providing average pass rates of those who trained in Canada by specialty or subspecialty.¹² I was unable to find any document comparable to the MCC Annual Technical Report, and unlike the MCC Annual Technical Report, these examination results were not broken down into cohorts making it unclear how CMGs versus IMGs who trained in Canada (or IMGs who trained outside Canada) performed on this exam relative to each other.

Fairness and the RCPSC exams

The SEPT document above describes critical elements necessary for fair testing including fairness as a lack of measurement bias and fairness in access to the constructs as measured. Without a transparent and detailed technical report such as MCC provides, and without transparent access to psychometric data and reliability and validity studies, it is difficult to assess how fair the RCPSC exams are to different cohorts. In particular, the lack of data regarding performance of CMGs relative to IMGs raises questions as to whether there are differences between the performance of IMGs relative to CMGs, and if so, the origins of any differences.

As discussed below, some IMGs who have completed RCPSC examinations have reported difficulty passing these exams. These respondents attribute their difficulty passing these exams to a variety of factors such as the inclusion of Canadian specific minutiae in the exams that they deem irrelevant to assessment of entry to practice levels of competency, as well as lack of access to Canadian study resources that CMGs and IMGs who trained in Canada have ready access to from their programs. Without a transparent report from the RCPSC regarding IMG performance, it remains a question whether IMGs do more poorly than CMGs on RCPSC exams, and we are forced to rely upon anecdotal reports. Further, if indeed IMGs do perform more poorly, we are unable to determine whether this is due to CMGs being more competent than IMGs, or whether this is due to other factors including failure to design an examination that meets SEPT standards of fairly assessing candidates from different cultural and linguistic backgrounds, thus providing CMGs and IMGs trained in Canada with an unfair advantage.

Prat, in Healthy Debate,¹³ describes what he experienced as a lack of fairness and transparency from the RCPSC regarding his request for a formal review of his exam scores based on a belief that he failed the psychiatry specialty exam due to factually inaccurate questions being included in the exam. Prat notes that, “The examination process and appeal do not permit anyone to review the content of the examination as it is deemed ‘confidential to the Royal College and not shared with candidates.’ There is therefore no possibility for the candidates to check the accuracy of the expected answers, not even for the sake of understanding where they failed.” Prat goes on to state that, “When a request for appeal is made, there is limited chance of success since candidates do not have access to anything tangible. Moreover, the Royal College does not accept appeal requests for what it considers to be an ‘alleged error in content.’”

This RCPSC policy precluding an appeal based on questions regarding inaccurate content is described in the RCPSC Policy on Formal Review of Examinations,¹⁴ which states,

Formal reviews of examinations are conducted only based on alleged significant procedural irregularities in the assessment process, not because of alleged errors in content. The exclusion of errors in content applies to allegations of errors in either the questions and structure of the examination, or the evaluation and content of the responses provided by the candidate.

Prat concludes that, “The Royal College examination process is not transparent,” and that, “A fair and transparent process needs to be in place.”

Fairness and the SOCASMA RCPSC exam experiences survey

Another indication of public and candidate perceptions regarding the RCPSC examination processes may be derived from a reading of the SOCASMA RCPSC Examination Experiences Survey, a survey undertaken by the Society for Canadians Studying Abroad (SOCASMA), a summary of which is posted on their website.¹⁵ This Survey was completed between December 2022 and February 2023. SOCASMA surveyed fifty-three anonymous participants who completed an online Google Forms survey about their experiences with a variety of RCPSC specialty exams. As their research referenced here relies exclusively on secondary use of anonymous information it does not require Research Ethics Board approval to be reproduced in this paper (see Article 2.4).¹⁶

Caution needs to be used in interpreting the results of the SOCASMA RCPSC Examination Experiences Survey. The number of respondents is low with only fifty-three respondents. The summary posted online contains only a brief description of the methodology of the survey. It states that “participants were recruited though invitations and notices on various medical forums and through interested stakeholder groups,” and goes on to caution that, “this survey may be subject to some degree of response bias.”^16(p1) Given the reported methodology of the survey, selection bias is likely. The combination of selection bias and low respondent numbers limit the generalizability of the survey. Also, based on the online summary, the authors conducted only a surface analysis of quantitative data. As an example, and as noted later in this paper, there is no reporting of how many CMGs vs IMGs had access to past MCQ questions, and no analysis of how such access may have impacted reported respondent pass rates on RCPCS exams. Further, the study relies entirely on self-reported data. The survey included considerable anecdotal qualitative data, which is subjective in nature and not subject to independent verification. Despite these limitations, in the absence of any transparent reporting by RCPSC of the results from their post examination survey reports, this survey appears to represent the only publicly available information regarding RCPSC examination participant’s experience of the RCPSC examination process. As such, the Survey feedback is worth considering.

In terms of Survey demographics, the RCPSC Examination Experiences Survey reported that “67.3% of respondents were International Medical Graduates (IMGs), 30.8% were Canadian Medical Graduates (CMGs) and 1.9% were US Medical Graduates (USMGs).”^15(p1) One of the questions the Survey asks respondents is, “In your opinion, do you believe the RCPSC exam you wrote was objective and fair?” Of 33 responses, 18 responded “Yes” and fifteen responded “No” resulting in 45.5% of respondents not believing that the RCPSC exam they wrote was objective and fair. This suggests a significant perception problem with the objectivity and fairness of RCPSC exams in the population sampled. This is concerning given that the Survey also reports that 34 of the 53 respondents or 64% of the sample reported passing their RCPSC exams, suggesting that respondents may not be simply complaining about exams they have failed. Again, caution should be used in interpreting these results given that the survey does not appear to have conducted any cross tabulations to determine how the respondents’ answers regarding whether the exam was objective and fair intersected with their having passed or failed their exam.

Another question asked in the Survey is, “When you think about the overall exam experience, do you believe it was fair and transparent?” Of 38 responses to this question, 18 responded “Yes” and 20 responded “No” resulting in 52.6% of respondents thinking the overall exam experience was not fair and transparent. As noted above, this data should be interpreted with caution given the methodological issues of the survey and the absence of reporting of any deeper data analysis. For perspective on the above data, while the MCCQE1 Candidate Survey in their 2020 Technical Report⁷ does not ask specifically for candidate’s experiences of exam fairness, there are several questions that are analogous and may offer points of comparison. The MCCQE1 Candidate survey asks respondents, “How would you rate your overall exam experience?^(p44). For the January 2021 examination cohort, 84% of candidates rated their exam experience as Good, Very Good, or Excellent.” The Candidate Survey also asks if, “The MCQ section provided an opportunity for me to demonstrate my level of medical knowledge.”^(p42) For the January 2021 examination cohort, 78% of candidates agreed or strongly agreed with this statement, with a further 18% neither agreeing nor disagreeing. Another question that speaks to MCCQE1 perceptions of fairness is, “The questions were clearly written.”^(p42) For the January 2021 examination cohort, 74% agreed or strongly agreed with this statement, with a further 17% neither agreeing nor disagreeing. Given the above MCCQE1 responses, reports that 52.6% of respondents do not think the overall RCPSC exam experience was fair and transparent appears to be a concerningly high percentage. It is also concerning that the RCPSC appears to conduct a candidate survey following examinations, but that they choose not to share the results of these surveys publicly resulting in a lack of transparency regarding how candidates view the various RCPSC examinations.

Respondents had an opportunity to provide comments regarding why they answered the above questions on the RCPSC Examination Experiences Survey the way they did. These qualitative responses are quoted below in Table 1,¹⁵ and provide some insight into areas of concern regarding the RCPSC exam process.

Table 1.

Qualitative survey comments

Feedback
“The questions were focused on minute details of subspecialties rather than general surgery questions that speak to safety to practice.”
“Some questions do not have a clear answer, do not reflect real clinical situations, no references found after the exam to support one particular answer.”
“I had some old exams but not the most recent couple years. The Canadians I met when I moved here had every question from every year which made the exam a cake walk for them.”
“Some questions were narrowly focused and based on single studies which at times appeared arbitrary.”
“The exam contained a lot of new data that is not relevant to a newly graduated non sub specialized surgeon. Also they asked about Hazard ratio. who memorize hazard ratio in trials!”
“Questions were very poorly written, in some cases clearly had been translated from French as the grammar was French (word order). Multiple answers were often correct and clinically inappropriate to choose one over another given that both interventions/treatments are critical e.g. fluids, epinephrine etc.”
“Many of the questions are apparently repeats that certain individuals and programs have access to for practice and comprises a majority of the questions on the exam. There were also many subspecialty questions not applicable to general practice”
“Canadian students/residents had access to questions well in advance and the ability to ask their supervisors (the ones writing the questions) what the answers were. These answers were shared around to Canadians. Speaking to colleagues who were CMGs and passed the exams well after the fact, they pretty uniformly say there’s no way anyone can pass that exam without the previous questions and specific coaching. One told me that even the course doesn’t really help (I took it), you need the past questions.”
“The RC do not provide the answers that are expected, so there is no way to know if they have made a mistake or not. We need to blindly rely on the knowledge content expert with no way to double checking. There is lack of transparency in their process to design the questions and no certainty that they use scientific references to design the questions.”
“No transparency at all. We do not know how the questions are constructed or reviewed.”
“Many exam questions focused on minutiae rarely encountered in actual practice, high degree of subjectivity with oral assessment.”
“The Canadian residents have access to a resource (past questions) that International students do not.”
“I have been told by numerous people who have taken the exam (CMGs and IMGs) that there is no way to pass the exam without having access to illegally shared databanks of remembered past exam questions, and that the Royal College recycles up to 80% of past exam questions, that they are overly detailed and lacking in clinical relevance and that simply studying will not be enough to pass. It feels like competing in the Olympics when it’s an open secret that everyone is taking steroids, and if you don’t take drugs you will not be able to compete/succeed.”
"Canadian trainees have the old exam questions and the exam is almost entirely old questions. If you don’t have some way to get them, it would be very difficult to pass the exam, almost impossible. Additionally, there are no standardized/accepted materials such as books or question banks to study from as are available in other countries. In the USA, there are books that are known to be the standard for studying, there are question banks provided by the board that are excellent preparation. The exam was poorly written and many questions focus on extremely subspecialty information that you would never manage

From the Table below, it is apparent that, at least based on the Survey sample, there is a candidate perception that RCPSC exams are not transparent, objective, or fair. Some of the apparent themes include a perception that a considerable number of questions are focused on minutiae rather than on assessing core competency for entering practice. From another Survey question, 55.5% of respondents did not believe the questions encountered on RCPSC exams were relevant and appropriate to assessing readiness to enter independent practice. Other issues include a perceived lack of transparency regarding how MCQ questions are created and, consistent with Prat’s comments, a perceived lack of transparency and candidate trust and confidence regarding how exams are scored and how potentially incorrect information in exam questions are dealt with by the RCPSC. According to the Survey, 17.2 % of respondents reported that 11% to 40% of questions contained factual errors, suggesting that concerns regarding test construction are a significant issue.

In contrast to the above candidate perceptions, the RCPSC describes a rigorous process for developing MCQ examination questions.¹⁷ However, despite these rigorous guidelines, the RCPSC does not transparently disclose any information from their psychometric analysis of concluded exams including information regarding number of questions discarded for statistical or psychometric reasons, a portion of which may have contained factual errors. It is not unusual that formal review processes for high-stakes exams do not include a review of factual errors in the questions asked. This appears to be the case for reconsiderations of the MCCQE1 which only conducts a reconsideration due to exam day incidents involving extenuating circumstances and procedural irregularities. Other high stakes exams in the United States such as the United States Medical Licensing Examination (USMLE) focus only on a recheck of scoring and does not include a manual review of the questions or candidate answers. In a Google Scholar scan of the literature pertaining to factual errors in other exams such as the MCCQE1 or the USMLE, I am unable to find any literature or research regarding concerns about factual errors in these exams. Given this lack of comparator data, there is no way to know whether the reports in this survey of between 11% and 40% of questions containing factual errors are significant or discrepant from candidate perceptions of other high stakes entry to practice exams.

Also of concern are the many qualitative comments suggesting that both CMGs and some IMGs have access to past exam questions and that access to these past exam questions provides an advantage to those in possession of these questions in achieving success on RCPSC Exams. While it is beyond the scope of this paper to verify this claim, it is worth noting that the RCPSC has a strict prohibition against sharing of copyrighted exam materials, with language stating,

During registration at the exam site, each candidate signs a statement confirming that they will respect the confidentiality of the exam. If this confidentiality is breached, exam results may be voided, and the Canadian licensing authorities will be informed.¹⁷

The RCPSC website also states,

Exam questions are protected by copyright and are the intellectual property of the Royal College of Physicians and Surgeons of Canada. Any reproduction or other disclosure of these exam questions, in whole or in part, is strictly prohibited. Our Royal College will take all available disciplinary measures and legal action against any candidate or others who violate this confidentiality provision.¹⁷

Despite the above prohibitions, of 37 responses from both CMGs and IMGs to the question, “Did you have access to past exam questions to help you study and prepare for the RCPSC exam?,” 27 responded “Yes” and 10 responded “No” with a total of 72.9% of those who answered indicating that they had access to past exam questions. Fourteen respondents answered this question by choosing “Not Applicable.” It appears that both CMGs and IMGs had access to past exam questions, although the qualitative comments quoted above suggest that there may be differential access to these past exam questions with IMGs having reduced access relative to CMGs. Unfortunately, the survey summary available online does not specify how many of these 72% were CMGs and how many were IMGs. This lack of a more in-depth analysis of data including cross tabulations between questions is a limitation of this survey.

Blew and colleagues report that on past RCPSC anesthesiology exams “Multiple-choice questions for each examination are a 50/50 mixture of banked questions that may have appeared previously on the written test and new questions submitted by anesthesiologists at large and edited by the Written Test Committee.”^18(p804) This paper dates from 2010 and hence this practice and the reported percentages for utilization of banked questions may have changed, however it is worth noting that on the SOCASMA RCPSC Examination Experiences Survey 53.8% of respondents reported finding that 31% or more of questions appeared to be from past exams. If it is true that a substantial number of RCPSC examination candidates have access to past exams, and in particular CMG candidates, this raises concerns regarding unfairness to IMGs, particularly IMGs who have not trained in Canada, in successfully completing RCPSC exams and gaining entry to practice. This would be entirely inconsistent with the principles of fairness outlined in the SEPT as they pertain to “fairness in access to the constructs as measured.”

Transparent, objective, impartial and fair: a legal standard for entry to practice

Eight Canadian provinces (British Columbia, Alberta, Saskatchewan, Manitoba, Ontario, New Brunswick, Nova Scotia, and Newfoundland and Labrador) have legislation that requires that regulatory Colleges assess qualifications for entry to practice in a manner that is transparent, objective, impartial and fair. Of these eight provinces, five (Alberta, Saskatchewan, Manitoba, Ontario and Newfoundland and Labrador) have language in fairness or other legislation that requires that if a health regulator relies on a third party to assess qualifications, it shall take reasonable measures to ensure that the third party makes the assessment in a way that is transparent, objective, impartial and fair. Given this legislative framework which recognizes the importance of an individual’s career and, given that most MRAs require RCPSC Certification as a condition of Registration, most MRAs have a duty to ensure that the RCPSC examination and certifications are transparent, objective, impartial and fair. It is of concern that the above discussion raises questions regarding the extent to which current RCPSC examination and certification processes meet this standard.

Transparent

As discussed above, the Prat article and the SOCASMA RCPSC Exam Experiences Survey both give rise to questions regarding the extent to which RCPSC exam processes are transparent. There appears to be a lack of public information regarding details of exam construction, validity, reliability, and outcomes; particularly outcomes related to different cohorts such as CMGs relative to IMGs. This lack of transparency is also reflected in concerns raised by Prat, and in the SOCASMA RCPSC Exam Experiences Survey regarding the creation and factual accuracy of examination questions and the level of transparency the RCPSC brings to this topic. While, as noted above, many organizations such as MCC who employ high stakes examinations do not permit formal reviews to focus on incorrect information in their examinations, a more transparent approach on the part of the RCPSC to issues of examination development, validity, reliability, and outcomes would likely have the effect of increasing public and candidate confidence in the RCPSC examination processes.

With respect to transparency, it appears that the RCPSC has much of the information the public might desire to ensure that RCPSC examination processes are objective and fair, however the RCPSC lack of transparency regarding this information means that the public does not have access to this data. As discussed above, the RCPSC document “Roles and Responsibilities of Examination Boards and Specialty Committees with Regards to Examinations” references many reports that the RCPSC compiles, including a psychometric exam report, a MCQ report, and a post exam survey report. Without access to these reports, it is impossible to know exactly what they contain. It is possible that having versions of these reports available to the public would enhance transparency and increase public confidence in the fairness of RCPSC examination processes. Instead, the current lack of transparency regarding RCPSC examination processes makes it extremely difficult to conduct a critical review of RCPSC examination processes and therefore almost impossible to accurately determine the fairness of RCPSC examination experiences.

Objective

Without transparency regarding examination construction, measures of validity, reliability, and outcomes, it is difficult to evaluate the objectivity of RCPSC examinations. Concerns have been raised by Prat and in the RCPSC Examination Experiences Survey results regarding the accuracy of some examination questions, as well as the relevance of these questions to entry to practice. In the interests of public confidence, it is important that transparent and publicly available validity and reliability research be undertaken to establish that RCPSC examinations measure factors important to establishing competence to enter independent practice, and not esoteric knowledge or minutiae unique to the Canadian medical context and culture. While an understanding of Canadian medical culture and context is important, undue focus on this creates bias and barriers which may be used to favor domestic trainees, whereas the intended purpose of the exam is reportedly to identify quality providers based on their entry to practice level knowledge of safe, effective practice in their specialty.

This point is made strongly in a recently released study of systemic discrimination experienced by international medical graduates.¹⁹ This study, using disaggregated data, notes that barriers experienced by IMGs include lack of transparency, and requirements that appear to be designed to ensure exclusion. This study also reports it is not unusual for an IMG doctor with five or ten years of experience as a doctor or a professor in their specialty to be equated to a recent Canadian graduate instead of being recognized for their expertise and extensive training. In the context of Canada’s current medical crisis and physician shortage where IMGs are seen by some, including in a recent Senate Report,²⁰ as a significant part of the solution to the physician shortage, a narrow focus on the Canadian context vs. competence to treat patients can create unnecessary barriers on physician recruitment and entry to practice. There are better ways to address any deficiencies in cultural knowledge than through entry to practice exams such as brief cultural competency workshops. It may be argued that an objective assessment of candidate competency should focus predominantly on medical skills and competence rather than on minutiae of Canadian knowledge or the nuances of Canadian medical practice.

Impartial

The concerns raised by qualitative comments in the RCPSC Exam Experiences Survey give rise to questions about the impartiality of current RCPSC examinations. From the comments in the SOCASMA RCPSC Exam Experiences Survey, it appears to be an open secret that past examination question banks exist, are critical to passing RCPSC examinations, and are available to both CMGs and IMGs studying and training in Canada, although they appear not to be available to all IMGs, nor to IMGs who trained outside Canada, and particularly Immigrant IMGs without Canadian connections. Given Blew and colleagues statement that in the past up to 50% of anesthesiology examination questions were reused, and the reports from the SOCASMA RCPSC Examination Experiences Survey that 31% or more of questions appeared to be from past exams, differences in access to banks of past examination questions would seem to have a high potential for creating differential outcomes for CMGs and IMGs. In the absence of transparent reporting by RCPSC of cohort differences in outcomes between CMGs and IMGs including differentiating between subgroups who did, and did not, train in Canada, it is unclear whether IMGs as a cohort, and particularly those without Canadian connections, are being treated impartially, or whether this is an issue that needs to be addressed to ensure an impartial and fair RCPSC examination process.

Fair

Again, without transparency regarding test construction, validity, reliability, and outcome reporting, it is difficult to establish whether the RCPSC examination process meets the standard of fairness. Certainly, the inequities identified regarding differential access to past examination questions for CMGs and IMGs raises questions regarding perceptions of fairness. These are issues that should be of concern to both the RCPSC and to MRAs who have the legal responsibility to ensure transparent, objective, impartial and fair registration processes, and fair access to the profession for all Canadians. Regardless of their place of education and place of training, Canadian citizens and permanent residents who have demonstrated medical competence should be entitled to fair access to the profession of medicine.²¹

Context: factors that may affect RCPSC ability to meet best practice standards

The RCPSC website lists thirty specialties and thirty-seven subspecialties that the RCPSC oversees.²² Each of these specialties and subspecialties likely requires the creation of a MCQ examination to assess candidates. This represents an incredibly broad spectrum of responsibility for the RCPSC in terms of examination development and credential recognition. In comparison, the MCC is responsible for the creation of only one high stakes MCQ exam, making it much easier for them to thoroughly research and establish the validity, reliability, and fairness of their MCCQE1 exam. In contrast to the RCPSC’s responsibility for some 67 specialty and subspecialty exams, in the United States, there are 24 certifying Boards organized under the American Board of Medical Specialties²³ with each Board responsible for the creation of their own specialty and subspecialty examinations. This is a much more achievable spectrum of responsibility.

Creation of objective, valid, reliable MCQ examinations is a resource intensive undertaking involving subject matter experts to create MCQ questions, oversight and exam security considerations, scoring, and psychometric analysis of examination results and score reporting. This process is an expensive one to undertake, and to some degree costs are passed on to certification candidates with estimated costs (not including examination preparation costs) of completing the RCPSC assessment and examination certification process of $8,130 for CMGs and $12,045 for IMGs.²⁴ Given these costs and the challenges of developing valid, reliable MCQ exams, it is possible that the RCPSC utilizes reused multiple choice questions in an effort to contain costs.

Regretfully, and based on the feedback from the SOCASMA RCPSC Exam Experiences Survey, reusing questions appears to have led to the creation of another problem, which is the use by both CMGs and IMGs of question banks made up of past exam questions that are still in circulation; contrary to RCPSC confidentiality provisions. Based on the RCPSC Exam Experiences Survey responses, there is a perception that CMGs have ready access to these past exams, while they are relatively unavailable to IMGs. This may be even more of an issue for immigrant IMGs who have fewer Canadian connections than Canadians studying medicine abroad (CSA) and are less familiar with Canadian culture and the nuances of the Canadian medical system. Again, as the RCPSC Exam Experiences Survey did not include an in-depth analysis of responses, it is not clear how many IMGs and CMGs had access to past exams and how this impacted their performance on exams.

Use of past sample questions is a common study tool, and most medical students use question banks as a primary study resource, however use of questions still in circulation raises serious ethical and fairness issues. Such a practice would be considered contrary to the rules of almost every standardized exam. Official question banks are made from old questions that are not still being used, although similar ones might appear. Accessing questions that are still in use as a study aid is highly problematic.

In the United States, standardized examinations such as the United States Medical Licensing Exam have commercial questions banks available from which candidates can study such as UWorld. Many United States specialty licensing Boards also have question banks available to candidates. In Canada, as noted above, the MCC has practice exams available for the MCCEQ1 and there are commercially available question banks such as the Canada Q Bank.⁹ It seems however, that the RCPSC has little available in the way of study resources such as practice questions or question banks for its various MCQ specialty and subspecialty exams. Further, there appears to be a paucity of commercially available question banks or study aids for RCPSC examinations. The RCPSC does provide candidates with a “Blueprint” or Exam Format document²⁵ as a study aid, but based on the RCPSC Exam Experiences Survey, many candidates are either unaware of the “Blueprint” documents, or do not find them helpful. Indeed, the RCPSC Exam Experiences Survey almost all respondents indicated they would find it helpful to have access to a bank of retired exam questions, and having access to exam preparation materials such as official question banks and review books was prominent among the answers to the question, “What other study resources would you find helpful to have available?”

Graduated licensing, practice ready assessments, and other approaches

Recently, Thoma and colleagues have discussed the merits of replacing high-stakes summative examinations with graduated medical licensure.²⁶ Thoma et al note that, “The current medical licensure practices in Canada depend on high-stakes, standardized, summative examinations that were developed to uphold the medical social contract to guard patient safety and benefit society. However, no evidence has shown that these exams contribute to this outcome.” They go on to state that, “the incredibly high pass rates of graduates from Canadian training programs suggest that this function is largely redundant.” Consistent with the discussion in the section above, they also remark that, “the exams remain painfully expensive for both the trainee and the regulatory body, at a time when student debt continues to soar.” They also note that, “The medical licensing process directs valuable and limited resources (e.g., time, energy, focus) toward passing an exam, potentially detracting from the more meaningful goal of preparing for independent practice.”

Instead, of high stakes exams, Thoma and colleagues suggest a graduated licensing system where, “specialty-specific examination boards would…be replaced with boards trained to review standardized practice audits.”²⁶ They further recommend that, “Credentialing and licensing authorities should advocate for the resources required to replace the current high-stakes summative assessments with graduated licensure; to develop quantifiable, nationally synergized, specialty-specific practice standards that support both the oversight of graduated licensure and maintenance of competence; and to streamline licensure requirements between jurisdictions.”

A paradigmatic shift in credentialing such as that suggested by Thoma and colleagues would offer a number of benefits to organizations such as the RCPSC that appear to be facing challenges in meeting best practice standards for transparency in examination construction, validity, reliability, and fairness. An approach consistent with that advocated by Thoma and colleagues would be consistent with the principles of Competence by Design developed by the RCPSC in post graduate medical education programs for CMGs. With regard to IMGs, the RCPSC is already developing alternative streamlined methods of assessing candidate competence such as Practice Ready Assessment and their Practice Eligibility Route.²⁷ Expanding these pathways for IMGs and phasing out problematic high-stakes examinations may help to reduce current perceived barriers IMGs face in obtaining RCPSC Certification. Current examination practices may be affected by fairness issues for internationally trained candidates who come from culturally diverse backgrounds that the current RCPSC examination process may not accommodate in a fair and equitable manner. Establishing these new pathways could also assist in bringing the RCPSC into alignment with many provincial MRAs who are actively seeking to reduce barriers for IMGs so that they can contribute to solving Canada’s current health care crisis.

Conclusion

The RCPSC is an institution that has accomplished much in 95 years and has much more to offer. It plays a critical role in developing national competency standards to achieve its vision of advancing learning for specialist physicians to deliver the best health care for all. However, a critical analysis of its examination and Certification practices raises questions as to whether the RCPSC is meeting best practice standards with respect to its broad mandate and usage of high-stakes examinations. The RCPSC’s lack of transparency regarding its examination processes and the inaccessibility of data related to reliability and validity runs the risk of generating mistrust on the part of examination candidates, the public, and the medical profession regarding the fairness and objectivity of the RCPSC’s examination processes.

It may be time for the RCPSC and the MRA’s who delegate the credentialing process to the RCPSC to reflect upon RCPSC’s role as a credentialing organization and more closely scrutinize RCPSC examination processes to ensure they meet best practices and legislated standards of being transparent, objective, impartial and fair. RCPSC, as an independent non-profit corporation, has no statutory duties or obligations, but as discussed above, the provincial MRAs do have a legal duty to ensure that third party assessors such as the RCPSC conduct their assessment in a way that meets fairness standards. If third party assessors fail to do so, these third parties such as the RCPSC must either be encouraged to change their practices, or MRAs need to find other methods of assessment that meet legislative standards of fairness. Such other methods might involve bringing specialty assessment in house with MRAs doing their own assessments of specialty qualifications.

Particularly in a changing environment marked by global mobility, RCPSC's traditional exam-based approach to certification may have become outdated. The exam process is both resource intensive in terms of creation and validation of exams and is increasingly time-consuming and expensive for both the RCPSC and candidates in an environment of fiscal restraint. Perhaps, particularly as Competence Based medical education is established for Canadian medical graduates, and as we recognize the validity of various internationally recognized approaches to medical training that result in competent practitioners in other countries, it is time for the RCPSC to move toward alternate methods of assessment suggested by other authors. Such approaches may include graduated medical licenses, Practice Ready Assessments, and maybe a portfolio approach to assessment of candidate credentials and competence in conjunction with periods of supervised practice.

Funding Statement

Funding: The author has no financial affiliations to declare.

Conflicts of Interest

The Author is a volunteer with SOCASMA, providing strategic guidance on topics related to this paper. He has engaged in dialogue with the RCPSC, the subject of this paper, regarding issues raised herein, and was part of the SOCASMA team that conducted the survey involving examination experiences referenced in this paper.

Edited by

Marcel D’Eon (editor-in-chief)