This article refers to the examinations formerly known as Membership of the Faculty of Accident and Emergency Medicine and Fellowship of the Faculty of Accident and Emergency Medicine. After the change in name of the Faculty of Accident and Emergency Medicine to College of Emergency Medicine, these examinations are now named Membership of the College of Emergency Medicine (MCEM) and Faculty of the College of Emergency Medicine (FCEM), respectively.
Purpose of the examinations
Part of the role of the National Health Service is in training the next generation of staff. This training facilitates young and inexperienced doctors in gaining knowledge and developing specific skills while delivering care under the supervision of senior doctors. Such in‐service training at a higher training level gives the doctor specialty‐specific skills, which he or she will in turn pass on to junior doctors. Thus, the specialty perpetuates its own expertise and identity. Training has to occur while continuing to provide care to patients and, in emergency medicine, in an often difficult and stressful environment. Workplace‐based assessment of a trainee actually “doing” what we want them to be able to do is important, and there is increasing attention to developing suitable methods in all specialties. However, the practicalities of such assessments and the work required to train the assessors and standardise techniques means that formal summative assessment outside the workplace remains a core part of assessment in medical education.
While recognising that the speed of progression of the trainee doctor from novice to expert is variable, the way in which career paths and the workforce are organised means that certain levels of autonomy are expected at particular levels in training (senior house officer, registrar). Doctors who are working at a level of autonomy higher than their capability present a considerable risk to patient safety. In emergency medicine, there are two major transitions in training: transition from immediate post‐registration working to semi‐autonomous working, with a limited supervisory role (the transition from senior house officer to specialist registrar (SpR)), and transition from SpR to the fully autonomous practitioner, with ultimate responsibility for delivery of care, as well as degrees of management, training and research responsibilities (SpR to consultant).
The College has developed two major examinations within the last 10 years—the Membership Examination (MCEM) and the Fellowship Examination (FCEM). The explicit purpose of any summative examination is to certify competence to proceed to the next level of practice. However, there are other less explicit but just as relevant effects of examinations. Learning and assessment are part of the same educational process, so that the inclusion of a topic or skill in any examination immediately alerts the candidate that the topic is important to know about. Candidates will prepare from past papers (official or unofficial black market created from the memory of previous candidates) and therefore concentrate on topics that have come up before. The examination must therefore reflect normal emergency medicine practice. In addition, any new specialty examination, particularly where we have had to rely on examinations of other colleges as our traditional standard, signifies a coming of age of the specialty. Acceptance by the Specialist Training Authority of both examinations as appropriate methods of assessment for our specialty was essential in the process of development. Lastly, the standard set must be such that candidates believe that they have successfully negotiated a tough goal, and that this examination is an important achievement in their career.
Content of the examinations
So what should the examination test? A traditional sequence in assessment design would be the construction of a specialty curriculum, followed by an assessment to match the curriculum contents. However, the desire to have our own certification methods, and the recognition that a curriculum is a living document that may take years to mature, was the driver behind undertaking the project in reverse order. Knowing the purpose of the examination could allow the college to at least set out the skills to be tested at each level. Thus, the potential SpR could be expected to have a broad clinical knowledge base, as well as some minimal practical skills. Imagine what list of conditions and skills necessary for practise can be drawn up by an SpR on his or her first night on call in the emergency department. At this junior level, there is little requirement of the SpR to have management skills (which will be gained during training) or research skills. However, the SpR wishing to become a consultant must be able to fulfil the role in its entirety, possessing management, teaching, research and interpersonal skills, apart from being a clinical expert.
As the examinations have developed, the designers have used a blueprint or matrix system. This allows the designer to ensure that in one examination paper, all relevant parts of emergency medicine are covered. For example, the MCEM Part A blueprint will ensure questions on anatomy, physiology and pathology in all types of conditions—medical, surgical, gynaecological, etc—are included. Similarly, for Part B, the short‐answer questions (SAQs) and Objective Structure Clinical Examination (OSCE) stations are commissioned together using a common blueprint to ensure that the breadth of the emergency medicine case mix is encompassed and that duplication is minimal. For example, having an OSCE station on thrombolysis and a SAQ on interpretation of an electrocardiogram and management of myocardial infarction would be unnecessary, and may prevent other important topics from being dealt with due to constraints of paper and OSCE length.
For the Fellowship, recognition of the importance of academic ability and management skills has meant inclusion of these as areas within the final assessment. Recent developments in this area include the increasing use of blueprints and more standardised objective marking sheets to allow additional feedback. At the same time, the examination has evolved to become more clinically focused, recognising the absolute requirement to show the clinical competence reflecting the importance of consultant clinical presence in our departments. This emphasis on demonstrable clinical skills is an important signal to the specialty and to the National Health Service as consultant revalidation is introduced. In addition, it shows the commitment of the College to patient care and to developing its own specialty, with unique skills in emergency assessment and treatment.
Selection of assessment methods
Having determined what should be tested, the selection of appropriate methods is crucial to allow the proper conditions for assessing what we want to assess. Clinical skills and verbal interpersonal communication skills cannot be tested on paper, and multiple‐choice questions are difficult to construct if you want to test problem‐solving skills. Analytical skills can be tested by oral examination or by written examination, but objectivity and reliability of marking can be difficult to achieve. A combination of methods is therefore appropriate to test a variety of aspects of competence; hence, each examination comprises several parts.
Emergency medicine as a specialty focuses on patients with acute conditions. By definition therefore, practical problems are inherent in designing a clinical examination to reflect “real life”. The traditional “long case” with an acute patient would not only raise ethical problems but also preclude appropriate planning, as specific patients would not be guaranteed to be available on the day. In addition, trying to find patients who present the same degree of complexity to the candidate may be difficult owing to the number of candidates, particularly at the MCEM level (144 candidates at one sitting). The assessment method selected for clinical skills assessment is therefore the OSCE, which is used increasingly in postgraduate examinations and is familiar to undergraduates. This uses only stable patients and professional role players. This is complemented by the short‐answer or modified essay question examination testing a broad range of knowledge, use of investigations, decision making, and problem analysis and solving. The OSCE format is time and resources heavy, particularly at the Fellowship level, where each station has two examiners. At membership, the quality control is maintained by double markers who randomly mark a proportion of stations along with the examiner, thus ensuring that “hawks” and doves” are identified. The OSCE stations are designed to reflect real life as much as possible, notwithstanding the necessity of using manikins in some cases. Each station has extensive “scripts” written for the role players, nurse assistants and examiners to ensure that each time a candidate performs on the station, the conditions are exactly the same. Examiner fatigue is a potential problem and having one examiner for all candidates in an examination is ideal, but after the 100th candidate at MCEM it can be difficult to concentrate.
Setting a standard
Both examinations have small groups responsible for the design, delivery and standardisation of the examination (Part A group, Clinical group, Management group, Academic group). These groups have evolved over the years with some changes in membership, but one of the key functions is to set a standard. Each group comprises several consultants who have widely differing backgrounds. This ensures that the standard set reflects common practice, not the expert with a particular interest whose ideal may be exemplary but unachievable in normal practice. Increasingly, these groups commission questions and stations from other consultants to ensure that this common standard is reached and that the examination truly reflects what happens in emergency departments.
Having set the questions and stations, the group then reviews the paper to identify ambiguous or poorly worded questions and to set a standard. The standard setting is based on the Anghoff method, whereby the examiners independently determine what the minimally competent candidate would achieve and then agree with discussion on a final score. This process may take a day for each examination but ensures that the pass mark is the criterion referenced (ie, the candidates achieve a specified standard, not merely compared with each other) and set before the examination. Thus, any discussion or change of a pass mark during the examination is only as a result of technical difficulties with conducting the examination.
In all sections of the examination, standardised answers are prepared before the examination, increasingly using marking sheets that give clear instructions to examiners as to what is an acceptable answer. This is a change in the FCEM viva sections particularly, and is designed to allow transparency in the results and enhanced feedback to the candidate.
Quality control
The clinical examination group (looking at the SAQ and OSCE for both MCEM and FCEM) often uses the same station or question stem for both examinations, with a higher pass mark set at the FCEM level. This allows the College to have internal quality control over projected levels of competence, and to compare performance at both levels to ensure the progress of candidates. This in turn feeds information to the training programmes about where the training should be focused. For example, an evaluation of results in both the examinations suggests that basic clinical skills (examination of systems, history taking) are in general poorly carried out. Training committees are given this information, and trainees then have the opportunity of focusing on reviewing these skills and obtaining local feedback on their performance.
Recently purchased software will allow us to scan results and build up a historical database of performance that allows rogue stations or questions to be identified. Each sitting of an examination contains both new and reused questions; this allows us to compare cohorts of candidates and ensure that the standard is maintained. The risk of reusing questions is that candidates preparing from “past papers” will be able to focus on that topic and therefore perform better. As a rough rule therefore, we assume that performance on a given question or station should always be improving.
The predictive validity of an examination (ie, whether it predicts the candidate who will perform in real life at the required level) is more difficult to determine. Detailed information on significant numbers is not yet available, but there is to date no evidence of candidates encountering serious problems in their practice after success in either examination. Thus far, feedback from candidates for MCEM is that it tests what they do as a registrar—content validity. Fellowship candidates have been more varied in feedback, and the examination has undergone so much change recently that objective reliable evaluation of the examination is difficult. The College is commissioning several pieces of work that will look at these aspects over the next few months, with such information being made available.
Evaluating the reliability of the examinations (ie, whether the same people would pass on a different day) is also difficult. There are candidates for MCEM who pass one station one year but fail the next; this is to be expected when there are so many stations. In real life, the case is not always exactly the same between one patient and another. In general terms, and in the experience of the Fellowship examination, candidates who pass one section usually pass it the next time. It is the clinical section that most candidates have difficulty with, and candidates often fail this repeatedly. The College is aware that the examination must discriminate between the candidate who has good “exam skills” and the candidate who genuinely is competent and “does do” in the workplace. The design of the objective marking scheme is key, as it must not focus on trivia but on observable essential steps in the skill. In addition, the revised regulations and guidance to candidates give more information on what to expect and how to prepare, thus encouraging candidates to access trainers to observe them and give feedback before the examination. In the OSCE, long checklists of individual steps are being replaced by more generic descriptions of a group of related tasks, thus both reflecting the higher level of performance and also allowing the examiner more time to observe the candidate. The College has developed examiner workshops that allow examiners to discuss aspects of examining—the difficult candidate, the ambiguous answer, as well as practising examining in low‐stakes situations. Thus, examiners will be “fit for purpose”, having agreed as a group what an acceptable standard is and knowing how to recognise that standard in a stressed candidate by using the simplified mark sheets.
Future work
Cost effectiveness in these examinations is important; any increase in cost will be passed on to the candidate (although Fellows and Members by subscription subsidise the examinations in part). Each part of the examination must contribute to the body of knowledge about any candidate's competence; else, this part of the examination must be changed. The College is looking at the feasibility of removing the academic section of FCEM and replacing it with a requirement to complete a critical appraisal course (with assessment) taken within the training period, with the publication of a peer‐reviewed paper (similar to the Fellowship of the Australasian College of Emergency Medicine). Similarly, the demonstration of management skills may be better tested as project work recorded as part of the portfolio, rather than as a viva, in addition to several OSCE stations with a management focus. These might be, for example, an interview with the business manager or a complaints meeting with an unhappy patient. These two initiatives may allow the College to further enhance the reliability and validity of the FCEM clinical section by increasing the time available for assessment.
Additional work is also planned to develop the content synergistically with the curriculum so that changes in practice in emergency medicine are reflected in the curriculum and in the examination. In addition, there will be close interaction with the training committee, so that the results of the examination inform training, and development in work‐based assessment complements the content of the summative assessment.
Summary
The current College examinations are judged to be “fit for purpose”. They reflect what we want the candidate to be able to do in real life, provide the proper conditions for assessing what we want to assess, and are practical to conduct with the time and resources available. The examinations make a statement on what the definition of competence is for our specialty and define a high standard to deliver good‐quality clinical care.
Abbreviations
FCEM - Faculty of the College of Emergency Medicine
MCEM - Membership of the College of Emergency Medicine
OSCE - Objective Structure Clinical Examination
SAQ - short‐answer question
SpR - specialist registrar
Footnotes
Competing interests: None declared.