Restarting training and examinations in the era of COVID-19: a perspective from the Federation of Royal Colleges of Physicians UK

ABSTRACT

COVID-19 has proven to be a potent disruptor of postgraduate training, assessment and learning. In so doing, it has equally proved to be a potent catalyst and has driven innovation. Here we discuss the response of the Federation of the three UK Royal Colleges of Physicians to the challenges presented in these areas by the COVID-19 pandemic.

Introduction

The contribution of our trainees to the fight against the COVID-19 pandemic has been immense. This has been a very difficult time for them physically, mentally and emotionally. Now, as the system starts to revert to a new ‘normal,’ their anxieties about training and assessment are also increasing: just as COVID-19 has disrupted service delivery, it has also disrupted postgraduate medical education, training and assessment.¹ The pandemic forced us to meet and overcome many challenges and to find novel ways of delivering services. Many of these changes are now embedded and it is unlikely that we will revert to our previous ways of working. This will challenge how we train and educate but it also provides us with a once-in-a-generation opportunity to modify training for the future.

At the height of the pandemic, there was less time for training and, in addition, clinical experiences changed. The usual rotations and training placements appropriate to each specialty were interrupted.² Workplace-based assessments (WPBAs) and supervised learning events (SLEs), normally carried out on the wards, in clinics and in other settings, were disrupted as the wards filled with COVID-19 patients and many hospitals cancelled face-to-face clinics, relying on greater use of virtual consultations via new platforms such as ‘Attend Anywhere’. Access to procedures in the craft specialties was, and remains, curtailed.³ Mandatory training courses have been cancelled and conferences and examinations run by most organisations have been temporarily suspended. It is no longer possible for examination candidates to congregate in examination halls or rooms and clinical examinations pose a risk to patients, surrogates, administrative staff, candidates and examiners alike.

Some principles

A lot of work has been, and is being, carried out at national, regional and local levels in the face of the above. The guiding principle for this work has been, as far as possible, to support our trainees and to maintain their training and career progression. Key aspects of this approach are the following:

• Facilitating trainee progression wherever possible.

• Reducing the burden of evidence collection for training.

• Carrying out assessments with fewer people.

• Focusing on those trainees who are at critical progression points in their training, who need reasonable adjustment or who are shielding.

• Recognising valid alternative experiences gained during the time of the pandemic.

• Maintaining equity in how trainees are treated across all specialties, including non-physician specialties.

• Adopting a ‘light touch’ approach.

It is vital that the entry of new trainees into training, and progression to CCT, are both maintained in order to secure the future GP and consultant workforce.

Training

Recruitment

When the COVID-19 pandemic arrived in the UK, most ST3 interviews had not yet started, some were part done and only a few were fully completed. Responsibility for recruitment lies with the Four Nation Statutory Education Bodies (SEBs) via the Medical and Dental Recruitment and Selection (MDRS) subgroup. It does not lie with JRCPTB or the colleges. The SEBs decided to cancel all further face-to-face interviews due to run under the auspices of the Physician Specialty Recruitment Office (PSRO). After consultation with junior doctor representatives, including the BMA Junior Doctors' Committee and the Academy of Medical Royal Colleges (AoMRC) trainees' committee, it was decided to introduce a new recruitment process based on self-assessment in four domains which would be scored and used to rank applicants: quality improvement, teaching experience, training in teaching and presentations. For ST3 recruitment, the MRCP(UK) examination would also be included.⁴

Publications, prizes, intercalated degrees and other academic achievements were considered but were not included as they had performed less well at predicting interview success in previous recruitment rounds. As they are likely to perform better for academic recruitment, concerns were raised in the academic community. It is likely that the forthcoming second round of recruitment will be modified and will include a face-to-face element in some form. The three physician colleges, Federation and JRCPTB strongly support endeavours to maintain and enhance academic recruitment going forward.

A new process for the Annual Review of Competency Progression (ARCP)

The SEBs decided to reduce both the evidence required from trainees and the number of ARCP panel members. The Educational Supervisor's Report (ESR) would be the key evidence and JRCPTB produced minimum data sets required for each physician specialty and decision aids for progression and non-progression in each year of each specialty.^5,6

New ARCP outcomes were also introduced to allow trainee progression to the next stage of training.^7,8 Where acquisition of some capabilities was delayed due to the pandemic, but trainees were not at a critical decision point and were otherwise making satisfactory progress, an Outcome 10.1 was awarded. The ARCP panel recorded the missing components of training and plans were to be made within the next placement for how the ‘missing’ capabilities were to be acquired. The next ARCP panel would review progress and determine whether any additional training time would be needed. Trainees at the end of core medical training (CMT) who had not passed PACES and/or Part 2 written MRCP(UK) were given a derogation to allow them to enter ST3, if successful in recruitment. However, they would have to complete the full Diploma by the end of ST3 in order to progress into ST4. Those nearing the end of the 7-year limit for obtaining all three parts of the MRCP(UK) Diploma were given a dispensation to allow them an extra year to pass it. Trainees at the end of higher specialty training who had not passed the relevant specialty certificate examination (SCE) could not, however, be awarded a certificate of completion of training (CCT) and were awarded an outcome 10.2. If no ARCP had taken place, an outcome N13 was applied.

Practice

COVID-19 demonstrated the need to have a workforce that is more broadly trained in general specialty areas and that is therefore more flexible. It also saw a reversion, in some places, to a modified form of the old firm structure, which did have some benefits for trainees and for training. The recently introduced new Internal Medicine curriculum has already started the process of strengthening generalist training in the physician specialties.

Future directions in training

COVID-19 has changed many aspects of practice, and many of these changes will endure. As examples, many/most outpatient consultations are now done remotely and, in the craft specialties, fewer procedures are being undertaken.³

Both of these changes have profound implications for trainee access, learning and experience. In addition, COVID-19 has highlighted the need to better address some other areas of training, including relevant aspects of infectious diseases, epidemiology, public health, health inequalities and health promotion.⁹

In parallel with the work being done to restart the service, JRCPTB is consulting all of its 30 Specialty Advisory Committees (SACs) to determine how the pandemic has affected them and how they plan to change training going forward. While this work is still ongoing, it has already resulted in a number of recommendations, some of which are general and applicable to all specialties (see Box 1).

Box 1.

Training recommendations for all specialties

Support: consider what support trainees will require in the future. Rotas: avoid ‘see-sawing’ trainees between specialty training and Covid-19 specific rotas. CREST: review the support for trainees who have not undertaken a relevant UK equivalent core training programme and who will therefore be entering training using the Certificate of Readiness for Specialty training (CREST) or, in the future, the new ‘alternative certificate’ which is to replace it. It is also important that the process for deciding on placement of trainees within localities is more transparent and doesn't disadvantage protected groups, particularly Black, Asian and minority ethnic (BAME) trainees and international medical graduates. Virtual outpatient clinics: consider how best to provide the skills required for virtual clinics. In addition, the nature of trainer supervision will be different and a new assessment tool for such clinics may be required. Trainers should consider how such clinics should be organised, including having dedicated training clinics, how trainers can ‘vet’ patients and how to organise pre- and post-clinic debriefing and discussion. Clinical reasoning: ensure that there is an increased focus on training in clinical problem solving and critical thinking. This will benefit both patients and the service since it will help to ensure that future consultants maximise these skills to focus investigation and promote efficient use of resources. Provide guidance: re-emphasise to trainers the importance of providing guidance and provide further training for them on the specific components of the new curricula that are being developed so that they are better able to do this Courses: review recommended courses and justify their necessity, bearing in mind equity of access across the four nations.

Training in certain specialties has been particularly badly affected. COVID-19 has had a devastating effect on procedures. For instance, data from the National Endoscopy Database indicate that total endoscopic activity fell to 5% of normal at the height of the pandemic, 1,700 procedures per week instead of 35,000.³ Other ‘craft specialties’ that have a specific emphasis on practical procedures, including cardiology, respiratory medicine, rheumatology and dermatology, have been similarly affected. In addition to the effect on service provision, this has had a profound effect on access to procedures for training purposes.

The SACs have recommended that specialty societies and the service should explore how to facilitate acquisition of procedural capabilities in the COVID-19 era when trainee exposure to procedures is reduced, and made the following specific suggestions:

• Simulation: training will require improved simulation and improved and more equitable availability of simulation and its more widespread use, particularly (but not exclusively) early in training, including before exposure to patients.

• Aptitude: tailor procedural training better to suit the individual trainee's aptitude including a greater focus on capability rather than simply numbers.

• Repository: consider developing a national repository of clinical images/videos that are acquired from endoscopic, and potentially other, procedures including cardiac procedures.

Other possible approaches include SACs and specialty societies exploring which procedures in their specialty are mandatory, bearing in mind that, in the future, not all consultants may need to be competent in all procedures, and investigating the feasibility of having dedicated training lists, recognising that this may be logistically challenging to deliver. There has also been some discussion about the development of boot camps or academies for procedural training and experience.

Examinations

MRCP(UK) is the division of Federation responsible for examinations in the physician specialties. It runs 14 examinations: the three parts of the MRCP(UK) Diploma, required to successfully complete CMT/internal medicine stage 1 training (IMS1), and 11 Specialty Certificate Examinations, the knowledge-based assessments required to successfully complete higher specialty training in 11 of the physician specialties. The MRCP(UK) exam assesses not just knowledge, but also capabilities, which include attitudes and behaviours. In 2019 MRCP(UK) examined 27,000 candidates in over 50 countries. Therefore, the decision taken to cancel all examinations from Monday 16 March until September 2020 has had far-reaching consequences. Federation tried hard to maintain some of its exams but, in the end, this proved impossible.

MRCP(UK) has been planning for the reopening of exams in September ever since they were cancelled. There are a number of challenges to overcome. Examinations will need to be implemented with heightened social distancing (SD) and infection control (IC) and current venues may not be suitable; finding alternatives will be hampered by estates already being under pressure from COVID-19-related demands. Clinician examiners and other senior clinicians are likely to have less availability for examination-related duties. Notwithstanding these resourcing issues, the forthcoming exam diets will have to cater not only for candidates who would normally be taking the exams at this time, but for the additional numbers who have been prevented from taking the exams due to cancellations; this will require a considerable increase in capacity.

There needs to be recognition that many trainees may be coming to terms with the physical and mental repercussions of the pandemic. They may also not have been able to study or have had the clinical experiences that would normally be part of exam preparation. Candidates at critical points in training or who have special needs will need to be identified and given priority.

The uncertainty of the situation presents an overarching challenge: all plans made will need to set out a hierarchy of contingencies in case of further waves and lockdown.

Restarting MRCP(UK) Part 1 and Part 2 written examinations

These exams are not currently computer-delivered. To minimise potential disruption from short-notice changes to government COVID-19 guidance, MRCP(UK) is planning to have two dates 6 weeks apart for each diet of each written MRCP(UK) paper, such that if one date has to be cancelled there is an alternative date already prepared for. It is also identifying venues to accommodate increased candidate numbers and is developing new IC and SD procedures to minimise the risk to all those involved. MRCP(UK) is also liaising with JRCPTB to identify trainees for whom obtaining the Part 2 written exam is crucial for career progression.

Restarting specialty certificate exams

These 11 examinations are delivered by computer-based testing using a UK based partner. MRCP(UK) is negotiating to secure increased capacity where necessary. MRCP(UK) has managed to secure that all 10 of these exams yet to take place will now run by the end of the year unless there are significant unpredicted further surges of COVID-19. Particular attention has been given to liaising with JRCPTB to identify and prioritise those who need the SCE to progress to CCT in 2020/2021.

Restarting PACES

It has already been shown that it is possible to run clinical examinations successfully during the COVID-19 pandemic. One model used the principle of cohorting of all participants.¹⁰

To reduce the risk of COVID-19 transmission, a new model for the PACES carousel will be introduced. At present the plan is to have two mini-cycles, instead of a five-station carousel. Stations 1 (respiratory, abdomen, 10 mins each), 3 (cardiology, neurology, 10 mins each) and 5 (two brief clinical consultations, 10 mins each), the so-called ‘clinical stations’, will form one mini-cycle with reduced patient numbers, one for each body system, and use of surrogates will be permitted in station 5. Increased time will be allowed for donning and doffing PPE. Stations 2 (history taking, 20 mins) and 4 (communications skills and ethics, 20 mins), the so-called ‘talking stations’, will be combined with a 20 min rest station to form the second mini-cycle. This second cycle will be delivered remotely on the same day, with the candidate in the examination centre and the surrogate/actor and examiners remote in different places. An online platform will be used, the same for all centres, together with high-quality webcams, microphones and speakers, with appropriate infection control. The candidates will not be expected to set up any of the equipment. To ensure consistency, a small pool of specifically trained examiners will be used.

There are potentially greater problems with venues, capacity and examiner availability than for the written exams. Non-NHS footprints will be preferred but some will have to be used and it will be necessary to avoid clinical areas and to define what areas of NHS estate can and cannot be used. The new model will require significant examiner and candidate training and communications to ensure that the validity, reliability and standard are not compromised.

MRCP(UK) is also drawing up a hierarchy of contingencies to allow for further modification to the above delivery models in the case that the pandemic worsens again and disrupts the above plans.

For the international PACES exams, the current carousel will be delivered, respecting local guidance on infection control and social distancing. The norm has been to use five UK examiners for each international centre, each paired with a local examiner. To allow for reduced UK examiner availability, however, the Clinical Examining Board (CEB) of MRCP(UK) has agreed a reduced examiner model of two UK examiners for mature, experienced international centres, and three UK examiners for less experienced centres.

Prior to COVID-19 it had been planned to introduce the new PACES carousel, known as ‘PACES 2020’, in diet 3 of this year. This has to be launched simultaneously domestically and internationally and so this will not now be possible. MRCP(UK) is drawing up contingency plans to introduce it in 2022 and to consider whether any further changes to it are necessary.

Future directions in the delivery of written exams

For some time, MRCP(UK) has been considering other ways of delivering its exams, particularly the written papers. It is now bringing forward plans to look at the possibility of doing this.

Online exams

The SCEs are already computer-delivered and, prior to COVID-19, MRCP(UK) was considering investigating whether and how quickly it could deliver the Part 1 and Part 2 written MRCP(UK) exams via computer-based testing. One particular issue is scale. The MRCP(UK) diploma is a huge exam in terms of candidate numbers which does present problems, not least in terms of finding a delivery partner able to provide for this size of candidate cohort three times per year and not just in the UK but also internationally. However, the pandemic has proven to be a potent catalyst, and Federation/MRCP(UK) is now planning to undertake pilots not merely of computer-based testing but of fully online exams.

MRCP(UK) is planning a pilot of a fully online exam for domestic delivery of both the Part 1 Written and the Part 2 Written exams this autumn, for up to 500 candidates in each exam, using remote proctoring (invigilation). This could eventually allow candidates to take the exam at a time and place of their own choosing. It could also facilitate a move towards other question styles, including very short answer; MRCP(UK) and the SCEs are all currently best-of-five multiple-choice question exams. The advantages of online delivery include:

• No travel required for candidates or invigilators.

• No limit to candidate numbers due to venue size or social distancing rules.

• No artificial exam diet restrictions; the exams could be rolled out continuously and taken whenever the candidate wishes.

• No disruption from travel constraints due to weather, war, politics – or viruses.

The main concern is exam security, both in terms of ensuring that the correct person is taking the exam and that they do not cheat. However, commercial organisations use remote invigilation which offers:

• pre-checks to ensure the candidate's system is properly and securely set up

• remote candidate authentication

• securing of the candidate's environment

• candidate supervision, which may be live or recorded; the former offers a lower invigilator-to-candidate ratio than could be provided in an exam hall (1:3–1:6) and can make use of biometric methods such as tracking eye movements and keystrokes

• reporting of any suspicious behaviour or infringement of rules pre-agreed with the exam body. Indeed, with live invigilation warnings can be delivered and candidates can be excluded if necessary.

The other main concern is IT failure. However, companies delivering online exams are able, to some extent, to mitigate for this. In addition, MRCP(UK) is developing alterations to its regulations to cope with this.

For any high-stakes examination it is always necessary to consider the effect of changes on exam validity, reliability, standard setting, reasonable adjustment and equality and diversity. Any change will have to be approved by the UK regulator, the General Medical Council (which has introduced a new ‘fast track’ process for the COVID-19 era) and by the regulators in other countries where the MRCP(UK) Diploma is a formal part of training. It will be essential to have a sophisticated communications package to inform prospective trainees and their trainers of the changes and to train them in what is expected.

If these pilots are successful, MRCP(UK) will be looking at incrementally increasing its online delivery offer to the point at which the entire domestic Part 1 and Part 2 Written exams can be undertaken in this manner. Following this, it will need to look at the possibility of doing this internationally, although there are concerns about internet availability and performance in some countries, meaning that universal online delivery may not be possible for all.

Open-book exams

Online exams delivery might also catalyse a shift, if desired, to ‘open book’ exams in which candidates are allowed access to resource materials.¹¹ This is thought by some to better simulate what candidates would face in real clinical practice and therefore to provide more meaningful/authentic assessment of their abilities.

Disadvantages of open book exams include the following:

• Different question styles may be required.

• Candidates spend less time preparing.

• Answering exam questions takes longer; this could impact reliability if a variable number of questions is asked between exam diets.

• Retention by the candidates of tested material may be shorter.

• Public perception; many regulators favour a proctored exam.

Very short answer questions (VSAs)

Single ‘best-of-five’ questions have drawbacks: they depend on having four plausible distractors and they are subject to ‘cueing.’ Moreover, clinical problem solving is often nuanced and depends on information gathering, initially though the history and examination. This is not assessed by the single best-of-five format. Hence there is interest in alternatives. Some universities have been using VSAs for some time¹² and their performance in medical education and training is being investigated.¹³

Future directions in the delivery of PACES

There is no doubt that the format of the clinical exam makes delivering it in a virtual environment much more challenging. There are a number of unique aspects of the PACES exam that contribute to its worldwide reputation as being of high quality:

• It tests seven clinical skills – ability to carry out a physical examination, ability to identify physical signs, clinical communication, differential diagnosis, clinical judgement, managing patients' concerns and maintaining patient welfare.

• All are of course important, but it is the second skill – identifying physical signs – that is the most discriminating of all. Although, during COVID-19, physicians used their stethoscopes less, they depended more on investigations, including CT scans, demonstrating how important proper use of highly developed clinical skills is to rational and economic use of expensive resources with potential side effects.

• There are two examiners for each of the five stations, at least one of whom is usually external to the centre, and they mark independently.

• The two examiners ‘calibrate’ the case – that is, they agree which areas of performance are necessary to pass and which will result in a fail for that station.

Alternative ways of assessing bedside clinical skills are currently being investigated by MRCP(UK).

Continuing professional development

Federation delivers and supports the CPD system (comprising the CPD diary, CPD app and an approvals system for CPD events) for consultant physicians, SAS (staff and associate specialists) doctors and physician associates. It recognised early on that the COVID-19 pandemic would result in the cancellation of many CPD events and that, in addition, clinicians might not have the time to undertake formal CPD activities. It therefore removed all numerical CPD credit targets for this year and next and issued a statement to this effect that could be shown to appraisers. The necessary requirement in the time of COVID-19 is to show evidence of engagement with CPD and of reflection.

Further, Federation has also introduced a system of fast-track approval for COVID-19-related events and is now approving more online modules and webinars, including approving the latter for up to one year after the live event. A further innovation is to approve E-Platforms and E-Libraries for CPD activities and credits.

Concluding comments

COVID-19 has presented us with many challenges and we are being, and will continue to be, tested by our response to them. The role of Federation is to deliver, on behalf of the three UK Royal Colleges of Physicians, a variety of activities in the fields of training, assessment and CPD. COVID-19 has disrupted some of this activity but, in challenging us, is leading us to consider and develop a number of solutions which will enable us to overcome the short term problems and better equip us to function even more efficiently in the twenty-first century. This will be to the benefit of our trainees and specialists both here in the UK and abroad and, ultimately and most importantly, to the benefit of our patients, wherever they may be.