‘Working together’ - A new approach to Reviewing the Quality of Postgraduate Medical Training Posts in Northern Ireland?

Abstract

Many of us are involved in the education and training of junior doctors. Maintaining and improving the quality of such training is the common goal of all medical educators, including those working in the Northern Ireland Medical and Dental Training Agency (NIMDTA) and within our hospitals – the Local Education Providers (LEPs). The development of NIMDTA’s Placement Quality Initiative (PQI) aims to create a more collaborative working relationship between NIMDTA and the LEPs, working together, to achieve a shared goal and develop and implement strategies to improve current practice. We review the PQI process, from both a trainee and trainer’s perspective, and ascertain if this approach has facilitated positive, reproducible changes in training programmes that are felt at ground level.

Introduction

The ability of trainees to maximise their potential is directly influenced by the quality of training they have access to; the common goal of all medical educators being the maintenance and continued improvement in the quality of education and training being delivered. The General Medical Council (GMC) stipulates that postgraduate deaneries must have effective educational governance systems and processes to monitor training sites, to ensure that the standards set out in ‘Promoting Excellence’¹ are being met. These should continuously improve the quality and outcomes of education and training by reviewing the quality of teaching, support, facilities and learning opportunities available on placements. Such reviews need to be robust, ongoing processes which ensure that training is taking place in a supportive and constructive environment, with evidence of good educational practice.¹

The Northern Ireland Medical and Dental Training Agency (NIMDTA) manages postgraduate medical training within Northern Ireland (NI). In November 2020 there were 24 hospital sites acting as Local Education Providers (LEPs) in NI, with 1734 trainees currently in a training post. In compliance with the GMC’s Quality Assurance Framework, a key component of NIMDTA’s Quality Improvement Framework is a Deanery visit to each LEP on a 5-yearly cyclical basis.² Primary, independent and qualitative data obtained through face-to-face interviews with trainees, trainers and educational management leads during visits, is triangulated with self-assessment LEP reports and Trainee surveys. In addition, visits provide the opportunity to explore areas of good practice and concerns and provide feedback.

Deanery visits have often generated a significant workload for both the Deanery and LEPs, which is out of proportion to the improvement achieved; repeat visits sometimes identifying unresolved issues. Therefore the existing visit process is not always effective in improving the quality of the training experience and LEPs can find engagement with the process difficult, as interactions can be perceived as hierarchical, if not confrontational, with the Deanery sitting ‘across the table’. Furthermore, available monitoring tools, such as the GMC’s National Trainee Survey (NTS), may not provide detailed local information, enabling only limited interpretation for smaller training units and there is more limited trainer engagement in the process. Three of the most recent large-scale NHS reviews (Keogh, Francis and Berwick), have highlighted that junior doctors’ views about their training experience need to be heard and that an effective programme of monitoring, including the use of trainee surveys, should be ensured.^3-5 All three reports support the thought that quality and safety in the NHS requires a simple monitoring process, delivered at local level.^3-5

In response to the administrative burden and limited improvements achieved using established practices, NIMDTA developed a Placement Quality Initiative (PQI) as an alternative method of meeting the GMC’s quality assurance standards. The objectives of the PQI are to improve the quality of medical training posts through review of current placements, active engagement with trainees, trainers and providers, and the development and implementation of strategies to improve current practice. This approach aims to create a more collaborative working relationship between NIMDTA and LEPs, working together, ‘around the table’, to achieve a shared goal.

In 2018 the PQI began by reviewing two training programmes – Obstetrics and Gynaecology and the Foundation Year 1 Programme. Review of Psychiatry specialities and Core Surgery were undertaken during 2019-20. Over the last two years the PQI team have obtained feedback on training placements from almost five hundred doctors in training.

We sought to answer the question: Does the PQI process, through a more collaborative approach, utilising shared and agreed goals, provide a more effective method to address the requirement on the Deanery to quality assure postgraduate medical training?

Method

The PQI process of reviewing training programmes is outlined in Figure 1. The process starts by collecting all existing information on training quality. This includes data available from previous Deanery visits, GMC NTS and specialty-specific data and reviews such as The Royal College of Obstetricians (RCOG) Training Evaluation Form (TEF) and Attrition: Solutions Focus Group Report.^6,7 The curriculum requirements of the training programme and the current training structures within NI are also considered. Following this a programme-specific, online, anonymised trainee questionnaire is developed in cooperation and agreement with trainees and trainers. Trainees are asked to provide their opinions on their current training post, including access to curriculum-specific areas of training. Additionally, the survey provides a number of opportunities for free-text comments, facilitating the broadest range of trainee feedback to be obtained. The data collected is then analysed and presented at a series of meetings both regionally to the Head of School and Training Programme Directors and then specific information is presented to Lead Educators, Directors of Medical Education (DMEs) and Education Managers in LEPs.

Figure 1.

Steps of the PQI process

Meetings with each LEP are attended by both educators and non-medical Trust leaders and administrators. Areas of strength and weakness are discussed and good practice from other units shared to enable development of collaborative improvement solutions. Actions to address identified areas for improvement are then agreed by both NIMDTA and the LEP. A report of the Regional PQI Review, along with key recommendations, is then made available to all educators and published on the Deanery website. Following a period to allow for the implementation of improvement strategies (12-18 months) trainees are re-surveyed to evaluate progress in achieving agreed development needs. A follow-up report is then disseminated and published.

At the beginning of the PQI process the team conducted an awareness campaign, presenting the PQI concept to Lead Educators, through educational committees and meetings (Department of Health Medical Leaders’ Forum, Trust Board meetings, Trainer Education Days) and to trainees through the NIMDTA Trainee Forum, BMA Junior Doctors’ Committee and articles in trainee newsletters. Engagement of trainees, trainers and LEPs in the process was considered essential in bringing about positive change in the quality of training placements and early socialising and promoting of the PQI concept was an important step in achieving this.

In order to address the question as to whether the PQI process provides a more effective method to deliver quality improvement in training placements than the current Deanery approach, feedback on the process was obtained through an online survey (Appendix 1). The ‘Experience of Placement Quality Review Questionnaire’ was released to all those involved in post-graduate medical education who had participated in any of the PQI processes. This included Clinical and Educational Supervisors, College Tutors, Training Programme Directors and Heads of School, DMEs and medical and non-medical LEP members involved in education such as Medical Directors, Clinical Directors and Education Managers. Trainee opinion on the PQI process was also obtained through a short 6-question anonymised trainee questionnaire (Appendix 2).

Results

Trainer opinion

The ‘Experience of Placement Quality Review Questionnaire’ was open online for a 3-week period in Oct 2020. The survey response rate was 40% (40 responses). Respondents had participated in the PQI review of at least one of the programmes reviewed with 13% being involved in the review of all programmes. There were responses from all categories of participant and from all five LEP Trusts in NI.

The majority of educators (70%) agreed or strongly agreed that an anonymised trainee survey provided an accurate picture of the quality of training within a unit and 91% indicated that the NIMDTA PQI team had a good understanding of the training programme and training structure under review. Comments were that the survey provided trainees with a ‘safe space’ to report issues and that repeating surveys at varying times could strengthen and improve the accuracy of findings. Feedback on the initial PQI meetings, during which the trainee survey results were presented, was very positive with the process being regarded as informative, supportive and collaborative, affirming current issues and providing useful evidence to facilitate change (Figures 2 & 3). The majority of educators (87%) reported that trainee feedback from the survey was disseminated to the wider clinical team through email or a local departmental meeting. Within the wider clinical team it was reported that 61% welcomed the feedback, two thirds of whom were prepared to implement changes in response. The PQI reports generated following the initial review were well received; with one educator highlighting that it was ‘useful as a starting point, especially when there is pushback from clinical management teams’.

Figure 2.

Educator feedback on the PQI process

Figure 3.

Educator feedback regarding the PQI initial meeting with LEPs

Almost all of the educators who responded to the PQI Review questionnaire (95%) reported having gained new insight into training in their unit through the process, with 87% agreeing that highlighting areas of regional and reproducible good practice was helpful. A further 57% agreed that trainee feedback accurately captured the quality of training. There were some concerns that a lower survey response rate in a few units might mean that the results were not reflective of training quality, that trainee dissatisfaction could drive increased participation and that some trainees might avoid providing negative feedback due to fear of loss of anonymisation.

Overall, 74% of trainers agreed that the PQI fulfilled its aims and has created a more collaborative approach to improving the quality of training posts. Only 22% felt that the process engaged trainers less than Deanery Visits.

Trainee opinion

The PQI review of the O&G training programme was one of the first PQI reviews in 2018 and has advanced beyond the initial PQI assessment and feedback phase. Trainees in O&G have had time to observe potential improvements in training practices and to complete a re-survey of their training experience. Trainee opinion of the PQI process was therefore determined by sending current O&G trainees a short, anonymised, online survey in October 2020. The response rate was 34% (30 trainees) with responses from all levels of training. Almost two thirds (65%) agreed or strongly agreed that an anonymised trainee survey provides an accurate picture of the quality of training within a unit, with a similar percentage feeling confident to express their opinion, both positive and negative, through the PQ surveys. In view of NI being a small Deanery, a few concerns were expressed that trainees might be more easily identifiable and comments might potentially impact future career prospects.

Of the respondents that still worked in the same unit, 31% saw definite improvements in training opportunities. Almost two thirds (61%) felt that the PQI process provided useful information to initiate positive change using a collaborative and supportive approach. Requests were made for the results of the PQI Reviews to be communicated directly to trainees.

Discussion

The primary objective of NIMDTA’s Placement Quality Initiative (PQI) is to produce meaningful improvement in the quality of the training experience provided by medical training posts. This new approach to reviewing training placements aims, through active engagement with trainees, trainers and LEPs, to develop and implement strategies to improve current practice and to enhance working relationships between the Deanery and LEPs. Trainee and trainer feedback, both formal and informal, has highlighted the PQI as a collaborative and less confrontational approach to assessing and improving training quality with measurable results already observed on the ground. A Regional Unit Prospectus for O&G Training in NI and a ‘Train in O&G in NI’ leaflet outlining the structure of the O&G training programme, were produced in collaboration with trainers and trainees in the five LEPs, following the O&G PQI Review in 2018.⁸ This has improved the information available to trainees about the training programme and training sites so that now the majority of trainees report having sufficient knowledge to make informed decisions in regard to placement options, compared to only a third prior to these publications.⁹

As highlighted in the Berwick, Francis and Keogh reports, junior doctors’ views about their training experience need to be heard.^3-5 The PQI provides a platform for this to occur. As part of the development of the PQI process, the PQI team reviewed the quality management infrastructure in other UK Deaneries. Health Education England (HEE) conduct the National Education and Training Survey (NETS) twice a year for all healthcare trainees and students to provide insight into the quality of training placements, identify areas of good practice and areas for improvement.¹⁰ NHS Education for Scotland developed the Scottish Training Survey in 2013 to add value to GMC NTS data, produce robust indicators for quality management and to capture data about each training post.¹¹ NIMDTA’s Placement Quality Initiative described here, is a novel and effective way of meeting the GMC standards for quality managing postgraduate training posts. It is a process which is replicable for other training programmes and regions and has been shown to promote collaborative working between trainees and trainers, LEPs and the Deanery.

A potential weakness of the PQI, suggested by trainee and trainer feedback, was the concern that lower survey response rates or participation being driven by trainee dissatisfaction may impact on the reliability of data obtained. In the six trainee surveys conducted to date, we have obtained opinions from almost five hundred NI trainees and the trainee survey response rate has ranged from 42% to 74% in the most recent survey. We highlight that the data obtained through the surveys is triangulated with other data sources, including the GMC NTS and published specialty-specific data, providing a broader view of the training experience; a key strength of the initiative. As more training programmes become involved in the PQI process and recommendations, good practice and improvement initiatives are disseminated and published, trainers and trainees are developing an increased familiarity with the process. It is anticipated that this will promote better engagement in the PQ process that will improve and maintain a high survey response rate in future training programme reviews.

The PQI approach represents a culture change in the monitoring and improving of the quality of training posts across Northern Ireland. Working together in a less hierarchical, ‘around the table’ relationship has improved the ability to affect change and has created engaging partnerships which align key medical education goals between LEPs and the Deanery. The improvement methods employed by PQI allow data-driven good practice to be shared, changing perspectives and moving away from a Deanery inspectorial approach. Doing so has promoted engagement and created a collaborative, outcome focused strategy to promoting training quality. We know that doctors who feel highly valued and motivated are better equipped to deliver high-quality care and meet the needs of patients and the wider NHS.¹² They can also be effective powerful agents for change and their views about their training are essential in the integration between clinical practice and education.

Conclusion

NIMDTA’s Placement Quality Initiative aims to improve the quality of medical training posts through review of current placements, active engagement with trainees, trainers and providers, and the development and implementation of strategies to improve current practice. It is a driver for change which diminishes hierarchical interactions between Local Education Providers and the Deanery. It promotes collaborative working relationships to generate practical improvement strategies and observable change in the quality of postgraduate medical training which is experienced at trainee level. The PQI approach has facilitated positive changes in the training programmes reviewed to date and continues to adapt and be tailored to upcoming specialty reviews. There is potential to integrate this method with the existing Deanery visit programme to enhance the monitoring of training placements and produce improvements in the overall quality of training, providing an enriched training experience, better quality of trainees and ultimately improved patient safety.

Appendices

1. Trainer survey

2. Trainee survey