A proposed new clinical assessment framework for diagnostic medical ultrasound students

Abstract

Current financial pressures within higher education institutions (HEIs) are driving new ways of delivering education and assessment. New technological developments are facilitating opportunities to rethink traditional educational methods and explore more innovative, effective approaches. Set against a background of increasing pressures to integrate technology to enhance learning, both in higher education and the NHS, education is moving towards greater integration of technology. Ultrasound education is an area which is currently being reviewed in many HEIs, as these programmes are expensive to administer for the relatively low numbers of students involved. Within ultrasound education, rigorous assessment of clinical competence is an area which is particularly expensive to undertake, and methods used in many training programmes are potentially unsustainable for HEIs in the current economic climate. In addition, clinical assessment methods used are often criticised for the difficulties encountered in trying to exclude subjectivity from the process, and ensure equity across all assessments. A new framework is therefore proposed, which has recently been accredited within a HEI ultrasound training programme by the Consortium for the Accreditation of Sonographic Education (CASE), and has been piloted during 2013. One of the components of this approach is the incorporation of an ultrasound simulator, which will help to standardise assessments and ensure students are assessed over a range of pathologies, rather than only those randomly presenting on the day of assessment. This paper discusses details of the newly accredited assessment process.

Introduction

The radical national changes that are underway for training of the healthcare workforce have major implications for both the providers of education and the employers of the healthcare workforce.^1,2 The model of healthcare training whereby the National Health Service (NHS) has responsibility for provision, is disappearing. The future direction is that of education providers needing to respond to commissioners’ requests and being able to evidence the capability to train.³ The intention is to establish a clear linkage between the educational needs of the future healthcare workforce and improved patient outcomes, along with developing a flexible approach to providing quality patient centred care.⁴

Training institutions will have greater accountability for the education of the future healthcare workforce, particularly with regard to quality metrics.³ There will be a requirement for an innovative approach to be adopted in terms of learning, teaching and assessment, and the training provision will have to constantly evolve to meet the changing needs of the healthcare workforce.⁴

There are significant concerns regarding the Department of Health (DH) proposal that funding for healthcare education will be restricted in the future to pre-registration programmes, and that there should no longer be funding available for post-registration and continuing professional development (CPD) provision.³ This poses a further risk to the future viability and availability of this provision. Funding for CPD will not be ring-fenced and may understandably not prove to be a high priority for Foundation Trusts and GP consortia during a period of radical structural change. Healthcare providers will be required to deliver efficiency savings over a 4-year time-scale, and CPD will be an easy target to cut. There are concerns over the reduction in funding for provision of postgraduate and CPD education,³ particularly in areas such as ultrasound, where there are acknowledged shortages of appropriately qualified staff.⁵

The current challenging economic climate, in which the health and education sectors are operating, is resulting in a continual search for more efficient methods of delivering healthcare education.^1,6,7 Universities have experienced severe reductions in budgets and can no longer afford to run courses for small numbers of students.^1,2 Many courses are being closed; however, these reductions in opportunities for formal training in small specialist areas, conflict with the requirement for improving the quality of care provided by healthcare workers, by curtailing the supply of appropriately qualified staff.^8,9 As a consequence, higher education institutions (HEIs) are looking for more efficient methods of running programmes, and this puts additional emphasis on the need for more creative methods of education delivery and learning opportunities.

In addition to these economic forces driving change there has been a rapid development of technology over the past few years, which has had dramatic effects on many aspects of society. In response, there have been a number of initiatives, from both the higher education sector and the National Health Service (NHS),^10–12 to integrate technology enhanced learning (TEL) into educational programmes. Current generations of students engage with technology in many aspects of their lives, and as a result expect it to form a large part of their education.^12,13 The challenge for HEIs has been to develop new approaches to teaching, learning and assessment, which incorporate technology, in order to improve the student experience. HEIs have therefore focused on using technology to enhance both the quality and the flexibility of learning opportunities, in order to meet student expectations.

As a result of the financial pressures, the rapidly evolving technological opportunities and associated student expectations, the further integration of technology into education programmes for healthcare workers is now generally viewed as the way forward, by enabling more efficient and cost-effective methods of delivering learning and assessment.^4,7,14

Ultrasound education

Ultrasound education is one of the small specialty areas under threat in many HEIs. The cost of running these programmes is high but numbers of students small and, although there is a continuing requirement to increase group size, current financial constraints within the NHS make it difficult to significantly further increase student numbers. However, demand for qualified sonographers currently outstrips supply,⁵ and there is a need to preserve education to ensure future sonography services can continue to be offered. Many departments are now in the position of being unable to provide sufficient staff or back-fill to train the future workforce. This need for additional sonographers, combined with the difficulty in increasing student numbers attending the training programmes, means that other methods of ensuring the courses are financially viable need to be explored.

Ultrasound education has evolved over the years, and will need to continue to change and adapt in response to changing pressures and opportunities as they arise. The Society of Radiographers (SoR) introduced an ultrasound training programme in 1977 which led to a Diploma in Medical Ultrasound for radiographers, or a Certificate in Medical Ultrasound for practitioners from other backgrounds.¹⁵ Assessment of competence took the form of a written examination paper and a viva. This education format continued to be offered until it was replaced in the early 1990s by a postgraduate HEI-based qualification open to all healthcare professionals.¹⁶ In 1993, an organisation was formed to oversee the provision of sonography education, known as the Consortium for the Accreditation of Sonographic Education (CASE). This consisted of representatives from a number of organisations, whose members were working as sonographers, reflecting the multidisciplinary nature of sonography.

Within ultrasound programmes, assessment of clinical competence has long been a contentious area. All programmes accredited by CASE are required to have a robust and transparent process for monitoring and assessing students, with the aim being to produce sonographers who are safe and competent to practise clinically.¹⁷ However, the methods for establishing this are not prescribed and, as a result, different HEIs have adopted a variety of methods for assessing students’ competence. Many in the field consider that a university assessor should perform a final summative assessment in the student’s clinical workplace. This, however, is an expensive, resource intensive process, which is potentially unsustainable for the majority of institutions in the current economic climate. In addition, critics of the process claim that, not only is it impossible to entirely exclude subjectivity from the process, but also it is difficult to ensure equity across all assessments when a variety of different assessors and patients are used for the assessments.

The University of the West of England (UWE), Bristol, currently offers a postgraduate training course in Medical Ultrasound. In December 2012 a Medaphor ScanTrainer (MedaPhor Ltd, Cardiff Medicentre, Cardiff, Wales) was purchased with the purpose of integrating its use into the teaching sessions in order to enhance the student experience. Simulation is offered as a response to the challenge of ensuring consistent learning and assessment in clinical practice, and has become increasingly attractive as an alternative education strategy in many settings,¹⁸ and one that may enhance clinical competence.¹⁹ Simulation provides a controlled learning environment, where the learner can engage in activities reflecting real life conditions, without exposing patients to risk.²⁰ The Medaphor system utilises a haptic device to replicate sensations of transvaginal and transabdominal real-life scanning applications (Figures 1 and 2). This was introduced into the programme for students beginning their modules in 2013 and was initially used for teaching, learning and formative assessments on the Obstetric and Gynaecology modules. However, it soon became apparent that the equipment also had the potential for carrying out assessment of student technical scanning ability and image interpretation skills. Discussions with the manufacturing company resulted in the formation of a user group. One of the first outcomes of this group was to request from the development engineers, a package of patient case studies, which could be designed to test the students’ knowledge and scanning ability over a wide range of subject specialty areas.

Figure 1.

The Medaphor transvaginal scanning simulator

Figure 2.

The Medaphor transabdominal scanning simulator

It was in this context that in 2012, the Ultrasound Programme team at UWE began to prepare for the quinquennial internal revalidation and external reaccreditation of the programme by CASE. During the design process for the new programme, the main objectives were to reduce costs of delivering the education programme, whilst trying to improve the student education experience and ensure equitable, standardised assessment processes. Various ideas were explored but further integration of technology into the learning and assessment process was selected as one of the most appropriate approaches. A new framework therefore evolved, which is being piloted during 2013, and has recently been accredited by CASE. One of the main components of this approach is the incorporation of an ultrasound simulator which, it is proposed, will offer a more equitable process by standardising assessments, and ensure students are assessed over a range of pathologies, rather than only those randomly presenting on the day of assessment.

The new framework for assessing student clinical competence

The clinical assessment scheme at UWE has always been a strong feature of the programme, with a clinical supervisor/appraiser identified within the clinical department, and a UWE appointed external assessor performing the final summative clinical assessment. The Course Team have always been strong advocates of this rigorous method of assessment, in order to provide complete transparency and confidence in the system for all stakeholders.

However, recently there have been incidents occurring during the end-of-module summative clinical assessment process, which resulted in the Course Team questioning their strongly held views on the appropriateness of this aspect of the assessment process. These involved students who have performed well throughout their training, but on finding multiple pathologies in an assessment situation, became flustered and lost confidence. Conversely there have been concerns with other students during their training about their ability to correctly identify and problem solve pathology, but that if all cases were normal during the assessment they may have passed.

These events caused the Course Team to question whether a more standardised assessment approach would perhaps result in a more equitable process to ensure students have achieved satisfactory levels of competence. This, together with the high costs of sending assessors to clinical departments over a wide geographical area, and increasing pressure on the University to ensure that costs of running the modules are covered by the income from the student fees, resulted in exploration of more equitable and cost-effective methods of carrying out the students’ final summative clinical assessment. As a result, the team consulted with clinical department managers, superintendents and appraisers to discuss proposals to replace the existing system for establishing student clinical competence, with a new framework.

The new framework that developed from these consultations consisted of:

• Inclusion of a simulator for teaching and formatively assessing obstetric, gynaecology and general medical ultrasound students.

• At the end of the module, a 1-hour assessment viva is to be held at the University for all clinical modules, and carried out by a member of the Ultrasound Programme Team. This will incorporate use of the simulator assessments and hands-on scanning where applicable. A variety of simulated case-studies and pathology examples will be used to test each student’s ability to obtain diagnostic images and interpret them.

• The final summative assessment is to be undertaken in the clinical workplace, performed by the student’s clinical supervisor experienced with the UWE system and if there are no particular concerns with a student’s progress. All clinical assessments should be performed with two department assessors present, and a consensus reached on the student’s level of competence. This is to demonstrate transparency in the assessment process. The viva has to be performed prior to the clinical assessment so that the assessors have the results of the test of the student’s underpinning knowledge before performing the clinical assessment.

• Where a department does not have assessors who are experienced with the University system, or where they feel there may be a problem student who is not progressing at the normally expected rate, an external assessor will undertake the assessment.

• UWE assessors will moderate 10% of all clinical department assessments, and these will be selected randomly.

• All clinical appraisers will need to attend an annual workshop at UWE, and this provides opportunities to update the training and assessment skills of these individuals.

• Students are required to complete a structured reflective clinical portfolio, designed to enhance the learning process and to demonstrate levels of progression during the course.

• It was also suggested that when selecting potential applicants for a trainee sonographer post, clinical supervisors may be interested in using the simulator for assessing potential ultrasound students’ hand-eye co-ordination and ability to differentiate features on ultrasound images.

The above changes to the clinical assessment process were all approved at the CASE reaccreditation event, and are being piloted in 2013. Feedback from students, clinical supervisors and the programme team will be obtained, to ensure the most effective system is implemented for the Programme from 2014. Further integration of the ultrasound scanning simulator into the programme is being planned. However, research into student experiences and perceptions of the simulator is currently being undertaken to ensure effective use of this technology for future deliveries of the modules.

Conclusion

The challenging economic climate in the UK is driving more innovative and cost-effective methods of delivering ultrasound education and assessment. Universities have experienced severe reductions in budgets and can no longer afford to run courses which are not financially viable. There are concerns over the reduction in funding for provision of postgraduate and CPD education, particularly in areas such as ultrasound where there are acknowledged shortages of appropriately qualified staff.⁵ As a consequence, HEIs are looking for more efficient methods of running programmes,⁴ and this puts additional emphasis on the need for more creative methods of education delivery and learning opportunities. In order to ensure robust education and assessment of the sonography workforce is preserved, the current system needs to adapt.

As a response to this, following consultation with relevant stakeholders, a new framework has been formulated for clinical ultrasound assessment processes. The intention is to establish robust education and assessment processes for future sonographers, whilst ensuring efficient use of resources. The course team are all committed to providing a high quality ultrasound programme that appropriately develops the ultrasound workforce of the future, to ensure the ultimate aim of achieving improved patient outcomes.

declarations

Competing interests: None

Funding: N/A

Ethical approval: N/A

Guarantor: VJ Gibbs

Contributorship: N/A