Using prescribing very short answer questions to identify sources of medication errors: a prospective study in two UK medical schools

Abstract

Objective

To assess the utility and ability of the novel prescribing very short answer (VSA) question format to identify the sources of undergraduate prescribing errors when compared with the conventional single best answer (SBA) question format and assess the acceptability of machine marking prescribing VSAs.

Design

A prospective study involving analysis of data generated from a pilot two-part prescribing assessment.

Setting

Two UK medical schools.

Participants

364 final year medical students took part. Participation was voluntary. There were no other inclusion or exclusion criteria.

Outcomes

(1) Time taken to mark and verify VSA questions (acceptability), (2) differences between VSA and SBA scores, (3) performance in VSA and (4) SBA format across different subject areas and types of prescribing error made in the VSA format.

Results

18 200 prescribing VSA questions were marked and verified in 91 min. The median percentage score for the VSA test was significantly lower than the SBA test (28% vs 64%, p<0.0001). Significantly more prescribing errors were detected in the VSA format than the SBA format across all domains, notably in prescribing insulin (96.4% vs 50.3%, p<0.0001), fluids (95.6% vs 55%, p<0.0001) and analgesia (85.7% vs 51%, p<0.0001). Of the incorrect VSA responses, 33.1% were due to the medication prescribed, 6.0% due to the dose, 1.4% due to the route and 4.8% due to the frequency.

Conclusions

Prescribing VSA questions represent an efficient tool for providing detailed insight into the sources of significant prescribing errors, which are not identified by SBA questions. This makes the prescribing VSA a valuable formative assessment tool to enhance students’ skills in safe prescribing and to potentially reduce prescribing errors.

Strengths and limitations of this study.

• There were a large number of participating medical students across two UK medical schools.

• We successfully assessed medical students’ ability to generate an authentic prescription and identified the sources of prescribing errors on a large scale using an automated marking system.

• The participants may be self-selecting to some degree; those that volunteered to participate may be more motivated or higher achievers.

• Further work is needed to examine the longer term impact of the use of very short answer questions and its effects on clinical practice at qualification or attainment in the prescribing safety assessment.

Background

Prescribing drugs forms a large part of the workload of doctors, and newly graduated doctors prescribe a significant proportion of those medications prescribed in hospital settings. It is a high-stakes task, with prescribing having significant implications for both hospitals and clinicians in terms of clinical risk and cost. Prescribing is a complex task for any doctor,¹ with prescribers having to select the correct drug, dose, frequency and route, while also taking into account interacting drugs and pre-existing comorbidities. Studies suggest an error rate of approximately 7%–10% among prescriptions written by clinicians in their first year after graduation, while more senior doctors have an error rate of around 5%.^2–4

Poor prescribing is not without consequence; medication errors are a common cause of harm to patients, with prescribing errors being the medication error most likely to cause moderate or severe harm to patients.^5–8 It has been estimated that 237 million medication errors occur per annum in England, with approximately 66 million of these being potentially clinically significant. These errors may have significant health and economic consequences with one study estimating that the burden of avoidable drug errors may cost the National Health Service approximately £1.6 billion per year and may contribute to 22 303 deaths.⁵ Developing interventions to reduce clinically important errors is therefore vital to improve patient safety and to reduce the financial burden on the National Health Service. Furthermore, the WHO has cited reducing harm from medication as one of its priorities since 2017.⁹

With such high stakes, it is crucial that undergraduate medical education prepares graduates to prescribe competently in a challenging work environment. However, many graduates report that they lack confidence in their prescribing abilities,^10–15 with only 29% of UK students feeling assured in their ability to achieve the General Medical Council’s (GMC) prescribing competencies on graduating medical school.¹⁰ The same study also found that the majority of students did not feel their prescribing knowledge and skills were thoroughly examined prior to graduation.¹⁰ This concern has been shared by both junior clinicians’ supervisors^16–19 and regulatory bodies.²⁰ Moreover, this appears to be a worldwide issue; medical students consistently appear to lack essential prescribing knowledge and skills.^{15 21}

At present, prescribing skills are mostly assessed using the written single best answer (SBA) examinations, the Objective Structured Clinical Examination (OSCE) or in Workplace-Based Assessments (WBAs).^{22 23} However, there are limitations to these assessment methods. While the SBA may allow broad sampling of the curriculum, it does not fully test the act of writing a prescription. Instead, it tests the ability to select a correct prescription out of a choice of five options. The SBA also gives no insight into the sources of errors among students.²⁴ The OSCE, conversely, can assess prescribing skills, but the scope of prescribing skills that can be tested is severely limited by the number of stations in the examination. WBAs, likewise, can assess prescribing skills; however, with the advent of electronic prescribing, undergraduates’ ability to achieve this competency has since been restricted. The Prescribing Safety Assessment, a national examination taken by medical students in the UK that is being adopted in Canada, Australia and New Zealand,^25–27 while going some way to address the issues described above, is an examination that is largely sat in the last few months of the undergraduate medical course. It is therefore not able to identify gaps in prescribing knowledge early enough nor does it provide early and longitudinal feedback for medical schools to be able to address deficiencies in prescribing knowledge and adjust the course content to strengthen skills in these areas. There is therefore a need to develop a means of formative assessment that facilitates learning by assessing students’ ability to prescribe across a broad sample of the undergraduate curriculum.

We have developed an online tool which allows thorough and authentic assessment of prescribing skills and medication management, in the form of the prescribing very short answer (VSA) question format. The aim of the prescribing VSA is to improve the validity of assessment of prescribing skills, and by extension the learning behaviour of prescribing among undergraduates, to enable safer and more confident prescribing on graduation.²⁸ Additionally, by identifying the types of error students’ make and areas of weaknesses in prescribing, the medical school curriculum can be adapted and improved. Identifying these deficiencies and remedying them is essential for both patient safety and health economics.

The prescribing VSA question format is based on similar principles to the VSA question, which has previously been shown to be a valid form of assessment with high reliability and discrimination when compared with SBAs.²⁹ Short answer questions (SAQs) have been shown to promote greater long-term information retention compared with SBAs,³⁰ but their use on a large scale has been restricted as they are not amenable to machine marking. VSAs, in which students provide an answer of one to four words in response to an open-ended question, are able to be marked electronically using new information technology, provide a way of using the benefits of SAQs while remaining feasible to mark efficiently on a large scale. The prescribing VSA format poses a clinical scenario and a lead-in question. The key difference in the prescribing VSA question is that the student must input free text answers for each of the medication name, dose, route and frequency answer fields.

The newly developed online software allows for wide sampling of the undergraduate curriculum for large numbers of students, using realistic clinical scenarios. The aim of this study is to evaluate the reliability and discrimination of prescribing VSA questions in prescribing skills assessment when compared with the traditional SBA question format, to assess the types of error undergraduates commonly make when prescribing and to assess the acceptability of using machine marking for prescribing VSA questions on a large scale.

Methods

Participants and assessment

This prospective study was approved by the Medical Education Ethics Committee at Imperial College London. Ethical approval was granted to invite all final year medical students at two medical schools (Imperial College London and University of Edinburgh) to sit the formative prescribing assessment. There were no other inclusion or exclusion criteria. The assessment was conducted on iPad tablets or fixed terminal computers using the newly developed online prescribing examination software (PRACTIQUE; Fry-IT, London, UK) and was held under examination conditions. All students had previous exposure to the VSA question format, through their use in formative assessments.

The students sat a formative examination in two parts. The first included 50 prescribing scenarios in the prescribing VSA format for which students had to generate a full prescription, including the medication name, dose, route and frequency. They were required to enter the medication name and dose in two separate free text fields, whereas the route and frequency were selected from two separate dropdown menus. The second part included the same 50 scenarios, in which the students selected the correct answer from five options in the traditional SBA format. Students were allowed to access the British National Formulary online throughout both parts of the assessment.

Each question consisted of a clinical scenario (which included the presentation, examination findings and investigation results, as necessary) and a lead-in question. Example prescribing VSA questions are available in the online supplementary file. The clinical scenarios were constructed such that they could be used in both the prescribing VSA and SBA format without any change to their content. The question topics were mapped to the final year undergraduate curriculum to ensure a broad sampling of the syllabus. The length of the VSA prescribing examination was 125 min, and the length of the SBA examination was 50 min.

Marking

The answers to the prescribing VSA questions were captured by the examination software (PRACTIQUE) and sent to a server via an encrypted connection. All identical responses were grouped in blocks by the examination software, and then machine marked using an automated matching algorithm. This compares the student’s answer against a set of preapproved acceptable answers for each question and uses a measure called Levenshtein distance³¹ to measure how closely a student’s given answer matches those preapproved correct answers. All student answers that were identical to the list of approved answers were automatically marked as correct. This list of preapproved answers normally consisted of a variety of correct drugs/doses/routes/frequencies, as determined by a group of clinicians. Students had to have entered the correct medication name, dose, route and frequency to score one mark. All match failures were highlighted by the software, and these responses reviewed by two clinicians simultaneously. Marks for responses deemed correct by the examiners could be awarded manually. Any responses marked manually as correct by the examiners would be applied to all identical answers. The examination software also permitted answers marked manually as correct to be added to the correct answer database for that question. The time taken by the two examiners to review the responses was recorded to assess acceptability. Responses to the SBAs were entirely machine marked using the examination software (PRACTIQUE).

Analysis

Statistical analyses were performed using PRISM V.8.0.0 (Graphpad Software, San Diego, California). Mann-Whitney test was used to compare the differences between VSA and SBA scores. Spearmann’s correlation coefficient was used to assess the correlation between the scores of the two formats. Cronbach’s alpha was used to assess the reliability of the assessments. The difference between proportion of correct and incorrect answers between the VSA and SBA question formats was examined using Fisher’s exact test.

Results

A total of 364 final year medical students sat the formative prescribing assessment.

Prescribing VSA utility

The total time spent by examiners (acceptability) to review the non-matching answers for 50 prescribing VSA questions for all 364 students (18 200 prescriptions) was 91 min. This is an average of 1 min and 49 s per question. The median percentage score for the prescribing VSA test (28%, IQR 20%–34%) was significantly lower than that of the SBA test (64%, IQR 54%–70%) (p<0.0001). There was a significant but modest correlation between VSA and SBA scores (r=0.66, p<0.0001). Reliability (Cronbach’s alpha) was 0.76 for the VSA test and 0.82 for SBA test.

Sources of error

Of the incorrect responses in the prescribing VSA assessment, 33.1% of these were due to incorrect medications being prescribed, 6.0% due to incorrect doses, 1.4% due to incorrect routes, 4.8% due to incorrect frequencies and 6.1% due to a combination of these errors.

Prescribing errors identified by the two formats

The scores on individual items were aggregated by prescribing area to allow comparison between the prescribing VSA and SBA question formats. There was a statistically significant difference between prescribing VSA and SBA student scores for all subject areas (table 1). Students consistently were less successful at writing a correct prescription compared with selecting the correct prescription from five options. In particular, they performed most poorly in prescribing fluids, insulin, anticoagulation, steroids and analgesia.

Table 1.

Student answers (correct and incorrect) to equivalent VSA and SBA questions in 10 prescribing areas

Grouped by subject	VSA correct	VSA incorrect	SBA correct	SBA incorrect	P value
Alcohol withdrawal	289	439	693	35	<0.0001
Analgesia	261	1559	928	892	<0.0001
Anticoagulation	292	1164	721	735	<0.0001
Antimicrobials	1168	2836	2625	1379	<0.0001
Emergencies	479	1341	1022	798	<0.0001
Fluids	80	1740	818	1002	<0.0001
Inhaled therapy	164	564	410	318	<0.0001
Insulin	26	702	362	366	<0.0001
Paediatrics	589	503	894	198	<0.0001
Steroids	98	994	620	472	<0.0001

SBA, single best answer; VSA, very short answer.

Discussion

Although prescribing skills are widely assessed through a variety of means in the undergraduate curriculum,^{22 23} until now there has not been an accepted method of assessing students’ ability to generate an authentic prescription on a large scale. SAQs have previously been acknowledged as a superior assessment format for testing prescribing skills but are labour intensive and time consuming to mark.³² The novel prescribing VSA question format overcomes these limitations while still requiring knowledge, judgement and skill to generate the correct answer. Furthermore, the rich data generated regarding the sources of error undergraduates make can be used to inform and improve prescribing skills teaching in the undergraduate curriculum. Additionally, personalised feedback can be sent out to the students, including what they have written for each question together with the correct answer. Our results suggest that the prescribing VSA question format is an acceptable and reliable assessment method for prescribing skills, with a number of advantages over using the traditional SBA.

Compared with the SBA, the prescribing VSA has allowed for a much more authentic and valid assessment process as students had to actually prescribe a medication rather than select the correct response from five possibilities. There was only a modest correlation between SBA and VSA, which suggests the assessment methods are measuring different constructs. Many of the prescribing errors made by students in the VSA format would have important clinical implications for patients; yet when answering the same question in an SBA format, they are able to select the correct answer. The corollary of this is that the SBA question format gives a falsely reassuring impression of students’ prescribing knowledge and skills.

Another significant advantage of the prescribing VSA questions compared with SBA questions is the rich feedback it gains from student responses. SBAs only show the examiner which questions students found more difficult but does not provide any insight into why it was more difficult. The prescribing VSA, however, allows examiners to pinpoint the specific areas of difficulty to the medication, dose, route or frequency of the prescription written. This allows educators to tailor teaching to target problematic areas and common prescribing mistakes.

For example, the prescribing VSA test was able to identify that some students prescribed large doses of rapid-acting insulin for a hyperglycaemia scenario, which in clinical practice would be a serious prescribing error. When prescribing fluids, students were frequently unable to select the appropriate fluid or duration of administration. Students were consistently unable to prescribe anticoagulation agents in a safe manner. Prescribing opiates, especially in a palliative care context, was another question in which doses with a potential to cause serious harm were often prescribed. The same questions in SBA format would not have yielded this important feedback. The students were at the beginning of their final year, so their performance may improve as they approach graduation. However, with the advent of electronic prescribing, it has become increasingly more difficult for students to practice in the workplace, as the system only permits qualified doctors to prescribe. This rich qualitative data can be used by medical schools to target interventions to improve prescribing education for undergraduates.

The prescribing VSA has also allowed 50 practical prescribing scenarios to be assessed in one sitting, which cannot be achieved using the time and resource-intensive OSCE examinations or opportunistic WBA methods.

The use of the iPad application as a platform for the prescribing VSA assessment has shown effective examination delivery. The machine marking is labour sparing as demonstrated by the 91 min taken to mark a large number of prescriptions. This study may be limited by the self-selecting nature of the sample; participation was not compulsory at either medical school, and it may be that those students who agreed to participate in the study are more motivated or higher achievers. While 18 200 prescriptions were generated across 364 students, weaker students are likely to make the same error repeatedly across the paper; this may give an artificial impression of the number of errors made. Furthermore, it is possible that students from the same institution have a tendency to make the same category of error, perhaps related to curriculum or teaching. This limits the generalisability of the results, and further work across a wider range of institutions is warranted. There are also inherent limitations in developing assessments, no matter how authentic, which take place in a controlled environment although with a time pressure. In real-life clinical practice, prescribing is often performed in a hurry while juggling other clinical or workload priorities.

Conclusions

Overall, VSA questions are an acceptable and reliable form of assessment of prescribing which provides detailed feedback, making it an excellent tool which supports students’ learning of safe prescribing, as well as the thorough assessment of prescribing skills. The rich feedback that can be derived from analysis of the sources of error that students make can be used to inform and improve the undergraduate curriculum. We hope that this intervention to improve junior clinicians’ prescribing has the potential to have a significant impact on patient safety.