A Scenario-Based Virtual Patient Program to Support Substance Misuse Education

Abstract

Objective. To evaluate virtual patient (VP) programs for injecting equipment provision (IEP) and opiate substitution therapy (OST) services with respect to confidence and knowledge among preregistration pharmacist trainees.

Methods. Preregistration trainee pharmacists pilot-tested the VP programs and were invited to complete pre/post and 6-month assessments of knowledge and perceived confidence.

Results. One hundred six trainees participated and completed the pre/postassessments. Forty-six (43.4%) participants repeated the assessments at six months. Scores in perceived confidence increased in all domains at both time points postprogram. Knowledge scores were greater posteducation than preeducation. Knowledge scores were also greater six months after education than preeducation. Knowledge scores at six months were lower than posteducation for both programs.

Conclusion. Virtual patients programs increased preregistration pharmacists’ knowledge and confidence with regard to IEP and OST immediately after use and at six months postprogram. There was a loss of clinical knowledge over time but confidence change was sustained.

INTRODUCTION

In Scotland, substance (drugs and alcohol) misuse services are delivered in nearly all community pharmacies, with approximately 84% offering IEP or OST.¹ Patients using these services may also attend hospitals for a range of reasons, from presentation at accident and emergency to scheduled procedures, and require support from the pharmacy staff in the hospitals. The vast number of patients means there is no stereotypical substance misuser. The range of substances being misused is wide, from caffeine or tobacco to alcohol, prescribed analgesics and strong opioids (eg, heroin). Prescription analgesics are becoming more commonly misused but variation exists between localities, populations, and cultures.² The drug using population is aging and as a result, there is an increase in the number of age-related conditions also being presented.³

From the turn of the twenty-first century, substance misuse services delivered at Scottish community pharmacies have increased in range and prevalence: not solely opioid substitution therapy (OST) but injecting equipment provision (IEP), the supervision of disulfiram, and alcohol brief interventions. All are intended to improve the health of the population and not just those at the highest risk.⁴

Given the increased provision of these services, in particular OST and IEP, pharmacist education is essential to deliver safe, effective, and patient-centered care. Many opioid users feel stigmatized⁵ as can others using IEP or other treatment services. Feelings of stigma can be a barrier to treatment progress⁵ and education should be targeted at the nature of addiction and the opportunities to put patients at ease.⁶

Historically, pharmacist education in substance misuse allayed some of the concerns and fears staff had and assisted in reducing prejudices. Staff attitude was correlated with service provision and enhanced service delivery.⁷ However, evaluation of structured educational sessions for pharmacists indentified minimal change in attitude to methadone programs.⁸ Traditionally, education aimed at addressing these issues has been either by didactic lecturing, role play scenarios, or written distance learning materials (eg, NHS Education for Scotland (NES) Substance Misuse Pack).⁹ Specialist Pharmacists in Substance Misuse (SPiSMs) deliver lectures at undergraduate level in Scotland, which are introductory in nature, and also a tutorial during preregistration training as part of the NES Pre-Registration Pharmacist Scheme (PRPS). Some students may obtain further experience from part-time employment in a community pharmacy.

Education is evolving with an increased use of technology. The potential advantages and disadvantages of using technology in health professions education are well documented.^10-14 The benefits of using technology include reduced staff time,¹² remote accessibility, self directed and self paced content,^8,12 provision of rapid and up-to-date feedback,¹⁰ easily updatable content,¹⁰ and delivering a consistent message.¹⁵ Educators are able to monitor learning activities and assess learning outcomes, while learners experience greater flexibility and interactivity compared to traditional large group education.¹⁴ Reviews attempting to evaluate the effectiveness of technology in education compared to traditional educational methods highlight that technology enhanced methods can be as effective as traditional method.^15-19

In the field of substance misuse, online education for pharmacists has been developed and evaluated in New Zealand.²⁰ The approach replaced a 3-4 hour training session and resulted in improved attitudes of trained community pharmacists toward patients receiving OST, as well as improvements in confidence and skills. Participants stated that the online format was “feasible and highly acceptable.” In addition to online modules, online virtual patients are used in educating health professionals,^21,22 with increased use in pharmacy education over the last decade.^23-31 Virtual patient programs allow learners to gain experience by making and learning from decisions in a safe environment,²⁵ thus supporting the active experimentation stage of experiential learning theory.³² The inclusion of “scenario-based” virtual patient programs can be effective where strategic, rather than procedural, learning goals are required. They also provide an opportunity to experience problems with no black and white solution,³³ a situation pharmacists report disliking.³⁴

Pharmacists providing substance misuse services are required to make ethical decisions, the consequences of which may be significant. For example, patients who are stabilized on long-term methadone treatment may request a supply of clean injecting equipment from pharmacists providing IEP services. Choosing to supply this equipment could increase the likelihood of overdose from the combination of illicit substances and prescribed methadone. Conversely, refusing to supply the equipment may lead to the transmission of blood borne infections from equipment sharing. Given the potential significance of these consequences, virtual patient technology appears to be well suited for education in this context. However, there are no published reports of their use.

The aim of this study was to develop and pilot-test two scenario-based virtual patient programs to educate preregistration pharmacists on injecting equipment provision (IEP) and opiate substitution therapy (OST) services in the community pharmacy setting. Preregistration training in Scotland is a 12-month experiential program within a patient-facing setting under the supervision of an appropriately trained pharmacist. Trainees have direct contact with patients, including those requiring substance misuse services. This paper describes the implementation and evaluation of these scenarios aimed at improving preregistration pharmacists’ knowledge and confidence in the provision of these services during a year of experiential learning.

METHODS

Two virtual patient scenarios relating to IEP and OST were developed using computer graphics technology. This included a motion capture system to acquire realistic human motion including complex human movement such as injecting drugs. Professional voice artists were recorded and dubbed onto the high definition prerendered computer animations before embedding all media into the Adobe Flash, (Adobe Systems Incorporated, San Jose, CA) runtime environment. This resulted in a virtual patient package that could run on a wider variety of computer specifications without the need for the latest graphics card technology (Digital Design Studios, Glasgow, Scotland, UK).

The learning outcomes were developed for NES by the SPiSMs. Content targeting these outcomes was story boarded then incorporated into the technology. The program was then reviewed by SPiSMs and NES. The program included scenarios during which users were presented periodically with 3-4 choices and the outcome of their decision was played out in the program (Figure 1). Users were allowed one attempt but all alternative outcomes were played out following the outcome relating to the user’s choice. No data were collected relating to choices made by individual users.

Figure 1.

Example Screenshot of Learner Choices in the Virtual Patient Program.

Preregistration trainees were chosen for this pilot program to give them the experience of providing substance misuse services without risk of personal or patient safety. All 106 trainees enrolled in the 2013-2014 NES preregistration training in the west of Scotland were included. All other trainees and staff were excluded. Trainees participated in the pilot as part of a mandatory educational program. Trainees were allocated to use the program on one of eight sessions, hosted in a class room environment with a capacity of 16 trainees in each session. This setting was selected because there were sufficient computers with adequate hardware and bandwidth specifications, for each trainee to work individually and IT support was also available.

To evaluate participant confidence and knowledge of subject matter, assessments for each scenario were designed by SPiSMs and piloted by six preregistration trainees from the year preceding the study cohort. Only minor typographical changes were made to the assessments, and these trainees were excluded from the study. The confidence domains were the learning outcomes for the program and related to respecting patients, identifying barriers to access of care, identifying drug misuse symptoms, developing empathy, and understanding harm reduction (Table 1). The SPiSMs determined the underpinning knowledge that is necessary for any pharmacists to be able to provide such services competently in practice, and this was used to inform the knowledge-based questions. The finalized assessments were made available before and immediately after each scenario was used in the classroom. Participants from each session took the assessments simultaneously under invigilation.

Table 1.

Self-rated Level of Confidence Preintervention, Immediately Postintervention and at Six Months Postintervention

The same assessments were made available, via an e-mailed web link, at six months after the trainees had participated in the classroom session. Participants could complete the 6-month assessment in their own time. There was no incentive for trainees to complete this assessment. Assessments contained the same questions at each time point but participants were unaware of this, prior to taking them, and participants were not permitted to take notes during the classroom assessments. For each scenario, confidence was measured using a group of four and five questions with a 5-point Likert scale for OST and IEP, respectively. Knowledge was assessed using 20 multiple-choice questions, subdivided in to five categories, for each scenario. All assessments were hosted on Questback (Questback, Olso, Norway).

Assessment data were extracted to Microsoft Excel and pre/post and 6-month data were matched according to individual participants before analysis. Wilcoxon signed rank test was used to investigate the difference in confidence and knowledge reported at each time point. For participants who completed all assessments, a Friedman’s pairwise comparison was used to investigate the overall difference in scores at each time point. The Wilcoxon signed rank test (with Bonferroni correction, p=0.0167) was used to determine the difference in score between each assessment. McNemars and Binomial exact tests were used investigate the difference between knowledge scores for individual questions, where appropriate. Formal ethical approval was not required as the NES research governance group considered this piece of work as course evaluation. Guidance from the Chief Scientist Office for NHS Scotland was consulted.

RESULTS

One hundred six preregistration trainees from the west of Scotland region participated in the pilot program and completed the pre/postassessments. Forty-six (43.4%) trainees completed the assessment at six months. All trainees self rated their perceived levels of confidence in specified domains preexperience, postexperience, and 6 months later. Table 1 shows that confidence increased significantly in all domains after using the program.

All trainees took the knowledge assessment before and after the program, and there was a significant increase in test scores immediately after using the program (Table 2). For the 46 participants who completed the knowledge assessment pre, post and at 6 months, the greatest knowledge scores were observed immediately after completing the program, followed by the scores at six months. Friedman’s pairwise comparison indicated a significant difference between the three assessments for both the IEP (χ²=64, p<0.001) and OST (χ²=77, p<0.001). A pairwise application of the Wilcoxon signed rank test (with Bonferroni correction, p=0.017) indicated that there was a significant improvement between prescores and postscores for IEP (Z=5.8, p<0.001) and OST (Z=5.9, p<0.001). This was also true for the preprogram and 6-month scores for IEP (Z=2.6, p<0.017) and OST (Z=4.8, p<0.001). However, there was a significant decrease in scores between the posttest and 6-month test scores for both scenarios (IEP, Z=5.50, p<0.001: OST, Z=5.8, p<0.001).

Table 2.

Knowledge Assessment Scores Preintervention, Immediately Postintervention and at Six Months Postintervention

Analysis of responses to individual questions revealed there was a significant improvement for the majority of questions in both scenarios when comparing preassessment to postassessment (McNemar’s test, Tables 3 and 4). There were three of 20 (15%) and five of 20 (25%) questions that were answered significantly better at six months than in the preassessment for the IEP and OST scenarios, respectively (Binomial exact test, Tables 3 and 4).

Table 3.

Injecting equipment provision (IEP) Individual Knowledge Question Scores Preintervention, Immediately Postintervention, and at Six Months Postintervention

Table 4.

Opiate substitution therapy (OST) Individual Knowledge Question Scores Preintervention, Immediately Postintervention, and at Six Months Postintervention

DISCUSSION

The aim of this study was to evaluate two virtual patient programs for IEP and OST services, with respect to confidence and knowledge, for preregistration pharmacist trainees. There was an increase in knowledge and confidence immediately after use and six months after. The self-reported level of confidence for learners was significantly higher immediately after using the virtual patient program than prior to use. The increased confidence level was maintained six months later. There was a significant increase in knowledge scores immediately after using the program than prior to use.

Almost half the trainees completed a knowledge test at six months after the program and for these participants the highest scores were observed immediately after completing the program, followed by the scores at six months. There was a significant improvement for the majority of questions in both scenarios when comparing pre/postassessments. Generally, at six months the knowledge scores for each question were higher than prior to using the program. Learning during their experience in practice may have contributed to the retention of knowledge.

Before participating in the program, it was expected that trainees had had limited experience of providing substance misuse services. During their 1-year preregistration training, which was work-based in a pharmacy setting, the participants were likely to have immediate increased exposure to this patient group. The program successfully increased the confidence of the trainees when working with and caring for this patient group with their self-reported level of confidence score being significantly higher after using the program in all areas assessed. Trainees need confidence supporting this patient group because it can influence pharmacist involvement in such services.²⁰ In Scotland, there is little opportunity for pharmacists to avoid providing substance misuse services.

Within the virtual patient scenarios, learners were able to watch the consequences of their decisions as pharmacists being played out within the patient’s home environment. Participants reported increased confidence in developing empathy with this patient group, which may be because they could appreciate the challenges these patients encounter at home. The experience may also allow future pharmacists to build realistic expectations of pharmaceutical care with this patient group. The program supports previous research where education increased confidence dealing with challenges associated with and improved attitudes toward this patient group.^7,20 This form of learning also supports the decision-making process when dealing with real-life scenarios.

There was a significant increase in knowledge scores immediately after using the program, with some retention six months later. In the United Kingdom, services are delivered more frequently in community pharmacies than specialist centres or clinics. Trainees predominantly experience this through OST supervised administration and instalment dispensing. Patients visit the pharmacy more frequently when initiating therapy, which is a time of greater risk. National guidelines recommend a minimum of three months of daily supervision as treatment is initiated to increase patient safety.^35,36 Trainees should be knowledgeable in this subject area because as they are likely to experience a daily consultation with these patients. One strength of the program is that the learners can test their clinical and ethical decision-making skills, dynamically controlling the patient story in a simulated environment.

With IEP, analysis of responses to individual questions revealed there were three questions answered significantly better at six months than in the preassessment. One was concerning the hepatitis C vaccine, which was topical at the time of the study. The other two questions related to heroin administration. This may be a topic that a substance misuse patient would discuss with a pharmacist. For the OST scenario, the questions answered significantly better at six months were again issues that trainees may have experienced in practice (eg, using haloperidol for psychotic episodes).

Although knowledge scores were lower at six months than immediately after using the program, the majority of the knowledge scores at this point were higher than the preassessment scores. Cook et al stated that the majority of published research outcomes with virtual patient technology were largely focused on short-term knowledge.²² Our research demonstrates knowledge gains immediately and at six months.

To our knowledge, there are no published evaluations of virtual patient technology to support substance misuse education, this is the only postgraduate research carried out on students with a master’s degree who undertook a year of practical experience afterwards. Jabbur-Lopes et al concluded that more resources should be invested in developing virtual patient technology specifically for teaching at a graduate level.³⁷ Cook et al concluded the role of virtual patients in postgraduate and continuing education requires further study.²²

Much of the literature looks at teaching students clinical skills with chronic disease conditions^25,38-43 or determines student satisfaction with the learning experience^44,45 and the perceived value and useability of the virtual patient tools.⁴⁶ Our research looked at not only increasing knowledge but improving confidence with a challenging patient group. Research could be carried out using this teaching method in other similarly challenging groups. In response to the findings of this study, a series of shorter scenarios relating to recovery and the provision of unscheduled care services are in development.

A limitation of this study was that not all the participants completed the 6-month assessment. In addition, the sample size and the fact that all participants were preregistration trainees may limit the generalizability of the findings. Although this study investigated knowledge and confidence, there was no assessment of competence in practice.

The trainees were practicing in a real-life environment, which varied depending on the training site(s) and patient population for each trainee. All trainees in the PRPS should have had real-life experience with OST/IEP service provision as part of their core training. Therefore, the conclusion that knowledge and confidence as a result of the program is maintained at six months is confounded.

Questions were kept consistent at each assessment but participants could have checked reference sources before the 6-month assessment. To minimize this, participants were not made aware that they would be asked the same questions at the 6-month point.

The questions in the assessments were worded in a multiple true/false format, which limited assessment to participants’ factual recall of information only. In addition, the four questions in section 5 of the IEP assessment (Table 3) only assessed the participants’ knowledge of one fact. Future studies would benefit from using a format that allows for the assessment of higher-order thinking.

Future research could also assess if this type of education improves competence in practice and, although this pilot-test was conducted in a class room setting, the same technology could be used in a large lecture theater, small group tutorial, or individually on a laptop or mobile device.

CONCLUSION

Using a virtual patient program as a means of training postgraduate trainees on how to deliver pharmaceutical care to substance misuse patients was successful at increasing confidence and knowledge immediately after and at six months after the intervention. This study showed a gradual loss of clinical knowledge over time despite students maintaining confidence.