Immersive technology in ophthalmology education: a systematic review

Abstract

Background

Despite the increasing use of immersive technology (IT) in ophthalmology, the effectiveness of this approach compared to other teaching practices is unclear. This systematic review aimed to determine the value of IT to teach students ophthalmic skills and whether it can supplement or replace conventional teaching practices.

Methods

A systematic search was performed of CENTRAL, MEDLINE, EMBASE, ERIC and PsychINFO databases. Randomised controlled trials comparing IT interventions versus (1) no training, (2) standard training, (3) different types of IT interventions, (4) different doses of IT interventions were eligible for inclusion.

Results

Seven trials involving 177 participants were included. IT offered some benefit compared to standard training as most trials demonstrated evidence of learning represented by composite performance score and performance time. Repetitive training with IT displayed similar results.

Conclusion

IT appears to improve the ophthalmic skill of healthcare trainees and should be considered as a supplement to training.

Introduction

The UK population is getting older with 18% aged 65 years and over.¹ Therefore, the demand for ophthalmologists will increase in line with an ageing population.² Teaching methods may need to change to match future demand.

The ophthalmic specialist training (OST), approved by the General Medical Council in the UK, ensures trainees are equipped with necessary skills equivalent to a consultant ophthalmologist by the end of the 7-year postgraduate training programme. While the term ophthalmic skill is not readily defined in the literature, it essentially embodies the 13 domains of clinical practice outlined by the OST.³ These can broadly be grouped into clinical skills or procedural skills. This systematic review defines ophthalmic skill as clinical or procedural skills relevant to the field of ophthalmology.

‘Situated learning’ coined by Lave and Wenger⁴ postulates that learning occurs unintentionally in an authentic environment. Learners become involved in a community of practice, gradually moving towards full membership. This transition from novice to expert is achieved via scaffolding,⁵ in which the teacher models how to approach a problem, and then withdraws their help, only offering support as needed. This approach to learning mirrors the principles of ‘see one, do one, teach one’ seen in some specialities.⁶ For example, one method for cataract surgery training sees trainees performing a single step multiple times under supervision. Once the step is mastered, the trainee advances to the next step.⁷ Despite simulation in ophthalmology focussing on surgical skills through the use of wet-labs, the complexity of ophthalmic surgery remains a challenge for trainees.⁸ Virtual reality (VR) may have the potential to improve performance in novice ophthalmic surgeons.⁹

Immersive technology (IT) is an umbrella term for several different technologies: VR, augmented reality (AR) and mixed reality (MR).¹⁰ Milgram and Kishino’s¹¹ reality-virtual continuum (RVC) provides a framework to understand the relationship among VR, AR and MR. The RVC is a continuous scale ranging from completely real to completely virtual. MR encompasses AR and VR with AR falling closer to the real world, whereas VR is grounded in the virtual world offering the user autonomy to control their actions, mimicking the real world.¹⁰

Learning involves the individual construction of knowledge within social communities. Loyens et al ¹² describe four assumptions of constructivist learning, which are in line with VR. The virtual experience is designed without a specified sequence so the learner can interact with the system how they see fit,¹³ leading to their knowledge construction. Learners become self-regulated because the virtual environment is a rich source of information,¹³ necessary for self-directed study. Authentic real-life situations are viewed as ideal learning circumstances; Chen¹³ points out that the VR environment creates problems to simulate the real world. Finally, VR can invoke cooperative learning by creating a shared space for a group of learners.¹³

To date, no published systematic reviews look at the influence of IT on healthcare education or ophthalmology education. Numerous reviews have looked at the effect of VR concerning surgical training^14–16 and rehabilitation medicine.^{17 18} Simulation reviews in ophthalmology do exist; however, VR is often a small component of these reviews,¹⁹ or the review looks at factors affecting the use of VR in ophthalmology.² This review addresses the gap in the literature by investigating the effectiveness of IT as an educational tool for trainees to improve their ophthalmic skills. The aims of this systematic review are to determine the value of IT to teach students ophthalmic skills and whether IT can supplement or replace conventional teaching practices.

Methods

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.²⁰

Systematic search strategy

Electronic databases were searched from inception to 3 June 2019 to identify eligible studies. The Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE (OvidSP), EMBASE (OvidSP), ERIC (EBSCOhost) and PsychINFO (EBSCOhost) databases were searched. The reference lists of included studies were also searched to identify eligible studies. The MEDLINE search strategy included a combination of keyword searching, controlled vocabulary (Medical Subject Headings) and a randomised controlled trial (RCT) filter available from the Cochrane handbook.²¹ The following search terms were combined with Boolean operators: (exp Virtual Reality/ OR exp User-Computer Interface/ OR exp Computer Simulation/ OR exp OR Imaging, Three-Dimensional/ OR simul*.mp OR interactiv*.mp OR ((virtual or augmented or mixed) adj3 realit*).mp OR (immersive adj3 techno*).mp OR VR.mp OR AR.mp OR MR.mp) AND (Ophthalmology/ OR Ophthalm*.mp OR Optomet*.mp OR Ocular.mp). The search strategy was adapted for the other databases.

Eligibility criteria

The following inclusion criteria were set: (1) RCTs, (2) trials in the English language, (3) study population consisting of healthcare trainees defined as physicians (medical student, residents, fellows and consultants), nurses (training and qualified) and allied health professionals (training and qualified), (4) ITs delivered to healthcare trainees, which compared (a) IT intervention versus no intervention, (b) IT intervention versus standard training, (c) IT intervention A versus IT intervention B, (d) IT intervention dose A versus IT intervention dose B. Trials using non-IT or non-IT combined interventions with IT were excluded.

Outcomes

Kirkpatrick four-stage evaluation was used to classify primary and secondary outcomes. Training can have an effect at four levels: reaction, learning, behaviour and organisational impact.²² Primary outcomes for this review were: Kirkpatrick level 1 (reaction), for example, satisfaction with the intervention measured via questionnaire, and Kirkpatrick level 2 (learning), for example, composite score measuring task performance. Secondary outcomes were: Kirkpatrick level 3 (behaviour) and Kirkpatrick level 4 (organisational impact), along with adverse events in trainees.

Study selection and risk of bias assessment

References were screened independently and in duplicate (SL 100%; KM 50%, ZK 50%) for studies that appeared to meet the inclusion criteria based on the title and abstract. Full papers of potentially eligible studies were ordered and screened by SL and decisions documented in detailed extraction forms, which were screened in duplicate by KM and ZK. Disagreements between researchers about study eligibility were resolved through discussion. The Cochrane risk of bias tool was used to assess the methodological quality of studies included in this systematic review. The internal validity of an RCT is examined by assessing the following features: ‘sequence generation (selection bias), allocation sequence concealment (selection bias), blinding of participants and personnel (performance bias), blinding of outcome assessment (detection bias), incomplete outcome data (attrition bias), selective outcome reporting (reporting bias) and other potential sources of bias’.²¹ Funding was considered under other biases.

Data extraction and analysis

Data were extracted independently and in duplicate on study characteristics and risk of bias using a standardised data extraction sheet and entered into RevMan5. Findings were presented through a narrative synthesis across all included studies, study characteristics were tabulated and risk of bias was presented graphically.

Results

Search strategy

Two thousand three hundred and twelve references were identified through searching electronic databases. One record was identified through searching reference lists of included studies. One thousand three hundred and seventy-one records remained after duplicates were removed. The full text of 26 studies was obtained and assessed for suitability using eligibility criteria. At this stage, 19 studies were excluded on the grounds of ineligible intervention and study design (see figure 1). A PRISMA flow diagram is presented in figure 1.

Figure 1.

PRISMA flow diagram. IT, Immersive technology; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT, randomised controlled trial.

Study characteristics

Seven studies were included in this systematic review.^{9 23–28} The characteristics of included studies and their results are presented in online supplemental table 1.

Outcomes

One study²⁶ addressed Kirkpatrick level 1 (reaction) using surveys. All included studies addressed Kirkpatrick level 2 (learning) using quantitative measures for practical skills (surgical training, fundus examination and laser treatment). One study²⁶ addressed Kirkpatrick level 3 (behaviour) by measuring operating room performance. No studies evaluated Kirkpatrick level 4 (organisational impact) or adverse events in trainees.

Risk of bias

Risk of bias summaries across the included studies are presented in figure 2.

Figure 2.

Risk of bias summary. Key: red circle symbolises high risk of bias, green circle symbolises low risk of bias, yellow circle symbolises unclear risk of bias.

IT versus no intervention

Three studies looked at the effect of IT on surgical skills.^{9 24 25} Most of the outcomes representing learning (4/6) showed a benefit of IT compared with no intervention, however, the vast majority of outcome results (5/6) were non-significant. See online supplemental table 1 for detail on outcomes and significance levels.

IT versus conventional teaching

Three studies compared the effect of IT on laser treatment,²³ funduscopy²⁸ and surgical skill.²⁶ Two out of three studies demonstrated evidence of learning in favour of IT compared with conventional teaching, and most outcomes (4/6) were statistically significant. One study²⁶ looked at the behavioural impact (Kirkpatrick level 3) among participants, which showed conventional teaching to be superior to IT, however, this result was not statistically significant.

IT intervention: regime A versus regime B

One study²⁷ assessed different levels of repetitive IT on surgical training (cataract). The regime with higher repetitions showed greater evidence of learning. Half of the outcomes (1/2) were statistically significant.

Discussion

The results from this systematic review indicate that IT appears to improve the ophthalmic skill of healthcare trainees when compared with standard training and when delivered at higher repetitions. Only one study tested the impact of IT in practice (Kirkpatrick level 3). Although standard training in this trial was found to be superior to IT, this result was not statistically significant.

No other reviews have investigated IT and learning in ophthalmology. Thomsen et al ¹⁹ conducted a similar review, but included all types of simulation, for example, including low fidelity models like inanimate models. They were unable to offer a definitive conclusion about the effectiveness of simulation training in the field of ophthalmology, stating more trials were needed to investigate ‘validity, efficacy and cost effectiveness of the various training models’. This systematic review agrees with Thomsen et al’s¹⁹ concerning validity; greater validity evidence is needed for outcome measures for standardised comparison. On the other hand, the findings of this systematic review suggest that IT may be of some benefit to students’ learning ophthalmic skills.

IT, and by extension, the studies involved in this review are underpinned by situated and constructivist learning.²⁹ Learning is embedded in activity, context and culture.⁴ This is exemplified well in Daly’s trial. The VR simulators provide the activity, assessment in the operating room provides an authentic context where knowledge can be applied and the learner engages within the culture, facilitated by the presence of the attending ophthalmic surgeons. Similar to constructivist learning, social interaction and collaboration are essential elements for situated learning. However, none of the trials incorporated these features into their methodology. For example, trained ophthalmologists were mainly used to assess learners, rather than for demonstration purposes. Also, learners did not have the opportunity to interact with other learners to share knowledge. IT trials involving a greater collaborative approach may be an area for future educational research.

Three included studies were limited by small sample sizes^{24 26 28} and only two studies^{9 27} calculated the optimal sample size for their trial. A type II error may have accounted for at least some of the negative results given the small sample sizes. The SE and CIs of small studies tend to be wide, leading to imprecise estimates of effect, meaning no firm conclusions can be drawn.³⁰ However, the absence of statistical significance does not mean there is no effect.³¹ The 95% CIs can indicate where the true effect lies. Many of the included studies did not report CIs for the mean difference between groups, so an effect cannot be suggested with certainty. The included studies were conducted in Germany, Denmark, France and the USA, where ophthalmic training may differ to the UK and therefore the findings may not be generalisable beyond these countries. Although many of the included studies reported some kind of performance score to measure differences between groups, the results could not be pooled together for meta-analysis, because each score was comprised of different items or evaluated different skills.

The method used to assess participants learning may have skewed the results in favour of the intervention or control. One study²³ allowed the control group to practice on patients under supervision and examined both groups using patients. We would expect the control group to do better than the intervention group for two reasons: priming and secondly, training with real patients is superior to any simulation model because their purpose is to mimic a real-life experience. However, this study found no statistical difference between groups. This unexpected result should be treated with caution due to the high risk of bias but does prompt further investigation into other contextual factors which may influence results. Ideally, it would be most beneficial to compare the acquisition of skills between models,⁹ for example, VR simulation versus an animal model, similar to the trial conducted by Daly et al.²⁶ Using patients to evaluate differences between groups is often not feasible; therefore, skills should be evaluated using validated equipment and outcome measures, so results can be interpreted accurately.

Most educational trials do not evaluate beyond Kirkpatrick level 2 outcomes.³² Only one trial²⁶ attempted to evaluate their intervention at a higher level. Trials which assess higher levels of Kirkpatrick’s model need to be conducted to establish whether learning from IT interventions translates into clinical practice.

Limitations of the review process

Despite creating a comprehensive search strategy with guidance from an information specialist, it is unlikely this review located all eligible studies for this topic. Grey literature sources were not searched and may have contributed additional trials for review. Language bias may have affected this review since studies not available in the English language were excluded.

Conclusion

Although this systematic review relies on studies with an unclear risk of bias, the findings do suggest IT interventions offer some advantages when compared with participants who received standard training, or increased doses of an IT intervention as measured by Kirkpatrick level 2 (composite performance score, performance time, knowledge score) outcomes. At the minimum, the role of IT in supplementing training for novice learners should be considered.

Data availability statement

Data sharing not applicable as no datasets were generated and/or analysed for this study.