A systematic review of simulation-based training tools for technical and non-technical skills in ophthalmology

Abstract

To evaluate all simulation models for ophthalmology technical and non-technical skills training and the strength of evidence to support their validity and effectiveness. A systematic search was performed using PubMed and Embase for studies published from inception to 01/07/2019. Studies were analysed according to the training modality: virtual reality; wet-lab; dry-lab models; e-learning. The educational impact of studies was evaluated using Messick’s validity framework and McGaghie’s model of translational outcomes for evaluating effectiveness. One hundred and thirty-one studies were included in this review, with 93 different simulators described. Fifty-three studies were based on virtual reality tools; 47 on wet-lab models; 26 on dry-lab models; 5 on e-learning. Only two studies provided evidence for all five sources of validity assessment. Models with the strongest validity evidence were the Eyesi Surgical, Eyesi Direct Ophthalmoscope and Eye Surgical Skills Assessment Test. Effectiveness ratings for simulator models were mostly limited to level 2 (contained effects) with the exception of the Sophocle vitreoretinal surgery simulator, which was shown at level 3 (downstream effects), and the Eyesi at level 5 (target effects) for cataract surgery. A wide range of models have been described but only the Eyesi has undergone comprehensive investigation. The main weakness is in the poor quality of study design, with a predominance of descriptive reports showing limited validity evidence and few studies investigating the effects of simulation training on patient outcomes. More robust research is needed to enable effective implementation of simulation tools into current training curriculums.

Introduction

Historically training in ophthalmology, as in other surgical specialties, has been based on a Halstedian model of apprenticeship learning. Trainees are assumed to be competent upon completing a minimum number of surgical procedures. Changes to the clinical environment and professional values have forced a review of this approach [1]. One of the problems associated with this model is the inconsistency in levels of knowledge and skills gained due to variations in clinical exposure and educational opportunities [2]. Using the total number of procedures that a trainee has performed as a benchmark for skill is also problematic as quantity does not equate to quality and competency cannot be accurately discerned in this way. Reductions in training hours due to regulations such as the European Working Time Directive further limit potential training opportunities [3]. Furthermore, growing ethical concerns over the use of patients for training purposes [4] are also having major impacts on training particularly in the early stages of the learning curve. Studies have shown close correlation between experience and complication rate [5, 6].

These issues highlight the need for improved training programmes with the development and objective assessment of proficiency prior to treating patients. Simulation models offer a platform for trainees to improve their clinical and surgical skills, enabling focussed, competency-based training without putting patients at risk. The healthcare sector is continually making rapid technological advances and the development of simulator models as safe and effective tools for training and assessment has risen dramatically. This trend has been observed within the field of ophthalmology [7], but the extent to which simulation is used varies widely between different training programmes. Its role remains limited by a lack of formal, standardised integration into existing curricula.

The purpose of this systematic review is to comprehensively evaluate the effectiveness and validity of all simulator models developed for ophthalmic training to date.

Methods

This review was carried out following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) statement and registered on the international prospective register of systematic reviews, “PROSPERO”, prior to conduction of this study (registration number: CRD42018087929).

Eligibility criteria

All original studies were included if they described simulation or e-learning for technical or non-technical skills development in ophthalmic training. Inclusion criteria for study participants were ophthalmologists of any grade and medical students who had completed or were completing their ophthalmology attachment. Studies were excluded if they did not provide original data; articles not specific to ophthalmology; and studies that did not use simulation for training or assessment purposes. We included all papers irrespective of the language.

Search methods

A systematic search of PubMed and Embase was carried out, using the terms “(simulat* OR virtual reality OR wet lab OR cadaver OR model OR e-learning) AND ophthalm* AND (training OR programme OR course)”. Search date was from inception to 01/07/2019. Reference lists from included articles and relevant reviews were hand searched for eligible studies.

Study selection

Two authors, RL and WYL, carried out independent, duplicate searches. All abstracts were reviewed and articles that were potentially eligible were read in full. A final list of studies meeting the eligibility criteria was compared and disagreements resolved by discussion (Fig. 1).

Fig. 1. PRISMA Flow Diagram.

Flow diagram of study selection process.

Data collection

The same two authors extracted data for each study separately and differences were resolved through discussion. Data collected included details of the simulator model, type of study design, number of participants and their training level, training task(s) involved, duration of training, and outcome data addressing validity and effectiveness of the model.

Data analysis

Studies were grouped according to simulator type: virtual reality; wet lab (live or cadaveric animal models and human cadaveric models); dry lab (synthetic models); and e-learning models. Validity was evaluated based on Messick’s modern validity framework [8] and the strength of each source of validity evidence was measured using a validated rating scale [9]. Effectiveness was quantified using an adaptation of McGaghie’s proposed levels of simulation-based translational outcomes (Table 1) [10]. Qualitative analysis was carried out due to the heterogeneity of study designs.

Table 1.

Details of the frameworks used for evaluation of validity and educational impact.

Framework	Parameter	Definition	Examples	Rating
Modern concept of validity—Messick	Content	Test items are relevant and representative of the intended construct	Using expert opinions to ensure all domains are accurately represented	N = no discussion of source of validity evidence 0 = discussion of source of validity but no data presented 1 = data weakly supports source of validity or is limited 2 = data strongly supports source of validity
	Response processes	Thought processes and actions of subjects and observers are made in accordance with the intended construct	Quality control of assessments, such as in standardising test administration and minimising examiner bias
	Internal structure	Test scores across tasks can be reliably reproduced	Calculating inter-item reliability and test-retest reliability
	Relations to other variables	Test scores correlate with external, independent measures which share a theoretical relationship	Comparing scores between groups with different levels of experience in the tested skill
	Consequences	The impact of using the assessment	Determining the pass-fail score and considerations for the subject on obtaining a pass or fail
Translational outcomes of simulation-based learning (adapted)—McGaghie et al.	Internal acceptability	The trainee’s satisfaction with using the simulator	Favourable responses from feedback forms or post-training survey questionnaires	Level 1
	Contained effects	Changes in performance in the simulation context	Development of knowledge and/or skills as measured by the simulator tool	Level 2
	Downstream effects	Behavioural changes in the clinical context	Transfer of knowledge/skills to clinical practice	Level 3
	Target effects	Direct changes to patient outcomes	Reduced rates of surgical complications	Level 4
	Collateral effects	Changes on a wider, systemic level	Cost saving; skill retention	Level 5

Results

A total of 3989 articles were screened, of which 3751 were excluded following abstract review. After reading the remaining 238 articles in full, a further 107 were excluded. A total of 131 original articles were included in this systematic review (Fig. 1). Details of findings are summarised in Tables 2–5 according to simulator type.

Table 3.

Wet-lab studies.

Model	Description	Reference	Area of training	Training task	Study design	Participants	Training time	Validity	Effectiveness
Rabbit eyes + human cataracts	Human cataract removed in its capsule and implanted into a rabbit eye	Tolentino and Liu [79]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eye	External tissue of a post-mortem porcine eye removed then placed in a microwave oven to induce cataract	van Vreeswijk and Pameyer [80]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes	Pig eyes filled with cooked chestnuts of varying hardness as pseudonuclei	Mekada et al. [71]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 1 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	2
Pig eyes	A range of formalin: alcohol ratios tested on pig eyes to simulate human lens	Sugiura et al. [78]	Cataract surgery	Not specified	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Goat eyes	Goat eyes injected with formalin and fixed on a stand	Dada and Sindhu [65]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes	Post-mortem pig eye injected with formalin and hydroxyethylcellulose to induce cataract	Hashimoto et al. [67]	Cataract surgery	Capsulorhexis	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Goat eyes	Goat eyes injected with formalin through the pars plana before capsulorhexis vs through a clear corneal side port into the nucleus after capsulorhexis	Sudan et al. [77]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes	Anterior chamber of pig eyes filled 75% with methylcellulose then injected with a formaldehyde-methanol solution to induce cataract	Saraiva and Casanova [75]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes + electronic sensor	Cup supporting an ex vivo human or porcine eye mounted on a 6 axis/torque sensor which detects direction and magnitude of forced applied by trainee	Leuschke et al. [69]	Cataract surgery	Not specified	Descriptive	n = 2 (ophthalmic surgeons)	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Rabbit eyes	Lens from enucleated rabbit eyes fixed with varying concentrations of paraformaldehyde	Ruggiero et al. [74]	Cataract surgery	Capsulorhexis	Experimental	n = 6 (cataract surgeons)	Not specified	Content: 2 Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	1
Goat eyes + human lens	Human cataractous nuclear implanted into a goat lens and mounted on rectangular polystyrene	Sengupta et al. [76]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes	Assessment of wet-lab performance using a modified surgical assessment tool (ICO- OSCAR)	Farooqui et al. [66]	Cataract surgery	Phacoemulsification	Pilot study	n = 12 (3rd year residents)	5 days	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes	Post-mortem human eyes with Karnovsky solution to induce cataract	Pandey et al. [72]	Cataract surgery	Not specified	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes	Medical lubricating jelly injected into in a human cadaver eye	Liu et al. [70]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Sheep + human lens	Human cataractous lens nucleus implanted in a sheep eye lens	Kayikcioglu et al. [68]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes	Survey of wet-lab training with pig eyes on residents’ perceived preparedness and difficulty with cataract surgery	Puri et al. [73]	Cataract surgery	Unspecified	Retrospective cross-sectional study	n = 116 (residents)	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	1
Rabbit eye	Rabbit eye used as a replacement for human eye	Abrams et al. [81]	Vitreoretinal surgery	Pars plana vitrectomy	Descriptive	N/A	2 h	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human/artificial eyes + Marty the Surgical Simulator	3 model systems: artificial eye with a plastic head model; human cadaver eye with the head model; and human cadaver eye without the model	Patel and Levin [84]	Glaucoma surgery	Goniotomy	Case series study	n = 4 (paediatric ophthalmology fellows)	2 h	Content: 0 Response processes: 2 Internal structure: N Relations to other variables: N Consequences: N	1
Pig eyes	Pig eyes soaked in 10% formaldehyde then mounted on a dummy head	Lee et al. [83]	Glaucoma surgery	Trabeculectomy	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes	Human donor cadaveric eyes with contact lens inserted into a surgical model mannequin head	Patel and Sit [85]	Glaucoma surgery	Trabeculectomy	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes + artificial orbit	Enucleated pig eyes placed into the orbit of a styrofoam model head; microsphere-based canalograms used to measure extent of outflow tract access	Dang et al. [82]	Glaucoma surgery (microincisional)	Ab-interno trabeculectomy (trabectome)	Case series	n = 7 (ophthalmology trainees)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	2
Human eyes	Human cadaveric corneoscleral rims used for angle surgery simulation	Arora et al. [86]	Glaucoma surgery	Microinvasive glaucoma surgery	Descriptive study	N/A	N/A	Content: 1 Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	N/A
Human eyes	Human cadaver corneoscleral rims fixated with a tack through the centre of the cornea to a styrofoam base	Nazarali et al. [87]	Glaucoma surgery	Bimanual skills with gonioscopy, microbypass stent insertion + removal, gonioscopy-assisted transluminal trabeculotomy	Experimental, feasibility study	n = 10 (ophthalmic residents)	Not specified	Content: 1 Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	1
Human eye + artificial anterior chamber	Human donor corneoscleral button placed over an artificial anterior chamber	Fontana et al. [89]	Corneal surgery	Deep Anterior Lamellar Keratoplasty using the big-bubble technique	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes	Pseudo-grafts created from lens capsule of enucleated porcine eyes and implanted into an intact globe	Droutsas et al. [88]	Corneal surgery	Descemet Membrane Endothelial Keratoplasty (DMEK)	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human cornea + artificial anterior chamber	One human cornea for donor graft preparation + one for practising graft insertion and unfolding in an artificial anterior chamber model	Vasquez Perez and Liu [90]	Corneal surgery	Descemet membrane endothelial keratoplasty (DMEK)	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes + artificial anterior chamber	Human corneas mounted on an artificial anterior chamber with a 3D-printed iris. Intraoperative OCT used to validate each step of the procedure.	Famery et al. [91]	Corneal surgery	Descemet membrane endothelial keratoplasty	Prospective, feasibility study	n = 5 (ophthalmic surgeons)	2 sessions (duration unspecified)	Content: 1 Response Process: N Internal Structure: N Relations to other variables: 2 Consequences: N	1
Pig eyes	Cadaveric pig eyes with bacon as extraocular muscles	White et al. [92]	Strabismus surgery	Steps for strabismus surgery	Case series	30 Residents	Not specified	Content: 0 Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	1
Pig eyes + chicken breast model	Wet-lab session using a chicken breast model for practice, followed by pig eyes	Vagge et al. [93]	Strabismus surgery	Partial-thickness scleral suture passes	Prospective cohort pilot study	n = 12 (8 first year and 4 second year residents)	2 h	Content: 0 Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	2
Pig eyelid	A rubber ball used to simulate the globe; a board with 4 metal screws mimicking the canthal tendons and arcus marginalis. Corners of a pig eyelid then sutured to the screws.	Pfaff [95]	Oculoplastic surgery	Eyelid margin repair	Descriptive	Oculoplastic staff and fellow, residents (numbers not specified)	Not specified	Content: 2 Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig head	Pig head split in half and rested on a surface for practising lid procedures	Kersey [94]	Oculoplastic surgery	Unspecified	Descriptive	Ophthalmologists of varying grades (numbers not specified)	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	1
Pig eyelids	Pig eyelids with surgically induced ptosis	Zou et al. [96]	Oculoplastic surgery	Ptosis repair	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes	Lecture on lateral cantholysis followed by video-demonstration, live demonstration on human cadaver eyes by an oculoplastic surgeon and practice on the same eyes	Patel et al. [97]	Oculoplastic surgery	Lateral cantholysis	Prospective study + survey	n = 12 (residents)	Not specified	Content: 1 Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	2
Sheep cranium	Intracranial and ocular dissection of 1-week-old sheep cranium	Altunrende et al. [98]	Orbital surgery	Micro-surgical skills	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes	Iron particles placed on cadaver cornea for rust ring formation before mounting on slit lamps. Removal of rust ring photographed and analysed using open-source computer software programme	Mednick et al. [99]	Ocular trauma	Corneal rust ring removal	Prospective	n = 22 (8 medical students, 10 residents, 4 attending ophthalmologists)	Not specified	Content: 1 Response processes: N Internal structure: N Relations to other variables: 0 Consequences: N	1
Goat eyes + artificial model head	Enucleated goats’ eyes are mounted on a model head. An incision is made using a scalpel along the corneoscleral limbus, simulating a full-thickness laceration.	Pujari et al. [100]	Ocular trauma	Corneoscleral perforation repair	Descriptive study	N/A	N/A	Content: 0 Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes + artificial orbit	Enucleated porcine eyes placed inside a metal orbit created using an adjustable eye support, cylinder and removable ring	Uhlig and Gerding [101]	Diagnostic examination	Direct and indirect ophthalmoscopy; gonioscopy	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes + formalin	Human autopsy eyes with cornea cleared with hyperosmotic dextran solution and fixed with formalin	Auffarth et al. [102]	General Ophthalmic Surgery	Not specified	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes + contact lens	Cyanoacrylate glue used to secure polymethylmethacrylate contact lens to the corneal rim of cadaver eyes	Lenart et al. [103]	General Ophthalmic Surgery	Not specified	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eyes with keratoprosthesis	Lander wide-field keratoprosthesis placed over cadaver eyes	Borirak-chanyavat et al. [104]	Anterior and posterior segment surgeries	Phacoemulsification, vitrectomy, panretinal laser	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Human eye + Spring-action Apparatus for Fixation of Eyeball (SAFE)	Hollow iron cylinder attached to a spring-action syringe forms a vacuum for fixation of human/animal cadaveric eyes	Ramakrishnan et al. [105]	Anterior and posterior segment surgeries	Various procedures (e.g., phacoemulsification, MSICS, LASIK, DALK, DSEK and trabeculectomy)	Descriptive	n = 2 (ophthalmic surgeons)	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	1
Sheep eyes	Sheep eyes mounted on an artificial orbit	Mohammadi et al. [106]	Anterior segment surgery	Range of anterior segment procedures (e.g., capsulorhexis; keratoplasty; trabeculectomy)	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes	Porcine eyes placed in an ocular bulb holder that is secured to a polyvinylchloride pillar on a modified polystyrene head	Porrello et al. [107]	Anterior and posterior segment procedures	Laser iridotomy, photocoagulation and all steps of cataract surgery	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes + SS Microscope-Integrated OCT (MI-OCT)	Real time 3D imaging to aid wet-lab microsurgery training	Todorich et al. [108]	Anterior segment surgery	Corneal suture passes and laceration repair	Randomised controlled study (with crossover)	n = 14 (6 first year, 4 second year and 4 third year residents)	Not specified	Content: N Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	2
Pig eyes	Micro-surgical skills course using pig eye models and a video-based scoring system for assessment	Ezra et al. [109]	General ophthalmic surgery	Micro-surgical skills	Prospective longitudinal cohort study	n = 14 (residents)	1 day	Content: N Response processes: N Internal structure: 2 Relations to other variables: 2 Consequences: N	2
Pig eyes and foot (ESSAT)	3-station wet-lab course: pig’s foot inserted with red plastic tubing to simulate temporal artery biopsy; pig eyes for muscle recession; pig eyes for cataract procedures.	Fisher et al. [110]	Ophthalmic surgery (a range of different areas)	Temporal artery biopsy, muscle resection and phacoemulsification	Survey	n = 22 (content experts: residency programme directors and faculty members)	N/A	Content: 2 Response processes: 1 Internal structure: N Relations to other variables: N Consequences: N	N/A
Pig eyes and foot (ESSAT)	Same as above	Taylor et al. [111]	Ophthalmic surgery (a range of different areas)	Temporal artery biopsy, muscle resection and phacoemulsification	Masked, prospective study	n = 29 (1 first year resident, 1 third year resident and 27 content experts)	N/A	Content: N Response processes: N Internal structure: 2 Relations to other variables: 2 Consequences: 2	N/A

Table 4.

Dry-lab studies.

Model	Description	Reference	Area of training	Training task	Study design	Participants	Training time	Validity evidence	Effectiveness
Aluminium foil with methacrylate support	Methacrylate for hand support, PVC sheet and aluminium foil for performing capsulorhexis on	Abellan et al. [112]	Cataract surgery	Capsulorhexis	Randomised controlled trial	n = 65 (ophthalmologists)	2 h	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	2
Japanese quail eggs	Sharp end of a quail egg cut and fitted with a silicone sclerocorneal cap; the yolk and albumen simulate the vitreous body and the inner eggshell membrane simulates the internal limiting membrane	Hirata et al. [113]	Vitreoretinal surgery	Membrane peeling	Case series	n = 8 (3 experienced vitreous surgeons and 5 inexperienced surgeons)	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	2
VitRet eye with fluid	An artificial eye model filled with vitreous-like fluid made of dairy creamer + balanced saline	Yeh et al. [114]	Vitreoretinal surgery	Three-port vitrectomy setup; intraocular tasks (e.g., core vitrectomy and membrane peel); wound closure	Case series	n = 13 (8 residents and 5 fellows)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	1
Artificial orbit with diascleral illumination	Eye support made from transparent polymethylmethacrylate, fitted onto cylinder and fixed with a metal ring	Uhlig and Gerding [115]	Vitreoretinal surgery	Not specified	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Reusable rubber eye	Rubber globe with removable plastic anterior segment for access to posterior segment; rubber bands used to simulate the rectus muscles and a coat of liquid skin bandage applied to simulate the membrane	Iyer and Han [116]	Vitreoretinal surgery	Epiretinal membrane peeling	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Medium-fidelity model	Wooden frame to simulate patient’s forehead; a table tennis ball to simulate the globe; pre-equatorial holes created light source insertions; tasks performed using real instruments and foot-pedal-controlled microscope	Rice et al. [117]	Vitreoretinal surgery	Sets of exercises including training single hand and bimanual dexterity	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
ILM peeling simulator	Artificial eye model placed in an ocular surgery simulator and an artificial ILM made using a PVA hydrogel. ILM peeling can be simulated under water.	Omata et al. [118]	Vitreoretinal surgery	Inner limited membrane (ILM) peeling	Descriptive study	N/A	N/A	Content: 0 Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	N/A
Nonbiologic Strabismus Surgery Simulator	Components include a rubber ball mounted to on a wooden based simulate the globe. An elastic band is attached to the eyeball, simulating the rectus muscle and a small piece of latex is attached to the eyeball with a thumbtack to simulate the conjunctiva and cornea.	Adebayo et al. [119]	Strabismus surgery	Steps for strabismus surgery	Randomised controlled trial	n = 41 (1st and 2nd year medical students)	1 week	Content: 1 Response Process: 2 Internal Structure: 2 Relations to other variables: N Consequences: N	2
Simulator for practising laser procedures	A model eye with artificial tissues	Simpson et al. [122]	Laser procedures	Peripheral iridotomy, posterior capsulotomy and laser retinopexy	Case series	n = 13 (6 inexperienced and 7 experienced residents)	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	N/A
Capsulotomy simulator	An adjustable artificial anterior chamber for fitting laser instrument; an intraocular lens coated with a crust at the posterior surface to simulate posterior capsule opacification	Moisseiev and Michaeli [121]	Laser procedures	Neodymium: YAG posterior capsulotomy	Descriptive	n = 3 (residents)	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	2
RETILAPP eye model	A ping pong ball simulating the globe is cut in half; a paper diagram of the lesion is cut out and placed between the two hemispheres of the globe; the eye is clamped to a slit lamp; contact lens is used over the model for practising lasers	Ganne et al. [120]	Laser procedures	Retinal laser photocoagulation	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
MIRA practice eye	Model eye stabilised onto a wooden mount; the optic nerve and fundal pattern are painted onto the globe for orientation	Weidenthal [123]	Laser procedures	Laser photocoagulation	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Child skull model	Anatomically correct model of the nasolacrimal duct drainage	Coats [124]	Oculoplastic surgery	Not specified	Descriptive	N/A	N/A	Content: 0 Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	N/A
3D-printed orbit models	Use of 3D printing to produce orbit models that replicate a patient’s bony anatomy for use in orbital surgical training	Scawn et al. [126]	Orbital surgery	Orbital decompression	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
3D-printed copies of human cadaveric orbital dissections	Surface mesh of orbit prosections created, processed using 3D laser scan, then printed	Adams et al. [125]	Orbital surgery	Not specified	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Newport eye corneal foreign body training phantom	A polyvinyl and gelatine-based model with resin used to secure a craft eye inside a plastic container; ground black pepper used to simulate a foreign body	Marson and Sutton [127]	Ocular Trauma	Corneal foreign body removal	Case series	n = 23 (6 ophthalmologists, 11 ED physicians and 2 ophthalmology + 4 ED nurse practitioners)	N/A	Content: 1 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	1
EYE Exam Simulator (Kyoto Kagaku Co.)	A mannequin head with adjustable pupil sizes and a holder to place slides showing different retinal conditions; a standard ophthalmoscope is used to simulate fundoscopy examination	McCarthy et al. [132]	Diagnostic examination	Direct ophthalmoscopy	Case series	n = 43 (32 emergency medicine and 11 ophthalmology residents)	N/A	Content: N Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	N/A
EYE Exam Simulator (Kyoto Kagaku Co.)	Same as above	Akaishi et al. [128]	Diagnostic examination	Direct ophthalmoscopy	Cross-sectional	n = 73 (3 medical students, 41 residents, 29 attending physicians)	Not specified	Content: N Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	N/A
Toy model eyes	Toy eyes cut around the pupil edge and everted; partial-thickness cuts made to simulate retinal tears; eye re-inverted and mounted on a wooden base; a 90-dioptre lens is mounted in the pupil and fixed with tape	Chew and Gray [129]	Diagnostic examination	Indirect ophthalmoscopy with scleral indentation	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Rubber ball eye	Eye made from rubber ball is cut in half and retinal details drawn on a painted orange background before sticking the 2 halves together; eyeball is inserted into a paperpulp head model	Kumar and Shetty [130]	Diagnostic examination	Indirect ophthalmoscopy	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Glass vial	Screw-top glass vial filled with mouthwash and face powder to simulate presence of cells and flare in the anterior chamber; holding the vial at different angles and positions in front of a slit lamp simulates appearance of an optical section and variations in thickness of the cornea	Morris [134]	Diagnostic examination	Slit-lamp examination	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Mannequin head model	Vacuum tubes with glue applied to the curved ends inserted into styrofoam mannequins to imitate slit lamp appearance of the anterior segment, flare and cells, hypopyon, hyphema, red reflex, cataract and corneal epithelial defects	Romanchuk [135]	Diagnostic examination	Slit-lamp examination	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Glass eyeball	A glass marble eye set onto a small bottle cap for stabilisation; piece of paper with letters placed behind the marble to assess visualisation; a hole punched in a separate piece of paper to simulate the pupil	Lewallen [131]	Diagnostic examination	Indirect ophthalmoscopy	Descriptive	N/A	N/A	Content: N Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Origami model	A sheet of letter paper with a retinal drawing or photograph on one side is folded into a box with a small aperture that acts as a pupil	Miller [133]	Diagnostic examination	Binocular indirect ophthalmoscopy	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Model for simulating indirect ophthalmoscopy and retinal photocoagulation	Model consists of a 60D lens, a bulb syringe to simulate the globe, card paper for the iris and a printed fundus photograph attached to the base	Kylstra and Diaz [136]	Diagnostic examination and laser procedures	Binocular indirect ophthalmoscopy and indirect laser retinal photocoagulation	Descriptive study	N/A	N/A	Content: 1 Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	N/A

Table 2.

Virtual reality studies.

Model	Description	Reference	Area of training	Training task	Study design	Participants	Training time	Validity	Effectiveness
Eyesi Surgical	Hardware: mannequin head; artificial eye with CCD camera; operative microscope; set of surgical instruments; foot pedals; touchscreen Software: VR platform; cataract and vitreoretinal modules; storage of performance metrics	Feudner et al. [14]	Cataract surgery	Forceps, anti-tremor and capsulorhexis	Randomised controlled trial	n = 63 (32 residents, 31 medical students)	3 weeks (±5 days)	Content: 0 Response processes: 2 Internal structure: 2 Relations to other variables: 0 Consequences: N	2
Eyesi Surgical	Same as above	Solverson et al. [27]	Cataract surgery	Phacoemulsification	Uncontrolled	n = 25 (18 residents, 7 ophthalmic surgeons)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	2
Eyesi Surgical	Same as above	Ahmed et al. [11]	Cataract surgery	Not specified	Cross-sectional survey	n = 56 (ophthalmology residency programme directors)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	2
Eyesi Surgical	Same as above	Privett et al. [22]	Cataract surgery	Capsulorhexis	Prospective, comparative case series	n = 23 (7 surgeons, 16 medical students and residents)	Not specified	Content: 0 Response processes: 0 Internal structure: N Relations to other variables: 1 Consequences: N	1
Eyesi Surgical	Same as above	Belyea et al. [12]	Cataract surgery	Phacoemulsification	Retrospective case-control study	n = 42 (3rd year residents)	Not specified	Content: 0 Response processes: N Internal structure: 1 Relations to other variables: N Consequences: 1	4
Eyesi Surgical	Same as above	Le et al. [16]	Cataract surgery	Forceps, anti-tremor and capsulorhexis	Multi-centre cross-sectional study	n = 65 (4 medical students, 4 technicians, 36 residents, 3 fellows, 18 staff ophthalmologists)	20 min	Content: N Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	N/A
Eyesi Surgical	Same as above	Nathoo et al. [20]	Cataract surgery	Forceps and anti-tremor	Retrospective cohort study	n = 10 (5 junior + 5 senior residents with no previous simulator use)	14 months	Content: N Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	2
Eyesi Surgical	Same as above	Selvander and Asman [25]	Cataract surgery	Navigation and capsulorhexis	Randomised uncontrolled trial	n = 35 (medical students)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	2
Eyesi Surgical + wet lab	Intensive training programme involving wet lab and Eyesi simulator experience	Baxter et al. [38]	Cataract surgery	Not specified	Case series	n = 3 (3rd year ophthalmology trainees)	Eyesi Surgical = 50 h; wet-lab not specified	Content: 2 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	4
Eyesi Surgical	Same as above	Daly et al. [13]	Cataract surgery	Capsulorhexis	Randomised controlled trial	n = 21 (residents—10 trained in the wet lab vs 11 on the simulator)	Not specified	Content: 0 Response processes: 0 Internal structure: N Relations to other variables: 1 Consequences: 1	3
Eyesi Surgical	Same as above	Li et al. [18]	Cataract surgery	N/A	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Eyesi Surgical	Same as above	Pokroy et al. [21]	Cataract surgery	Phacoemulsification	Retrospective cohort study	n = 20 (residents)	≥6 h training (mean = 21.2 h)	Content: N Response processes: N Internal structure: N Relations to other variables: 1 Consequences: 1	4
Eyesi Surgical	Same as above	Saleh et al. [24]	Cataract surgery	Navigation, anti-tremor, bimanual, cracking + chopping, capsulorrhexis	Prospective	n = 18 (1st year ophthalmic trainees)	3 repeats (session duration not specified)	Content: N Response processes: N Internal structure: 2 Relations to other variables: 1 Consequences: 0	2
Eyesi Surgical	Same as above	Selvander and Asman [26]	Cataract surgery	Capsulorhexis, hydromaneuvers and phacoemulsification	Uncontrolled	n = 24 (7 cataract surgeons, 17 medical students)	Not specified	Content: 0 Response processes: N Internal structure: 0 Relations to other variables: 1 Consequences: N	N/A
Eyesi Surgical	Same as above	Spiteri et al. [28]	Cataract surgery	Forceps, anti-tremor, capsulorhexis and phacoemulsification	Uncontrolled	n = 30 (10 novice, 10 intermediate, 10 experienced surgeons)	2 sessions an hour apart	Content: N Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	N/A
Eyesi Surgical	Same as above	Thomsen et al. [30]	Cataract surgery	Phacoemulsification (all modules except chopping)	Uncontrolled	n = 42 (26 ophthalmic trainees, 16 ophthalmic surgeons)	≤2 h	Content: 1 Response processes: N Internal structure: 2 Relations to other variables: 1 Consequences: 1	2
Eyesi Surgical	Same as above	Gonzalez-Gonzalez et al. [15]	Cataract surgery	Capsulorhexis	Prospective, comparative case series	n = 14 (3 attending physicians, 11 trainees)	Not specified	Content: N Response processes: N Internal structure: N Relations to other variables: 0 Consequences: N	2
Eyesi Surgical	Same as above	Li et al. [18]	Cataract surgery	N/A	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Eyesi Surgical	Same as above	McCannel [19]	Cataract surgery	Capsulorhexis	Retrospective case-control study	n = 38	Within a span of 4 years	Content: 0 Response processes: N Internal structure: 0 Relations to other variables: 1 Consequences: 1	3
Eyesi Surgical	Same as above	Roohipoor et al. [23]	Cataract surgery	Anti-tremor, bimanual, capsulorhexis, forceps and navigation training	Retrospective cohort study	n = 30 (residents)	≤3 months	Content: 0 Response processes: N Internal structure: 0 Relations to other variables: 1 Consequences: 1	2
Eyesi Surgical	Same as above	Thomsen et al. [31]	Cataract surgery	Navigation, anti-tremor, forceps, bimanual, capsulorhexis, divide and conquer	Cross-sectional study	n = 11 (surgeons)	1 h warm up before assessment	Content: 0 Response processes: N Internal structure: 0 Relations to other variables: 2 Consequences: N	N/A
Eyesi Surgical	Same as above	Bozkurt et al. [37]	Cataract surgery	Navigation, forceps, bimanual, anti-tremor, capsulorhexis	Prospective cohort study	n = 16 (ophthalmic residents + faculty members)	Not specified	Content: 1 Response Process: 1 Internal Structure: N Relations to other variables: 1 Consequences: 0	2
Eyesi Surgical	Same as above	Staropoli et al. 29]	Cataract surgery	Phacoemulsification	Retrospective case series	n = 22 (3rd year residents)	Within span of 3 years	Content: 0 Response processes: 0 Internal structure: N Relations to other variables: N Consequences: 2	4
Eyesi Surgical	same as above	Ng et al. [33]	Cataract surgery	Navigation, anti-tremor, capsulorhexis, cracking + chopping	Cross-sectional, multi-centre study	n = 19 (ophthalmic trainees)	4 weeks	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	3
Eyesi Surgical	Same as above	Colné et al. [34]	Cataract surgery	Irrigation + aspiration, capsulorhexis, cracking	Prospective study	n = 18 (12 residents + 6 cataract surgeons)	Not specified	Content: 1 Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	N/A
Eyesi Surgical	Same as above	Ferris et al. [35]	Cataract surgery	Cataract training modules (unspecified)	Retrospective cohort study	n = 265 (1st and 2nd year trainees)	N/A	Content: 1 Response Process: N Internal Structure: N Relations to other variables: 1 Consequences: 2	5
Eyesi Surgical	Same as above	La Cour et al. [36]	Cataract surgery	Eyesi cataract modules	Prospective, uncontrolled study	n = 19 (cataract surgeons)	Mastery learning (time taken for the trainee to reach a pre-defined pass score)	Content: 1 Response Process: 1 Internal Structure: 2 Relations to other variables: 1 Consequences: 1	3
Eyesi Surgical	Same as above	Lucas et al. [32]	Cataract surgery	Cataract training modules (unspecified)	Retrospective cohort study	n = 14 (2nd year residents)	Not specified	Content: 1 Response Process: N Internal Structure: N Relations to other variables: N Consequences: 2	4
Eyesi Surgical	Same as above	Rossi et al. [41]	Vitreoretinal surgery	Navigation and membrane peeling	Prospective, comparative case series	n = 44 (6 medical student, 24 residents, 14 vitreoretinal surgeons)	Not specified	Content: N Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	2
Eyesi Surgical	Same as above	Park et al. [40]	Vitreoretinal surgery	Navigation, forceps, anti-tremor and vitrector	Prospective cohort study	n = 14 (12 residents, 1 medical retina fellow, 1 vitreoretinal surgeon)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	2
Eyesi Surgical	same as above	Koch et al. [39]	Vitreoretinal surgery	Not specified	Cross-sectional survey	N = 156 (108 residents, 48 ophthalmologists with more experience)	Not specified	Content: 0 Response processes: 1 Internal structure: N Relations to other variables: N Consequences: N	1
Eyesi Surgical	Same as above	Vergmann et al. [42]	Vitreoretinal surgery	Navigation, forceps, bimanual, laser coagulation, posterior hyaloids, membrane peeling	Prospective	n = 35 (20 medical students, 10 residents, 5 surgeons)	2 sessions with up to 2 weeks apart	Content: 1 Response processes: 1 Internal structure: N Relations to other variables: 1 Consequences: N	2
Eyesi Surgical	Same as above	Cissé et al. [43]	Vitreoretinal surgery	Navigation, forceps, vitrector, epiretinal membrane peeling	Prospective study	n = 21 (15 residents + 6 VR surgeons)	2 × 60-min sessions	Content: 1 Response Process: 1 Internal Structure: N Relations to other variables: 1 Consequences: N	N/A
MicroVisTouch	Hardware: mannequin head; blunt-tipped handpiece; robotic arm; footpedals Software: VR platform; haptic feedback interface; cataract surgery modules	Banerjee et al. [44]	Cataract surgery	Capsulorhexis	Prospective	n = 8 (4th year residents)	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	N/A
MicroVisTouch	Same as above	Sikder et al. [45]	Cataract surgery	Capsulorhexis	Prospective	n = 78 (residents)	6 months	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	N/A
MicroVisTouch	Same as above	Kozak et al. [46]	Vitreoretinal surgery	Epiretinal membrane + internal limiting membrane peeling procedures	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
PhacoVision	A personal computer with 3D visual interface, phacoemulsification handpiece, a nucleus manipulator and foot pedals for control of the phacoemulsification procedure and microscope adjustments	Laurell et al. [54]	Cataract surgery	Phacoemulsification	Experimental	n = 7 (medical students + ophthalmic surgeons)	Not specified	Content: 0 Response processes: 0 Internal structure: N Relations to other variables: N Consequences: N	1
Phantom Phaco-simulator	Simulator with Phantom haptic device	Agus et al. [52]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Cataract surgery stimulator	Low-cost simulator using computer-based algorithms for tissue deformation, surface cutting and volume sculpting; two-handed device with six degrees-of-freedom for human–computer interactions	Choi et al. [53]	Cataract surgery	Phacoemulsification	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Pars plana vitrectomy simulator	A vitrectomy probe and handpiece of an intraocular illumination probe tracked by CCD cameras within a mechanical eye, housed inside a mannequin head	Jonas et al. [56]	Vitreoretinal surgery	Pars plana vitrectomy	Descriptive	n = 14 (residents and medical students)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	2
Sophocle	Binocular microscope with a slit lamp and 3D translation controlled by a swingle bar	Peugnet et al. [58]	Vitreoretinal surgery	Retinal photocoagulation	Randomised controlled trial	n = 10 (residents)	N/A	Content: 0 Response processes: N Internal structure: 0 Relations to other variables: N Consequences: N	3
VR surgery simulator	3D position tracking stylus, Pentium II desktop, Open GL and Microsoft Visual C + + languages to control the interaction and update the visual feedback tracking the instruments	Verma et al. [59]	Vitreoretinal surgery	Unspecified	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Vitrectomy simulator	Computer software with special hardware.	Neumann et al. [57]	Vitreoretinal surgery	Vitrectomy	Descriptive	N/A	N/A	Content: 1 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Vitreous surgery simulator	High-resolution colour stereo binoculars, haptic devices, foot switches and a high-speed graphics computer	Hikichi et al. [55]	Vitreoretinal surgery	Vitrectomy	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Endoscopic Endonasal Surgery Simulator (EESS)	VR software to convert endoscope and surgical instrument to a video display that can be simultaneously seen by instructor and trainee.	Weiss et al. [60]	Endoscopic endonasal surgery	Endoscopic navigation, endonasal injection and middle turbinate medialization	Randomised controlled trial	n = 15 (residents)	5 h	Content: 0 Response processes: N Internal structure: 1 Relations to other variables: N Consequences: N	2
Eye surgery simulator	High-speed computer graphics workstation, a stereo operating system, a wrist rest and a position tracking stylus connected to force feedback motors	Sinclair et al. [62]	General ophthalmic surgery	Unspecified	Descriptive	N/A	N/A	Content: 1 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Micro-surgical robot	A virtual environment; micro-surgical master and slave; mannequin	Hunter et al. [61]	General ophthalmic surgery	Unspecified	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Ophthalmic Retrobulbar Injection Simulator (ORIS)	Use of QuickTime to create digital video sequences for instructing residents on retrobulbar injection; the user can control the viewing angles and video sequence using controls on the screen.	Merril et al. [63]	Ophthalmic anaesthesia	Retrobulbar injection	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
Ocular ultrasound using VR software Blender	A 3D virtual model built using open-source software used to generate movie clips to simulate different movements and orientations of an ocular ultrasound scanner head.	Mustafa et al. [64]	Ocular Ultrasound	Imaging	Descriptive	N/A	N/A	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	N/A
EyeSi Direct Ophthalmoscope	Simulator consists of an ophthalmoscope handpiece with built-in display, a patient model head and a PC with touchscreen. Performance metrics for different components of the examination are calculated and recorded	Borgersen et al. [48]	Fundoscopy examination	Direct ophthalmoscopy	Prospective validation study	n = 21 (13 medical students; 8 ophthalmic consultants)	Not specified	Content: 1 Response Process: 2 Internal Structure: 2 Relations to other variables: 2 Consequences: 1	N/A
EyeSi Direct Ophthalmoscope	Same as above	Boden et al. [49]	Fundoscopy examination	Direct ophthalmoscopy	Randomised, controlled study	n = 34 (medical students)	Not specified	Content: 1 Response Process: 1 Internal Structure: N Relations to other variables: N Consequences: 0	2
Eyesi Indirect Ophthlamoscope	Simulator consists of diagnostic lenses, a model patient head and an ophthalmoscope headband with mounted stereo display, showing a 3D virtual patient and virtual lens when the trainee’s hand is placed over the patient’s eyes. Software comes with a range of patient cases and pathologies	Chou et al. [50]	Fundoscopy examination	Indirect ophthalmoscopy	Prospective	n = 42 (25 medical students, 17 trainees)	Not specified	Content: 0 Response processes: N Internal structure: N Relations to other variables: 1 Consequences: N	1
Eyesi Indirect Ophthlamoscope	Same as above	Loidl et al. [51]	Fundoscopy examination	Indirect ophthalmoscopy	Prospective study + survey	n = 292 (medical students)	1 week	Content: N Response Process: N Internal Structure: N Relations to other variables: N Consequences: N	1

Table 5.

E-learning studies.

Model	Description	Reference	Area of training	Training task	Study design	Participants	Training time	Validity	Effectiveness
Computer-Assisted Learning Ophthalmology Programme	(1) A software programme delivering a multi-media tutorial for learning about the pupillary light reflex (2) A Macintosh-connected mannequin model housing motor-driven camera diaphragms to simulate pupil response to the swinging flashlight test (3) A multiple-choice quiz to test the user’s understanding of the material	Kaufman and Lee [137]	Diagnostic examination	Swinging flashlight test	Evaluation study	n = 29 (medical students)	2 weeks	Content: 2 Response processes: 2 Internal structure: N Relations to other variables: N Consequences: N	1
E-learning modules	Case presentations based on an interactive Q&A game format	Stahl et al. [138]	Ophthalmology education	Ophthalmology-related patient cases	Prospective study	n = 272 (medical students)	2 terms	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: 1	2
Ophthalmic Operation Vienna	(1) Surgical videos accompanied by 3D animations of 5 surgical procedures: cataract, glaucoma, retinal detachment, vitrectomy and ablative refractive surgery (2) A multiple-choice test on cataract and glaucoma surgery topics	Prinz et al. [139]	Anterior and posterior segment surgery	Cataract and glaucoma surgery knowledge	Randomised controlled trial	n = 172 (medical students)	2 weeks	Content: 0 Response processes: N Internal structure: 2 Relations to other variables: N Consequences: N	2
3D computer animations	Software and hardware systems used to create 3D animations (e.g., Cinema 4D XL Studio Bundle to create the ocular muscles) into interactive computer programmes to simulate eye movements, pathologies and neuro-surgical techniques	Glittenberg and Binder [140]	Ophthalmology education	Neuro-ophthalmology and oculomotor anatomy knowledge	Case-control comparative study	n = 40 (medical students)	~45 min lecture presentations	Content: 0 Response processes: N Internal structure: N Relations to other variables: N Consequences: N	2
Virtual Mentor	An interactive, computer-based simulation of the cognitive components of performing hydrodissection	Henderson et al. [141]	Cataract surgery	Hydrodissection	Single-masked, randomised controlled trial	n = 68 (residents)	Not specified	Content: 2 Response processes: N Internal structure: 0 Relations to other variables: 1 Consequences: N	2

Virtual reality

Eyesi Surgical

The Eyesi Surgical (VRmagic, Mannheim, Germany) is a high-fidelity virtual reality simulator designed for practising intraocular procedures. It consists of a mannequin head that houses a model eye connected to a computer interface and an operating microscope. The movements and positions of surgical instruments are tracked by internal sensors, producing a virtual image that is viewed through the microscope, as well as on separate touchscreen. The software contains training modules that simulate different steps in cataract and vitreoretinal surgeries. The system records performance metrics, enabling scores and feedback to be generated [11]. Of all virtual reality simulator models developed for use in ophthalmology training, the Eyesi has been the most extensively assessed, with a total of 33 validity studies.

Cataract surgery

[Summary: content = 2; response processes = 1; internal structure = 2; relations to other variables = 2; consequences = 2; translational outcomes = level 5].

Twenty-eight studies assessed the Eyesi cataract training modules, collectively demonstrating all five sources of validity evidence, with data strongly supporting each parameter (score = 2) except for response processes, which had more limited evidence (score = 1) [11–38]. A randomised controlled trial (RCT) by Feudner et al. showed that those who trained with the Eyesi achieved significant improvements in their capsulorhexis performance in the wet lab compared with the no-training, control group [14]. Another RCT suggested that virtual reality training was comparable to training using wet lab [13]. Residents were assessed on their first capsulorhexis in the operating room following either Eyesi or web-lab training. Overall technical scores were equivalent. The study also provided evidence of predictive validity with a direct correlation between time taken to complete the training modules on the Eyesi and true operating room time, as well as overall performance score.

Regarding patient outcomes, five studies demonstrated the transfer effects of Eyesi with reduced complications in live cataract surgery following training [12, 21, 29, 35, 38]. Of note, a multi-centre retrospective study involving 265 ophthalmology trainees across the UK showed that complication rates dropped from 4.2 to 2.6% (38% reduction) following the introduction of Eyesi simulators into training programmes [35]. Similarly, a study by Baxter et al. demonstrated that the use of a structured curriculum with wet lab and Eyesi training led to a considerable reduction in complication rates compared with reported figures for traditional training programmes [38]. However a recent study also testing transfer of skills showed some limitations to Eyesi training [36]. Performance during Eyesi training was comparing to subsequent performance in theatre. Results showed that improvements in OR performance was only observed for ophthalmologists who were less experienced and that the ability for Eyesi scores to discriminate between novice and experienced surgeons could only be seen in the first few training sessions.

Vitreoretinal surgery

[Summary: content = 1; response processes = 1; internal structure = N; relations to other variables = 1; consequences = N; translational outcomes: level 2].

Only four studies have evaluated the vitreoretinal modules on the Eyesi Surgical Simulator [39–42]. These studies support the content validity for vitreoretinal surgery training, as well as response processes, and relations to other variables. Similar to cataract surgery training, scores on the vitreoretinal modules were able to discriminate between experienced and inexperienced surgeons. One study reported evidence for response processes through the standardisation of testing and assessment such as allocating set time periods for training, standardised instructions and using the same supervisor. This evidence remains limited at the best [43]. Studies on the vitreoretinal modules also demonstrated a learning curve with overall scores increasing and completion time decreasing with repeated attempts, indicating contained effects in using the Eyesi for vitreoretinal training. No evidence has been published to support internal structure and consequences or transfer of skills to the operating room.

MicroVisTouch

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = 1; consequences = N; translational outcomes = N].

The MicroVisTouch (ImmersiveTouch, Inc, Chicago, USA) is another commercially available virtual reality simulator that was introduced after the Eyesi, with a report of the prototype published in 2012 [44]. Unlike the Eyesi, the MicroVisTouch features a single handpiece that is attached to a robotic arm and is used to control the appropriate instrument according to the procedure being simulated. It also differs from the Eyesi in that it has an integrated tactile feedback interface, reportedly the first ophthalmic simulator to have this feature [45]. Currently, simulation is limited to three key steps in cataract surgery (clear corneal incision, capsulorhexis and phacoemulsification), although further modules are being developed.

Compared with the Eyesi, fewer studies have assessed the MicroVisTouch. Two groups have reported, implicitly, that the simulator demonstrates content validity for simulating capsulorhexis and that there is evidence of relations to other variables [44, 45], but other sources of validity evidence are lacking. Evidence supporting the effectiveness of using the simulator is also lacking. A third group adapted the MicroVisTouch by customising the algorithm and integrating OCT (Optical Coherence Tomography) scans of varying vitreoretinal conditions to the simulator, enabling patient-specific simulation training of vitreoretinal procedures (epiretinal membrane and internal limiting membrane peeling) [46]. However, the validity and effectiveness of this model was not tested in the original study and no further reports have been found.

Eyesi Ophthalmoscopes

Direct

[Summary: content = 1; response processes = 2; internal structure = 2; relations to other variables = 2; consequences = 1; translational outcomes: level 2].

The Eyesi Direct Ophthalmoscope (VRmagic, Mannheim, Germany) is a virtual reality simulator that enables fundoscopy examination practice, consisting of an ophthalmoscope handpiece with built-in display and a patient model head connected to a touchscreen. A range of patient cases and pathologies can be selected from the programme and objective feedback is provided based on the trainee’s performance [47].

Although only two studies were found evaluating this simulator, there was strong evidence for its validity. Borgersen et al. published the only study in this review to assess validity using all five parameters in Messick’s framework, and showed that the consequences of using a set pass/fail score to accurately discriminate between inexperienced participants (medical students), who were given a fail compared with the experienced participants (ophthalmology consultants) who all passed [48]. The second study showed that participants who trained with the simulator achieved higher scores in an  OSCE (Objective Structured Clinical Examination)   assessment compared with a control group who only received classical training, thus demonstrating contained effects for translational outcomes [49].

Indirect

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = 1; consequences = N; translational outcomes: level 1].

The Eyesi Indirect Ophthalmoscope (VRmagic, Mannheim, Germany) is similar to the Eyesi Direct, an ophthalmoscope headband that is connected to a display showing a 3D virtual patient and virtual lenses when physical, diagnostic lenses are placed over the model head. As with the Eyesi Direct, physiologic and pathologic functions for the virtual patient can be controlled and varied.

Only two studies were found for this simulator [50, 51]. In contrast to the Eyesi Direct, validity evidence was limited to relations to other variables as one study showed that the simulator could discriminate between medical students and ophthalmology trainees [50]. Effectiveness was limited to internal acceptability as participants gave positive feedback of their experience in using the simulator.

Others

A variety of different virtual reality simulators have also been described, including three models for cataract surgery [52–54]; five for vitreoretinal surgery [55–59]; one for endoscopic endonasal surgery [60]; two for general ophthalmic surgery [61, 62]; 1 for ophthalmic anaesthesia [63]; 1 on ocular ultrasound [64]; and 1 for indirect ophthalmoscopy [54]. However, these have all been stand-alone reports with limited evidence of content validity only (scores of 0 or 1). An exception is the Endoscopic Endonasal Surgery Simulator by Weiss et al., which was tested in an RCT and demonstrated good internal structure [60]. Effectiveness was only tested in four models, with the Sophocle retinal photocoagulation simulator shown to be the most effective (downstream effects) as live assessment on real patients showed that the simulator group performed similarly to the control group who had previously practised on patients [58]. As with the other descriptive study models, these simulators have not been further investigated.

Wet lab

A total of 47 studies on wet-lab models were found, of which 12 were mixed models used in conjunction with an inanimate device or artificial system. From the animal model studies, 22 used porcine-related specimens, 3 used sheep specimens, 4 used goat eyes and 3 rabbit eyes. The number of studies using human cadaveric eyes or isolated lens were 17, of which 3 were used in combination with animal tissue.

Cataract surgery

[Summary: content = 2; response processes = 0; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = level 2].

There were 16 studies describing the use of wet-lab models for cataract surgery [65–80]. These demonstrated content validity only, with no evidence for other validity parameters. Models which showed the strongest evidence for validity were pig eyes filled with cooked chestnuts for practising phacoemulsification [71] and rabbit eyes fixed with paraformaldehyde for simulating capsulorhexis [74]. These two models demonstrated contained effects and internal acceptability respectively.

Vitreoretinal surgery

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = N].

One study described the use of rabbit eyes for performing pars plana vitrectomy, from which content validity could be inferred [81]. However, all other sources of validity evidence and indications of effectiveness were lacking.

Glaucoma surgery

[Summary: content = 1; response processes = 2; internal structure = N; relations to other variables = 1; consequences = N; translational outcomes = level 2].

Six studies were found for glaucoma surgery [82–87], with the majority lacking formal validity assessment. One study, which tested placement of human cadaveric eyes into a model head Marty the Surgical Simulator (Iatrotech Inc., Del Mar, USA) for goniotomy simulation, demonstrated good response processes and evidence of internal acceptability [84]. Dang et al. also showed that performing trabeculectomies on porcine eyes with added canalograms for outflow quantification had some evidence for relations to other variables and contained effects [82].

Corneal surgery

[Summary: content = 1; response processes = N; internal structure = N; relations to other variables = 2; consequences = N; translational outcomes = 1].

The use of wet-lab models for practising corneal surgery has been described in four studies [88–90]. Content validity and relations to other variables were demonstrated in one study [91], which tested the feasibility of simulating Descemet’s membrane endothelial keratoplasty on human corneas with an artificial anterior chamber with a 3D-printed iris. However, evidence of other validity parameters and effects were not demonstrated in the other studies.

Strabismus surgery

[Summary: content = 0; response processes = 0; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = level 2].

Two wet-lab models were found for strabismus surgery, both using porcine eyes. White et al. added bacon to the eyes to simulate extraocular muscles [92], whereas Vagge et al. asked residents to practice on a chicken breast model followed by the pig eyes [93]. Discussion of content validity and response processes was made in both studies but no data were reported. Internal acceptability and contained effects were demonstrated for the two models respectively.

Oculoplastic surgery

[Summary: content = 2; response processes = 0; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = level 2].

Four studies described the use of wet-lab oculoplastic simulators [94–97]. These all demonstrated content validity, with one study by Pfaff showing strongest evidence for this parameter [95]. One group showed that using a split pig head for practising lid procedures had good internal acceptability [94] and another group using human cadaver eyes showed that trainees had improved comfort, confidence and technical skills in performing canthotomy and cantholysis procedures [97].

Orbital surgery

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = N].

Altunrende et al. describe using a sheep cranium to practise ocular dissection for orbital surgery. Content validity was reported but any further effectiveness of the model was not testes [98].

Ocular trauma

[Summary: content = 1; response processes = N; internal structure = N; relations to other variables = 0; consequences = N; translational outcomes = level 1].

A recent study by Mednick et al. showed that placing iron particles on human cadaver eyes for corneal rust ring removal simulation had evidence of content validity and relations to other variables [99]. Internal acceptability was shown to be high. Another study on ocular trauma surgery described the use of goats’ eyes for practising corneoscleral perforation repair [100]. However, as the study was purely descriptive, it was not possible to assess its validity or effectiveness.

Diagnostic examination

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = N].

One Study by Uhlig and Gerding tested the use of porcine eyes placed inside an adjustable, artificial orbit for practising direct and indirect fundoscopy, as well as gonioscopy [101]. As this was a descriptive study, no evidence for validity or effectiveness was given.

Others

The remaining wet-lab models were either used to simulate a wide range of anterior and/or posterior segment surgeries or general micro-surgical skills [102–111].

Only two models, both using porcine eyes for micro-surgical skills assessment, provided data supporting their validity. Ezra et al. investigated the use of a video-based, modified Objective Structured Assessment of Technical Skill (OSATS) assessment tool. They demonstrated good internal structure, with high inter-rater reliability, and relations to other variables, with significant correlation between the OSATS scores and results from a separate motion-tracking device [109].

The Eye Surgical Skills Assessment Test (ESSAT), involving the use of porcine eyes and feet as part of a three-station assessment, demonstrated all five sources of validity evidence. One study showed, via a panel of ophthalmic surgery experts, that there was strong evidence of content validity [110]. A further masked study demonstrated that the ESSAT showed strong inter-rater reliability (internal structure) and that the senior resident in the study scored higher than the junior resident (relations to other variables) [111]. Unlike other models, the study authors also went on to discuss the potential consequences of using the ESSAT as an assessment tool, weighing up the benefits of setting a competence score that trainees would need to meet before performing on real patients, with the potential problems of the ESSAT becoming a stressful test preventing less confident residents from entering the operating room. The effectiveness of using this test, however, was not tested.

Altogether, the wet-lab studies, which assessed effectiveness only evaluated responses to participant surveys (internal acceptability) [105] and performance improvements on the models themselves (contained effects) [108, 109]; downstream effects were not demonstrated.

Dry lab

Twenty-six studies on synthetic models were identified, of which eight were developed for practising diagnostic examination techniques (slit lamp, direct and indirect ophthalmoscopy), six for vitreoretinal surgery, one for strabismus surgery, four for laser procedures, two for orbital surgery, one for cataract surgery, one for oculoplastic surgery, one for ocular trauma, one for general ophthalmic surgery and one for combined fundoscopy examination and laser procedures.

Cataract surgery

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = level 2].

Abellán et al. developed a low-cost cataract surgery simulator using a methacrylate support and aluminium foil for capsulorhexis simulation [112]. This was the only inanimate simulator to be tested in an RCT and demonstrated transfer effects as those who trained using the model achieved a higher percentage of satisfactory capsulorhexis in subsequent practice with animal eye models compared with those who had begun training with the animal eyes. Further validity evidence was lacking from the study.

Vitreoretinal surgery

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = 1; consequences = N; translational outcomes = level 2].

For vitreoretinal surgery, two different validated models were found. Hirata et al. used quail eggs within a silicone cap to simulate membrane peeling. The model was shown to discriminate between experienced and inexperienced surgeons in terms of operating time and the success rate of membrane peeling (relations to other variables) [113]. This was similarly tested in a study by Yeh et al. where, using the artificial VitRet Eye Model (Phillips Studio, Bristol, UK) filled with vitreous-like fluid to simulate a variety of vitreoretinal surgery procedures, a positive correlation was observed between the trainees’ level of experience and total score [114]. In terms of effectiveness, the models by Yeh and Hirata showed internal acceptability and contained effects, respectively.

Other dry-lab models for vitreoretinal surgery included the use of an artificial orbit with diascleral illumination [115]; a modified rubber eye [116]; a medium-fidelity model constructed using a wooden frame and tennis ball to simulate the globe [117]; and an artificial eye with inner limited membrane made using hydrogel [118]. However, these models were only described, with no assessment of validity or effectiveness.

Strabismus

[Summary: content = 1; response processes = 2; internal structure = 2; relations to other variables = N; consequences = N; translational outcomes = level 2].

One study was found on the use of a low-fidelity, dry-lab model for strabismus surgery simulation [119]. The model consisted of a rubber ball simulating the globe; elastic band simulating the recti muscles, a piece of latex to simulate the conjunctiva and cornea. Results showed no significant differences between this model and a higher-fidelity, wet-lab model and this dry-lab model. The study showed strong evidence of valid response processes and internal structure as the authors performed a pre-randomisation test to determine baseline dexterity and ensured stratified randomisation of participants into two different groups with equal baseline dexterity. The process for evaluating the participants’ skills after training was also robust as their performance was evaluated by two independent ophthalmologists using three different validated assessment scales (ICO-OSCAR, OSATS and ASS).

Laser surgery

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = 1; consequences = N; translational outcomes = level 2].

Out of the four laser simulators found [120–123], only the model designed by Simpson et al. showed evidence of validity through relations to other variables [103]. The effectiveness of training with this model, however, was not investigated. Conversely, a capsulotomy simulator by Moisseiev and Michaeli demonstrated contained effects but did not test for validity [121].

Oculoplastic surgery

[Summary: content = 0; response processes = 0; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = N].

One oculoplastic surgery dry-lab model was found, using an anatomically correct skull model for simulating nasolacrimal duct surgery [124]. However, this was descriptive only, with no assessment of validity or effectiveness.

Orbital surgery

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = N].

There were two studies using 3D-printed orbit models for simulating orbital surgery [125, 126]. However, as these were also descriptive only, evidence for their validity and effectiveness was not shown.

Trauma management

[Summary: content = 1; response processes = N; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = level 1].

The Newport Eye is a simple training phantom using a craft eye, resins and ground black pepper to simulate corneal foreign body removal [127]. This demonstrated evidence for content validity as experts agreed that the model was realistic in terms of its tissue colour, consistency and anatomy. Trainees also reported being more confident with the procedure after using the model, demonstrating internal acceptability. However, this simulator does not appear to have been used by other groups and no further reports were identified.

Diagnostic examination

[Summary: content = 0; response processes = 0; internal structure = N; relations to other variables = 1; consequences = N; translational outcomes = 0].

The largest proportion of dry-lab models were designed for practising examinations, including slit lamp and fundoscopy (direct and indirect) [128–136]. Two studies tested the validity and effectiveness of the EYE Exam Simulator (Kyoto Kagaku Co., Japan), a popular tool for fundoscopy practice. This consists of a head model with adjustable pupil sizes and changeable fundus slides to represent different retinal conditions. McCarthy et al. showed that response processes were generally poor as letters were added to the slides to check the participant’s field of vision through the ophthalmoscope and the majority were unable to identify the markers or pathology on the slides [132]. Survey responses also indicated that there was low user satisfaction with the model as trainees did not feel it was realistic or that the exercise improved their skills. On the other hand, Akaishi et al. showed that there was a strong correlation between accuracy of examination on the EYE simulator and previous experience of performing fundoscopy in the clinic, providing some evidence for its validity [128].

All other studies of ophthalmoscopy and slit-lamp simulators were descriptive, showing internal content validity only.

E-learning

Aside from training technical skills, tools have also been developed for improving cognitive and other non-technical skills such as teamwork and leadership. A total of five studies were found using this modality, with the majority testing its use amongst medical students. All studies incorporated both training and assessment as part of the course.

Computer-Assisted Learning Ophthalmology Program

[Summary: content = 2; response processes = 2; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = level 1].

The Computer-Assisted Learning Ophthalmology Program designed by Kaufman and Lee is a multi-media, interactive tutorial, which aims to help medical students learn about the pupillary light reflex [137]. Content validity was demonstrated extensively by experts and response processes were thoroughly assessed as each student’s experience and thought process during the training was evaluated by an external interviewer after the programme. Despite the positive responses from all groups showing that it was a valid and effective simulator (internal acceptability), it has not been used by other medical schools and no further reports have been found.

Case-based e-learning modules with Q&A games

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = N; consequences = 1; translational outcomes = level 2].

A study by Stahl et al. also tested the consequences of using e-learning modules as a part of ophthalmology teaching for a group of 272 medical students [138]. Although validity parameters were not formally tested, the authors found that students who used e-learning more frequently achieved better exam results.

Ophthalmic Operation Vienna

[Summary: content = 0; response processes = N; internal structure = 2; relations to other variables = N; consequences = N; translational outcomes = level 2].

An RCT by the Medical University of Vienna evaluated the use of a 3D animated programme for learning different steps in ophthalmic surgery [139]. This demonstrated strong internal structure as a reliability analysis of the multiple-choice questions used at the end of the programme showed a Cronbach’s α coefficient of 0.7, indicating high reliability. Those in the simulation group also outperformed the control group in the final test, showing contained effects.

3D computer animations for learning neuro-ophthalmology and anatomy

[Summary: content = 0; response processes = N; internal structure = N; relations to other variables = N; consequences = N; translational outcomes = level 2].

A similar study by Glittenberg and Binder was carried out, investigating the use of a combination of various 3D design software for teaching complex topics in neuro-ophthalmology [140]. No evidence was provided supporting its validity. However, effectiveness was demonstrated as students responded very positively to the programme in a satisfaction questionnaire (internal acceptability) and also achieved significantly better results in a post-lecture test compared with the control group (contained effects).

The Virtual Mentor

[Summary: content = 2; response processes = N; internal structure = 0; relations to other variables = 1; consequences = N; translational outcomes = level 2].

Whilst most e-learning studies were designed to help medical students, one model was developed for ophthalmology residents to develop non-technical skills. A multi-centre RCT tested the effects of using The Virtual Mentor, an interactive, computer-based programme teaching the cognitive aspects of performing hydrodissection in cataract surgery, including decision making and error recognition [141]. Test questions demonstrated good content validity as they were developed and modified by cataract surgery experts across nine academic institutions. Test scores also demonstrated relations to experience, with correlation between total marks and residency year of training. Despite the lack of data quantifying the reliability of this model, the study showed a degree of internal structure as residents were randomised using a stratified design according to their academic centre and residency year, factors which would likely have influenced the test scores. Internal acceptability was demonstrated by positive user feedback and contained effects through higher post-test scores and a greater mean increase in pre- to post-test results in the simulator group compared with the control group.

Discussion

This systematic review of simulation training in ophthalmology provides a comprehensive evaluation of all available simulation tools using the modern taxonomy. Virtual reality simulators were the most widely evaluated and the Eyesi Surgical Simulator in particular. For cataract surgery, evidence to support all aspects of content validity have been reported. Critically data support the collateral effects of using the Eyesi with training being shown to result in improve operating room performance and lower complications. In contrast, only a much more limited assessment of other ophthalmic simulation training tools has been undertaken including the vitreoretinal training modules for the Eyesi Surgical system. A wide variety of dry-lab and wet-lab training models were reported. Use of dry-lab models in ophthalmology was more limited compared with other surgical specialities [142] with no evidence to suggest any model was particularly effective. In contrast, a relatively high number of wet-lab models was reported. In general, acceptability was high with positive participant feedback and there was evidence, albeit limited, to support the educational impact of wet-lab training. Cadaveric animal tissue was most commonly used and no significant benefits of human over animal cadaveric models were reported. Only five studies reported the use of e-learning. These results do support its potential for ophthalmology training but further assessment needs to be undertaken before incorporation into the training curriculum. Lastly, there was a paucity of studies addressing non-technical skills training in this area. The impact of human factors on patient safety is well-recognised corresponding to the rapid increase in non-technical skills training in medicine [143]. One study in this review, the Virtual Mentor e-learning programme, included cognitive components of cataract surgery training [141]. A pilot study by Saleh et al. also demonstrated the feasibility of using high-fidelity, immersive simulation for cataract surgery, using scenarios based on previous patient safety incidents and evaluating the cross-validity and reliability of four established assessment tools (OTAS, NOTECHS, ANTS and NOTSS) [144].

Simulation tools are increasingly being used for assessment of technical and non-technical skills, both formative and summative. In this review, only one assessment tool, ESSAT, has been described. Strong validity evidence has been shown for the ESSAT but further research on the development of standards and application of the ESSAT tool has not yet been performed. Effective skills assessment is becoming increasingly important both to support competency-based training, as well as enable objective proficiency assessment. In response to growing calls for greater transparency and accountability, formal ongoing credentialing and certification are being considered to ensure doctors maintain the necessary skills and knowledge throughout their professional careers. Simulators are being used to provide objective skills assessment but especially in such high-stakes assessment, rigorous validation of the assessment tools is required before they can be implemented.

Overall, the majority of studies lacked a formal validation process, with 45% of studies (n = 59) being purely descriptive. Furthermore, most validity assessments used the outdated validity frameworks which greatly limits the value of these results. “Face validity” was commonly reported as validity evidence despite the recognition that such subjective assessment of the perceived realism of a simulator is largely irrelevant to its educational impact [145]. Likewise the concept of construct validity using expert–novice comparisons remains widely used but again offers little useful insight into a simulator’s educational impact. The lack of validation studies appears greater than in other specialties. Similar systematic reviews of otolaryngology and orthopaedic simulation training reported rates of descriptive studies of 23% and 38%, respectively [146, 147]. This resonates with findings from a recent review of simulation-based validation studies across all surgical specialities that reported that only 6.6% used Messick’s validity framework [148]. Evidence for a number of components were particularly deficient. Internal structure was rarely assessed, a fundamental area evaluating the reliability and generalisability of scores. For the wet-lab and dry-lab groups, a large number of authors have attempted to establish validity through feedback from study participants on whether the simulator was a valid representation of the surgical correlate. However, this is flawed since the majority of these participants are inexperienced and input should be made from those who have more expertise in the procedure of interest. Effectiveness and translational outcomes were also not extensively tested. In particularly wet-lab and dry-lab simulation studies predominantly reported evidence from user satisfaction surveys, with few assessing for skill improvement and none investigating the relationship to OR performance or patient-related outcomes. Although several studies have linked Eyesi training with reduced complication rates, the majority of these have been retrospective studies which did not control for important confounders such as participants undertaking other forms of training. A few studies explored the collateral effects of simulation training on a systemic level, such as cost saving or policy changes. Two separate preliminary analyses on cost effectiveness were carried out in 2013, both suggested that the cost to benefit ratio was unfavourable. One study predicted, on the basis of cost modelling, that residency programmes would not be able to recoup the costs of purchasing one Eyesi model within 10 years under the most optimistic scenario [149]. The other study suggested that, realistically, it would take 34 years to make a cost recovery [150]. In contrast, the most recent study by the Royal College of Ophthalmologists in the UK argued that the Eyesi was a cost-effective method if costs of complication were include. Access to an Eyesi simulator led to a 1.5% decrease in complication rates, which were inferred to result in an estimated 280 fewer cases of posterior capsular rupture complications alone per year. This would amount to a saving of roughly £560,000 per year and, using this figure, the authors calculated that the cost of purchasing 20 Eyesi simulators would be regained within 4 years. Due to the contrast in findings between these three studies and the implications for both patient safety and costs for healthcare providers, further attempts should be made to provide an updated reflection of current cost effectiveness of Eyesi simulators. In addition, there should be more studies to test whether the same potential benefits gained from Eyesi training can be achieved with a lower-cost model.

The limitations to this study are that, although a broad search criterion was applied using comprehensive search terms, it is possible that some reports using different terminology may have been missed. As discussed above, a large proportion of studies suffered from poor methodologies, utilising outdated concepts of validation and greatly limiting the conclusions that can be drawn. The heterogeneity in methodology and outcomes across the studies also prevented the use of quantitative analysis. In addition, the majority of e-learning studies included in this review recruited medical students rather than ophthalmic professionals, thus results obtained may not be reflective of specialised training.

Conclusion

The increasing importance of simulation training in ophthalmology is reflected by the number and variety of models described in the literature. The Eyesi Surgical remains the only model to have undergone extensive testing and the necessary evidence supporting its use has been reported. The main limitations of current research lie in the use of outdated validity frameworks, a lack of attempt made to establish the collateral, systemic effects of using simulator models and the low quality of validation study designs. Future studies need to follow current recommendations on the assessment and validation of educational tools to ensure that simulation-based training is successfully incorporated into current systems of training in ophthalmology, especially for high-stakes applications such as credentialing and assessment.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.