Assessment of a high-fidelity, virtual reality-based, manual small-incision cataract surgery simulator: A face and content validity study

Abstract

Purpose:

To establish the face and content validity of the HelpMeSee Eye Surgery Simulator – a virtual reality-based cataract surgery simulator for manual small-incision cataract surgery (MSICS).

Methods:

The face and content validity were assessed on the sclero-corneal tunnel construction course. A questionnaire with 11 questions focused on the visual realism, with resemblance to real life surgery, and the training value of the simulator was developed. Thirty-five experienced MSICS surgeons participated in the study. Responses were recorded using a seven-point scoring system.

Results:

Overall, 74.3% (26/35) of the respondents agreed that the overall visual representation of the eye and the instruments in the simulator were realistic. The task of injecting a visco-elastic through the paracentesis was reported to be the most visually realistic task with a mean score of 5.78 (SD: 1.09; range: 2–7). With regard to content validity, 77.1% (27/35) of the subjects felt agreed that the errors and complications represented throughout the entire tunnel construction module were similar to those encountered in real life; the task of entering the anterior chamber with the keratome had a mean score of 5.54 (SD: 0.98; range 1–7), being rated the highest in that aspect. Overall, 94.3% (33/35) of the subjects agreed that the simulator would be useful in developing hand–eye co-ordination. A similar number of 94.3% (33/35) agreed that based on their experience, they would recommend cataract surgical training on this simulator.

Conclusion:

The results suggest that the HelpMeSee Eye Surgery Simulator appears to have sufficient face and content validity for cataract surgical training.

Cataract surgery is one of the most commonly performed surgeries worldwide.[1] With over 15 million cataract extraction surgeries performed annually globally, it is the most frequently performed day-care procedure.[2] With the anticipated demographic changes in the developing world, the cataract-related visual morbidity and the cataract subject pool are expected to increase significantly. In order to address this change, surgical training programs will be required to ensure that ophthalmology residents in training achieve acceptable levels of competence in cataract surgery prior to graduation.[2] It has been observed that in developing countries, while the occasional ophthalmology residency program is generous and well rounded, a significant number do not offer even basic cataract surgical training to their residents.[3]

Wet lab training and surgical simulators have been an integral part of surgical training curriculums for quite some time now.[4,5] The ongoing coronavirus disease 2019 (COVID-19) pandemic has impacted ophthalmic education significantly. A survey by Mishra et al.[6] showed that nearly 81% of the surveyed trainees were of the opinion that COVID-19 had negatively impacted their surgical training, with fewer surgeries and increased stress levels. With fewer elective surgeries and training opportunities, trainees may need to rely more on surgical videos, wet-lab training, and simulators to hone their surgical skills. In the United Kingdom, ophthalmology trainees are allocated time to complete two cataract modules on an Eyesi® Surgical (VRmagic, Mannheim, Germany) simulator, a popular virtual reality ophthalmic simulator.[7] This mandatory simulation training has been found to reduce the unadjusted posterior capsule rupture rate in phaco-emulsification for novice surgeons by 38% from 2009 to 2015.[8] As a result of the COVID-19 pandemic, an increase in the use of ophthalmic simulators was observed in the UK.[9]

However, phaco-emulsification is not feasible and available everywhere. Manual small-incision cataract surgery (MSICS) is the recommended substitute technique to replace phaco-emulsification owing to the lack of the necessary machinery, surgical instrumentation, consumables, and surgical expertise, especially in developing countries.[10] Surgeons with skill in MSICS are essential to address the global backlog in cataract surgical services.[5] Most of the available ophthalmic surgical simulators focus on surgical skills required in phaco-emulsification and vitreo-retinal procedures.[11] The HelpMeSee Eye Surgical Simulator (HelpMeSee Inc., NY, USA) is a high-fidelity, virtual reality-based simulator specifically built to support the training for MSICS [Fig. 1].[5] This simulator combines high-quality computer graphics and the ability to provide real-time tactile feedback integrated with a physics model of various surgical tasks in MSICS to provide a realistic experience of the surgical task performance as necessary for skills training. Owing to a realistic physics-based modeling of the virtual eye, the HelpMeSee Eye Surgery simulator is able to not just support the task performance in the intended way but also show errors and complications that are likely to occur during surgery. For example, while creating a sclero-corneal tunnel, a trainee may end up creating a superficial tunnel leading to a buttonhole. Conversely, a very deep tunnel can lead to a perforation of the sclera with the underlying uveal tissue being visible. Similarly, during the tunnel dissection, the trainee may also end up having a premature entry in to the anterior chamber. All these errors are not only demonstrated visually on the simulator but also perceived by the trainee through the handpieces that have real-time tactile feedback. Additionally, errors are also recorded and displayed on the screen and through the eyepiece when the trainee makes them. These errors include ‘iris touch’ and ‘lens touch’ while making a paracentesis or during visco-elastic injection. These features allow the trainee to know how to avoid complications and additional features such as a performance summary including error counts, and video playback features allow trainees to review previous attempts and assess their errors and outcomes. The simulation tasks have a standard level of difficulty. This is made considering a normal eye with no unusual pathology. The instructor-led training course for MSICS on the HelpMeSee Simulator is a comprehensive 6-day long program with a well-rounded, exhaustive curriculum comprising didactic classroom lectures, lab activities, interactive debrief sessions, and simulator sessions (which comprise 80% of the course time). At the end of the course, the trainee undergoes an assessment of the tasks on the simulator. The first day of this course comprises mainly of the scleral groove and tunnel dissection tasks, which were the assignments used to assess the face and content validity in this study.

Figure 1.

The HelpMeSee Eye Surgery Simulator

With this background, the present study was conceived to assess the face and construct validity of the HelpMeSee Eye Surgery Simulator. In general, ‘face validity’ is expressed as the assessment of virtual realism or the extent of a simulator’s realism and appropriateness when compared to the actual task. ‘Content validity’ is an assessment of the suitability of a simulator as a teaching tool, essentially the extent to which a simulator’s content is representative of the knowledge or skills that have to be learnt in the real environment.[12,13,14,15,16,17] With respect to cataract surgery, the face content would address the issue of how realistic the visuals of the virtual eye, the surgical instruments, and their interaction appear through the eyepieces of the simulator while performing surgical steps and how realistic is the simulated feel (haptic feedback) that is perceived during the surgical steps. Similarly, in this context, content validity addresses the question of how useful the cataract surgical simulator is in helping learn relevant skills such as scleral dissection, anterior chamber entry, and visco-elastic injection.

Methods

Thirty-five ophthalmology experts volunteered for and participated in the present study. The study was conducted at the Comprehensive Cataract Conference 2^nd World Conference on MSICS and Comprehensive Cataract Conference in Chennai, India. The demographics and surgical experience of the experts are summarized in Table 1. Each delegate was given a comprehensive introduction on the purpose of the study, an overview of the simulator and its components, and instructions on how to use the simulator. The course chosen for the demonstration and assessment was scleral tunnel construction course (STCC) in MSICS. This course included the following tasks: dissecting the scleral tunnel with a crescent blade [Fig. 2a], creating a paracentesis at the limbus with a stab blade [Fig. 2b], injecting the visco-elastic material through the paracentesis injection [Fig. 2c], and finally entering the anterior chamber with a keratome [Fig. 2d].

Table 1.

Demographics and details of the participants

Cumulative number of surgeries performed by the experts: 910,000 cases

Gender: Male: 27 (77%); Female: 8 (23%)

Average surgical experience (MSICS cases): 26,000 cases

Average surgical experience of the experts (in years): 24 years (range: 7-42 years)

Average age of the experts (in years): 48 years (range: 34-70 years)

Figure 2.

(a) is a screenshot of a trainee dissecting the scleral tunnel with a crescent blade. (b) shows the visuals of a paracentesis being created with a stab blade. (c) shows the microscope view of the visco-elastic material being injected into the anterior chamber through the paracentesis. (d) is the keratome being used to enter the anterior chamber following tunnel dissection

A written consent to participate in the study was obtained from all subjects. Participants were orientated to the simulator and supervised by four investigators (AGN, CA, AEB, and TS). The initial attempt was performed by the investigator with the expert watching the attempt on the screen. Before the first attempt, the experts were asked to ‘feel’ the virtual eye and to move it around using the Colibri forceps to grasp the conjunctiva. Most experts were experiencing this for the first time. Following this, the experts were allowed to perform the above enumerated tasks multiple times over. At the conclusion of the session that on an average lasted 45 minutes, the subjects completed a short questionnaire designed to evaluate face and content validity [Table 2]. The data points in the questionnaire were entered into a Microsoft Excel Spreadsheet (Microsoft Corporation, Medmont, USA) for further analysis.

Table 2.

The questionnaire and the responses of the participants

	<100	101-500	501-1000	1001-2500	>2500
How many MSICS cases do you perform annually?	1	15	5	9	5
	Yes	No
Are you currently/previously involved in MSICS training?	33	2
Do you currently perform MSICS?	34	1
Do you have any previous experience with surgical simulators?	20	15
Based on your experience, would you recommend simulation-based cataract surgery training on this simulator?	33	2
Face Validity:
The visual representation of the following was realistic: (7 - Strongly agree to 1 - strongly disagree)	Strongly Agree	Agree	More or Less Agree	Undecided	More or Less Disagree	Disagree	Strongly Disagree	Mean Score (1-7)
Overall instrumentation and the eye	5	9	12	5	3	1	0	5.37
Tunnel dissection using a crescent blade	2	10	9	7	5	1	1	4.71
Making a paracentesis with a stab blade	7	10	7	4	4	2	1	5.63
Injecting the visco-elastic through the paracentesis	9	15	6	4	0	1	0	5.78
Entering the anterior chamber using a keratome	4	13	8	4	4	1	0	5.03
Content Validity:
The representation of typical errors and complications that a novice is likely to encounter are depicted on the simulator:
Overall procedure	4	13	10	5	2	1	0	5.23
Tunnel dissection	2	14	10	5	3	0	1	5.17
Creating a paracentesis	2	17	9	4	1	2	0	5.26
Injecting the visco-elastic through the paracentesis	7	11	9	7	1	0	0	5.46
Anterior chamber entry using a keratome	5	15	10	4	1	0	0	5.54
The simulator would be useful in developing hand-eye co-ordination for trainees who use them.	15	13	5	1	1	0	0	6.14

MSICS: Manual Small-Incision Cataract Surgery

Results

Demographics and experience

Table 1 summarizes the demographics and experience of the experts who participated as subjects in this study. The group of experts was highly experienced in numbers of years practicing as ophthalmic surgeons as well as in terms of the number of cataract surgeries performed. Among the experts, 97.1% (34/35) reported that they perform at least 100 MSICS procedures annually. The average age of the cohort was 46.1 years. The mean number of years in practice as operating surgeons was 24 years (range: 7–42 years). The surgeons were also asked to enter the approximate number of MSICS surgeries that they have independently performed till date (rounded off to the nearest hundred). The cumulative number of MSICS procedures performed by all the subjects combined was 910,000.

Face validity

In the questionnaire, the experts were asked if they agreed that the visuals of the tasks depicted on the simulator were realistic. A seven-point Likert scale was used with one being the lowest (strongly disagree) and seven being the highest (strongly agree). Overall, 74.3% (26/35) of the respondents agreed that the overall visual representation of the eye and the instruments in the simulator were realistic. Thirty subjects (85.7%) were in agreement that the task of injecting a visco-elastic substance into the eye through the paracentesis was most realistically represented. This was followed by keratome entry (71.4%), paracentesis creation (68.6%), and sclero-corneal tunnel dissection, where 60% of the subjects agreed that the visuals were realistically depicted on the simulator.

Content validity

Overall, 33/35 (94.3%) of the subjects agreed that the simulator would be useful in developing hand–eye co-ordination in trainees who trained on it. A similar number of experts 33/35 (94.3%) agreed that based on their experience on the HelpMeSee Eye Surgery Simulator, they would recommend training on this simulator for all trainees. To assess the content validity of the simulator, the subjects were asked if the representation of errors and complications on the simulator were comparable to those that a novice surgeon would likely encounter in real-life surgery. In all, 77.1% (27/35) of the subjects felt agreed that the errors and complications represented throughout the entire tunnel construction module were similar to those encountered in real life. The maximum agreement was noted for keratome entry (80%) and paracentesis creation (80%), followed by visco-elastic injection (77.1%) and tunnel dissection (74.3%).

Discussion

The results presented here are significant because the HelpMeSee Eye Surgery Simulator is the only cataract surgical simulator that is built and designed specifically to support MSICS.[5,6] Most virtual-reality simulators and tissue simulators for cataract surgery have been primarily developed for training in phaco-emulsification, including the Eyesi simulator, MicroVisTouch, PhacoVision, Bioniko, Kitaro, SimulEYE, and the Phantom Phaco simulator, among others.[11,18,19,20,21,22,23] As a surgical procedure, MSICS has been found to be more economical than phaco-emulsification. Additionally, it was also reported to be as safe and nearly as effective as phaco-emulsification. From a surgeon’s perspective, apart from being an additional surgical procedure in the surgeon’s repertoire, MSICS is recommended as an alternative to phaco-emulsification for the rehabilitation of cataract patients in developing countries where requisite equipment and trained personnel for phaco-emulsification may not always be available.[24]

Structurally, as is seen in Fig. 1, the simulator is designed to resemble an actual patient undergoing cataract surgery, which adds to the immersive experience of surgery. The HelpMeSee Eye Surgery simulator also features haptic feedback in the handpieces [Fig. 3]. Haptics refers to the process of recognizing objects through touch, delivered in the form of vibrations and force feedback, created by moving components of a device which is controlled by integrated software.[25] Haptic technology is used to augment VR-based simulation learning, especially in enabling novices to appreciate tissue structures and in developing basic skills such as tissue tension by providing live artificial tactile feedback and increasing the overall “realism” of the simulation.[26] A comprehensive systematic review of randomized controlled studies comparing VR training with and without haptics reported that overall haptic feedback has been shown to improve the fidelity, realism, and thus the training effect of VR simulators.[26] Of the previously studied cataract surgery simulators, none of the available VR simulators have real-time haptic feedback.

Figure 3.

The multi-purpose handpieces on the HelpMeSee eye surgery simulator which are used to navigate the ‘virtual instruments’ in the eye on the simulator and provide real-time haptic feedback

The HMS Eye Surgery Simulator is a unique simulator that aims to train trainees in a procedure that has previously not been taught on a simulator with haptic feedback. To ensure that any new simulator provides a realistic comparison to real-life surgery, it must undergo scientific validation. This study aims to establish face and content of the HMS Eye Surgery Simulator in order to determine its value as a training tool. Authors regard that face validity is expressed as the assessment of virtual realism by novices, while content validity refers to experts’ assessment of the suitability of a simulator as a teaching tool.[12,13,27] Given that the creation of the sclero-corneal tunnel is unique to MSICS, this representative task was chosen for assessment of face and content validity of the simulator. This tunnel when constructed well is self-sealing and therefore is very crucial with regard to the subsequent surgical steps and eventual outcome of the surgery. Therefore, in order to be proficient at SICS surgery, it is imperative that a trainee possesses the skill and the right technique to create and is clear of the corneal tunnel without any errors. There is a paucity of similar comparable data points for the HelpMeSee Eye Surgery Simulator.

Assessment of face validity is an inherently subjective topic. Experts who found the simulator easy to use and who performed well tend to rate the simulator experience higher than those who did not perform the tasks well.[28] Previous studies evaluating face validity of other simulators have only basic questions on the realism and ease of use of the simulator.[11,29] The present study included these questions and further included additional questions about each surgical task separately in the questionnaire, thereby adding granularity to the results.

Validation studies conducted on surgical simulators at large conventions or scientific meetings have been reported previously.[30,31] This setting offers an environment where subjects and experts with different backgrounds and varying levels of experience are present at the same time. However, the present study has some inherent limitations. The ideal sample size required to obtain a reliable result for face and content validation was not ascertained beforehand. However, in the literature, it has been found that there is no clear agreement on the adequacy of sample size in such validity studies;[14,15] many face and content validity studies of simulators have been conducted with a smaller sample size compared to the present study.[13] The HelpMeSee Eye Surgery Simulator supports all surgical steps that constitute the entire MSICS procedure for training, namely, capsulorrhexis, nucleus delivery, cortex aspiration, and intra-ocular lens implantation. Additionally, the simulator will soon allow for other variable conditions such as deep-set eyes, small pupils, variable grades of cataract, and different iris colors. However, the present study looked only assessed the task of tunnel creation because of time constraints and availability of experts. Therefore, it is possible that the results may not be entirely representative of all the tasks of the simulator. On the other hand, the basic skills such as using a crescent blade, holding the conjunctiva, creating a paracentesis, injecting the visco-elastic material into the eye, and the use of a keratome are not restricted to MSICS alone. Therefore, it is possible that the HMS simulator could have wider applicability and use in training residents to acquire basic ophthalmic surgical skills and to learn tissue handling.

The study has some drawbacks: it assessed the viewpoints of only experts and not real trainees, who would be the eventual users of the simulator for training. Additionally, simulation has the inherent flaws of not always being able to completely re-create every real-life scenario that can occur. Additionally, the simulator as of now supports only right-handed surgeons and with the groove/incision made superiorly. Therefore, only right-handed expert surgeons were invited to try, and hence, there was no feedback or comment received about the difficulty based on the location of the groove.

Conclusion

To summarize, The HelpMeSee Eye Surgery Simulator has demonstrated face and content validity as a virtual reality simulator for training in MSICS. With increasing popularity and integration of simulation into surgical curricula worldwide, this surgery simulator can be a valuable adjunct to surgical training. This study also demonstrates that real-time haptic feedback, when incorporated into a virtual reality simulator, can improve the training value of a surgical simulator. The present study provides a strong basis for further research and additional validation for this simulator. Although the preliminary data are promising, reliability validation studies would be needed to determine whether results are consistent across multiple measures.[32] Furthermore, the ultimate assessment of a simulator is to quantify the transfer of skills acquired on a simulator to the operating room, which would be needed in the future.[33]

Financial support and sponsorship

The study was fully funded by HelpMeSee Inc.

Conflicts of interest

The authors of this manuscript have the following competing interests:

• AGN has received lecture fees from Carl Zeiss Meditec.

• AGN, CA, AEB, TS, VCL receive consulting fees from HelpMeSee Inc.