Validity of the Rezum Simulator for Trainees

Abstract

Background:

Simulations have revolutionized surgical training and are an invaluable adjunct tool for augmenting the proficiency of surgeon and patient safety. Before being included in a practical assessment process, simulators need to be using various validity inference frameworks.

Objective:

We examine the construct validity and reliability of the Rezum simulator.

Methods:

Seventeen candidates of different professions voluntarily participated in the Rezum simulation workshop. The simulator provides a variety of variable metrics and challenges. Each candidate performed three cases of different difficulty levels with three trials of each case. Validity was measured statistically through a one-way analysis of variance (ANOVA) test, and a p-value of < 0.05 was considered significant. Additional reliability tests were provided, including intraclass correlation coefficients, a Cronbach test (0.7 is considered acceptable), and standard error of measurement.

Results:

The ANOVA of total scores among candidates was significant (p = 0.029). Senior registrars and consultants had the highest total scores. Procedure times did not differ significantly among candidates (p = 0.169). The reliability test for the total score was 0.899 (0.831–0.942), with a standard error value of 2.75, a standard deviation of 8.67, and a Cronbach alpha value of 0.915.

Conclusion:

We confer the primer evidence of Rezum simulation as a valid, reliable simulator of most of its metrics.

1. BACKGROUND

Integration of surgical simulations into training programs considerably adopts beneficial influences on trainers, mentors, and patients in direct and indirect means. The high costs associated with other teaching modalities (e.g., live-animal or cadaver training models) and restricted access for rehearsed practice have fostered interest in virtual learning experiences via innovative simulation platforms. Simulations are a vital adjunct to the classic master-apprentice teaching model and shorten the learning curve (1).

Surgical technologies in urology have progressed dramatically over the past decade. In this respect, education and training have moved toward implementing innovative methods such as simulated platforms to enhance physicians’ performance, confidence, and patient safety before practicing such producers on actual patients (2). Rezum is a minimally invasive device primarily for bladder outlet obstruction secondary to the prostate. This would be one of the top surgical options that need to be trained adequately amid residents, marked by its simplicity and 5-year evidence-based effectiveness (3).

Validity has been classified into many forms: face, concurrent, content, and construct. Assessment of a simulation’s validity and reliability are vital before integration into a training curriculum (4, 5).

2. OBJECTIVE

Here we examine the construct validity and reliability of the Rezum simulation platform.

3. MATERIAL AND METHODS

Simulator specifications

The Rezum simulator was manufactured by the Boston Scientific company and runs the VirtaMed platform. It contains a collection of training cases in a 3-D virtual setting. The software has varied metrics, including the number of full treatments, contagious lesion, procedure time, cumulative saline installed, partial treatments, treatments with vapor leakage, treatments in poor visualization, and treatments with excessive torque against the lobe. The simulation automatically determines each metric’s score according to a specific goal established by the software and ultimately calculates a total score that considers all of the above metrics.

Study design

Staff, senior registrars, and trainee residents participated in the prospective observational study during a Rezum workshop hosted by our training center in April 2021. The participants were grouped according to their profession into consultants (N = 4), senior registrars (N = 4), senior residents (N = 4), and junior residents (N = 5). All participants were tutored about the device and any related troubleshooting. The simulation provides many cases of several challenges. We chose three cases (Table 1). Each participant has to practice three trials for each case in a time goal pre-sit by the simulation according to the case level. No assistance was given during the study.

Table 1. Rezum case details.

	Reliability	Cronbcha Alpha	SEM
Number of full treatments	0.79	0.80	0.89
Contiguous lesion	0.87	0.89	3.11
Total score (out of 100)	0.89	0.91	2.75
Procedure time for this level (min)	0.61	0.63	0.03
Cumulative saline instilled goal	0.79	0.80	0.37
Partial treatment	-0.21	-0.21	0.26
Treatments with vapor leakage	-0.05	-0.06	1.26
Treatment in poor visualization	0.37	0.38	0.22
Treatments with excessive torque against lobe	-0.04	-0.04	1.31

Statistical analysis

Metrics data of participants were obtained from the simulation and analyzed through IBM SPSS Statistics, v.27. Construct validity of the simulator was measured using a one-way analysis of variance (ANOVA), and p-values of < 0.05 were considered significant. Reliability was assessed using intraclass correlation coefficients (ICC). Internal consistency among the groups was measured using a Cronbach alpha () test ( = 0.7 is considered acceptable). The standard error of measurement (SEM) was concluded from the Cronbach alpha and standard deviations (SD) of the tests using the following equation:

SEM=SD × √1 − reliability

The smaller the SEM, the higher the reliability of the data, with a confidence interval of 95% (Z-value = 1.96).

4. RESULTS

Table 2 shows the candidates’ profession and case difficulty. The total scores by profession were: Junior resident (89.47 ± 11.83), Senior resident (89.86 ± 7.16), Senior registrar (94.49 ± 6.31), and Consultant (92.94 ± 6.21). The ANOVA test detected a significant variance in the total scores among candidates (p = 0.02). Nevertheless, no statistically significant result in procedure time was detected between the candidates (p = 0.16). The ICC reliability test for the total score revealed a good reliability result, 0.89 (range, 0.83–0.94; SEM, 2.75; Cronbach alpha, 0.91). Additional details for reliability are given in Table 3.

Table 2. Experiment characteristics.

Candidates	Frequency	Percent	Level of cases	Frequency	Percent
Junior residents	45	29.8	Easy	51	33.8
Senior residents	36	23.8	Intermediate	51	33.8
Senior registrars	35	23.2	Difficult	49	32.5
Consultants	35	23.2

Table 3. Reliability and standard error of measurement of the Rezum metrics.

Parameter	Easy case	Intermediate case	Difficult case
Prostate volume	45 cm3	35 cm3	78 cm3
Median lobe	No	No	Yes
Prostatic urethral angle	Normal	Elevated	Elevated
Challenging rate (1-3)	1	2	3

5. DISCUSSION

The 21st century is characterized by technological disruption of various sectors, among which education is no exception. Teaching methods have been refined and enhanced through the use of simulators, devices with built-in software that aim to mimic the intended procedure or skill. The practice of technical and non-technical skills to reach proficiency through simulations has been proven to improve healthcare delivery by advancing patient safety and lowering hospitalization costs (2). However, objective assessment and structured feedback are crucial aspects of assessment tools; hence simulations need to be tested for their validity and reliability (6).

The process of validating simulations is complicated and needs to be understood thoroughly to avoid misleading educators and learners (7). Validity is a dynamic and flexible process that involves a wide range of developed frameworks. Conventionally, validity is divided into face, content, construct, and criterion validities. Face and content validities are subjective tests, while construct and criterion validities are objective (5). However, the tests mentioned above have been reframed into a contemporary validity inferences framework, including content, response process, internal structure, relationship to other variables, and consequences. The relationship between the test contents and the intended construct to measure is reflected in the recent definition of content validity. However, the links among data metrics and how to outreach the construct, and degree of the relation of the construct to the interpretation of test score are referred to as internal structure and relationships with other variables source of validities, respectively (8 ,9).

the earlier mentioned modern sources of validities can be considered collectively under construct validity, According to Educational and Psychological Measurement Standards. The ability of the simulation to discriminate between experienced and non-experienced participants is often used to define construct validity. Cook et al. have provided a broader definition: “as intangible attributes (constructs) are linked with observable attributes based on a conception or theory of the construct”.

Nevertheless, not all data sources and evidence are needed for all assessments (9). We propose the basic internal structure of Rezum simulation through the adjunct of statistical analysis of simulation metrics as initial validity of the simulator and reliability of its variables. Validated simulation is a step toward its usage for structured evaluation and builds a comprehensive construct validity inference with meaningful impaction as an educational tool.

Measuring the precision of the tests or simulation’s metrics has been pointed as reliability, which is now part of construct validity. Its role is invaluable in conform simulations study, which can be tested in numerous methods. Reliability helps ensure the consistency and reproducibility of the simulation tests and assures application and transformation of simulation metrics into a real practice (10, 11). We further enhance our reliability test with the SEM as integral method, determining the variation error during test measurement. When the SEM is close to zero, the test is correlated with the present result as an acceptable one. However, if the result of SEM is approaching the SD of the tests, this suggests that variation error is present. A simulation would be unreliable when the SEM is equal to the SD of particular variable measures. Our variables’ reliability and SEM are adequate and strengthen the reliability of the Rezum metrics, except for partial treatment, treatments with vapor leakage, treatment in poor visualization, and treatments with excessive torque against the lobe. Our study had a limited sample size for further involvement of candidates in the advanced implication of the modern validity assessment process.

6. CONCLUSION

Here we validated the Rezum simulator. Our assessments should be considered a primer toward further validation studies of Rezum simulation as a solid future assessment and educational tool in the field of prostatic surgery.

Author’s contribution:

Both authors were involved in all steps of preparation this article, including final proofreading.

Conflicts of interest:

There are no conflicts of interest.

Financial support and sponsorship:

Nil.

Permission to reproduce any copyrighted material:

Granted.