Assessment of the Learning Curve for Virtual Surgical Planning in Orbital Fractures

Abstract

Virtual surgical planning (VSP) is becoming more widely used in maxillofacial reconstruction and can be surgeon-based or industry-based. Surgeon-based models require software training but allow surgeon autonomy. We evaluate the learning curve for VSP through a prospective cohort study in which planning times and accuracy of 7 otolaryngology residents with no prior VSP experience were compared to that of a proficient user after a single training protocol and 6 planning sessions for orbital fractures. The average planning time for the first session was 21 minutes 41 seconds ± 6 minutes 11 seconds with an average maximum deviation of 2.5 ± 0.8 mm in the lateral orbit and 2.3 ± 0.6 mm in the superior orbit. The average planning time for the last session was 13 minutes 5 seconds ± 10 minutes and 7 seconds with an average maximum deviation of 1.4 ± 0.5 mm in the lateral orbit and 1.3 ± 0.4 mm in the superior orbit. Novice users reduced planning time by 40% and decreased maximum deviation of plans by 44% and 43% in the lateral and superior orbits, respectively, approaching that of the proficient user. Virtual surgical planning has a quick learning curve and may be incorporated into surgical training.

Introduction

Computer-aided surgery (CAS) is gaining greater acceptance in maxillofacial reconstruction. Computer-aided surgery can be divided into 3 steps: (1) planning: also known as virtual surgical planning (VSP), (2) execution: carrying out the surgical plan (ie, models, guides, and patient specific implants), and (3) verification: confirming that the proposed plan has been achieved (ie, intraoperative navigation or computed tomography [CT]). This study focuses on VSP; the technique of using detailed CT imaging and planning software to generate a preoperative plan in a virtual environment.^1–3 Virtual surgical planning modalities may be surgeon-based (ie, performed by the surgeon) or industry-based (ie, performed by the surgeon and an engineer via teleconferencing). While the surgeon-based model requires an upfront financial investment in planning software and surgeon training, it allows the individual surgeon to autonomously generate the plan. The industry-based model does not require a financial investment or surgeon training. However, engineer costs, prolonged planning time, and diminished surgeon input are limitations in this model.^4–6 The industry-based model has gained a strong foothold, however as planning software becomes more user friendly, it is likely that surgeon-based workflows will become more common and gain greater acceptance.⁶ The aim of this study is to evaluate the learning curve for novice VSP software users. The authors hypothesize that VSP software can be learned quickly and effectively without the need for extensive training, making it a feasible tool to incorporate as part of resident surgical training.

Materials and Methods

Study Design

Institutional review board approval was obtained for this prospective study. Study participants included 7 otolaryngology residents (2 postgraduate year 1 [PGY-1], 3 PGY-2, 1 PGY-3, 1 PGY-5) at a single academic institution. The participants had no experience with VSP. The study was broken into 2 parts, a single training protocol where participants were educated on the use of the VSP tool (iPlan—Brainlab AG) and 6 planning sessions where the participants’ virtual surgical plans were evaluated for duration and accuracy. Only the planning sessions were used for data collection/analysis. The senior author (EBS), having previously created >500 plans, acted as a proficient user/control.

Training protocol

The training protocol was developed to educate inexperienced participants in a uniform fashion. It was completed individually and proctored by the same investigator (TYH). Each participant went through the training protocol at their own pace (range 30-60 minutes). The training protocol was broken into 2 parts: user interface training and case training.

User interface training

The user interface training involved viewing an online training manual and basic interaction with the user interface options on a workstation.

Case training

Case training involved planning a single orbital reconstruction. Four tasks were included: (1) placement of fiducial markers to be used for intraoperative registration, (2) segmentation of the uninjured right orbit, (3) mirroring of the uninjured right orbit across the midline plane, and (4) precise alignment of the uninjured mirrored orbit onto the contralateral injured orbit (Figure 1A).

Figure 1.

A, Example user interface for VSP demonstrating segmented unaffected orbit on the right and the mirrored orbit plan superimposed on the left fractured side. The red markers are fiducial points for intraoperative navigation. B, Coronal view: measurement of maximum deviation of mirrored orbit plan from bone superior (blue double-ended arrow) and lateral (yellow double-ended arrow) orbit. VSP indicates virtual surgical planning.

Planning sessions

Each participant independently completed 6 planning sessions. Each planning session required the participant to complete the same 4 tasks demonstrated in the training protocol. One case was provided to each participant every other day. Each case represented a similar but unique orbital fracture amenable to VSP. The order, setting, and timing of presentation were uniform and completed over an 11-day period. Recorded data points included (1) total time of each planning session, (2) accuracy of the mirroring task, and (3) the number of times each participant referred to the training manual per session. Accuracy was determined by comparing the largest deviation (in mm) of the mirrored orbit to the injured orbit. The measurements were taken along the orbital roof and lateral orbital wall, where there were no fractures (Figure 1B). These measurements were performed independently and averaged between 2 authors (TYH and RD) who were blinded to the participant and trial number.

Data Analysis

Ordinary least squares regression analysis with exponential fitting using RStudio (RStudio) was used for data analysis. To assess for proficiency and accuracy in the learning curve of VSP, the participants’ planning times and maximum deviation in the orbital roof and lateral orbital wall were compared with a proficient user’s data (EBS). A P value less than .05 was considered statistically significant. The data were plotted using Microsoft Excel.

Results

All participants completed the entire 6 case series. None of the participants referred to the BrainLab user manual during their planning session. The average planning time for the first case was 21 minutes 41 seconds ± 6 minutes 11 seconds with an average maximum deviation of 2.5 ± 0.8 mm in the lateral orbit and 2.3 ± 0.6 mm in the superior orbit (Table 1 and Figures 2 –4). With the completion of subsequent planning cases, regression analysis showed an overall downward trend in the planning time (P = .0037, R ² = 0.19; Figure 2) and in the maximum deviation in the lateral orbit (P = .0149, R ² = 0.14; Figure 3) and the superior orbit (P = .0301, R ² = 0.11; Figure 4). The average planning time for the last case was 13 minutes 5 seconds ± 10 minutes and 7 seconds with an average maximum deviation of 1.4 ± 0.5 mm in the lateral orbit and 1.3 ± 0.4 mm in the superior orbit (Table 1 and Figures 2 –4). Overall, there was a 40% decrease in total planning time with a decrease of 44% and 43% in the maximum deviation of plans in the lateral and superior orbits, respectively.

Table 1.

The Planning Time and Maximum Deviation in the Lateral and Superior Orbits in Plans Created by Each Novice User Throughout the Study.

	Trial number	1	2	3	4	5	6
Participant 1	Superior deviation, mm	3.1	2.3	2.1	4	2.7	2
	Lateral deviation, mm	2.3	2.1	3.4	3	1.2	1.4
	Planning time, seconds	1517	1261	1153	889	572	554
Participant 2	Superior deviation, mm	2.9	2.3	2.6	2.4	2.4	1.4
	Lateral deviation, mm	3.4	2.5	2.4	2.4	3.7	1.6
	Planning time, seconds	1671	1285	1295	1203	1738	2008
Participant 3	Superior deviation, mm	1.6	1.2	1.4	2.3	1.6	0.8
	Lateral deviation, mm	1.5	1.1	1.1	0.8	0.2	0.6
	Planning time, seconds	1740	1228	389	534	371	383
Participant 4	Superior deviation, mm	2.4	2.6	1.8	1.6	1.3	1.6
	Lateral deviation, mm	2.5	3.2	1.6	1.2	1.3	1.2
	Planning time, seconds	1198	662	567	589	410	468
Participant 5	Superior deviation, mm	1.6	1	1.7	2	1.2	1.5
	Lateral deviation, mm	2.6	1.3	2.2	2.3	2	2
	Planning time, seconds	679	572	521	521	655	621
Participant 6	Superior deviation, mm	1.8	1.7	1.6	1	1.8	1.1
	Lateral deviation, mm	1.6	1.3	2	1.2	1.4	2
	Planning time, seconds	1213	1061	662	755	565	775
Participant 7	Superior deviation, mm	2.4	1.2	1.2	0.8	1.4	1
	Lateral deviation, mm	3.4	1.4	0.8	0.6	1.4	0.8
	Planning time, seconds	1092	807	830	740	1057	692
Average	Superior deviation ± SD, mm	2.3 ± 0.6	1.8 ± 0.6	1.8 ± 0.5	2.0 ± 1.0	1.8 ± 0.6	1.3 ± 0.4
	Lateral deviation ± SD, mm	2.5 ± 0.8	1.8 ± 0.8	1.9 ± 0.9	1.6 ± 0.9	1.6 ± 1.0	1.4 ± 0.5
	Planning time ± SD, seconds	1301 ± 371	982 ± 299	774 ± 338	747 ± 241	767 ± 483	786 ± 554

Abbreviation: SD, standard deviation.

Figure 2.

A, Exponential fitted regression analysis demonstrated a downward trend in the planning time with subsequent trials. Control: y = 298.04e ^0.0224x. Participants: y = 1204.2e ^−0.114x. R ² = 0.1925. P = .0037. B, The novice users’ average planning times approached a proficient user’s planning times (control). *Within 2 standard deviations; **within 1 standard deviation.

Figure 3.

A, A scatter plot and exponential fitted regression analysis showed improvement in the maximum deviation in the lateral orbit in preoperative plans for orbital fractures with successive trials. Control: y = 1.2774e ^0.011x. Participants: y = 2.4626e ^{− 0.127x}. R² = 0.1393. P = .0149. B, There was no significant difference in the novice users’ average maximum deviation in the lateral orbit compared to the proficient user’s measurements for all the trials. *Within 2 standard deviations; **within 1 standard deviation.

Figure 4.

A, Exponential fitted regression analysis of the maximum deviation in the superior orbit in virtual preoperative plans decreased as the number of created plans by each novice user increased. Control: y = 0.6052e ^−0.061x. Participants: y = 2.2291e ^−0.08x. R ² = 0.1122. P = .0301. B, There was variation in the novice users’ average maximum deviation in the superior orbit compared to the proficient user’s data. *Within 2 standard deviations; **within 1 standard deviation.

Discussion

Virtual surgical planning is an evolving tool in maxillofacial reconstruction that facilitates surgical repair and may optimize patient outcomes.^7–9 Bly et al demonstrated that mirror image overlay in unilateral orbital fractures had a 5-fold decrease in revision surgeries and a 2-fold improvement in postoperative diplopia.⁷ The use of intraoperative surgical planning has been proposed to improve intraoperative accuracy and reduce the incidence of revision procedures.⁸ While this technology continues to grow, there is very little known about the learning curve for use of these tools. This study evaluated the learning curve for the use of iPlan software in orbital reconstruction by novice users.

The results of this study demonstrated a marked reduction in planning time from 21 minutes 41 seconds (± 6 minutes 11 seconds) to 13 minutes 5 seconds (±10 minutes and 7 seconds) after completion of 6 cases; a 40% reduction in time. Accuracy improved from 2.5 ± 0.8 mm to 1.4 ± 0.5 mm (lateral orbit) and 2.3 ± 0.6 mm to 1.3 ± 0.4 mm (superior orbit) after 6 cases; resulting in a 44% and 43% reduction, respectively. The average participant planning times and the maximum deviations in the lateral and superior orbits began to approach those of a proficient user (Figures 2 –4). It was noted that lateral orbit wall accuracy was improved more than the superior orbit. This difference may be reflective of the learning curve required to recognize facial asymmetry and geometric variability between the 2 orbits.¹⁰ Given the rapid improvements in total time and accuracy, as well as regression analysis with P values <.05, we conclude that the data demonstrate both statistically significant decreases and practical improvements in planning time and planning accuracy. While improvement is expected with repetition, our results suggest that novice users can approach the proficiency of a competent user after a relatively short timeframe with only 1 training session and 6 planning sessions. Following the study, participating residents were ultimately able to independently use VSP software to develop preoperative surgical plans.

A potential limitation with VSP is cost. However, Rodby et al demonstrated that VSP can decrease operative times while improving the accuracy of reconstruction; thus the overall cost savings may offset the technological cost.¹¹ Resnick et al reported that VSP can significantly decrease planning time and overall planning cost associated with orthognathic surgery.¹² Finally, Mendez et al demonstrated that surgeon-based VSP could potentially eliminate the costs and time associated with industry-based VSP.¹³

Limitations of this study include (1) use of a single medical center, (2) small sample size—likely resulting in low R² values which may influence the fit of the data in the regression analysis, and (3) use of single VSP software.

Conclusion

There is a quick learning curve for VSP with iPlan software. After 1 training session and the completion of 6 cases, novice users reduced their planning time by 40% (from 21 minutes 41 seconds to 13 minutes 5 seconds). Accuracy was improved by 44% for the lateral orbit (2.5 ± 0.8 mm to 1.4 ± 0.5 mm) and 43% for the superior orbit (2.3 ± 0.6 mm to 1.3 ± 0.4 mm). Novice users were able to approach the proficiency of a competent user relatively rapidly, suggesting VSP software and application may be a valuable addition to resident surgical training.