A Machine Learning approach to predict surgical learning curves

Abstract

Background:

Contemporary surgical training programs rely on the repetition of selected surgical motor tasks. Such methodology is inherently open-ended with no control on the time taken to attain a set level of proficiency, given the trainees’ intrinsic differences in initial skill levels and learning abilities. Hence, an efficient training program should aim at tailoring the surgical training protocols to each trainee. In this regard, a predictive model utilizing information from the initial learning stage to predict learning curve characteristics should facilitate the whole surgical training process.

Methods:

This paper analyzes learning curve data to train a multivariate supervised machine learning model. One factor is extracted to define the trainees’ learning ability. An unsupervised machine learning model is also utilized for trainee classification. When established, the model can predict robustly the learning curve characteristics based on the first few trials.

Results:

We show that the information present in the first ten trials of surgical tasks can be utilized to predict the number of trials required to achieve proficiency (R² = 0.72) and the final performance level (R² = 0.89). Furthermore, only a single factor, learning index, is required to describe the learning process and to classify learners with unique learning characteristics.

Conclusions:

Using machine learning models, we show, for the first time, that the first few trials contain sufficient information to predict learning curve characteristics and that a single factor can capture the complex learning behavior. Using such models holds the potential for personalization of training regimens, leading to greater efficiency and lower costs.

Introduction

Bimanual motor skill learning is an important aspect of surgical training. Surgeons learn technical skills through repeated practice. Though, most residency programs provide the opportunity to practice skills without ensuring that a certain level of proficiency has been reached. Technical surgical skills have been traditionally assessed using in-training evaluation reports (ITER), procedural-based assessment (PBA) or surgical log books. All these approaches are based on the traditional model of apprenticeship with the faculty responsible for assessing technical proficiency based on direct observation. However, problems including leniency/severity errors, central tendency errors, and “halo effects” associated with such approaches are well-known^1,2. Moreover, trainees have inherent differences in initial skill levels and learning rates as well as variations between surgical procedures which limits the effectiveness of the time-limited training approach.

Technical skill testing for certification and competency-based curricular are increasingly becoming popular. Demonstrating proficiency in basic laparoscopic and endoscopic skills is now pre-requisites for certification in general surgery³. Starting in 2018, the fundamentals of laparoscopic surgery (FLS) program is also required for board certification for obstetric and gynecological surgery (www.flsprogram.org). Realizing the inherent problems with the approach of repeated practice, there is significant interest in proficiency-based training^4–7. In this approach, repetition is continued until a certain level of proficiency is achieved. However, the procedure is cumbersome and time consuming as the number of repetitions is not known in advance. To develop structured training programs that account for individual variability in skills and learning abilities, a more personalized method is needed which can predict individual learning curves for any surgical procedure based on initial performance. This requires a deeper understanding of learning curves for surgical skills.

Different techniques to analyze and model the learning curves of surgical procedures have been presented in the literature. A review paper⁸ has summarized those approaches. First, without any statistical analysis, a simple graph or a table displaying the outcome of the surgery against the number of operations was presented to show the learning curve in a substantial number of studies⁸. As a higher level of analysis, basic statistical tools such as t-test, analysis of variance, or chi-squared test was applied to two or three groups of data split by a number of practices⁸. However, the splitting points in these studies were arbitrarily selected, and the underlying curves in each group were not analyzed⁸.

Besides statistically analyzing the learning curves, analytical modeling methods have also been presented in literature. A commonly used modeling method is to fit a curve to the learning curve data using least squares regression, with or without an adjustment for other confounding factors including age and sex⁸. Both linear and exponential curves have been used to fit learning curves, without much justification for the choice of these curves⁸.

The cumulative summation (CUSUM) technique was initially suggested to monitor surgical performance⁹ but was recently applied to analyze the learning curve of a surgical skill^10–15. CUSUM has a simple formulation that positive or negative increments are added to a cumulative score according to failure or success of the successive trial¹⁶. A graph representation of CUSUM is intuitive in that a declining trend indicates successive success and an increasing trend indicates successive failures¹⁶. The two boundary limits h₁ and h₀ in CUSUM graphs represent whether or not the observed failure rate is significantly different from the desired acceptable failure rate¹⁶. The number of trials to achieve proficiency is derived by counting the number of attempts before crossing the boundary limits¹⁷. With different specific aims, the design of CUSUM schemes varies across studies. In some studies^18–20, CUSUM is calculated as the cumulative difference between observed and expected outcome values, such as operation time or blood loss, instead of binary results of success or failure. A change-point in these CUSUM graphs was determined to derive the number of trials to achieve proficiency^18,20 or learning phases¹⁹. Some studies set the downward part of CUSUM graphs to zero to monitor only failures²¹.

A unique disadvantage of all these existing approaches is that they require information regarding training that has already taken place. While they are academically of interest, their utility in designing individualized training programs is limited. To the best of our knowledge, there are no approaches in the literature that focus on predicting learning curve features including the final performance level after a certain number of trials. Hence, our goal in this study is to overcome these limitations in the existing literature by testing two hypotheses. First, we hypothesize that the performance of a trainee during the first few trials of a bimanual motor task has sufficient information to predict the number of trials required to achieve proficiency and the final performance level. Secondly, we hypothesize that it is possible to define a single factor that can describe how these parameters including the initial skill level of the trainee are related to each other. To accomplish that, we have performed a meta-analysis of bimanual skill acquisition in pattern cutting task which is a part of the FLS program^13,15. Learning data on the physical FLS trainer box as well as on a virtual basic laparoscopic skill trainer (VBLaST) replicating the FLS tasks^14,22–25 have been utilized.

Materials and methods

Data sources

In this study, we performed a retrospective analysis of the learning curve data from three IRB approved studies^13,15,26 The studies were associated with the FLS pattern cutting task which involves cutting a gauze following a marked circle^13,15and the FLS intracorporeal suturing²⁶. The three studies were selected because of the similarity in experimental setup and procedures. Novice medical students, with no prior surgical experience, were selected for these studies^13,15,26. The training was carried out over a three-week period using the FLS trainer box or VBLaST replicating these tasks. The FLS program has been shown to be reliable to quantify surgical skill level^27,28 and metrics for these tasks are established in the surgery literature²⁹ and utilized in Board certification in general surgery. The performance score of each trial was calculated from task performance time and performance error using the accredited FLS scoring methodology with consent under a nondisclosure agreement from the FLS Committee. The metrics for the VBLaST are derived from the FLS metrics and discussed elsewhere²³. In both cases, the metrics are aggregated into a final score, either being the FLS score or the VBLaST score. These two scores are the quantities used to build the surgical learning curves. We excluded as outliers those curves that did not exhibit a clear initial learning stage or a learning plateau. Fifteen learning curves were then selected for this study (Table 1).

Table 1.

The source of the data^13,15

Study	Platform	Motor task	No. of learning curves
Nemani et al. 201713	FLS*	Pattern cutting	4
Nemani et al. 201713	VBLaST*	Pattern cutting	6
Linsk et al. 201715	VBLaST*	Pattern cutting	2
Fu et al. 201926	FLS*	Suturing	3

^*FLS is a physical training box and VBLaST is a virtual reality version of it.

Variables

This study involved three variables: the number of trials required to achieve proficiency, the initial performance level, and the final performance level. We defined the number of trials required to achieve proficiency based on the Technical Skill Proficiency-Based Training Curriculum⁴ for the FLS program. For the pattern cutting task, proficiency is achieved if the task can be performed within 98 seconds on two consecutive repetitions; for the intracorporeal suture, trainees achieve proficiency when they perform 10 additional trials after two consecutive trials within 112 seconds with allowable errors. A similar definition was employed with the VBLaST data. We defined the initial performance level as the average score of the first three trials. To define the final performance level, multiple two-tailed t-tests were performed between different trial intervals. From the result in Table 2, the fifth 10 trials performance is not significantly different from the fourth 10 trials, but the other pairs of trial intervals are significantly different. This indicates that from the fortieth trial, the performance scores do not significantly change. Thus, we defined the final performance level as the average score after the fortieth trial.

Table 2.

Significant test results between trial intervals

Trial intervals	Significant test
First vs. second 10 trials	p = 0.000*
Second vs. third 10 trials	p = 0.004*
Third vs. fourth 10 trials	p = 0.023*
Fourth vs. fifth 10 trials	p = 0.205

^*Significance α = 0.05

Multivariate supervised learning model

To test our first hypothesis, we used a multivariate supervised machine learning approach known as kernel partial least squares (KPLS)^30,31 to predict the learning curve features including the number of trials to achieve proficiency and the final performance level from the initial learning performance. KPLS first computes a non-linear transformation of the ten initial trials performance scores (denoted as X) from the thirteen learning curve entries into a high dimensional feature space. Once transformed, the projected observations are then linearly regressed to maximize their covariance with the final performance score or the number of trials to achieve proficiency (denoted as y). In this way, the non-linear relationship could be modeled optimally between the input X and the output y.

We used the coefficient of determination (denoted as R²) as defined below to quantify the accuracy of the proposed model:

$$R^2=1-\frac{SSE}{SST}$$ (2)

$$SSE=\frac{{\sum }_{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}{n-1}$$ (3)

$$SST=\frac{{\sum }_{i=1}^n{\left(y_i-\overline{y}\right)}^2}{n-1}$$ (4)

where y_i is the true outcome value, ${\widehat{y}}_i$ is the predicted outcome value, $\overline{y}$ is the mean of the true outcome value, and n is the total number of samples. R² = 1 indicates that all the variance of the data is explained by the model³⁰. Conversely, smaller values or even negative values indicate a poor predictability is considered.

Considering the small sample size of the dataset, we validated our modeling results using the leave-one-out cross-validation (LOOCV) scheme³¹. In LOOCV, we excluded one learning curve from the training dataset and tested the model on the learning curve left out, repeating the analysis for all learning curves to ensure robustness of the model and to avoid overfitting.

Since the dataset is from subjects practicing on two types of training platforms, a physical FLS trainer box as well as on a VBLaST, it is necessary to validate whether the model works across different platforms. To achieve this, a platform-wise cross-validation scheme was designed. In this scheme, we trained the model on one platform and tested it on the other one to demonstrate the effectiveness of the model across the two platforms.

Wright³² was the first to discuss learning curve modeling, and his log-linear model has been used in prior literature³³. Other log-linear based models have also been developed considering different factors affecting learning curves, like the S-Curve model which takes a gradual start-up into consideration³³. In the experimental setup in this study, the subjects recruited were novice medical school students with no prior surgical experience, and the training paradigm was consistent across trials. Thus, we adopted the conventional log-linear model as a comparison, given by the following equation:

$$Y=Y_0{N}^{\theta }$$ (5)

where N is the number of trials, Y is the performance, θ is the learning rate, and Y₀ is the initial performance level. The model variables are independently assessed for each subject.

Factor analysis model

Factor analysis is a statistical method to explore latent variables or factors from the observed variables. In our case, we have three observed variables - initial performance level, number of trials to achieve proficiency and final performance level. The question we asked is whether a single variable can be used to represent all three variables. Here, we use a factor analysis model known as the kernel principal component analysis (KPCA) approach³¹ to extract a representative factor, learning index (LI), from the three learning curve features. We first mapped the three features into a high dimensional non-linear space. Then, we extracted one principal component as LI from these high-dimensional data by principle component analysis (PCA). To test whether LI represents the three learning curve features, we used LI to predict the three features by KPLS. If LI could predict all three features accurately, then the information compressed in LI is enough to represent the three learning curve features.

Unsupervised classification of skill level

An unsupervised learning approach known as k-means clustering analysis^34,35 was adopted to separate the trainees according to their different learning curve characteristics. By analyzing the learning curve features of different trainee groups, the learning characteristics of each group were summarized. Furthermore, grouping results derived from LI was compared to the grouping result from all the three features, to see whether LI could indicate unique learning characteristics.

Results

Learning curve data and features

The learning curve data for all subjects are presented (4 trained on FLS physical training box and 9 trained on VBLaST virtual reality trainer) in Figure 1 and the three features calculated from the learning curves are summarized in Table 3.

Figure 1.

The learning curves are from 3 studies: (a) FLS pattern cutting study in Nemani et al. 2017¹³; (b) VBLaST pattern cutting study in Nemani et al. 2017¹³; and (c) VBLaST pattern cutting study in Linsk et al. 2017¹⁵; (d) FLS intracorporeal suture study in Fu et al. 2019²⁶. (The permission to reuse the data has been acquired from the journals).

Table 3.

The learning curve feature values for all subjects (performance levels refer to the FLS/VBLaST scores)

Subject No.	Initial performance level (average from trial #1 to trial #3)	Number of trials required to achieve proficiency	Final performance level (average from trial #40 to trial #50)
1	13.10	57	65.46
2	4.88	15	83.36
3	2.38	33	72.86
4	18.57	22	69.32
5	26.07	45	70.50
6	34.88	114	66.84
7	16.90	50	62.28
8	25.36	67	59.64
9	26.31	52	68.21
10	42.98	10	74.93
11	21.43	20	80.32
12	40.83	50	71.14
13	16.13	56	83.50
14	0	114	66.77
15	0	97	76.95

Prediction performance

Table 4 represents the R² values for KPLS and the log-linear models when they are used to predict the number of trials to achieve proficiency and the final performance level based on the performance in initial several trials. The performance of the KPLS model indicates that the initial performance pattern can be used to predict the two learning curve features with a high degree of accuracy. Conversely, using the log-linear model, the resultant R² values for predicting the number of trials to achieve proficiency are negative and those for predicting the final performance level indicate the need for a considerable number of initial trials to be close to 1.

Table 4.

Accuracy of KPLS and log-linear model

Learning curve variable	KPLS	log-linear model
Number of trials to achieve proficiency	R2 = 0.72 (first 10 trials)	R2 = −4.21 (first 50 trials)*
		R2 = − 9.27 (first 40 trials)*
		R2 = − 15.17(first 30 trials)*
		R2 = −49.53 (first 20 trials)*
		R2 = − 109.55 (first 10 trials)*
Final performance level	R2 = 0.89 (first 10 trials)	R2 = 0.76 (first 50 trials)
		R2 = 0.58 (first 40 trials)
		R2 = 0.37 (first 30 trials)
		R2 = − 0.27 (first 20 trials)
		R2 = −3.36 (first 10 trials)

^*For first 10 trials, the predicted learning curves of subject 4, 6, 12 and 13 do not achieve proficiency in 1000 trials and are excluded from the R² calculation; for first 20 trials, the predicted learning curve of subject 6 does not achieve proficiency in 1000 trials and is excluded from the R² value calculation; for first 30, 40 and 50 trials, the predicted learning curve of subject 14 does not achieve proficiency in 1000 trials and is excluded from the R² value calculation

The log-linear model performance was further explored with a different number of initial trials for which data were used for model development. One example of the learning curve from subject 1 is shown in Figure 2. The figure shows that the log-linear model works well when the FLS scores for the first 50 trials are known. However, when fewer trials are used as input to the model, the log-linear curve clearly becomes less accurate. This is particularly evident when data from only the first 10 trials are used, and the log-linear curve underestimates the learning effect considerably. The underestimation of the log-linear model also explains why the R² values for predicting the number of trials to achieve proficiency are all negative using this approach (Table 4). The predicted curves are grossly underestimated, implying that a much larger number of trials is required to achieve proficiency than what is actually needed. Some predicted curves could not achieve proficiency even after 1000 trials and are excluded from the analysis (Table 4). Although the variables being included in this work are very simple, it is shown to be hard to predict those using existing models, such as the log-linear model.

Figure 2.

The performance of the log-linear model developed using the scores for the first 50, 40, 30, 20, and 10 of trials for subject 1. ‘o’s represent the trials for which the scores are assumed to be known; ‘x’s represent the remainder of the trials for which the data are not used in developing the model. The solid line represents the log-linear model.

Considering the two different platforms the subjects practiced on (physical and virtual), platform-wise cross-validation tests were performed on the KPLS model. The results are listed in Table 5. When predicting the number of trials to achieve proficiency and the final performance, the R² of the KPLS model are all above 0.70 in the platform-wise cross-validation testing. This indicates that the learning curve data patterns are consistent across the physical FLS training box and the VBLaST box, and the datasets could be meta-analyzed.

Table 5.

Cross-validation results of KPLS

Learning curve variable	Platform-wise
	Trained on FLS platform	Trained on the VBLaST platform
	But tested on the VBLaST platform	But tested on FLS platform
Number of trials to achieve proficiency	R2 = 0.72	R2 = 0.78
Final performance level	R2 = 0.73	R2 = 0.78

Factor analysis

We derived a single factor, which we referred to as LI, from the three learning curve features (initial performance level, number of trials to achieve proficiency and final performance level), based on the KPCA method. LI is a latent variable which depicts the learning characteristics of the learners. The R² values when using LI to predict the three learning curve features by a KPLS model are listed in Table 6. Since all the R² values are above 0.8, the extracted feature could be determined as a representative feature of the three features.

Table 6.

the R² values using the learning ability to predict the three learning curve features

Case	R2
Predicting all the three features	0.92
Initial performance level	0.87
Number of trials to achieve proficiency	0.93
Final performance level	0.94

K-means clustering

Next, we further grouped the subjects by their learning curve features (initial performance level, number of trials to achieve proficiency and final performance level) using the k-means clustering algorithm. Two groups naturally emerged from this analysis: group 1 with subjects 2, 10, 11, and 13 and group 2 consisting of the remaining ones. To understand the implication of this grouping, we plotted the number of trials to achieve proficiency and the final performance level, against the initial performance level in Figure 3(a). The crosses represent subjects in group 1, and the circles represent subjects in group 2. From these plots it is clear that trainees in group 1 have higher initial performance levels, require fewer trials to achieve proficiency and achieve higher final performance levels, while the trainees in group 2 have lower initial performance levels, need more trials to achieve proficiency and achieve lower final performance levels. When we use the same k-means clustering algorithm to group the subjects based on the feature values of LI, the same grouping result is derived as shown in Figure 3(b). The two groups are clearly separable and clustered by the extracted features. This result supports that LI is sufficient to classify learners with unique learning characteristics.

Figure 3.

The trainees were clustered into two groups by the k-means clustering algorithm. ‘x’s represent trainees that are clustered into Group 1 and ‘o’s represent trainees that are clustered into Group 2. The grouping results are from (a) the learning curve features and (b) the extracted factor “learning ability”.

Discussion

Our findings highlight that the use of machine learning enables assessing the performance of a trainee by evaluating his or her performance during the first ten repetitive trials. Based on only ten trials, we can predict (i) the number of trials required to achieve proficiency and (ii) the final performance level, as defined as the average FLS score after fortieth trial, with a high degree of accuracy. Furthermore, a single factor, LI, which we refer to as the learning ability here, can be derived from a non-linear factor analysis model. The single factor describes common variation within these two parameters and the initial performance level. This, in turn, implies that the number of trials required to achieve proficiency, the final performance level and the initial performance are not independent of each other.

These findings are related to earlier work in this area, as the initial learning stages are related to the later stages^36,37. Moreover, Jirapinyo et al. ³⁸ showed that a log-linear regression model can describe surgical training learning curves reasonably well when all learning data are used. However, other studies contract this ^19,39. With respect to our study, even though we extracted very simple learning curve characteristics, we still found limitation in the use of log-linear models. One potential reason is that the log-linear regression model is derived from group data, but individual learning progress may be distinct from the group trend ⁴⁰. Another, more intuitive reason is that the log-linear model assumes a predefined form for the learning curves prior to fitting the curves. In sharp contrast, the KPLS model does not assume such an assumption. In addition to that, it is well known that a larger set of variables that are highly correlated or collinear can yield difficulties in identifying regression models. KPLS is a nonlinear regression tool that has been developed in the field of chemometrics with the aim to handle such situations³⁰. Therefore, the KPLS regression model is able to capture the complex process of surgical skill learning in a data-driven format, which is missed by simplistic analytical models including the log-linear model. Once a KPLS regression model has been developed for a surgical task, the trained model may, consequently, be employed to predict the number of trials needed to achieve proficiency by any new learner based on scores of the initial few trials.

Moreover, the use of machine learning for learning curve prediction has the potential to redesign surgical training programs and has implications for skill decay and retraining throughout the entire professional life of surgeons. Predicting the learning curve variables early in the training process would help to provide more focused feedback and implement adaptive learning strategies. The idea of adaptive training is not new and has been suggested in the surgical literature. For example, Stefanidis et al.⁴¹ pointed out that by establishing skill learning curves, the training curricula could be tailored to provide additional training to those that need more training than others. Another study⁴² compared adaptive curricula and volume-based curricula in surgical training and demonstrated that the group trained with adaptive curricula achieved the same level of performance, but required fewer training hours.

Another important finding of our study is that the single factor allows clustering the trainees into two groups based on their distinct learning curve characteristics. These groups have clearly unique characteristics, where the participants in Group 1 had a higher initial ability to carry out the FLS task, showed higher final performance level and did not require a considerable number of trials to converge from the initial to their final performance level. On the other hand, the subjects in Group 2 had a lower initial ability, showed a lower final performance level and required more repetitive trials to reach the final performance level. The surgical motor learning difference between individuals has also been demonstrated and studied in other studies. For example, Louridas et al.⁴³ showed that with the same training curriculum, the participants demonstrated different learning results and could be divided into three groups with top, moderate and low performance. Similar grouping results of surgical residents were reported⁴⁴. Our study provides a quantitative understanding that individuals with different initial skill levels require different practice to reach a final performance level. Differences in learning characteristics may be due to innate factors including handedness, gender, visual-spatial ability, and confidence level^45–51 or extrinsic factors including research experience, selection of specialty, and grouping^49,52,53.

Our study has several limitations which may suggest conducting further research in this area. First, the learning curves were quantified by task performance scores which are calculated from performance time and performance error. While these same metrics are used in the FLS, there could be other kinematic (e.g., hand trajectory) or physiological (e.g., eye motion or skin conductance) metrics which could provide further information regarding performance. In separate work, we have shown that functional brain imaging that relies on neuro-vascular coupling, provides much more accurate quantification of bimanual motor skill learning than the traditional FLS metrics^54,55. Second, the bimanual motor task in this study is limited to the pattern cutting task and intracorporeal suture task. Extending the analysis procedures to other FLS tasks and actual surgical procedures is left for future work. It is important to note that the results reported can assist in planning individualized training regimes. It is not intended to share the predicted number of trials with the learners, as this may affect their performance negatively. In addition to that, physiological measurement such as functional brain imaging could play an important role here to monitor the workload and attention level to determine whether the trainees are trying their best to learn. Finally, data from only a few learning curves have been used in this study. Learning curve studies are inherently difficult to do due to the extended time commitment of the medical students and the limited number of willing participants. As summarized in a comprehensive review paper ⁵⁶, a sample size of 8–23 is reported in previous simulation-based surgical training studies, indicating the difficulty to recruit participants in multiple days training protocols. We included three of our previous studies collecting the learning curve information to have a sample size of fifteen. To set up a machine learning model based on such a small sample size, we selected KPLS, which is a machine learning method suitable for small sample size. To determine machine learning models for a relatively small sample size, we selected KPLS, which is a kernel-based projection method. Such methods have an extensive record of applications in data chemometrics^57–59 where small sample sizes are the norm rather than the exception. Moreover, we report results that are based on an independent assessment of the model performance to ensure that the established regression models are not compromised by overfitting. Future multi-center studies may mitigate some of these issues.

In conclusion, we propose the use of sophisticated machine learning models for predicting the learning curve features from the initial few trials of bimanual surgical motor tasks. A single factor, LI, which we define as the learning ability, is able to capture complexities of learning behavior. Use of such models holds the potential for personalization of training regimens, leading to greater efficiency and lower costs.