An interpretable whole-stream video based deep learning framework for surgical skill assessment of basic procedural elements

Abstract

Background:

Conventional surgical skill assessment, conducted by expert surgeons in operating rooms, is often time-consuming and subjective. An automated and objective system capable of differentiating surgeon expertise levels (novice, intermediate, and expert) and providing feedback would significantly enhance surgical training. However, most existing automated frameworks prioritize deep learning performance over interpretability.

Materials and methods:

We propose I3D-SAP, an interpretable video-based surgical skill assessment framework of basic procedural elements using a 3D convolutional neural network. The model processes surgical videos and classifies surgeons into three expertise levels. Experiments are conducted on the public JIGSAWS dataset, evaluating performance across suturing, needle passing, and knot tying tasks.

Results:

Our method has achieved high accuracy: 100% (suturing), 96.3% (needle passing), and 97.2% (knot tying), comparable to the state-of-the-art models. Our framework’s interpretability is demonstrated through Class Activation Map (CAM) visualization and snippet-wise analysis, offering dual perspectives to explain classification decisions. However, evaluation under a more rigorous Leave-One-User-Out (LOUO) protocol revealed a significant performance drop, and testing on an external dataset (ROSMA) resulted in near-chance-level accuracy, indicating that the current model’s ability to generalize to entirely unseen surgeons is limited.

Conclusion:

I3D-SAP provides both high accuracy and interpretability, addressing a critical gap in automated surgical assessment of basic procedural elements. By visualizing the model decisions, it enhances transparency and trust, paving the way for practical adoption in surgical training. Moreover, the model’s current generalizability challenges should be addressed through future work involving larger, more diverse datasets and techniques for surgeon-invariant feature learning before practical clinical deployment can be considered.

Introduction

Compared to traditional open surgery, minimally invasive surgery (MIS), such as laparoscopic surgery and robot-assisted minimally invasive surgery (RMIS), benefits patients by less pain, less risk of infection, and less blood loss. The individual skill of a surgeon is critical in MIS[1] and has a significant impact on the surgical safety and well-being of patients. To improve surgical training and feedback will help reduce the incidence of complications. Accordingly, intensive education for novice surgeons is essential to ensure that surgical interventions are safe and effective. Therefore, it is crucial to assess surgeons’ operative skills.

HIGHLIGHTS

• We introduce a video analysis approach that can process the entire surgical training video.

• We develop a model (I3D-SAP) automatically classifying surgical skill levels from videos.

• Our model achieves comparable performance on the widely used JIGSAWS benchmark dataset.

• We enhance interpretability of the model by providing visual reasoning.

• We supply annotations to the ROSMA dataset and carry out experiments on it.

Conventionally, the evaluation of laparoscopic surgery skills is carried out in the operating rooms by expert surgeons with manual rating via using Objective Structure Assessment of Technical Skills (OSATS)[2], McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTEL)[3], Global Operative Assessment of Laparoscopic Skills (GOALS)[4], or other rating measures^[2,5^]. However, the evaluation process is manual, subjective, and time-consuming.

Surgeons require lengthy and expensive training to become proficient and safe, with better outcomes demonstrated better training and feedback^[6–8^]. To address these limitations, it is becoming important to develop accurate and objective automated assessments of surgical skills[9]. With rapid development in machine learning and increasing scale of the operation video recordings and robotic kinematic data, developing an objective and accurate machine learning based automated surgical skill assessment system has become possible^[10–12^]. Herein, we intend to develop an interpretable surgical skill analysis framework and provide a way to objectively and efficiently assess operative skills in laparoscopic surgery.

In the traditional method of assessing surgical skills, manual efforts are strenuous[13]. Recently, researchers proposed deep learning methods for the automated MIS surgical assessment[14]. Kinematic data were used in surgical skill evaluation by some researchers. For instance, Fawaz et al proposed a fully connected convolutional neural network (FCN) to assess the surgery skill level through the kinematic data[15]. Wang et al presented an end-to-end deep convolutional neural network (CNN) framework for skill assessment of surgical evaluation[16]. A CNN-Long short-term memory (LSTM) network was introduced by Nguyen et al to classify the surgical skills[17]. Beyond the kinematic data, some researchers also used the video data to carry out surgical skill assessments. A novel video-based method with two correlated multi-task learning frameworks was designed by Wang et al, including recognized surgical gestures and surgical skill levels[18]. Funke et al achieved video-based surgical skill assessment using a fine-tuned inflated 3D CNN and proposed a temporal segment network (TSN) framework[19]. Lavanchy et al provided a three-stage machine learning approach for surgical skill assessment by extracting motion features of the instruments from video with instruments detection[20]. Liu et al put forward a unified multi-path framework for automated surgical skill evaluation, considering various components of surgical performance such as instrument handling, intraoperative event patterns, and additional skill-related indicators[21]. Yanik et al used a self-supervised contrastive model to extract spatiotemporal features from suturing videos, which were then classified and scored using a 1D residual neural network[22].

However, most existing surgical skill assessment works are short of interpretability. The lack of transparency of these deep neural networks makes it hard for the senior surgeons to know whether the assessment decision is reliable, which compromises trust from senior surgeons^[23,24^]. Consequently, the application of automated surgical skill evaluation needs to be explainable. Interpretability of the model is essential for these medical AI applications to become accepted and integrated into clinical practice. According to the statement[25], explainability is the ability to explain AI decision-making in understandable terms for humans, with a broader range of end-users on how a decision has been drawn. Much research has been carried out in the field of Explainable Artificial Intelligence (XAI) and interpretability to overcome this limitation^[26,27^]. For instance, using XAI and interpretability methods, we can understand which part of the input data accounts for the classification decision in any classifier[28]. Examples of the XAI methods include SHapley Additive exPlanations (SHAP)[29], Local interpretable model-agnostic explanations (LIME)[30], guided backpropagation (GBP)[31], class activation mapping (CAM)[32], and Gradient-weighted class activation mapping (Grad-CAM)[33]. CAM uses global average pooling and a fully connected layer to visualize the neural network and display the most important features related to the predictions. In other words, CAM can visually highlight the decision strategies of the model’s prediction. In this work, we propose a new video-based framework named I3D-SAP for surgical skill assessment of basic procedural elements. Moreover, Sequential Average Pooling (SAP)-based CAM can explain the contributions of the spatial–temporal regions of the surgical video that drive each decision for surgical skill classification. A snippet-wise analysis breaks the video into segments to identify the skill level of each snippet out of the whole video as well as which portions most influence the model’s prediction. Furthermore, the proposed I3D-SAP can provide interpretability by visualizing the surgical trial’s trajectory.

In this study, we developed and validated a novel video-based AI framework, termed I3D-SAP, for automated surgical skill classification. The framework enhances interpretability through class activation mapping and snippet-wise visualization, enabling visual reasoning of model decisions. We assessed its performance on the JIGSAWS dataset, where it achieved competitive results, and further carried out additional experiments on the ROSMA dataset to evaluate its generalizability.

Additionally, the use of AI in this manuscript is compliant with the TITAN Guidelines 2025[34].

Methods

In this study, we propose a novel model using whole-stream surgical videos as input for surgical skill evaluation. Figure 1 illustrates the overall pipeline of the work consisting of data pre-processing, I3D-SAP video classifier, interpretability, and visual reasoning.

Figure 1.

Overview of our proposed framework taking surgical video data as input data and then applying pre-processing procedure. The I3D-SAP video classifier uses the processed video as input and is trained for surgical skill classification (novice, intermediate, expert). The interpretability of the model is achieved through visualizing the trial’s trajectory (visual reasoning process) with SAP-based CAM visualization and snippet-wise visualization.

Datasets

The JIGSAWS dataset is applied for the evaluation of our proposed method. It has surgical video data captured from the da Vinci Surgical System[35], which is the public dataset for surgical skill level assessment[36]. More specifically, it includes three elementary surgical tasks, i.e., suturing, needle passing, and knot tying in surgical skills curricula. Eight surgeons with different self-proclaimed skill levels (novice, intermediate, expert) perform the three sub-surgical tasks on bench-top models. The detailed statistical analysis of the dataset is shown in Table 1.

Table 1.

Statistical analysis of videos in JIGSAW dataset.

	Label	SU	KT	NP
Number of trails	Expert	10	10	9
	Intermediate	10	10	8
	Novice	19	16	11
Subjects	Expert	D, E	D, E	D, E
	Intermediate	C, F	C, F	C, F
	Novice	B, G, H, I	B, G, H, I	B, G, H, I

KT, knot tying; NP, needle passing; SU, suturing.

The dataset also provides two official cross-validation setup, namely Leave One Super-trial Out (LOSO) and Leave One User Out (LOUO). The LOSO strategy is implemented by organizing the dataset into trial, where each trial consists of the five super-trials which are repetitions of the same task performed by all participants. Since each task is repeated five times, this produces five super-trials in total. In each fold of LOSO cross-validation, one super-trial is held out as the test set, while the remaining four are used for training. This design ensures that the same repetition index across all participants is consistently excluded, thereby avoiding any overlap of trial-specific temporal patterns between training and testing. Apart from LOSO cross-validation, the LOUO cross validation is also provided by the JIGSAWS dataset. The LOUO strategy is implemented by splitting the dataset based on participants. In this setup, all data from one participant are withheld as the test set, and the data from the remaining participants are used for training. With eight participants in the dataset, this results in eight folds. This design allows to directly assess how well the model generalizes for a completely unseen surgeon, ensuring no subject-level overlap between training and testing.

Another dataset is the Robotic Surgical Maneuvers (ROSMA) dataset, a large dataset collected using the da Vinci Research Kit (dVRK) in collaboration between the University of Malaga (Spain) and the Biorobotics Institute of the Scuola Superiore Sant’ Anna (Italy)[37]. The ROSMA dataset is relatively newer than the JIGSAWS dataset and has more participants and trials.

Meanwhile, both the JIGSAWS and ROSMA datasets are publicly available benchmark datasets. Ethical approval, subject consent procedures, and proper de-identification have already been obtained and managed by the original data holders prior to release. Therefore, no additional ethical approval was required for our use of these datasets.

Pre-processing

The videos used in this work are captured at the rate of 30 frames per second and the spatial dimensions are 640 pixels in width and 480 pixels in height. However, the raw video as input causes out of memory (OOM) error on the 16 GB memory NVIDIA GPU. Thus, down-sampling at both temporal and spatial dimensions is performed on the videos. The video is first temporally re-sampled to 6 frames per second by selectively retaining only the first frame from every consecutive block of five original frames. Subsequently, each of these frames is spatially down sampled to a maximum border length at 128 while retaining the original aspect ratio by using linear interpolation. Zero-padding is then applied to the resized frames to fulfill the frame into a size of 128 × 128. The pre-processed videos are eventually saved with a size of 128 × 128 pixels and a frame rate of 6 per second to retain the same video length as the raw videos for better reviewing.

I3D-SAP video classifier

We propose a 3D CNN based model that takes the whole stream as input instead of snippets to reduce the bias of defining the labels for segmented snippets. Meanwhile, using the whole stream as input can accelerate the inference time since it requires only one time of computation instead of multiple times of sequential computation like recurrent neural network (RNN)[38] based video classification solutions[39]. Our proposed model has two major components, one is the 3D CNN backbone for feature extraction and another is SAP for feature reduction over time sequence.

3D CNN backbone

Since the number of data is limited in JIGSAWS dataset, we use the pre-trained 3D CNN based backbone called Inflated 3D CNN (I3D), which is inflated from 2D CNN by expanding all the filters and pooling kernel in the network with an additional temporal dimensionality to enable it to extract temporal relation between video frames[40]. Typically, the $N\times N$ kernel is inflated into $N\times N\times N$ kernel. However, some pooling kernels’ inflation is designed to be asymmetric to avoid losing lower-level temporal information.

The I3D, whose initial weights are bootstrapped from an Inception-V1[41] model pre-trained on ImageNet[42], has two pre-trained models with RGB video stream and optical flow as inputs. It is then further pre-trained on the Kinetics dataset[43], a large-scale and high-quality dataset focused on human action, and achieves the state-of-the-art performance with fine-tuning on UCF-101[44], HMDB-51[45], and mini Kinetics datasets. Moreover, averaging the prediction of the two pre-trained networks with RGB video stream and optical flow as inputs even achieves higher accuracy while comparing with those that only use one as input[40].

However, our I3D-SAP model only uses the pre-trained RGB model as the backbone, since optical flow is a time-consuming pre-process step compared with the other conventional video pre-process steps such as zooming and temporal resampling. Moreover, according to Funke et al[19], using the optical flow as input of the model does not enhance the model’s performance in an overall perspective compared to RGB while increasing lots of processing time to compute the optical flow from original video.

Sequential Average Pooling

As aforementioned, the whole stream is used as the input to the 3D CNN based backbone for feature extraction. Therefore, the size of the feature extracted can vary due to the difference in input size and the 3D CNN based backbone only conducts the size of the feature representation at specific rates. More specifically, the spatial size of the final feature representation is the same for different inputs while the sequential length should be different, which makes it impossible to feed into a classification layer based on the fully connected layer which requires input with fixed size.

Inspired by Global Average Pooling (GAP) proposed in a study by Lin et al[46], which takes the average value of each feature map as feature representation. We propose SAP as shown in Figure 2 to eliminate the problem of feature map size variance. The equation is as follows:

$$SAP\left(F\left(x,y,z\right)\right)=\frac{{\displaystyle \sum zF\left(x,y\right)}}{z}$$ (1)

Figure 2.

The illustration of SAP.

where F denotes the feature representation, x and y denote the two spatial dimension and z denotes the temporal dimension.

After SAP, the 3D feature representation with different sequential lengths will be reduced to a 2D feature representation with the same spatial size while maintaining more information than using GAP as a feature reduction strategy.

Model structure

The final model structure is shown in Figure 3 with the 3D CNN based backbone followed by an SAP layer and a fully connected layer before the Softmax function to generate the class probability from logit output. Moreover, as the limited data exists in the datasets and only three surgical skill levels need to be classified, shallower models may also perform well on this task. Therefore, we cut the backbone at different points to construct the lighter and faster models.

Figure 3.

The three I3D-SAP models (top) and the detailed description of the inception (Inc) module (bottom). The strides of the pooling layer (Max-Pool) and the convolutional layer (Conv) are default 1 if not specified. Meanwhile, Relu activation is included in the convolutional layer though it is not shown. The three I3D-SAP models end with the module (SAP + FC + Softmax) and the corresponding down-sampling rates are shown below the model name with the format of (frames, x, y).

SAP based class activation map

CAM is a technique that makes the network explainable by correlating the focusing area on the specific input with a specific category in classification tasks. It requires the last two layers of the network to be a GAP layer and an FC layer so that the weights can be projected back on the output of the last convolutional layer to generate the class activation maps. These then highlight the discriminative class-related regions[32].

For a given image, let $f_k\left(x,y\right)$ denote the value at spatial location $\left(x,y\right)$ in feature map k of the last convolutional layer. The average value of each feature map is $\frac{{\sum }^x,y{f}_k\left(x,y\right)}{N}$ where $N$ is the total pixels in a feature map. Thus, the class score $S_c$ for a specific class is ${\sum }^{F^k}{w}_k^c$ where $w_k^c$ denotes the $k^{th}weight$ of specific class $c$. By replacing $F^k$ with $\frac{{\sum }^x,y{f}_k\left(x,y\right)}{N}$, we obtain

$$S_c=\frac{\sum x,y\sum k{f}_k(x,y)W_k^c}{N}$$ (2)

After projecting the result back to feature maps, we obtain

$$M_c\left(x,y\right)=\frac{\sum k{f}_k\left(x,y\right)W_k^c}{N}$$ (3)

where $M_c$ denotes the activation map of a specific class.

As the activation map is in the format of a heat map which considers relative intensity among the map, the equation can be simplified by eliminating $N$.

$$M_c(x,y)=\sum f_k(x,y)W_k^c$$ (4)

Since our models use SAP instead of GAP, the feature maps are pooled over the temporal dimension instead of spatial dimensions as GAP applied on 2D images. Therefore, we revise the equation to

$$M_c(x,y,z)=\sum _{k,x,y}{f}_k(x,y,z)W_{k,x,y}^c$$ (5)

where $z$ in our case represents the temporal dimension, $w_{k,x,y}^c$ represents the weight at position $\left(x,y\right)$ in $k^{th}$ weight matrix for class $c$.

Furthermore, as SAP performs on the temporal dimension, SAP based CAM should also provide a coarse saliency visualization over the temporal dimension similar to the original CAM which provides coarse focusing on the spatial dimensions. Therefore, we average the generated saliency map over spatial dimensions and obtain the final equation as follows:

$$M_c(z)=\frac{{\sum }_{x,y}{\sum }_{k,x,y}{f}_k(x,y,z)W_{k,x,y}^c}{xy}$$ (6)

The general illustration of our proposed SAP based CAM is shown in Figure 4 and $w$ in the figure represents the weight matrix as aforementioned in Equation 5.

Figure 4.

The illustration of our SAP based CAM.

Experiments

Implementation details

The experiments are conducted on Google Colab Tesla P100 GPU. All the experiments are implemented in Python. The model is optimized by using gradient descent with momentum to accelerate the gradient descent in the relative direction and suppress oscillations in the parameter space. Moreover, due to the different sequential lengths of the input videos, the input data cannot be batched and fed into the network. Therefore, instead of using a mini-batch, we accumulate the gradient every 10 times to achieve the same effect as batched training. Furthermore, all the experiments share the same training hyperparameter settings with learning rate set to 0.01, momentum set to 0.9, weight decay set to 0, both L1 and L2 regularization set to 1e⁻¹⁰. During our training, the early stopping strategy is also applied when both the training accuracy stops increasing and the training loss stop decreasing for more than 3 epochs. Moreover, we conduct experiments on three model structures of different depths as shown in Figure 3, namely model 1, model 2, and model 3.

Leave One Super-trial Out Cross-validation

In the experiments, the LOSO cross-validation setting is used to evaluate the generalizability of model structures. Our proposed video classifier with whole stream video as input has achieved the best accuracy of 100, 96.3, and 97.2% over three models with different lengths on the three surgical skills tasks, i.e., suturing, needle passing, and knot tying in surgical skills curricula, respectively. Moreover, the 95% confidence interval for

The detailed experimental results in precision, recall and F1-score are shown in Tables 2, 3, and 4. The confusion matrixes of the classification results are shown in Figures 5, 6, and 7.

Table 2.

LOSO cross-validation result for three elementary tasks (model 1).

	Knot tying			Needle passing			Suturing
	precision	recall	f1	precision	recall	f1	precision	recall	f1
Novice	88.9	100.0	94.1	95.2	100.0	97.6	100	100	100
Intermediate	100.0	80.0	88.9	93.8	93.8	93.8	100	100	100
Expert	100.0	100.0	100.0	94.1	88.9	91.4	100	100	100
Average	96.3	93.3	94.4	94.4	94.2	94.3	100	100	100
95% CI	0.89–1.00	0.80–1.00	0.89–1.0	0.94–0.95	0.89–1.00	0.92–0.97	1.00–1.00	1.00–1.00	1.00–1.00
Accuracy (%)	94.4			94.4			100
AUC	0.94			0.99			1.0

Table 3.

LOSO cross-validation result for three elementary tasks (model 2).

	Knot tying			Needle passing			Suturing
	precision	recall	f1	precision	recall	f1	precision	recall	f1
Novice	94.1	100.0	97.0	95.2	100.0	97.6	100	100	100
Intermediate	100.0	90.0	94.7	94.1	100.0	97.0	100	100	100
Expert	100.0	100.0	100.0	100.0	88.9	94.1	100	100	100
Average	98.4	96.7	97.2	96.5	96.3	96.2	100	100	100
95% CI	0.94–1.00	0.90–1.00	0.95–0.99	0.94–0.96	0.89–1.00	0.94–0.97	1.00–1.00	1.00–1.00	1.00–1.00
Accuracy (%)	97.2			96.3			100
AUC	0.96			0.99			1.0

Table 4.

LOSO cross-validation result for three elementary tasks (model 3).

	Knot tying			Needle passing			Suturing
	precision	recall	f1	precision	recall	f1	precision	recall	f1
Novice	91.4	100.0	95.5	100.0	80.0	88.9	97.4	97.4	97.4
Intermediate	89.5	85.0	87.2	72.7	100.0	84.2	95.0	95.0	95.0
Expert	100.0	90.0	94.7	100.0	88.9	94.1	100	100	100
Average	93.6	91.7	92.5	90.9	89.6	89.1	97.5	97.5	97.5
95% CI	0.90–1.00	0.85–1.00	0.87–0.95	0.72–1.00	0.80–1.00	0.84–0.94	0.95–1.00	0.95–1.00	0.95–1.00
Accuracy (%)	93.0			88.9			97.4
AUC	0.94			0.91			0.96

Figure 5.

Confusion matrix of the classification results for model1 with LOSO cross-validation.

Figure 6.

Confusion matrix of the classification results for model2 with LOSO cross-validation.

Figure 7.

Confusion matrix of the classification results for model3 with LOSO cross-validation.

Ablation study

We conduct ablation studies on the proposed I3D-SAP model by systematically removing certain components of the I3D-SAP model. One of the ablation studies is to explore the contribution of different model depths and the other is to validate the effect of different input quality. The three models’ performances are evaluated on three independent elementary tasks. Results are shown in Tables 2, 3, and 4. Models 1 and 2 achieve 100% accuracy in recognizing the suturing task, while model 2 performs slightly better than model 1 in both the needle passing and knot-tying tasks. With further inspection of the predictions, we have found that the wrong predictions often have a confidence score around 50%, while most correct predictions have a confidence score over 90% on the predicted and ground-truth classes. Meanwhile, we have also found that the wrong predictions among both models’ cross-validation results overlap, which means the wrongly predicted items are similar in the two models. Since the two different depth models have similar wrong predictions, we assume the same wrong predictions might be at the periphery of the distribution.

Consequently, we believe the shallower model 2 outperforms model 1 according to its higher accuracy and faster inference time which takes advantage of its shallower architecture. Model 3, which has a much shallower architecture than either model 1 or model 2 (Fig. 3), is much more efficient when inferencing due to a significant reduction in computation required. However, the shallow architecture also leads to learned features containing less task-related information, which further causes a drop in the accuracy in the other three evaluation matrices for all three independent elementary tasks (Table 4).

The main implementation place would be hospitals or medical school either in small or large scale, and the implementation cost for either training or inferencing should not exceed too much. According to our experiments, the resolution of 128 × 128 and sampling rate of 6 frames per second is the maximum quality that an GPU with 16 GB memory can handle, which indicates that this video quality setting is also the maximum input quality for most consumer level GPU. However, we have further conducted ablation study on input with larger frame sample rate and frame resolution with the results shown in Table 5 on NVIDIA A6000 GPU (48G). The results indicate that with both larger frame sampling rate and frame resolution, the overall performance can be improved, specifically for the needle-passing task. However, with larger frame sampling rate and frame resolution, the request for GPU memory is also raised significantly, which means the implementation cost of infrastructure will raise significantly as well. Compared to the cost and the improvements, the resolution of 128 × 128 and sampling rate of 6 frames per second would be the best solution for the current stage according to its comparable performance and low cost to implementation, therefore it is feasible to be implemented in more places to accelerate the surgical training process.

Table 5.

Experimental result of ablation study on input with larger sample rate and resolution using JIGSAWS dataset and LOSO cross-validation.

Experimental setting		Min GPU memory (GB)	Average inference time (second per video)	Accuracy (%)
Sample rate	Resolution			Knot tying	Needle passing	Suturing
10	128	32	3.6	97.2	98.2	100.0
6	128	16	2.2	97.2	96.3	100.0
6	192	48	4.1	97.2	100.0	100.0

Leave One User Out cross-validation

Although carrying out experiments with the LOUO setup is important to evaluate the model’s ability to generalize for unseen operators, our results (41.7% accuracy on average, Table 6) reveal a large performance gap compared to LOSO. This suggests that the model is strongly dependent on subject distribution and struggles to learn subject-invariant representations. A key reason lies in the limitations of the JIGSAWS dataset: it contains only eight participants, with just two experts and two intermediates, resulting in unbalanced training sets when one participant is left out. For instance, when an expert is used as the test subject, the training data contain very few expert-level samples, making generalization especially difficult.

Table 6.

LOUO cross-validation result for three elementary tasks.

	Knot tying			Needle passing			Suturing
	precision	recall	f1	precision	recall	f1	precision	recall	f1
Novice	0.51	0.66	0.58	0.50	0.45	0.47	0.78	0.92	0.84
Intermediate	0.06	0.05	0.05	0.14	0.19	0.16	0.00	0.00	0.00
Expert	0.43	0.30	0.35	0.00	0.00	0.00	0.50	0.50	0.50
Average	0.33	0.34	0.33	0.21	0.21	0.21	0.43	0.47	0.45
Accuracy (%)	38.9	22.2					57.7
Averaged Accuracy (%)				41.7

This limitation reflects a broader challenge of evaluating cross-user generalization on JIGSAWS rather than a shortcoming of our specific model. Nevertheless, it highlights the importance of methodological advances to address distributional biases. Potential directions include cross-user or cross-institution training with multiple datasets following the same data collecting strategies, advanced data augmentation using high-quality generative data, and transfer learning with a strong video-based model under reduced-sample conditions. Most importantly, validation on larger and more diverse surgical datasets will be critical to demonstrate meaningful cross-user generalizability in automated surgical skill assessment.

SAP based CAM results for I3D-SAP

The general screenshot of the CAM visualization is shown in Figure 8 including temporal class activation map and trajectory with class activation information based on colormap turbo where the colors range from cold to hot representing saliency from low to high. The results are then analyzed together with video synchronized under the supervision of surgical experts.

Figure 8.

Visualization of the trail’s trajectory. The top sub-figure is the 3D trajectory visualization of the two captures, where the axes represent for the spatial position measure X, Y, Z. The bottom sub-figure is the temporal visualization of the two captures, where the horizontal axis represents for the time.

Regardless of tasks, the model focuses more on the movement of the instruments when a video is classified as novice while focuses less and less on the movement when a video is classified as intermediate or even expert. We assume it is because a novice surgeon is less familiar with the operation and the movement of the instrument is less smooth and fluent than that of intermediate and expert surgeons. Meanwhile, spatially the model always shows more interests in the background and less in the action. However, we have found the little spatial interests in the action still affect a lot in a temporal context. In other words, the less interested action parts mostly determine the classification results of the videos.

Apart from generally analyzing the CAM visualization regardless of tasks, we further analyze it with specific tasks: suturing, knot tying, and needle passing. As for suturing, the model focuses more on the significant amplitude of handling the sutures and the rotation of the needle when a video is classified as a novice, while it shows no special interests when a video is classified as intermediate. However, when a suturing video is classified as an expert, the model shows interest in the cooperation of the left-hand instrument with the right-hand one, such as passing the needle. Therefore, we assume an expert surgeon performs better instrument collaboration and smaller amplitude in suturing tasks than that of intermediate and novice. For knot tying, the model focuses more on the first half part of the knot tying action, including the movement of the instruments and twisting the sutures to a circle when a video is classified as a novice.

The same for suturing, the model shows no special interests for the class of intermediate. For the knot tying video classified as an expert, the model focuses more on the second half part of knot tying action, including passing the needle across the circle twisted in the former part and sutures handling follows. We assume a novice surgeon mostly struggles to twist the sutures in knot tying while an expert surgeon is more skillful for that and can deal with the following operation well. As for needle passing, the model shows no particular interest for the class of expert while shows interest in passing needles across the small hole for both novice and intermediate. Though the model focuses on the same action for novice and intermediate, there is still a difference. The model focuses on the slack action for the novice while the action seems more fluent for the intermediate. Since the model shows no particular interest in expert, which is quite different from suturing and knot tying, we assume needle passing is quite different from the other two tasks, and the classification is mostly based on the fluency of passing the needle across the small hole.

Moreover, combining the sequential CAM visualization with the video can help the trainees understand which parts of their operations lead to their surgical skill level predictions as novice, intermediate or expert. The results can be used to guide their future practice in surgical skill training and help them improve quickly.

Snippet-wise visualization for I3D-SAP

Furthermore, ascribed to the nature of our proposed I3D-SAP model accepting different ranges of input length, it is easy to adapt our model to an implementation of snippet-based inference without changing the model’s structure and parameters. We divide the whole-length videos into small snippets using the sliding windows method with 50% overlap each and feed the small snippets into the model to obtain the corresponding predicted skill level categories probabilities. We also sequentially visualize the predicted probabilities with an example of visualization for LOSO 2 validation set is shown in Figure 9.

Figure 9.

Snippet-wise visualization for I3D-SAP of a randomly selected LOSO validation set (LOSO 2). The predicted probabilities for each skill level category are visualized in blue, yellow and red for novice, intermediate and expert, respectively. “K” denotes knot-tying, “N” denotes needle passing, and “S” denotes suturing. The horizontal axis of each sub-figure represents for the time and the vertical axis represents for the probability of the snippet to be identified as certain class.

Compared with the overall prediction results, the snippet-wise visualization can provide a sequential explanation of the final prediction by giving the skill level predictions of relatively small periods. Meanwhile, unlike the CAM visualization which tries to explain the model’s results from the saliency map derived from the top predicted category, the snippets-wise visualization explains the model differently by subdividing the input. From the sequential visualization, where an example of visualization for LOSO 2 validation set is shown in Figure 9, we have found that for both novice and expert categories, the snippet-wise visualization produces a clearer result compared with that of intermediate. More specifically, for the novice category, most of the snippets appear to be a novice with high probability while a small part of snippets might be predicted as intermediate or even expert. In other words, the blue curves are at the top for almost all the time when a video is predicted as novice. A similar situation happens for the expert category with the red curves at the top for almost all the time. However, as for the intermediate category, it seems to be a more complex situation that the three color curves often cross each other, which indicates that the intermediate category is somehow more difficult to be recognized compared with the novice and expert classes. From the overall perspective, the results indicate that a novice would probably have most of the operations at the novice skill level, an expert would probably have most of the operations at the expert level, an intermediate would probably have the operations at a mixture of all three levels from novice to expert, which matches the normal cognition.

Moreover, combining the snippet-wise visualization with the video allows the trainees to review their operations and know which parts of operations are qualified to be retained and which parts of operations need to be improved. And even for the experts, it can also remind them of their unqualified operations and help them improve their skills continuously.

Additional experiments on ROSMA dataset

Since the ROSMA dataset does not provide a specific skill level annotation for the trials, we formulate it as a binary classification problem (high skill or low skill) with two senior surgeons manually annotating the dataset independently. Two board-certified senior surgeons independently annotate all trials. The annotation is conducted in a double-blinded manner, where reviewers are unaware of each other’s scores to prevent bias. The labeling criteria were based on a holistic assessment of surgical skill, integrating key components such as the economy of motion, instrument handling and overall procedural fluency. While subjective, these criteria are well-established in surgical assessment and grounded in the reviewers’ extensive clinical expertise. The inter-rater reliability, measured by Cohen’s Kappa coefficient, was 0.37, indicating fair agreement in the initial independent annotations. All disagreed cases are then discussed in a consensus meeting between the two surgeons by jointly reviewing until a unanimous final label is agreed upon for each. These consensus labels serve as the ground truth for our study. The subjective nature of the annotations introduces a potential limitation; therefore, the results should be interpreted with an understanding that model performance is benchmarked against expert human judgement.

To evaluate the performance of our model on the ROSMA dataset, we mimic the LOUO design of the JIGSAWS dataset to perform the cross-validation process, with one participant’ data as testing set and the remaining as training set. The results are shown in Table 7 with an averaged accuracy of 56.8% for two level surgical skill classification. For three tasks, namely Wire Chaser, Pea on a Peg, Post and Sleeve, our model performs similarly with the accuracy slightly over 50%. Similar to the JIGSAWS dataset, the ROSMA dataset also has limited number of participants and trials, specifically 12 participants and 206 trials in total, which introduces bias to both of the annotation and the model training process. In other words, we only have 11 participants’ data for training and 1 participant’s data for testing, which most likely results in lower accuracy.

Table 7.

LOUO cross-validation result for the ROSMA dataset.

	Knot Tying			Needle Passing			Suturing
	precision	recall	f1	precision	recall	f1	precision	recall	f1
Low skill	0.62	0.62	0.62	0.48	0.42	0.45	0.51	0.60	0.55
High skill	0.59	0.59	0.59	0.58	0.64	0.61	0.60	0.51	0.55
Average	0.61	0.61	0.61	0.54	0.54	0.54	0.56	0.55	0.55
Accuracy (%)	60.6	54.3					55.4
Averaged Accuracy (%)				56.8

Discussion

Comparison with the counterparts of the SOTA results

With LOSO experimental setup, Table 8 shows the performance comparison of our results with others for predicting the self-proclaimed surgical skill level. Using JIGSAWS kinematic data, Wang et al[16] reported that the CNN method achieved the accuracy of 94.1%, 90.3%, and 86.8% on the three surgical tasks, Nguyen et al[17] reported that the CNN-LSTM method achieved the accuracy of 98.4%, 98.4%, and 94.8% on the three surgical tasks, and Fawaz et al[15] reported that the FCN method achieved 100%, 100%, and 92.1% on the three surgical tasks, respectively. Using the JIGSAWS video data, Funke et al[19] reported that the 3D CNN method achieved the accuracy of 100%, 100%, and 95.1% on the three surgical tasks, and Wang et al[18] reported that the multi-task learning framework achieved the accuracy of 100%, 100%, and 97.5% on the three surgical tasks, respectively. Compared to these works, our work has achieved a state-of-the-art performance.

Table 8.

Performance comparison of the proposed method with some of the state-of-the-art results on surgical skill level classification.

Author	Method	Accuracy (%)
		Suturing	Needle passing	Knot tying
Wang et al (2018)[16]	CNN (kinematic)	94.1	90.3	86.8
Nguyen et al (2019)[17]	CNN-LSTM (kinematic)	98.4	98.4	94.8
Fawaz et al (2018)[15]	FCN (kinematic)	100	100	92.1
Funke et al (2019)[19]	3D CNN (video)	100	100	95.1
Wang et al (2020)[18]	Multi-task learning framework (video)	100	100	97.5
Ours	I3D-SAP	100	96.3	97.2

The bold values indicated the best performance.

Advantages

In this study, we propose a novel approach to use the whole stream video for surgical skill level prediction and also provide interpretations to predictions. We put forward an I3D-SAP model which uses I3D as the backbone for feature extraction and follows an SAP module for sequential pooling to eliminate the problem of length variance. Our model achieves 100% accuracy for the suturing task, 96.3% accuracy for the needle-passing task and 97.2% for the knot tying task with LOSO setup. With the SAP module, we further propose a SAP based CAM to explain the decision strategies of the model how to classify a video into a specific category. Furthermore, ascribed to the nature of our model can have inputs with different lengths, we also provide snippet-wise visualization to explain the model’s decision strategies from a different perspective. Due to the absence of widely accepted quantitative metrics for evaluating explainable artificial intelligence methods, we have conducted a structured questionnaire-based assessment to evaluate the clinically perceived quality of our two proposed approaches: CAM and snippet-wise visualization. Two senior surgeons were invited to complete the questionnaire, and they were asked to rate each method along two key dimensions: satisfaction with the interpretability output and trust in the proposed explainable artificial intelligence method. The questionnaire items are listed in Table 9. Responses were collected on a 5-point Likert scale (1 = very dissatisfied/untrustworthy; 5 = very satisfied/trustworthy). The results showed that CAM received an average satisfaction score of 4.0 and a trust score of 3.5, whereas snippet-wise visualization achieved the same level of satisfaction 4.0 but a higher trust score 4.0. These results show that both explainable artificial intelligence methods are helpful for surgical skill evaluation.

Table 9.

Questionnaire details for the explainable artificial intelligence evaluation.

Question1: How satisfied are you with the proposed explainable artificial intelligence method? Please rate on a scale of 1 to 5.

Question2: How much do you trust the proposed explainable artificial intelligence method? Please rate on a scale of 1 to 5.

Limitations

In addition to the advantages of our approach, there is another point to mention its limitations. Compared to other snippets-based video classifiers, the main limitation of our whole-stream-based strategy is memory-consuming, for which we down-sample the videos either in both spatial and temporal dimensions to resolve. In addition, since there is only one publicly available benchmark dataset, JIGSAWS, to evaluate robotic surgical skills, our experiments can only be performed on it. Meanwhile, due to the relatively small sample number of participants and the unbalanced distribution of their surgical skill level, the performance of the proposed method is not reliable for LOUO validation. More data is needed to test the generalization ability of our proposed framework.

In particular, the substantial gap between the LOSO accuracy and the LOUO accuracy (average 41.7%) on JIGSAWS dataset indicates that the current model exhibits limited ability to generalize for unseen surgeons. This discrepancy likely arises from several factors: the small number of participants in JIGSAWS, which encourages models to learn surgeon-specific execution patterns rather than task-invariant skill representations; inherent stylistic differences among surgeons, which may dominate the visual and motion cues and confound the model when no samples from the target surgeon appear in training; and the whole-stream input design, which, while capturing richer temporal information, may also amplify characteristic patterns that do not generalize across individuals. These factors collectively suggest that the strong performance observed under LOSO does not necessarily translate to simulated real-world deployment where new surgeons are encountered. Therefore, caution is needed when interpreting the current model’s generalizability, and additional evidence is required before its use in practical skill assessment settings. More data covering a wider range of participants, surgeon styles, and task executions is crucial to improving the robustness of the proposed framework and mitigating potential overfitting to individual surgeons.

Furthermore, the model’s performance on the ROSMA dataset, with an average accuracy of 56.8%, was only marginally above chance level. While the limited number of participants in ROSMA is a contributing factor, this result necessitates a more comprehensive discussion. The near-chance performance suggests that the model’s ability of surgeon-independent generalization is currently weak. This aligns with and reinforces the concerns about surgeon-independent generalization identified in the JIGSAWS LOUO evaluation. It indicates that the model’s learned representations may be susceptible to dataset-specific biases or may not adequately capture fundamental, transferable skill features across varied surgical contexts.

Beyond the generalization issues, the evaluation of interpretability in this study also has inherent subjectivity. Although we employed CAM visualization and snippet-wise analysis to illustrate the model’s decision rationale, these observations largely rely on qualitative interpretation. The questionnaire-based assessment with two senior surgeons further provides initial quantitative insights, but the very small sample size limits the statistical persuasiveness and may not represent broader clinical perspectives. Therefore, the current interpretability evaluation should be regarded as exploratory rather than definitive.

Future directions

Future work should focus on enhancing the generalizability, usability, and scalability of the proposed I3D-SAP framework. One common way is to extend the evaluation to more diverse and complex datasets, either public or private^[47,48^]. Besides, integrating additional modalities such as robotic kinematics, force/torque signals, or eye-tracking may provide a richer and more nuanced understanding of surgical skill^[49,50^]. Moreover, to address data privacy concerns, federated learning and privacy-preserving techniques should be adopted to enable collaborative model training across institutions without sharing sensitive surgical data^[51,52^].

In addition, given the observed LOUO performance gap, future studies should explore strategies specifically targeted at improving surgeon-independent generalization, such as domain generalization techniques, style-invariant representation learning, meta-learning, or explicit disentanglement of personal style from objective skill indicators. Larger multi-center datasets and prospective clinical validations will be essential steps toward real-world translation.

Meanwhile, to strengthen objectivity for the evaluation of interpretability in future, more rigorous and quantifiable validation methods should be incorporated. Potential directions include computing the spatial overlap between the model’s highlighted attention regions and manually annotated critical operative areas by multiple expert surgeons; performing insertion/deletion tests to measure the causal contribution of highlighted regions to the final prediction; and conducting multi-center user studies with a larger pool of surgeons to assess the trustfulness of the interpretability results statistically. These approaches can help establish a more reliable and reproducible evaluation framework for explainable surgical skill assessment.

Conclusion

We propose an interpretable video model named I3D-SAP for surgical skill assessment of basic procedural elements. It has achieved comparable performance on JIGSAWS with accuracy of 100%, 96.3%, and 97.2% for suturing, needle passing, and knot tying, respectively. The proposed I3D-SAP model takes the whole video stream as input for surgical skill classification, and provides visual reasoning and interpretability by both CAM visualization with gradient perspective and snippet-wise visualization with the perspective of small-time period prediction.

However, the study identifies important limitations that temper the initial high performance. The model exhibits a significant performance decline under the more realistic LOUO validation scheme, and its accuracy falls to near chance levels on the external ROSMA dataset. These results highlight a critical challenge: the model’s current susceptibility to surgeon-specific patterns and difficulty in generalizing to entirely novel operators and surgical contexts. This underscores the need for caution in interpreting these results for real-world application, where generalization is significant.

Therefore, while I3D-SAP provides a foundation for interpretable automated assessment, its practical translation requires future work focused on improving generalization. This includes validation on larger, multi-center datasets, the exploration of domain generalization and style-invariant learning techniques, and more rigorous clinical evaluation. Overall, the model represents a step towards objective, transparent skill assessment, but its current generalizability constraints must be explicitly acknowledged and addressed for it to be reliably integrated into surgical training environments.

Ethical approval

This research study does not involve patients. All of the used data are public available with relevant ethical approval.

Consent

This study does not contain patient data.

Sources of funding

This research is supported by National Key R&D Program of China (Grant No. 2022YFB3503404) and Ningbo Science & Technology Innovation 2025 Major Project (Grant No. 2022Z126).

Conflicts of interest disclosure

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this study.

Author contributions

K.C. and Y.W.: conceptualization; K.C., Y.W., and Y.Z.: data curation; K.C. and Y.W.: formal analysis; K.C. and Y.W.: methodology; Y.W.: project administration; K.C.: software; Y.W. and B.W.: supervision; K.C., Y.W., Y.Z., B.W., Z.L., and K.T.: validation; K.C.: visualization; K.C.: writing – original draft; Y.W., Y.Z., B.W. Z.L., and K.T.: writing – reviewing & editing. All authors have read and approved the current version of the manuscript.

Research registration unique identifying number (UIN)

This research study does not involve human subjects, so there is no need for UIN.

Guarantor

Ying Weng.

Provenance and peer review

This paper was invited

Data availability statement

All datasets used in this study are publicly available.

Statement of generative AI usage

In this work, generative AI is neither used for code generation nor manuscript writing.

Acknowledgements

Not applicable.