Assessing Surgical Skill in Orthopaedic Trauma Surgery Training

Abstract

Background:

Surgical skill assessment in orthopaedic trauma surgery still relies on subjective expert ratings, which limits consistency and scalability. While digitalization offers a path toward objective and scalable assessment, the highly manual and haptic nature of surgery makes tool use difficult to capture digitally, keeping such approaches underdeveloped. This study introduces a digital assessment framework for orthopaedic trauma training that derives digital behavioral metrics (DBM) from tracked surgical tool motion. Specifically, it investigates (1) which DBM indicate technical proficiency and (2) to what extent these DBM are capable of predicting expert-rated surgical performance.

Methods:

Twenty-eight participants performed 3 standardized fracture fixations on synthetic bone models of the radius, ulna, and fibula. Tool motion was captured and transformed into a digital twin from which metrics such as path length, smoothness, and task duration were derived. These metrics were statistically compared with the average Global Rating Scale (GRS) obtained from 4 experts who rated each surgical performance. (1) Correlation analysis identified skill-relevant metrics and (2) a predictive model was trained to estimate performance from DBM evaluating its accuracy against the individual expert ratings.

Results:

(1) Several DBM were found to be indicative of surgical performance. Measures based on tool path length and time per activity showed strong correlations with expert ratings, reaching coefficients of up to 0.6. Correlation strength varied across tools and procedures. (2) The predictive model achieved a mean absolute difference of 3.8 points from the average GRS score (scale range: 28-70), outperforming the mean interexpert difference of 4.6 points.

Conclusion:

DBM were identified as valid indicators of surgical skill. The study further demonstrated their predictive value, showing closer alignment with experts’ average GRS score than individual expert ratings. These findings highlight the feasibility of objective, expert-independent performance assessment in orthopaedic trauma surgery training.

Introduction

Accurate assessment of surgical skill is central to surgical education, guiding both feedback for technical development and trainee selection^1,2. The current standard in surgical performance assessment still relies primarily on pen-and-paper-based tools. Structured questionnaires have been developed for different surgical procedures and have demonstrated high reliability and validity across simulation models and real operating room settings^3,4. Such tools face 2 key challenges related to expert raters: subjective scoring with interrater variability⁵, and limited expert availability which restricts regular practice and skill refinement⁶.

In recent years, digital training and assessment methods have been developed to address these limitations of expert-based evaluations. The approaches assess surgical performance by analyzing instrument motion during task execution^7-9. From tracked tool movements, quantitative indicators of execution quality can be derived, with general movement characteristics such as smoothness, regularity, and efficiency shown to correlate with technical performance. Such digital assessment concepts are widely established in laparoscopic, arthroscopic, and robotic surgery training, where surgical instruments are inherently suited for motion tracking^9-11.

In orthopaedic trauma, however, digital performance assessment is still at an early stage of development, despite prior work demonstrating the feasibility of using motion analysis for objective skill assessment in trauma simulation¹². The specialty depends highly on physical interactions and haptic feedback which creates difficulties when translating the training environment into a fully digital space¹³. Unlike in minimally invasive or robotic surgery, the tools used in fracture fixation—such as drills, screwdrivers, and forceps—are not inherently equipped with sensors, and meaningful training requires hands-on practice on physical models to preserve the haptic experience. Consequently, defining and capturing of objective digital metrics for performance evaluation become challenging¹⁴. Even with tool motion data, it remains unclear (1) which metrics meaningfully reflect surgical proficiency, and (2) to what extent these metrics are capable of predicting surgical performance.

To address these challenges, this study introduces a digital assessment framework for orthopaedic trauma. Using an optical tracking system, movements of relevant surgical tools are recorded to create a digital representation that mirrors the physical fracture fixation task, referred to as a digital twin. Within this digital twin, tool motion can be analyzed and clinically relevant digital behavioral metrics (DBM), such as speed, smoothness, and efficiency of tool use, are computed for a performance assessment.

In the study, 28 participants performing 3 standardized procedures (radius, ulna, and fibula) were evaluated by 4 raters using the Global Rating Scale (GRS)¹⁵, with averaged scores serving as reference ratings. Based on this data set, the study investigates DBM in 2 ways: (1) by correlating DBM with averaged expert ratings to identify performance indicators, and (2) by developing and testing a DBM-based predictive model to evaluate the capability of these metrics to estimate surgical performance.

By addressing these challenges, this work may advance efforts to establish objective assessment standards in orthopaedic trauma surgery and supports the development of scalable, expert-independent training solutions.

Methods

Part 1: Digital Training Technology

The proposed training setup (see Fig. 1) consists of standard surgical tools and implants (drills, screwdrivers, pointed forceps, bone forceps, plates, and screws); synthetic bone models (Synbone AG, Zizers, Switzerland) of the radius, ulna, and fibula; an optical tracking system (FusionTrack 250, Atracsys LLC); and a fixture that enables realistic anatomical positioning of the bones.

Fig. 1.

Training setup with synthetic bone models, surgical tools, and implants. Motion tracking via reflective markers creates a real-time digital twin for objective performance assessment.

Tool motion is captured with the optical tracking system positioned approximately 1.2 m from the setup and angled downward at 45° relative to the horizontal. Reflective marker sets, each with 3 retroreflective spheres, are rigidly attached to surgical tools and the bone fixture, enabling 6-degree-of-freedom tracking.

The tracking data are used to generate a digital twin of the surgical scene, providing a real-time digital representation of the physical training procedure. Implemented in the Unity game engine (Unity Technologies, San Francisco, California), this representation enables visualization of tool trajectories and extraction of DBM describing movement patterns. An example visualization of tool trajectories is shown in Fig. 2.

Fig. 2.

Digital twin visualization of tool trajectories (pointed forceps orange, drill violet, screwdriver green) during the radius case. The system enables full workflow reconstruction and extraction of DBM for objective performance assessment. DBM = digital behavioral metrics.

Digital Behavioral Metrics

DBM were selected based on prior work in surgical motion analysis, particularly in minimally invasive and robotic procedures^10,16, and on performance indicators reported in orthopaedic trauma surgery⁵. The set (Table I) captures key dimensions of tool motion, including how fast the tools move (velocity), how smooth and consistent the movements are (movement regularity), how efficiently space is used during motion (spatial efficiency), and how the movements are organized over time (procedural efficiency).

TABLE I.

Extracted DBM Derived from the Digital Twin, Categorized by Motion Type and Computed at Both Activity and Procedure Level to Characterize Surgical Performance

Activity-based
Movement speed
Avg. translational velocity	Describes how fast the tool moves on average	m/s
Avg. rotational velocity	Describes how fast the tool rotates on average	Deg/s
Movement regularity
Std. translational velocity	Describes how much the speed of the tool varies	m/s
Std. of rotational velocity	Describes how much the rotational speed varies	Deg/s
Translational velocity entropy	Describes how variable the speed of the tool is	-
Rotational velocity entropy	Describes how variable the rotational speed of the tool is	-
Translational centroid of frequency	Describes whether tool motion is predominantly continuous or characterized by frequent changes in motion	Hz
Rotational centroid of frequency	Describes whether tool rotation is predominantly continuous or characterized by frequent changes in motion	Hz
Spatial efficiency
Tortuosity	Describes how straight or indirect the tool path is during an activity	-
Point density	Describes how densely tool positions are distributed in space	Points/m3
Overall procedure-based
Spatial efficiency
Accumulated tool path	Describes the total distance the tool travels during the procedure	m
Accumulated tool rotation	Describes the total amount of tool rotation during the procedure	degrees
Procedural efficiency
No. of activities	Describes how often the procedure is broken into separate activity segments	-
Avg activity time	Describes the average duration of a single-activity segment	s
Std activity time	Describes how much the duration of activity segments varies throughout the procedure	s

Avg = average, Std = standard, Deg = degrees (deg), m = meter, s = seconds, and Hz = Hertz.

Metrics were computed separately for each tracked tool (drill, screwdriver, bone forceps, and pointed forceps). Tool use was segmented into activity episodes, defined as continuous use of a tool within the predefined working area. For each episode, DBMs describing movement speed, regularity, and spatial efficiency were extracted. Procedure-level metrics were then derived by aggregating all episodes, including measures describing spatial and procedural efficiency.

Part 2: Experimental Study

Staff members from the Department of Orthopedic and Trauma Surgery at Cantonal Hospital Lucerne were invited to voluntarily participate. Individuals with prior exposure to the setup were excluded. The Cantonal Ethics Committee confirmed the study was not subject to the Swiss Human Research Act; no formal approval was required (Req-2025-00367). Informed consent was obtained from all participants.

Study Protocol

Participants performed 3 standardized fracture fixations on synthetic bone models: (1) a simple oblique radial shaft fracture (AO/OTA classification 2R2A1) treated with a compression plate, (2) a simple malleolar fracture (AO/OTA classification 42A3) treated with a neutralization plate, and (3) a multifragmentary ulnar shaft fracture (AO/OTA classification 2U2C2) treated with a bridge plate.

Before performing the tasks, all participants received standardized instruction based on AO principles¹⁷ through written guidelines and videos. Participants were guided according to these materials but performed all tasks independently. Screws were provided upon request.

The generation of a real-time digital twin of the surgical scene with motion tracking during each procedure enabled continuous extraction of DBM. All procedures were video-recorded and anonymized for expert review.

Expert Ratings

Procedure recordings were independently reviewed by 4 consultants with 8 to 40 years of surgical experience serving as expert raters. Performances were rated using the GRS which covers domains such as tool handling, motion efficiency, and procedural flow. The soft tissue handling item was omitted as not applicable. Scores from the 4 experts were averaged to create a consensus-based reference for validating the digital behavioral metrics. To optimize review time, videos were shown at 4× playback speed. All evaluators were blinded to DBM as well as to participant identity and experience level.

Statistical Analysis

Statistical analyses were performed using Python (version 3.10; Python Software Foundation, Wilmington). All recorded procedures were included and linked to their corresponding DBM and the average GRS score from the expert ratings. Interrater reliability was assessed using the intraclass correlation coefficient (ICC). The data set was used to identify DBM indicative for surgical performance and create a predictive model for automated skill assessment.

Correlation Analysis

Performance-relevant metrics were identified using Spearman rank correlation between individual DBM and average GRS scores. Correlations were computed separately for each tool and case to account for tool-specific and case-specific performance patterns.

Predictive Capabilities of Digital Behavioral Metrics

To predict surgical performance from DBM, a linear mixed-effects regression model was trained using the benchmark GRS score as the dependent variable. DBM served as predictors after dimensionality reduction via principal component analysis (PCA). A random intercept for participant ID accounted for repeated measurements. Model performance was evaluated using leave-one-out cross-validation (LOOCV)¹⁸. In each iteration, the model was fitted using data from all participants except 1 and then tested on that remaining participant. This process was repeated until every participant had been used once as the test case. Prediction accuracy was measured as the mean absolute error (MAE) between predicted and average GRS scores and compared against the average MAE of individual expert ratings relative to the average score.

Results

A total of 83 procedures performed by 28 participants were analyzed (28 radius, 28 fibula, and 27 ulna procedures). One participant was excluded for prior exposure, and another completed only 2 procedures due to time constraints. The final cohort (11 female, 17 male) included 6 medical students, 13 residents, 6 senior physicians, and 3 chief physicians. The average age was 33.8 years (SD = 9.7), with a mean of 6.6 years of surgical experience (SD = 8.6 years).

Evaluation of the procedures by 4 experts using the GRS resulted in 332 individual assessments. The average GRS rating across all participants was 50.6 (SD = 9.57), ranging from 28 to 70 points, with case-specific averages of 48.6 (SD = 9.35) for the radius, 50.7 (SD = 9.55) for the fibula, and 52.7 (SD = 8.17) for the ulna procedures. ICC among expert assessments was 0.88, indicating good consistency¹⁹. For all procedures, a corresponding set of DBM was generated.

Correlation Analysis for Performance Indicators

Correlations between average GRS scores from expert ratings and DBM were calculated across all procedures and analyzed by tool and case type. Strong correlations were particularly observed for time-based and path length metrics across all tools and procedure types, with Spearman correlation coefficients reaching a magnitude of 0.6 (Fig. 3).

Fig. 3.

Spearman correlation coefficients between average GRS scores and DBM, stratified by tool and procedure type (radius, malleolus, and ulna). Each bar shows the strength and direction of the correlation. Higher absolute values indicate stronger associations with surgical performance. DBM = digital behavioral metrics, and GRS = Global Rating Scale.

Other metrics exhibited tool-specific variability but also showed notable correlations with expert ratings. Particularly strong correlations were found for the drill and pointed forceps, whereas the bone forceps showed comparatively weaker association. When broken down by case type, correlations were higher for radius and malleolus procedures than for the ulna case.

Predictive Capabilities of Digital Behavioral Metrics

Leave-one-out cross-validation of the linear mixed-effects model predicting the GRS score yielded a MAE of 3.8 (SD ± 3.2) points (on the GRS scale: 28-70) compared with the average GRS score. Experts differed on average by 4.6 (SD ± 3.8) from the average GRS rating. They are individually plotted in Fig. 4.

Fig. 4.

The average GRS score is plotted against the GRS scores predicted by the linear mixed-effects model using leave-one-out cross-validation (blue dots). Individual expert ratings (light-gray dots) show a broader spread around the average GRS score, which reflects their higher mean absolute error compared with the model (MAE 4.6 vs. 3.8 points). GRS = Global Rating Scale.

Discussion

This work introduces a training framework for orthopaedic trauma that translates real-world surgical training procedures with realistic haptic feedback into a digital representation, enabling the extraction of skill-relevant DBM for surgical skill assessment.

The results show that simple DBM, such as time-based measures and tool path length, correlate strongly with average expert GRS ratings. Time-based metrics quantify temporal efficiency where shorter completion times indicate greater procedural familiarity and fluency. Path length reflects spatial efficiency where shorter paths indicate more direct movements¹⁰. Correlation strength varied across tasks and tools, with the radius fracture showing the strongest alignment between DBM and average expert ratings. By contrast, the ulna case, being less complex without the need for lag screw placement, resulted in weaker correlations. This finding emphasizes the importance of selecting tasks with sufficient difficulty to discriminate between varying levels of expertise. This is consistent with prior studies showing that overly simple tasks can cause performance plateaus and reduce discriminative power^20,21. Tool-specific analysis revealed that metrics from the drill and pointed forceps had higher discriminative power than those from the screwdriver or bone forceps, suggesting that selecting technically demanding tools may streamline assessments without sacrificing accuracy. Focusing on such tools could allow for reduced setup complexity while still obtaining sufficient information on performance.

The potential of DBM was demonstrated through a predictive linear mixed-effects model, which estimated surgical performance with an average MAE of 3.8 points on the GRS scale—outperforming the experts’ MAE of 4.6 points. Thus, the digital assessment showed closer agreement with the consensus than individual raters, with an error margin well within natural human variability. This highlights the feasibility of using DBM for automated assessment in orthopaedic trauma surgery. These findings advance simulation-based orthopaedic trauma training by showing that even simple behavioral metrics from training procedures can reliably predict expert evaluations. While such predictions primarily provide a summative assessment, the framework also supports formative learning. Because the behavioral metrics can be analyzed at the level of individual tools and procedural steps, they could provide targeted feedback during self-directed practice to help trainees identify specific technical or procedural deficits. Ultimately, this dual utility supports the development of autonomous, instructor-free training environments as proposed in recent surgical simulation research.^22,23

While the digital framework enables objective assessment of real-world procedures and preserves realistic haptics, it presents limitations. Soft tissue—an essential component of orthopaedic trauma surgery—is not yet integrated. Incorporating it would increase procedural complexity and improve discriminative power. The framework also focuses on process-based behavioral metrics rather than final surgical outcomes, although procedural performance does not necessarily correlate with outcome quality²⁴. Integrating outcome measures would therefore complement DBMs and enable more comprehensive assessment. Recent work has shown that surgical implants can be included in the digital twin, allowing evaluation of outcome quality such as implant position or fracture alignment²⁵. A further limitation is the current reliance on high-precision optical tracking, as the associated setup effort may limit integration into routine clinical environments²⁶; more accessible and cost-effective tracking solutions could support broader adoption in surgical training.²⁷

Conclusion

This work presents a digital training setup for orthopaedic trauma surgery that preserves haptic feedback while enabling objective performance assessment through optical tracking. Several DBM showed strong correlations with expert ratings, and a predictive model achieved closer agreement with average GRS scores than individual evaluations (3.8 vs. 4.6 GRS points, range 28-70), demonstrating the potential of data-driven assessment. This work may contribute to the groundwork for standardized, expert-independent evaluation in orthopaedic trauma surgery training and supports the shift toward simulation-based, competency-driven medical education.