Patient-specific fine-tuning of a self-distilled transformer model for normal tissue segmentation in abdominal magnetic resonance guided adaptive radiotherapy

Abstract

Background and Purpose:

Accurate segmentation of organs-at-risk is critical for adaptive Magnetic Resonance guided radiotherapy (MRgRT) in upper gastrointestinal (GI) treatments. Large anatomical variations pose challenges for automated segmentation methods, requiring considerable manual editing. Our goal was to develop and evaluate the effectiveness of patient-specific (PS) fine-tuning of a pretrained transformer model for segmenting upper GI structures.

Materials and Methods:

A PS fine-tuning approach was implemented by adapting a pretrained transformer model called Self-distilled Masked Image Transformer (SMIT). SMIT was trained using T2-weighted MR images from 30 patients with pancreatic cancers who underwent MRgRT to create a general model (GM). This GM was subsequently adapted to each new individual patient using data from their first treatment fraction, producing a PS model. Its performance was evaluated across the subsequent treatment fractions in 10 patients using Dice similarity coefficient (DSC), Hausdorff distance (HD95), and added path length (APL) by comparing against expert delineations.

Results:

PS model outperformed the GM across all structures, with the largest improvements for duodenum-stomach (DSC: 0.84 to 0.90, HD95: 15.9 to 4.9 mm). This approach reduced potential contour editing efforts, decreasing median APL by 4%, 7% and 26% for large bowel, small bowel, and duodenum-stomach.

Conclusion:

PS fine-tuning was most effective for scans with low soft-tissue contrast and challenging anatomy. This method provides a practical, computationally feasible approach to enhance online adaptive workflow by improving segmentation accuracy and potentially reducing the need for manual editing that overall improves on-table treatment times.

Graphical abstract

Highlights

• •Hausdorff distance reduced from 15.9 mm to 4.9 mm for duodenum and stomach structures.
• •Dice similarity coefficient improved from 0.53 to 0.88 for post-surgical anatomy.
• •One treatment fraction sufficient for patient-specific segmentation model training.

1. Introduction

Magnetic resonance-guided radiotherapy (MRgRT) has enhanced abdominal radiotherapy treatments by providing high-contrast visualization of tumors and organs at risk (OARs) during treatment [1]. Many MR-Linear accelerator (Linac) platforms are now commercially available, integrating a Linac with an onboard MR imager to enable daily adaptive replanning tailored to anatomical variations [2], [3]. This capability is particularly beneficial for tumors abutting the upper gastrointestinal (GI) organs, which exhibit substantial inter and intra-fractional motion and deformation [4]. A prominent application of this modality is the stereotactic MR-guided adaptive radiation therapy (SMART) trial, which leverages MRgRT to safely deliver ablative doses in only five fractions for pancreatic cancer. While conventional radiation treatments for pancreatic cancer typically require 5–6 weeks, the SMART trial demonstrated that an ablative 5-fraction adaptive MRgRT regimen could be delivered safely [5], [6], [7], [8], [9], [10], [11].

Accurate segmentation of OARs on daily MR scans is essential for plan optimization and dose calculation within adaptive workflows [12]. In radiation treatments involving upper GI organs such as the stomach, duodenum, small bowel, and large bowel accurate delineation is essential to minimize treatment-related toxicity [13]. Manual contouring could substantially extend the duration of each adaptive fraction, impacting clinical efficiency and patient access to adaptive treatments [14].

Deep learning approaches have shown promising results for automated segmentation of abdominal organs on MR images [15]. Transformer-based architectures, in particular, demonstrated superior performance compared to traditional convolutional neural networks (CNNs) due to their ability to capture long-range dependencies and contextual information [16], [17]. One example of a pretrained transformer applied to GI organ segmentation was the Self-distilled Masked Image Transformer (SMIT) that was shown to provide accurate segmentation of abdominal organs both from Computed Tomography (CT) and MR images [18].

Nevertheless, models trained on diversified datasets have been shown to be challenging on anatomically complex structures, such as bowel loops, emphasizing the shortcomings of purely general models [19], [20]. Ultimately, extensive manual editing is often required to correct patient-specific (PS) anatomical variations [14]. To address these issues, recent research explored PS adaptation methods that fine-tune general models (GM) on a small number of annotated images from the same patient, significantly improving segmentation accuracy [21], [22], [23], [24]. In the context of radiotherapy, where patients undergo multiple treatment fractions, this approach was especially well-suited, as high-quality contours created during simulation or the first treatment fraction could be leveraged to enhance segmentation performance on daily MR scans. In this study, the aim was to evaluate transformer-based PS fine-tuning for upper GI OAR segmentation, with the goal of improving segmentation accuracy in adaptive MRgRT, which would potentially reduce adaptive contouring time.

2. Materials and methods

2.1. Dataset and model training

This study utilized T2-weighted MR images acquired from upper GI radiotherapy patients treated on a 1.5 $T$ hybrid MR-Linac platform (Elekta Unity, Elekta AB, Stockholm, Sweden) using a compression belt to minimize respiratory motion. MR images were acquired using a standardized Elekta Unity protocol (TR = 1300 ms, TE = 86.75 ms, flip angle = 90 degrees, slice thickness = 2 mm, in-plane resolution = 1 × 1 mm${}^2$). These images were normalized [0,1] following intensity clipping using 5% and 95% signal intensity values for individual cases.

This study was performed under our institution approved IRB (#21-129). All patients underwent Stereotactic Body Radiotherapy (SBRT) of the pancreas, delivered in five fractions, consisting of two dose levels: an intermediate region prescribed to 33.0 Gy and a boosted region receiving 50.0 Gy. The dataset was structured to support both GM and PS training.

For GM training, the dataset included MR images from 30 upper GI patients, each with expert-delineated contours for the stomach-duodenum (combined as a single structure), small bowel, large bowel, and liver. The data were randomly divided into 70% for training and 30% for validation.

For PS training, an independent dataset of 10 additional patients treated with 5-fraction SBRT was used. Expert-verified contours were available for all fractions (totaling 50 scans). The first fraction’s contours were used to fine-tune the GM weights, adapting the model to each patient. The remaining fractions served as an independent test set to evaluate segmentation performance. The overall PS fine-tuning workflow was illustrated in Fig. 1.

Fig. 1.

Overview of the proposed PS segmentation workflow based on the SMIT AI model. Step 1: GM segmentation training. Step 2: Fraction 1 contours AI generated with GM. Step 3: Contours were manually verified and edited post treatment. Step 4: Verified contours sent for PS model fine-tuning, with selective layer freezing. Step 5: The PS model segmented subsequent treatment fraction images (fractions 2–5).

Additionally, a deformable registration workflow was implemented for comparison with the GM and PS models. The first fraction was manually rigidly registered to each subsequent fraction using bone-based alignment, followed by deformable contour propagation using the MIM Maestro v7.3.7 deformable registration algorithm. The resulting propagated contours were compared against the AI-generated segmentations.

2.2. Model architecture

Our approach built upon the SMIT architecture, selected for its versatile applicability to segmentation of GI OARs from CT and MR images [18], [25], [26]. The code for this model is available on GitHub (https://github.com/The-Veeraraghavan-Lab/SMIT). Further details on the model can be found in the Supplementary Materials A.

The model processed input volumes of size 128 × 128 × 128 voxels, generating segmentation predictions at the original resolution through sliding window inference with 0.5 patch overlap. The transformer encoder processed images using a patch size of 2 × 2 × 2 voxels with a feature embedding dimension of 1024.

2.3. Patient-specific fine-tuning strategy

Our PS fine-tuning strategy adapted the general SMIT model to an individual patient’s anatomy using data from a single treatment fraction. To optimize computational efficiency and model accuracy, we systematically evaluated different selective layer-freezing schemes within the segmentation model’s architecture. Specifically, we explored six distinct layer-freezing patterns, summarized in Table 1.

Table 1.

Evaluation of selective layer-freezing patterns during PS fine-tuning.

Pattern	Trainable Components	Mean DSC
1	Decoder layers only	0.84
2	Decoder + Stage 1 transformer and feature encoder blocks	0.85
3	Decoder + Stages 1 and 2 transformer and feature encoder blocks	0.85
4	Decoder + Stages 1, 2, and 3 transformer and feature encoder blocks	0.85
5	Decoder + Stages 1–4 transformer and feature encoder blocks	0.86
6	All encoder and decoder layers	0.86

Each configuration was trained using the PS fine-tuning workflow and evaluated on the subsequent fraction (Fraction 2) across all 10 test patients. The overall Dice Similarity Coefficient (DSC), averaged across all structures, was used as the performance metric to identify the optimal freezing strategy.

To maximize the utility of the single training fraction, we implemented a data augmentation pipeline for model training. Our approach incorporated comprehensive spatial augmentations including random flipping along all three axes with a probability of 0.2, random 90°rotations with 0.2 probability, and random affine transformations with small rotation ranges of 0.05 radians, translation of up to 5 voxels, and scaling factors of 0.1. Additionally, we applied random intensity scaling with factors of 0.3 at a probability of 0.3 and random intensity shifting with offsets of 0.3 with 0.3 probability. Our training process utilized patch-based sampling with random cropping of 128 × 128 × 128 patches using balanced sampling between positive and negative examples. To optimize memory usage during PS training, we extracted 2 crops per iteration, while maintaining data diversity through the enhanced augmentation pipeline.

The PS fine-tuning was performed using a combined DSC and Cross-Entropy loss function with AdamW optimizer [27], a learning rate of 1e-5, weight decay of 1e-5, 300 epochs and batch size of 1. For this specific PS approach, traditional validation using separate datasets was not applicable.

2.4. Training efficiency and computational requirements

The model was implemented using the PyTorch (v1.12.1, CUDA 11.6, cuDNN 8.3.2) and MONAI (v0.8.0) frameworks in Python (v3.9.12). Training and inference were performed on NVIDIA A40 GPUs. For PS fine-tuning, the complete process took approximately 1–2 h per patient on a single GPU for roughly 300 epochs.

For the GM and PS models, inference time was approximately 3 min per scan, which included exporting DICOM data from MIM to AI pipeline, preprocessing the image, running inference and exporting back into MIM.

2.5. Evaluation metrics and statistical analysis

We evaluated segmentation performance of both the GM and PS model on subsequent treatment fractions using DSC and 95th percentile Hausdorff Distance (HD95), which quantify volumetric overlap and boundary accuracy, respectively [28]. To estimate clinical editing effort, we calculated the Added Path Length (APL) using 2 mm arc-length resampling, a metric shown to correlate with manual contouring time and served as a surrogate for quantifying contouring workload [29], [30]. We additionally assessed model performance within the treatment volume, defined as the region extending 2 cm beyond the Planning Target Volume (PTV), thereby excluding distant peripheral areas whose segmentation accuracy would have negligible impact on treatment plan optimization.

Statistical analysis was performed using paired patient-level comparisons between the PS model and both the GM and deformable methods for each segmentation metric (DSC and HD95). For each patient, a median value was first computed across fractions 2–5, yielding a single representative value per patient per structure. The overall summary statistics reported represented the median of these patient-level medians across all 10 patients. Comparisons between models were performed using the non-parametric Wilcoxon signed-rank test on patient-level medians, with statistical significance defined as $p<0.05$. Percent differences were computed from the overall medians (median of patient-level medians), comparing PS with GM and deformable models for each structure and metric.

3. Results

Overall, the selective layer-freezing strategies had minimal impact on segmentation accuracy, showing only marginal variations across configurations. Pattern 2, where the decoder, Stage 1 transformer and feature encoder blocks remained trainable, offered the best balance between computational efficiency and performance. Although the fully trainable configuration (Pattern 6) achieved slightly higher scores (DSC = 0.86), the selective approach provided comparable accuracy with reduced computational cost. Accordingly, we adopted Pattern 2 as a practical compromise to maintain stable performance while facilitating efficient memory loading during training.

PS fine-tuning improved segmentation accuracy over GM for all structures. Fig. 2 shows boxplot distributions for each metric across all three methods, both for all structures and for those within 2 cm of the PTV.

Fig. 2.

Performance comparison by structure showing per-patient median DSC and HD95 distributions for deformable contours (purple), GM (salmon), and PS (blue) models across large bowel, small bowel and duodenum-stomach structures. The boxplots on the left represented results over the full volume, whereas the dashed boxplots on the right showed performance restricted to the region within 2 cm of the PTV. Each patient data point corresponded to the median value of the evaluated metric for a given patient across fractions 2-5. Boxes indicated the interquartile range, while the whiskers showed the minimum and maximum values after excluding outliers.

Across fractions 2–5, the PS model outperformed both the GM and deformable methods across all OARs and segmentation metrics (Table 2; Table S1). The most important improvements were observed for duodenum-stomach and small bowel, with HD95 reductions of 69% and 51%, respectively. Large bowel and liver showed smaller improvements (HD95 reductions of 43% and 11%, respectively). Compared to deformable contour propagation, the PS model achieved higher DSC values across all structures, with small bowel and duodenum-stomach showing the greatest differences.

Table 2.

Segmentation performance summary showing median [range] of patient-level medians for DSC, HD95 (mm), and APL ($\times 10^3\mathrm{mm}$) across all 10 patients. For each patient, the median was first computed across fractions 2–5; reported values represent the median and range of these patient-level medians. Percentage differences compare the PS model to deformable contours and GM.

Structure	Model	DSC	HD95 (mm)	APL ($\times 10^3\mathrm{mm}$)
Liver	Deformable	0.96 [0.83, 0.98]	3.8 [2.0, 10.4]	20.2 [14.4, 33.8]
	General	0.96 [0.64, 0.98]	2.8 [2.0, 24.3]	18.6 [12.8, 39.6]
	Patient-Specific	0.97 [0.95, 0.98]	2.5 [1.4, 4.9]	14.0 [11.8, 26.3]
	PS–Deform (%)	＋1⁎	–34⁎	–31
	PS–GM (%)	＋1⁎	–11⁎	–25
Duodenum-Stomach	Deformable	0.71 [0.49, 0.87]	13.6 [4.2, 28.8]	14.5 [11.2, 32.0]
	General	0.84 [0.50, 0.90]	15.9 [5.7, 42.8]	14.1 [10.2, 29.1]
	Patient-Specific	0.90 [0.72, 0.94]	4.9 [2.0, 18.3]	10.5 [7.0, 28.0]
	PS–Deform (%)	＋27⁎	–64⁎	–27
	PS–GM (%)	＋7⁎	–69⁎	–26
Large Bowel	Deformable	0.71 [0.40, 0.81]	15.7 [12.0, 53.0]	35.5 [28.7, 59.2]
	General	0.89 [0.34, 0.94]	8.2 [2.1, 41.2]	19.1 [13.1, 65.2]
	Patient-Specific	0.90 [0.71, 0.94]	4.7 [2.5, 23.4]	18.4 [13.1, 51.0]
	PS–Deform (%)	＋27⁎	–70⁎	–48
	PS–GM (%)	＋1⁎	–43⁎	–4
Small Bowel	Deformable	0.58 [0.35, 0.79]	25.4 [15.5, 133.1]	47.2 [13.3, 71.2]
	General	0.75 [0.39, 0.91]	19.3 [5.2, 56.9]	34.2 [11.8, 52.3]
	Patient-Specific	0.80 [0.65, 0.92]	9.5 [4.8, 21.0]	31.7 [9.0, 48.7]
	PS–Deform (%)	＋38⁎	–63⁎	–33
	PS–GM (%)	＋7⁎	–51⁎	–7

Wilcoxon signed-rank test comparing PS to Deform/GM on patient-level medians.

^*p $<$ 0.05.

The evaluation restricted to the 2 cm region surrounding the PTV showed similar overall trends when comparing the PS model to the GM (Table S2). Again, the most important improvements were observed for duodenum-stomach and small bowel, with HD95 reductions of 80% and 58%, respectively, while large bowel and liver showed smaller improvements (HD95 reductions of 14% and 25%, respectively).

Individual patient analysis revealed heterogeneous responses to the PS fine-tuning approach, with the magnitude of improvement varying across patients (Table S1). The greatest gains with PS models relative to GMs were observed for Patients 2, 4, and 7, as illustrated in Fig. 3. Patient 2 presented with abnormally dark liver imaging, which caused the GM to achieve a median DSC of only 0.64; the PS model improved liver DSC to 0.95. Additionally, duodenum–stomach DSC increased from 0.50 (GM) to 0.80 (PS) for this patient.

Fig. 3.

Representative cases demonstrated improved performance of PS models in challenging clinical scenarios. Case A (Patient 2, axial): Liver segmentation failure in GM due to poor image quality was corrected by PS adaptation, while deformable contours also performed well. Case B (Patient 7, axial): Post-Whipple surgery anatomy with displaced stomach location confounded GM but was accurately segmented by PS model; deformable contours were also partially successful, though less accurate than PS. Case C (Patient 4, sagittal): Poor duodenum-stomach contrast limited GM performance was overcome by PS model, which achieved better ground truth agreement. Deformable contours showed some success but remained inferior to PS. Case D (Patient 10, axial): Representative example illustrating improved segmentation of the duodenum–stomach structure by the PS model compared to both the GM and deformable contours.

Patient 7 demonstrated significant post-surgical anatomical changes due to Whipple surgery, displacing the stomach. The GM achieved a duodenum–stomach DSC of 0.53, whereas the PS model reached 0.88. Small bowel segmentation also improved from 0.69 (GM) to 0.82 (PS).

Patient 4 presented compromised image quality with reduced soft-tissue contrast. The PS model improved large bowel DSC from 0.34 (GM) to 0.71 and small bowel DSC from 0.39 (GM) to 0.65.

In comparison with deformable contour propagation, cases where the GM underperformed exhibited improved performance with the propagated contours; nevertheless, the PS model achieved the highest overall accuracy. Conversely, Patients 1, 6, and 9 showed minimal benefit from PS adaptation, as their anatomical features were already well represented by the GM, with baseline DSC values exceeding 0.90 for most structures. In these cases, deformable contours yielded the lowest accuracy among the three methods.

4. Discussion

This study demonstrated that PS fine-tuning of a general task-specific hybrid transformer-convolutional network model created from a pretrained transformer encoder improved the accuracy and consistency of OAR delineation in adaptive MRgRT for upper GI treatments. By leveraging only a single treatment fraction for adaptation, our approach achieved notable improvements, particularly in challenging anatomical scenarios.

When examining the APL results, we observed overall reductions between the GM and PS models. Using APL as a surrogate for potential reductions in contour editing time, these findings were encouraging. Therefore, based on our study, we believe a clinical implementation is warranted to properly evaluate the impact of the proposed approach on contouring time reduction in clinical practice [29], [30].

Our performance results demonstrated similar advantages to those reported by Li et al. [24] and Kawula et al. [23], though key differences exist in the segmentation structures used. We selected the four structures listed above because they represented critical OARs prioritized in the SMART trial, where the large bowel, small bowel, duodenum, and stomach were treated as separate structures [7]. In our analysis, the duodenum and stomach were combined into a single structure to reflect our current clinical practice, where physicians agreed to evaluate them jointly based on their shared dosimetric endpoints. While separate segmentation would be ideal, particularly for metrics involving large volume constraints such as D${}_{5{\mathrm{cm}}^3}$, combining them can lead to more conservative plans that may unnecessarily compromise target coverage. As this study was intended to showcase improvements relative to our existing workflow, we therefore maintained the evaluation of this combined structure.

Compared to Li et al. [24], our model achieved superior small bowel segmentation after training on a single fraction, with a DSC of 0.80 and HD95 of 9.5 mm. In contrast, they reported approximately 0.73 DSC and 40.0 mm HD95. Their performance approached our DSC only after training their model with four fractions of images and achieved comparable HD95 after two. Although their study included the stomach, a direct comparison is non-trivial since we combined the stomach and duodenum in our evaluation.

Unlike Kawula et al. [23], who appeared to evaluate the bowel as a single structure, our study separated it into the large and small bowel. When comparing their bowel structure to our individually segmented counterparts, our model showed better agreement, with higher DSC and lower HD95 values for both the large and small bowel. However, these comparisons should be interpreted with caution given differences in MR systems (0.35 $T$ vs. 1.5 T MR-Linac) and dataset sizes. While Kawula et al. used a larger dataset, they noted potential inaccuracies in contours outside the treatment field. In our study, we fully recontoured each case to address concerns about the accuracy of manually corrected deformed contours from daily MR images, particularly beyond 2 cm from the target volumes. Although such inaccuracies may have limited dosimetric impact, they could introduce misleading anatomical features that hinder model training [23]. Our goal was to develop a workflow that performed well across the entire patient volume, as this will be beneficial for future work on dose accumulation using deformable registration. Therefore, it was important in our study to validate that the proposed method achieved convincing results across the full patient dataset.

This study had a few limitations. First, it represented a single-center investigation conducted on a single Elekta MR-Linac scanner. Although its applicability across institutions was not evaluated, previous studies have indicated that PS model training can be applied effectively across diverse clinical environments [21], [22], [23], [24]. Another limitation was the absence of a comparison with a other existing convolutional architectures, including an nnU-Net. Our goal was not to demonstrate superiority over existing architectures but to illustrate that the proposed methodology can be successfully applied to large foundation models with convincing results. Furthermore, comparison of a pretrained model to a scratch trained nnUnet is not meaningful as the former makes use of pretraining to extract useful features. Future work could include a systematic evaluation of multiple architectures using this approach to determine whether one model consistently outperforms others.

Additionally, during PS fine-tuning, no separate validation dataset was used to monitor performance or guide hyperparameter selection (e.g., number of epochs). Default training parameters from the GM setup were used for retraining. Although overfitting may be a concern, it is important to emphasize that this technique was intentionally designed to adapt the model to a specific patient at the cost of generalizability. Any application beyond the fine-tuned case would be at the user’s own discretion and risk, as model reliability cannot be ensured outside the intended domain. Future work could further investigate strategies to balance patient-level adaptation with overfitting control or quantify the extent to which PS models lose general performance.

The patient cohort in this study was relatively small, and a larger population would be valuable to assess not only segmentation performance gains but also potential reductions in contouring time. While previous studies have validated APL as an efficiency metric correlated with manual contouring time [29], [30], its use here served primarily as a surrogate measure of efficiency. Future clinical implementation on a larger patient cohort will be necessary to determine whether the observed improvements translate into measurable reductions in contouring time in routine practice.

In this study, we demonstrated that PS fine-tuning of transformer-based segmentation models significantly improved auto-segmentation accuracy for adaptive MR-guided upper GI radiotherapy. By leveraging the first treatment fraction for model adaptation, our approach achieved improvements across all evaluated OARs compared to the GM. This personalized approach to auto-segmentation represents a practical advancement toward more efficient adaptive radiotherapy workflows by reducing the burden of manual contouring.

CRediT authorship contribution statement

Nicolas Côté: Conceptualization, Methodology, Software, Formal analysis, Investigation, Writing – original draft. Jue Jiang: Software – review & editing. Sharif Elguindi: Software, Validation – review & editing. Marsha Reyngold: Validation, Data curation – review & editing. Christopher Crane: Validation, Data curation – review & editing. Ifeanyirochukwu Onochie: Data curation – review & editing. Harini Veeraraghavan: Supervision – review & editing. Arthur Lalonde: Supervision – review & editing. Neelam Tyagi: Supervision, Project administration – review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary material related to this article.

MMC S1

SMIT model architecture details and per-patient segmentation metrics for all abdominal OARs.