Risk stratification and cross-center prediction of thoracoabdominal 6D setup errors

Abstract

Purpose

Thoracoabdominal CBCT-derived six-degree-of-freedom (6DoF) setup errors vary substantially across centers. Using record-level setup events as the unit of analysis, we examined whether pre-treatment workflow variables could stratify the risk of large translational events, how they performed for infrequent rotational events, and whether model performance held up across centers.

Methods

This retrospective two-center cohort study included CBCT-derived 6DoF setup records collected during 2022–2025 at Center 1 and 2020–2022 at Center 2. After exact matching and deduplication, 6,033 unique records were retained from the original 6,584 records. The primary endpoint was three-dimensional composite translational displacement ≥ 5 mm (Y5mm), calculated using tvec. Secondary endpoints were tvec ≥ 10 mm (Y10mm), maximum rotational magnitude (rmax) ≥ 3° (R3°), and the composite endpoint Yaction, defined as either Y5mm or R3°. Center, anatomic site, tumor type, and immobilization method were used as predictors. CatBoost and XGBoost were evaluated with three-fold stratified cross-validation and bidirectional cross-center external validation.

Results

Event rates for Y5mm, Y10mm, R3°, and Yaction were 39.3%, 6.1%, 5.4%, and 40.9%, respectively. Key endpoint rates differed significantly between centers, suggesting marked domain shift. In internal validation, CatBoost and XGBoost performed similarly for Y5mm (AUPRC, 0.620 and 0.622, respectively), whereas prediction of R3° was limited. In bidirectional cross-center validation, performance for the primary and composite endpoints declined compared with internal validation and showed asymmetric transfer patterns. Calibration also worsened externally. SHAP analysis identified immobilization method as one of the most important features for Y5mm.

Conclusion

Thoracoabdominal CBCT-derived 6DoF setup errors show clear risk stratification patterns and substantial cross-center heterogeneity. Models based on pre-treatment workflow variables provided modest stratification for large translational events, but were less useful for rare large rotational events. For workflow deployment, local recalibration at the target center may be preferable to direct transfer of raw predicted probabilities.

Introduction

Accurate and reproducible patient setup is fundamental to maintaining target dose coverage and minimizing unintended irradiation of normal tissues in thoracoabdominal radiotherapy. Compared with treatment sites with relatively stable anatomy, the thoracic and abdominal regions are more susceptible to respiratory motion, soft-tissue deformation, patient comfort, and differences in immobilization, making interfraction geometric uncertainty inherently more complex. CBCT-based image-guided studies have shown that setup deviations are not uniformly distributed across anatomic sites, and thoracoabdominal targets often exhibit greater setup uncertainty than more anatomically stable regions [1, 2].

Cone-beam computed tomography (CBCT) has become an integral component of image-guided radiotherapy for thoracoabdominal treatment, allowing setup deviations to be detected and corrected before irradiation through registration with the planning CT. In this way, CBCT helps reduce geometric mismatch and also provides a practical basis for institution-specific evaluation of setup error patterns [1, 2]. In thoracoabdominal cases, CBCT is used not only for routine position correction, but also to compare the stability of different initial setup strategies and immobilization approaches. Previous studies suggest that immobilization devices, respiratory management, and certain clinical characteristics may all be associated with the magnitude of setup error, implying that setup risk is not evenly distributed across patients or treatment fractions [3, 4]. At the same time, more frequent CBCT verification is not always warranted. Beyond the additional imaging dose, intensified image verification also increases treatment-room occupancy and workflow burden, making optimization of verification frequency a practical issue in modern IGRT workflows [5].

Accordingly, a key clinical challenge is not only to maintain setup accuracy, but also to identify high-risk setup scenarios more selectively so that verification intensity and workflow resources can be allocated more efficiently [5]. What is needed in practice is not merely retrospective recognition of setup errors after they have occurred, but prospective stratification of the risk of large setup deviations before treatment delivery. If high-risk scenarios for large translational or rotational deviations could be identified using only routinely available pre-treatment workflow information—such as center, anatomic site, tumor type, and immobilization method—this approach could provide a more direct basis for selectively intensified CBCT verification, auditing of immobilization workflows, and optimization of image-guidance resources.

In recent years, artificial intelligence and machine learning have been increasingly applied in image-guided radiotherapy. Prior studies and reviews suggest that AI may contribute to target localization, motion monitoring, image registration, misalignment detection, and workflow automation, with the potential to reduce treatment uncertainty and improve delivery precision [6, 7]. Even so, most existing work has focused on image-level or registration-level tasks, such as misalignment detection, image correction, or setup-shift prediction in specific technical settings. By contrast, relatively few studies have addressed real-world thoracoabdominal CBCT-derived six-degree-of-freedom (6DoF) setup errors using routinely available pre-treatment workflow variables for risk stratification, let alone examined whether such models remain valid across centers [6, 7].

Against this background, we performed a retrospective two-center analysis using CBCT-derived 6DoF setup error records collected during routine thoracoabdominal radiotherapy workflows to investigate risk stratification and cross-center prediction of large translational and rotational events. We specifically evaluated the discriminative performance of pre-treatment workflow variables for key setup endpoints, the generalizability and interpretability of the models across centers, and the calibration behavior observed during cross-center transfer. By doing so, we sought to provide quantitative evidence for identifying high-risk scenarios, selectively intensifying verification, and optimizing setup workflows. Unlike image-level tasks that detect misalignment only after image acquisition, our goal was to prospectively stratify the risk of large setup deviations before treatment delivery using routine workflow variables alone, thereby offering earlier support for selective verification decisions.

Methods

Study design and data source

This was a retrospective two-center cohort study based on CBCT-derived six-degree-of-freedom (6DoF) setup error/correction records collected during routine thoracoabdominal radiotherapy workflows at Ganzhou Cancer Hospital (Center 1) and Sun Yat-sen University Cancer Center (Center 2). The data collection period was 2022–2025 for Center 1 and 2020–2022 for Center 2. At both centers, the initial setup workflow was generally consistent: therapists first positioned the assigned immobilization device on the treatment couch and then aligned the immobilization device and patient position using body-surface marks together with room lasers. For the CBCT-guided setup events included in this study, a CBCT scan was acquired and followed by online grayscale registration, which was reviewed before treatment; manual adjustment could be performed if the initial auto-match was considered clinically inappropriate. The system then recorded couch correction values in three translational and three rotational directions prior to couch shift, representing the final clinically accepted corrections rather than raw auto-match outputs.

Both centers used the Varian VitalBeam CBCT system (Varian Medical Systems, Palo Alto, CA, USA). The software version was 2.7 MR4 at Center 1 and 2.5 MR3 at Center 2. The systems entered routine clinical use in 2021 and 2019, respectively, and were in routine operation during the corresponding study periods. Both centers followed generally consistent setup and registration standards, ensuring comparable procedures for CBCT acquisition and couch corrections. As this was a retrospective study based on exported historical setup records, detailed equipment-level calibration logs were not available in the analytic dataset; however, both systems were operated within the institutions’ standard clinical calibration and quality assurance workflows.

The original exported database contained 6,584 records, including center, anatomic site, tumor type, immobilization method, and 6DoF correction values. In this study, each record represented a single system-recorded CBCT-guided setup error/correction event before couch shift, rather than an individual patient or a complete treatment fraction. Each record included the corresponding workflow descriptors together with the three translational and three rotational correction values recorded after online grayscale registration and before couch shift. In routine clinical practice, CBCT-guided setup verification was performed according to institutional protocols and clinical judgment; therefore, the exported database did not represent all treatment fractions and did not reliably link any additional verification CBCT scans within the same fraction. Accordingly, the exported database should be interpreted as a collection of clinically recorded CBCT-derived 6DoF setup error/correction events rather than as a complete treatment-fraction log.

Because the exported historical database was designed as a setup-record database rather than a patient-level treatment-course database, it did not consistently retain a stable de-identified patient identifier. Therefore, the exact number of unique patients contributing to these records could not be reliably reconstructed. Prescription dose and planned fractionation information were also not uniformly available. Accordingly, the present analysis was conducted at the setup-record level and focused on workflow-level risk stratification of CBCT-derived 6DoF correction events rather than patient-level treatment-course outcomes.

Because historical system exports may contain repeated exports or duplicate aggregation, rule-based deduplication was performed before the main analysis. As the analytic database did not consistently retain a unified unique record identifier, duplicate records were identified by exact matching on a prespecified set of key fields. After deduplication, 551 duplicate records were removed, leaving 6,033 unique records for the main analysis.

The primary unit of analysis was the individual record, reflecting the practical workflow scenario in which each setup event may correspond to one potential verification decision. Overall, the dataset provides a record-level view of clinically reviewed and accepted CBCT-derived 6DoF corrections, suitable for workflow-level risk stratification and machine learning modeling. Because the current dataset did not include a stable patient-level identifier, strictly patient-independent data splitting was not feasible. We therefore performed an additional sensitivity analysis using a more conservative stratified evaluation based on grouping keys derived from combinations of center, anatomic site, tumor type, and immobilization method, in order to examine the extent to which model performance depended on structural differences in case-mix composition. This analysis was intended only to provide a robustness bound and was not used to infer generalizability at the patient-independent level.

Quantification of setup errors and study endpoints

In this study, both translational and rotational setup errors refer to the six-degree-of-freedom correction values recorded by the system after CBCT acquisition and online grayscale registration, but before couch shift.

Three-dimensional translational error was quantified as the Euclidean norm of the three translational components:

1

The original translational values were recorded in centimeters and converted to millimeters for analysis. Based on tvec, a translational threshold spectrum was defined as:

2

Among these endpoints, Y5mm was prespecified as the primary endpoint and Y10mm as a secondary endpoint.

Rotational error was quantified as the maximum absolute value among the three rotational components:

3

with values expressed in degrees. Based on rmax, a rotational threshold spectrum was defined as:

4

Among these, R3° was prespecified as a key secondary endpoint.

A composite endpoint, Yaction, was further defined as:

5

to represent events that might warrant intensified setup verification or additional imaging attention. In addition to the primary and key secondary endpoints, event rates across the translational and rotational threshold spectra were also reported to characterize the risk gradient of setup errors. These thresholds were used for workflow-level risk stratification and should not be interpreted as new clinical margin recommendations. The endpoint thresholds were selected a priori as workflow-level operational thresholds rather than as new clinical margin recommendations. The 5-mm translational threshold was chosen as the primary endpoint because it represents a commonly used magnitude for clinically meaningful setup deviation and potential workflow attention in image-guided radiotherapy. The 10-mm threshold was included to capture rarer but more severe translational tail events. For rotational errors, the 3° threshold was selected as a pragmatic cutoff for large rotational deviations in 6DoF couch workflows. In addition, the use of threshold spectra for translation (1, 2, 3, 5, and 10 mm) and rotation (2°, 3°, and 5°) was intended to characterize the gradient from frequent small deviations to less frequent larger deviations, rather than to rely on a single binary cutoff.

Predictors, data preprocessing, and model development

Model inputs were restricted to routinely available pre-treatment workflow variables, including center, anatomic site, tumor type, and immobilization method; all were categorical. Before model development, categorical entries underwent string cleaning, category harmonization, and unified handling of missing values to reduce noise arising from inconsistencies in the original records. The 6DoF setup error values were used only for endpoint construction and descriptive analyses, and were not entered into the models in order to avoid target leakage.

CatBoost and XGBoost were used to develop binary classification models. The primary modeling task was Y5mm, and secondary tasks included Y10mm, R3°, and Yaction. For low-incidence endpoints, class-imbalance handling was applied during training: CatBoost used class weights, whereas XGBoost used scale_pos_weight. Model outputs were event probabilities and were used for subsequent evaluation of discrimination, probability calibration, and supplementary analyses.

Internal validation

Internal validation was performed using three-fold stratified cross-validation. Specifically, all records were stratified by endpoint label and split into three subsets. In each round, two subsets were used for model training and the remaining subset was used for validation. Validation predictions from all folds were then combined to generate pooled out-of-fold predictions, which served as the basis for internal performance evaluation and presentation of the main results.

Cross-center external validation

To assess model transportability across institutions, we performed bidirectional cross-center external validation. Models were trained on data from Center 1 (Ganzhou Cancer Hospital) and tested on data from Center 2 (Sun Yat-sen University Cancer Center), and vice versa. This design was intended to reflect a real-world external deployment scenario in which a model developed at one institution is transferred to another.

Because the variable “center” may capture both case-mix differences and potential shortcut information, we further considered two input settings—with-center and without-center—as a sensitivity analysis. This analysis was used to examine the extent to which inclusion of the center label influenced cross-center generalization.

Statistical analysis and model performance evaluation

Categorical variables are presented as counts and percentages. Between-center differences in key endpoints were compared using contingency table methods, and absolute risk differences with 95% confidence intervals (CIs) were calculated.

Model discrimination was assessed primarily by the area under the precision-recall curve (AUPRC), with the area under the receiver operating characteristic curve (AUROC) reported as a supplementary metric. Because several endpoints were relatively infrequent, AUPRC was prespecified as the main measure of discrimination. Probability calibration was evaluated using the Brier score and expected calibration error (ECE). 95% CIs for performance metrics were estimated by bootstrap resampling with 800 iterations.

Because AUPRC is strongly influenced by endpoint prevalence in the validation set, cross-center external validation results are reported not only as absolute AUPRC values, but also together with validation-set prevalence and ΔAUPRC (AUPRC − prevalence), to reflect the additional precision-recall gain over a random baseline. Because this was a retrospective study based on all eligible historical records available during the predefined study periods, no formal sample size calculation was performed before study initiation. Instead, the reliability of the findings was assessed by reporting the size of the analytic cohort and endpoint event counts, together with three-fold stratified cross-validation, bidirectional cross-center external validation, bootstrap-based confidence intervals, calibration assessment, and grouping-key-based sensitivity analysis.

Supplementary analyses and model interpretation

For the primary endpoint, predicted probabilities were further post-processed using isotonic regression, and calibration before and after recalibration was compared with reliability plots. We also examined how sensitivity, positive predictive value, and related operating characteristics changed across different decision thresholds, in order to illustrate how model outputs might be used in a selectively intensified verification workflow.

To assess robustness across clinical settings, subgroup analyses were performed by anatomic site and immobilization method. To further characterize the geometric pattern of immobilization-related risk differences, translational errors were decomposed by direction, and the median, 95th percentile, and tail-event rates of absolute errors were compared separately for the LR, SI, and AP axes.

Because the current dataset did not provide a stable patient-level identifier, a more conservative stratified sensitivity analysis was additionally performed using grouping keys derived from combinations of center, anatomic site, tumor type, and immobilization method, and the results were compared with the record-level analysis. Model interpretability was assessed using SHAP. For the primary endpoint, global feature importance was quantified and the directional contributions of major features to predicted risk were examined.

Ethics statement

This study was approved by the Medical Ethics Committee of Ganzhou Cancer Hospital (Approval No. [2026] Kelunshen No. 6) and was reviewed and approved by the Ethics Committee of Sun Yat-sen University Cancer Center (Approval Letter No. B2026-167-01; Document No. SL-B2026-167-01). The study was conducted in accordance with the Declaration of Helsinki, the International Council for Harmonisation Good Clinical Practice (ICH-GCP) principles, and relevant national regulations. Given the retrospective nature of the study and the use of de-identified clinical data, the requirement for informed consent was waived by the ethics committees.

Results

Cohort assembly and baseline composition

The original database included 6,584 CBCT-derived six-degree-of-freedom (6DoF) setup error records collected during thoracoabdominal radiotherapy workflows. After rule-based deduplication using exact matching on prespecified key fields, 551 duplicate records were removed, leaving 6,033 unique records in the main analytic cohort. Record-level analysis was used for both statistical evaluation and model development, reflecting the practical workflow setting in which each setup event may correspond to a potential verification decision (Fig. 1).

Fig. 1.

Study flowchart of cohort derivation and validation design. Note: The raw database included 6,584 CBCT-derived 6-DoF setup records. After exact deduplication based on 10 prespecified fields, 6,033 unique records were retained for the record-level main analysis. Model development used three-fold stratified cross-validation with pooled out-of-fold predictions, and external validation was performed bidirectionally between the two centers. Models were evaluated under both with-center and without-center settings. CBCT, cone-beam computed tomography; OOF, out-of-fold

In the main analytic cohort, 3,826 records came from Center 1 and 2,207 from Center 2. Overall, thoracic records accounted for 53.1% of the cohort and abdominal records for 46.9%. The case mix differed substantially between the two centers: Center 1 was dominated by thoracic records, whereas abdominal records were more common in Center 2. The distribution of immobilization methods also differed markedly between centers, pointing to real-world heterogeneity in both case mix and workflow characteristics (Table 1).

Table 1.

Baseline characteristics of the record-level main analysis cohort

Characteristic	Overall(N = 6,033)	Center1(N = 3,826)	Center2(N = 2,207)
Site
Thorax	3,205 (53.1)	2,440 (63.8)	765 (34.7)
Abdomen	2,828 (46.9)	1,386 (36.2)	1,442 (65.3)
Disease
Cervical cancer	1,191 (19.7)	674 (17.6)	517 (23.4)
Lung cancer	1,167 (19.3)	912 (23.8)	255 (11.6)
Rectal cancer	1,043 (17.3)	118 (3.1)	925 (41.9)
Esophageal cancer	845 (14.0)	335 (8.8)	510 (23.1)
Breast cancer	810 (13.4)	810 (21.2)	0 (0.0)
Other	977 (16.2)	977 (25.5)	0 (0.0)
Immobilization
Vacuum bag	3,366 (55.8)	2,084 (54.5)	1,282 (58.1)
Cerv-thor mask + thermoplastic pad	1,315 (21.8)	1,315 (34.4)	0 (0.0)
Orfit frame	925 (15.3)	0 (0.0)	925 (41.9)
Other	427 (7.1)	427 (11.2)	0 (0.0)

Data are presented as n (%). Percentages were calculated within each column. The main analysis cohort was defined at the record level after exact deduplication. Less frequent disease and immobilization categories were grouped as “Other”

Event spectrum of setup endpoints and between-center differences

In the main analytic cohort (N = 6,033), event rates across the translational threshold spectrum declined monotonically as the threshold increased. The event rates of Y1mm, Y2mm, Y3mm, Y5mm, and Y10mm were 98.6%, 90.1%, 75.1%, 39.3%, and 6.1%, respectively. A similar decreasing pattern was observed across the rotational threshold spectrum, with event rates of 16.6%, 5.4%, and 1.1% for R2°, R3°, and R5°, respectively. The composite endpoint Yaction occurred in 40.9% of records (Fig. 2).

Fig. 2.

Threshold spectrum of translational and rotational setup error endpoints. Note: Event prevalence declined monotonically as the threshold increased for both translational and rotational setup error endpoints. Translational thresholds were defined at 1, 2, 3, 5, and 10 mm using tvec, whereas rotational thresholds were defined at 2°, 3°, and 5° using rmax. Percentages shown next to each point indicate the observed prevalence in the record-level main analytic cohort

Marked between-center differences were observed for all key endpoints. Compared with Center 1, Center 2 showed higher event rates for Y5mm, Y10mm, R3°, and Yaction, with absolute risk differences of 28.1%, 12.8%, 9.6%, and 30.2%, respectively. All of these differences were statistically significant (all P < 0.001; Table 2). Taken together, thoracoabdominal CBCT-derived 6DoF setup errors showed a characteristic long-tailed distribution, along with clear cross-center domain shift.

Table 2.

Prevalence of key endpoints and between-center differences

Endpoint	Overall, n (%)	Center 1, n (%)	Center 2, n (%)	RD (95% CI)	P value
Y5mm	2,368 (39.3)	1,108 (29.0)	1,260 (57.1)	28.1 (25.0–30.6)	< 0.001
Y10mm	370 (6.1)	56 (1.5)	314 (14.2)	12.8 (11.1–14.3)	< 0.001
R3°	328 (5.4)	73 (1.9)	255 (11.6)	9.6 (8.3–11.0)	< 0.001
Yaction	2,469 (40.9)	1,143 (29.9)	1,326 (60.1)	30.2 (27.6–32.7)	< 0.001

Data are presented as n (%). Percentages were calculated within each center. RD indicates the absolute risk difference (Center 2 minus Center 1), with 95% confidence intervals estimated from contingency table comparisons

Internal validation and probability calibration

Using pooled out-of-fold predictions from three-fold stratified cross-validation, CatBoost and XGBoost showed very similar performance on the primary task, Y5mm, with AUPRCs of 0.620 and 0.622 and corresponding AUROCs of 0.718 and 0.721, respectively (Fig. 3A; Table 3). For the more severe translational endpoint Y10mm, both models achieved an AUPRC of 0.234 (Fig. 3B; Table 3). Performance dropped more noticeably for the low-incidence rotational endpoint R3°. By contrast, results for the composite endpoint Yaction were relatively stable, suggesting that models based on pre-treatment workflow variables retained some discriminative value for large translational events and workflow-triggering endpoints (Table 3).

Fig. 3.

Internal validation and calibration of key translational endpoints. Note: (A) Precision-recall curves for Y5mm based on pooled out-of-fold (OOF) predictions from three-fold stratified cross-validation. B Precision-recall curves for Y10mm based on pooled OOF predictions. C Calibration of the CatBoost model for Y5mm, comparing raw probabilities with cross-fitted isotonic calibration; the background histogram shows the distribution of predicted probabilities. D Threshold sweep for Y10mm, showing sensitivity and positive predictive value (PPV) across candidate decision thresholds. Dashed horizontal lines in panels A and B indicate baseline prevalence. Shaded bands in panels A and B represent bootstrap 95% confidence intervals. Threshold markers in panel D are shown for illustration only and do not represent recommended clinical thresholds

Table 3.

Internal validation and calibration metrics for key tasks

Task	Baseline prevalence	Model	AUPRC	AUROC	Brier score	ECE
Y5mm	0.393	CatBoost	0.620	0.718	0.208	0.080
		XGBoost	0.622	0.721	0.207	0.089
Y10mm	0.061	CatBoost	0.234	0.838	0.155	0.253
		XGBoost	0.234	0.841	0.152	0.252
R3°	0.054	CatBoost	0.133	–	0.183	0.288
		XGBoost	0.135	–	0.202	0.317
Yaction	0.409	CatBoost	0.651	0.734	0.205	0.077
		XGBoost	0.653	0.737	0.203	0.074

Performance was evaluated using pooled out-of-fold (OOF) predictions from three-fold stratified cross-validation. Baseline prevalence refers to the event rate in the record-level main analytic cohort. Lower Brier scores and lower expected calibration error (ECE) indicate better calibration

AUPRC Area under the precision-recall curve, AUROC Area under the receiver operating characteristic curve

For the primary endpoint Y5mm, reliability plots showed better agreement between predicted probabilities and observed event rates after post hoc isotonic regression than with the raw model outputs (Fig. 3C). Calibration metrics from internal validation likewise indicated that probability estimates were relatively well calibrated for Y5mm and Yaction, whereas Y10mm and R3° showed clearly higher ECE values, pointing to less stable probability estimates for low-incidence endpoints (Table 3). Changes in sensitivity and positive predictive value across different operating thresholds further illustrated the practical trade-offs involved in using model outputs to support selectively intensified verification workflows (Fig. 3D).

A comparison of internal and external calibration for the primary and composite endpoints showed that probability estimates degraded more readily in the external setting than in internal validation (Table 4). In particular, in the non-center 2 → center 2 direction, both the Brier score and ECE increased noticeably for Y5mm and Yaction, suggesting that under cross-center transfer, the interpretability of predicted probabilities may be affected earlier than discrimination by differences in baseline risk and workflow heterogeneity.

Table 4.

Calibration metrics for the primary and composite endpoints under internal and external validation

Setting	Task	Brier score	ECE
Internal OOF	Y5mm	0.207	0.089
non-center 1 → center 1	Y5mm	0.213	0.065
non-center 2 → center 2	Y5mm	0.324	0.323
Internal OOF	Yaction	0.203	0.074
non-center 1 → center 1	Yaction	0.216	0.081
non-center 2 → center 2	Yaction	0.323	0.319

Lower Brier scores and lower expected calibration error (ECE) indicate better calibration. Internal results were based on pooled out-of-fold (OOF) predictions from three-fold stratified cross-validation, whereas external results were obtained from bidirectional cross-center validation

Cross-center external validation and sensitivity analysis of the center label

In bidirectional cross-center external validation, model performance generally declined relative to internal validation (Fig. 4; Table 5). Because AUPRC is strongly influenced by endpoint prevalence in the validation set, absolute AUPRC values should not be compared across transfer directions without accounting for differences in prevalence. We therefore report not only absolute AUPRC, but also validation-set prevalence and ΔAUPRC, in order to reflect the additional precision-recall gain over a random baseline.

Fig. 4.

Bidirectional cross-center external validation performance for key prediction tasks. A External validation performance for the rotational endpoint (R3°). B External validation performance for the translational endpoint (Y5mm). C External validation performance for the composite endpoint (Yaction). External validation was performed bidirectionally, with each center serving in turn as the held-out test set. The figure shows absolute AUPRC values under the with-center setting. Error bars indicate 95% confidence intervals. Because precision–recall performance depends on validation-set prevalence, prevalence-adjusted interpretation is provided in Table 5. AUPRC, area under the precision–recall curve

Table 5.

Bidirectional cross-center external validation for the primary and composite tasks with prevalence-adjusted precision-recall metrics

Transfer direction	Task	Prevalence	AUPRC	ΔAUPRC
non-center 1 → center 1	Y5mm	0.290	0.320	0.030
non-center 2 → center 2	Y5mm	0.571	0.499	-0.072
non-center 1 → center 1	Yaction	0.299	0.319	0.020
non-center 2 → center 2	Yaction	0.601	0.525	-0.076

Prevalence refers to the event rate in the external validation set. ΔAUPRC, defined as AUPRC minus prevalence, represents the gain over the random precision-recall baseline for that validation set. Center 1 denotes Ganzhou Cancer Hospital, and Center 2 denotes Sun Yat-sen University Cancer Center. Additional results for other tasks and models are provided in the Supplementary Materials

For the primary endpoint Y5mm, the models still retained some improvement over baseline in the non-center 1 → center 1 direction. In the non-center 2 → center 2 direction, by contrast, absolute AUPRC remained relatively high, but ΔAUPRC was negative because prevalence in the validation set was substantially higher. A similar pattern was observed for the composite endpoint Yaction. In contrast, external validation performance for the key rotational endpoint R3° was generally weak in both directions. Taken together, these findings suggest that asymmetry in cross-center external validation cannot be judged from absolute AUPRC alone, but needs to be interpreted alongside validation-set prevalence and ΔAUPRC.

In external testing, performance differences between the with-center and without-center input settings were small overall. This suggests that the main constraint on cross-center generalization was not the center label itself, but more likely deeper domain shift arising from differences in case mix, immobilization patterns, and local workflow characteristics.

Subgroup analyses

After stratification by anatomic site, model performance for the primary endpoint Y5mm differed clearly between subgroups. Using XGBoost as an example, the abdominal subgroup showed a higher AUPRC than the thoracic subgroup, and the same pattern was observed with CatBoost. A similar advantage for the abdominal subgroup was also seen for the composite endpoint Yaction (Fig. 5).

Fig. 5.

Subgroup performance of the Y5mm model by site and immobilization. Note: AUPRC values with 95% confidence intervals are shown for subgroups defined by site (panel A) and immobilization method (panel B), based on pooled out-of-fold predictions from the XGBoost model. N denotes the number of records and E the number of positive events in each subgroup. HNS, head-neck-shoulder; thermo pad, thermoplastic pad; cerv-thor, cervicothoracic; AUPRC, area under the precision-recall curve

When stratified by immobilization method, subgroup separability appeared even more heterogeneous. The Orfit frame subgroup showed the highest AUPRC, followed by the vacuum bag subgroup, whereas subgroups such as HNS mask + vacuum pad, HNS mask + thermo pad, and cerv-thor mask + thermo pad performed less well and had wider confidence intervals. Taken together, these findings suggest that anatomic site and immobilization method were associated not only with baseline event risk, but also with the degree to which the underlying risk structure could be learned by the models. Among the currently available workflow variables, immobilization method appeared to be one of the strongest observable discriminative signals (Fig. 5).

Directional decomposition analysis

To further explore the geometric pattern underlying immobilization-related risk differences, translational errors were decomposed by direction. Differences between immobilization methods were not distributed evenly across the LR, SI, and AP axes, but were concentrated mainly in the AP direction (Fig. 6; Table 6).

Fig. 6.

Directional decomposition of translational setup errors by immobilization method. Note: (A) Axis-specific P95 absolute translational error in the LR, SI, and AP directions. B Axis-specific tail-event rate for absolute translational error > 5 mm. Compared with the vacuum bag, the Orfit frame showed the most pronounced excess tail risk in the AP direction, with more moderate differences in LR and little difference in SI

Table 6.

Direction-specific translational error metrics for the Orfit frame and vacuum bag

Axis	Orfit P50 (mm)	Vacuum P50 (mm)	Orfit P95 (mm)	Vacuum P95 (mm)	Orfit > 5 mm (%)	Vacuum > 5 mm (%)
LR	2.5	2.0	7.70	5.00	20.1	4.2
SI	1.5	2.0	5.58	6.00	7.4	8.6
AP	5.9	2.0	14.58	5.10	58.4	5.1

LR Left-right, SI Superior-inferior, AP Anterior-posterior, P50 median absolute error, P95 95th percentile of absolute error

Compared with the vacuum bag, the Orfit frame showed a more pronounced rightward shift and broader tail in the AP direction, with a median absolute error of 5.9 mm versus 2.0 mm, a 95th percentile of 14.58 mm versus 5.10 mm, and a markedly higher event rate for |AP| > 5 mm (58.4% vs. 5.1%). By contrast, differences in the LR direction were only moderate, and those in the SI direction were relatively small. Taken together, these findings suggest that the association between immobilization method and the risk of large translational events was driven mainly by enrichment of tail events in the AP direction, rather than by a parallel increase across all three axes.

Sensitivity analysis based on grouping keys

Because the current dataset did not provide a stable identifier for uniquely tracking individual patients, strictly patient-independent data splitting was not feasible. As an alternative, we performed a more conservative stratified sensitivity analysis using grouping keys derived from combinations of center, anatomic site, tumor type, and immobilization method. Compared with the record-level pooled out-of-fold results, model performance for the primary endpoint Y5mm declined noticeably under this setting, and a similar pattern was observed for Y10mm (Fig. 7; Table 7).

Fig. 7.

Grouping-key-based sensitivity analysis for key translational endpoints. Note: Precision-recall curves for Y5mm (panel A) and Y10mm (panel B) are shown from grouping-key-based grouped cross-validation. Dashed horizontal lines indicate baseline prevalence, and shaded bands represent group-bootstrap 95% confidence intervals

Table 7.

Record-level versus grouping-key-based sensitivity analysis for key translational endpoints

Task	Model	Record-levelAUPRC(95%CI)	Grouping-key-basedAUPRC(95%CI)
Y5mm	CatBoost	0.620 (0.599–0.640)	0.387 (0.304–0.490)
	XGBoost	0.622 (0.601–0.642)	0.442 (0.328–0.590)
Y10mm	CatBoost	0.234 (0.204–0.265)	0.095 (0.016–0.219)
	XGBoost	0.234 (0.204–0.265)	0.088 (0.018–0.194)

Record-level results were based on pooled out-of-fold predictions from three-fold stratified cross-validation. Results for the more conservative analysis were obtained from grouped cross-validation using derived grouping keys

CI Confidence interval, AUPRC Area under the precision-recall curve

These findings suggest that when records are randomly split at the record level, performance estimates may benefit in part from repeated structural combinations appearing in both the training and validation sets, which can make model performance look somewhat more optimistic than it would be under a more conservative evaluation scheme. In this context, the record-level results are better suited to describing workflow event-level risk distributions and potential verification burden, whereas the grouping-key sensitivity analysis provides a more conservative bound on robustness.

SHAP-based interpretability analysis

In the CatBoost model for the primary endpoint Y5mm, global SHAP importance ranked immobilization method highest, with a mean absolute SHAP value greater than those of center, tumor type, and anatomic site (Fig. 8A). This suggests that, among the workflow variables considered in the present study, immobilization method was one of the strongest observable signals for distinguishing the risk of large translational events.

Fig. 8.

SHAP-based interpretability analysis for the Y5mm CatBoost model. Note: (A) Global feature importance ranked by mean absolute SHAP value, showing that immobilization method contributed the largest share of model importance. B Category-level SHAP dependence for the top-ranked feature (immobilization method), showing the direction and magnitude of SHAP contributions across immobilization categories. Positive SHAP values indicate a higher predicted risk of Y5mm, whereas negative SHAP values indicate a lower predicted risk. SHAP, Shapley additive explanations; HNS, head-and-neck–shoulder; thermo, thermoplastic

Further category-level dependence analysis showed clear differences in the direction of SHAP contributions across immobilization categories (Fig. 8B). Taken together with the subgroup and directional decomposition analyses, these findings suggest that immobilization method functioned not only as an important predictive feature, but also as a composite marker that may reflect center composition, disease distribution, and local workflow differences.

Discussion

Using CBCT-derived six-degree-of-freedom (6DoF) setup error records from routine thoracoabdominal radiotherapy workflows at two centers, we developed a threshold-spectrum-based framework for risk stratification and evaluated the predictive value of pre-treatment workflow variables for key setup endpoints. Four main findings emerged. First, thoracoabdominal setup errors showed a clear long-tailed distribution: low-threshold events were common, whereas higher-threshold events became progressively less frequent. Second, event rates for key endpoints differed substantially between the two centers, pointing to evident cross-center domain shift. Third, using only pre-treatment workflow variables—namely center, anatomic site, tumor type, and immobilization method—the models achieved moderate and relatively stable discrimination for Y5mm and Yaction, but performed poorly for the low-incidence rotational endpoint R3°. Fourth, immobilization method appeared to be one of the strongest observable discriminative signals, and its associated risk differences were expressed primarily as tail broadening in the AP direction. Taken together, these findings suggest that although workflow-level interpretable models cannot replace image guidance itself, they may still provide a quantitative basis for identifying high-risk scenarios and selectively intensifying verification. In this sense, our approach is methodologically complementary to prior work that has focused more heavily on image-level misalignment detection and automated audit tasks [8].

One of the more distinctive design features of this study was the use of a threshold spectrum rather than a single endpoint. The workflow implications of thoracoabdominal setup errors are not uniform: smaller deviations are common and may often be partially absorbed by existing online correction strategies, whereas larger translational or rotational events are more likely to trigger repositioning, rescanning, or escalation of the verification process. On that basis, we treated thresholds such as 5 mm, 10 mm, and 3° as workflow-level nodes for risk stratification rather than as proposals for new clinical margin recommendations. The 5-mm translational threshold was selected because it represents a practical magnitude often used for clinically meaningful setup deviation and workflow attention in image-guided radiotherapy workflows. This is consistent with the broader radiotherapy principle that the significance of geometric deviation must be interpreted in relation to target coverage and intended clinical use [9]. The 10-mm threshold was included to capture rarer but more severe translational tail events. For rotational errors, the 3° threshold was chosen as a pragmatic large-rotation cutoff, because rotational deviations of this magnitude may have relevant geometric consequences, particularly for off-isocenter anatomy or elongated thoracoabdominal targets [10, 11]. The use of threshold spectra (1, 2, 3, 5, and 10 mm for translation; 2°, 3°, and 5° for rotation) was intended to characterize the gradient from frequent small deviations to less frequent larger deviations rather than relying on a single binary cutoff. In addition, the practical impact of rotational error is not determined by angle alone, but also depends on off-axis distance and local dose gradient; the same angular deviation may therefore have different consequences across thoracoabdominal settings [10, 11]. In our data, event rates for both translational and rotational endpoints decreased monotonically as the threshold increased, supporting the presence of a clear risk gradient in thoracoabdominal CBCT-derived 6DoF setup errors. This also provides a basis for center-specific threshold recalibration in line with local workflows and verification strategies.

Bidirectional cross-center external validation was another central finding of this study. Compared with internal validation, model performance declined overall in the external setting, and the degree of decline was clearly asymmetric across transfer directions. This pattern was broadly consistent with the between-center differences observed in baseline event rates, case mix, and immobilization distributions, suggesting that the main barrier to generalization was not the algorithm itself, but case-mix shift and workflow heterogeneity [12, 13]. The comparison between with-center and without-center settings further showed that external performance differed little overall between the two, indicating that the center label was neither the sole source of model performance nor the main explanation for limited generalization. A more plausible interpretation is that “center” functions only as a proxy for multiple institutional differences; when case mix, immobilization patterns, and local workflows differ systematically, model transportability may remain constrained even without explicitly including center as a predictor [14]. It is also important to note that the 6DoF values analyzed here were derived from couch correction quantities displayed and stored by the system after online grayscale registration of CBCT and before couch shift, and the study database consisted of direct exports of these historical system records. Accordingly, between-center differences may reflect not only case-mix composition and local workflow differences, but also some inconsistency in system recording conventions and IGRT practice. For this reason, we interpret cross-center differences as composite workflow differences rather than as a simple comparison of geometric error levels alone.

The reliability of between-center comparisons should also be interpreted in the context of anatomical-site composition. Thoracoabdominal CBCT-derived setup corrections are not expected to be uniform across anatomical regions. Abdominal positioning may be more susceptible to respiratory motion, soft-tissue deformation, and day-to-day variation in gastrointestinal filling, whereas thoracic positioning may be relatively more constrained by bony anatomy depending on immobilization and registration practice. Therefore, part of the observed between-center difference may reflect differences in site composition rather than center-related factors alone. In our cohort, thoracic and abdominal records were distributed differently between centers (Table 1). However, descriptive stratified analyses showed that center-related differences remained observable within both thoracic and abdominal subgroups, with larger separations in the abdominal subgroup. These findings suggest that the observed cross-center heterogeneity is likely attributable to a combination of case-mix differences and workflow-related factors rather than to anatomical-site composition alone.

Pathological subtypes also exhibited substantial heterogeneity across centers (Table 1). Several disease categories were represented predominantly in only one center, limiting the interpretability of disease-specific cross-center comparisons. Nevertheless, among the major disease categories represented in both centers, including cervical, lung, esophageal, and rectal cancers, between-center differences in event rates remained evident. Therefore, tumor type was retained as a workflow-level predictor in the primary models, while disease-specific findings were interpreted descriptively rather than as independent validation results.

These findings should therefore not be interpreted simply as model failure, but rather as evidence that workflow-level prediction models are sensitive to real-world domain shift during cross-center transfer. Several practical strategies may help mitigate this limitation in future deployment. First, model outputs should be assessed and, when necessary, recalibrated using local data from the target center before being used for threshold-based workflow decisions [12, 13]. Second, harmonization of workflow descriptors, immobilization categories, CBCT acquisition settings, registration practice, and record-export conventions may reduce avoidable sources of between-center heterogeneity [14]. Third, rather than assuming a single universal model across all thoracoabdominal settings, a more realistic implementation strategy may be to use the model as a local workflow-audit and risk-stratification tool, with site-specific or workflow-specific threshold adjustment. Finally, because setup practice may evolve with new equipment, updated protocols, or staff experience, periodic local validation would be needed to maintain model reliability over time [13, 14].

The interpretation of immobilization-related risk differences also requires the subgroup, directional decomposition, and SHAP results to be considered together. Subgroup analyses showed clear heterogeneity in model separability across immobilization methods, and directional decomposition further indicated that these differences were not distributed uniformly across all three axes, but were expressed mainly as tail broadening in the AP direction. More specifically, compared with the vacuum bag, the Orfit frame showed more pronounced high-quantile inflation and tail-event enrichment in the AP direction, whereas differences in the LR direction were more modest and no consistent pattern was observed in the SI direction. This suggests that the predictive value of immobilization method did not arise from a general amplification of translational error across all axes, but rather from its ability to capture risk for specific directional high-risk events, particularly large anterior-posterior displacements [15, 16]. At the same time, SHAP analysis ranked immobilization method highest among the workflow variables considered here. Even so, this “importance” is better understood as a composite workflow-level proxy than as evidence of a single directly modifiable mechanism, because it may simultaneously reflect differences in body-position control, degree of trunk constraint, respiratory management intensity, and local registration practice [17, 18].

By contrast, prediction of the low-incidence large rotational endpoint R3° was clearly weaker than prediction of Y5mm. This was not unexpected. First, the event rate for R3° was low, making stable discrimination inherently more difficult under class imbalance. Second, all predictors in this study were static workflow variables available before treatment, and did not include more direct dynamic or imaging-based descriptors of rotational risk. The information available to the model was therefore limited from the outset. Third, rotational errors in thoracoabdominal cases are more likely to be influenced by dynamic factors such as momentary posture, breathing pattern, and trunk torsion, which are difficult to capture using static pre-treatment workflow information alone [19, 20]. The present findings should therefore be interpreted to mean that pre-treatment workflow variables may support limited risk stratification for large translational events, but are not yet sufficient for reliable prediction of low-incidence large rotational events.

The intended application level of the current prediction task also needs to be made explicit. Because all predictors were static variables available before treatment, the model is conceptually closer to one-time pre-treatment risk stratification than to a real-time decision model informed by intra-fraction or inter-fraction updating. We adopted the record as the primary unit of analysis because the endpoint was defined at the level of a single setup event crossing a predefined threshold, and one aim of the study was to characterize the overall burden of setup events with particular pre-treatment features and their potential relevance to selectively intensified verification. Within this framework, record-level results are better understood as estimates of workflow event-level risk distributions rather than as evidence of strictly patient-independent generalizability. The grouping-key sensitivity analysis supports this interpretation. Compared with the main analysis, AUPRC for Y5mm and Y10mm declined substantially under the more conservative grouped setting, suggesting that purely record-level random splitting may partly benefit from repeated structural combinations appearing in both training and validation sets, thereby producing somewhat optimistic performance estimates. In this sense, record-level results are more suitable for describing workflow event-level risk burden and potential verification demand, whereas the grouping-key sensitivity analysis provides a more conservative robustness bound. Importantly, this analysis should not be interpreted as evidence of patient-independent generalization.

Another implication of this study is that probability calibration may degrade more readily than discrimination during cross-center transfer. Although internal validation suggested relatively good calibration for Y5mm and Yaction, the Brier score and ECE worsened substantially in external validation, particularly in the non-center 2 → center 2 direction. This suggests that differences in baseline risk and workflow heterogeneity across centers may not only affect ranking performance, but may also erode the interpretability of predicted probabilities even earlier when those probabilities are used to support threshold-based workflow decisions. For applications that rely on fixed probability cutoffs to trigger workflow escalation or intensified verification, a more reasonable strategy would be to recalibrate the model locally at the target center rather than directly transferring the raw predicted probabilities. This view is also consistent with broader work on external validation of clinical prediction models and radiotherapy outcome modeling, both of which have repeatedly emphasized that practical deployment requires attention not only to discrimination, but also to calibration and uncertainty assessment [21, 22].

In this context, precise local calibration may be more appropriate than pursuing a single universal cross-center predictive model. The present results suggest that a model trained in one workflow environment should not be transferred directly to another center as a fixed probability-based decision system. Instead, a more reproducible implementation pathway would be to use the externally transferred model as an initial risk-ranking tool, followed by calibration of predicted probabilities and decision thresholds using a small representative local dataset from the target center. Such local calibration would allow the model output to better reflect center-specific baseline risk, case mix, immobilization practice, and registration workflow, thereby improving the interpretability of probability-based workflow decisions [13, 21].

Staff experience and temporal workflow evolution may also influence setup-error patterns. In routine radiotherapy practice, setup consistency can be affected by therapist experience, onboarding of new staff, changes in immobilization practice, updates to CBCT acquisition or registration protocols, and increasing familiarity with a given workflow. Because the present retrospective setup-record database did not contain staff-level identifiers, onboarding information, or explicit timestamps for workflow changes, we could not perform a reliable longitudinal analysis of setup-error trends over time. Therefore, the current findings should be interpreted as reflecting the aggregate workflow conditions during the respective data collection periods. For future implementation, model performance should be re-evaluated and, when necessary, recalibrated after major workflow changes, equipment updates, or staff onboarding [13, 14].

This study has several strengths. First, the data were drawn from two real-world thoracoabdominal radiotherapy workflows spanning multiple disease types, sites, and immobilization methods, which enhances clinical relevance. Second, we did not limit the analysis to endpoint spectra, internal validation, and bidirectional cross-center external validation, but complemented these with calibration analysis, subgroup analysis, directional decomposition, grouping-key sensitivity analysis, and SHAP-based interpretation, thereby forming a relatively coherent chain of evidence. Third, all model inputs were routinely available pre-treatment workflow variables, making the framework closer to a potentially deployable clinical scenario. At the same time, several limitations should be acknowledged. First, the main analysis was record-based rather than patient-based, and the current dataset did not include a stable identifier for uniquely tracking patients; accordingly, true patient-independent modeling and validation were not possible. In addition, the exported database did not contain complete treatment-fraction logs, which limited our ability to distinguish initial setup records from any additional verification records acquired within the same fraction. The database also did not include a record-level indicator of whether manual adjustment had been performed during online registration. The existing grouping-key sensitivity analysis can only provide a more conservative robustness bound and cannot substitute for patient-level validation. Second, the predictors were limited to categorical workflow variables and did not include dynamic or imaging-derived information more directly related to motion mechanisms, which likely contributed to the weak performance for low-incidence large rotational events. Third, although bidirectional cross-center validation was performed, the number of centers remained limited, and transportability across a broader institutional range still requires further study. Finally, the thresholds and their potential workflow applications were derived from retrospective data and should not be interpreted as clinical threshold recommendations. Future work should therefore include validation in additional centers and incorporation of deployable prior information that more directly reflects dynamic setup mechanisms, with the aim of improving both robustness and clinical usefulness.

Conclusion

Thoracoabdominal CBCT-derived six-degree-of-freedom setup errors showed a clear long-tailed distribution and substantial cross-center heterogeneity. Interpretable models built from pre-treatment workflow variables provided modest risk stratification for large translational setup events and may offer quantitative support for identifying high-risk scenarios, selectively intensifying verification, and auditing setup workflows. However, the current variable set carried limited information for predicting low-incidence large rotational events, and probability calibration during cross-center transfer appeared to degrade more readily than discrimination. Further validation in additional centers, together with local recalibration and more deployable prior information that better reflects dynamic setup mechanisms, will be needed to improve model generalizability and practical clinical value.

Abbreviations

AUPRC

Area under the precision-recall curve

AUROC

Area under the receiver operating characteristic curve

CBCT

Cone-beam computed tomography

CI

Confidence interval

ECE

Expected calibration error

IGRT

Image-guided radiotherapy

OOF

Out-of-fold

PPV

Positive predictive value

SHAP

Shapley additive explanations

6DoF

Six degrees of freedom

Authors’ contributions

W.L., D.X., and Y.F. contributed equally to this work and share first authorship.L.X., C.X., and J.F. share corresponding authorship.W.L., D.X., and Y.F. conceived and designed the study.W.L. curated the data, conducted the main analyses, and drafted the manuscript.D.X. and Y.F. contributed to data analysis, model development, and manuscript revision. L.W., S.L., and X.L. assisted with methodology development, data collection, and validation.C.X. and J.F. contributed to study supervision and data interpretation.L.X. led the overall project, supervised the research, guided the study design, and critically revised the manuscript.All authors reviewed and approved the final manuscript.

Funding

This research received no external funding.

Data availability

The datasets generated and/or analyzed during the current study are not publicly available because they were derived from routine clinical CBCT-guided radiotherapy workflows and are subject to institutional and ethical restrictions. Although the data were de-identified, they contain detailed setup correction records that may still carry a residual risk of re-identification. Data may be made available from the corresponding author upon reasonable request, subject to approval by the relevant institutional ethics committees and in accordance with applicable data protection regulations. The analytical methods are described in sufficient detail in the Methods section to support reproducibility.

Declarations

Ethics approval and consent to participate

This study was approved by the Medical Ethics Committee of Ganzhou Cancer Hospital (Ganzhou, China; Approval No. (2026) KELUNSHEN No. (6)) and was reviewed and approved by the Ethics Committee of Sun Yat-sen University Cancer Center (Approval Letter No. B2026-167-01; Document No. SL-B2026-167-01). The study was conducted in accordance with the Declaration of Helsinki, the International Council for Harmonisation Good Clinical Practice (ICH-GCP) principles, and relevant national regulations. Given the retrospective nature of the study and the use of de-identified clinical data, the requirement for informed consent was waived by the ethics committees.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.