AI-based framework to fuse pre-RT brain metastases contours with follow-up MRI to improve post-RT assessment

Abstract

Background

Brain metastases (BMs) patients often receive multiple courses of radiotherapy (RT). Comparing follow-up imaging with RT plans is time-intensive and currently not possible in picture archiving and communication system (PACS). In this study, we aim to develop an automated tool to overlay pre-RT tumor contours onto follow-up MRI in the PACS to support response assessment and identification of untreated metastases, reducing review complexity and time.

Methods

We built an AI-driven workflow that registers planning CT to follow-up MRI and propagates treated-lesion contours; outputs are exported as PACS-compliant DICOM. Performance was evaluated in 40 patients: 20 underwent quantitative comparison between manual and automated registrations using Dice similarity coefficient (DSC) and mean surface distance (MSD), following AAPM TG-132 guidelines; 20 additional patients (5-35 lesions per patient) underwent clinical evaluation of follow-up and pre-RT lesions contours visualization in PACS. Three physicians assessed these cases to measure review time and inter-observer agreement in treatment response classification (improved/stable vs indeterminate).

Results

The workflow successfully registered all CT-MRI pairs. Mean DSC/MSD was: brain 0.97 ± 0.01/0.0 ± 0.0 mm, brainstem 0.89 ± 0.03/0.1 ± 0.1 mm, and gross tumor volumes 0.65 ± 0.18/0.6 ± 0.4 mm. Average physician review time per case decreased from 7.97 to 3.95 min with the automated workflow, and full inter-physician agreement increased from 72.4% to 93.5%.

Conclusion

We developed and validated an AI-based tool that accurately fuses pre-RT contours with follow-up MRI for BMs, addressing a key gap in current PACS systems. The workflow significantly reduced review time per lesion and enhanced inter-physician agreement, and has the potential to enable faster, more consistent multidisciplinary follow-up assessment.

Key Points.

• An AI-based framework was developed to fuse pre-RT brain metastases contours with follow-up MRI, enabling integration into PACS.

• The tool reduced physicians’ average review time per follow-up case by 50.43% and increased inter-physician agreement on treatment response from 72% with conventional review to 93% with the AI-assisted pipeline.

• The framework addresses a critical gap by making radiotherapy planning data accessible to the broader multidisciplinary team without requiring access to the RT planning system.

Importance of the Study

Follow-up assessment of brain metastases after radiotherapy (RT) becomes increasingly complex as both the number of metastases and the number of RT courses per patient rise. Conventional workflows lack access to RT planning information in PACS, requiring physicians to spend extra time in assessing treated lesions and distinguishing them from new or progressing disease. This study demonstrates a PACS-integrated, AI-assisted workflow that brings RT context (ie, prior RT contours and target geometry) directly into routine neuro-oncology imaging. Rather than introducing another planning-system tool, our approach operates inside the enterprise imaging environment: it automatically registers and fuses pre-RT structures to follow-up MRI and renders them in the diagnostic viewer, eliminating multi-platform exports, duplicate contouring, and license/log-in bottlenecks. In doing so, it enables faster case review and more consistent, RT-aware interpretations across the multidisciplinary team, complementing retreatment utilities in planning systems by serving the earlier surveillance and decision-making steps where most follow-up imaging is read.

Recent reports highlight an increased rate of new cancer cases observed in the USA.^1,2 Brain metastases (BMs), the most common variant of malignant brain tumors, develop in 10-40% of cancer patients in the USA.^3–5 Available modalities for treatment of BMs include surgery, radiosurgery, conventional radiotherapy (RT), and systemic therapies.⁶ Patients receiving RT to treat BMs can often go through multiple courses of treatment. After each RT course, BMs’ progression or tumor response is evaluated on follow-up diagnostic imaging every 2-3 months.^7–9 These follow‑up scans may reveal any newly developed BMs and help to clarify whether further evaluation therapy is needed. Importantly, radiation treatment-related records (images, dose maps, tumor contours, etc) where physicians may also need to register prior treatment plans on follow-up scans to accelerate their review process. Notably, this treatment-related information is not typically stored within institutional picture archiving and communication systems (PACSs). These records are available in a separate clinical database (ie, oncology information systems (OISs)) within radiation oncology departments; therefore, individual treatment information is not easily accessible to multidisciplinary clinical teams (medical oncology, surgery, radiology, neurologists, neurosurgeons), who routinely review follow-up imaging. This separation can hinder efficient clinical interpretation and collaborative decision-making.

To our knowledge, there is limited literature related to this problem. Previously, Savjani et al.developed a tool to display isodose lines within the PACS⁶; though it relied on manual image registrations, between follow-up images and RT simulation images, a time-intensive and expertise-dependent process that limits routine clinical use. Similarly, Hutten et al.incorporated biologically effective dose (BED) maps for lung stereotactic body radiation therapy (SBRT) patients in PACS, demonstrating improved recognition of radiation-induced lung injury.¹⁰ In their workflow, the BED maps were first fused with the corresponding planning CT and uploaded to the PACS, where the multidisciplinary clinical team could visualize the fused images alongside diagnostic or follow-up CT scans. However, while this allowed for side-by-side review of pretreatment dose distributions and diagnostic findings, the treatment-dose information was not directly overlaid on the diagnostic images themselves—an integration that becomes particularly important for patients with an increasing number of lesions, where direct visualization of dose-to-anatomy relationships is essential for accurately correlating radiation effects with anatomical changes and assessing treatment response. Consequently, there remains a need for an automated, AI-driven framework that can seamlessly fuze pretreatment RT data (such as tumor contours) with follow-up diagnostic imaging (integrated directly into PACS) to enable efficient, multidisciplinary review and improve confidence in post-RT response assessment. Moving beyond these approaches, we developed an AI‑driven framework that automatically registers and propagates planning tumor contours to follow‑up MRI and integrates them within the PACS making these visualizations readily accessible to multidisciplinary clinical teams and increasing confidence in subsequent treatment decisions through enhanced post-RT evaluation. In this retrospective study, we applied the developed framework to 40 patients with BMs. Planning CT and MR were automatically segmented to delineate key brain structures, guiding CT-MR registration and subsequent contour propagation. The geometric accuracy of automated registration was compared with clinician-defined alignments using Dice similarity coefficient (DSC) and mean surface distance (MSD) metrics. For qualitative evaluation, clinical GTV contours were overlaid on follow‑up MRIs and reviewed within research PACS, where physicians assessed clinical utility and reported a marked reduction in patient evaluation time along with improved inter-physician agreement on tumor response.

Methods

Patient Data

We retrospectively identified stereotactic radiosurgery (SRS) cases, including both upfront and postoperative treatments from our institution’s clinical database under Institutional Review Board approval (IRB #120703005). Eligible cases had to meet al. of the following criteria: (i) adult patients (≥18 years) treated for BMs between January 2018 and December 2023; (ii) no prior cranial (ie, whole-brain) conventional RT; (iii) complete planning dataset available (simulation CT, RTSTRUCT, RTPLAN, and RTDOSE files); and (iv) at least 1 diagnostic follow‑up MRI acquired within 90 days after treatment. From 67 eligible cases, 40 patients meeting these criteria were selected for analysis via simple random sampling without replacement using a computer-generated sequence using a fixed seed for reproducibility. For each selected case, we retrieved the full set of clinically approved planning records and the corresponding follow‑up MR scans from our institutional archive. The planning CT for all included SRS cases was a contrast-enhanced simulation CT, consistent with our institutional standard for BMs unless contraindicated.

Software Development

We first curated all imaging data (CTs, MRIs) and resampled them to isotropic 3D volumes. Figure 1 shows the analysis pipeline which takes 3 inputs: (1) the CT-simulation scan associated with the desired treatment plan, (2) the list of GTVs defined in the same RT plan, and (3) the follow-up MRI. The output is a DICOM MR image that contains the overlay of the rigidly propagated pre-RT tumor contours on the follow-up MRI.

Figure 1.

Proposed approach used for improved follow-up assessment of BM patients.

We leverage AI-based auto-contouring to guide the CT-to-MRI registration process. First, the brainstem is segmented in each modality: on CT with a model trained using nnU-Net framework and on MRI with the open-source SynthSeg framework.^11,12 Next, we compute the center of mass (CoM) of each brainstem. Assuming the brainstem CoM remains stable throughout treatment, we translate the CT volume so that its CoM coincides with that of the MRI. This translation provides a robust, anatomy-anchored initialization that compensates for gross inter-scan positioning offsets and narrows the search space for the subsequent rigid optimization. A subsequent rotation about this shared CoM, optimized with a multi-resolution Euler-3D rigid transform, refines the alignment and preserves fine anatomical detail. The registration process begins with an initial 3D Euler transform, which sets the center of rotation/translation to the brainstem’s CoM. Mutual information is then used as the similarity metric to capture the statistical relationship between CT and MRI intensities, which is essential for multimodal registration. A multi-resolution registration pyramid is applied using shrinking factors, where the images are progressively downsampled: first by a factor of 4 (coarse), then 2 (medium), and finally at full resolution (fine). This strategy allows the optimizer to perform rapid global alignment on coarse images, reducing the risk of getting trapped in local minima, before refining the transformation at higher resolutions. Smoothing factors introduce Gaussian blurring at each level of the pyramid: heavy smoothing at the coarse level, less at the medium level, and none at the finest level. This step suppresses high-frequency noise, enabling the optimizer to prioritize global structural alignment in the early stages and then focus on fine anatomical details in the final stage. The final rigid-registration matrix is applied to resample each treatment-plan GTV into the follow-up MRI space, after which the fused series is exported as axial DICOM images.

End-to-End Testing

An end-to-end test was designed to quantitatively and qualitatively review (via an observational study) the utility of the developed software tool. To quantify registration accuracy, we compared 20 patients’ automatically propagated structures with a manually-registered reference set created by an experienced clinician at the time of planning. Registration performance was quantified using the DSC and MSD (mm) in line with the American Association of Physicists in Medicine Task Group 132 guidelines.¹³ The DSC ranges from 0 to 1, where 1 indicates perfect spatial overlap between automated and manual contours and 0 indicates no overlap. The MSD measures the average distance between corresponding surface points of 2 contours, with lower values indicating better geometric agreement. For anatomical structures such as the brain and brainstem, DSC values ≥0.8 and MSD values around 2-3 mm are generally considered acceptable for clinical registration accuracy.¹³ Specifically, the brain, brainstem, and treatment-plan GTVs were rigidly mapped to the follow-up MRI with our automated pipeline and evaluated against the corresponding manually-registered structures.

A blinded observational study was designed to qualitatively assess the clinical utility of the propagated GTV contours on follow-up imaging within institutional PACS. Twenty BMs patients, each containing 5-35 BMs, were stratified into 2 size-balanced observational study cohorts (Groups A and B) (Figure 2). These 20 cases were entirely distinct from those used in the quantitative analysis. All datasets were fully de-identified, and the acquisition dates were normalized to a single fictitious timestamp so that every series within a case (and across cases) displayed identical scan dates. For every case, the PACS provided 2 series to the reviewing physicians: (i) the pre-RT MRI and (ii) the post-RT follow-up MRI. In the enhanced cohort (Group B) a third series was additionally available: the follow-up MRI with automatically propagated GTVs over-laid. In both cohorts, the simulation CT, corresponding structures/contours, to the treatment plan, and dose distribution were also available for reference within the treatment-planning system (TPS) to simulate a typical post-RT assessment workflow. It is important to note that radiology reports (both for pre- and post-RT imaging) were excluded from this analysis. Three board-certified radiation oncologists (ROs), blinded to cohort assignment, independently evaluated every BM treatment response in both workflows. For each case ROs recorded review time and classified each lesion’s post-RT response using response assessment in neuro-oncology (RANO) Criteria 2.0 Criteria.¹⁴ The mean review time per metastasis was the primary endpoint. Differences in review time between Group A and Group B were assessed using the Mann-Whitney U test, with a P-value <.05 considered statistically significant. Secondary endpoints were inter-observer agreement on lesion response and themes derived from physician feedback, which would guide refinements needed for routine clinical deployment. To assess inter-rater reliability among the participants, we evaluated observers’ lesion-level response classifications (eg, improved/stable, indeterminate) for the 2 observational study cohorts. The rationale for grouping improved and stable lesions was that both represent clinically favorable outcomes that do not necessitate additional testing or treatment. We computed Fleiss’ kappa coefficient to quantify overall agreement among raters within each group. In addition to kappa, we calculated the proportion of lesions for which all 3 physicians gave identical responses (“full agreement”) and cases where 2 out of 3 agreed while the third differed (“2/3 agreement”). This 2-level agreement breakdown enabled us to assess not just consistency, but also the extent of unanimity in clinical interpretation between the conventional and AI-assisted workflows.

Figure 2.

Samples distribution into 2 groups for getting review’s comments.

Results

Automated propagation of CT-based contours was successful on 100% of cases. This process took on average 9.04 ± 1.29 min per patient. Figure 3 shows DSC and MSD comparisons between contours propagated using automatic and manual registrations. Average DSC (±standard deviation) values were 0.97 ± 0.01, 0.89 ± 0.03, and 0.65 ± 0.18 for brain, brainstem, and GTVs, respectively. Average MSD values were 0.0 ± 0.0, 0.1 ± 0.1, and 0.6 ± 0.4 mm for the same structures, respectively. Individual GTV volumes ranged from 0.010 to 16.206 cc, reflecting the lesion-level variability across patients.

Figure 3.

Box and whisker plot of, (A) Dice score between automatically transformed brain structure by our tool and ground truth (manual) by clinical experts, (B) mean surface distance between automatically transformed (automatic) brain structures by our tool and ground truth (manual) by clinical experts. Central line represents the median, border of the box represents Q1 (25%) and Q2 (75%) and markers represents outliers.

Observational Study

Three ROs reviewed both observational study cohorts (Groups A and B) and the time required to assess each GTV was recorded. Figure 4A demonstrates that the enhanced workflow (ie, displaying automatically propagated contours on the follow-up MRI) consistently shortened review time relative to the conventional workflow (MRI alone): on average, RO-1 saved 38.7 s per lesion (75%, P = .045), RO-2 saved 9.9 s (27%, P = .045), and RO-3 saved 19.8 s (59%, P = .0001). Overall, average review time per follow-up case dropped from 7.97 to 3.95 min using conventional versus automated workflow. To assess inter-rater reliability in post-RT lesion responses, we computed Fleiss’ kappa and agreement distributions (Figure 4B and C). Group A (no contours on follow-up imaging) showed moderate agreement with a Fleiss’ kappa of 0.36, while Group B (AI-driven contours’ propagation) demonstrated slightly lower kappa (0.33). However, the rate of full agreement in lesion classification was substantially higher in Group B (93.5%) compared to Group A (72.5%), with a corresponding decrease in partial agreement (2 out 3 observers’ agreement: 6.5% [Group B] vs. 27.5% [Group A]). This suggests that while the overall kappa value remains similar, presentation of treated targets overlaid on follow-up imaging leads to more consistent ratings amongst observers.

Figure 4.

(A) Time spent by physicians to review treatment response in Group A vs Group B. Each physician’s total review time per follow-up MRI was normalized by the number of visible brain metastases, yielding the mean review time per lesion. Statistical comparison between the groups was performed using Mann-Whitney U-test. Asterisks indicate the level of statistical significance where *, **, and *** mean P < .05, P < .01, and P < .001, respectively. (B) Inter-physician agreement for treatment response assessment (ie, improved/stable vs indeterminate) per lesion in Group A (with no contours on follow-up imaging in PACS. (C) Inter-physician agreement for treatment response assessment (ie, improved/stable vs indeterminate) per lesion in Group B (with AI-driven contours on follow-up imaging in PACS).

Discussion

In this study, we developed an automated tool that overlays RT target contours onto follow-up imaging within the PACS (an enterprise, vendor-agnostic environment) to streamline post-RT assessment during patient visits and avoid planning-system silos and access-restricted, department-specific databases/commercial solutions. Developed specifically for SRS workflows, the proposed tool underwent separate quantitative and qualitative evaluations. Quantitative results showed that the automated workflow generated propagated lesion contours comparable to manual propagation, with mean surface-distance differences being less than 1 mm for most target lesions. Observational study showed that overlaying targets on post-RT MR images directly in the PACS significantly reduced review time (50%, P < .001) per follow-up case compared to manual process, achieving the study’s primary endpoint. By bringing RT context to the point of first read across the enterprise, the approach also enhanced inter-observer agreement in case assessments.

The enhanced workflow translated into meaningful efficiency gains and steadier clinical decisions. In a busy SRS clinic that reviews ≈15 lesions per patient, this corresponds to roughly 6 min reclaimed per follow-up visit, freeing physician time for counseling or additional cases. Just as important, the proportion of full 3-way agreement on lesion response increased from 72.5 % with the conventional MRI-only approach to 93.5 % when contours were displayed. Although Fleiss’ κ remained comparable, the marked increase in unanimous ratings implies fewer equivocal findings and, potentially, less need for adjudication, thereby streamlining downstream management decisions such as the timing of salvage therapy or interval imaging. An additional advantage is that, by integrating treatment context directly into the PACS, the contours become accessible to the broader neuro-oncology team without requiring access to the radiation oncology TPS. This not only facilitates multidisciplinary review but may yield even greater time savings for non-ROs. Together, these gains suggest that integrating treatment information within the PACS can accelerate routine follow-up and enhance the consistency of clinical judgments.

To date, only 2 groups have explored how RT data can be displayed within the clinical PACS.^6,10 Savjani et al.overlaid planning-CT isodose lines onto follow-up MRI, but their workflow depended on a manual rigid registration between the planning CT and each subsequent scan.⁶ Hutten et al.extended the concept for lung SBRT by importing BED maps into the PACS; clinicians could visualize BED on the pretreatment CT and correlate it with subsequent MRI changes, yet the BED maps themselves were not shown directly on the follow-up MRI.¹⁰ Both solutions therefore require either manual registration steps or accept that dose information is confined to the planning CT. In contrast, our approach provides an AI-driven, automated pipeline that registers planning CT to the follow-up scan, propagates target contours onto the new imaging, and exports the result as DICOM objects that are immediately viewable in the PACS (see Figure 5). While some commercial solutions exist to fuze and review tumor changes across serial imaging, the resulting information often remains confined to department-specific applications with restricted access. In contrast, our approach renders pre-RT contour overlays natively in PACS, enabling enterprise-wide visibility without department-specific planning software logins or exports.

Figure 5.

Treated GTVs on follow-up MRI are available in PACS, enabling clinicians in assessing treatment response effectively.

It is important to note that anatomical changes remain a recognized limitation for any registration method. Like Savjani et al. we observed that substantial post-treatment changes, as well as variations in MRI acquisition parameters, can lead to misalignment. During tool development, these challenges highlighted the need for a robust QA/QC framework to ensure reliable CT-MR registration. In particular, we found that using lower-resolution 2D series instead of 3D acquisitions matched to the planning CT increased the risk of misregistration (Figure 6). To overcome these issues, we implemented automated pre-transfer checks to confirm image dimensionality, slice spacing, and transformation residuals, along with a registration-confidence score to flag cases for manual review. Routine spot audits and version-controlled logging were also incorporated to ensure that potential errors are intercepted before derived objects enter the PACS. Together, these measures mitigate known limitations of registration, safeguard data integrity, and strengthen clinician trust in the automated workflow. Implementation is underway using a Plan-Do-Study-Act (PDSA) framework.¹⁵ In the current phase, the radiation oncology team triggers the application and selects the relevant DICOM studies (planning CT/RTSTRUCT and desired post-RT MRI sequence); the pipeline runs automated QA checks, performs CT-MR registration, and returns PACS-ready overlays for review and publish. This implementation strategy has been designed to build trust in the tool as well as enable a controlled and safe deployment before broader release. Broadening access beyond radiation oncology (eg, MR technologists) is feasible but non-trivial, as many RT treatment records (plans, structures, approvals) are not universally visible in enterprise systems; resolving cross-department data availability and permissions is the principal hurdle. In an ideal next state, we envision the process could become event-driven (ie, automatically launching after an imaging study completes) and, following a brief human verification step, pushing the RT-aware overlays to PACS for institution-wide use; however, we believe that earning trust through verification in a controlled environment is a logical first step.

Figure 6.

3D MR with 1.0 mm slice spacing improves alignment with planning CT than 2D axial MR series with 3.0 mm slice thickness.

While our AI-based framework demonstrates significant potential in improving post-RT assessment of BMs, several limitations must be acknowledged. First, the current workflow relies on rigid CT-to-MRI registration which assumes stability of the brainstem center-of-mass for alignment. In practice, inferior deformation of the brainstem due to neck positioning, postoperative changes, or edema can occur, leading to potential misregistration that rigid transforms cannot fully compensate. Incorporating deformable registration in future could improve robustness in such cases. Second, this was a single-institution study, which may limit generalizability. Future multi-institutional validation will be essential to confirm performance across varied imaging systems and clinical environments. Third, by focusing solely on BMs, the framework’s applicability to other disease sites or metastatic patterns remains to be demonstrated. Fourth, the dataset included a limited number of very small lesions (<5 mm); which often appeared indistinct on lower-resolution MRI, challenging both automated propagation and physician review. To mitigate this, we implemented matched resampling to a common isotropic voxel size, to enhance small-lesion conspicuity. Nonetheless, the algorithm’s performance and reviewer confidence for these borderline-sized lesions warrant validation in larger and more diverse cohorts. Finally, MRI acquisition parameters varied among patients (particularly in sequence type, slice thickness, and in-plane resolution) introducing protocol heterogeneity that may affect registration accuracy and contour visibility. Standardizing MRI protocols or developing sequence-adaptive registration strategies could further enhance reliability. Clinical deployment is underway, and future developments will include deformable registration, multi-course treatment tracking, isodose line propagation, and validation across institutions and anatomical sites to broaden clinical utility.

Conclusion

Post-RT assessment of BMs is often hindered by the lack of integrated RT plan data within PACS. This study presents an AI-based automated framework that registers and overlays pre-RT tumor contours onto follow-up MRI, enabling accurate identification of treated lesions and improved differentiation from new or progressive disease. The workflow significantly reduced review time per lesion and enhanced inter-physician agreement, and has the potential to enable faster, more consistent multidisciplinary follow-up assessment.

Author Contributions

Conceptualization: Y.K., C.E.C; Data Curation: Y.K., K.A.H.; Formal Analysis: Y.K., C.E.C., H.C., G.N.R., R.N., F.R.L., R.C.; Funding Acquisition: C.E.C.; Investigation: C.D.W., D.H.B., J.B.F., R.A.P., C.E.C.; Methodology: Y.K., C.E.C; Project Administration: C.E.C; Resources: Core Facility for AMIPA, UAB Radiation Oncology Department; Software/AI Framework Development: YK, C.E.C.; Supervision: CEC; Validation: C.D.W., D.H.B., J.B.F; Visualization: Y.K., C.E.C, J.B.F, K.L.S.; Writing—Original Draft: Y.K, C.E.C.; Writing—Review & Editing: All authors.

Conflict of Interest Statement The authors declare that they have no conflicts of interest related to the subject matter or materials discussed in this article.

Funding

Y.K. received partial funding from the O’Neal Comprehensive Cancer Center NextGen Scholar Award, and additional support was provided by the Department of Radiation Oncology at the University of Alabama at Birmingham (PI: C.E.C.).

Ethics

This study was approved by Institutional Review Board of the Department of radiation Oncology, University of Alabama at Birmingham, USA (IRB#120703005).

Data Availability

The imaging datasets used in this study were anonymized but contain patient-derived data and are therefore not publicly available due to institutional and ethical restrictions. De-identified data may be made available from the corresponding author on reasonable request and subject to approval by the institutional review board.

Prior Presentations

Preliminary results of this study have been partially presented as poster presentation at the American Association of Physicists in Medicine (AAPM) annual meeting, July 21-25, 2024, AAPM annual meeting, July 27-30, 2025, and the American Society for Radiation Oncology, September 27-Oct 01, 2025.