Automatically tracking brain metastases after stereotactic radiosurgery

Abstract

Background and purpose

Patients with brain metastases (BMs) are surviving longer and returning for multiple courses of stereotactic radiosurgery. BMs are monitored after radiation with follow-up magnetic resonance (MR) imaging every 2–3 months. This study investigated whether it is possible to automatically track BMs on longitudinal imaging and quantify the tumor response after radiotherapy.

Methods

The METRO process (MEtastasis Tracking with Repeated Observations was developed to automatically process patient data and track BMs. A longitudinal intrapatient registration method for T1 MR post-Gd was conceived and validated on 20 patients. Detections and volumetric measurements of BMs were obtained from a deep learning model. BM tracking was validated on 32 separate patients by comparing results with manual measurements of BM response and radiologists’ assessments of new BMs. Linear regression and residual analysis were used to assess accuracy in determining tumor response and size change.

Results

A total of 123 irradiated BMs and 38 new BMs were successfully tracked. 66 irradiated BMs were visible on follow-up imaging 3–9 months after radiotherapy. Comparing their longest diameter changes measured manually vs. METRO, the Pearson correlation coefficient was 0.88 (p < 0.001); the mean residual error was −8 ± 17%. The mean registration error was 1.5 ± 0.2 mm.

Conclusions

Automatic, longitudinal tracking of BMs using deep learning methods is feasible. In particular, the software system METRO fulfills a need to automatically track and quantify volumetric changes of BMs prior to, and in response to, radiation therapy.

1. Introduction

Brain metastases (BMs) are the most common form of brain tumors. It is estimated that 10–40% of all cancer patients will develop brain metastases, with an estimated 70,000–400,000 cases/year in the United States [1], [2], [3], [4]. Historically, the standard of care for patients with multiple BMs was whole brain radiation therapy (WBRT), which controls the disease for some time, but carries the possibility of cognitive side effects and reduced quality of life [5], [6], [7]. In addition, delivering additional courses radiation may increase the risk of radionecrosis [8], [9], [10]. Recent advances in technology have allowed treatment of multiple BMs with frameless single-fraction or hypofractionated stereotactic radiosurgery (SRS/HYPO) [11]. Evidence of reduced cognitive decline has led to the primacy of SRS/HYPO for treating BMs [12]. Moreover, there is an increasing trend to manage multiple or recurrent BMs with multiple courses of SRS/HYPO [13], [14], [15], [16], [17], [18], [19], [20], [21].

Patients with BMs are typically monitored with magnetic resonance (MR) imaging performed every two–three months [22], including T1 post-Gd (T1C+). The complexity of longitudinally tracking each BM and determining treatment response increases with each course of radiation or follow-up MR. Key challenges include classifying BMs as progressing, stable, or recurring, separating new BMs from treated BMs, and differentiating recurrent metastases from delayed radiation necrosis or post-treatment effects.

Limited prior work exists on automatic BM tracking. Shearkhani et al. applied a Jacobian operator field to detect size changes in brain metastases on longitudinal MRs, with significant challenges from new and resolved BMs and false positive detections from blood vessels [23]. Chitphakdithai et al. performed deformable intrapatient registration to estimate BM size changes, but relied on manual segmentation of the initial tumor [24]. Some commercially available software platforms can track BMs with significant manual user input, but do not provide a feasible or practical solution for patients with multiple BMs and multiple prior treatments. Recently, Petersen et al. evaluated a PACS-integrated tracking tool for tracking BMs [25], and demonstrated time savings and reduced manual clicks for physicians. However, there are currently no available tools which integrate automatic image registration, detection, segmentation, and tracking, and can provide information about all treated and untreated lesions in a timely manner to the radiologist and radiation oncologist.

The aim of this study was to investigate whether it is possible to automatically track size and radiation dose for multiple BMs, which could help clinicians make the optimal care decisions. We validated and investigated the efficacy of the METRO software (MEtastasis Tracking with Repeated Observations), which detects and segments all BMs using deep learning, tracks the BMs across co-registered MR timepoints, and reports the radiation dose from prior treatments.

2. Materials and methods

2.1. Overview of METRO process

Fig. 1 shows an overview of METRO. The inputs are DICOM data (Digital Imaging and Communications in Medicine): longitudinal MR image series from the picture archiving and communication system (PACS) and DICOM-RT data from the treatment planning system (TPS). A recent MR series is chosen as the fixed image and rigid registration is performed with all the patient’s other MRs. After registration, artificial intelligence (AI) inference is performed at each MR series using a deep convolutional neural network to produce longitudinal maps of BM gross tumor volumes (GTVs) in the fixed MR frame of reference. Next, BM tracking is performed using the treatment plan structure sets and the AI segmentations. Depending on overlap with existing GTVs from treatment plans, the detected GTVs are associated with previously treated GTVs or classified as new AI GTV candidates. The size of each GTV is tracked over time. The calculated dose distributions from prior treatment plans are overlaid on the GTVs to track the physical dose over time. Finally, a report document is generated with separate pages for each tracked BM. Raw data is also saved for downstream analysis, including tracking information such as volume and 3D longest diameter.

Fig. 1.

Schematic showing an overview of the METRO software. Planning and follow-up MR scans are obtained from the PACS. Treatment plans with calculated dose and CT scans are obtained from the TPS.

2.2. Registration method

As months or years elapse, the patient’s brain may exhibit large changes over time, including tumor control and progression, edema, midline shift, and resection cavities. Furthermore, patients who survive longer have several follow-up imaging studies. METRO co-registers all longitudinal MR imaging using a six-degree-of-freedom (6DOF) rigid registration. A multiple-stage registration procedure is used, balancing speed and accuracy, while avoiding catastrophic failures requiring manual user intervention.

The choice of fixed image for the registration depends on the treatment plans. If no co-registered treatment plans are provided, the most recent T1C+ MR is the fixed image. Otherwise, it is the MR registered to the most recent treatment plan. The mutual information similarity metric was used as the cost function [26].

A brief description of the registration stages follows. The registration translation is initialized based on intensity moments, followed by a three-dimensional exhaustive search for initial rotation angle. Two successive stages align the entire head using gradient descent with 6DOF. Next, a fine 6DOF search is performed using only the brain parenchyma as the region of interest—this brain contour is obtained either from the structure sets of the co-registered treatment plan or an MR-based deep learning model [27]. Lastly, a fine-tuning 6DOF search is performed using a rectangular zone centered on the lateral ventricles. After registration, all images are resampled to the fixed image with isotropic 1 mm voxel spacing.

The registration method was validated by comparing with manually verified registrations performed by a medical physicist on 20 BM patients. Each patient had two T1C+ brain MR scans with a large time interval: median 392 days, interquartile range (IQR) 334–887. Seven anatomical points were chosen, and manually located on multi-planar views of the first scan. The XYZ position of each control point was passed through the spatial transformations from the manual and automatic registrations, and the registration error was quantified by differences in resulting position. Further details are given in Supplementary Materials.

2.3. AI segmentation

BM GTVs are automatically detected and segmented on co-registered T1C+ scans using a previously published AI model [27]. Using GTVs authored by the radiation oncologist as ground truths, this 3D V-Net convolutional neural network was trained (tested) on statistically independent samples of 409 (1 0 2) SRS/HYPO patients with 1345 (3 6 7) BMs. Dense evaluation of the neural network on each MR timepoint produces 3D BM probability maps in the fixed image frame of reference. To obtain discrete BM segmentations from the probability maps, a binary threshold is applied, followed by a connected-component analysis. The average patient sensitivity, false positive rate, and Dice coefficient were 95% ± 3%, 2.4 ± 0.5 per patient, and 0.76 ± 0.03, respectively (95% confidence).

A heuristic is applied to handle instances where two BMs are conjoined into one segmentation by a sandbar or isthmus of lower probability. A morphological opening operator is applied once to each initial segmentation blob of volume $V[\mathrm{m}\mathrm{L}]$. Assuming the blob contains two spherical lesions each of volume roughly $V/2$, an adaptive ball kernel radius of $0.4r_{\mathrm{s}\mathrm{p}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}}$ is chosen, where $r_{\mathrm{s}\mathrm{p}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}}[\mathrm{c}\mathrm{m}]={\left(\frac{3V}{4\pi }\right)}^{\frac{1}{3}}$. If the result has two separate parts, they are now each counted as separate BM detections; otherwise, the original component is kept.

2.4. Brain metastasis tracking

After registration and AI segmentation, METRO tracks BMs over time. First, consider physician-authored GTVs with radiation prescriptions from prior treatment plans. Each expert GTV from the structure sets is rasterized, then expanded by a default 1 mm margin for longitudinal tracking. At each timepoint, any segmentation components that overlap with this expanded GTV are found; their union constitutes the tracked lesion on that scan. This search is applied to all scan timepoints—BMs are tracked and measured even at MRs preceding treatment. Measurements of (non-expanded) GTV volume and 3D longest diameter (3LD) at each timepoint are computed and saved. Longitudinal size changes are measured using AI segmentations.

BMs which are newly appeared or not prescribed treatment are also tracked. To mitigate false positive detections, an increased detection probability threshold (80%) was required to identify such lesions. METRO checks each MR timepoint and finds new or untargeted AI GTVs that do not overlap with AI GTVs found at previous timepoints. Like the expert GTV tracking, these AI GTVs are tracked on past and future timepoints, using the expansion margin to check for overlap with AI segmentations. False negative, true positive, and false positive detections were identified based on clinical radiology reports. The radiologist’s axial 2D diameter measurements were recorded for false negatives. False positives were measured using 3LD.

Next, dose metrics are computed to each BM for patients with prior treatment data. This includes BMs not targeted for radiation by a given SRS/HYPO or WBRT plan, using the BM segmentations and the calculated dose images. Dose tracking is handled slightly differently for targeted and untargeted lesions. For linac-based SRS plans, a targeted lesion is defined as an expert GTV which has an associated planning target volume and a mean physical dose sum > 10 Gy. Otherwise, it is an untargeted BM which received spillage dose. Using the original GTV volume that initiated the tracking, physical dose metrics are recorded for each treatment, including fractionation, dose delivered to 99% volume (D99%), and mean dose.

2.5. Longitudinal patient dataset for clinical validation

32 patients with BMs were retrospectively identified under a patient consent waiver approved by our Institutional Review Board. They were treated in 2018–2021 with VMAT SRS/HYPO on linear accelerators using multiple arcs, couch rotations, and optical surface monitoring [11]. Patients were imaged with T1C+ MR prior to treatment and received regular follow-up imaging (183 total scans). METRO was used to longitudinally track each of the 187 lesions treated within this cohort. 53 lesions were excluded due to lack of follow-ups within a chosen time window of 90–270 days. Three resection cavities, one skull lesion, and seven missing PTVs were also excluded. For the remaining 123 lesions, the follow-up MR closest to 180 days post-treatment was chosen. Median time between treatment start date and the chosen follow-up was 185 days (IQR 149.5–196).

2.6. Brain metastasis response assessment

To evaluate the accuracy measuring the size changes of tracked BMs, the pre-treatment 3LD and volume for each lesion was compared to the size at follow-up. The percent changes of 3LD and equivalent sphere diameter (ESD) between pre-treatment and follow-up scans were calculated. To validate the tracking performance, a trained operator contoured the same 123 BMs on the pre-treatment and follow-up MR scans using MIM Maestro 6. Volume and longest diameter measurements were extracted from each manual contour. The ESD was derived from the manual and automatic volume measurements. 3LD and ESD were compared between METRO and manual measurements using linear regression and the Pearson correlation coefficient. For percent size changes, the residual error was defined as the software observation minus the human observation, with size changes determined either by 3LD or ESD. Distributions of the size change residual errors were analyzed for both size metrics.

3. Results

3.1. Registration accuracy

Evaluating the registrations qualitatively, differences between scan timepoints were handled well by the registration method, including ventricle size, edema, tumor progression, tumor response, and motion and susceptibility artifacts. No BM tracking failures were caused by registration uncertainty. The spatial shifts between manual and automatic registration at each anatomical control point were assessed to quantify the registration accuracy. The average shift per point in millimeters was: Point A, 1.8 ± 0.7; Point B, 1.2 ± 0.5; Point C, 1.3 ± 0.5; Point D, 1.6 ± 0.6; Point E, 1.7 ± 0.7; Point F: 1.2 ± 0.5; Point G: 1.3 ± 0.5 (95% confidence). The average shift across all points was 1.5 ± 0.2 mm (95% confidence) with median 1.1 mm (IQR 0.6–1.6). Registration examples for three patients with time intervals of about three years are shown in Figure S2.

3.2. Report document

Several example reports are shown for anonymized patients. In Fig. 4, a larger enhancing lesion in the right frontal lobe was treated with 27 Gy over three fractions, and the report correctly identifies that the lesion responded to treatment on follow-up images. Fig. 5 shows a BM in the right parietal lobe which was treated with 21 Gy (single-fraction) and is stable for some time, but exhibits an increase in size over one year later. Lastly, Fig. 6 shows a new BM identified by METRO in the right parietal lobe after four previous courses of radiation treatments to other BMs. See Figures S3 and S4 for additional examples.

Fig. 4.

This 2 cm enhancing lesion in the right frontal lobe was treated with 27 Gy in three fractions using the MR scan from 11/12/2017 for planning. The planning target volume (PTV) is shown in red, and the longitudinal AI segmentations in blue. The lesion size is decreasing at all follow-up MR images with the last MR from 8/15/2018. The METRO report shows both the 3D longest diameter and the volume based on the AI segmentation at each MR. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 5.

A lesion in the right parietal lobe is targeted with single-fraction SRS using the MR from 6/12/2018 for treatment planning. There is not much size change in the first nine months after treatment, but a year and a half after treatment, at the MR scan from 12/24/2019, the lesion size is increasing. The increase in size must be evaluated for potential progression or necrosis.

Fig. 6.

This patient received three courses of SRS elsewhere in the brain, followed by WBRT (Rx 30 Gy). Then, the METRO workflow identified a new lesion in the right parietal lobe at the 7/10/2019 MR, which is increasing in size at subsequent follow-up on 1/7/2020.

3.3. Detection

72% (38/54) of new or unirradiated BMs were detected in the longitudinal tracking dataset. 92 false positive new lesions were tracked (0.5/scan). The median sizes of true positive, false negative, and false positive detections of unirradiated BMs were respectively: 0.8 cm (IQR 0.6–1.2), 0.5 cm (IQR 0.3–0.7), and 0.7 cm (IQR 0.6–0.9). The median initial size of all tracked lesions was 0.9 cm (IQR 0.6–1.3). The isthmus rule was applied 64 times (0.3/scan) and affected measurements of 15 irradiated BMs. For example, two nearby BMs in the left cerebellum and temporal lobe were measured at 7.58 and 0.73 cm³ pre-treatment, after an isthmus was removed between their conjoined segmentations.

3.4. Treatment response

With complete BM disappearance as the quantity of interest, compared to the human observer, METRO produced 48 true positives (i.e. truly disappeared BMs), 66 true negatives, four false positives, and five false negatives. For the 66/123 BMs still visible on follow-up, linear regression (Fig. 2) shows a correlation (R² = 0.80) between the size responses measured by human versus METRO. Comparing the size changes of non-disappeared lesions measured by both observers, the Pearson correlation coefficient was 0.88 for 3LD and 0.86 for ESD (p < 0.001). Distributions of the size change residual errors are shown in Fig. 3. The mean residual of 3LD changes was −8 ± 17% (95% confidence), with standard deviation 70%, median 2%, IQR (-7%, 12%). The mean residual of ESD changes was −12% ± 17% (95% confidence) with standard deviation 81%, median 2%, and IQR (-7%, 11%). The residual distributions were characterized by a central mode plus large outliers.

Fig. 2.

Linear regression of size response for treated BMs that were still visible on follow-up, as measured automatically by METRO versus manually. One outlier with diameter change greater than + 200% is suppressed.

Fig. 3.

Distributions of residual errors for the size changes measured by automatic METRO method for the BMs still visible at follow-up. The residual is the difference of the percent change measured with METRO minus that measured manually. Left: 3D longest diameter. Right: equivalent sphere diameter.

4. Discussion

The METRO process was capable of reading the patient data and tracking BMs over time. The longitudinal intrapatient registration method was reliable when tested on numerous clinical cases, including those with significant time intervals and notable changes in brain anatomy, and its accuracy was sufficient for longitudinal tracking. The information gathered from each patient’s imaging and treatment history is compiled in a concise, lightweight report document accessible to clinicians, and data are organized for downstream analysis.

We observed a strong correlation between the BM size changes measured by METRO and the human operator. Furthermore, positive versus negative size changes were well-differentiated by METRO—this can be observed by the lack of points in the upper-left and lower-right quadrants of Fig. 2. The distribution of residuals for size changes measured by human versus METRO (Fig. 3) were well-centered at zero, but highly non-normal with large variance due to outliers. We observed a strong correlation coefficient of 88% for the measured change in longest diameter, but there is room for improvement. One potential avenue is a longitudinal segmentation model trained using multiple annotated longitudinal scans. We noticed several cases with non-spherical BMs where the 3LD remains stable, but the BM volume is increasing. This indicates that automatic measurements of BM volume could be more clinically relevant to tumor burden than the manually-measured 2D longest diameter frequently employed in radiology practice.

The accuracy for tracking new sub-centimeter BMs was limited; several false positives and false negatives were observed. The sample of unirradiated BMs was also biased towards smaller size, since larger, growing BMs are more likely to be targeted with radiation. The driving factor is the imperfect detection accuracy of the AI model used to analyze individual MR scans. Unfortunately, this problem is omnipresent in the BM AI literature [27], [28], [29], [30], [31], [32], [33], [34]. There is ongoing work studying how to mitigate false positives [35], [36].

Varying imaging protocols and quality standards can hamper the generalizability of AI-based work such as this across different centers. Consensus guidelines are emerging for BM studies [37], [38]. MR imaging parameters such as pixel spacing and slice thickness can be automatically detected, but other quality aspects require standards developed by experts.

It is currently difficult to differentiate post-treatment tumor volumes from radionecrosis treatment effects. Despite the current ambiguity, post-treatment tracking of abnormal volumes associated with tumors could be an important tool to help distinguish recurrences from radionecrotic volumes.

Manually identifying multiple, potentially irradiated BMs on longitudinal imaging is time-consuming and prone to human error [39], [40], [41], [42]. Measuring or segmenting primary and metastatic brain tumors is similarly challenging [31], [43]. With the aim of aiding clinical practice, we are currently working to integrate this tracking workflow with our existing clinical systems. To address potential inaccuracies, a radiologist or trained operator could view and modify the results, while referencing other synchronized image series from the MR studies. The user would delete false positives and adjust inaccurate longitudinal segmentations, then an approved report would be generated. The approved structure sets would be available if further SRS treatment is indicated, and confirmed new BM appearances would be monitored automatically. These data would be invaluable not only for longitudinal retrospective studies, but also as a continuous source of BM annotations for developing better longitudinal AI models. While manual corrections do cost time, there is great potential for time savings and clinical insights as the tracking accuracy is further improved.

In conclusion, we show that automatic longitudinal tracking of brain metastases using deep learning methods is feasible. In particular, the software system METRO fulfills a need to automatically track and quantify volumetric changes of brain metastases prior to, and in response to, radiation therapy. The accuracy achieved in detecting and tracking tumor volumes, excepting tumors smaller than 1 cm, appears to be adequate to support clinical workflows. Further development is needed to improve the detection of small BMs and the response measurement accuracy. Future work may explore how to distinguish post-treatment tumor volumes from radionecrosis treatment effects. We invite interested researchers to discuss potential multicenter collaborations. In the future, the addition of interactivity for clinicians can make this a routine part of the patient’s medical record.

Funding: MSKCC Support Grant/Core Grant P30 CA008748.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: