Development and prospective in-patient proof-of-concept validation of a surface photogrammetry + CT-based volumetric motion model for lung radiotherapy

Abstract

Purpose:

We develop and validate a motion model that uses real-time surface photogrammetry acquired concurrently with four-dimensional computed tomography (4DCT) to estimate respiration-induced changes within the entire irradiated volume, over arbitrarily many respiratory cycles.

Methods:

A research, couch-mounted, VisionRT (VRT) system was used to acquire optical surface data (15 Hz, ROI = 15 × 20 cm²) from the thoraco-abdominal surface of a consented lung SBRT patient, concurrently with their standard-of-care 4DCT. The end-exhalation phase from the 4DCT was regarded as reference and for each remaining phase, deformation vector fields (DVFs) with respect to the reference phase were computed. To reduce dimensionality, the first two principal components (PCs) of the matrix of nine DVFs were calculated. In parallel, ten phase-averaged VRT surfaces were created. Surface DVFs and corresponding PCs were computed. A principal least squares regression was used to relate the PCs of surface DVF to those of volume DVFs, establishing a relationship between time-varying surface and the underlying time-varying volume. Proof-of-concept validation was performed during each treatment fraction by concurrently acquiring 30 s time series of real-time surface data and “ground truth” kV fluoroscopic data (FL). A ray-tracing algorithm was used to create a digitally reconstructed fluorograph (DRF), and motion trajectories of high-contrast, soft-tissue, anatomical features in the DRF were compared with those from kV FL.

Results:

For five of the six fluoroscopic acquisition sessions, the model out-performed 4DCT in predicting contour Dice coefficient with respect to fluoroscopy-derived contours. Similarly, the model exhibited a marked improvement over 4DCT for patch positions on the diaphragm. Model patch position errors varied from 5 to −15 mm while 4DCT errors ranged between 5 and −22.4 mm. For one fluoroscopic acquisition, a marked change in the a priori internal–external correlation resulted in model errors comparable to those of 4DCT.

Conclusion:

We described the development and a proof-of-concept validation for a volumetric motion model that uses surface photogrammetry to correlate the time-varying thoraco-abdominal surface to the time-varying internal thoraco-abdominal volume. These early results indicate that the proposed approach can result in a marked improvement over 4DCT. While limited by the duration of the fluoroscopic acquisitions as well as the resolution of the acquired images, the DRF-based proof-of-concept technique developed here is model-agnostic, and therefore, has the potential to be used as an in-patient validation tool for other volumetric motion models.

1. INTRODUCTION

Intrafraction motion and deformation of the internal anatomy induced by respiration represent an outstanding problem in thoracic and abdominal cancer radiotherapy (RT).¹ If not adequately accounted for, these movements can cause an under-dosage of the tumor or undesired radiation dose and, therefore toxicity to healthy tissue.² While there are a variety of clinical and investigational motion management techniques described in the literature,^3–8 a common limitation to current clinically deployed techniques is that they rely on a single-point representation of the tumor target for real-time position monitoring.

As we transition toward more potent forms of lung RT [stereotactic body radiotherapy (SBRT), particle and proton therapy], it is no longer sufficient to monitor the tumor target alone and more accurate and effective motion management becomes significantly important. Effective motion management requires us to capture the complex spatiotemporal changes in the entire irradiated patient volume, for example, translation/rotation/deformation, baseline shifts, and interplay of tumor and organs at risk (OARs), at an adequately high spatial and temporal resolution, and over arbitrarily many respiratory cycles. These changes need to be captured pretreatment for planning, as well as during treatment throughout the delivery of each RT fraction.

The current standard-of-care for capturing motion-related changes for treatment planning is to acquire a respiratory-correlated four-dimensional computed tomography (4DCT) scan, where CT projections are acquired over several respiratory cycles and then sorted into 6–10 volumes over a single average cycle.^9–11 The major limitation of this approach is that it does not capture cycle-to-cycle variations such as baseline drifts or changes in amplitude and frequency, and the effect these phenomena have on the motion and deformation of the tumor and surrounding critical organs.^12,13 For example, Cai et al. found that, as a result of cycle-to-cycle variations, the internal target volume (ITV) derived from 4DCT was underestimated by >20%.^14,15 Similar limitations are encountered with real-time position monitoring during RT delivery. Most clinical solutions monitor the tumor target using external or internal surrogates that yield three-dimensional (3D) translation of (and sometimes rotation around) a single point, that is, the tumor centroid. Thus, complex real-life effects such as rotation and deformation of the tumor target and its interplay with surrounding organs are not captured or accounted for.

In this context, volumetric surrogate-based motion models (SMMs) offer a possible solution by relating the internal motion to a measurable surrogate.¹⁶ In such methods, a correspondence model is built from observation of the relationship between measured internal anatomic motion and a simultaneously measured surrogate data. In the forthcoming treatments, the surrogate data are measured and the correspondence model is used to relate it to the internal motion and estimate the position of organ(s) of interest at any given point in time. The accuracy of the estimated motion depends on the (a) choice of surrogate, (b) truthfulness of the captured anatomic motion (used for training), (c) correspondence model that determines relationship of internal motion and measured surrogate, and (d) the patient- and fraction-specific complexity and variability of respiratory motion.

A variety of volumetric models have been investigated. A comprehensive review of various motion models may be found in McClelland et al.,¹⁷ and are briefly summarized here. Models introduced by Blackall et al and McClelland et al are the first ones to permit modeling of an entire volume of interest by employing affine and deformable registration, respectively.^18,19 Blackall et al. developed a model by adopting the assumption of repeatable breath cycles. With the motion of diaphragm derived from fast MRI methods as surrogate, a model was proposed that distinguished between inhalation and exhalation and therefore was able to capture intracycle variations.¹⁸ They tried to estimate the level of errors that may be induced into RT planning by intra- and intercycle variability of respiratory motion and the choice of imaging strategy used to obtain planning information form ITV. They used this model to study how the motion varied due to intra- and intercycle variations and in breath-hold vs free breathing. McClelland et al proposed a model for image-guided RT and 4DCT image acquisition. They used AP displacement of the chest, extracted from 4DCT, as surrogate and respiratory phases served to capture intercycle variations. Zhang et al. proposed a motion model that used diaphragm motion as surrogate to extract a low-quality respiration correlated cone beam CT (RC-CBCT) set from the acquired CBCT projections and used this set to remove respiration-induced artifacts in CBCT images.²⁰

Fayad et al. proposed extracting information from imaging data or by measuring the motion of the surface instead of a single point on it.²¹ However, the information extracted from imaging data usually suffers from limited amount of data and basically carries all the disadvantages of the images that were not good enough to represent the motion by themselves (e.g., lack of contrast). Additionally, because of difficulty measuring the surface at the same time as the data used to image the internal motion, investigators have used the imaging data to export the surface of their interest.^21,22 McClelland et al. actually measured the surface motion but they extracted a scalar surrogate from it rather than using the surface itself.²³

An important aspect of motion models is for them to allow capturing the intercycle variations of patient’s breathing pattern. To do so, the surrogate and the motion data need to be acquired for more than a single breathing cycle. Low et al. proposed a five-dimensional lung motion model that uses the two biomechanical parameters of air filling, and the imbalances of dynamical stress distributions to simulate lung tissue displacement.^24,25 As surrogate, they use the tidal volume measured by spirometry and its time derivative. The model returns estimation of complex hysteresis and irregular breathing patterns and is the first one to have the potential of capturing both inter- and intracycle variations. They used this model to quantify and analyze lung motion pattern at several points of interest and they were able to verify model estimations using the continuity equation.²⁵ Zhao et al.²⁶ examined the behavior of these two model parameters for a large population of patients (49 patients). The large variation of the biomechanical parameter across the patient population highlights the necessity for patient-specific motion models.

A mathematical model that uses principal component analysis of deformation vector fields (DVFs) computed from 4DCT images was first proposed by Zhang et al. to represent volumetric deformations of the lung.²⁰ This model uses the SI displacement of the diaphragm and its precursor (i.e., a time-lagged value of the signal) acquired from 4DCT and FL images to construct and parameterize the model. Subsequently, Li et al. use tumor displacement measured from implanted fiducials in FL images, as a surrogate to parameterize Zhang’s mathematical model.²⁷ Their model uses the maximum likelihood hypothesis to evaluate model coefficients and generate DVFs that map the fiducial positions matching the surrogate measurement.

In this work, we use a similar formalism as that proposed by Zhang to develop a lung motion model that monitors the real-time variations in the thoraco-abdominal surface to generate estimates of the motion and deformation of the underlying anatomical volume.^28,29 In contrast to the Zhang model, which uses a one-dimensional (1D) diaphragm motion and its precursor as surrogate, we use whole surface motion and deformation (15 × 20 cm² area by VRT camera), acquired concurrently with but independent from 4DCT. We use principal component analysis (PCA) to represent DVFs and a learning model based on partial least squares regression (PLSR) to link the surfaces observed at the time of the 4DCT acquisition with the observed DVFs. In this manner, our model is able to capture the internal volumetric anatomical motion (including the OARs). In addition, we develop a generalized validation method for volumetric motion models such as ours based on comparing the positional accuracy of anatomic features as observed from model-generated digitally reconstructed fluoroscopy (DRF) with those from concurrently acquired in-room fluoroscopy scans. Finally, we demonstrate the proof-of-concept of constructing and validating the model in a dataset acquired from a human lung cancer patient.

2. MATERIALS AND METHODS

This section is divided into three parts. First, we describe the theoretical framework for the motion model and develop a model-agnostic proof-of-concept validation technique based on digitally reconstructed DRFs. Second, we show proof-of-concept of the motion model using surface photogrammetry and 4DCT data acquired from a lung cancer patient. Finally, based on the proof-of-concept methodology developed earlier, we quantitatively characterize the accuracy of our patient-specific motion model.

2.A. Theoretical framework for the model

2.A.1. Principal component analysis of 4DCT deformation vector fields

Our motion model is based on the correspondence between the surface photogrammetry and internal anatomical deformations at the time of 4DCT simulation. Figure 1 illustrates the key steps in the creation of the motion model.

Fig. 1.

Flowchart of training and construction of the model.

The end-exhalation phase of the 4DCT (P50%) is assigned as the nominal reference phase and, using a free-form B-Spline deformable image registration (DIR) Nifty-Reg,³⁰ the DVFs between each remaining respiratory phase and the reference phase (total of 9 DVFs) are computed. Each DVF is stored as a row in a motion matrix V. Instead of a simple one-to-one correspondence model (e.g., McClelland et al. 2006),¹⁹ we use an approach based on PC decomposition. Given the large dimensionality of the DVFs (order of 1E7 variables), the traditional approach of performing PCA using the eigenvalue decomposition of the covariance matrix can become numerically unstable.^31,32 To circumvent this potential problem, PC decomposition is performed using singular value decomposition (SVD) and the jth row ${\overrightarrow{v}}_j$ (corresponding to jth respiratory phase) of the motion matrix V is represented as a weighted sum of the mean deformation, ${\overrightarrow{v}}_{\mathit{\text{mean}}}$, and the first few principal components such that:^31,32

$$\overrightarrow{v}={\overrightarrow{v}}_{\mathit{\text{mean}}}+\sum _{i=1,\dots ,N}{\alpha }_i^j{\overrightarrow{U}}_i\text{ for }j=1,\dots ,9$$ (1)

where ${\overrightarrow{v}}_{\mathit{\text{mean}}}$ is the mean deformation defined as the row average of the motion matrix V, and ${\overrightarrow{U}}_i$ is the unit vector corresponding to the ith principal component and N is the number of components used. In this work, we use the first three components to approximate the DVFs. Using the orthonormality condition of the PCs, the scalar coefficients ${\alpha }_i^j$ are computed using the vector dot product as projections of the residual deformation ${\overrightarrow{v}}_j-{\overrightarrow{v}}_{\mathit{\text{mean}}}$ along each PC vector ${\overrightarrow{U}}_i$ such that:

$${\alpha }_i^j=\left({\overrightarrow{v}}_j-{\overrightarrow{v}}_{\mathit{\text{mean}}}\right)\cdot {\overrightarrow{U}}_i$$ (2)

Equations (1) and (2) are used to estimate the rows of the motion matrix V from coefficients ${\alpha }_i^j$ and PCs (${\overrightarrow{U}}_i$).

2.A.2. Formation of phase-averaged surface displacement vector field

Surfaces acquired via photogrammetry using the VisionRT (VRT) system are used as independent predictor variables to drive the motion model. Due to the nature of the photogram-metric acquisition, the captured point clouds can be prone to missing patches and noise. To address this problem, we reconstruct these data as watertight surfaces. In this work, we have implemented the Poisson surface reconstruction technique developed by Kazhdan et al.^33–35 The technique is briefly described here for convenience. For each point-cloud, the algorithm uses the captured points (vertices), and their orientations (gradient) derived from vertex connectivity to generate a watertight surface. Using the Euler-Lagrange formalism, the surface that best approximates the captured points, with an orientation that matches the measurement, is found to be the solution to the Poisson Equation. The Galerkin discretization formulation³⁵ results in a system of equations which are solved using a tri-variate B-Spline basis function. With our current implementation, a watertight surface originating from a point-cloud with approximately 15 000 points can be obtained within 20 s.

Despite the use of surface reconstruction, integration of regions with large missing patches in the point-cloud results in undesired noise at the edges of the surface. To circumvent this problem, a 15 × 20 cm² region of interest (ROI) covering the thoraco-abdominal surface of the patient is selected and used as input into the model.

We used the first PC coefficient of the surfaces to generate ten phase-averaged surfaces. A dataset is compiled from the nine phase-averaged surface displacement vector field (SDVF), calculated as the algebraic difference between the phase-averaged surfaces corresponding to P50% and each of the remaining nine respiration phases. In this study, the dimension of the surface ROI reconstructed on a 0.67 × 0.68 cm² grid amounted to a few hundreds of predictor variables.

2.A.3. Generation of the training set and learning model

After computation of the prerequisites for the training set consisting of nine volumetric DVF-PC coefficients ${\alpha }_i^j$ via Eq. (2) and the corresponding nine SDVFs, the relationship between the two sets is estimated. We use the partial least square regression (PLSR) technique to relate the predictor variables (i.e., SDVFs) with the observed DVF-PC coefficients. PLSR is a multivariate regression technique used in a variety of fields such as medicinal chemistry, process control, and tumor classification.^36–39 It is designed for situations where a large number of possibly correlated predictor variables along with relatively few samples of their observed relation with the dependent variable and are used to estimate the dependent variable. Using PLSR we construct a regression that relates the predictor variables (time-varying surface) with the dependent variable (time-varying internal volume, ${\alpha }_i^j$). To this end, the observed sets of predictor and dependent variables are standardized to have zero mean and unit standard deviation and stored in matrices X and Y, respectively. The matrix of independent variables X is decomposed as:

$$X~T{P}^T$$ (3)

where T is an orthonormal matrix referred to as the score matrix. The dependent variables Y are decomposed in terms of T as:

$$Y~TB{C}^T$$ (4)

where B is a diagonal matrix of regression weights and C is a weight matrix of the dependent variables. Using the orthonormal property of the score matrix T and Eqs. (3) and (4), the matrix B_PLS can be computed as:

$$B_{PLS}={\left(P^T\right)}^{-1}B{C}^T$$ (5)

Equation (5) presents a relation between the predictor and the dependent variable. Upon its computation, B_PLS is used to generate new estimates of the dependent variable α_i(t) for newly observed surface predictors $\overrightarrow{x}(t)$.^40,41

2.B. Empirical demonstration using patient data

Under an IRB-approved protocol, a lung SBRT patient (male, 74 yr old, secondary malignant neoplasm of bilateral lung, tumor in right lower lobe) was consented for this study. In accordance with our institutional standard-of-care, CT simulation data in the form of a phase-sorted 4DCT scan (Philips 16-slice Brilliance Big Bore) were acquired from the patient. The 4DCT scan consisted of ten average phases each composed of 104 3-mm thick slices, reconstructed on a 512 × 512 matrix (1.08 × 1.08 mm²). Concurrent to the 4DCT simulation, real-time surface photogrammetry data were acquired from the thoraco-abdominal surface (including the ROI) at ~15 Hz using a prototype, research, couch-mounted, VRT system.

The reason for developing a couch-mounted VRT version was to be able to maintain a constant optical field-of-view (FOV), unlike the ceiling-mounted system, where the camera’s FOV changes continuously with the movement of the CT couch. The couch-mounted VRT used a camera and software identical to those incorporated in the commercially available ceiling-mounted, single-camera pod VRT systems. The prototype VRT computer also incorporated a high-speed image capture (HSIC) card and a high-speed disk, both of which enabled the system to capture and store real-time surface point clouds. The couch-mounted system was calibrated using the same calibration protocol as the clinical ceiling-mounted system. The accuracy of the couch-mounted system was validated against that of the ceiling-mounted system and determined to be within ±0.5 mm in all three cardinal directions.

The 4DCT images in this study were phase binned using the clinical RPM system, which was placed at the umbilicus according to our institution’s guidelines. Presence of the RPM box resulted in geometric distortions on the surface. In order to use the VRT surfaces, an 8 cm × 4 cm rectangular section was removed from the 4DCT acquired surfaces prior to their reconstruction. This missing patch was then “filled in” using the Poisson surface reconstruction process described above, and the reconstructed surface was used for model construction. Previous phantom studies performed by our group have found this technique to have submillimeter accuracy (RMSE) and maximum point errors lower than 2 mm.³³

Figure 2(a) shows the patient setup for the 4DCT + VRT data acquisition. Prior to patient placement on the couch, the VRT system underwent daily calibration. Following patient setup on the CT couch, the VRT system was placed over the patient’s legs, ~80 cm from the Xiphoid process. Further adjustments of the speckle pattern were performed according to the room light and patient skin tone. For a typical 4DCT scan that lasts 100 to 120 s, a surface dataset consisting of 1500–1800 observations was acquired. One example of a VRT surface, reconstructed using a 100 × 100 mesh grid with voxel size 6.7 × 6.8 mm² is shown in Fig. 2(c). The 15 × 20 cm² ROI, covering parts of the thorax and the abdomen, is highlighted in blue and was used as model input. We found this ROI to be the largest, noise-free, rectangular surface region that could be extracted from the acquired VRT raw data. The 4DCT and the optical surface data were used to construct the volumetric model using the methodology described in the section on theoretical framework for the model.

Fig. 2.

Measurement setup. (a) Surface acquisition setup during four-dimensional computed tomography (4DCT) measurement: a prototype couch-mounted vision radiotherapy (VRT) system is used to monitor the patient’s thoraco-abdominal surface while the patient is scanned. The system is mounted on the couch and has the benefit of maintaining the same field-of-view for each couch position. (b) Surface acquisition setup during RT delivery; fluoroscopic data projections are acquired simultaneous with surface recording. The measurements are performed at the start and the end of treatment using the kV imager of a TrueBeam linac and VRT system. (c) A surface example reconstructed by Poisson method. A region of interest (highlighted in blue) that covers parts of the thoraco-abdominal surface is used as input for the model.

2.C. Proof-of-concept validation of the model

A proof-of-concept validation of the model was performed using the lateral and posterior–anterior kV fluoroscopy data which captured patient motion when the patient was on the treatment table. The kV fluoroscopic sequences, which served as the nominal ground truth, were acquired using the on-board kV imager (OBI) on the Truebeam linac (ABC on). For a given treatment fraction, the FL time series were acquired concurrently with VRT surfaces in two sessions — immediately prior to (FX1-start, FX2-start, and FX3-start) and immediately following RT delivery (FX1-end, FX2-end, and FX3-end). A source to detector distance (SDD) of 150 cm and source to iso-center distance of 100 cm were used. Fluoroscopic acquisition (7 mAs, 100 kVp) was performed for ~30 s at ~7 fps with the kV source placed at 270° (i.e., lateral projections) for the first and third fractions (FX1 and FX3) and kV source (7 mAs, 85 kVp) placed at 180° during the second fraction (FX2). Contrast enhancement was performed by applying adaptive histogram equalization using 256 bins.⁴² For each fluoroscopic frame (768 × 1024 pixels), a radiation oncologist manually selected the border of lungs with the diaphragm using the 3D slicer image computing toolbox.⁴³ In the lateral projections, this region is referred to as liver here after. For the posterior–anterior projections, the interface between the diaphragm and the left and right lobes was visible and contoured separately. These contours were stored as binary masks to be used in the process. Simultaneous with the FL acquisitions, surface photogrammetry data were acquired using the couch-mounted VRT system, placed on the linac couch [Fig. 2(b)]. The setup and acquisition of surface data were identical to those used during sim CT. Thus, for this patient, a total of six pairs of concurrent fluoroscopic and photogrammetry acquisitions are reported.

Two key aspects of the motion model were examined — (a) how much additional motion information does the model yield compared to 4DCT, and (b) how robust and accurate is the model from fraction to fraction, that is, to what extent does the association between the external surface and the internal volume established during model creation hold true during dose delivery. Toward both objectives, we generated digitally reconstructed fluorograph (DRF) matching the FL geometry by applying forward projection ray-tracing operators available from the python interface of the ASTRA toolbox.⁴⁴

Figure 3 shows the steps in process of proof-of-concept of the model. To compare the potential benefit associated with the use of the model over 4DCT and characterize the model performance, for each of the ten 4DCT volumes, right lateral projections were generated according to the FL imaging geometry with the tumor centroid (planned tumor volume centroid) placed at imaging iso-center using the forward projection. Additionally, for each surface acquired at the time of 4DCT, a DRF was constructed from the corresponding model-generated 3D image.

Fig. 3.

Flowcharts of frame work for proof-of-concept validation of the model.

The day-to-day robustness and accuracy of the model was characterized by deforming the P50% liver mask using the model-estimated DVFs. A DRF of the deformed liver mask was generated and used to track the shadow of the liver in the projections. Accuracy of the model was quantified by comparing model-estimated displacement of patches on the diaphragm, and the overlap between the estimated liver contour with respect to the patch displacement and the segmented liver contour observed from ground truth FL. The Dice coefficient of the contours in the two DRFs was also calculated as a similarity measure according to Eq. (6) and compared with that in the corresponding ground truth FL.

$$\mathit{\text{Dice coefficient}}(A,B)=\frac{2\times |(A\cup B)|}{|A|+|B|}$$ (6)

where A and B are the contours in the reference DRF (ground truth FL) and the 4DCT-DRF.

3. RESULTS

3.A. Surface training dataset

Convergence of the RMSE and maximum absolute error for the surface ROI with increasing number of principal components is shown in Fig. 4(a). After using three surface PCs, the RMSE and the maximum absolute error converged rapidly to sub-1 and 1 mm respectively and plateaued. Analysis of the coefficients for higher order components showed the signal to be contaminated with noise and lacked any visible pattern or relation with respiratory motion.

Fig. 4.

Properties of acquired surfaces. (a) Convergence of error with increasing number of surface principal components (PCs). (b) Evolution of first three surface PC coefficients in time. (c) Relation between first two surface PC coefficients. (d) Relation between surface PCs with mean surface AP displacement.

Evolution of the first three surface PCs in time is shown in Fig. 4(b). Variations in amplitude and frequency were observed in all three signals. The first component was found to exhibit a high correlation with mean AP displacement of the surface ROI. This relation is shown in Fig. 4(d) where a linear relationship (R² = 0.998) is observed between the first PC and the mean ROI AP displacement. The second surface PC coefficient exhibits little correlation with the surface AP (R² = 0.138).

The second surface component was found to vary in a circular pattern with the first surface component. Phase-averaged values of these components, standardized to have zero mean and unit deviation, were computed from surfaces acquired at the time of the 4DCT simulation and are shown in Fig. 4(c). Overlaid in black are surface coefficients observed during FX1-start. Each loop traced by the black curve corresponds to one breathing cycle. Changes in the breathing pattern including shallower and deeper breaths produced different loops which often deviated from the relation observed at the time of the 4DCT simulation.

3.B. DVF training dataset

Convergence of RMSE and maximum absolute error (mm) with increasing the number of DVF-PCs included in Eq. (4) are shown in Fig. 5(a). A rapid convergence in RMSE was observed with the first two components and a subsequent plateau in convergence was reached thereafter (RMSE = 0.98 mm). Similarly, the maximum absolute error associated with PCA reconstruction reduced from 6.3 mm after using a single component to 2.21 mm when two PCs were included. The incremental benefit of using higher order PCs (i.e., beyond 2) was negligible due to the nonlinear nature of the deformations. The voxels with the largest error magnitude were found at the left lateral interface of the lung and the diaphragm. Variations in the first two DVF-PC coefficients are shown in Fig. 5(b). The two components traced out a circular pattern that resembled the trajectory of the surface components [Fig. 4(c)]. Model-generated PC coefficients estimated from surfaces that were acquired during FX1-start are shown in black in Fig. 5(b). Each respiratory cycle is marked as one loop in this curve, and the presence of cycle-to-cycle variations produced a different loop for each cycle.

Fig. 5.

Properties of volumetric deformation vector fields (DVFs). (a) Convergence of displacement error with increasing number of DVF principal components (PCs). (b) Relation between the first two DVF-PC coefficients.

3.C. Model Performance

Figure 6 highlights the observed relationship between the first DVF-PC coefficients and the first phase-averaged surface PC coefficients. Both datasets were standardized to have zero mean and unit variance. Both the predictor (surface PC coefficients) and predicted variables (DVF-PC coefficients) exhibit a high correlation with each other. Appearing in red are PLSR-estimated values when the training sets were used as input for the model.

Fig. 6.

Relation between first deformation vector fields-principal components coefficient and first surface coefficient.

Figure 7(a) shows a FL projection during FX1-start. Similar contrast and grayscale values were chosen for all the images in Fig. 7 to aid in visual comparison. A DRF of the deformed liver masks was generated and used to track the shadow of the liver in the projections. An example of the model-generated DRF overlaid with the liver contour is shown in Fig. 7(b). Additionally, the liver was contoured in each 4DCT phase and its projection was used to track the liver in each DRF. An example of the P50% DRF overlaid with the liver contour is shown in Fig. 7(c). Similarly, Fig. 7(d) shows a FL projection acquired during FX2-start, with the model-estimated DRF, and corresponding 4DCT-DRF shown in Figs. 7(e) and 7(f). The interface of the diaphragm for the left and right lung lobes are visible in these images and were examined separately.

Fig. 7.

Measured and generated right lateral, and posterior–anterior projections and position of tracked patches. (a) Fluoroscopic data (FL) projection acquired at the start of the first radiotherapy (RT) delivery fraction. A radiation oncologist segmented the shadow of the liver (in red). (b) Digitally reconstructed radiograph (DRF) generated using a right lateral projection geometry from the model-estimated computed tomography (CT) image. A projection of the deformed liver mask is used to track the liver contour in the DRF. (c) DRF generated using the corresponding phase assigned four-dimensional CT (4DCT) image. (d) FL projection acquired at the start of the second RT delivery fraction. (e) DRF generated using a anterior — geometry from the model-estimated CT image. Masks of the liver-lung interface (right) and stomach/spleen–lung interface (left) were tracked and used for evaluation. (f) DRF generated using the corresponding phase assigned 4DCT image.

Figure 8(a) presents model and 4DCT liver Dice coefficients as a function of time. Dice coefficients comparing the model-estimated contour (blue curve) with those from FL (red curve) were consistently above 0.9; indicating a high degree of overlap of the FL measurements. In contrast, Dice coefficients comparing the 4DCT-derived contour (green curve) with those from FL had a minimum of 0.86, indicating lower similarity to the ground truth FL measurements. 4DCT similarity to FL were found to exhibit larger oscillations with the breathing cycle and showed rapid falls at the end of inspiration. Two snapshots of the liver contours at t = 8.4 s and t = 14 s are shown in Figs. 8(b) and 8(c). At t = 8.4 s, the patient was at the end of inhale. However, due the shallow nature of the breath, the 4DCT contour overestimated the actual diaphragm motion and resulted in the low Dice coefficients appearing in the green curve in Fig. 8(a). Manually reviewing the FL images near t = 14 s and t = 22 s, we observed that along with the expected liver motion in the SI direction, the liver also exhibited an intrusive posterior–anterior motion at these times. This motion was not captured by neither the model nor the 4DCT, so that both contours resulted in a similar Dice coefficient.

Fig. 8.

Evaluation of model performance using the first fluoroscopic acquisition of first radiotherapy delivery. (a) Comparison of model and four-dimensional computed tomography derived liver Dice coefficients. (b) Liver contours at t = 8.4 s. (c) Liver contours at t = 14 s.

Figures 9(a) to 9(c) compare model-estimated (blue) and 4DCT-derived (green) patch positions with FL measurements (red). For all three patches, a gain in performance over 4DCT was observed at the end of exhale phase. However, while good agreement between model and FL was found over the first cycle, both the model and 4DCT overestimated the patch position at the end of inhale during the second and third cycles. Similarly, comparison of 4DCT and model-estimated patch positions for the left and right lobes are shown in Figs. 9(d) and 9(e). For this fluoroscopic acquisition, the 4DCT measurement overestimates patch motion at end of inhale while the model is able to account for the shallower respiration. Therefore, while good agreement with fluoroscopy is seen at the end of exhale phase for both 4DCT and model-estimated patch positions, a better agreement between model and FL images is seen during the peak inhale phases. For patch #5, acquired concurrently to patch #4, both the model and 4DCT underestimate the magnitude of motion.

Fig. 9.

Comparison of model (blue) and four-dimensional computed tomography (green) estimated patch position with flouro measurement (red). Displacements are reported in voxel #. A magnification factor of 0.28 mm/voxel (measured at ISO) is used to report values in mm. (a) Estimated patch displacements for patch #1. (b) Estimated displacement for patch #2. (c) Estimated displacement for patch #3. (d) Estimated displacement for patch #4. (e) Estimated displacement for patch #5.

Figure 10(a) shows a box plot comparison of model and 4DCT Dice coefficients computed with respect to FL. Median Dice coefficients for each respiratory phase are marked in yellow, and the range covered by the whiskers indicates coverage of the 99th data percentile. While improvements in Dice coefficients were seen at all instances of the respiration cycle, the most marked improvement over 4DCT was observed at the end of exhale phase (P50%) and was followed by mid-inspiration phases (P10% to P40% and P60% to P80%). A similar pattern was seen for the patch displacement errors reported in Fig. 10(b); a noticeable reduction in the error spread (range of whiskers) was seen at P50%, followed by the mentioned mid-inspiration and expiration phases. For this acquisition, both the model and the 4DCT overestimated displacements at inspiration (P0% and P90%).

Fig. 10.

Performance in estimated Dice coefficient and patch displacement error as a function of respiratory phase. Mid expiration phases (P10% to 40%) and mid-inspiration phases (P60% to 80%) have been coalesced into one data point each. (a) Comparison of model and four-dimensional computed tomography (4DCT) estimated liver Dice coefficients. (b) Comparison of model and 4DCT patch displacement errors.

3.D. Impact of observed changes in the external–internal correlation

Figures 11(a) and 11(b) present comparisons between model- and 4DCT-estimated Dice coefficients, and patch displacement errors from four lateral FL scans, FX1-start, FX1-end, FX3-start, and FX3-end. Similarly, Figs. 11(c) and 11(d) present comparison results between model- and 4DCT-estimated Dice coefficients and patch displacement errors for the left and right lobes in posterior–anterior scans. In comparison to 4DCT, use of the model consistently resulted in higher Dice coefficients, and lower patch displacement errors.

Fig. 11.

Comparison of model (blue) and four-dimensional computed tomography (4DCT) (green) estimated Dice coefficients and patch displacement errors during four fluoroscopic data scans. Right lateral projections were acquired at the start (FX1-start) and end (FX1-end) of radiotherapy (RT) fraction 1, and start (FX1-start) and end (FX3-end) of RT fraction 3. Posterior–anterior projections were acquired at the start (FX2-start), and end (FX2-end) of fraction 2. (a) Liver Dice coefficients calculated from lateral projections. (b) Lateral projections patch #1–3 displacement errors. (c) Dice coefficients for diaphragm interface with right and left lobes calculated from posterior–anterior projections (FX2). (d) Patch displacement errors for the right (patch #4) and left (patch#5) lobes calculated form posterior–anterior projections.

The model proposed in this work relies on a-priori external–internal correlation observed during the 4DCT acquisition. A by-product of its application was the assumption that this correlation is preserved during all subsequent stages of RT. When this condition was loosely satisfied, marked improvements in Dice coefficients and patch displacement errors over the current standard of care 4DCT were observed. However, violation of this assumption (due to interfraction variations in the breathing pattern) led to errors that were comparable to those observed with 4DCT. Figures 12(a) and 12(b) show an example, where the correlation between the first or second surface PC coefficients and the voxel location of patch #1 changed during different acquisitions. The relation that was observed at the time of 4DCT simulation is shown in blue. Overlaid in the figure are model-derived (blue stars) and FL-measured (red points) values during the acquisition FX1-start. While considerable overlap was observed near the end of exhale (negative values of surface PC #1), the correlation changed in such a way that the patch displacement was overestimated by the model at the end of exhale. This resulted in the large discrepancies in patch displacement error observed near the end of inspiration (P0% and P90%) in Fig. 10(b). A more notable example of the violation of this assumption was seen for FX3-start.

Fig. 12.

Observed change in the external–internal correlation as a function of time. (a) Surface PC#1 coefficient vs. patch#1 position as observed during initial four-dimensional computed tomography simulation (blue triangles), estimated by model for FX1-start (dark blue stars), measured by using fluoroscopy during FX1-start (red stars) and FX3-start (orange).

3.E. Processing and computational time

Table I lists the processing times (in minutes) for some of the tasks performed in this study. Processing times include disk read/write operations and were reported using the time difference between the start and termination of the process performed on a workstation with a 4-core intel i7 CPU(4.4 GHz) and 32 GB of RAM. Potential solutions to significantly increase the reconstruction speed include implementation of the surface reconstruction on graphic processor units (GPUs) or improving the quality of the captured surfaces by using multiple cameras which can potentially remove the requirement for surface reconstruction.

Table I.

Processing times (in min) for surface reconstruction and image processing tasks performed in this study.

	Task	Time (minutes)
Surface reconstruction	Extract.bmp files from VRT.raw data	50 mins (for a 3-min recording)
	Extract.obj files from processed.bmp data	30 mins (for a 3-min recording)
	Water-tight surface reconstruction	240 mins (for 2750 surfaces)
PCA of 4DCT	Calculate volumetric DVFs	40 mins (for 9 DVFs)
	PC Decomposition	4 mins (for 9 × 512 × 512 × 101 DVFs)
Proof-of-concept validation	Construct DRRs and write to disk	85 mins (for 200 frames)
	Contour DRRs and FLs in 3D Slicer	80 mins (for 200 frames)

3D, three-dimensional; 4DCT, four-dimensional computed tomography; DVF, deformation vector fields; FL, fluoroscopic data; PCA, principal component analysis; VRT, vision radiotherapy.

4. DISCUSSION

This work considered the feasibility of using regions of the thoraco-abdominal surface to aid inform 4DCT images and capture some of the encountered variations when monitoring respiratory motion over several cycles. Our motion model used real-time measurements of the surface for the first time to estimate corresponding volumetric DVFs of the underlying anatomy. By performing fluoroscopic scans that were well spread out over the course of the patient’s treatment, we provided a clinically relevant and in-situ evaluation of our model’s performance. We then performed an in-patient proof-of-concept validation prospectively. We quantified the gain in model performance by comparing its accuracy with respect to the current standard-of-care 4DCT. As an additional consequence of using scans spread out over two treatment fractions, our results highlighted the ever-changing nature of the internal–external correlation and emphasized the need for frequent model validation and update. We expect that the ability to capture such cycle-to-cycle variation opens the possibility for more advanced managements.

Analysis of VRT captured surfaces indicated the presence of three independent PC components in the surface signal. Only a single component would be present and needed to capture the motion of a surface if it moved in a perfectly correlated manner along a single direction. Therefore, while further investigations regarding the interpretation of these signals is required, this presence of the second and third PC components may be associated with the involvement of different muscle actuators, such as the abdominal and intercostal muscles, that aid respiration. These components may be necessary to capture deformations such as expansion and compression of the surface. Our results suggest that a three PC-representation of the surface ROI is adequate for capturing surface motion to within submillimeter accuracy. Consequently, for the surface ROI considered in this study, the use of PCA is found to be a viable option for reducing the dimension of VRT surface ROIs.

A limiting factor related to the model developed in this study is the quality of surface images that are acquired when using the single unit, couch-mounted VRT system. Unlike clinical VRT systems, which are typically equipped with two to three ceiling-mounted units and permit the acquisition of large and relatively noise-free surface ROIs, the surfaces acquired from the single unit couch-mounted system are more prone to geometric distortions and shadowing artifacts. As a result, the acquired surfaces required reconstruction prior to their use. One notable example of such distortions is shadowing effects caused by protruding breasts that are common to female patients. Our experience with surfaces captured from female volunteers using the couch-mounted VRT system has been that large missing patches in the breast areas and superior thoracic region are commonly present. Such areas of missing data result in noisy surfaces and can degrade model performance when included in the ROI. Additionally, we found that small surface ROIs (e.g., smaller than 5 × 5 cm²) provided only one noise-free principle component, and were essentially equivalent to a 1D surrogate. The quality of the higher order surface PCs improved with the increasing ROI size. For these reasons, we selected the largest, noise-free, rectangular ROI that could be extracted from the raw data. A possible solution to improve the size and quality of the acquired surfaces is the placement of a second camera system superior to the patient which monitors the thoracic region. This technique has the potential to permit the acquisition of accurate thoraco-abdominal surfaces for both male and female patients.

To register the surface and volume dataset pairs, accurate landmarks are needed. One of the challenges we encountered was finding reliable and consistent landmarks between 4DCT and VRT time series (throughout both time series). A potential approach to mitigate this challenge would be to use surface markers that are reflective as well as radio-opaque (e.g., attached onto a form-fitting vest).

As part of this study, model performance and accuracy were evaluated using the diaphragm motion observed from manual contouring of 1300 × 2 fluoroscopic and DRF images. Both lateral and posterior–anterior projections were used in this process. Generally, the largest magnitude of respiration-induced motion is in the SI direction. However, several studies have reported that the magnitude motion in the AP direction can be as large as the SI direction.^12,45 The contoured dice coefficients and patch displacement errors reported from lateral projections provided an integrated evaluation of model performance in both SI and AP directions. However, lateral projections also compound the motion of each lung onto one image. In lung patients, motion of each lobe can be different depending on the affected lung, as well as the reproducibility of organ positions at the diaphragm interface such as the liver (right lung) and the stomach and spleen (left lung). The use of PA projections in this study enabled a quantitative analysis of model-estimated diaphragm motion in the SI direction for separate lung lobes.

There are several limitations in the proposed proof-of-concept validation methodology which stem from the use of kV FL images. kV images are commonly aid us in RT for patient simulation, motion assessment, and setup. The wide availability of this modality during RT delivery sessions, and the potential real-time information of the internal anatomy in the form of planar projections make these images a valuable tool in RT. Two drawbacks associated with kV fluoroscopy are low image contrast and convoluted nature, as well as the additional imaging dose.⁴⁶ These drawbacks limit the generality and applicability of our results. For example, as mentioned earlier, model performance should be evaluated for several landmarks, including clinically relevant organs and sites such as the GTV. However, the low contrast nature of these images prevented us from delineating other relevant structures including the GTV. Additionally, to reduce the associated imaging dose, the durations of fluoroscopic acquisitions presented in this study are short in comparison to the RT delivery session (which typically lasts 7–10 min). During these scans, the end of exhale phase for this patient was fairly reproducible and the patient did not exhibit baseline shifts. As a result, model performance in the presence of base-line shifts remains ambiguous and warrants further investigation. In a previous study, we used a similar methodology to account for cycle-to-cycle variations including baseline shifts in an externally and internally deformable lung phantom that used two actuators to generate variable internal–external correlation.⁴⁷ However, the well-controlled experimental nature of phantom-measurements and the different nature by which motion is generated in the human body limit the applicability of that methodology to the current study. In the present study, the impact of change in the external–internal correlation that was seen in FX3 [Figures 12(a) and 12(b)] suggests that deviations such as baseline shifts, which impact the a priori observed internal–external correlation, can degrade model performance.

It is important to note that for the patient reported in this study, we were unable to see the GTV itself in the FL scans and the DRFs. While observing the GTV in the FL as well as in the DRFs would of course be ideal, we contend that this is not absolutely essential for the proof-of-concept step of the model. The primary objective of the proof-of-concept step was to ensure that the surface-to-internal correlation is spatiotemporally consistent, over the entire volume, with that established during the model creation stage, and multiple high-contrast, soft tissue-based features are adequate to validate the model. Clinically, an analogous philosophy is followed when we validate tumor position (single-point monitoring) using orthogonal x-rays of implanted radio-opaque fiducial markers.

To date, the most common validation technique for CT-based motion models has been to compare model performance using volumetric CT images from the same session or a 4DCT from another session. In contrast, in the present study, the variation in the internal and external anatomy correlation is observed in all FL acquisitions.¹⁷ Deviations are largest at the end of inhale and highlighted the limitation of models built from a priori correlation. The end-exhalation phase was most reproducible although we observed comparable uncertainties to end of inspiration phase for the first acquisition of fraction 3 (FX3-start). Such findings are only possible when examining the internal–external correlation at different times and emphasize the need for model updates through additional imaging of the internal anatomy. These uncertainties will not be encountered when validating model with the leave one out approach where a dataset of the 4DCT session is used for validation.⁴⁸ Use of 4DCT of a different session provides a validation of the patient, due to the averaged nature of the 10-phase representation, variations particularly at the end of inspiration phase can be limited.

Also noteworthy is the fact that our proof-of-concept validation strategy is agnostic to the underlying motion model. Any surrogate-based volumetric motion model can be validated so long as there is access to in-room kV fluoroscopy.⁴⁹

Whereas, in the case of this patient, the model largely retained its fidelity over multiple fractions, there were instances [Figs. 12(a) and 12(b)] where this correspondence changed. In general, one cannot assume that this correspondence will hold true for all patients and/or for cases involving conventional fractionation (~30 fractions compared to 3–5 for SBRT). In fact, in the case of conventionally fractionated RT, it is also possible the internal/external correlation varies due to changes in anatomy such as tumor shrinkage and weight loss. For these reasons, it is important to extend the current modeling framework so that the kV FL of the day can be used not just to validate, but also to update the surface-internal correspondence. 4D-CBCT data acquired during the RT delivery session can also serve to update and evaluate the model. In principle, 4D-CBCT could help verify the fitness of the model throughout the treatment, and highlight deviations from the simulation run. In the presence of deviations, the acquired 4D CBCT dataset can permit model update. Additionally, projections acquired from such a dataset, which are time-stamped and correlated with the breathing signal have the potential to aid in model uncertainty quantification.

5. CONCLUSION

We described the development and a proof-of-concept for a volumetric motion model that uses surface photogrammetry to correlate the time-varying thoraco-abdominal surface with the time-varying internal thoraco-abdominal volume. These early results indicate that the proposed approach can result in a marked improvement over 4DCT. However, because the model is built from data acquired during a single 4DCT simulation session, it may not remain completely valid from fraction to fraction, and during the same fraction. In future work, this limitation may be addressed using the kV fluoroscopy in-room data to periodically update model parameters in order to maintain its accuracy.