A pilot study using kernelled support tensor machine for distant failure prediction in lung SBRT

Abstract

We developed a kernelled support tensor machine (KSTM)-based model with tumor tensors derived from pre-treatment PET and CT imaging as input to predict distant failure in early stage non-small cell lung cancer (NSCLC) treated with stereotactic body radiation therapy (SBRT). The patient cohort included 110 early stage NSCLC patients treated with SBRT, 25 of whom experienced failure at distant sites. Three-dimensional tumor tensors were constructed and used as input for the KSTM-based classifier. A KSTM iterative algorithm with a convergent proof was developed to train the weight vectors for every mode of the tensor for the classifier. In contrast to conventional radiomics approaches that rely on handcrafted imaging features, the KSTM-based classifier uses 3D imaging as input, taking full advantage of the imaging information. The KSTM-based classifier preserves the intrinsic 3D geometry structure of the medical images and the correlation in the original images and trains the classification hyper-plane in an adaptive feature tensor space. The KSTM-based predictive algorithm was compared with three conventional machine learning models and three radiomics approaches. For PET and CT, the KSTMbased predictive method achieved the highest prediction results among the seven methods investigated in this study based on 10-fold cross validation and independent testing.

Graphical Abstract

1. Introduction

Non-small cell lung cancer (NSCLC) is the most common type of lung cancer worldwide (World Health Organization, 2017). Approximately 15 – 20% of NSCLC patients present with early-stage, localized disease that can be controlled upon receiving treatment with curative-intent (Howlader N, 2016), although this number may rise with the recent implementation of CT-based screening programs (Team, 2013). In general, surgery has been the standard treatment for early stage NSCLC with long-term local control rates of 80–94% (Miller et al., 2016), but systemic failure rates range from 15–20% (Ginsberg et al., 1995; Martini et al., 1995). For patients who are not surgical candidates, stereotactic body radiation therapy (SBRT) has demonstrated local control rates over 95% (Nikolaev et al., 2016; Timmerman et al., 2010), becoming the standard of care for medically inoperable patients (Chi et al., 2010; Ettinger et al., 2010; Liu et al., 2015). Nevertheless, as with surgical patients, distant failure is common in this patient population. In the RTOG 0236 trial, the distant failure rate five years after SBRT was 31% (Timmerman et al., 2014). Another retrospective study on lung SBRT patients also found distant failure rates of 14.7%and 19.9% within two and five years, respectively (Senthi et al., 2012). Other prospective trials with small samples sizes also showed that SBRT for early-stage primary lung cancer achieved local control rates over 95% with high distant failure rates of about 29% (Nath et al., 2011). Unlike surgical cases, where features of the resected tumor specimens are the basis for identifying patients who may benefit from systemic therapy, risk models for identifying analogous patients in the SBRT population have not gained wide acceptance. Analogous to surgical patients, accurate prediction of early distant failure in NSCLC patients receiving SBRT has the potential to identify patients who would benefit from systemic therapy.

Medical imaging acquired as part of standard of care, such as computed tomography (CT) and fluorodeoxyglucose positron emission tomography (FDG-PET), has the potential to predict outcomes in early stage NSCLC treated with SBRT (Clarke et al., 2012; Lian et al., 2016; Lovinfosse et al., 2016; Tan et al., 2013). With recent advances in radiomics techniques, many approaches have been developed for predictive modeling based on medical images (Cook et al., 2014; Gillies et al., 2015; Kumar et al., 2012; Lambin et al., 2012; Madabhushi and Lee, 2016; Tajbakhsh et al., 2016; Tan et al., 2013), mainly involving features/input determination and classifier learning.

In general, current methods of classification modeling based on medical images can be divided into three categories: 1) handcrafted feature-based approaches (Aerts et al., 2014; Lian et al., 2016; Liu et al., 2017; Namburete et al., 2015; Parmar et al., 2015; Vallières et al., 2015; Wimmer et al., 2016; Wu et al., 2016; Zhou et al., 2017); 2) feature learning-based approaches (Ciompi et al., 2015; Kooi et al., 2017; Kumar et al., 2015; Roth et al., 2016; Shen et al., 2015; Shin et al., 2016); and 3) voxel/pixel-based approaches (Huang et al., 2017; Khamis et al., 2017; Zuluaga et al., 2015).

In handcrafted feature-based methods, quantitative imaging features are manually extracted first. Feature selection methods are then used to select optimal feature subsets t5o reduce the redundancy of handcrafted features. Finally, the selected features are used to construct a predictive model. Although quantitative imaging-based features have shown to be effective in building predictive models, the generic nature of their features may not fully reflect the unique characteristics of a particular cancer (Chung et al., 2015). Moreover, different feature selection methods often obtain different optimal features subsets (Zhou et al., 2016), hindering the robustness of the classification.

Recently, feature learning-based approaches, such as the discovery radiomics method (Chung et al., 2015) and the deep convolutional neural network (CNN) (Shin et al., 2016) have been developed using the intrinsic features learned automatically for the predictive model. These methods require large scale annotated datasets (such as ImageNet (Krizhevsky et al., 2012)) to achieve acceptable prediction accuracy. However, obtaining such a large annotated dataset for radiation treatment outcome prediction is still challenging (Miotto et al., 2017).

Investigators have proposed constructing models using original images without handcrafted features and feature learning, i.e., voxel/pixel-based models. The conventional voxel/pixel-based approaches first vectorize the image and then use the vectorization data as input for modeling. In this case, the dimensionality of the input sample is usually high. High space dimensionality may induce small sample size problems (Raudys and Jain, 1991) and the curse of dimensionality (Keogh and Mueen, 2011) for medical data with small training sample size, leading to overfitting and decreasing classification accuracy. Dimensionality reduction methods, such as the down-sample method (Khamis et al., 2017; Magnin et al., 2009), were developed to improve the performance of classifiers. However, image vectorization with an intrinsic tensor structure breaks the geometry structure and correlation in the original image so that its information cannot be fully taken into consideration. Meanwhile, reducing dimensionality also induces the loss of available tumor information.

In contrast to the methods mentioned earlier, the tensor space model (Kolda and Bader, 2009; Liu et al., 2010; Shashua and Levin, 2001) uses high order imaging with an intrinsic high order tensor structure as input for modeling, eliminating the need for handcrafted imaging features, feature learning, or image vectorization. Three dimensional (3D) medical imaging methods, such as PET and CT, have an intrinsic tensor structure suitable for tumors as 3D objects. Support tensor machine (STM) is a common tensor space model that is the generalization of support vector machine (SVM) from vector space to tensor space. In contrast to conventional radiomics approaches that rely on handcrafted imaging features, the STM-based classifier uses 3D imaging as input for the model, taking full advantage of the information from whole images without using handcrafted features. The STM-based classifier does not break the natural 3D geometry structure of the image and correlation in the original image.

Tao et al. introduced first the STM framework based on the multilinear algebra theory with a brief discussion on potential kernelization (Tao et al., 2005). They further gave the alternating projection optimization procedure with a convergent proof for STM without kernelization (Tao et al., 2007). This algorithm and its developed forms have been applied successfully to pedestrian detection, face recognition, and remote sensor (Biswas and Milanfar, 2016; Hao et al., 2013; Zhang et al., 2011). In most of these applications, STM models are trained by either iterative projection techniques (Zhang et al., 2011) or rank decomposition methods (Hao et al., 2013) because the training model of STM is not a convex optimization problem in tensor space and its closed form solution is unavailable. In these training algorithms, the classification hyper-planes are trained in the original tensor space. Thus these STMs are called as linear STMs (LSTM) since they only involve multilinear operations without the transformation from original tensor space to the feature tensor space. However, similar to kernel-based SVM models, it could be beneficial to construct the STM model in an adaptive feature tensor space transformed from original tensor space. Motivated by the application of kernels in SVM, we develop a kernelled STM algorithm (KSTM) for the STM in an adaptive feature tensor space that uses 3D tumor tensors as input to predict distant failure in early stage NSCLC treated with definitive SBRT in this pilot study. The KSTM iterative algorithm with a convergence proof is given and used to train the weight vectors for each tensor mode for the classifier.

2. Materials and Methods

2.1. Patient cohort

Our study included a cohort of 110 patients with inoperable early stage (IA and IB) NSCLC. They had been treated with SBRT from 2006 to 2016 with a median follow-up of 18 months. Twenty-five (22.73%) of these patients experienced distant failure. For each patient, pre-treatment CT and PET images were acquired by one of three clinical scanners: the SIEMENS Biograph 64/1094 (Siemens Medical Solution, Malvern, PA), the Philips Gemini TF/Dual GS (Philips Healthcare, Andover, MA) or the GE Discover ST (GE Healthcare, Waukesha, WI). The CT volume was composed of 274 to355 slices with 512 × 512 pixels. The PET volume was composed of 274 to355 slices with 168 × 168 pixels, 144 × 144 pixels or 128 × 128 pixels.

2.2. Tensor Construction

Tumor tensors were constructed from both PET and CT images. The standard uptake value (SUV) (Vansteenkiste et al., 1999) was first calculated for each PET image. Tumors were then segmented from both PET and CT images in a semi-automatic way as follows. The middle slice was segmented using the object information based interactive segmentation method (OIIS) that requires a seed region of tumor identified manually (Zhou et al., 2013). Once the central slice was segmented, the other slices were segmented by the well-known OTSU method (Otsu, 1979) that considers the similarity between two adjacent slices. For each imaging modality after segmentation, the 3D tumor tensors were constructed with the same size that was the biggest value among all tumors sizes, which center the tumors were located in and which periphery zeros were filled in. The tensor size of PET imaging was 17 ×17 × 17 and the tensor size of CT imaging was 53 × 56 × 20.

All tumor ROIs were interpolated to the same resolution using a 3D spline interpolation method (Sathyamurthy and Raff, 1975). To determine the tensor size after interpolation, the product between tumor ROI size (number of voxels) and physical size of each voxel at the same dimension was calculated. The quotient of the product was then divided by the physical size of voxel interpolated tumor ROI at its corresponding dimension became the size of the interpolated tumor. After interpolation, the tensor size of PET imaging was 34 × 34 × 43 and the tensor size of CT imaging was 55 × 56 × 80.

Consider the noise from interpolation processing, the three-dimensional Gabor filter banks were used to denoise the images (Ahmmed, 2011) by obtaining the frequency information of the local space domain, which are similar to that of the human visual system. The four constructed tumor tensors for PET imaging and CT imaging without interpolation and Gabor filtering are illustrated in Fig. 1. The tensors with 3D spline interpolation and 3D Gabor filtering were used as the input of the KSTM algorithm.

Fig. 1.

The four constructed tumor tensors (bottom row) and their corresponding 2D patches of the tumor from one of slice (upper row): (a) and (b) for PET imaging; (c) and (d) for CT imaging; (a) and (c) show the same patient without distant failure; (b) and (d) show the same patient with distant failure.

2.3. Kernelled Support Tensor Machine (KSTM)

The transformation from original tensor space $R^{J_1\times J_2\times \cdots \times J_N}$ to an adaptive feature tensor space $R^{I_1\times I_2\times \cdots \times I_N}$ is defined as follows

$$\varnothing :R^{J_1\times J_2\times \cdots \times J_N}\to R^{I_1\times I_2\times \cdots \times I_N}$$

satisfying

$$\varnothing (X)={\left({\varnothing }_{i_1,i_2,\cdots ,i_N}(X)\right)}_{I_1\times I_2\times \cdots \times I_N,}$$

where $X\in R^{J_1\times J_2\times \cdots \times J_N}.$

For tensor $X\in R^{J_1\times J_2\times \cdots \times J_N}$ , a hyper-tensor plane is constructed in the adaptive feature tensor space $R^{I_1\times I_2\times \cdots \times I_N}$ based on the mode product (De Lathauwer, 1997) between tensor and vector:

$$\varnothing (X){\prod }_{n=1}^N{\times }_n{w}^{(n)}+b=0,$$ (1)

where ${\text{w}}^{(\text{n})}\in {\text{R}}^{{\mathrm{I}}_{\mathrm{n}}},\text{n}=1,2,\cdots ,\mathrm{N}$ is a weight vector for mode n and b is the bias to be determined. A visual illustration of the mode product between a 3D tensor and three vectors is presented in Fig. 2.

Fig. 2.

The mode product between a 3D tensor and three vectors.

Given the training samples set ${\left\{X_m\in R^{J_1\times J_2\times \cdots \times J_N},y_m\in \{-1,1\}\right\}}_{m=1}^M,$ where y_m is the class label of sampleX_m, the hyper-tensor plane (1) is determined by solving the following optimization problem the similar to linear STM:

$$argminJ\left(w^{(n)}(n=1,2,\cdots ,N),b,{\xi }_m(m=1,2,\cdots ,M)\right)=\frac{1}{2}{\prod }_{n=1}^N{||w^{(n)}||}_F^2+C{\sum }_{m=1}^M{\xi }_m,$$ (2)

subjected to

$$y_m(\varnothing \left(X_m\right){\prod }_{n=1}^N{\times }_n{w}^{(n)}+b)\ge 1-{\xi }_m,$$ (3)

$${\xi }_m\ge 0,m=1,2,\cdots ,M,$$ (4)

where ξ_m is the misclassification error of the th training sample X_m (m = I,2, ···, M), and C ≥ 0 is the trade-off between the classification margin and the misclassification error.

Because the optimization problem defined above is not convex in the linear space, a closed form solution is unavailable. Moreover, it is hard to give the particular transformation ∅ since it is adaptive. Motivated by the iterative procedure of LSTM and SVM solver with kernel, a kernelled STM iterative algorithm (KSTM) is proposed by orderly fixing N - 1 weight vectors in (2)–(4)–(3) inherited from the previous step. This alternating approach is equivalent to the following N training models of SVM in feature space (n = 1,2, ··· , N):

$$argminJ(w^{(n)},b^{(n)},{\xi }_m^{(n)}(m=1,2,\cdots ,M))=\frac{1}{2}{||w^{(n)}||}_F^2{\prod }_{1\le l\le N}^{l\ne n}{||w^{(l)}||}_F^2+C{\sum }_{m=1}^M{\xi }_m^{(n)}$$ (5)

subjected to

$$y_m({\left(w^{(n)}\right)}^T\left({\varnothing }_n\left(X_m{\prod }_{1\le l\le N}^{l\ne n}{\times }_l{P}_l\left(w^{(l)}\right)\right)\right)+b^{(n)})\ge 1-{\xi }_m^{(n)},$$ (6)

$${\xi }_m^{(n)}\ge 0,m=1,2,\cdots M,$$ (7)

where w^(l), l = 1,2⋯, N and l ≠ n are fixed, ${\varnothing }_n:R^{J_n}\to R^{I_n}$ is an adaptive projective mapping from $R^{J_n}$ to $R^{I_n}$ and $P_l:R^{I_l}\to R^{J_l}$ is a projective mapping from $R^{I_l}$ to $R^{J_l}$ defined as (8) and (12).

The detailed iterative procedures to solve equations (5)–(7) are described in Table 1.

Table 1:

The KSTM iterative algorithm

Input: A set of tensor samples $\{X_m\in R^{J_1\times J_2\times \cdots \times J_N},y_m\in \{-1,1\}{\}}_{m=1}^M$, threshold parameters ε.

Output: $w^{(l)}\in R^{J_l}(l=1,2,\cdots ,\mathrm{N},l\ne \mathrm{n}),{\alpha }_m^n\in R^{I_n}(m=1,2,\dots ,M)$ and b(n).

Step 1: Set $w^{(l)}\in R^{J_l}$ equal to unit vector with same elements $w_0^{(l)},l=1,2,\cdots ,\text{N}$, and the initial projective mappings Pl in (5)–(7) are defined as $$P_l\left(w_0^{(l)}\right)=w_0^{(l)}.$$ (8)

Step 2: Fix w(l) (1 ≤ l ≤ N, l ≠ n) to be the known weight vectors.

Step 3: For n = 1,2,..,N, set $Y_i^{(n)}=X_i{\prod }_{1\le l\le N}^{l\ne n}{\times }_l{P}_l\left(w^{(l)}\right)$, i = 1,2,…,M.

Step 4: Assume $w^{(n)}={\sum }_{i=1}^M{\alpha }_i^{(n)}{y}_i{\varnothing }_n\left(Y_i^{(n)}\right)$, and solve the dual problem of the quadratic program problem (5)–(7) by SVM solver with kernel to obtain ${\alpha }_i^{(n)},i=1,2,\dots ,M$ as follows: $$argmaxJ({\alpha }_i^{(n)}(i=1,2,\cdots ,M))={\sum }_{i=1}^M{\alpha }_i^{(n)}-\frac{1}{2}{\sum }_{i,j=1}^M{\alpha }_i^{(n)}{\alpha }_j^{(n)}{y}_i{y}_{j}K(Y_i^{(n)},Y_j^{(n)}),$$ (9)

subjected to $$0<{\alpha }_i^{(n)}\le C,i=1,2,\dots ,M,$$ (10) $${\sum }_{i=1}^M{\alpha }_i^{(n)}{y}_i=0,$$ (11)

where K is a kernel function satisfying $K\left(Y_i^{(n)},Y_j^{(n)}\right)=<{\varnothing }_n\left(Y_i^{(n)}\right),{\varnothing }_n\left(Y_j^{(n)}\right)>.$

Step 5: Set $$P_l\left(w^{(l)}\right)={\sum }_{i=1}^M{\alpha }_i^{(l)}{y}_i{Y}_i^{(l)},l=n.$$ (12)

Step 6: Let n ← n +1 if n < N - 1and n ← 1 if n = N.

Repeat Step 3-Step6 until convergence i.e. satisfying (13).

Step 7: If $$|{\sum }_{l=1}^N(<P_l\left(w_t^{(l)}\right),P_l\left(w_{t-1}^{(l)}\right)>{\left|\left|P_l\left(w_t^{(l)}\right)\right|\right|}_F^{-2}-1)|<\epsilon ,$$ (13)

set $w^{(l)}=P_l\left(w^{(l)}\right)\in R^{J_l}(l=1,2,\cdots ,N,l\ne \mathrm{n})$ and output them, then go Step 8. Otherwise, go to Step 3 to Step 7. Here t represents the tth iteration.

Step 8: Set $Y_i^{(n)}=X_i{\prod }_{1\le l\le N}^{l\ne \mathrm{n}}{\times }_l{w}^{(l)},i=1,2,\dots ,M$ and solve the optimization problem (9)–(11) by SVM solver to obtain the final ${\alpha }_i^{(n)},i=1,2,\dots ,M$ andb(n). Then output ${\alpha }_i^{(n)},i=1,2,\dots ,M,b^{(n)},$ and stop.

Once the iterative KSTM training model defined in equations (5)–(7) is solved, the classification hyper-tensor plane is fixed and the class label of tensor sample X can be predicted using the following classification function:

$$\text{f}(\text{X})=\mathrm{sign}({\sum }_{i=1}^M{\alpha }_i^{(n)}{y}_i\text{K}\left(Y_i^{(n)},\text{X}{\prod }_{1\le l\le N}^{l\ne \text{n}}{\times }_l{w}^{(l)}\right)+b^{(n)}),$$ (14)

where the kernel function K is optional, which is similar to the kernel function applied in SVM. The radial basis function (RBF) kernel has demonstrated superior performances in various applictions. Thus our proposed algorithm also coupled with the RBF kernel in our all experiments.

The convergence of the iterative procedure is proved below.

Theorem 1. The iterative procedure (5)–(7) will monotonically decrease the objective function values in the optimization problem defined in equations (2)–(4), converging the KSTM iterative algorithm (5)–(7).

Proof: Equation (2) can be rewritten as:

$$J(w^{(1)},w^{(2)},\cdots ,w^{(N)})=\frac{1}{2}{\prod }_{n=1}^N{||w^{(n)}||}_F^2+C{\sum }_{m=1}^M{\xi }_m.$$ (15)

First, we set the initial weight vectorsw^(l) and projective mapping P_l in equation (5)–(7) in Step 1 and fix w^(l) in Step 2 of Table 1. Without loss of generality, starting n = 1 , we assume $Y_i^{(n)}=X_i{\prod }_{1\le l\le N}^{l\ne n}{\times }_l{P}_l(w^{(l)})$, i = 1,2,…, M. At the point of the minimum on the Lagrangian of the optimization problem (5)–(7), the one order derivative of this Lagrangian with respect to w⁽¹⁾ must be zero. Thus

$$w^{(1)}={\sum }_{i=1}^M{\alpha }_i^{(1)}{y}_i{\varnothing }_1(Y_i^{(1)}),\text{n}=2,3,\dots ,\text{N}.$$ (16)

and the optimization problem (5)–(7) is equivalent to its dual problem defined in (9)–(11) with a kernel function K (Cortes and Vapnik, 1995). We then solve the quadratic program problem (9)–(11) to obtain the unique solution ${\alpha }_i^{(1)}(i=1,2,\dots ,M)$ and update w⁽¹⁾ with $w_1^{(1)}={\sum }_{i=1}^M{\alpha }_i^{(1)}{y}_i{\varnothing }_1(Y_i^{(1)})$. Second, set $P_1\left(w^{(1)}\right)={\sum }_{i=1}^M{\alpha }_i{y}_i{Y}_i^{(1)}$ and $Y_i^{(2)}=X_i{\prod }_{1\le l\le N}^{l\ne 2}{\times }_l{P}_l(w^{(l)}),$ i = 1,2, … ,M with the newest P₁(w⁽¹⁾) .Input P_l(w^(l)) 1 ≤ l ≤ N, l ≠ 2 to the optimization problem (5)–(7) to obtain ${\alpha }_i^{(2)}(i=1,2,\dots ,M)$ by solving its dual problem (9)–(11). Update w⁽²⁾ with $w_1^{(2)}={\sum }_{i=1}^M{\alpha }_i{y}_i{\varnothing }_2(Y_i^{(2)})$. We repeat the second steps above until we obtain the last unique solution ${\alpha }_i^{(N)}(i=1,2,\dots ,M)$ and update w^(N) with $w_1^{(N)}={\sum }_{i=1}^M{\alpha }_i{y}_i{\varnothing }_N(Y_i^{(N)})$ in the first iteration. We can then obtain the weight vectors $\{w_1^{(n)}|n=1,2,\cdots ,N\}$ in the first iteration. For the second iteration, we repeat these steps in the first iteration by using $\{P_n(w_1^{(n)})|n=1,2,\cdots ,N\}$ as fixed weight vectors, and we obtain weight vectors $\{w_2^{(n)}|n=1,2,\cdots ,N\}$ in the second iteration. As we continue the iterations process, a sequence of weight vectors $\{w_t^{(n)}|n=1,2,\cdots ,N{\}}_{t=1}^{\infty }$ is obtained, where t is the iterative number. Since (9)–(11) is a quadratic program problem ${\alpha }_i^{(n)}(i=1,2,\dots ,M)$ is the globally optimal solution to the corresponding maximum dual optimization problem (9)–(11) which is equivalent to the optimization problem (5)–(7). Thus $w_i^{(n)}$ is the globally optimal solution to the corresponding minimum optimization problem. Thus,

$$\begin{array}{c}J\left(w_1^{(1)},w_0^{(2)},w_0^{(3)}\cdots ,w_0^{(N)}\right)\ge J\left(w_1^{(1)},w_1^{(2)},w_0^{(3)}\cdots ,w_0^{(N)}\right)\ge \cdots \ge J\left(w_1^{(1)},w_1^{(2)},w_1^{(3)}\cdots ,w_1^{(N)}\right)\ge \\ J\left(w_2^{(1)},w_1^{(2)},w_1^{(2)}\cdots ,w_1^{(N)}\right)\ge J\left(w_2^{(1)},w_2^{(2)},w_1^{(3)}\cdots ,w_1^{(N)}\right)\ge \cdots \ge J\left(w_2^{(1)},w_2^{(2)},w_2^{(3)}\cdots ,w_2^{(N)}\right)\ge \\ \cdots \cdots \ge \\ J\left(w_t^{(1)},w_{t-1}^{(2)},w_{t-1}^{(3)}\cdots ,w_{t-1}^{(N)}\right)\ge J\left(w_t^{(1)},w_t^{(2)},w_{t-1}^{(3)}\cdots ,w_{t-1}^{(N)}\right)\ge \cdots \ge J\left(w_t^{(1)},w_t^{(2)},w_t^{(3)}\cdots ,w_t^{(N)}\right)\ge \\ \cdots \cdots \ge 0.\end{array}$$

Therefore, $\{J\left(w_t^{(1)},w_t^{(2)},w_t^{(3)}\cdots ,w_t^{(N)}\right){\}}_{t=1}^{\infty }$ is a monotonically decreasing sequence with down boundary, and therefore converges. #

The change in function values of the left formula in equation (13) was plotted with different iteration numbers for PET and CT, in Fig. 3 (a) and (b), respectively. The function value decreased rapidly to zero within a few iterations, illustrating the fast convergence of the STM iterative algorithm (Fig. 3).

Fig. 3.

The function value of the left formula in (13) with respect to different iterations: (a) for PET tumor imaging with SUV and (b) for original CT tumor imaging.

2.4. Three dimensional data balancing

Class imbalance had to be addressed since distant failure occurred in a minority of cases. The Synthetic Minority Over-sample Technique (SMOTE) (Chawla et al., 2002) is a common method to solve the class imbalance problem. It can augment the decision region of the minority class by generating a synthetic sample based on minority class information. Typically, SMOTE is produced in the vector space using the K-nearest neighborhood (KNN) graph based on Euclidean distance. However, because we used 3D imaging as input for the STM-based algorithm, the conventional SOMTE is no longer applicable. Thus, we propose a high order SMOTE by using the KNN graph in the tensor space based on the tensor distance defined as follows:

$$\text{d}(\text{X},\text{Y})=\Vert \text{X}-\text{Y}{\Vert }_F=\sqrt{{\sum }_{i_1=1}^{I_1}{\sum }_{i_2=1}^{I_2}\cdots {\sum }_{i_N=1}^{I_N}{(x_{i_1,i_2,\cdots ,i_N}-y_{i_1,i_2,\cdots ,i_N})}^2}.$$ (17)

This high order SMOTE can generate synthetic tensor samples, but not vector samples, based on minority class information.

3. Experimental Setup

3.1. Workflow of the KSTM-based classifier

The overall workflow of the KSTM-based classifier is illustrated in Fig. 4. SUVs are first calculated for PET. For each patient, the primary tumors are segmented semi-automatically on both pre-treatment CT and PET. Three-dimensional tumor tensors for each imaging modality are constructed and used as input for the KSTM-based classifier. The 3D spline interpolation is used to equalize the resolution and the 3D Gabor filter is applied to filter the noise. High order SMOTE is used to balance the training samples with tensor structure. A KSTM iterative algorithm is used to train the weight vectors for every mode of the tensor for the classifier.

Fig. 4.

Workflow of the KSTM-based classifier.

3.2. Multi-modality fusion for PET and CT

As PET and CT measure different intrinsic characteristics of a tumor, the combination of information learned from PET and CT could improve the predictive performance. In this work, we combined the output scores (i.e. the binary probabilities) of the KSTM-based classifier for PET (KSTM-PET) and the KSTM-based classifier CT (KSTM-CT) using a reliable classifier fusion (RCF) strategy (RCF-PET-CT) (Zhiguo Zhou, 2017). The RCF strategy does not only consider the relative importance of output scores from different modalities, but also takes the reliability of output scores from different modalities to get more reliable fusion results.

3.3. Methods for comparative testing

The comparative methods include three conventional machine learning methods (SVM, SVM-PCA and LSTM) and three conventional radiomics methods (SVM-ReliefF, SVM-SFS and RF-WLCX).

1). SVM:

As the basic model of STM, SVM with vectorization of the tumor image as input for modeling was compared with our proposed algorithm. Performance was improved using the 3D spline interpolation to equalize the resolution and the 3D Gabor filtering to denoise on both PET and CT imaging. We used vectorization of the tumor image after 3D spline interpolation and 3D Gabor filtering as input for the SVM-based classifier with radial basis function (RBF) kernel.

2). SVM-PCA:

Considering the high dimensionality of the image vectorization, principal component analysis (PCA) (Abdi and Williams, 2010) was used to reduce sample dimensionality and extract the principal components. The number of the principal components was selected using the smallest size with a variance contribution rate of 100% (PET: 44; CT: 47). The principal components extracted by PCA from vectorization of the tumor image with 3D spline interpolation and 3D Gabor filtering were used as input for the SVM-based classifier with radial basis function (RBF) kernel.

3). LSTM:

Linear support tensor machine constructs the classification model in original data tensor space. It is a simplified form of KSTM, thus it is compared with the proposed KSTM algorithm. The tensors after 3D spline interpolation and 3D Gabor filtering are used as the input for modeling.

4). SVM-ReliefF:

Conventional radiomics methods referring to handcraft-based extraction and analysis of high-throughput quantitative features mainly include three steps: 1) feature extraction; 2) feature selection; 3) classification. In this method, 240 texture features were extracted based on the Grey Level Concurrence Matrix (GLCM) (Davis et al., 1979). In particular, for each distance from {1, 2, 4, 8} and each gray level from {8, 16, 32, 64, 128}, twelve texture features (Zhou et al., 2017) were calculated based on the averaging GLCM at thirteen directions. Nine intensity features based on the intensity histogram, and eight geometry features associated with tumors were also extracted. Furthermrore, 257 wavelet features were calculated using a coiflet transformation (Parmar et al., 2015), which were the transformed domain representation of the intensity and textural features. A total of 514 radiomic features were used in this analysis. The classical feature selection method ReliefF (Kononenko et al., 1997) was used to select the optimal feature subset based on the training dataset. The selected feature subset on the training dataset was used as the input for the SVM model with a RBF kernel to train the classification model by the 10-fold cross validation method. The selected feature subset on the testing dataset was used to independently evaluate the performance of the trained classification model.

5). SVM-SFS:

As the traditional feature selection method, the sequential forward feature selection (SFS) method (Kohavi and John, 1997) coupled with the SVM classifier using the above 514 handcrafted features as input was also compared with our proposed algorithm. SFS with the minimal classification error as criterion based on the quadratic discrimination was used to select the optimal feature subsets with 10-fold cross validation method in the training dataset. We obtained 20 optimal feature subsets after the 20 times of feature selection process. The frequency of each feature appearing in these 20 subsets was then calculated and sorted decreasingly. Let {p_iǀi = 1,2, ··· , n} denote the set consisting of such frequencies. Subsequently, new feature subsets are constructed by collecting the features with a frequency higher than the value p_i(i = 1,2, ··· , n) in the training dataset. Then, n new feature subsets and n groups of the predictive results can be obtained by using the new feature subset as input for the SVM-based classifier with the10-fold cross validation method. The best results are selected from the n groups of predictive results, and the corresponding optimal features subset is obtained, which was also used as the optimal features subset in the independent testing.

6). RF-WLCX:

The 514 features extracted as above (SVM-ReliefF) was used here. Parmar et al. examined twelve classification methods and fourteen feature selection methods (Parmar et al., 2015). They found that the Wilcoxon test based feature method (WLCX) and the classification method random forest (RF) had the highest prognostic performance. Thus we used WLCX to rank the importance for each feature. Subsequently, we selected the feature subset consisting of the top important features with different number from {5, 10, ··· , 100} and obtain 20 feature subsets. The RF classifier was used to perform the classification with the selected feature subset as input in the training dataset by the 10-fold cross validation method. Then 20 results can be obtained for 20 selected feature subsets. The best results were selected from the 20 results and the corresponding optimal features subset is obtained, which was also used as the optimal features subset in independent testing.

3.4. Experimental settings

Firstly, 33 patients (30% over all participants) were selected randomly as an independent testing dataset, in which 7 participants experienced distant failures. The remaining patients were used for model training and validation, in which 18 patients experienced distant failure. A 10-fold cross validation method was employed to evaluate performance in the training dataset. All patients in the training dataset were randomly partitioned into 10 subsets with a size either 7 or 8. For each experimental round in the training processing, one subset was designated as the testing data and the remaining subsets as the training data. This process was repeated 5 times to avoid any bias introduced by random data partitioning. The optimal parameter group was selected and the corresponding classification results were obtained by averaging the results 5 times. An independent testing method was used to evaluate performance in the testing dataset with the model trained by the training dataset.

For the KSTM and SVM algorithms, the trade-off parameter C was chosen from {2⁻²⁰, 2⁻¹⁹ _, ···, 2²⁰} and the free parameter in the RBF kernel was also chosen from {2⁻²⁰, 2⁻¹⁹, ··· , 2²⁰}. For the LSTM algorithm, the trade-off parameter was also chosen from {2⁻²⁰, 2⁻¹⁹, ··· , 2²⁰}. For the random forest algorithm, the number of tree was chosen from {250, 255, 260, ··· , 600}. The threshold in the stopping condition (13) was empirically set to10⁻².

We adopted four metrics as evaluation criteria, including the area under the receive operating characteristic curve (AUC) (Fletcher et al., 2012), classification accuracy (ACC), sensitivity (SEN), and specificity (SPE). Let TP, TN, FP, and FN indicate true positive rate, true negative rate, false positive rate, and false negative rate, respectively. These evaluation metrics are defined as ACC = (TP + TN)/(TP + TN + FP + FN), SEN = TP/(TP + FN), SPE = TN/(TN + FP).

4. Results

4.1. Performance with 3D interpolation and 3D Gabor filtering

We use the results of 10-fold cross validation in training dataset to evaluate the performance with 3D interpolation and 3D Gabor filtering. The performance is presented in Fig. 5. The KSTM-based classifier coupled with 3D interpolation and 3D Gabor filtering (KSTM-IN3-Gabor) consistently outperformed the KSTM-based classifier without interpolation and Gabor filtering, and the KSTM-based classifier coupled with 3D interpolation (KSTM-IN3) for both PET and CT. KSTM-IN3 consistently outperformed KSTM except for sensitivity in PET. The corresponding ROC curves for the KSTM-based classifier are shown with and without 3D spline interpolation and 3D Gabor filtering in a one-time 10-fold cross validation experiment (Fig. 6). Also, KSTM-IN3-Gabor outperformed KSTM-IN3, and KSTM-IN3 outperformed KSTM. Thus, the tensors with 3D spline interpolation and 3D Gabor filtering were used as the input for the KSTM algorithm.

Fig. 5.

Performance of the STM-based classifier with 3D spline interpolation and 3D Gabor filtering: a) performance for PET imaging; b) performance for CT imaging.

Fig. 6.

ROC curves for the STM-based classifier with and without 3D spline interpolation and 3D Gabor filtering: (a) PET imaging and (b) CT imaging.

4.2. Comparison with conventional methods

The results of the comparative testing of seven classification algorithms are shown in table 2 for PET imaging and table 3 for CT imaging. For PET in 10-fold cross validation on training dataset, the KSTM-based predictive algorithm achieved the highest AUC (0.81), accuracy (76.10%), sensitivity (78.89%) scores among the seven methods investigated in this study, although its specificity score (75.25%) was slightly lower. For PET on the independent testing dataset, the KSTM-based predictive algorithm also achieved the highest AUC (0.74), accuracy (75.76%) and sensitivity (71.43%) scores, although specificity score (76.92%) was lower than LSTM that had imbalanced sensitivity (42.86%) and specificity (80.77%). For CT, the KSTM-based predictive algorithm also achieved the highest AUC (0.81), accuracy (78.44%), sensitivity (77.78%) scores in 10-fold cross validation on training dataset as well as the highest AUC (0.80), accuracy (75.76%), sensitivity (71.43%) and specificity (76.92%) scores in independent dataset among the seven methods investigated in this study.

Table 2:

Performance of different predictive models for PET.

Methods	10-fold cross validation				Independent testing
	AUC	ACC (%)	SEN (%)	SPE (%)	AUC	ACC (%)	SEN %)	SPE
SVM	0.72±0.03	73.51±3.96	64.44±6.33	76.27±6.23	0.58	60.61	42.86	65.38
SVM-PCA	0.70±0.06	72.47±3.94	58.89±16.94	76.61±9.08	0.61	66.67	71.43	65.38
LSTM	0.74±0.03	73.77±3.10	62.22±11.39	77.29±4.58	0.72	72.73	42.86	80.77
SVM-ReliefF	0.79±0.01	73.25±4.91	76.67±7.24	72.20±6.74	0.73	69.70	71.43	69.23
SVM-SFS	0.77±0.03	71.43±3.67	72.22±7.86	71.19±7.19	0.69	72.73	71.43	73.08
RF-WLCX	0.65±0.06	66.49±1.93	55.56±8.78	69.83±3.26	0.64	63.64	71.43	61.54
KSTM	0.81±0.02	76.10±3.51	78.89±5.69	75.25±7.45	0.74	75.76	71.43	76.92

Table 3:

Performance of different predictive models for CT.

Methods	10-fold cross validation				Independent testing
	AUC	ACC (%)	SEN (%)	SPE (%)	AUC	ACC (%)	SEN (%)	SPE (%)
SVM	0.75±0.05	77.92±3.67	71.11±12.67	80.00±7.89	0.60	57.58	42.86	61.54
SVM-PCA	0.75±0.05	76.36±2.13	67.78±6.09	78.98±4.42	0.69	72.73	57.14	76.92
LSTM	0.69±0.08	64.94±4.11	52.22±16.48	68.81±8.09	0.62	66.67	57.14	69.23
SVM- ReliefF	0.76±0.04	70.65±3.51	73.33±4.65	69.83±4.65	0.67	72.73	71.43	73.08
SVM-SFS	0.74±0.03	75.84±2.69	64.44±6.33	79.32±4.70	0.71	66.67	71.43	65.38
RF-WLCX	0.63±0.01	65.45±2.17	52.22±7.45	69.46±3.17	0.64	69.70	57.14	73.08
KSTM	0.81±0.03	78.44±3.63	77.78±7.86	78.64±6.95	0.80	75.76	71.43	76.92

The corresponding ROC curves for the seven predictive algorithms are shown in a one-time 10-fold cross validation experiment in training dataset and a one-time independent testing experiment in testing dataset (PET: Fig. 7; CT: Fig. 8), indicating that the proposed KSTM-based algorithm coupled with 3D spline interpolation and 3D Gabor filtering outperformed the other six methods.

Fig. 7.

ROC curves for seven compared predictive algorithms for PET imaging: (a) 10-fold cross validation and (b) independent validation.

Fig. 8.

ROC curves for seven compared predictive algorithms for CT imaging: (a) 10-fold cross validation and (b) independent validation.

We used the unpaired t-test with a 95% confidence interval to assess whether the difference in performance between the KSTM-based algorithm and each competing method was statistically significant in 10-fold cross validation. The corresponding p-values are reported in Table 4 for training results. These results show that our KSTM-based algorithm is significantly better than the other methods, as shown by the p-values < 0.05.

Table 4:

The p-values in the unpaired T-test between the performances of the KSTM based classifier and each competing method.

	10-fold cross validation
Method	PET	CT
SVM	< 0.0001	0.0434
SVM-PCA	0.0028	0.0496
LSTM	0.0011	0.0016
SVM-ReliefF	0.0487	0.0294
SVM-SFS	0.0234	0.0032
RF-WLCX	< 0.0001	< 0.0001

4.3. Multi-modality fusion of PET and CT based on RCF

The results of the fusion of PET and CT based on RCF strategy (RCF-PET-CT) on the independent testing dataset were summarized in Table 5. Compared with the results from KSTM-based classifier using PET or CT imaging only as the input, RCF-PET-CT obtained higher AUC (0.84), accuracy (84.85%) and specificity (88.46%) and the same sensitivity (71.43%). The ROC curves of the KSTM- PET and KSTM-CT, and RCF-PET-CT shown in Fig. 9 also demonstrate that the RCF-PET-CT had a better performance.

Table 5:

The results of the multi-modality fusion of PET and CT based on RCF.

Method	AUC	ACC (%)	SEN (%)	SPE (%)
RCF-PET-CT	0.84	84.85	71.43	88.46

Fig. 9.

ROC curves of the KSTM-based classifier for PET and CT, and the multi-modality fusion of PET and CT based on RCF.

5. Discussion

We evaluated an KSTM-based classifier that uses 3D tumor images as input to predict distant failure in early stage NSCLC patients treated with SBRT. The experimental results showed that the KSTM-based predictive algorithm outperforms traditional machine learning approaches and conventional radiomics approaches. It achieved the following better metrics on CT than PET for AUC, accuracy, sensitivity, and specificity: 0.81, 78.44%, 77.78%, and 78.64% in 10-fold cross validation, respectively. It also achieved the following better metrics on CT than PET for AUC, accuracy, sensitivity, and specificity: 0.80, 75.76%, 71.43%, 76.92% for CT in the independent testing, respectively. These results have implications for patients with early stage NSCLC who might benefit from the addition of systemic therapy based on a calculated increased risk for distant failure. These short-term predictive models will be useful for stratifying patients for risk-adapted prospective therapeutic protocols.

The proposed model can fully characterize the tumor properties through tensor structure, potentially overcoming the limitations of conventional methods that rely on handcrafted features. The KSTM-based classifier takes full advantage of the image information from whole images without using handcrafted features. Additionally, for conventional radiomics approaches, the feature selection is needed to reduce the redundancy of handcrafted features. However, the same feature selection method coupled with different classifiers or the same classifier coupled with different feature selection methods often yield different selection results, affecting the robustness of the classification results. Therefore, the KSTM-based algorithm is more robust than conventional radiomics approaches.

Compared with the voxel/pixel-based approaches that are restricted to one-dimensional vectors unfolded from tensors as input for modeling, the KSTM-based classifier learns fewer parameters compared to the SVM-based models. The KSTM model involves a reduced number of parameters for optimization. SVM with vectorization of the image as input needs learning $\left({\prod }_{n=1}^N{I}_n+M+1\right)$ parameters for the training samples mentioned in subsection 2.3. However, the STM-based classifier with the high order image tensor as input only needs learning $\left({\sum }_{n=1}^N{I}_n+M+1\right)$ parameters, which are less than $\left({\prod }_{n=1}^N{I}_n+M+1\right)$, the number of parameters learned in the SVM model. Additionally, the weight vector dimensionality in the KSTM iterative algorithm (13)–(15) is I_n (n = 1,2, ··· , N ) , which is lower than the dimensionality ${\prod }_{n=1}^N{I}_n$ of the SVM weight vector with vectorization of the image as input. For example, if the original tensor space is projected to the feature tensor space with the same dimensionality, the number of parameters is calculated as55 + 56 + 80 + 48 + 1 = 239 and the dimensionality of the weight vectors in the STM iterative process is55, 56, 80 in the KSTM-IN3-Gabor algorithm for CT imaging (the tensor size is55 × 56× 80, as seen in section 2.2). The number of parameters is 55 * 56 * 80 + 48 + 1= 246449 and the dimensionality of the weight vector is 55 * 56 * 80 + 246400 in the algorithm SVM-IN3-Gabor for CT imaging.

The KSTM-based classifier holds all imaging information without dimensionality reduction usually needed for the voxel/pixel based approaches. Meanwhile, the KSTM-based classifier maintains the natural 3D geometry structure of the image and correlation in the original image. Additionally, the KSTM based algorithm trains the classification model on the adaptive feature tensor space based on the kernel function. It improves the performance compared to the LSTM based algorithm that trains the classification model on original tensor space. Although only 3D KSTM based algorithm is used in this study, the proposed algorithm can solve the classification problem with any dimensionality. The KSTM based algorithm is applicable to the dataset with small scale. This is in contrast to feature learning method such as deep learning, which in general requires a large scale dataset to learn features effectively for classification.

In this study, the tensors as the input of the proposed algorithm were constructed based on the tumor ROI segmented using a semi-automatic way. As the KSTM does not require the calculation of texture or geometric features explicitly, we expect the segmentation accuracy of tumor will not dramatically affect the results as compared to the standard radiomic approaches. On the other hand, even though KSTM uses the original image (after segmentation) as the input, it does not involve auto-feature extraction process that is typically involved in convolutional neural network. Therefore, a large volume that involves too much background (e.g., whole lung slice) will adversely affect the performance of KSTM. Nevertheless, a region of interest (ROI) covering the tumor with minimal involvement of background can be potentially used as the input in KSTM. The influence of segmentation as well as input ROI is worthy of investigation in a future study.

The 3D spline interpolation was used to obtain the tensor with the same solution along different directions and the three-dimensional Gabor filter banks were used to denoise the images in this work. Experimental results in Figures 5 and 6 demonstrate that both 3D interpolation and noise filtering can improve the performance of KSTM. However, different interpolations schemes and noise reduction filters may potentially affect the performance of KSTM, which requires further investigation.

It is noted that pre-treatment PET and CT images were acquired at different hospitals even though patients received SBRT treatments from a single institute. Thus acquisition protocol variations exist among images we used in this study. As shown in many radiomic approaches, some radiomic features can be sensitive to scanning protocols. The KSTM method used in this work does not require the calculation of features explicitly, thus it may be less sensitive to imaging protocol variation as compared to other radiomics approaches that rely on extracted quantitative features. Nevertheless, the performance of KTSM may be still affected by scanning protocol or acquisition parameter variations as reconstructed PET/CT images are used as the input for KTSM. However, due to a relatively small patient size we currently have, we are not able to quantify the influence of PET and CT scanning protocols to the performance of KSTM by further grouping patients according to scanning protocols. If the scanning protocols and acquisition parameters are harmonized for PET and CT, the performance of KSTM could be further improved.

The proposed KSTM-based classifier uses the KSTM iterative procedure (Table 1) to train the weight vectors for every mode of the tensor for the classifier. We proved the convergence of the KSTM iterative procedure, although the solution may not necessarily be optimal globally. Because the optimization problem defined by equations (2)–(4) is not convex in the linear space, a closed form solution by the linear algebra theory is unavailable. The convex optimization theory (Li et al., 2009) on Riemannian manifolds may be used to study the closed form solution by constructing a Riemannian manifold and proving that the optimization problem defined by equations (2)–(4) is convex in this constructed Riemannian manifold. The key step would be the definition of the Riemannian metric in the feasible solutions set as $\mathcal{W}=\{\{w^{(1)},w^{(2)},\cdots ,w^{(N)}\}|w^{(n)}\in {ℝ}^{I_n},n=1,2,\cdots ,N\}.$

While the number of patients used in this work is comparable to a similar study (Wu et al., 2016), it’s still relative small as compared to some other radiomic studies. Thus large steps in the ROC curves are observed due to the limited number of patients in both training and testing dataset. For the same reason, there is a possibility that overfitting may happen in the model training. However, our results indicate that for many models (e.g., LSTM and RF-WLCX using PET, and SVM-SFS, RF-WLCX and KSTM models using CT), the performance was comparable between training and testing datasets. The presented KSTM model performed fairly well on the test dataset, achieving AUCs of 0.74, 0.80 and 0.84 for PET, CT and fused PET-CT model, respectively. Furthermore, the results on the independent dataset demonstrate the KSTM model outperforms all other comparative methods investigated in this work. However, a larger independent patient cohort preferably including patients from other institutions is needed in order to validate the performance of KSTM.

Highlights:

• A kernelled support tensor machine (KSTM) is proposed to assess SBRT for NSCLC patients.

• The KSTM based model uses native tumor imaging as input.

• The KSTM trains the classification hyper-plane in an adaptive feature tensor space.

• An iterative procedure with the proof of convergence for KSTM is developed.

• The KSTM achieved the highest prediction results among the seven investigated methods.