ARMO: Automated and reliable multi-objective model for lymph node metastasis prediction in head & neck cancer

Abstract

Objective:

Accurate diagnosis of lymph node metastasis (LNM) is critical in treatment management for patients with head & neck cancer. Positron emission tomography (PET) and computed tomography (CT) are routinely used for identifying LNM status. However, for small or less fluorodeoxyglucose (FDG) avid nodes, there are always uncertainties in LNM diagnosis. We are aiming to develop a reliable prediction model is for identifying LNM.

Approach:

In this study, a new automated and reliable multi-objective learning model (ARMO) is proposed. In ARMO, a multi-objective model is introduced to obtain balanced sensitivity and specificity. Meanwhile, confidence is calibrated by introducing individual reliability, whilst the model uncertainty is estimated by a newly defined overall reliability in ARMO. In the training stage, a Pareto-optimal model set is generated. Then all the Pareto-optimal models are used, and a reliable fusion strategy that introduces individual reliability is developed for calibrating the confidence of each output. The overall reliability is calculated to estimate the model uncertainty for each test sample.

Main Results:

The experimental results demonstrated that ARMO obtained more promising results, which the area under the curve (AUC), accuracy, sensitivity and specificity can achieve 0.97, 0.93, 0.88 and 0.94, respectively. Meanwhile, based on calibrated confidence and overall reliability, clinicians could pay particular attention to highly uncertain predictions.

Significance:

In this study, we developed a unified model that can achieve balanced prediction, confidence calibration and uncertainty estimation simultaneously. The experimental results demonstrated that ARMO can obtain accurate and reliable prediction performance.

1. Introduction

Head & neck (H&N) cancer is one of most common sites worldwide (Cognetti et al., 2008). Lymph node metastasis (LNM) has been an important diagnostic factor because it may decrease survival time and may lead to distant metastasis (Pantel and Brakenhoff, 2004). Radiation therapy has become a standard way to control locoregional failure (Moore et al., 2005). Accurately identifying malignant lymph nodes (LNs) plays a vital role in H&N cancer management (Chen et al., 2019). Computed tomography (CT) and fluorodeoxyglucose (FDG) positron emission tomography (PET) are routinely used for evaluating LNM status in clinical practice. However, uncertainties exist when identifying LNM based on PET&CT, especially for evaluating small or less FDG-avid LNs. This is further compounded when factoring a physician’s years of experience (Zhou et al., 2018a). As such, building a reliable LNM prediction model to overcome the above uncertainties is needed.

In prediction modeling, the intrinsic uncertainty associated with PET/CT-based prediction can be handled by obtaining an actual probability for the correct prediction, termed confidence calibration (Zhang et al., 2019) which is defined as the ability of a model to provide an accurate probability of correctness for any of its predictions. For example, a LN predicted with a confidence of 85% should have a chance of 85% being correctly predicted (Carneiro et al., 2020), and the model output probability should be close to 0.85. Meanwhile, the uncertainty associated with each model (mimicking a physician’s experience) be handled by estimating the uncertainty of model output to measure the reliability of a particular prediction result for a given sample. This is termed as uncertainty estimation which is defined as estimating the degree of confidence for each model output given the available data and the inherent variability of the model. In other words, a “third-party” evaluation (e.g., another model) can be performed to measure how reliable the prediction result is, and clinicians can determine whether they should re-evaluate the model output with high uncertainty. Furthermore, building a model with balanced sensitivity and specificity is also desired for LNM prediction (Zhou et al., 2017a), as an imbalanced model is unacceptable, even when obtaining high accuracy. Developing a prediction model that integrates balanced sensitivity and specificity, confidence calibration and uncertainty estimation into a unified framework is desirable.

We have previously developed an automated multi-objective model (AutoMO) (Zhou et al., 2019) to improve model performance with balanced sensitivity and specificity. In AutoMO, sensitivity and specificity are simultaneously considered as objective functions to generate the Pareto-optimal model set in the training stage. All the Pareto-optimal models are used in the testing stage, and their output probabilities are fused through an evidential reasoning (ER) approach (Wang et al., 2006). Compared with traditional multi-objective and single-objective models(Zhou et al., 2017a), experimental results showed that AutoMO could obtain better performance and more balanced sensitivity and specificity.

However, confidence calibration and uncertainty estimation (Gawlikowski et al., 2021) have not been investigated in AutoMO, and uncertainty estimation plays an important role in building a safe model. As such, a new automated and reliable multi-objective (ARMO) model for LNM prediction in H&N cancer is proposed in this study, which unifies balanced sensitivity and specificity, confidence calibration and uncertainty estimation into one framework. Moreover, individual reliability which measures the effect of local information from a given test sample on each Pareto-optimal model is calculated, and the confidence (final fused output probability) is calibrated when integrating the individual reliability during the fusion process. As most of available fusion strategies (Meng et al., 2020) only rely on the weight or relative importance of different models without considering the reliability of the output from individual model, they cannot be readily used for reliable fusion in ARMO. Since the evidential reasoning rule (ER²) (Yang and Xu, 2013) is a more generalized probabilistic inference engine that integrates the reliability into the evidence combination, it is adopted for model fusion in our study. The proposed ER²–based reliable fusion strategy does not only take the relative importance of different Pareto-optimal models into account, but also considers the individual reliability of each model output for a test sample. Furthermore, the overall reliability is calculated based on each model's individual reliability to estimate the uncertainty of each test sample.

Our contribution to the proposed ARMO framework is summarized as follows: (1) we develop a general model which integrates balanced sensitivity and specificity confidence calibration, and uncertainty estimation into a unified framework; (2) we define a new individual reliability to calibrate confidence; (3) we define a new overall reliability to estimate the uncertainty of model output; and (4) ARMO achieves promising results for LNM prediction in H&N cancer.

2. Related Work

2.1. Lymph node metastasis prediction

Several deep learning-based models have been investigated that attempt to identify LNM. In one study (Zhou et al., 2018a), a 3D CNN was combined with radiomics to identify LNM in H&N cancer based on PET and CT. In another study, ultrasound images were used to predict LNM in breast cancer through three different deep learning models (Zhou et al., 2020a). A multiple instances learning with deep graph convolution model was developed for LNM prediction on histopathological images (Zhao et al., 2020b) in colorectal cancer. To predict LNM in rectal cancer, a faster region-based convolutional neural network (Faster R-CNN) was employed to build a prediction model (Ding et al., 2019). To predict LNM in lung cancer, a DensePriNet combining 3D DenseNet and prior clinical features was developed (Zhao et al., 2020a).

Meanwhile, several radiomics-based strategies have also been developed for LNM prediction. For example, radiomics was used to predict axillary LNM in breast cancer by using 2D and 3D features extracted from dynamic contrast-enhanced magnetic resonance imaging (Arefan et al., 2020). In another study, support vector machine was used to predict LNM in esophageal cancer treatment with neoadjuvant chemotherapy based on features extracted from pre-operative CT (Wang et al., 2017). Meanwhile, transfer learning was introduced into radiomics to build a LNM prediction model for papillary thyroid carcinoma using preoperative ultrasound (Yu et al., 2020). To predict axillary LNM in breast cancer, a mammography-based radiomics strategy was developed (Yang et al., 2019). A radiomics nomogram method was developed for predicting LNM in colorectal cancer based on preoperative CT (Huang et al., 2016).

2.2. Confidence calibration

Due to the overconfident prediction in most of the current deep learning methods, confidence calibration is becoming more important to avoid potential risks in high stake applications such as in the medical domain. Confidence calibration is used to obtain an accurate probability of correctness for the prediction (Mehrtash et al., 2020). Several methods have been developed for improving confidence. A widely used post-processing method termed temperature scaling calibrates the prediction probability by inserting a scaling parameter into softmax formulation and turning it into a validation set to “soften” the probabilities (Guo et al., 2017). In addition, a regularization term was also introduced into the confidence calibration, such as Mixup (Thulasidasan et al., 2019) and label smoothing (Müller et al., 2019). Mixup is a typical data augmentation method that randomly mixes two samples at both the image and label space, whilst label smoothing revises the true labels by fusing them with a uniform distribution. Bayesian inference was also explored for calibration. For example, Monte Carlo Dropout (Gal and Ghahramani, 2016) was applied in training and testing through dropout. In deep ensemble (Lakshminarayanan et al., 2017), the confidence can also be calibrated by averaging the output of each base deep learning model.

2.3. Uncertainty estimation

Uncertainty estimation plays an important role in evaluating the reliability of a model output (Dohopolski et al., 2020). There are generally four types of strategies in uncertainty estimation, including single deterministic methods, Bayesian methods, ensemble, and test-augmentation methods.

In single deterministic methods, the uncertainty can be directly obtained from the model itself or through external methods. For example, an evidential neural network [29] was developed by introducing the evidence theory into CNN to estimate the uncertainty directly. A commonly used external method builds two models simultaneously: one for actual prediction and the other for predicting the uncertainty of the first model’s prediction (Raghu et al., 2019).

There are three types of approaches in Bayesian methods for uncertainty estimation. The first approach is Variational inference, which approximates the posterior probability through optimizing over a family of tractable distributions (Barber and Bishop, 1998). A well-known method is Monte Carlo Dropout (Gal and Ghahramani, 2016). The second approach is based on sampling methods. Among the multiple sampling methods, Markov chain Monte Carlo (MCMC) sampling is commonly used in uncertainty estimation (Bishop and Nasrabadi, 2006). The third approach is based on Laplace approximation. The uncertainty is estimated by calculating the approximation of the Hessian matrices (Ritter et al., 2018).

A deep ensemble model refers to an ensemble of multiple deep neural networks. To deal with model uncertainty, the member networks were designed for two purposes: performing the prediction and obtain a value of data uncertainty of the prediction results (Lakshminarayanan et al., 2017). Furthermore, several enhanced versions were developed to build more efficient ensemble model, such as sub-ensembles (Valdenegro-Toro, 2019) and batch-ensembles (Wen et al., 2020).

The basic idea of test time augmentation is to create a couple of test samples for each test sample via data augmentation and perform testing on all the created samples to calculate the output probability for measuring uncertainty (Ayhan and Berens, 2018). This method has been widely used in medical image analysis (Wang et al., 2018) (Wang et al., 2019) such as cell segmentation (Moshkov et al., 2020).

3. Methods

3.1. Overview

The ARMO framework is illustrated in Fig. 1, which consists of training and testing stages. In training stage, the aim is to produce a Pareto-optimal model set by maximizing sensitivity- and specificity-based objective functions through a multi-objective optimization algorithm. “Pareto-optimal” solution is defined as a set of “non-inferior” solutions in the objective space that represents a boundary beyond which none of the objectives can be improved without sacrificing at least one of the other objectives (Chinchuluun and Pardalos, 2007). In this study, we produce a couple of prediction models which are non-dominated by one another. Therefore, the objective function is a vector other than a scaler value. Correspondingly, a number of Pareto-optimal solutions should be achieved instead of one single solution. Meanwhile, the weight for each Pareto-optimal model is determined from the training and validation sets. In testing stage, the lymph node features are fed into the models and the final calibrated confidence is obtained by integrating estimated individual reliability into reliable fusion. Furthermore, the overall reliability is measured through averaging individual reliability to estimate the uncertainty of the final output. The details of the proposed framework are described in the following subsections.

Fig. 1:

ARMO framework for LNM prediction. In training stage, the Pareto-optimal model set is generated after multi-objective optimization, whilst the calibrated output probability and overall reliability are obtained through ER² based fusion.

3.2. Training stage

In training stage, the features are extracted from lymph nodes in PET and CT images, respectively. Then a deep neural network (DNN) with multiple hidden layers is used for model construction. A multi-objective model which considers sensitivity $f_{sen}$ and specificity $f_{spe}$ as the objective functions simultaneously is built, they are:

$$f_{sen}=\frac{TP}{TP+FN},f_{spe}=\frac{TN}{TN+FP},$$ (1)

where $TP$ is the number of true positives, $TN$ is the number of true negatives, $FP$ is the number of false positives, and $FN$ is the number of false negatives. We aim to maximize the two objective functions simultaneously in ARMO, that is:

$$f={\max }_{\alpha ,\beta }(f_{sen},f_{spe}).$$ (2)

where $\alpha$ denotes model parameters and $\beta$ denotes all the selected features.

To solve the above multi-objective optimization problem in Eq. (2), we used an iterative multi-objective immune algorithm (IMIA) (Zhou et al., 2017b). IMIA is a type of multi-objective evolutionary algorithm which has shown superior performance for multi-objective optimization (Zhou et al., 2016; Zhou et al., 2015). The IMIA procedure is summarized below and detailed implementation has been described previously (Zhou et al., 2017b).

1) Initialization. A hybrid initialization is performed, and the parameters are randomly initialized. Assume that the solution set size is $P$ and the maximal generation is $G$.

2) Clonal operation. Copy the solution with a larger crowding-distance several times to keep the best solutions.

3) Mutation operation. Perform the static mutation operation to generate better solutions.

4) Deleting operation. Because a newly generated solution set may include the same solutions (same $f_{sen}$ and $f_{spe}$), the unique one should be kept to increase the diversity of search space by avoiding duplicate solutions.

5) Updating the solution set. Select $P$ solutions to keep the size of the solution set, using the utility-based fast non-dominated sorting approach.

6) Termination. The algorithm ends once it reaches the maximal generation $G$; Otherwise, go to step 2).

After training, the Pareto-optimal models denoted by $F=\{F_1,\cdots ,F_N\}c$ are generated, where $N$ is the number of Pareto-optimal models.

3.3. Weight

After generating the Pareto-optimal model set, the relative weight of each model which measures the relative importance among the Pareto-optimal models, should be measured as well. This is necessary because each Pareto-optimal model is unique and may exhibit varying levels of accuracy and balance performance between sensitivity and specificity. Assume the weight is $w=\{w_1,\cdots ,w_L\}$, which satisfies the following constraints:

$$0\le w_l\le 1\text{and}{\sum }_{l=1}^N{w}_l=1,$$ (3)

The output probability for each Pareto-optimal model is denoted by $p=\{p_l^1,p_l^2,l=1,\cdots ,N\}$, where $p_l^1$ and $p_l^2$ indicate the output probability, respectively and $p_l^1+p_l^2=1$. Because we are aiming to obtain the balanced results between sensitivity and specificity, a model with a well-balanced two-objective function values (i.e., sensitivity and specificity) has non-zero weights, while the other models have zero weights. Furthermore, since AUC is a good measurement of model performance, it is taken as the second factor in determining the weight, which is denoted by $AU{C}_{val}$. In summary, the weight $w_l$ for a model with test sample $x$ is calculated as:

$$w_l\{\begin{array}{cc}\frac{\lambda \frac{f_{sim\_sen}^l}{s_{sim\_spe}^l}+(1-\lambda ){AUC}_{val}^l}{2}\hfill & when0.5\le \frac{f_{{sim}_{sen}}^l}{f_{{sim}_{spe}}^i}\le 1\hfill \\ \frac{\lambda \frac{f_{sim\_spe}^l}{s_{sim\_sen}^l}+(1-\lambda ){AUC}_{val}^l}{2}\hfill & when0.5\le \frac{f_{{sim}_{spe}}^l}{f_{{sim}_{sen}}^i}\le 1\hfill \\ 0\hfill & Othersituation\hfill \end{array}\phantom{\}}.$$ (4)

where $\lambda$ is weight vector to indicate the importance of two parts in Eq. (4), and $0\le \lambda \le 1$.

3.4. Individual reliability

In testing stage, the selected features are fed into the trained Pareto-optimal DNN models and the probability output for each model can be obtained. Then individual reliability for each model is estimated for calibrating confidence during the reliable fusion. As the concept of individual reliability is introduced for the first time in this work, its definition is given below.

Definition 1 (Individual reliability). The individual reliability for a model is defined as the similarity between the individual model’s output probability and other model’s output probabilities, which satisfies the following conditions:

$$r_i=\{\begin{array}{c}0when{l}_i\ne l_j;j=1,\cdots ,N,j\ne i\hfill \\ 1when{l}_i=l_j\wedge p_{l_j}=1;j=1,\cdots ,N,j\ne i\hfill \\ 0<r_i<1inothersituations\hfill \end{array}\phantom{\}},$$ (5)

where $r_i$, $i=1,\cdots ,N$ represents the reliability, and $l_i$ represents the predicted label and $p_{l_j}$ represents the probability output. Based on equation (5), when all the labels of other individual models are different from $l_i$, reliability $r_i$ is defined as 0. When all the labels of other models are the same as $l_i$ and the corresponding output probabilities are 1, reliability $r_i$ is defined as 1. Any other situation between these two extreme situations will have a reliability $0<r_i<1$.

The definition of reliability indicates the degree to which an individual model is reliable according to the output probability of other models. It depends on whether its output probability is close to the output probabilities of other models. The formula for calculating the reliability is given as follows.

We first define the dissimilarity $D_i$ between $C_i$ and $C_j$, $j=1,\cdots ,N$, $j\ne i$ as:

$$D_i={\prod }_{j=1,j\ne i}^N\left(1-p_j\right),i=1,\cdots ,N,$$ (6)

where $p_j$ is the output probability of model $C_i$. The similarity $S_i$ is calculated as:

$$S_i=1-D_i=1-({\prod }_{j=1,j\ne i}^N(1-p_j)),i=1,\cdots ,N,$$ (7)

Assuming that $SL$ is the number of models ($SL\le L\text{-}1$) that have the same labels with $C_i$, the reliability is calculated as:

$$r_i(x)=\frac{SL}{N-1}\cdot S_i=\frac{SL}{N-1}\cdot (1-({\prod }_{j=1,j\ne i}^N(1-p_j))),i=1,\cdots ,N.$$ (8)

In the following, we will prove our reliability calculation method can meet the definition.

Lemma 1. The reliability calculation formula meets Definition 1.

Proof.: For model $C_i$:

Situation 1: when all the labels of other classifiers are different from $l_i$, $SL=0$. According to Eq. (8), $r_i=0$.

Situation 2: when all the labels of other individual models are the same as $l_i$, $SL=N\text{-}1$. Meanwhile, as $p_j=1$, $j=1,\cdots ,L$, $j\ne i$, $S_i=1$. Therefore, based on Eq. (6), $r_i=1$.

Situation 3: As $0<p_j<1$,

$$0<{\prod }_{j=1,j\ne i}^L(1-p_j))<1,$$ (9)

Then

$$0<1-({\prod }_{j=1,j\ne i}^L(1-p_j))<1,$$ (10)

As $0<SL<N-1$,

$$0<\frac{SL}{N-1}<1,$$ (11)

So

$$0<r_i=\frac{SL}{N-1}\cdot (1-(\prod _{j=1,j\ne i}^N(1-p_j)))<1,i=1,\cdots ,N.$$ (12)

3.5. Reliable fusion

After obtaining individual reliability and weight for each Pareto-optimal model, reliable fusion strategy is used to obtain calibrated confidence and overall reliability. Since reliable fusion is different from traditional fusion strategies, its definition is given as following:

Definition 2 (Reliable Fusion). The reliable fusion fuses the output probabilities of all individual models with both weight and reliability to obtain more reliable probability output and overall reliability.

An overview of the proposed strategy is shown in Fig. 2. Assume that there are $N$ individual models, and the corresponding output probability is $P_l$, $l=1,\cdots ,N$. For each individual model, the relative weight denoted by $w_l$, $l=1,\cdots ,L$ is obtained by training, whilst individual reliability for each model is denoted by $r_l$, $l=1,\cdots ,N$. Given a test sample, the final fused output probability $P_h$ and overall reliability $r_{all}$ can be obtained using the analytic ER² denoted by $AE{R}^2$:

$$[P_h,r_{all}]={AER}^2(P_l,w_l,r_l),l=1,\cdots ,N,$$ (13)

Fig. 2:

Reliable fusion workflow. Multiple individual models are built by feeding into all the possible features. Then the final calibrated output probability and overall reliability can be obtained by combining the output probability of all the individual models through analytic ER².

Particularly, $AE{R}^2$ is:

$$p_h=\frac{k\left[{\prod }_{i=1}^N\left(\frac{w_i{p}_{h,i}}{1+w_i-r_i}+\frac{1-r_i}{1+w_i-r_i}\right)-{\prod }_{i=1}^N\left(\frac{1-r_i}{1+w_i-r_i}\right)\right]}{1-k{\prod }_{i=1}^N\left(\frac{1-r_i}{1+w_i-r_i}\right)},h=1,\cdots ,H,$$ (14)

And normalized factor $k$ is:

$$k={\sum }_{h=1}^H\left({\prod }_{i=1}^N\left(\frac{w_i{p}_{h,i}}{1+w_i-r_i}+\frac{1-r_i}{1+w_i-r_i}\right)\right){\left[-(H-1){\prod }_{i=1}^N\left(\frac{1-r_i}{1+w_i-r_i}\right)\right]}^{-1}.$$ (15)

where $H$ is the number of Pareto-optimal model, and $N$ is the number of predicted label. Then overall reliability denoted by $r_{all}$ is estimated by averaging the individual reliability, that is:

$$r_{all}=\frac{{\sum }_{i=1}^N{r}_i}{N}.$$ (16)

Based on the overall reliability, the physicians can determine whether they should accept the model prediction result and pay particular attention to the output with low reliability.

To perform reliable fusion directly, analytic ER² (Zhou et al., 2020b) is used, which will be described in the following subsection.

4. Experimental and Analysis

4.1. Materials

Data from 129 surgical H&N cancer patients with pathological ground truth of LNs status were used. Those patients underwent surgery between 2009 and 2018 at our institute (UT Southwestern Medical Center, Dallas, TX, USA) and have preoperative PET and CT. Both benign and malignant status of 543 LNs were determined from pathological reports. Two examples are illustrated in Fig. 3. All the nodes were contoured on preoperative contrast-enhanced CT under PET guidance. We used 130 malignant nodes and 413 benign nodes. The features, including intensity, texture, and geometry, are extracted from PET and CT images, respectively, using the same approach used in our previous study (Zhou et al., 2017a).

Fig. 3:

An illustration of benign and malignant nodes in H&N cancer. (a) and (b) are the malignant and benign LNs, respectively. The first row shows the nodes (arrows) on CT images, and the second row shows the nodes (red markers) on fused PET and CT images.

4.2. Setup

In total, 256 features were extracted. The population size was set to 50, and the generation number was set to 20 when training ARMO by IMIA (Miikkulainen and Forrest, 2021). The mutation probability was indicated as 0.9. Due to the limited data set scale, a deep feedforward neural network with three hidden layers is used for building the predictive model and three models are built for PET, CT and PET&CT, respectively. The number of nodes in the three hidden layers is 64, 32 and 16, respectively.

Meanwhile, we compared with a traditional multi-objective model (MO) optimized by IMIA. Because convolutional neural network (CNN) has achieved huge success in multiple clinical use, it is compared as well (Zhou et al., 2018b). The corresponding architecture is shown in Table I. Note that CNN model is trained in an end-to-end way. The categorical cross entropy is used as loss function. Support vector machine with radial basis function kernel is used as base model to build ARMO (ARMO-SVM) for comparison. Five-fold cross-validation was performed on a patient level, and all the experiments were run five times to measure the mean and standard deviation values of AUC, accuracy (ACC), sensitivity (SEN) and specificity (SPE) with a probability threshold setting at 0.5 for the evaluation. Meanwhile, the p-value of unpaired t-test is used to indicate the statistical significance.

Table I.

CNN architecture

Layer	Kernel Size	Stride	Output Size	Features Volumes
Input	-		48×48×32	1(or 2)
C1	5×5×5	[1 1 1]	44×44×28	64
MP1	2×2×2	[2 2 2]	22×22×14	64
C2	3×3×3	[1 1 1]	20×20×12	64
C3	3×3×3	[1 1 1]	18×18×10	64
MP2	3×3×3	[2 2 2]	7×7×3	64
C4	3×3×3	[1 1 1]	5×5×3	64
C5	3×3×1	[1 1 1]	3×3×3	64
C6	3×3×1	[1 1 1]	1×1×1	64
GAP	2× 2×2	[2 2 2]	1×1×1	64
FC1			1×1×1	256
FC2			1×1×1	3

^*C indicates Convolution layer + ReLU layer +Batch Normalization layer; MP indicates Max-pooling layer; GAP indicates Global-average-pooling layer and FC indicates Fully-connected layer

Additionally, the confidence calibration results are evaluated through commonly used expected calibration error (ECE) and maximal calibration error (MCE) (Nixon et al., 2019). The accuracy can be evaluated through dividing prediction results into $M$ bins and the accuracy in each bin is calculated. In our study, $M$ is set as 15. Assume that the set of sample indices is denoted by $B_m$, and its confidence is in $(\frac{m-1}{M},\frac{m}{M}]$. The accuracy for each bin is calculated as:

$$acc(B_m)=\frac{1}{\mid B_m\mid }{\sum }_{i\in B_m}1({\widehat{y}}_i=y_i),$$ (27)

where $1(.)$ is an indicator function and ${\widehat{y}}_i=arg{\max }_{y\in Y}P(y\mid x)$. The confidence for each bin $B_m$ is defined as

$$conf(B_m)=\frac{1}{\mid B_m\mid }{\sum }_{i\in B_m}P(y_i\mid x_i),$$ (28)

Then ECE is measured by:

$$ECE={\sum }_{m=1}^M\frac{\mid B_m\mid }{\mid \mu \mid }\mid acc(B_m)-conf(B_m)\mid .$$ (29)

where $\mid \mu \mid$ is the number of samples in the testing set. Then MCE is calculated as:

$$MCE={\max }_{m\in \{1,\cdots ,M}\frac{\mid B_m\mid }{\mid \mu \mid }\mid acc(B_m)-conf(B_m)\mid .$$ (30)

ECE is used to measure the weighted average of all calibration gaps across all bins, while MCE is the largest gap across all gaps. Note that the closer to 0 the ECE is, the better the confidence calibration is.

4.3. Prediction results

The ROC curves for ARMO with three modalities are illustrated in Fig. 4. The average AUC value for the PET-based model is 0.960, and the CT-based model is 0.964. PET&CT model outperforms two single modality models, in which the average AUC value achieves 0.970. These results demonstrate that ARMO is a promising model for LNM prediction in H&N cancer.

Fig. 4:

ROC curves for ARMO with three modalities.

4.4. Model comparison

To validate the model performance, we compared ARMO and ARMO-SVM with MO and CNN on three modalities, and the results are summarized in Table II. ARMO performed the best among four methods measured by AUC. ARMO outperforms MO (p-value<0.01) because it increases the diversity by using multiple Pareto-optimal models to get the final results. The experiments showed that CNN’s performance is not as good as ARMO (p-value<0.01). This may be caused by the limited training samples in our study. As ARMO used hand-crafted features to build the model, it does not need a large-scale training dataset. Meanwhile, ARMO can obtain better performance than ARMO-SVM (p-value<0.01), which shows that DNN is a better base classifier for model construction.

Table II.

Predictive results for three methods

		AUC	ACC	SEN	SPE
PET	MO	0.92±0.01	0.87±0.01	0.76±0.04	0.90±0.01
	CNN	0.92±0.02	0.89±0.01	0.85±0.02	0.89±0.02
	ARMO-SVM	0.95±0.00	0.90±0.00	0.88±0.00	0.91±0.01
	ARMO	0.96±0.01	0.91±0.01	0.87±0.02	0.92±0.01
CT	MO	0.92±0.01	0.87±0.01	0.76±0.04	0.92±0.01
	CNN	0.93±0.01	0.89±0.01	0.83±0.02	0.91±0.01
	ARMO-SVM	0.95±0.00	0.90±0.00	0.88±0.01	0.90±0.01
	ARMO	0.96±0.00	0.92±0.00	0.89±0.02	0.93±0.01
PET&CT	MO	0.94±0.01	0.91±0.01	0.78±0.03	0.94±0.01
	CNN	0.93±0.01	0.90±0.01	0.83±0.01	0.93±0.01
	ARMO-SVM	0.96±0.01	0.91±0.01	0.87±0.02	0.92±0.01
	ARMO	0.97±0.00	0.93±0.00	0.88±0.01	0.94±0.00

4.5. Confidence calibration validation

Since the confidence calibration is performed in the fusion stage, we mainly measure the calibrated results in reliable fusion (RF). Meanwhile, since weighted fusion (WF) is a classical classifier fusion strategy and averaging fusion (AF) is an outstanding confidence calibration method in ensemble learning (Mehrtash et al., 2020), they are compared with RF in this study. WF is implemented as follows:

$$P_{fin}={\sum }_{i=1}^N{w}_i{p}_i.$$ (31)

The evaluation results on ECE and MCE are shown in Fig. 5 and 6, respectively. The ECE value in RF on three modalities is the smallest, showing that our new reliable fusion strategy can obtain a more reasonable output probability. Meanwhile, RF can obtain lower MCE than the other fusion strategies, indicating better confidence calibration performance as well.

Fig. 5:

ECE for Confidence calibration evaluation.

Fig. 6:

MCE for confidence calibration evaluation. WF: weighted fusion, AF: averaging fusion, RF: reliable fusion.

4.6. Uncertainty estimation validation

To evaluate the effectiveness of overall reliability as a method of uncertainty estimation, all the prediction results of testing samples are divided into two groups by using the median value of overall reliability: group one is the samples with the overall reliability lower than median value, and group two are the samples with the reliability higher than median value. Then the ROC curves for these two groups on three inputs are plotted (Fig. 7). The AUC values from group two (above median value) are greatly larger than group one (below median value) (p-value<0.01) on three modalities, which indicates that the predicted result with higher overall-reliability is more reliable. Therefore, overall reliability is an excellent measurement to estimate the model uncertainty, and physicians can determine whether they should accept the model prediction results based on overall reliability or pay particular attention to those predictions with lower reliability.

Fig. 7:

ROC curves below and above median overall reliability values on three modalities.

5. Discussions and Conclusions

In this study, ARMO was proposed for reliable LNM prediction in H&N cancer. ARMO is a general and unified framework that considers balanced sensitivity and specificity, confidence calibration, and uncertainty estimation simultaneously. To obtain balanced model output, multi-objective model which considers sensitivity and specificity as the objective functions simultaneously is introduced. To calibrate confidence, individual reliability is defined and integrated into the reliable fusion. In reliable fusion, to perform the fusion in a straightforward way, an analytic ER² with the analytic expression is used. The analytic ER² is also a generalized probability inference tool, which can take the reliability into the consideration and can also be applied to other probabilistic machine learning fields. To estimate the uncertainty, we defined overall reliability which is estimated by averaging the individual reliability. Finally, by considering calibrated confidence and overall reliability, physicians can determine whether the ARMO output should be accepted. The experimental results demonstrated that ARMO outperforms the comparative models, and our developed strategies are more reliable than other currently available methods.

As noted in the introduction section, AutoMO was initially developed in a preliminary study to fuse feasible solutions from Pareto-optimal solutions in a MO radiomics model. Compared to AutoMO, which only considers model balance, the proposed ARMO simultaneously considers solution balance, confidence calibration and uncertainty estimation. Specifically, there are several differences between two models: (1) During the training stage, AutoMO employs a basic classifier SVM, while ARMO employs a deep neural network which has demonstrated exceptional performance in various applications. (2) During the testing stage, AutoMO employs ER to perform fusion, while ARMO employs a new and reliable fusion strategy. Compared to ER, reliable fusion can not only obtain calibrated confidence by introducing individual reliability, but also can estimate the uncertainty of output through overall reliability. (3) By introducing the overall reliability, physicians can determine whether to accept the model output, thus facilitating translation into clinical practice. However, AutoMO does not offer such capability. In summary, ARMO is a more reliable and practical model than AutoMO.

Furthermore, building a reliable artificial intelligence (AI) system is required in medicine and other safety-critical applications. Individual reliability can not only help improving model performance, but also calibrate the confidence. Meanwhile, the overall reliability can facilitate the translation of AI models into clinic. For example, most current models focus on achieving better performance and some models even achieve 90% accuracy or higher. However, in medical applications, even 10% error could lead to severe consequences thus clinical adoption of AI models are much slower compared to other domains. As shown in Figure 6, the quantification of reliability could help to identify those potentially incorrect predictions such that clinicians could pay closer attention to these samples with lower reliability. It is hoped that the development of reliable models and reliability/uncertainty quantification of model output could facilitate the clinical adoption of reliable AI strategy for clinical decision support.

Appendix: Analytic evidential reasoning rule

ER² is a generalization of the ER approach (Yang and Xu, 2002) that was originally developed for multiple criteria decision analysis based on the D-S theory (Shafer, 1976). $\Theta =\{h_1,h_2,\cdots ,h_H\}$ includes a set of mutually exclusive and collectively exhaustive hypotheses, where $\Theta$ is referred to as a frame of discernment (e.g. malignant or benign for a lymph node). The power set of $\Theta$ consists of all its subsets, denoted by $P(\Theta )$ or $2^{\Theta }$, as following:

$$P(\Theta )=2^{\Theta }=\{\varnothing ,\{h_1\},\cdots ,\{h_H\},\{h_1,h_2\},\cdots ,\{h_1,h_M\},\cdots ,\{h_1,h_{H-1}\},\Theta \},$$ (A.1)

where $\{h_1,h_2\},\cdots ,\{h_1,h_M\},\cdots ,\{h_1,h_{H-1}\}$ are called the local ignorance. The output probability of each individual model is considered as one piece of evidence. Assuming that a piece of evidence $e_i$ is represented as a random set and profiled by a belief distribution ($BD$), the following is true:

$$e_i=\{(\theta ,p_{\theta ,i}),\forall \theta \subseteq \Theta ,{\sum }_{\theta \subseteq \Theta }{p}_{\theta ,i}=1\},$$ (A.2)

where $(\theta ,p_{\theta ,i})$ is an element of evidence $e_i$, indicating that the evidence points to proposition $\theta$, which can be any subset of $\Theta$ or any element of $P(\Theta )$ except from the empty set. $p_{\theta ,i}$ represents probability output. $(\theta ,p_{\theta ,i})$ denotes a focal element of $e_i$ if $p_{\theta ,i}>0$.

The reasoning process in the ER² is performed by defining a weighted belief distribution with reliability ($WBDR$):

$$m_i=\{(\theta ,{\tilde{m}}_{\theta ,i}),\forall \theta \subseteq \Theta ;(P(\Theta ),{\tilde{m}}_{P(\Theta ),i}\},$$ (A.3)

where ${\tilde{m}}_{\theta ,i}$ measures the degree of support for $\theta$ from $e_i$ with both weight and reliability being taken into account, defined as follows:

$${\tilde{m}}_{\theta ,i}=\{\begin{array}{c}0,\theta =\varnothing \hfill \\ c_{rw,i}{m}_{\theta ,i},\theta \subseteq \Theta ,\theta \ne \varnothing \hfill \\ c_{rw,i}(1-r_i),\theta =P(\Theta )\hfill \end{array}\phantom{\}},$$ (A.4)

$$or{\tilde{m}}_{\theta ,i}=\{\begin{array}{c}0,\theta =\varnothing \hfill \\ {\tilde{w}}_i{p}_{\theta ,i},\theta \subseteq \Theta ,\theta \ne \varnothing \hfill \\ 1-{\tilde{w}}_i,\theta =P(\Theta )\hfill \end{array}\phantom{\}},$$ (A.5)

$m_{\theta ,i}=w_i{p}_{\theta ,i}$ and $c_{rw,i}=1∕(1+w_i-r_i)$ is a normalization factor, which satisfies ${\sum }_{\theta \subseteq \Theta }{\tilde{m}}_{\theta ,i}+{\tilde{m}}_{P(\theta ),i}=1$. ${\tilde{w}}_i=c_{rw,i}{w}_i$ is acting as a new weight. Then the ER rule combines multiple pieces of evidence recursively. If two pieces of evidence $e_1$ and $e_2$ are independent, they jointly support proposition $\theta$ denoted by $p_{\theta ,e(2)}$, which is generated as follows:

$$p_{\theta ,e(2)}=\{\begin{array}{c}0\theta =\varnothing \hfill \\ \frac{{\widehat{m}}_{\theta ,e(2)}}{{\sum }_{D\subseteq \Theta }{\widehat{m}}_{D,e(2)}}\theta \subseteq \Theta ,\theta \ne \varnothing \hfill \end{array}\phantom{\}},$$ (A.6)

$${\widehat{m}}_{\theta ,e(2)}=[(1-r_2)m_{\theta ,1}+(1-r_1)m_{\theta ,2}]+{\sum }_{B\cap C=\theta }{m}_{B,1}{m}_{C,2}\forall \theta \subseteq \Theta ,$$ (A.7)

When $L$ pieces of independent evidence are present, the jointly supported proposition $\theta$ denoted by ${\widehat{m}}_{\theta ,e(L)}$ can be generated by the following two equations:

$${\widehat{m}}_{\theta ,e(L)}=[(1-r_i)m_{\theta ,e(i-1)}+m_{P(\Theta ),e(i-1)}{m}_{\theta ,i}]+{\sum }_{B\cap C=\theta }{m}_{B,e(i-1)}{m}_{C,i}\forall \theta \subseteq \Theta ,$$ (A.8)

$${\widehat{m}}_{P(\Theta ),e(L)}=(1-r_i)m_{P(\Theta ),e(i-1)},$$ (A.9)

After normalization, the combined $BD$ denoted by $p_{\theta }$ can be calculated by the following equation:

$$p_{\theta }=\frac{{\widehat{m}}_{\theta ,e(L)}}{1-{\widehat{m}}_{P(\Theta ),e(L)}},\forall \theta \subseteq \Theta .$$ (A.10)

The recursive formula of the ER² can combine multiple pieces of evidence in any order. To perform inference in a straightforward way, we inferred analytic ER², that is

$$p_h=\frac{k\left[{\prod }_{i=1}^N\left(\frac{w_i{p}_{h,i}}{1+w_i-r_i}+\frac{1-r_i}{1+w_i-r_i}\right)-{\prod }_{i=1}^N\left(\frac{1-r_i}{1+w_i-r_i}\right)\right]}{1-k{\prod }_{i=1}^N\left(\frac{1-r_i}{1+w_i-r_i}\right)},h=1,\cdots ,H,$$ (A.11)

And normalized factor $k$ is:

$$k={\sum }_{h=1}^H\left({\prod }_{i=1}^N\left(\frac{w_i{p}_{h,i}}{1+w_i-r_i}+\frac{1-r_i}{1+w_i-r_i}\right)\right){\left[-(H-1){\prod }_{i=1}^N\left(\frac{1-r_i}{1+w_i-r_i}\right)\right]}^{-1}.$$ (A.12)

where $H$ is the number of Pareto-optimal model, and $N$ is the number of predicted label. The detailed inference can be seen in (Zhou et al., 2020b). Eq. (11) is the AER² expression and Eq. (12) is the normalized factor, which provides an explicit aggregation function.