Adversarial training and deep k-nearest neighbors improves adversarial defense of glaucoma severity detection

Abstract

Glaucoma is an eye disease that can cause irreversible blindness to people if not treated properly. Although deep learning models have shown that they can provide good results in identifying diseases from medical imagery, they suffer from the vulnerability of adversarial attacks, making them perform poorly. Several techniques can be applied to improve defense against such attacks. One of which is adversarial training (AT) which trains a deep learning model using the input's gradient used to generate noises to the input image and Deep k-Nearest Neighbor (DkNN) that enforces prediction's conformity based on nearest neighbor voting on each layer's representation. This work tries to improve the defense against adversarial attacks by combining AT and DkNN. The evaluation performed on several adversarial attacks show that given an optimum k, the combination of these two methods is able to improve most models' overall classification result on the perturbed retinal fundus image.

Highlights

• •A robust deep learning (DL) model for glaucoma severity detection is proposed.
• •It combines a vision-based DL with adversarial training and deep k-nearest neighbor (DkNN).
• •Evaluations are performed on models trained using normal and three adversarial attacked datasets.
• •The effect of DkNN is sensitive depending on the selected k.
• •DkNN combined with adversarially trained model improves the robustness.

Glaucoma severity; Multiclass classification; Deep learning; Adversarial training; Adversarial attack; Nearest neighbors

1. Introduction

Glaucoma is an eye condition that causes blindness to around 250 people every day, which equals one person every six minutes [24]. It is a disease that damages the optic nerve and Retinal Nerve Fiber Layer (RNFL), causing blindness [19]. Shinde states that by 2040, the total cases of people with glaucoma are predicted to reach 111.8 million [19], [24]. Due to the irreversibility of blindness caused by glaucoma, its recognition and treatment must be treated quickly. Similar to Zulfira et al., this research primarily focuses on the most common types of glaucoma, which is the primary open-angle glaucoma defined by its severity, to find the best treatment and prevent blindness caused by the disease.

Diseases usually diagnosed using medical imagery like Magnetic Resonance Imaging (MRI), Chest Radiography, and Retinal Scan are primarily solved by a vision-based, fully deep learning model like Convolutional Neural Network (CNN). Although it works primarily with good results, deep learning models tend to be sensitive to noises, especially when trained in a small dataset. In the case of imagery, noises can either be based on the image quality itself or artificially generated. Examples include a non-mydriatic fundus camera (NMFC), which has been commonly used to evaluate diabetic retinopathy (DR) for almost 26 years. A study has shown that a large amount of poor quality-ungradable images and low sensitivity has limited the use of NMFC for screening, especially with people with small pupils, dark irises, and media opacities like cataract [10], [16]. A handheld fundus camera also showed poorer image quality than a proper desktop camera [16]. We would like to note that from the examples shown in Rajalakshmi et al. works, a smartphone-based fundus camera quality will depend on the smartphone camera quality itself, and screening might be done in a non-clinical condition. Hence, under these conditions, the overall image quality will differ such that it will be more susceptible to noise in the case of a handheld or smartphone-based fundus camera. Artificially, an example method to generate noise is through adversarial attacks that are intended to make a deep learning model perform poorly [1], [22]. Adversarial attacks in vision-based tasks generate “adversarial examples” to cause a model to make a mistake intentionally by generating a synthesized image that looks similar to the original image but can hinder the prediction result of the model [23].

Several methods exist to defend a deep learning model against adversarial attacks. The most common method is adversarial training which trains a deep learning model with a perturbed image input [1], [13], [22]. Another technique is applying a Deep k-Nearest Neighbor (DkNN) proposed by Papernot and McDaniel. DkNN is initially intended to enforce a prediction's conformity on input based on the original training data that a deep learning model has trained. It performs a nearest neighbor search to find the training points based on each layer's representation and can defend a deep learning model from adversarial attacks by identifying changes in label distribution of nearest neighbor training points from lower to higher layer representations to indicate misclassification within the deep learning model [14].

In this work, we combine DkNN with several vision-based deep learning models trained on several adversarial training such as Fast Gradient Sign Method (FGSM) Training with random starts (RS) [22], FGSM with gradient alignment (GradAlign) [1], and Projected Gradient Descend (PGD) [13] to observe how it will improve the defense against adversarial attacks further.

In highlights, the research contributions of this work are: a robust deep learning (DL) model for glaucoma severity detection is proposed; it combines a vision-based DL with adversarial training and deep k-nearest neighbor (DkNN); evaluations are performed on models trained using normal and three adversarial attacked datasets; the effect of DkNN is sensitive depending on the selected k; DkNN combined with adversarially trained model improves the robustness. Furthermore, this work is split into several chapters. Chapter 2 briefly reviews glaucoma detection and adversarial defense. Chapter 3 presents the methods we apply to our work. Chapter 4 provides the evaluation result of our work. Lastly, Chapter 5 gives our conclusion about this work and suggestions for future works to improve or explore the methods applied in this work more in-depth.

2. Related work

From what we have discovered, we have not found any research related to defense against adversarial attacks in the glaucoma detection field yet. However, there have been different kinds of research for glaucoma detection and adversarial attacks. The related researches include:

2.1. Glaucoma detection

Most research uses segmentation techniques to extract the Cup-to-Disc Ratio (CDR) from the input images. Shinde proposed the use of U-Net [17] as a feature extraction model for extracting CDR, Inferior Superior Nasal Temporal (ISNT) ratio for Neuro-retinal Rim (NRR), and ISNT blood vessel ratio from the U-Net's segmentation result. The features extracted use an ensemble of models for doing majority votes between three classifiers, Support Vector Machine (SVM), Adaptive Boosting (Adaboost), and a Multilayer Perceptron (MLP). The proposed segmentation method achieves a dice score of 0.93 for outer disc (OD) segmentation and 0.87 for outer cup (OC) segmentation. The classification evaluation achieves, on average, across multiple datasets over 98.8% accuracy [19].

Zulfira et al. utilizes several techniques for feature extraction since it does not utilize a deep learning model. The techniques used include segmentation techniques, including Active Contour Snake for tracing the OC and OD outlines, Optical Nerve Head (ONH) masking to mask the OD region, and Otsu thresholding for binarizing the segmentation result to black and white. Since it does not utilize learnable feature extraction found in CNNs, it uses Gray Level Occurance Matrix (GLCM), which calculates the occurrence of the same matrices in images and extracts features by determining the probability from the neighboring relationship between two pixels within a distance and its angle orientation. The classification model uses an ensemble model for imbalanced data called Dynamic Ensemble Selection with Multiclass Imbalanced datasets (DES-MI) that generate balanced training data and select the optimal classifier by weighting the competence. From the 5-fold cross-validation used to evaluate the model, the proposed methods obtained an overall accuracy of 0.96 compared to the U-Net with 0.90 accuracies. The proposed method also obtains a faster inference time where it obtains 2.56s compared to U-Nets 3.52s [24].

2.2. Defense against adversarial attacks

Deep learning models are noise-sensitive, especially when trained on a small dataset. One very vulnerable and dangerous noise to a deep learning model is a noise generated from adversarial perturbation, which utilizes the input's gradient from the model to fool the model intentionally [1], [22], [23]. Many types of research have allowed deep learning models to perform more optimally when given an adversarial attack. However, some methods suffer from the obfuscated gradient, a particular type of gradient masking that leads to a false sense of security against adversarial examples [2]. One method to improve adversarial defense without obfuscation is through adversarial training, which uses the perturbed input generated from the attack [13], [22]

Athalye et al. states that PGD adversarial training [13] does not cause obfuscated gradients [2]. Although it is beneficial to use PGD training, it requires high compute resources and training time because it takes multiple steps of gradient calculation for a single input. Hence, a faster adversarial training method with FGSM perturbation [1], [23] exists. Xu et al. utilize FGSM training by initially adding uniformly random noise. However, FGSM training has a problem called catastrophic overfitting, where the robust accuracy of an FGSM-trained model on PGD perturbation suddenly drops to 0% on the training data. Such a problem can be salvaged by applying early stopping by measuring accuracy on PGD perturbation [22]. Andriushchenko and Flammarion states that the catastrophic overfitting problem can be solved by implementing gradient alignment (GradAlign) that calculates using cosine similarity between normal and perturbed input gradient. However, it leads to a cost of increasing the training time since it calculates gradient two times as compared to Wong et al. approach since it needs the gradient of both normal and perturbed input to regularize [1].

Papernot and McDaniel proposed a DkNN model calibrated with each layer's representation in a trained neural network for use during inference to enforce conformity in a deep learning model. Thus, it also reduces the computational cost during training since it can be applied using an existing trained model without additional training. It can also provide a defense to perturbation by detecting changes in the labeling of nearest neighbor points from the training data between each layer representation within the deep learning model allowing for more robust classification output from the model [14].

3. Methodology

For this research, we use the same dataset used by Zulfira et al. The entire dataset has over 250 RF images where the ground truths are labeled by ophthalmology experts [24]. Due to the privacy policy of Dr. YAP Hospital in Yogyakarta, Indonesia, we cannot publicly post the dataset. We split the data into 200 data for training and 50 for testing. Then, we split the train set by doing a 5-fold cross-validation for the training and validation phase. We generate a model for each fold that takes the best weights according to the best validation loss of the fold. After training, using the same folds used for training, we generate a DkNN model with the same training data and calibrate it using the validation set. We evaluate both models first with and without adversarial attacks while simultaneously evaluating with different k values to find the optimum parameters for the DkNN model.

3.1. Data augmentation

Data augmentation is adding or augmenting additional data to train deep learning models. It is shown that using data augmentation can substantially improve deep learning performance. It also does not add additional overhead in inference since it is only applied during the model training [3].

We utilize RandAugment [3] as the augmentation technique used for this research which removes the need for a separate search phase for finding optimal parameters for augmentation. It applies parameter-free augmentation with transformation based on uniform probability $\frac{1}{K}$. Thus giving $K^N$ where N is the number of transformations for an image. Cubuk et al. states that there are 14 operations randomly sampled for RandAugment, which include:

• •No Operation (Identity)
• •Auto Contrast
• •Equalize
• •Rotate
• •Solarize
• •Color
• •Posterize
• •Contrast
• •Brightness
• •Sharpness
• •ShearX
• •ShearY
• •TranslateX
• •TranslateY

Fig. 1 shows the example of RandAugment operations, where the original and augmented normal samples are shown in Figs. 1a and 1b, the original and augmented mild samples are illustrated in Figs. 1c and 1d, the original and augmented severe samples are depicted in Figs. 1e and 1f. We take one sample for each label within the dataset and provide the transformations to be visualized. For our research, we utilize the RandAugment implementation from PyTorch's Torchvision library [15], where we only change the default number of sampled operations from two to four random operations.

Figure 1.

RandAugment examples on randomly sampled images for each label: (a) the original normal, (b) the augmented normal, (c) the original mild, (d) the augmented mild, (e) the original severe, and (f) the augmented severe.

3.2. Normalization

Normalization of image data is a procedure that ensures that comparisons between data-collecting methods and texture instances are as accurate as possible. It is advised to normalize pixel values for imaging modalities that do not correlate to absolute physical quantities [4]. Fig. 2 shows the normalization we have done for both before and after data augmentation using RandAugment as explained in Section 3.1. The original, normalized, and normalized-augmented normal samples are shown in Figs. 2a, 2b, and 2c. The original, normalized, and normalized-augmented mild samples are illustrated in Figs. 2d, 2e, and 2f. The original, normalized, and normalized-augmented mild samples are illustrated in Figs. 2g, 2h, and 2i. We normalized the image according to the mean and standard deviation based on Equation (1).

$$\mathrm{normalize}(x,\mu ,\sigma )=\frac{x-\mu }{\sigma }$$ (1)

where x, μ, and σ are derived as the input image, mean, and standard deviation of all channels throughout the full dataset. For this research, we use the default ImageNet's [18] mean and standard deviation since we utilize models that are pretrained with the ImageNet dataset.

Figure 2.

Normalized examples on randomly sampled images for each label: (a) the original normal, (b) the normalized normal, (c) the normalized-augmented normal, (d) the original mild, (e) the normalized mild, (f) the normalized-augmented mild, (g) the original severe, (h) the normalized severe, (i) the normalized-augmented severe.

3.3. Deep learning architectures

For this research, we proposed the use of three distinct vision-based deep learning models for comparison, including:

3.3.1. Residual network

ResNet is a CNN architecture proposed by He et al. Known for its place for winning ImageNet Large Scale Visual Recognition Challenge 2015 [18] competition on classification task [7]. One of the most widely used CNN models for transfer learning on many vision-based tasks.

It contains three components: the input stem, four successive stages of a convolutional block, and the output layer. Fig. 3, adopted from [8], shows that the input stem contains a 64 output channel convolutional layer with $7\times 7$ kernel size and two strides succeed by a max-pooling layer with $3\times 3$ pool size and the same two strides. Thus, the width and height of the input are reduced four times, and the channel is increased to 64.

Figure 3.

Residual Network.

Each stage begins with a downsampling block from the second stage, followed by some residual blocks. There are two pathways identified as path A and path B within the downsampling block. Path A consists of three convolution layers, with a kernel size of $1\times 1$, $3\times 3$, and $1\times 1$. The first convolution uses two strides to halve the input width and height, and the final output channel uses a fourfold greater output channel than the previous two. This technique is known as bottlenecking. Path B consists of a $1\times 1$ convolution with two strides to change the input shape to match the output shape for the skip connection, which sums the outputs of both paths to generate the output of the downsampling blocks. For this research, we use the ResNet-D variant, which is a modified version of ResNet that adds a $2\times 2$ average pooling layer with two strides before the convolution, in which the convolution layer's stride is changed to one which does not ignore 3/4 of the feature map inputs [8].

3.3.2. Vision transformer

Vision Transformer (ViT) is a transformer-based model proposed by Dosovitskiy et al. for computer vision tasks. It is composed of encoder stacks used in the original Transformer architecture. It is shown that evaluation in ImageNet [18] that ViT performs comparably to recent CNN models with fewer compute resources to train [5].

Fig. 4 shows the model overview as provided in [5]. The original Transformer has a one-dimensional input sequence of token embeddings. In the case of ViT, the image is transformed into flattened two-dimensional patches using Equation (2).

$$x\in {\mathbb{R}}^{H\times W\times C}\to x_p\in {\mathbb{R}}^{N\times (P^2•C)}$$ (2)

where height, width, and channel are derived as H, W, and C. P is defined as the resolution of patches in the image, and $N=\frac{HW}{P^2}$ denotes the number of patches as the sequence length for input to the Transformer. The patches are flattened and mapped to trainable linear projections on D dimensions, which the output is referred to as the patch embeddings.

Figure 4.

Vision Transformer.

The encoder architecture in ViT is similar to the encoder within the original Transformer proposed by Vaswani et al. [21]. The encoder has multiple layers of multi-head self-attention and MLP blocks. Additionally, Layer Normalization is performed prior to each block with skip connections after each block [5].

3.3.3. Gated multilayer perceptron

Gated Multilayer Perceptron (gMLP) is a deep learning architecture proposed by Liu et al. [20]. It is a Multilayer Perceptron-based alternative to Transformer since it does not use the self-attention that Transformers are known for. The MLP layer in gMLP consists of a statically parameterized channel and spatial projections. Evaluation in ImageNet [18] shows that gMLP with 66% fewer parameters performs better than the recent MLP-based vision model MLP-Mixer proposed by Tolstikhin et al. with an increase of 3% accuracy and comparable performance to ViT [11].

Fig. 5 shows an overview of gMLP architecture [11]. Liu et al. states that gMLP consists of L blocks with identical size and structure stacked. Suppose $X\in {\mathbb{R}}^{n\times d}$ is the representation of a token with sequence length n and dimension d. The block is defined as shown in Equation (3).

$$Z=\sigma (XU),\tilde{Z}=s(Z),Y=\tilde{Z}V$$ (3)

where σ defines a function of activation, such as Gaussian Error Linear Unit (GELU). U and V are linear projections along channel dimensions similar to Transformers' Feed Forward Neural Network (FFN). $s(•)$ is a layer used to capture the spatial interactions depending on the mapping. When s is an identity mapping, the transformation becomes similar to that of a regular FFN.

Figure 5.

gMLP Layer.

In order to do cross-token interactions, layer $s(•)$ must contain an operation that contracts over the spatial dimension with a linear projection based on Equation (4).

$$f_{W,b}(Z)=WZ+b$$ (4)

where $W\in {\mathbb{R}}^{n\times n}$ is a matrix with same size as the sequence length n and token-specific bias b. Spatial projection $W(Z)$ is independent of the input representations, unlike self-attention in Transformers, which is dynamically generated from Z.

The $s(•)$ layer can be defined as the linear gating output using Equation (5).

$$s(Z)=Z\odot f_{W,b}(Z)$$ (5)

where ⊙ refers to an element-wise multiplication operation. Furthermore, Z is split into two independent parts ($Z_1$, $Z_2$) along the channel dimension for the gating function and multiplicative pass, which is referred to as the Spatial Gating Unit (SGU) in Equation (6).

$$s(Z)=Z_1\odot f_{W,b}(Z_2)$$ (6)

where $f_{W,b}$ input is normalized to stabilize the model which is shown in [11] able to narrow the performance gap with self-attention.

3.4. Adversarial attack

The adversarial attack is a technique to generate adversarial examples intentionally designed to cause a deep learning model to make mistakes. In computer vision, images or frames input is synthetically generated to look similar to the original input but misleads a deep learning model to provide the wrong output [23]. For this research, we evaluate our trained models with the addition of using DkNN using three methods which include:

3.4.1. Fast gradient sign method

FGSM is a one-step adversarial attack proposed by Goodfellow et al. The formula for generating adversarial examples using FGSM is given in Equation (7)

$$\mathrm{FGSM}(\theta ,x,y)=x+ϵ\cdot \mathrm{sign}({\mathrm{\nabla }}_x\mathcal{L}(\theta ,x,y)),$$ (7)

where θ is the model parameters, x is the input, y is the label, and ϵ is the maximum perturbation. FGSM is called “fast” since it generates adversarial examples from the input's gradient from only one backpropagation. Thus, FGSM solves the problem of generating many adversarial examples [6], [23].

3.4.2. Basic iterative method

BIM is an iterative adversarial attack proposed by Kurakin et al. It is an iterative version of FGSM, with the formula for generating adversarial examples written in Equation (8).

$$\mathrm{BIM}(\theta ,x^{t+1},y)={\mathrm{clip}}_{x,ϵ}(x^t+\alpha \cdot \mathrm{sign}({\mathrm{\nabla }}_x\mathcal{L}(\theta ,x^t,y))$$ (8)

where t is the current iteration which starts at $t=0$ thus $x_0=x$, α is the step size which is usually set in a relatively small number, and consider the number of steps as T that when $t<T$, guarantees perturbations can reach the defined border. The clip function denotes the argument projection to the surface of x's ϵ-neighbor ball $B_x:\{x^{\prime }:||x^{\prime }-x|{|}_{\infty }\le ϵ\}$ [9], [23].

3.4.3. Projected gradient descent

PGD is a variation of BIM proposed by Madry et al. It adds a uniformly random initialization with a range of $[-ϵ,ϵ]$ before the same perturbation process as BIM [13]. Hence, PGD has a more substantial perturbation than BIM, although with a low visual difference.

Fig. 6 illustrates some examples of those three adversarial attacks. The original normal sample and the three attacked-normal samples with FGSM, BIM, and PGD are shown in Figs. 6a, 6b, 6c, and 6d. The original mild sample and the three attacked-mild samples with FGSM, BIM, and PGD are depicted in Figs. 6e, 6f, 6g, and 6h. The severe normal sample and the three attacked samples with FGSM, BIM, and PGD are illustrated in Figs. 6i, 6j, 6k, and 6l.

Figure 6.

Examples of the adversarial attacks: (a) Original normal sample, (b) Normal with FGSM attack, (c) Normal with BIM attack, (d) Normal with PGD attack, (e) Original mild sample, (f) Mild with FGSM attack, (g) Mild with BIM attack, (h) Mild with PGD attack, (i) Original severe sample, (j) Severe with FGSM attack, (k) Severe with BIM attack, (l) Severe with PGD attack.

3.5. Adversarial training

Adversarial training is one of the ways to have a robust deep learning model. Generally, it can be expressed as the following optimization problem given a network $f_{\theta }$ with learnable parameter θ, loss function l, a dataset of $(x_i,y_i)$, and a threat model Δ in Equation (9),

$$\underset{\theta }{\min }\sum _i\underset{\delta \in \mathrm{\Delta }}{\max }l(f_{\theta }(x_i+\delta ),y_i)$$ (9)

this explains that given $\mathrm{\Delta }=\{\delta :{\Vert \delta \Vert }_{\infty }\le ϵ\cap ϵ>0\}$ as an example, the objective is to minimize a given input $x_i$ with perturbation or noise δ which derived from maximization of Δ and target $y_i$ to find the optimal parameters θ for network $f_{\theta }$ [1], [13], [22].

3.5.1. PGD training

Proposed by Madry et al., it is an adversarial training technique that utilizes PGD perturbation explained in Section 3.4.3, which generated over N steps [13]. It is one of the most optimal techniques to train a robust network. However, it leads to an increase in training time due to N number of gradient calculations. The algorithm for PGD training is shown in Algorithm 1.

Algorithm 1.

PGD Training.

3.5.2. FGSM-RS training

Proposed by Wong et al., it is a variation of standard FGSM training that initializes uniform sampling of $[-ϵ,ϵ]$. It is shown that this method can produce an almost similar level of robustness as opposed to PGD training with significantly less amount of time. However, FGSM training suffers from a catastrophic overfitting condition where the robust accuracy w.r.t PGD attack suddenly drops to 0% [22]. The algorithm of FGSM-RS is shown in Algorithm 2.

Algorithm 2.

FGSM-RS Training.

3.5.3. FGSM-GradAlign training

Proposed by Andriushchenko and Flammarion it is a variation of FGSM training which adds a gradient alignment (GradAlign) as a regularizer by calculating the cosine similarity between expected and perturbed input gradient. Although it prevents catastrophic overfitting [22] and allows for a closer result to a multistep adversarial training like PGD [13], the use of GradAlign comes at an additional training cost since it needs to calculate the input gradient two times. The algorithm for FGSM-GradAlign is shown in Algorithm 3.

Algorithm 3.

FGSM-GradAlign Training.

3.6. Deep k-nearest neighbor

DkNN is an algorithm proposed by Papernot and McDaniel to enforce conformity of a deep neural network's (DNN) prediction on an input to the training data. The nearest neighbor search is performed for each layer within a DNN to find the k closest representation between the extracted input and train data from the same layer. After a layer's closest representations have been calculated, the labels found are analyzed to ensure each intermediate computation performed in each layer remains conformal to the model's final prediction [14].

DkNN allows for more robust deep learning models since it creates a novel characterization for confidence which Papernot and McDaniel calls credibility. Credibility spans the representation hierarchies within a DNN, and any credible classification must be supported by evidence from the training data. Evaluation from experiments that Papernot and McDaniel perform shows that using DkNN on several datasets and perturbations show that DkNN can improve a DNN's defense against adversarial attacks based on the input's non-conformity and empirical p-value taken from the nearest neighbors of the training set [14]. The steps on how DkNN works are shown in Algorithm 4.

Algorithm 4.

Deep k-Nearest Neighbor.

4. Result and discussion

4.1. Experiment parameters

In the experiment, we use three distinct pretrained deep learning models with a similar number of parameters with the default ImageNet classifier head, ResNet-50d (CNN), ViT-S (Transformer), and gMLP (MLP) model, as shown in Table 1. We trained all models with the same random seeding and mixed precision on NVIDIA GeForce RTX 2070 SUPER using a PyTorch implementation. To increase the reproducibility of our result, we configure our PyTorch environment to use deterministic algorithms as much as possible when training in parallel using the GPU with random seeding set to 2022. Although the result we gathered around several runs during training was consistent, the cost of reproducibility is that our model's training time is increased.

Table 1.

Experimented model number of parameters.

Model	Number of parameters
ResNet-50d	25,576,264
ViT-S	21,974,632
gMLP-S	24,721,096

For the hyperparameters, we did several trials and errors to find the best hyperparameters shown in Table 2. We discovered that the models overfitted the training set when we tried with or without freezing the pretrained models. To reduce overfitting, we tried several techniques, such as exponential moving average (EMA) parameter update. For adversarial training, we use the default parameters used in [12], [22] where α is the step size for adversarial training steps, and ϵ is the perturbation or clipping limit. Additionally, we use Nesterov momentum for the optimizer with the momentum value set to 0.9 and weight decay of 0.05. We also checkpoint the model's weights based on the best validation loss saved for each fold during training to get the best model from each fold's training session.

Table 2.

Training hyperparameters.

Hyperparameters	Value
Training Batch size	32
Testing Batch size	16
Total epoch	200
Freeze pretrained models	False
Optimizer	SGD
Learning Rate (ResNet)	5 × 10−3
Learning Rate (ViT+gMLP)	5 × 10−4
Learning rate scheduler	Cosine Annealing
Scheduler step	Every batch to 50 epochs
Use EMA	True (0.999 decay)
Adv. Training HP	α = 10/255,ϵ = 8/255
GradAlign λ	0.2
PGD Training Step	7

Additionally, we add additional layers before the classifier layer to improve our results. As shown in Fig. 7, after the pretrained model that we considered as the encoder, we added two fully connected (FC) layers where in between, we added a Gaussian Error Linear Unit (GELU), Batch Normalization (BN), and a Dropout layer where both BN and Dropout are used to reduce overfitting. We also use skip connections that add the values from the first FC layer to the second FC layer to improve our classifier's learning ability. Finally, the skip connection's output goes to the classifier layer defined with the specified number of classes as the final output.

Figure 7.

Classification model.

For DkNN, the training and calibration set representations are from inputs without any perturbations. All adversarial evaluations are done during the model's testing on the test set. Evaluation on adversarial attacks is evaluated with ϵ ranging from 0.2 to 1.0, and for BIM and PGD, we set additional parameters, α to 0.01 and T to 10. The DkNN models are evaluated with k from 1 to 100 in five steps.

4.2. Evaluation results

We evaluate each model based on the weights trained and validated from the five-fold split. For the evaluation, we evaluate with and without adversarial attacks that have been explained in Section 3.4. The DkNN models generated are the best generated from our hyperparameter tuning result for each test result. Due to the few data samples in the dataset, the standard deviations between different random data subsets for some metrics across models are considerably higher, as shown in the results.

Overall, the differences are negligible in the effects of DkNN when we evaluate the models typically with optimum k. In Table 3, we see that the adversarially trained models generally have lower performance when evaluated without attacks. We see that the results for the adversarially trained model are improved when applying DkNN, even though it is still lesser than most typically trained models.

Table 3.

5-fold average test set evaluation.

Model	Train Method	DkNN	Accuracy	Precision	Recall	F1-Score
ResNet-50d	Normal	N/A	0.7600 ± 0.0400	0.7153 ± 0.0691	0.6204 ± 0.0681	0.6496 ± 0.0715
		k = 1	0.7600 ± 0.0283	0.7328 ± 0.0722	0.5992 ± 0.0478	0.6307 ± 0.0501
	FGSM-RS	N/A	0.6480 ± 0.0179	0.3484 ± 0.3021	0.3483 ± 0.0335	0.2884 ± 0.0631
		k = 5	0.6760 ± 0.0167	0.5570 ± 0.0376	0.5187 ± 0.0330	0.5250 ± 0.0403
	FGSM-GradAlign	N/A	0.7200 ± 0.0529	0.5675 ± 0.1289	0.5496 ± 0.1004	0.5485 ± 0.1142
		k = 45	0.7400 ± 0.0616	0.6658 ± 0.1363	0.5646 ± 0.0832	0.5797 ± 0.0885
	PGD	N/A	0.6400 ± 0.0000	0.2142 ± 0.0020	0.3333 ± 0.0000	0.2608 ± 0.0014
		k = 1	0.6400 ± 0.0583	0.4779 ± 0.0849	0.4679 ± 0.0613	0.4663 ± 0.0716
ViT-S	Normal	N/A	0.8360 ± 0.0089	0.7981 ± 0.0288	0.7263 ± 0.0198	0.7441 ± 0.0134
		k = 10	0.8280 ± 0.0460	0.8256 ± 0.0655	0.6967 ± 0.0876	0.7222 ± 0.1016
	FGSM-RS	N/A	0.6400 ± 0.0141	0.2812 ± 0.1508	0.3396 ± 0.0203	0.2753 ± 0.0349
		k = 1	0.7360 ± 0.0358	0.5899 ± 0.0931	0.5721 ± 0.0620	0.5740 ± 0.0762
	FGSM-GradAlign	N/A	0.6320 ± 0.0179	0.2743 ± 0.0848	0.3587 ± 0.0451	0.2965 ± 0.0551
		k = 1	0.7000 ± 0.0648	0.5365 ± 0.1151	0.5217 ± 0.0949	0.5208 ± 0.1035
	PGD	N/A	0.6440 ± 0.0089	0.3146 ± 0.1510	0.3446 ± 0.0159	0.2837 ± 0.0324
		k = 5	0.7160 ± 0.0740	0.5898 ± 0.1986	0.4912 ± 0.1129	0.4875 ± 0.1377
gMLP-S	Normal	N/A	0.8000 ± 0.0583	0.7302 ± 0.1010	0.6950 ± 0.0908	0.6900 ± 0.0960
		k = 10	0.8160 ± 0.0456	0.7783 ± 0.0433	0.7067 ± 0.0679	0.7083 ± 0.0902
	FGSM-RS	N/A	0.6560 ± 0.0219	0.3845 ± 0.2421	0.3650 ± 0.0435	0.3156 ± 0.0758
		k = 1	0.6200 ± 0.0316	0.4960 ± 0.1330	0.4437 ± 0.0787	0.4464 ± 0.0821
	FGSM-GradAlign	N/A	0.6520 ± 0.0179	0.3280 ± 0.1612	0.3533 ± 0.0298	0.2954 ± 0.0512
		k = 5	0.7000 ± 0.0346	0.5925 ± 0.0756	0.4946 ± 0.0472	0.5086 ± 0.0525
	PGD	N/A	0.6400 ± 0.0000	0.2133 ± 0.0000	0.3333 ± 0.0000	0.2602 ± 0.0000
		k = 10	0.7040 ± 0.0261	0.7395 ± 0.1066	0.4854 ± 0.0640	0.4966 ± 0.0692

When we evaluate with FGSM attack, Table 4 shows the benefit of using DkNN for improving a deep learning model robustness. With DkNN, we see that it improves most models' robust prediction ability. However, ResNet-50d trained with FGSM-RS without DkNN achieves the best result with the highest score across all metrics. We also see that the results show that models with PGD training tend to prefer lesser k values.

Table 4.

5-fold average test set evaluation with FGSM perturbation (ϵ = 0.4).

Model	Train Method	DkNN	Accuracy	Precision	Recall	F1-Score
ResNet-50d	Normal	N/A	0.2080 ± 0.0832	0.1773 ± 0.0445	0.1333 ± 0.0341	0.1498 ± 0.0406
		k = 75	0.5440 ± 0.0434	0.2757 ± 0.1396	0.2896 ± 0.0136	0.2545 ± 0.0206
	FGSM-RS	N/A	0.7480 ± 0.0701	0.6823 ± 0.1336	0.6154 ± 0.1138	0.6277 ± 0.1299
		k = 10	0.7080 ± 0.0944	0.5765 ± 0.2293	0.5521 ± 0.1488	0.5446 ± 0.1724
	FGSM-GradAlign	N/A	0.6600 ± 0.0510	0.4953 ± 0.1607	0.4367 ± 0.0771	0.4316 ± 0.0984
		k = 35	0.6840 ± 0.0358	0.5870 ± 0.1251	0.4771 ± 0.0529	0.4909 ± 0.0720
	PGD	N/A	0.6400 ± 0.0000	0.2133 ± 0.0000	0.3333 ± 0.0000	0.2602 ± 0.0000
		k = 10	0.6480 ± 0.0179	0.4850 ± 0.0657	0.4133 ± 0.0397	0.3927 ± 0.0594
ViT-S	Normal	N/A	0.2160 ± 0.1987	0.1656 ± 0.1545	0.1312 ± 0.1245	0.1461 ± 0.1373
		k = 80	0.3280 ± 0.1301	0.1885 ± 0.0705	0.2075 ± 0.0691	0.1903 ± 0.0714
	FGSM-RS	N/A	0.3000 ± 0.2950	0.1244 ± 0.0953	0.1562 ± 0.1536	0.1349 ± 0.1175
		k = 1	0.6240 ± 0.1108	0.4958 ± 0.3660	0.3712 ± 0.0866	0.3499 ± 0.1221
	FGSM-GradAlign	N/A	0.2560 ± 0.2809	0.1052 ± 0.0930	0.1333 ± 0.1463	0.1153 ± 0.1157
		k = 50	0.6200 ± 0.0283	0.2108 ± 0.0036	0.3229 ± 0.0147	0.2551 ± 0.0073
	PGD	N/A	0.4840 ± 0.2523	0.1755 ± 0.0715	0.2521 ± 0.1314	0.2052 ± 0.0965
		k = 1	0.5480 ± 0.1016	0.4324 ± 0.1585	0.3442 ± 0.0919	0.3458 ± 0.1034
gMLP-S	Normal	N/A	0.1520 ± 0.1361	0.1516 ± 0.1219	0.0917 ± 0.0870	0.1119 ± 0.1002
		k = 85	0.2840 ± 0.1299	0.1500 ± 0.0577	0.1571 ± 0.0784	0.1521 ± 0.0672
	FGSM-RS	N/A	0.5400 ± 0.1908	0.1948 ± 0.0373	0.2812 ± 0.0993	0.2281 ± 0.0634
		k = 80	0.6360 ± 0.0089	0.2137 ± 0.0025	0.3312 ± 0.0047	0.2598 ± 0.0030
	FGSM-GradAlign	N/A	0.3440 ± 0.2762	0.1366 ± 0.0874	0.1792 ± 0.1439	0.1526 ± 0.1120
		k = 65	0.6240 ± 0.0261	0.2455 ± 0.0776	0.3342 ± 0.0282	0.2744 ± 0.0436
	PGD	N/A	0.4880 ± 0.2257	0.1801 ± 0.0560	0.2542 ± 0.1175	0.2088 ± 0.0813
		k = 1	0.4760 ± 0.1276	0.3678 ± 0.1500	0.3604 ± 0.1448	0.3492 ± 0.1376

Interestingly, the gMLP-S performs noticeably worse with adversarial training when evaluated on FGSM, even though fast adversarial training is similar to training the model on inputs with FGSM's perturbation. Another notable result is that when we combine both methods on ResNet-50d, we see a noticeable increase in precision, recall, and F1 with nominal accuracy costs. However, with ViT-S, we also see a decrease in performance when applying both methods.

BIM and PGD are known to have more substantial perturbation than FGSM since both attacks add more noise based on the model's gradient since it iterates several times, unlike FGSM, which is only done once. Table 5, Table 6 shows how higher k can benefit from defending against more substantial perturbation and the limited extent of adversarial training on defending perturbation since it depends on the hyperparameters used for training the model. Based on our evaluation, we show that both BIM and PGD perturbation make the models with or without adversarial training unable or barely able to predict any sample of the test set since adversarial training only provides defense on weaker perturbations due to the low ϵ used for adversarial training. DkNN allows our model to predict some test set samples with noticeable improvement for each evaluation score. With both methods combined, we achieve more significant improvements across all models, with the best results achieved by ViT-S on FGSM-RS training for both evaluations.

Table 5.

5-fold average test set evaluation with BIM perturbation (ϵ = 0.4).

Model	Train Method	DkNN	Accuracy	Precision	Recall	F1-Score
ResNet-50d	Normal	N/A	0.0120 ± 0.0110	0.0147 ± 0.0146	0.0188 ± 0.0214	0.0145 ± 0.0150
		k = 70	0.0080 ± 0.0110	0.0711 ± 0.1469	0.0104 ± 0.0181	0.0177 ± 0.0321
	FGSM-RS	N/A	0.0040 ± 0.0089	0.0035 ± 0.0078	0.0021 ± 0.0047	0.0026 ± 0.0059
		k = 100	0.6160 ± 0.0537	0.2100 ± 0.0073	0.3208 ± 0.0280	0.2538 ± 0.0144
	FGSM-GradAlign	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 90	0.5200 ± 0.2683	0.1773 ± 0.0805	0.2708 ± 0.1398	0.2133 ± 0.1049
	PGD	N/A	0.1080 ± 0.1397	0.0612 ± 0.0673	0.0563 ± 0.0728	0.0580 ± 0.0695
		k = 100	0.6360 ± 0.0089	0.2128 ± 0.0011	0.3312 ± 0.0047	0.2592 ± 0.0023
ViT-S	Normal	N/A	0.1560 ± 0.1472	0.1421 ± 0.1350	0.0938 ± 0.0966	0.1120 ± 0.1109
		k = 95	0.2840 ± 0.0219	0.1541 ± 0.0115	0.1479 ± 0.0114	0.1507 ± 0.0090
	FGSM-RS	N/A	0.0080 ± 0.0179	0.0067 ± 0.0149	0.0042 ± 0.0093	0.0051 ± 0.0114
		k = 50	0.6160 ± 0.0684	0.2779 ± 0.1568	0.3346 ± 0.0580	0.2833 ± 0.0774
	FGSM-GradAlign	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 90	0.6080 ± 0.0610	0.2089 ± 0.0085	0.3166 ± 0.0318	0.2516 ± 0.0166
	PGD	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 90	0.5480 ± 0.0795	0.2544 ± 0.0761	0.3221 ± 0.0647	0.2741 ± 0.0581
gMLP-S	Normal	N/A	0.1240 ± 0.1260	0.1354 ± 0.1293	0.0771 ± 0.0826	0.0959 ± 0.0989
		k = 1	0.1240 ± 0.1260	0.1354 ± 0.1293	0.0771 ± 0.0826	0.0959 ± 0.0989
	FGSM-RS	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 90	0.5720 ± 0.0912	0.2035 ± 0.0141	0.2979 ± 0.0475	0.2414 ± 0.0262
	FGSM-GradAlign	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 95	0.6400 ± 0.0000	0.2133 ± 0.0000	0.3333 ± 0.0000	0.2602 ± 0.0000
	PGD	N/A	0.0880 ± 0.1397	0.0490 ± 0.0726	0.0458 ± 0.0728	0.0471 ± 0.0723
		k = 55	0.6040 ± 0.0607	0.2774 ± 0.1583	0.3192 ± 0.0379	0.2637 ± 0.0399

Table 6.

5-fold average test set evaluation with PGD perturbation (ϵ = 0.4).

Model	Train Method	DkNN	Accuracy	Precision	Recall	F1-Score
ResNet-50d	Normal	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 85	0.2120 ± 0.2468	0.0918 ± 0.0927	0.1104 ± 0.1286	0.0989 ± 0.1086
	FGSM-RS	N/A	0.0440 ± 0.0607	0.0312 ± 0.0428	0.0229 ± 0.0316	0.0264 ± 0.0364
		k = 100	0.6320 ± 0.0110	0.2123 ± 0.0013	0.3291 ± 0.0057	0.2582 ± 0.0028
	FGSM-GradAlign	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 90	0.5120 ± 0.2326	0.1826 ± 0.0617	0.2667 ± 0.1211	0.2151 ± 0.0868
	PGD	N/A	0.1320 ± 0.2339	0.0567 ± 0.0879	0.0687 ± 0.1218	0.0610 ± 0.1014
		k = 100	0.4680 ± 0.2115	0.1788 ± 0.0476	0.2437 ± 0.1101	0.2042 ± 0.0730
ViT-S	Normal	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 95	0.3480 ± 0.3094	0.1284 ± 0.1050	0.1812 ± 0.1611	0.1487 ± 0.1291
	FGSM-RS	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 100	0.6440 ± 0.0089	0.2808 ± 0.1510	0.3400 ± 0.0149	0.2730 ± 0.0285
	FGSM-GradAlign	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 85	0.5880 ± 0.0657	0.2324 ± 0.0605	0.3200 ± 0.0483	0.2671 ± 0.0520
	PGD	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 40	0.6160 ± 0.0261	0.2690 ± 0.0942	0.3667 ± 0.0982	0.3024 ± 0.0818
gMLP-S	Normal	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 80	0.1080 ± 0.2198	0.0436 ± 0.0870	0.0654 ± 0.1128	0.0511 ± 0.0982
	FGSM-RS	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 90	0.6240 ± 0.0261	0.2113 ± 0.0033	0.3250 ± 0.0136	0.2561 ± 0.0067
	FGSM-GradAlign	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 85	0.6240 ± 0.0219	0.2113 ± 0.0027	0.3250 ± 0.0114	0.2561 ± 0.0056
	PGD	N/A	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000	0.0000 ± 0.0000
		k = 5	0.6400 ± 0.0000	0.2142 ± 0.0020	0.3333 ± 0.0000	0.2608 ± 0.0014

4.3. Finding the optimum k for DkNN

KNN models, in general, tend to be sensitive to the number of k chosen as the optimum to select the nearest neighbor. From the best result we have shown in Table 3, Table 4, Table 5, Table 6, we show how the best results of DkNN can either be equal, worse, or better than its non-DkNN counterpart.

Fig. 8 shows how without any perturbation during the testing, the performance of DkNN based on Accuracy in Fig. 8a, Precision in Fig. 8b, Recall in Fig. 8c, and F1 in Fig. 8d tends to decrease as we increase k due to the model having sufficient samples for each layer representation.

Figure 8.

Performance of the normally trained model based on 5-fold cross-validation for the original test set using: (a) Accuracy, (b) Precision, (c) Recall, and (d) F1.

However, as we increase the levels of perturbation, we see that generally, in Figure 9, Figure 10, Figure 11, the optimum k tends to increase as we scale up the attack from FGSM to PGD. The performance of DkNN based on Accuracy in Fig. 9a, Precision in Fig. 9b, Recall in Fig. 9c, and F1 in Fig. 9d tends to higher as the k increases. Its performance is shown in Figs. 10a, 10b, 10c, and 10d also tends to increase as we increase k. Except for PGD training, as shown in Figs. 11a, 11b, 11c, and 11d, where it does not need to use more samples due to the training scheme is the same as the attack itself with different hyperparameters between training and testing.

Figure 9.

Performance of the normally trained model based on 5-fold cross-validation for the test set with FGSM attack (ϵ = 0.4) using: (a) Accuracy, (b) Precision, (c) Recall, and (d) F1.

Figure 10.

Performance of the normally trained model based on 5-fold cross-validation for the test set with BIM attack (ϵ = 0.4) using: (a) Accuracy, (b) Precision, (c) Recall, and (d) F1.

Figure 11.

Performance of the normally trained model based on 5-fold cross-validation for the test set with PGD attack (ϵ = 0.4) using: (a) Accuracy, (b) Precision, (c) Recall, and (d) F1.

We believe this result is due to DkNN generally favoring more samples for calculating nearest neighbors for each representation to provide better results due to perturbation effects. Using DkNN on adversarially trained models, we see that it can reduce the optimum k for certain perturbations with like FGSM as shown in 4, and 5 with most models.

With a combination of adversarial training and DkNN, we see that in Figure 8, Figure 9 the optimum k tends to be identical or lower for most models. In higher perturbations like Figure 10, Figure 11, the evaluation shows that the performance still aligns similarly to the normally trained models that it benefits more from higher k. As we increase the k, we see that several models that are adversarially trained have significant improvement over the normally trained models, where the results are closer to the normal evaluation result.

The trend follows when we check the correlation between the scores and hyperparameters used for the model and attack. In Table 7, we see that there is a very high negative correlation between k and the evaluation scores, which means that the effects of increasing k harm the model performance as we increase k. Table 8 shows that DkNN is able to increase several models' performance significantly on gMLP-s and minorly on ViT-S. However, the performance on most ResNet-50d models shows that with a combination of adversarial training can harm the performance. As we evaluate further by increasing the attack's ϵ on FGSM, we did not see any further changes in the model's evaluation.

Table 7.

Correlation between testing score, DkNN's k, and adversarial attack parameter ϵ (No Attack).

Train Method	Model	Parameter	Accuracy	Precision	Recall	F1-Score
Normal	ResNet-50d	k	-0.8872	-0.8172	-0.9221	-0.9227
		ϵ	N/A	N/A	N/A	N/A
	ViT-S	k	-0.8277	-0.8577	-0.8739	-0.8816
		ϵ	N/A	N/A	N/A	N/A
	gMLP-S	k	-0.7415	-0.6964	-0.8487	-0.8316
		ϵ	N/A	N/A	N/A	N/A
FGSM-RS	ResNet-50d	k	-0.0839	-0.5169	-0.6290	-0.6842
		ϵ	N/A	N/A	N/A	N/A
	ViT-S	k	-0.5625	-0.4958	-0.6520	-0.6663
		ϵ	N/A	N/A	N/A	N/A
	gMLP-S	k	-0.0769	-0.4596	-0.5237	-0.5764
		ϵ	N/A	N/A	N/A	N/A
FGSM-GradAlign	ResNet-50d	k	-0.4852	-0.7595	-0.7537	-0.7902
		ϵ	N/A	N/A	N/A	N/A
	ViT-S	k	-0.4862	-0.6215	-0.6105	-0.6660
		ϵ	N/A	N/A	N/A	N/A
	gMLP-S	k	-0.5126	-0.4698	-0.6801	-0.6896
		ϵ	N/A	N/A	N/A	N/A
PGD	ResNet-50d	k	-0.2815	-0.5172	-0.7135	-0.7431
		ϵ	N/A	N/A	N/A	N/A
	ViT-S	k	-0.4893	-0.4776	-0.6080	-0.6168
		ϵ	N/A	N/A	N/A	N/A
	gMLP-S	k	-0.5717	-0.5466	-0.6830	-0.6958
		ϵ	N/A	N/A	N/A	N/A

Table 8.

Correlation between testing score, DkNN's k, and adversarial attack parameter ϵ (FGSM).

Train Method	Model	Parameter	Accuracy	Precision	Recall	F1-Score
Normal	ResNet-50d	k	0.7972	-0.0660	0.7642	0.4884
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.2665	-0.0329	0.1871	0.0767
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.3666	-0.0470	0.2561	0.1394
		ϵ	0.0000	0.0000	0.0000	0.0000
FGSM-RS	ResNet-50d	k	-0.3317	-0.6663	-0.6613	-0.6950
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.2334	-0.3104	-0.0442	-0.2537
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.6756	-0.0702	0.6270	0.3817
		ϵ	0.0000	0.0000	0.0000	0.0000
FGSM-GradAlign	ResNet-50d	k	-0.2282	-0.6132	-0.6591	-0.7147
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.1504	-0.1222	0.1315	0.0627
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.5675	-0.1863	0.3623	0.1283
		ϵ	0.0000	0.0000	0.0000	0.0000
PGD	ResNet-50d	k	0.3045	-0.5412	-0.5144	-0.5733
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.3453	-0.1866	0.0116	-0.2306
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.5962	-0.0069	0.1769	-0.0971
		ϵ	0.0000	0.0000	0.0000	0.0000

We see the benefits of DkNN performing at its optimum on iterative attacks. Table 9 shows many high positive correlations as we increase k for many models, especially with ResNet-50d on PGD and FGSM-GradAlign training, which benefits the most from using DkNN. In contrast, as shown in Table 9, we did not see any correlation as we increased the ϵ on the BIM attack. A similar pattern follows on evaluation with PGD attack shown in Table 10, we see that ResNet-50d benefits from the DkNN. Oddly, on specific models, the increase of perturbation ϵ on PGD attacks has a minimum correlation that may improve the model's evaluation result.

Table 9.

Correlation between testing score, DkNN's k, and adversarial attack parameter ϵ (BIM).

Train Method	Model	Parameter	Accuracy	Precision	Recall	F1-Score
Normal	ResNet-50d	k	0.1772	0.1432	-0.1637	0.0389
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.2530	-0.0407	0.1576	0.0728
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.0473	-0.1758	-0.0390	-0.0842
		ϵ	0.0000	0.0000	0.0000	0.0000
FGSM-RS	ResNet-50d	k	0.5468	0.4777	0.4794	0.5225
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.5298	0.0462	0.4092	0.2322
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.5505	0.3172	0.4683	0.4574
		ϵ	0.0000	0.0000	0.0000	0.0000
FGSM-GradAlign	ResNet-50d	k	0.6293	0.6659	0.6918	0.6666
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.6019	0.4645	0.5932	0.5652
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.5384	0.3579	0.4781	0.4442
		ϵ	0.0000	0.0000	0.0000	0.0000
PGD	ResNet-50d	k	0.8729	0.8290	0.8549	0.8537
		ϵ	0.0000	0.0000	0.0000	0.0000
	ViT-S	k	0.7048	0.2157	0.4900	0.4801
		ϵ	0.0000	0.0000	0.0000	0.0000
	gMLP-S	k	0.3948	0.0831	0.2336	0.1787
		ϵ	0.0000	0.0000	0.0000	0.0000

Table 10.

Correlation between testing score, DkNN's k, and adversarial attack parameter ϵ (PGD).

Train Method	Model	Parameter	Accuracy	Precision	Recall	F1-Score
Normal	ResNet-50d	k	0.5524	0.5830	0.5518	0.5683
		ϵ	0.2731	0.2410	0.2737	0.2580
	ViT-S	k	0.5390	0.5458	0.4984	0.5325
		ϵ	0.1659	0.1815	0.1859	0.1819
	gMLP-S	k	0.3301	0.3573	0.3617	0.3523
		ϵ	0.0426	0.0472	0.0636	0.0490
FGSM-RS	ResNet-50d	k	0.5086	0.3065	0.2784	0.3716
		ϵ	0.0352	0.0651	0.1536	0.0840
	ViT-S	k	0.5145	-0.0668	0.3899	0.1896
		ϵ	-0.0093	0.0032	-0.0019	-0.0127
	gMLP-S	k	0.4127	0.0992	0.2185	0.2411
		ϵ	0.1472	0.1845	0.2092	0.1944
FGSM-GradAlign	ResNet-50d	k	0.6163	0.5619	0.6310	0.6232
		ϵ	0.1587	0.0858	0.1505	0.1423
	ViT-S	k	0.4443	0.0751	0.3038	0.2443
		ϵ	0.2879	0.2606	0.2839	0.2770
	gMLP-S	k	0.4059	0.3142	0.3768	0.3860
		ϵ	-0.0629	-0.0757	-0.0615	-0.0750
PGD	ResNet-50d	k	0.6394	0.4457	0.3152	0.4679
		ϵ	0.1547	0.1526	0.1821	0.1584
	ViT-S	k	0.3644	-0.0391	0.1856	0.0954
		ϵ	0.0385	-0.0059	0.0327	0.0401
	gMLP-S	k	0.1817	-0.1997	-0.0105	-0.1096
		ϵ	-0.0412	-0.0030	-0.0246	-0.0314

5. Conclusion

We have shown that adversarial training and DkNN can improve the model's robustness in glaucoma severity detection. The results show that given optimum k, a model with DkNN can perform similarly with minor differences when no adversarial attacks occur. When evaluated with adversarial attacks, we have shown that for most models, it can further improve the adversarial defense. However, further analysis will be required on a larger dataset, given that the results shown are using models trained and evaluated on a small dataset. The results may differ when DkNN has more representations stored and calibrated with greater subsets of data. Other factors include the model's performance when given perturbed training and calibration data representations stored in DkNN. We will leave these as future work.

Declarations

Author contribution statement

Suyanto Suyanto: Conceived and designed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data.

Lalu M. Riza Rizky: Performed the experiments; Analyzed and interpreted the data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

The data that has been used is confidential.

Declaration of interest's statement

The authors declare no competing interests

Additional information

No additional information is available for this paper.