Classification of Infection and Ischemia in Diabetic Foot Ulcers Using VGG Architectures

Abstract

Diabetic foot ulceration (DFU) is a serious complication of diabetes, and a major challenge for healthcare systems around the world. Further infection and ischemia in DFU can significantly prolong treatment and often result in limb amputation, with more severe cases resulting in terminal illness. Thus, early identification and regular monitoring is necessary to improve care, and reduce the burden on healthcare systems. With that in mind, this study attempts to address the problem of infection and ischemia classification in diabetic food ulcers, in four distinct classes. We have evaluated a series of VGG architectures with different layers, following numerous training strategies, including k-fold cross validation, data pre-processing options, augmentation techniques, and weighted loss calculations. In favor of transparency and reproducibility, we make all the implementations available through the Generally Nuanced Deep Learning Framework (GaNDLF, github.com/CBICA/GaNDLF. Our best model was evaluated during the DFU Challenge 2021, and was ranked 2^nd, 5^th, and 7^th based on the macro-averaged AUC (area under the curve), macro-averaged F1 score, and macro-averaged recall metrics, respectively. Our findings support that current state-of-the-art architectures provide good results for the DFU image classification task, and further experimentation is required to study the effects of pre-processing and augmentation strategies.

1. Introduction

Diabetic foot ulcers (DFUs) are the most common complications of diabetes mellitus that usually take long time to heal and are among the leading causes of hospitalization and morbidity of patients with diabetes [1,2]. According to published estimates, DFU accounts for roughly 20% of hospital admissions of diabetic patients [3]. In addition, DFU leads to substantial emotional, physical, and financial distress that deteriorates the quality of life of patients and their caregivers [4]. If not managed properly, DFU combined with ischemia and infection can cause gangrene, lower limb amputation, and even death [1,2].

For healthcare systems with limited resources, DFU diagnosis can impose a substantial economic burden. As such, the clinical translation of computational methods for the automated assessment of DFU could be beneficial for all stakeholders in the healthcare system, namely, the clinical sites, patients, and caregivers. Such translation could specifically contribute in the early detection of DFU, as well as in the regular monitoring of patients’ condition by themselves or by their caregivers. For this purpose, several mobile device applications have been designed and developed for standardization and collection of DFU images, and for promoting self-care of DFU [5-8].

Recent advances in the fields of computer vision and machine learning have had a growing impact on medical imaging, including radiology, histopathology, and dermatology [9-25]. Of special note is the proliferation of several deep learning (DL) methods, which have shown superior performance in numerous computer vision and medical image computing tasks [26-31]. Lately, it is also shown that DL models succeed in classification, detection, and segmentation tasks on DFU images [32-38]. Specifically, previous work from Goyal et al. [33] focuses on the recognition of ischemia and infection on DFU images, but their work tries to solve binary classification problem of ischemia (ischemia vs. non-ischemia) and infection (infection vs. non-infection) separately.

This study aims to solve a multi-class (4-class) classification problem for Diabetic Foot Ulceration (DFU), by leveraging the GenerAlly Nuanced Deep Learning Framework (GaNDLF)¹ [39]. The specific 4 classes considered are: i) infection, ii) ischemia, iii) both infection & ischemia, and iv) controls (i.e., neither infection, nor ischemia) (Fig. 1), as defined by the DFU Challenge (DFUC) 2021 [40], conducted in conjunction with the Annual Scientific Meeting of Medical Image Computing and Computer Assisted Interventions (MICCAI) 2021. GaNDLF facilitated our work by providing simple application programming interfaces to rapidly and robustly incorporate techniques such as cross-validation [41], data pre-processing, data augmentation, and weighted loss calculation into our experimental design. Our best model, with which we participated at the DFUC 2021, was compared to the baseline models provided by Yap et al. [40], and was ranked in the 2^nd, 5^th, and 7^th place in the DFUC 2021, based on the macro-averaged AUC (area under the curve), macro-averaged F1 score, and macro-averaged recall metrics, respectively.

Fig. 1.

Examples from each class considered in the DFUC2021 challenge. GT: ground truth, pred: prediction of our best model VGG11 5-fold

2. Materials and Methods

In this section, we describe the provided dataset in detail, illustrate the examples of infection, ischemia, both infection & ischemia, and control cases from DFU patients with images. Additionally, we explain the methods used in the work, their configuration, the overall VGG architecture, and the various training strategies we followed.

2.1. DFU Dataset

The DFUC2021 dataset describes a multi-institutional collection for analysis of specific pathologies, focusing on infection and ischemia [40]. Specifically, Manchester Metropolitan University and Lancashire Teaching Hospitals established a repository that contains DFU images of infection and ischemia cases for the purpose of supporting research on more advanced methods of pathology detection and recognition of DFU. These DFU images are collected from Lancashire Teaching Hospitals, where photographs were taken from patients during their clinical visits. The three cameras used for capturing the foot images are Kodak DX4530, Nikon D3300 and Nikon COOLPIX P100. The images taken were close-ups of the whole foot from a distance of approximately 30–40 cm with parallel inclination to the ulcer plane. Thereafter, the DFU regions are cropped from the original images and natural data augmentation is performed by preserving the case ids and splitting them to train and test sets.

The complete DFUC2021 dataset comprises of a total of n = 15, 683 DFU images. The provided ground truth labels, defining the four classes considered by DFUC2021 are i) infection, ii) ischemia, iii) both infection & ischemia, and iv) controls (i.e., neither infection, nor ischemia). Representative example cases for each class are shown in Fig. 1. To quantitatively evaluate the performance of algorithms developed for this task, the complete set of n = 15, 863 images are partitioned into three distinct independent subsets. The training set includes n = 5, 955 images, provided with their ground truth labels dividing the training set in n = 2, 555 cases with only infection, n = 227 cases with only ischemia, n = 621 cases with both infection & ischemia, and n = 2, 552 control cases. For algorithmic evaluation on unseen data, the DFUC2021 dataset further provides n = 5, 734 cases for testing, and n = 3, 994 unlabeled cases. The utilization of unlabelled data for the ischemia and infection classification is left for future work.

2.2. DL Framework

We leveraged the GenerAlly Nuanced Deep Learning Framework (GaNDLF)² [39] to conduct all experimentation and training for this study. GaNDLF has been developed in Python using the well-known DL library PyTorch [42]. It enables researchers to target various machine learning (ML) and artificial intelligent (AI) workloads (such as segmentation, regression, classification, and synthesis) using different types of imaging modalities (such as RGB, radiographic, and histopathologic imaging), by providing a complete end-to-end solution for training and deploying robust DL models [39]. GaNDLF makes DL accessible for researchers who do not have extensive experience in designing and implementing DL pipelines, while making it straightforward for computational researchers to make their algorithms available for a wider array of applications. It also aims to deploy DL workflows in clinical environments with relative ease. With properties such as an end-to-end application programming interface, ease of training robust and generalizable models with different configurations, robust data pre-processing & augmentation techniques, and nested k-fold cross validation, GaNDLF aims to fill gaps of other popular DL libraries. This allows a user to easily design different experiments by simply editing text parameter files without requiring any additional coding, and when combined with the rich metrics library for validating trained models, GaNDLF provides deeper insights into model robustness.

2.3. Architecture

We trained three versions of the VGG architecture [43] for the DFUC2021, namely the VGG11, VGG16, and VGG19 [43]. VGG architectures use very small convolutional filters, and apply spatial padding with the intention of preserving the original resolution of the input image. A total of 5 max-pooling operations are performed over a 2 × 2 window size, with a stride of 2, to ensure that each max-pooling operation reduces both the image height and width in half. The classifier component of our VGG variants use the ReLU activation function [44], along with a global average pooling [45], two drop-out layers, and a penultimate linear layer. To ensure that the network performs classification, a softmax layer is added as the final layer, which enables us to extract the likelihood for each class. Schematic representations for the overall architecture of VGG11, VGG16, and VGG19, can be found in Figs. 2, 3, and 4, respectively.

Fig. 2.

VGG11 architecture (Figure constructed using PlotNeuralNet [46])

2.4. Training

We used two distinct approaches to train the model. First, we consider a more clinically oriented paradigm, and we split the training data into two equal halves, using one as a retrospective/discovery cohort (i.e., training) and the other as an unseen prospective/replication cohort (i.e., blinded validation). For this partitioning, all subjects of each label/class were proportionally and randomly divided across the 2 halves (retrospective/prospective). This approach was used to yield our baseline results and enabled us to tune the hyperparameters of the model for the task at hand.

Once we obtained these baseline results, we considered a more computationally oriented paradigm, to ensure generalizability of the trained model and prevent overfitting. We specifically employed a k-fold cross validation schema [41], which is a technique widely-used in ML to ensure reporting unbiased performance estimates, and help capture information from an entire dataset by training k different models on k corresponding non-overlapping folds/subsets of the complete training data. Using k-fold cross-validation one can test the model’s ability to make accurate predictions on unseen data, detect problems like overfitting or selection bias [47], and provide an understanding on how well the model will generalize to the real distribution of data. For all experiments that used cross-validation, we set the number of folds as k = 5. We have performed an equal non-intersecting 5-way split of the training data, ensuring that the model trains on the full training data without overfitting.

Based on the knowledge of the model hyperparameters obtained from our initial experiment using the labelled training data split in equal halves, we proceeded to split the training data into a set of n = 5 randomized splits. Each split was used to train the VGG11, VGG16, and VGG19 variants of the network architectures. During the training step, a single patch of 128 × 128 is extracted from a random location from each image and processed by the model for loss back-propagation. For the forward pass of the model (i.e., the validation/inference phases), enough patches of the specified size are generated from each image to ensure that every pixel is processed at least once by the model, and the final prediction is generated by averaging all the predictions from each patch. To generate the final predictions during the testing phase, we have averaged predictions from every fold. The batch size was chosen as b = 256 for VGG11 and VGG16 architectures, and b = 128 for VGG19 to ensure maximal utilization of the available hardware resources. For each model architecture, configurations with a standard set of data augmentations and data pre-processing techniques were also evaluated. For data augmentation, bias, blur, noise, and swapping techniques are used with a maximum probability p = 0.5. For data pre-processing, a z-scoring normalization mechanism was used. The choice of the loss function stayed the same across these experiments, as the original VGG architecture, which is softmax followed by categorical cross entropy loss, and has shown to work better for multi-class classification problems [48].

To increase the variability of our experiments, we further used weighted cross-entropy loss [49], which ensures that misclassifications of the class with the smallest number of labels generates the largest loss, and could thus be better suited for datasets with imbalanced classes. We used ADAM [50] as the optimizer with an initial learning rate of 0.001. For the half split configurations, the set the total number of epochs to n = 200 and set the patience value to n = 50. For the 5-fold cross validation configurations, we set the number of epochs to n = 300 and patience value to n = 50. All of the experimentation was performed on a high performance computing cluster with NVIDIA P100 GPU (which has 11 GB of dedicated video memory), using 32 GB RAM, and 10 CPU threads.

3. Results

In this section, we first analyze the validation performance of the models trained with the different training strategies. We perform our final model selection based on the performance of our partitioned validation set, and submit the inference results from the best model for the challenge evaluation and ranking. Thereafter, we analyze our best models’ performance in terms of macro-averaged F1 score, and Macro-AUC, macro-averaged recall and accuracy.

3.1. Training/Validation Dataset Performance

We conducted n = 12 different experiments with various training strategies and VGG architectures (Table 1). The models trained without any data augmentation and pre-processing and without using the weighted cross entropy loss outperformed the rest. The overall best model, in terms of validation loss and validation accuracy, was the standard VGG11 architecture trained using 5-fold cross-validation. This model, trained without any data augmentation and pre-processing, was able to reach an average cross entropy loss of 0.24 after averaging over the outputs of all the folds. Table 1 includes a detailed overview of the training results.

Table 1.

Validation performance of VGG variations. DA: Data augmentation, DP: Data pre-processing, WCE: Weighted Cross-Entropy. Bold values imply the best performance and the underlined values imply the models that are selected for experimenting in real test set

Model architecture	Training strategy	Validation loss
VGG11	Half-Split	0.49
VGG16	Half-Split	0.51
VGG19	Half-Split	0.59
VGG11	5-fold (DA, DP)	0.52
VGG16	5-fold (DA, DP)	0.63
VGG19	5-fold (DA, DP)	0.72
VGG11	5-fold	0.24
VGG16	5-fold	0.30
VGG19	5-fold	0.31
VGG11	5-fold (WCE)	0.27
VGG16	5-fold (WCE)	0.39
VGG19	5-fold (WCE)	0.52

3.2. Testing Dataset Performance

Once we obtained results for all our cross-validated experiments (Table 1), we measured the performance of our best 6 models with the DFUC2021 testing data, as provided by the challenge organizers. The best configuration, in terms of accuracy, macro-averaged F1 score, and Macro-AUC, was the standard VGG11 architecture with 5-fold cross validation. Our best configuration placed in the 5^th place in the DFU Challenge 2021, where the participants were ranked according to macro-averaged F1 score. In addition, we analyzed our results according to macro-averaged AUC and macro-averaged recall metrics. Our model ranked in the 2^nd and 7^th place, respectively (for details, please see the official DFU leaderboard³). Considering that the DFUC2021 dataset is an imbalanced set (i.e., the number of samples for each of the classes are not similar), it was surprising to us that standard VGG11 model trained without any pre-processing, data augmentation, and weighted cross-entropy outperformed the other configurations we tested during both validation and testing phases of the challenge. This may require some future meta-analysis to gain a deeper understanding on the exact driving factors. We conducted a visual demonstration of our model’s performance on the test set with example samples given by the providers. Samples with ground truth values and predictions can be found in Fig. 1. A detailed illustration of these results can be found at Table 2.

Table 2.

Test set performance of the 6 best models, selected according to the average performance of during cross validation training.

Model	Accuracy	Macro-AUC	Macro-Recall	Macro F1-Score
VGG11 5-fold	0.640	0.870	0.576	0.561
VGG16 5-fold	0.617	0.869	0.575	0.543
VGG19 5-fold	0.615	0.870	0.572	0.551
VGG11 5-fold WCE	0.624	0.845	0.561	0.541
VGG16 5-fold WCE	0.601	0.858	0.583	0.534
VGG19 5-fold WCE	0.595	0.859	0.562	0.521

We also conducted an analysis based on the class-individual F1 scores. The standard VGG11 model performed well based on the ischemia F1 score and control F1 score, achieving ranks of 3^rd and 5^th, respectively. On the other hand, the performance based on infection F1 score and both F1 score was relatively poor, with rank of 8^th for both metrics. Additionally, we compared the VGG19 5-fold model’s performance with the other models in the challenge leaderboard and observe that it had the highest 2^nd and 5^th score based on the results on ischemia F1 score and both F1 score, respectively. Additional details can be found in Table 3.

Table 3.

Class-individual F1 scores of the selected models.

Model	Infection F1-Score	Ischemia F1-Score	Both F1-Score	None F1-Score
VGG11 5-fold	0.547	0.522	0.440	0.736
VGG16 5-fold	0.493	0.531	0.424	0.723
VGG19 5-fold	0.475	0.548	0.461	0.719
VGG11 5-fold WCE	0.521	0.488	0.428	0.726
VGG16 5-fold WCE	0.448	0.515	0.462	0.712
VGG19 5-fold WCE	0.417	0.502	0.450	0.716

4. Discussion

In this study, we modified a well-known DL neural network architecture, namely VGG, to classify images containing diabetic foot ulcerations (DFUs), into infection, ischemia, both infection & ischemia, and control. Classification of ischemia and infection of DFU patients is an important task which would help early diagnosis and prevent serious illness in the future. Although 4 classes can be considered as small, the training dataset is not balanced, which makes it harder to learn for the classes with small number of samples. Our results indicate that the best approach, in the set of experiments we performed, was the one that did not rely upon weighted loss calculation or any pre-processing methods.

All architectures evaluated in this study were implemented in GaNDLF, which is a high level framework for training robust DL models. For training, we used n = 12 different configurations of the general VGG architecture, with different number of weight layers and different training strategies. The number of weight layers were 11, 16 and 19. The two major training strategies we leveraged was 1) splitting the training data into retrospective/training and prospective/-validation datasets, as halves, and 2) following a 5-fold cross validation schema. Effect of data pre-processing and augmentation was explored and quantified in these experiments. It is observed that the 5-fold configuration of the VGG11 without any data augmentation or pre-processing performed the best out all experiments, with an average loss of 0.24 across 5 folds. We hypothesise that VGG16 and VGG19 architectures performed worse than VGG11 because they are simply too large for the given task, and addition of more data is required to properly optimize their weights. The best model in our experimentation was ranked as the 5^th place in DFU Challenge 2021, in which the participants were ranked according to macro-averaged F1 score. In addition, our best model was ranked as 2^nd and 7^th place based on macro-averaged AUC and macro-averaged recall metrics, respectively. Additionally, we further analyzed the class-individual F1 scores and observe that even though our model has performed better on ischemia F1 score and both F1 score, it has performed poorly on infection F1 score and both F1 score metrics.

We believe that there is plenty of room for improvement and further analysis in related future work. Especially, by taking RGB-specific augmentations [51] and pre-processing methods [52] into account, we would expect to significantly improve the model performance. Considering DFU 2021 dataset is imbalanced, exploration of the reasons why weighted cross-entropy loss did not increase the performance would be insightful. Additionally, exploration of custom loss functions, which are specifically designed for optimizing the macro-averaged F1 score could improve the performance [53]. We would also like to see how other popular architectures would perform on this task. Experimentation with architectures like residual networks [54], Efficient-Net [55], Xception-Net [56], InceptionRes-Net [57] and vision transformers [58] would be insightful. Finally, the use of the unlabelled datasets (n = 3, 994) to augment the training using weakly supervised training [59] could also help the model performance.

5. Conclusions

DFU can cause critical health problems when it is combined with ischemia and infection. Thus, early diagnosis of potential severe can save lives, and contribute to improvement in the quality of life for all stakeholders in the healthcare system. Automated computational approaches targeting on providing classification suggestions could contribute in early disease detection and the management of DFU patients. Our proposed VGG11 model, trained using a 5-fold cross validation configuration, without any data augmentation or pre-processing, demonstrated superior performance when compared to the other evaluated models, and placed in the 5^th place on DFU Challenge 2021, where the rankings were determined by the macro-averaged F1 score. Future work, towards further improving our obtained results, should explore custom loss functions, RGB-specific augmentations (for example, using contrast, brightness, and scale augmentations) along with RGB-specific pre-processing.

A. Illustrations of Various VGG Variants

A.1. VGG16

Fig. 3.

VGG16 architecture (Figure constructed using PlotNeuralNet [46])

A.2. VGG19

Fig. 4.

VGG19 architecture (Figure constructed using PlotNeuralNet [46])