Multiclassification Analysis of Volumetric, Protocol, and Application Layer DDoS Attacks

Abstract

In today’s digital landscape, Distributed Denial of Service (DDoS) attacks represent a constantly evolving and significant threat, interrupting online services via many attack vectors. These attacks universally aim to render websites and services inoperable. Developing effective detection strategies is imperative as DDoS attacks become more frequent, varied, and destructive. This paper introduces an in-depth multiclassification analysis of DDoS attacks, categorizing them into Volumetric, Protocol, and Application Layer attacks. Utilizing the extensive CICDDoS2019 dataset, which includes eleven specific DDoS attack variations, we systematically classify these variations. Our analysis employs six diverse Machine Learning (ML) models: Logistic Regression (LR), Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), Neural Networks (NN), and Extreme Gradient Boosting (XGB). We aim to identify the most effective model for predicting DDoS traffic across each attack category and to ascertain the model with superior overall performance. The findings of this study provide valuable insights into the effectiveness of various ML techniques in countering DDoS attacks, thereby contributing to the fortification of digital infrastructures against this pervasive threat.

1. Introduction

Recently, there has been much growth in networked devices in use globally and an apparent rise in demand. People, businesses, and governments are increasingly relying on this technology. While many benefits come from this, it opens the door to many potential vulnerabilities and attack types [1]. One of the most prominent attack types seen regularly is distributed denial-of-service (DDoS) attacks [2][3]. A distributed denial of service (DDoS) attack constitutes a malicious effort to render an online service inaccessible to users, typically achieved by temporarily disrupting or suspending the functions of its hosting server [4]. While these attacks have a simple enough concept, their effects can be detrimental to a person or company. With a few compromised networked devices, malicious traffic can be sent to disrupt and crash web-based or network resources. Moreover, DDoS attacks can be difficult to detect manually as there are many possible strategies for carrying out these attacks. They can be implemented at different layers within the TCP/IP model, such as application, transport, and network layers) and exploit several protocols, such as UDP, TCP, SNMP, and LDAP.

As a result, these attacks can swiftly render the services of a target effectively useless until a remediation process can be completed. This can be extremely difficult depending on the severity, type of DDoS attack, and the victim. Mirai is a malicious software that infects smart devices using ARC processors, transforming them into remotely controlled bots or “zombies.” These infected devices form a network known as a botnet, commonly utilized for initiating DDoS attacks [5][6]. In September 2016, Mirai executed its inaugural major assault targeting a French technology firm, OVH. The attack reached an unparalleled peak of 1 terabit per second (Tbps) and is approximated to have engaged approximately 145,000 devices in the offensive maneuver [7]. AWS reported mitigating a massive DDoS attack in February 2020. At its peak, this attack saw incoming traffic at 2.3 terabits per second (Tbps) [8]. A notable example of a severe DDoS attack against Google Cloud occurred in August 2023 [9][10]. This attack is noted to be one of the largest attacks in history, with its peak traffic volume of 398 million requests per second (RPS), indicating that the capabilities of attackers are growing rapidly, which must be addressed by creating new DDoS detection methods with fast and accurate performance. By contrast, in 2022, the largest recorded DDoS attack peaked at 46 million reps [11].

This paper performs two multi-class classification experiments based on this growing concern. Using the CICDDoS2019 dataset, we have grouped the available attack data into Volumetric, Protocol, and Application Layer DDoS attacks [22]. Within these three categories, there are eleven different types of DDoS attacks, which are DNS Flood, UDP Flood, UDP Lag, SYN Flood, TFTP, LDAP, MSSQL, NetBIOS, NTP, SNMP, and SSDP Attacks. Before analyzing our ML models, we performed feature selection to identify some of the most influential and relevant features contributing to detecting an attack. After selecting these features, we then analyze and compare the performance of six ML algorithms on our datasets to find three answers to two fundamental questions: (1) which algorithm can perform best for each of the three DDoS attack categories; (2) which algorithm can perform best when classifying based on the eleven individual attack types and then explore how these happen. The ML algorithms we use are Logistic Regression (LR) [25], Decision Tree (DT) [26], Random Forest (RF) [27], Support Vector Machine (SVM) [28], K-Nearest Neighbors (KNN) [29] and Extreme Gradient Boosting (XGB) [30].

2. Related Work

Recent studies have explored various machine learning techniques for detecting DDoS attacks, emphasizing accuracy, efficiency, and robustness [12][13]. Approaches include Higher Order Singular Value Decomposition, Random Forest, and Gradient Boosting with datasets like CICDDoS2019 and BoT-IoT [23]. Some focus on IoT networks, employing unique feature extraction methods and classifiers. Others explore neural networks and advanced algorithms like CatBoost for traffic classification [24]. In this section, we highlight the evolving nature of DDoS attack detection, with a trend towards high-accuracy, real-time capable models.

The study in [14] presents a new DDoS attack detection method using higher-order singular value decomposition and machine learning techniques like decision trees, achieving better accuracy and lower false positives. It highlights the method’s robustness against errors but notes its higher computational demand, suggesting a focus on real-time application viability and future exploration of deep learning algorithms. In [16], Batchu and Seetha’s research focuses on improving DDoS attack detection in the face of increasing network traffic and sophisticated attack methods. Addressing data quality issues affecting model accuracy, they utilize the CICDDoS2019 dataset with KNORA-E and KNORA-U algorithms, achieving exceptional accuracy between 99.9878% and 99.9909%. Their study thoroughly compares their model with other methods, demonstrating its superior accuracy, recall, precision, and efficiency, positioning it as an effective solution for DDoS attack detection in diverse. Chu et al. [15] utilize a Random Forest model to detect DDoS attacks in vulnerable network servers, focusing on the transport layer. This model, featuring 24 key attributes, outperforms previous ones with a 97% accuracy rate across various attack types. It shows high precision, recall, and F1 scores, particularly effective in HTTP attack detection. The study emphasizes the model’s simplicity and accuracy, illustrating the effectiveness of their approach in DDoS attack detection.

In [17], Ma et al. developed a comprehensive DDoS detection model termed “feature and model selection” (FAMS). This methodology, divided into data preprocessing, feature selection, machine model selection, and parameter optimization, resulted in a Random Forest model with a notable 99.99% accuracy and swift processing time (0.216050 seconds). FAMS’s structured, multi-stage approach ensures high accuracy and offers adaptability for future models and datasets, showcasing its practicality and efficiency in optimizing machine-learning models for DDoS detection. Parfenov et al. [18] focus on a machine learning method to classify network traffic in IoT networks, distinguishing benign activities from DDoS attacks. The study introduces a novel approach using altered hash values in device configuration files for identifying malicious activities, highlighting their method’s effectiveness, especially with CatBoost and Gradient Boosting algorithms, achieving high scores in precision, recall, and F1 metrics. The research concludes successfully, with plans to refine the accuracy for multiclass classification scenarios.

In [19], Talaei Khoeiand and Kaabouch thoroughly evaluated nine machine learning models to identify the most effective for specific tasks. The study highlights the total length of forward packets, flow byte, flow packet, flow IAT mean, and flow IAT standard deviation as key features. Among the models, Alex Neural Networks (ANN) and Variational Autoencoder (VA) stand out, with ANN achieving a higher accuracy of 98.71% and faster processing time (1.29 seconds) compared to VA’s 96.7% accuracy and 3.3 seconds processing time. In [20], Li et al. developed a strategy for detecting volumetric DDoS attacks, focusing on volume-based attacks like HTTP and TCP/UDP floods. It identifies key characteristics such as traffic size, source information, and response time to malicious traffic. The approach uses sliding time windows and information entropy for accuracy. The model, employing a Long Short-Term Memory (LSTM) system, undergoes a three-stage process of traffic analysis, entropy calculation, and attack detection. The results show high accuracy and low false positives, with future research directions including IoT network applications and varied DDoS attack types. In [19], Navruzov and Kabulov tackle detecting DDoS attacks in large networks, highlighting difficulties due to device diversity and high traffic. It proposes using data mining for enhanced security models, focusing on big data preprocessing and machine learning. The study uses the CICDDoS2019 dataset to analyze relationships between attack types and normal traffic, underscoring the importance of feature stability. Findings indicate that Random Forest and K-NN are effective for DDoS detection, especially when feature selection is stability-focused.

3. Volumetric, Protocol, and Application Attack Taxonomy

Utilizing the CIC-DDoS2019 dataset, we introduce a novel taxonomy to categorize eleven types of attack packets into three distinct classes: volumetric, application, and protocol attacks [22], which are explained in the following.

Volumetric Attacks:

This category includes DNS, NTP, UDP, UDP-Lag, and Syn attacks. The unifying characteristic of these attacks is their strategy to overwhelm a network with a deluge of illegitimate packets. By grouping these attacks, we aim to refine the model’s capability to effectively classify volumetric attacks, which might be less discernible when treated as separate entities.

Application Layer Attacks:

This classification encompasses TFTP and MSSQL attacks. MSSQL attacks specifically exploit a system’s SQL server to deplete resources, while TFTP attacks manipulate the trivial file transfer protocol. Both target the application layer of a computer’s OSI model. Grouping these attacks enhances the model’s precision in identifying threats targeting the application layer.

Protocol Attacks:

This category includes SSDP, SNMP, NetBIOS, and LDAP attacks. Although targeting the application layer, they are distinguished by their method of monopolizing system resources using protocols, resulting in a packet structure different from application layer attacks. This categorization is designed to improve the model’s generalization capabilities in identifying new types of protocol-based attacks.

The SDG Classifier was employed in our analysis to extract feature significance across the established attack categories. Figure 1 presents an in-depth breakdown of how the machine-learning model interprets various attack types. This includes a mapping of attack-type labels to coefficients using a predefined dictionary. The top 5 features with the highest absolute weights are identified for each attack type. These weights are crucial as they indicate the features’ substantial impact on the prediction accuracy of the attack. The absolute weight of each feature quantifies its contribution, with higher values signifying a more pronounced influence on the model’s decision-making process. Furthermore, the mean values of these significant features in the training dataset are provided. These averages offer valuable insights into the typical characteristics associated with each type of attack, thereby enhancing our understanding of the attack patterns. This detailed exposition not only improves the transparency of the model but also serves as a vital resource for cybersecurity professionals. It aids them in comprehending the relevance and impact of specific features, thereby bolstering effective threat-detection strategies.

Fig. 1.

Breakdown of the machine learning’s interpretation for different attack types

4. Methodology

Various datasets are currently utilized in developing Machine Learning models for DDoS attack detection. Among these, the CIC-DDoS2019 dataset stands out due to its comprehensive nature. It encompasses a substantial amount of data, approximately 30 GB, and includes a detailed array of 88 distinct features. This dataset is particularly notable for having eleven different types of DDoS attacks, providing a broad spectrum for analysis and model training. The covered attack types are NTP, DNS, LDAP, MSSQL, NetBIOS, SNMP, SSDP, UDP, UDP-Lag, Syn, and TFTP. The diversity and volume of data in the CIC-DDoS2019 dataset make it an invaluable resource for developing robust and effective DDoS detection algorithms in ML.

The optimization of the CIC-DDoS2019 dataset was our initial focus, aiming to ensure a robust and reliable dataset for testing our machine learning models. This process involved several critical steps: (1) Data Cleaning: We began by removing all null values, duplicated entries, and infinite values from the dataset. This step was crucial to enhance the dataset’s reliability and accuracy; (2) Feature Selection and Omission: Certain features were omitted due to their lack of relevance or potential to contribute meaningfully to our analysis. These include ‘Unnamed: 0’, ‘Flow ID,’ and ‘Inbound.’ Additionally, we excluded features containing source/destination IP and port numbers and the ‘Timestamp’ feature. The rationale was to prevent data leakage, which could otherwise result in model overfitting; (3) Label Encoding: For most of the dataset, label encoding was conducted. This was particularly essential for the ‘Label’ variable for classifying DDoS types. We programmatically assigned an integer representation to benign traffic (labeled as 0) and various attack types (marked as 1–11); and (4) Data Scaling: The final preprocessing step involved scaling our data using the StandardScaler method. This standardization ensures that the scaled features have a mean of 0 and a standard deviation of 1. Such scaling is beneficial for better handling outliers and improving the model’s performance. These preprocessing steps were carried out to prepare the dataset for effective and accurate machine learning model application, laying a solid foundation for our subsequent analysis.

Two datasets were required, one for binary classification and one for multi-classification. This is because the data needs to be balanced to ensure no bias in the machine learning model. Each category must have the same number of entries as the others to mitigate the bias. To elaborate, binary classification has two classifications: benign and attack packets. The attack packets are made up of eleven types of attacks, and benign is made up of 60,000 entries. This means the attack category cannot total over 60,000. One can first find the smallest entries for the eleven attack types. This came to be UDP-Lag with 4,818 entries after data preprocessing. Next, 4,818 packets of the other attack types must be compiled together. After this was completed, the attack category totaled 52,998 entries. To create the final binary classification dataset, 52,998 benign entries were taken to balance the data. The final dataset for binary classification counted 105,996 entries. A binary label column was also added to represent the type of attack. Benign was represented by 0, and malicious packets were noted by 1.

Similar steps are being followed to extract the dataset for our multi-class classifications. However, because the categories differed, the data must be balanced differently. The smallest traffic category was benign at 60,000 entries, so the other categories must contain 60,000 entries. The volumetric category is made up of five attacks. This means that 12,000 entries from each specific attack needed to be included in the final category for the volumetric classification. Next, the protocol category is required to be created. It comprises four attacks, so 15,000 entries must be made from each attack. Then, the application layer category was formed by taking 30,000 packets from the TFTP and MSSQL attacks. Lastly, all the categories were combined into one CSV file, and a new column named ‘Multi_Class’ was appended. This column represented the packet type by assigning an integer 0–3 to a row. Benign was represented by 0, volumetric was represented by 1, protocol was represented by 2, and application layer attacks were denoted by 3. With the sub-datasets extracted, we moved on to further preprocessing and feature extraction.

One aspect to note in the classification based on DDoS types is that each traffic attack type has widely varying amounts of entries. Benign traffic consists of 60,000 out of 240,000 entries, while some attack types, such as NetBIOS, only contain 12,000. To resolve such a significant imbalance, we randomly dropped values from classes until each class had 12,000 values, leading to 144,000 entries.

The CIC2019DDoS dataset is notably large, comprising over 80 features across more than 200,000 traffic records. Not all features may significantly contribute to the detection of DDoS attacks. Considering the increased computational demands of using the entire feature set, it was essential to devise a strategy for identifying the ten most influential features. To address this challenge of high dimensionality, we employed the Sequential Forward Selection (SFS) method. SFS is a wrapper-based feature selection algorithm, and for our purposes, it was configured to utilize a Random Forest Classifier as its estimator. This algorithm starts from a baseline of zero selected features. It iteratively adds one feature at a time, basing the selection on each feature’s improvement to the model’s accuracy. Crucially, we integrated a 5-fold cross-validation within the SFS process. This approach helps prevent overfitting and offers a more accurate estimation of the model’s performance as the feature selection unfolds. For consistency and to maintain a unified approach across our study, we decided to use the same top ten features identified by SFS for all classifications in our research, which includes one binary classification and two multi-class classifications. This systematic approach in feature selection is pivotal, as it ensures that our model is both efficient and effective, focusing only on the most impactful features for DDoS attack detection.

Now that our dataset is optimized and we have established our ten most influential features for DDoS detection, we can proceed to modeling strategies. We utilize the ML algorithms LR, DT, RF, KNN, SVM, and XGB, as mentioned. For each model, we use a Bayesian Search with 30 iterations to optimize our models’ hyperparameters. The Bayesian Search algorithm explores various hyperparameter combinations for each given ML model, repeatedly training models until completion. Once completed, the best results are chosen based on the highest accuracy rating. The results will contain the top ten feature names and the training accuracy rating associated with that model. We then create the base model for each algorithm, where no parameters are given and default values are used. This will establish a baseline accuracy we can compare to our optimized model. The optimized model can be constructed with the hyperparameters given by Bayesian Search, from which we then test this model and compare its accuracy, recall, F1, and precision to the base. We repeat these steps to classify DDoS attacks into three categories and 11 attack types. Figure 2 below illustrates the flowchart of our methodology.

Fig. 2.

Framework of the proposed methodology.

5. Result Analysis

The initial phase of our results analysis involves examining the critical features selected through our Sequential Forward Selection (SFS) implementation. These features are essential in differentiating between benign and DDoS-related traffic. We will delve into the first three features, which are ranked by their overall influence on the model:

Flow Bytes/s:

This feature indicates the byte rate transmitted through a network flow. An abnormally high value in this metric could suggest involvement in a DDoS attack, particularly those of the Volumetric category. Such patterns are often observed in attacks where large volumes of data are sent to overwhelm the target.

Fwd Header Length:

Representing the total length of a packet’s header, this feature is particularly telling in specific DDoS strategies, such as DNS Amplification attacks. These attacks typically utilize disproportionate packet headers, where a small DNS query triggers a substantially larger response packet, overwhelming the victim’s network. This feature is, therefore, a strong indicator of DDoS activity, especially in attacks like NTP and LDAP.

Min Packet Length:

The shortest packet length within a flow is denoted by this feature. Its significance stems from certain DDoS attack types that use small, maliciously crafted packets to congest system resources. Examples include UDP and SYN Floods, categorized as Volumetric attacks in our dataset.

The patterns and behaviors identified by these features predominantly align with the detection of Volumetric DDoS attacks, likely due to the similar characteristics shared among these types of attacks. Following this feature analysis, we tested baseline binary classification with various ML algorithms. This testing yielded encouraging results across almost all algorithms, setting a positive foundation for further in-depth analysis and model refinement.

As illustrated in Table 1, the results of our binary classification show outstanding performance across all metrics displayed. While LR has the lowest performance out of our selection, a 96.32% accuracy is quite good. Despite these high scores, we believe there may be some issues about overfitting. This could be caused by errors within our dataset and preprocessing or potential data leakage during model training. All three of our tree-based algorithms score 100% accuracy, which is unusually high. Given that these were allowed to run with no specified parameters, these algorithms may have created trees with enough complexity to make perfect classifications, which could cause additional overfitting.

Table 1.

Results given from base ML models using binary and 3-category multiclassification.

Results given from ML models using binary classification						Results given from base ML models with 3-category multiclassification.
Model	LR	DT	RF	KNN	XGBoost	LR	DT	RF	KNN	XGBoost
Accuracy	96.32	100	100	99.93	100	55.59	87.58	87.71	88.03	88.01
Precision	93.52	99.98	100	99.98	99.99	56.68	88.43	88.56	88.37	88.69
Recall	99.50	99.97	99.99	99.97	99.99	55.59	87.58	87.71	88.03	88.01
F1-Score	98.64	99.97	99.99	99.93	99.99	52.98	87.54	87.66	88.15	87.99

Upon testing our base models with no given hyperparameters, we see a good performance from the DT, RF, KNN, and XGBoost algorithms, as illustrated in the right part of Table 1. However, we noted that LR appears to perform poorly in this case. This makes sense, as LR tends to struggle with highly complex datasets. Additionally, LR is known to work under multi-class settings, which could also contribute to its poor performance. Granted, this model had no specified parameters, which further contributed to the poor accuracy, precision, recall, and F1 score. On the other hand, we see that of all algorithms, KNN scored the highest accuracy rating of 88.03%. We did not expect to see this, but ultimately, it is a sensible result. In short, KNN works by classifying incoming data entries based on the “k-nearest-neighbors.” Depending on one entry’s values, it will be classified based on which existing class holds many similar values. Since these DDoS attacks work similarly in their respective categories, it’s sensible that KNN performs well here. Lastly, our tree-based algorithms (DT, RF, and XGB) have notable performances. DTs alone excel under these use cases of multi-class classification with highly complex data in large volumes. Given that RF and XGB are advancements of the DT algorithm, their improved performance over DT is expected.

Upon testing our optimized models with hyperparameters given through the Bayesian search, we do not see much improvement in any algorithms (for some algorithms, there is a decrease) except LR and KNN, as illustrated in Table 2. We were left fairly confused by these results, as our implementation of Bayesian Search allows for almost all algorithm parameters to be tested at large ranges. We see definite performance improvements in optimizing LR, albeit still achieving a low accuracy rating of 66.40%. One notable parameter contributing to the performance increase is ‘multi_class,’ in which we test with the values ‘auto,’ ‘over,’ and ‘multinomial.’ This parameter is dedicated to multi-class use cases in which ‘ovr’, or the One-vs-Rest approach, trains a binary classifier to predict which class a data entry will be selected. The choice ‘multinomial’ uses a different approach, where probabilities are calculated for each class, and the current data entry will be classified based on the highest probability. Beyond the improvements in LR, we see slight decreases in the performance of our optimized tree-based models. We are unsure exactly why this is, as it could stem from several reasons. The parameter ranges we specified could be too large or small, the models could be overfitted in training, and issues could exist in our dataset, to name a few. We did note a minimal improvement in the optimized KNN model’s accuracy of 0.03%. We aren’t sure of what caused this, but we assume that the tested hyperparameters contributed to this improvement. This could have led to minimal improvement since KNN has few parameters to test. Also, the default parameters in our base model could have ended up like those in our optimized model.

Table 2.

Results given from optimized ML models with 3- and 11-type multiclassification.

Results given from optimized ML models with 3-category multiclassification.						Results given from base ML models with 11-type multiclassification.
Model:	LR	DT	RF	KNN	XGBoost	LR	DT	RF	KNN	XGBoost
Accuracy:	66.40	87.51	87.64	88.06	87.97	56.93	72.11	81.61	70.33	73.89
Precision	61.09	88.46	88.52	88.44	88.69	53.94	74.30	75.11	71.48	75.91
Recall	61.40	87.51	87.64	88.06	87.97	56.932	72.11	72.31	70.33	73.89
F1-Score	59.15	87.41	87.58	88.19	87.94	52.22	68.72	68.84	69.32	72.51

Table 2 illustrates the multiclassification results on DDoS attacks by individual type, in which there is a notable drop in the performance of all algorithms except LR. Compared to classifying based on the 3 DDoS categories, LR performs much better in this case of categorizing based on the DDoS types. The remaining algorithms perform worse in this classification compared to their former classification by attack categories. A potential reason behind this could be that when split by individual types, specific attacks may have more detailed, prominent patterns/behavior than other types. This could likely aid the LR model in identifying particular classes. However, we see that this LR model has low precision and F1-score, which probably means that this model also has difficulty in accurately classifying true/false positives for one or more attack types. The drop in performance metrics could also be caused by the increase in complexity, classifying into 12 classes instead of 4. We also see reduced performance metrics in our tree-based models and KNN, which we somewhat attribute to the increase in complexity. Ultimately, our RF model scored the highest testing accuracy of 81.61%, but we see that other performance metrics are subpar. With an F1-score of 68.84%, this tells us that this model is struggling to make accurate classifications.

6. Conclusions

Our findings indicate that the most effective approach for multi-class classification of DDoS attacks involves categorizing based on the three broad attack categories rather than differentiating among the 11 specific attack types. This strategic shift has yielded superior performance metrics across all algorithms in our study. The models developed for the 3-category classification consistently outperform those designed for classifying the 11 individual DDoS attack types across various performance metrics. Notably, the K-Nearest Neighbors (KNN) model has emerged as the top performer. It excels particularly in recall and F1 score, albeit with marginally lower precision than our tree-based models. The effectiveness of KNN can be attributed to its proficiency in scenarios where classes exhibit closely related values and characteristics. This aligns well with the three-category classification approach, where each category embodies distinct, recognizable strategies and behaviors during an attack. While slightly overshadowed by KNN in this context, the XGBoost algorithm has also demonstrated commendable performance. It stands out as a robust choice for our multi-class classification needs, further substantiating the effectiveness of our chosen methodology.

These insights are instrumental in guiding our ongoing efforts to refine DDoS attack detection and classification. The distinct advantages of the 3-category classification strategy and the strengths of the KNN and XGBoost models offer promising avenues for future enhancements in cybersecurity. In the future, we will further refine and optimize the KNN and XGBoost models and explore other approaches for detecting, evaluating, and managing the alerts generated by our detection system [31][32][33].