Modeling multivariate cyber risks: deep learning dating extreme value theory

Abstract

Modeling cyber risks has been an important but challenging task in the domain of cyber security, which is mainly caused by the high dimensionality and heavy tails of risk patterns. Those obstacles have hindered the development of statistical modeling of the multivariate cyber risks. In this work, we propose a novel approach for modeling the multivariate cyber risks which relies on the deep learning and extreme value theory. The proposed model not only enjoys the high accurate point predictions via deep learning but also can provide the satisfactory high quantile predictions via extreme value theory. Both the simulation and empirical studies show that the proposed approach can model the multivariate cyber risks very well and provide satisfactory prediction performances.

1. Motivation and introduction

Cyber risk has become one of the most emerging risks in recent decades which can cause disastrous consequences and tremendous monetary losses [16,24,29]. This is fundamentally caused by the fact that cyberspace is extremely difficult to secure because of its complexity (e.g. an attacker can launch cyber attacks from anywhere in the world, and cyber systems have a very large ‘surface’ of software vulnerabilities that can be exploited to penetrate into them). What makes things worse is that cyber systems have become underlying pillars for various infrastructures and physical systems. For example, there are growing concerns about cyber threats to critical infrastructures because of the linkage between cyber and physical systems [7]. In addition, cyber attacks causes billions of dollars in losses each year based on the research by the Ponemon Institute and published by IBM Security [25]. In light of the risk and potential consequences of cyber risks, securing cyberspace has become not only a mission of homeland security but also an emerging task for private companies. This clearly calls for research into cyber risk modeling, of which risk prediction is an important task.

In this work, we investigate a particular challenge that is encountered when modeling and analyzing cyber risk, namely the modeling of multivariate cyber risks. This challenge is imposed by the high-dimensional dependence among cyber attacks and extreme attacks during a short time period. The dependence among cyber attacks is very common which is the nature of cyber risks. For example, the Distributed Denial of Service (DDoS) attack, a large-scale DoS attack where the perpetrator often uses thousands of hosts infected with malware to launch attacks to a target or multi targets (e.g. ports, computers, severs), is in nature high-dimensional and dependent. In particular, the DDos attacks are often extreme during a short time period. The defender often needs to estimate or predict the cyber risks for the purpose of adjusting the defense posture in practice. However, this is very challenging because of the high-dimensionality and heavy tails (i.e. extreme attacks during a short period) exhibited by cyber attack data [24,30–32], which hinders the development of modeling multivariate cyber risks. Therefore, in the literature of statistical modelings, there are only few studies investigating multivariate cyber risks. For example, Xu et al. [30] proposed a vine copula approach for modeling the dependence among the time series of the number of cyber attacks, and the dependence between the time series of the number of attacked computers. Peng et al. [24] developed a copula-GARCH model, which also uses vine copulas to model the multivariate dependence among cyber attacks. Ling et al. [21] proposed a vector autoregression (VAR) approach to identifying a geospatial and temporal patterns in the cyberattacks by considering the long range dependence.

In recent years, there is a growing interest in developing deep learning models for time series forecasting [6,19,26]. Fang et al. [9] developed a deep learning framework by utilizing the bi-directional recurrent neural networks with long short-term memory to predict the univariate cyber attack time series. For multivariate time series forecasting, Goel et al. [11] proposed a hybrid R2N2 (residual recurrent neural networks) model, which first models the time series with a simple linear model (e.g. VAR) and then models its residual errors using recurrent neural networks (RNNs). Lai et al. [20] developed a long- and short-term time-series network (LSTNet) to model the multivariate time series. The main idea is to use the convolution neural network and the recurrent neural network (RNN) to model short-term local dependence and long-term trends for time series. Coulibaly and Baldwin [4] applied an RNN-based model to the problem of modeling complex time-varying water resources in terms of time series forecasting. Che et al. [1] studied a deep learning model based on gated recurrent unit (GRU) for modeling multivariate time series with missing data. Zhang et al. [33] used a GRU-based model to forecast multivariate time series after the decomposition. Wang et al. [28] presented an end-to-end framework called deep prediction interval and point estimation, which can simultaneously perform point estimation and uncertainty quantification for multivariate time series forecasting. The main idea is to model the loss function by penalizing the loss of point estimation and the loss of prediction interval based on the deep learning model of long short-term memory network. Please refer to [8] for a recent review on the deep learning models for cybersecurity applications. However, to the best of our knowledge, there is no deep learning framework for modeling the multivariate cyber attack data in the domain of cyber security.

In this work, we propose a novel framework which is different from those in the literature for modeling and predicting the multivariate cyber attack data. Specifically, we develop a hybrid model by combining deep learning and extreme value theory [5,23]. That is, we first model the multivariate attack time series via a deep learning model which aims to capture the high-dimensional dependence via the deep learning network. We then model the residuals exhibiting heavy tail via the extreme value theory. The proposed approach not only can provide accurate point prediction but also capture the tails very well (i.e. predicting the extreme attacks). This novel framework is particularly useful in practice as it can guide the defender to prepare the resource for both the regular attack (in terms of mean prediction) and worst attack (in terms of tail prediction) scenarios.

The rest of the paper is organized as follows. In Section 2, we present the preliminaries on the deep learning model and extreme value theory. In Section 3, we discuss the proposed the framework for model fitting and prediction. Section 4 assesses the proposed approach via simulation studies. In Section 5, we study two real honeypot attack data. In the final section, we conclude our results and present a discussion toward some limitations and future work.

2. Preliminaries

2.1. Long short-term memory (LSTM)

The LSTM network, a special kind of RNNs, introduced in [13] has been received remarkable attention because of the ability to learn long term patterns in sequential data and tremendously improve the prediction accuracy compared to other deep learning models. Therefore, this deep learning model has been successfully applied in many areas [12].

Figure 1 illustrates a LSTM block. Compared to regular RNN, the LSTM employs a different approach for the activation. Specifically, given time series data ${\mathit{y}}_1,\dots ,{\mathit{y}}_t$ where ${\mathit{y}}_i=(y_{1,i},\dots ,y_{n,i}{)}^{\mathrm{\top }}$, $i=1,\dots ,t$, the activation $h_t$ of LSTM at step t is computed based on four pieces: gate input (a.k.a, information gate), the forget gate, the output gate, and the cell gate. The information gate input at step t is

$$i_t=\sigma ({\mathbf{W}}_{ia}\cdot h_{t-1}+{\mathbf{W}}_{iy}\cdot {\mathit{y}}_t),$$

where σ is the sigmoid function, and $\mathbf{W}$'s are the parameter matrices of the network learned during the training. The forget gate input and the output gate input are computed as

$$\begin{array}{rl}{f}_t& =\sigma ({\mathbf{W}}_{fa}\cdot h_{t-1}+{\mathbf{W}}_{fy}\cdot {\mathit{y}}_t),\\ o_t& =\sigma ({\mathbf{W}}_{oa}\cdot h_{t-1}+{\mathbf{W}}_{oy}\cdot {\mathit{y}}_t).\end{array}$$

The cell gate input is computed as

$$C_t=f_t\odot C_{t-1}+i_t\odot k_t,$$

where ⊙ represents the element-wise Hadamard product, the $C_{t-1}$ is the cell state information from the previous step, and $k_t$ is

$$k_t=\tanh ({\mathbf{W}}_{ca}\cdot h_{t-1}+{\mathbf{W}}_{cy}\cdot {\mathit{y}}_t)$$

where $\tanh$ is the hyperbolic tangent function. Finally, the activation at step t is computed as

$$h_t=o_t\odot \tanh (C_t),$$

where $h(t)\in {\mathcal{R}}^n$ is the final cell output.

Figure 1.

A LSTM block at step t.

2.2. GRU and mLSTM

As the variants of LSTM, GRU [2] and Multiplicative LSTM (mLSTM) [18] have also received much attention recently due to their extraordinary prediction abilities. The GRU uses less trainable parameters than LSTM, by having only two gates, the reset gate ( $r_t$) and the update gate ( $z_t$). Its computational process is as follows

$$\begin{array}{rl}{z}_t& =\sigma ({\mathbf{W}}_{zy}\cdot {\mathit{y}}_t+{\mathbf{W}}_{za}\cdot h_{t-1}),\\ r_t& =\sigma ({\mathbf{W}}_{ry}\cdot {\mathit{y}}_t+{\mathbf{W}}_{ra}\cdot h_{t-1}),\\ {\hat{h}}_t& =\tanh ({\mathbf{W}}_{hy}\cdot {\mathit{y}}_t+{\mathbf{W}}_{ha}(r_t\odot h_{t-1})),\\ h_t& =(1-z_t)\odot h_{t-1}+z_t\odot {\hat{h}}_t.\end{array}$$

Compared to the LSTM, the mLSTM simply replaces $h_{t-1}$ with an intermediate state, $m_t$, in order to calculate $i_t$, $f_t$, $o_t$, and $k_t$, where $m_t$ is

$$m_t={\mathbf{W}}_{ma}\cdot h_{t-1}\odot {\mathbf{W}}_{my}\cdot {\mathit{y}}_t.$$

For more and detailed discussions on the deep learning models, please refer to [12,13]

3. Deep learning framework for multivariate cyber risks

Let ${\mathit{y}}_t=(y_{1,t},\dots ,y_{n,t}{)}^{\mathrm{\top }}$ be a vector of attack time series. In practice, the dimension of n can be very large, and particularly, the dependence among cyber risks can be very nonlinear. This motivates us to use the following approach for analyzing multivariate cyber risks, which utilizes the advantage of high accurate point estimates of deep learning and the advantage of EVT for modeling the high quantiles. Specifically, the proposed model is as follows,

$${\mathit{y}}_t=f({\mathit{y}}_{t-1},\dots ,{\mathit{y}}_{t-p})+{\mathit{ϵ}}_t,$$

where ${\mathit{ϵ}}_t=({ϵ}_{1,t},\dots ,{ϵ}_{n,t}{)}^{\mathrm{\top }}$ is an unobservable zero mean white noise vector process, and f is a function by mapping the vector $({\mathit{y}}_{t-1},\dots ,{\mathit{y}}_{t-p})$ to the mean vector of ${\mathit{y}}_t$. Note that a special case is the VAR,

$${\mathit{y}}_t={\mathit{\mu }}_0+A_1{\mathit{y}}_{t-1}+\dots +A_p{\mathit{y}}_{t-p}+{\mathit{ϵ}}_t,$$

where $A_i$ are coefficient matrices, $i=1,\dots ,p$, and ${\mathit{\mu }}_0$ is an n-dimensional constant vector. We propose to model the mean of ${\mathit{y}}_t$ by

$${\hat{\mathit{y}}}_t=\hat{f}({\mathit{y}}_{t-1},\dots ,{\mathit{y}}_{t-p})$$

via deep learning approach, while the tails of ${\mathit{y}}_t$ are estimated by EVT approach.

The proposed framework involves the following three stages.

Stage 1: capturing the multivariate dependence via deep learning

In multivariate time series modeling and prediction, the most challenging part is to find a suitable f for capturing the complex dependence pattern. Deep learning as an emerging tool for multivariate time series modeling and forecasting has an extraordinary forecasting performance [10,11]. Therefore, we develop the following procedure for modeling the mean function f via deep leaning.

Given time series ${\mathit{y}}_1,\dots ,{\mathit{y}}_{t-1}$, we define p as the number of input data points in each of them, and p also indicates the number of time steps to look back (i.e. lags). The prediction process can be formally described as:

$${\hat{\mathit{y}}}_t=\hat{f}({\mathit{Y}}_{t-1,p}),$$

where ${\mathit{Y}}_{t-1,p}=({\mathit{y}}_{t-1},\dots ,{\mathit{y}}_{t-p})\in R^{n\times p}$ is the input matrix. We apply the following objective function to measure the error generated by the model,

$$J=\frac{1}{(T-t+1)n}\sum _{i=t}^T\Vert {\mathit{y}}_i-{\hat{\mathit{y}}}_i{\Vert }_2^2+\lambda \cdot \Vert \mathbf{W}{\Vert }_2^2,$$ (1)

where λ is the tuning parameter, $\Vert \cdot {\Vert }_2^2$ represents the squared $L_2$ norm, and $\mathbf{W}$ represents all the training parameters in the learning process. For instance, if the network is LSTM, then $\mathbf{W}=\{{\mathbf{W}}_{za},{\mathbf{W}}_{zy},{\mathbf{W}}_{ra},{\mathbf{W}}_{ry},{\mathbf{W}}_{ha},{\mathbf{W}}_{hy}\}$. The optimization is defined as

$${\mathbf{W}}^{\ast }=\arg \underset{\mathbf{W}}{min}J,$$

which can be solved by using the gradient descent method [17]. To select the best deep learning model, we perform a manual grid search by setting the hyper-parameters of the networks as follows:

• Network models: $M_1$-LSTM, $M_2$-mLSTM, $M_3$-GRU;

• Number of layers: {1, 2, 3}; Size of each layer: {16, 32, 64};

• Batch size: {5,10}; Bidirectional: {True, False}; $p=\{1,2,3,4,5\}$;

• Penalty parameter: $\lambda =\{0.01,0.001\}$; Initial learning rate: $\gamma =\{0.01,0.001\}$;

• Number of training epochs: {40, 50, 60, 70, 80}.

Algorithm 1 presents the detailed procedure for selecting the best deep learning model.

State 2: modeling high quantiles via extreme value theory

After fixing the deep learning model, the fitted values at time t are

$${\hat{\mathit{y}}}_t=\hat{f}({\mathit{Y}}_{t-1,p}),$$

and the residuals are

$${\mathit{e}}_t={\mathit{y}}_t-{\hat{\mathit{y}}}_t,$$

where ${\mathit{e}}_t=(e_{1,t},\dots ,e_{n,t}{)}^{\mathrm{\top }}$. The second stage is to model the residuals by some statistical distribution. Since the heavy tails are often observed in the attack data, we propose to model the high quantiles of the residuals by the EVT approach. This is in principle in line with the two-stage pseudo-maximum-likelihood approach in [22].

Recall that a popular EVT method is known as the peaks over threshold approach [5,23]. Specifically, given a sequence of i.i.d. observations $X_1,\dots ,X_n$, the excesses $X_i-\mu$ with respect to some suitably large threshold μ can be modeled by, under certain mild conditions, the Generalized Pareto Distribution (GPD). The survival function of the GPD is:

$$\overline{G}(x)=1-G(x)=\{\begin{array}{cc}{(1+\xi {\displaystyle \frac{x-\mu }{\sigma }})}_{+}^{-1/\xi },& \xi \ne 0,\\ \exp \{-\frac{{\displaystyle x-\mu }}{{\displaystyle \sigma }}\},& \xi =0.\end{array},$$ (2)

where $x\ge \mu$ if $\xi \in {\mathbb{R}}^{+}$ and $x\in [\mu ,\mu -\sigma /\xi ]$ if $\xi \in {\mathbb{R}}^{-}$, and ξ and σ are, respectively, called the shape and scale parameters.

Stage 3: prediction and evaluations

After we determine the deep learning model and residual distribution, then we can use the forward propagation to predict the mean

$${\hat{\mathit{y}}}_{t+1}=\hat{f}({\mathit{Y}}_{t,p}).$$

The q-quantile of $y_{i,t+1}$ can be predicted as

$${\hat{\mathit{y}}}_{i,t+1,q}={\hat{y}}_{i,t+1}+G_i^{-1}(q),$$ (3)

where $G_i^{-1}(q)$ is the q-quantile of $G_i$ as defined in Equation (2) for time series i, $i=1,\dots ,n$. Algorithm 2 presents the detailed procedure for the prediction.

The prediction performance for the mean prediction are evaluated based on MSE and mean absolute percentage error (MAPE) [14].

In order to assess the prediction accuracy of quantiles, we propose using the Value-at-Risk (VaR) [23] metric because it is directly related to the quantities of interest. Recall that for a random variable $X_t$, the VaR at level α for some $0<\alpha <1$ is defined as ${\mathrm{VaR}}_{\alpha }(t)=inf\{l:P(X_t\le l)\ge \alpha \}$. An observed value that is greater than the predicted ${\mathrm{VaR}}_{\alpha }(t)$ is called a violation, indicating inaccurate prediction. In order to evaluate the prediction accuracy of the VaR values, we adopt the following two popular tests [3]: (i) the unconditional coverage test, denoted by LR ${}_{uc}$, which evaluates whether or not the fraction of violations is significantly different from the model's violations; (ii) the conditional coverage test, denoted by LR ${}_{cc}$, which is a joint likelihood ratio test for the independence of violations and unconditional coverage.

4. Simulation study

In this section, we perform a simulation study on assessing the performance of proposed approach. To mimicking the attack data with heavy tails, we randomly generate two data sets with size $5000\times 5$ from the following models. In each experiment, the simulated data is split into three parts: the first 3500 is used for training, and the following 500 observations is used for the validation. The testing data is the last 1000 observations.

• VAR with heavy tail. The VAR model is set as follows.

  $${\mathit{y}}_t={\mathit{\mu }}_0+A_1{\mathit{y}}_{t-1}+A_2{\mathit{y}}_{t-2}+{\mathit{ϵ}}_t,$$

  where $t=1,\dots ,5000$, ${\mathit{\mu }}_0=({\mu }_{1,0},\dots ,{\mu }_{5,0})$, ${\mathbf{y}}_t=(y_{1,t},\dots ,y_{5,t})$, $A_1$ and $A_2$ are $5\times 5$ coefficient matrices. In the experiment, $A_1$ and $A_2$ are generated from uniform distribution $[-0.2,0.2]$, and ${\mu }_{i,0}=100$, $i=1,\dots ,5$. The error ${\mathit{ϵ}}_t=({ϵ}_{1,t},\dots ,{ϵ}_{5,t})$ has zero mean, and ${ϵ}_{i,t}/\sigma$, $i=1,\dots ,5$, follows a skewed-t distribution with density as follows

  $$g(z)=\frac{2}{\xi +{\xi }^{-1}}[t_{\nu }(\xi z)\mathrm{I}(z<0)+t_{\nu }\left({\xi }^{-1}z\right)\mathrm{I}(z\ge 0)],$$ (4)

  where $\mathrm{I}(\cdot )$ is the indicator function, $\xi >0$ is the skewness parameter, and

  $$t_{\nu }(z)=\frac{\mathrm{\Gamma }((\nu +1)/2)}{\sqrt{\nu \pi }\mathrm{\Gamma }(\nu /2)}{[1+z^2/\nu ]}^{-(\nu +1)/2}$$

  with the shape parameter $\nu >0$. In the experiment, the parameters are set as $\sigma =20$, $\xi =1.5$, and $\nu =3$.

  Figure 2 plots the simulated time series, and we observe there exist large values which indicate the heavy tail.

  We employ Algorithm 1 to train and validate the deep learning model on the training and validation data sets. The residuals are used to fit the distribution G in Equation (2) with the threshold setting to be $.9$-quantile of residuals for each time series which is determined via the mean residual life plot [23].

  The prediction is performed via Algorithm 2, where the threshold is set to $.9$-quantile of residuals. The prediction performance is reported in Table 1. For the comparison purpose, we also report the prediction performance of VAR, where the lag p is selected via AIC criterion. The VAR model is considered as a benchmark model in the sequel discussion. It is seen from Table 1 that the proposed model outperforms the benchmark model for all the time series in terms of MAPE and MSE for the point prediction. Particularly, the MSEs are significantly improved by the deep learning approach.

  For the quantiles prediction, we plot the violations for the proposed model and benchmark model in Figure 3(a,d), respectively. It is seen that the observed number of violations are fairly close to the expected ones by the proposed approach from 0.95 to 0.98 levels. The benchmark model underestimates the number of violations at the 0.95 level. For the 0.99 level, it is seen that the proposed approach underestimates the number of violations for time series 4 and 5, which result in relatively smaller p-values of LRuc and LRcc tests in Figure 3(b–c). For the benchmark model, we observe that it underestimates four time series at level 0.99 from Figure 3(d).

  We conclude that the proposed approach has an overall satisfactory prediction performance for high quantiles from level 0.95 to 0.98.

• Copula+AR-GARCH with heavy tail. For this simulation, we assume that the mean part follows AR(1) process

  $$Y_{i,t}-\mu ={\varphi }_1(Y_{i,t-1}-\mu )+{ϵ}_{i,t},$$

  where $i=1,\dots ,5$, and

  $${ϵ}_{i,t}={\sigma }_{i,t}{Z}_{i,t}$$

  with $Z_{i,t}$ being the innovations that are identically distributed with skewed-t density $g(\cdot )$ in Equation (4) and the dependence structure is specified via R-vine copula, and ${\sigma }_t$ follows a standard GARCH(1,1) model, i.e.

  $${\sigma }_{i,t}^2=w+{\alpha }_1{ϵ}_{i,t-1}^2+{\beta }_1{\sigma }_{i,t-1}^2.$$

  In the experiment, the parameters are set as follows

  $$(\mu ,{\varphi }_1,w,{\alpha }_1,{\beta }_1,\xi ,\nu )=(50,.6,.5,.05,.8,1.5,3).$$

  To simulate the dependence among $Z_{i,t}$, we generate 5-dimension multivariate uniform distribution, where the dependence structure is specified via the R-vine copula. Specifically, the R-vine tree matrix is as follows

  $$\left(\begin{array}{ccccc}2& 0& 0& 0& 0\\ 5& 3& 0& 0& 0\\ 3& 5& 4& 0& 0\\ 1& 1& 5& 5& 0\\ 4& 4& 1& 1& 1\end{array}\right)$$

  and the family matrix is set as

  $$\left(\begin{array}{ccccc}0& 0& 0& 0& 0\\ 1& 0& 0& 0& 0\\ 3& 3& 0& 0& 0\\ 4& 4& 4& 0& 0\\ 4& 1& 1& 3& 0\end{array}\right).$$

  The parameter matrix is set as follows

  $$\left(\begin{array}{ccccc}0& 0& 0& 0& 0\\ 0.2& 0& 0& 0& 0\\ 0.9& 1.1& 0& 0& 0\\ 1.5& 1.6& 1.9& 0& 0\\ 3.9& 0.9& 0.5& 4.8& 0\end{array}\right).$$

  A $5000\times 5$ matrix from the above R-vine structure is generated, which is used for generating the dependent $Z_{i,t}$, $i=1,\dots ,5$.

  The simulated time series plots are displayed in Figure 4, and the extreme values can also be observed.

  We again employ Algorithm 1 to train and validate the deep learning model, and the residuals are used to fit the GPD distribution in Equation (2). It is discovered that the threshold of 90th percentile of residuals can produce satisfactory fitting performance for all the time series. Therefore, in the prediction procedure of Algorithm 2, the threshold is fixed at 90th percentile of residuals for each time series.

  Table 2 presents the performances of point predictions for both proposed model and benchmark model. It is seen that the proposed model can significantly outperform the benchmark model in terms of both metrics for all the time series.

  For the high quantiles, the number of violations are displayed in Figure 5(a,d). It is seen that the proposed model has a very satisfactory prediction performance compared to that of benchmark model. The large p-values of LRuc and LRcc tests in Figure 5(b–c) compared to those in Figure 5(e–f) also confirm that the proposed approach has an accurate prediction performance for the high quantiles.

Figure 2.

Time series plots of simulated VAR with skewed-t tails. (a) Time series 1 (b) Time series 2 (c) Time series 3 (d) Time series 4 (e) Time series 5.

Table 1.

Prediction performances based on MAPE and MSE for the proposed model and benchmark model.

	Deep		Benchmark
Series	MAPE	MSE	MAPE	MSE
1	0.1116	225.4169	0.1768	356.6563
2	0.1152	211.2983	0.1908	357.2598
3	0.0542	285.8806	0.0841	408.9762
4	0.0771	226.0736	0.123	382.4075
5	0.1716	191.3857	0.2839	378.5063

Figure 3.

Numbers of violations and p-values of LRuc and LRcc tests for the proposed model and benchmark model, where ‘Exp’. represents expected number of violations, and x-axis represents the VaR level α. (a) Violations-Deep+EVT (b) LRuc-Deep+EVT (c) LRcc-Deep+EVT (d) Violations-Benchmark (e) LRuc-Benchmark (f) LRcc-Benchmark.

Figure 4.

Time series plots of simulated Copula+AR-GARCH with student-t tail. (a) Time series 1 (b) Time series 2 (c) Time series 3 (d) Time series 4 (e) Time series 5.

Table 2.

Prediction performances based on MAPE and MSE for the proposed model and benchmark model.

	Deep		Benchmark
Series	MAPE	MSE	MAPE	MSE
1	0.0185	2.4612	0.032	4.8339
2	0.0189	2.3415	0.0324	4.6922
3	0.0198	2.5678	0.0325	4.7743
4	0.0190	2.2672	0.0313	4.3606
5	0.0183	2.3662	0.0323	4.7806

Figure 5.

Numbers of violations and p-values of LRuc and LRcc tests for proposed model, and true model. (a) Violations-Deep+EVT (b) LRuc-Deep+EVT (c) LRcc-Deep+EVT (d) Violations-Benchmark (e) LRuc-Benchmark (f) LRc-Benchmark.

To conclude, the proposed model has satisfactory prediction performances for both the point prediction and high quantile prediction. It can significantly outperform the benchmark model.

5. Empirical study

In this section, we study two real attack data which are collected by the honeypot instrument [27].

5.1. Honeypot data I with 9 dimensions

This honeypot data is publicly available on the web [15], which was collected via Amazon Web Service (AWS) virtual honeypots across the world. The dataset has 9 honeypot hosts (EU, Oregon, Singapore, SA, Tokyo, Norcal1, Norcal2, US-East, Sydney), and the attacks were recorded between 03/03/2013 to 09/08/2013, which includes 451,581 events. The recorded attack data includes attack time, targeted host, attackers' IP addresses and origin countries. This dataset was discussed in [21] based on the daily aggregation, where the long range dependence model was discovered and incorporated into the modeling process.

Daily aggregation. The daily aggregated data has 188 observations, we leave the last 38 observations as the prediction evaluation as that in [21] where the predicted MSE is reported. We employ Algorithm 1 to build the deep learning model where the training data has 100 observations, and validation data has 50 observations. The prediction performance based on Algorithm 2 is reported in Table 3.

Table 3.

Prediction performances for log-transformed daily aggregated honeypot attack data.

	Deep		LRD+VAR in [21]
Series	MAPE	MSE	MSE
EU	0.0685	0.2058	0.2540
Oregon	0.0118	0.0096	0.0138
Singapore	0.0144	0.0145	0.0133
SA	0.0284	0.0319	0.0567
Tokyo	0.0418	0.4225	0.4733
Norcal1	0.0262	0.0282	0.0445
Norcal2	0.0287	0.0358	0.0692
US-East	0.0666	0.3726	0.3848
Sydney	0.0554	0.1126	0.1374

The proposed deep learning has a satisfactory prediction performance. In particular, it outperforms the approach in [21] for all the attack time series except for Singapore based on MSEs (0.0145 vs. 0.0133). Since the daily aggregated time series only has a total of 188 observations for each host, the high quantile predictions are not performed.

Hourly aggregation. In practice, the network defender is often interested in assessing the hourly attacks [24,31,32]. In this section, we study the performance of proposed model based on the hourly aggregated data which has 4512 observations for each host. The hourly aggregated time series are plotted in Figure 6. It is observed that there are extreme attacks for all the hosts. Particularly, there exist tremendous numbers of attacks during some short periods for Oregon, Singapore, and Tokyo hosts. The data is split into three parts: training with 3500 observations, validation with 500 observations, and the last 512 observations are used for the prediction evaluation. The Algorithm 1 is employed on the training and validation datasets to select the best deep learning model, and Algorithm 2 is used for the prediction.

Figure 6.

Time series plots of hourly aggregated attack data for all the hosts. (a) EU (b) Oregon (c) Singapore (d) SA (e) Tokyo (f) Norcal1 (g) Norcal2 (h) US-East (i) Sydney.

The prediction performances are reported in Table 4. For the comparison purpose, the prediction performances of benchmark VAR, GRU [1], and RNN [4] models are also reported. For the proposed approach, we can observe that it can significantly outperform the other models in terms of MAPE metric. It is interesting to see that the benchmark model can also outperform the GRU and RNN models for EU, Oregon, Singapore, and Sydney. For the MSE metric, it is seen that the proposed approach can outperform the other models for all the time series except for SA. After carefully checking the data, it is found that the poor prediction performance of SA is mainly caused by a few points which are not well predicted. It is also interesting to observe that the benchmark model performs better than the GRU and RNN models for most of time series. For Tokyo time series, the MSEs are very large for all the models, which is due to one extreme observation. By comparing Tables 3 and 4, it is seen that the MSEs in Table 4 are much larger than those in Table 3. This is because that the MSEs in Table 3 are based on the log-transformed daily aggregated data while the MSEs in Table 4 are based on the hourly aggregation. The other possible factor is that the hourly data has more variabilities which introduces the prediction challenge.

Table 4.

Prediction performances for daily aggregated honeypot attack data based on proposed approach (‘Deep’), benchmark VAR, GRU, and RNN models.

	Deep		Benchmark		GRU		RNN
Series	MAPE	MSE	MAPE	MSE	MAPE	MSE	MAPE	MSE
EU	0.3067	15.5700	0.4055	15.7607	0.4912	23.3754	0.4934	23.7104
Oregon	0.1826	34.6819	0.2446	35.9174	0.2460	40.8180	0.2570	79.5017
Singapore	0.1756	29.5333	0.2372	34.3774	0.2664	80.6857	0.3020	464.3414
SA	0.3248	23.7633	0.4643	14.6417	0.4601	15.6584	0.4544	14.7642
Tokyo	0.6727	230,565.6248	1.0707	321,396.0317	0.7494	300,824.4188	0.7357	294,587.3091
Norcal1	0.3225	12.0416	0.4668	16.1691	0.4539	16.3385	0.4503	16.3793
Norcal2	0.3568	23.3879	0.4857	25.7881	0.4778	25.9137	0.4706	26.0152
US-East	0.4124	12.4906	0.5408	18.9909	0.5038	16.9006	0.5007	20.2726
Sydney	0.2916	13.6534	0.4649	21.834	0.4865	24.5399	0.5036	31.7841

For the high quantile prediction, we study the proposed model at the aggregate level. Specifically, we take the log-transformation on both the observations and aggregated fitted values on the training and validation dataset, and calculate the residuals,

$$e_t=\log \left(s_t\right)-\log \left({\hat{s}}_t\right),$$

where $s_t=\sum _{i=1}^t{y}_i$, ${\hat{s}}_t=\sum _{i=1}^t{y}_i$, and $t=1,\dots ,4000$. The GPD distribution is then fitted to $\{e_t{\}}_{t=1,\dots ,4000}$ with the threshold of $.9$-quantile of $\{e_t{\}}_{t=1,\dots ,4000}$. The estimated parameters are $\xi =0.3710$ with standard error 0.0827, and $\sigma =0.0981$ with standard error 0.0100, which are significant. The QQ- and PP- plots are displayed in Figure 7. It is seen that PP- plot is satisfactory. However, the tails of QQ-plot is off the diagonal line, which is mainly caused by the extreme large values in the attack data. Although there are several points off the diagonal line, we still use the fitted GPD distribution to predict the high quantiles as it does not affect the main conclusion.

Figure 7.

QQ- and pp-plots of honeypot data fitted by the GPD distribution. (a) QQ-plot (b) PP-plot.

Algorithm 2 is employed on the attack data at the aggregated level. The assessment of predicted ${\hat{s}}_t$, $t=4001,\dots ,4512$ is shown in Table 5. It is seen that the prediction performance is very satisfactory. The predicted numbers of violations are quite close to the true numbers of violations. They pass all the tests based on LRuc and LRcc tests except that at $.98$ level of LRcc test is relatively small (0.0241).

Table 5.

The p-values of the VaR tests of the predicted violations for $\alpha =.92,.94,.94,.96,.98$.

α	Exp.	Ob.	LRuc	LRcc
0.92	41	41	0.9948	.7118
0.94	31	31	0.9585	0.9944
0.95	26	23	0.5919	0.5828
0.96	21	17	0.4192	0.2124
0.98	10	9	0.6894	0.0241

Note: ‘Ob.’ represents the observed number of violations and ‘Exp.’ represents the expected number of violations.

We conclude that the proposed approach has an overall satisfactory prediction performance.

5.2. Honeypot data II with 69 dimensions

In this section, we study the other honeypot data which was used in [24,31]. The dataset was collected by a low-interaction honeypot during 4 November 2010 to 21 December 2010 with a total number of 1123 hours, which has 69 consecutive IP addresses. Each TCP flow initiated by a remote computer and an unsuccessful TCP handshake are deemed as attacks [31]. The data is split into three parts: the training with 750 observations, the validation with 150 observations, and the rest 223 observations are used for the prediction evaluation. The data is log-transformed to reduce the skewness.

We employ Algorithm 1 for training and validating the deep learning model, and Algorithm 2 for the prediction. The performance of point predictions are shown in Table 6.

Table 6.

Summary statistics of predictions of proposed approach (‘Deep’), benchmark VAR, GRU, and RNN models for MAPE and MSE based on the log-transformed attack data.

Statistic	$N$	Mean	St. Dev.	Min	$Q_1$	Median	$Q_3$	Max
Deep
MAPE	69	0.091	0.017	0.036	0.084	0.088	0.095	0.142
MSE	69	0.316	0.180	0.114	0.197	0.253	0.324	1.028
Benchmark
MAPE	69	0.150	0.033	0.041	0.139	0.162	0.167	0.189
MSE	69	0.718	0.184	0.152	0.625	0.716	0.861	1.092
GRU
MAPE	69	0.118	0.018	0.037	0.115	0.120	0.125	0.156
MSE	69	0.454	0.176	0.123	0.344	0.399	0.499	1.069
RNN
MAPE	69	0.122	0.023	0.035	0.124	0.129	0.134	0.167
MSE	69	0.468	0.187	0.108	0.376	0.431	0.537	1.438

It is observed that the proposed deep learning model has a very satisfactory prediction performance. Particularly, the mean (0.091) and median (0.088) of MAPEs by the proposed approach are much smaller than the means and medians of MAPEs by the other models. For MSEs, the proposed approach also significantly outperforms the benchmark model in terms of means and medians. It is observed that the benchmark model performs worse than the GRU and RNN models in terms of mean and median. The boxplots are further shown in Figure 8. It is seen that the proposed approach indeed significantly outperforms the other models.

Figure 8.

Boxplots of MAPEs and MSEs based on various models for the log-transformed attack data. (a) MAPE-Deep (b) MAPE-Benchmark (c) MAPE-GRU (d) MAPE-RNN (e) MSE-Deep (f) MSE-Benchmark (g) MSE-GRU (h) MSE-RNN.

For the high quantiles, we also study the prediction performance at the aggregate level as it corresponds to the network level in practice where the defense can take place. Specifically, we calculate the residuals,

$$e_t=s_t-{\hat{s}}_t,$$

where $s_t=\sum _{i=1}^t{y}_i^{\prime }$, ${\hat{s}}_t=\sum _{i=1}^t{\hat{y}}_i^{\prime }$, and $y^{\prime }$s are the log-transformed attack data, $t=1,\dots ,900$. The GPD distribution is then fitted to $\{e_t{\}}_{t=1,\dots ,900}$ with the threshold to be 0.9 -quantile of $\{e_t{\}}_{t=1,\dots ,900}$. The estimated parameters are $\xi =0.2116$ with standard error 0.1226, and $\sigma =12.5542$ with standard error 2.0010, which are significant at $.1$ level. The QQ- and PP-plots are displayed in Figure 9. It is seen that PP- plot is satisfactory. The tails of QQ-plot is slightly off the diagonal line. This is again caused by the extreme large values in the attack data. But the overall fitting is fairly well.

Figure 9.

QQ- and pp-plots of the log-transformed attack data fitted by the GPD distribution. (a) QQ-plot (b) PP-plot.

For the high quantile prediction, Algorithm 2 is employed to the testing data, where the threshold is set to be the 0.9-quantile of residuals. During the prediction process, we retrain the deep learning model three times by using Algorithm 1 to further improve the prediction accuracy. The p-values of VaR tests are presented in Table 7. We first observe that the proposed model slightly underestimates the number of violations, particularly at the level 0.95 (11 vs. 17), which is mainly due to the extreme large attacks. The other levels have large p-values which indicates that the proposed model can predict them well.

Table 7.

The p-values of the VaR tests of the predicted violations for $\alpha =.95,.96,.97,.98,.99$.

α	Exp.	Ob.	LRuc	LRcc
$.95$	11	17	0.0846	0.0235
$.96$	9	12	0.2960	0.2022
$.97$	7	11	0.1117	0.2392
$.98$	4	5	0.7773	0.8548
$.99$	2	4	0.2738	0.5100

Note: ‘Ob.’ represents the observed number of violations and ‘Exp.’ represents the expected number of violations.

To conclude, the proposed model has overall satisfactory fitting and prediction performances. Since we model the multivariate dependence via deep learning architectures, the dependence is treated as a ‘black-box’. This causes the loss of interpretation of the dependence among cyber risks compared to the vine copula approach [24], which is the common issue of deep learning approach.

6. Conclusion

In this work, we propose a novel approach to predicting the multivariate cyber risks. The key idea is to provide the accurate point prediction by training the deep learning network, while employ the extreme value theory for modeling and predicting the high quantiles. One particular advantage of proposed approach is that it can easily handle high-dimensional cyber risks thanks to the deep learning architectures, and further, the deep learning model can be retrained to improve the prediction accuracy if needed. The simulation and empirical studies confirm the feasibility of proposed approach and satisfactory fitting and prediction performances.

There are some limitations for the current study. The real attack datasets studied in the current work were collected by honeypots. The other attack data (e.g. collected by telescope or other instruments) may exhibit different phenomena, which should be carefully analyzed before employing the approach developed in our work. The current work can be extended in several directions. For example, the deep learning model(s) may be further developed for more accurate fitting and prediction [6]. Second, for the complex multivariate cyber risks, the GPD distribution may not be enough for capturing the tail behavior. Then, the non-stationary extreme value distribution may be exploited for the possible modeling [5,23].

Funding Statement

Peng Zhao was supported in part by the National Natural Science Foundation of China under Grant 11871252, and A Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.

Disclosure statement

No potential conflict of interest was reported by the author(s).