Dynamic structural models with covariates for short-term forecasting of time series with complex seasonal patterns

Abstract

This work presents a framework of dynamic structural models with covariates for short-term forecasting of time series with complex seasonal patterns. The framework is based on the multiple sources of randomness formulation. A noise model is formulated to allow the incorporation of randomness into the seasonal component and to propagate this same randomness in the coefficients of the variant trigonometric terms over time. A unique, recursive and systematic computational procedure based on the maximum likelihood estimation under the hypothesis of Gaussian errors is introduced. The referred procedure combines the Kalman filter with recursive adjustment of the covariance matrices and the selection method of harmonics number in the trigonometric terms. A key feature of this method is that it allows estimating not only the states of the system but also allows obtaining the standard errors of the estimated parameters and the prediction intervals. In addition, this work also presents a non-parametric bootstrap approach to improve the forecasting method based on Kalman filter recursions. The proposed framework is empirically explored with two real time series.

1. Introduction

In modern management operations, forecasting plays a key role. Researches on forecast generally consider three main categories: long-term, medium-term and short-term forecasts [15,17]. An efficient prediction, for example, can allow a company commit its resources with greater security to make long-term profits (long-term forecast), since it helps to identify future demand patterns and facilitates the new products development. The short-term forecast, for example, is important for studying the balance of national power grid, which requires a balance between the electricity produced and consumed at any moment in the day [13].

Forecasting an event depends on how well we understand the factors that contribute to its occurrence and how much unexplained variability is involved, as well as the factors determining actual outcomes, types of data patterns, and so on. As for the method (quantitative or qualitative), the choice depends on what data are available and the predictability of the quantity to be forecast. Since the numerical information is available about the past of the phenomenon, it is reasonable to assume that some aspects of the past patterns will continue into the future, the quantitative forecasting can be applied.

There is a wide range of quantitative forecasting models for specific purposes, such as the models designed in the state-space form, which has deserved much attention by researchers, among them [2,14,21,22,24,25,27,30,31,37]. This attention is justified, on the one hand, by the fact that state-space structures are optimal and flexible for class of exponential smoothing models [25]; on the other hand, such models are very flexible to incorporate covariates effects [13,16,37,44], as well as to accommodate resampling methods such as bootstrap methodology [12,28,37,38].

Regardless of how these methods model a time series, implicitly some of these models assume that past observations of a time series contain all information required for forecasting its future; that is, they forecast the future of a time series using only its past observations [44]. For this class of models without covariates (with reference to short-term forecast), the most common models of state-space structure include those underlying the well-known additive and multiplicative methods of Holt-Winters [18,22,25,33,39–42].

It is obvious that the history of a time series certainly contains information about its future. However, other information beyond what is available in a time series history can also shed light on the series movements over time, therefore, lead to more accurate forecasting of its future if incorporated [44]. Such other information can be provided by so-called external influence variables (or covariates). Proposals of this type of models can be seen in [8] with SARIMA (Seasonal Auto-Regressive Integrated Moving Average) and [2,13,16,27,44].

As Wang [44] also found, the projection of forecast models with covariates has two fundamental advantages: (i) to take into account the history of the time series of interest and the information hidden in quantifiable covariates, may lead to a more accurate forecast of the time series of interest; (ii) to forecast, one only needs to know how the time series of interest move over time. But, in order to make the series to move in the desired directions, for example, to estimate (in order to reduce or increase) the maximum production of electric power due to demand, we need to understand the reasons behind this demand. Such knowledge can often be learned from its relation to other variables. In this context, the essential methodological challenge is the ability to relate the history of the time series to exogenous factors or covariates [16]. Still within the advantages (or importance) of using covariates for short-term forecasting, we highlight the research works of [7,13,15,16,23,32,36].

The focus of this work is primarily to explore the use of covariates on short-term forecasting of time series with complex seasonal patterns. The framework proposed is inspired by De Livera et al. [1], who introduced two structures, BATS as an acronym for the main features of the model: Box-Cox transformation, ARMA errors, Trend, and Seasonal components and TBATS with the initial T connoting Trigonometric. The authors estimate the parameters of these models by exponential smoothing. Among the advantages presented by TBATS we highlight the following: (i) the ability to accommodate data with non-integer seasonal periods, high-frequency data and dual calendar effects data; (ii) the Box-Cox transformation that allows to deal with the non-linearity of the data; (iii) the ARMA process on residuals to solve the autocorrelation problem. However, we notice that TBATS models are related to ETS models, tbats() is fully automatic, is unlikely to over include covariates. The fact that TBATS does not allow for covariates which may be important for short-term forecasting may be pointed out as a disadvantage of these models.

Our proposed framework differs from TBATS approach in four key aspects, namely: (i) the framework is a state-space formulation with multiple sources of randomness (MSR); (ii) the models do not incorporate smoothing parameters; (iii) the models deal with covariates, but they can work without the covariates; (iv) in the estimation procedure, we used the Kalman filter (with recursive adjustment of the covariance matrices) to obtain one-step-ahead forecasting errors and associated variances needed for evaluating fitting criteria for given trial values of the parameters. For the proposed framework, the Kalman filter is appropriate in particular. The choice of MSR formulation is justified by the fact that it is more flexible way to treat the covariates in the proposed framework and using Kalman filter as an algorithm for statistical treatment. In addition, once our proposed framework does not incorporate the smoothing parameters, this fact limits the possibility of having many parameters to estimate. From the forecast point of view, there is a growing interest among researchers in combining the resampling methods with the state-space models [4,11,12,19,28,37,38,46]. With these ideas, we formulated a bootstrap procedure based on the residuals trigonometric structural model with covariates.

Our subsequent study has three objectives: (i) to explore the use of covariates in short-term forecasting of time series with complex seasonal patterns; (ii) to construct a Kalman filter with recursive adjustment of covariance matrices and to project a computational procedure for estimation of the proposed models; (iii) to construct a bootstrap procedure for forecasting. The rest of the work is organized as follows: In Section 2, we provide a brief review of TBATS models. We then introduce in this Section the new proposed structural model with covariates (TSCov–Trigonometric Structural models with Covariates) including its state-space representation and the Kalman filter with recursively covariances computed. Section 3 contains the empirical fitting of proposed model. Specifically addresses the maximum likelihood estimation, the new computational procedure and the model selection criterion. Section 4 provides the forecasting strategies. Two procedures are presented: the first is related to the direct use of Kalman filter recursions, the second is based on bootstrap method. The proposed model is then applied in Section 5. Conclusion and future direction are drawn in Section 6.

2. Models for time series with complex seasonal patterns

2.1. TBATS models

TBATS models which is BATS model plus Trigonometric Seasonal models was proposed by De Livera et al. [1] in order to overcome some weaknesses found on the traditional seasonal exponential smoothing models. The BATS model is the most obvious generalization of the traditional seasonal innovations models to allow for multiple seasonal periods [1], formulated as follows:

$$\begin{array}{rl}{y}_t^{(\omega )}& =\{\begin{array}{ll}\frac{y_t^{\omega }-1}{\omega }& \mathrm{se}\omega \ne 0\\ \log y_t& \mathrm{se}\omega =0\end{array}\end{array}$$ (1a)

$$\begin{array}{rl}{y}_t^{(\omega )}& ={\ell }_{t-1}+\varphi b_{t-1}+\sum _{i=1}^T{s}_{t-m_i}^{(i)}+d_t\end{array}$$ (1b)

$$\begin{array}{rl}{\ell }_t& ={\ell }_{t-1}+\varphi b_{t-1}+\alpha d_t\end{array}$$ (1c)

$$\begin{array}{rl}{b}_t& =(1-\varphi )b+\varphi b_{t-1}+\beta d_t\end{array}$$ (1d)

$$\begin{array}{rl}{s}_t^{(i)}& =s_{t-m_i}^{(i)}+\gamma {\gamma }_i{d}_t\end{array}$$ (1e)

$$\begin{array}{rl}{d}_t& =\sum _{i=1}^p{\phi }_i{d}_{t-i}+\sum _{i=1}^q{\theta }_i{\epsilon }_{t-i}+{\epsilon }_t\end{array}$$ (1f)

Furthermore, De Livera et al. [1] found that BATS model cannot accommodate non-integer seasonality. In the quest for a more flexible parsimonious approach, the authors introduced the following trigonometric representation of seasonal components based on Fourier series:

$$\begin{array}{rl}{s}_t^{(i)}& =\sum _{j=1}^{k_i}{s}_{j,t}^{(i)}\end{array}$$ (2a)

$$\begin{array}{rl}{s}_{j,t}^{(i)}& =s_{j,t-1}^{(i)}\cos {\lambda }_j^{(i)}+s_{j,t-1}^{\ast (i)}\sin {\lambda }_j^{(i)}+\gamma {\gamma }_1^{(i)}{d}_t\end{array}$$ (2b)

$$\begin{array}{rl}{s}_{j,t}^{\ast (i)}& =-s_{j,t-1}^{(i)}\sin {\lambda }_j^{(i)}+s_{j,t-1}^{\ast (i)}\cos {\lambda }_j^{(i)}+\gamma {\gamma }_2^{(i)}{d}_t\end{array}$$ (2c)

where $m_1,\dots ,m_T$ denote the seasonal periods, ${\ell }_t$ is the local level in period t, b is the long-run trend, $b_t$ is the short-run trend in period t, $s_t^{(i)}$ represents the ith seasonal component at time t, $d_t$ denotes an ARMA(p, q) process, and ${\epsilon }_t$ is a Gaussian white-noise process with zero mean and constant variance ${\sigma }^2$. The smoothing parameters are given by $\alpha ,\beta$, $\gamma {\gamma }^{(i)}$, $\gamma {\gamma }_1^{(i)}$ and $\gamma {\gamma }_2^{(i)}$ for $i=1,\dots ,T$, and ${\lambda }_j^{(i)}=2\pi j/m_i$. The stochastic level of the ith seasonal component is described by $s_{j,t}^{(i)}$, and the stochastic growth in the level of the ith seasonal component that is needed to describe the change in the seasonal component over time is described by $s_{j,t}^{\ast (i)}$. The number of harmonics required for the ith seasonal component is denoted by $k_i$. So the new class designated by TBATS, is obtained by replacing the seasonal component $s_t^{(i)}$ in Equation (1) by the trigonometric seasonal formulation, and the measurement equation by

$$y_t^{(\omega )}={\ell }_{t-1}+\varphi b_{t-1}+\sum _{i=1}^T{s}_{t-1}^{(i)}+d_t.$$ (3)

From a practical point of view, as stated above, tbats() is fully automatic and is unlikely to over include covariates. In contrast, we introduce a new state-space model with multiple sources of randomness and covariates, as shown in the next section.

2.2. The TSCov model

Let $\{Y_t\}=\{y_1,\dots ,y_n\}$ be the observed time series and $z_{\kappa ,t}$ ( $\kappa =1,\dots ,r$) the set of regressor variables. We notice that our study is limited for cases of additive trends and seasonality. In addition, to deal with non-linearity problems, it is assumed that TSCov model is applicable to a Box-Cox transformation. Using the same notation as in (1) and (2), the regressor variables may be incorporated into the TSCov model as follows:

$$\begin{array}{rl}{y}_t^{(\omega )}& =\{\begin{array}{ll}\frac{y_t^{(\omega )}-1}{\omega }& \mathrm{se}\omega \ne 0\\ \ln y_t& \mathrm{se}\omega =0\end{array}\end{array}$$ (4a)

$$\begin{array}{rl}{y}_t^{(\omega )}& ={\ell }_{t-1}+\varphi b_{t-1}+\sum _{i=1}^T{s}_{t-m_i}^{(i)}+\sum _{\kappa =1}^r{\beta }_{\kappa }^{\ast }{z}_{\kappa ,t}+{\epsilon }_t;{\epsilon }_t\sim iid\mathcal{N}(0,{\sigma }_{\epsilon }^2)\end{array}$$ (4b)

$$\begin{array}{rl}{\ell }_t& ={\ell }_{t-1}+\varphi b_{t-1}+{\xi }_t;{\xi }_t\sim iid\mathcal{N}(0,{\sigma }_{\xi }^2)\end{array}$$ (4c)

$$\begin{array}{rl}{b}_t& =(1-\varphi )b+\varphi b_{t-1}+{\zeta }_t;{\zeta }_t\sim iid\mathcal{N}(0,{\sigma }_{\zeta }^2)\end{array}$$ (4d)

$$\begin{array}{rl}{s}_t^{(i)}& =\sum _{j=1}^{k_i}{s}_{j,t}^{(i)}\end{array}$$ (4e)

$$\begin{array}{rl}{s}_{j,t}^{(i)}& =s_{j,t-1}^{(i)}\cos {\lambda }_j^{(i)}+s_{j,t-1}^{\ast (i)}\sin {\lambda }_j^{(i)}+e_{j,t}^{(i)}\end{array}$$ (4f)

$$\begin{array}{rl}{s}_{j,t}^{\ast (i)}& =-s_{j,t-1}^{(i)}\sin {\lambda }_j^{(i)}+s_{j,t-1}^{\ast (i)}\cos {\lambda }_j^{(i)}+e_{j,t}^{\ast (i)}\end{array}$$ (4g)

We assume that $e_{j,t}^{(i)}=e_{j,t}^{\ast (i)}\sim iid\mathcal{N}(0,{\sigma }_e^{2(i)})$ and ${\epsilon }_t,{\xi }_t,{\zeta }_t,e_{j,t}^{(i)}$ are independent processes. As well as in (1) and (2), $m_1,\dots ,m_T$ represent the seasonal periods and T denotes the seasonal patterns; ${\lambda }_j^{(i)}=2\pi j/m_i$ ( $j=1,2,\dots ,k_i$ and $i=1,\dots ,T$); ${\ell }_t$ and $b_t$ are the local level and the short-term trend in period t; b is the long-term trend. $s_t^{(i)}$ and $s_{j,t}^{(i)}$ denotes also the seasonal component in period t and the stochastic level of the ith seasonal component and $s_{j,t}^{\ast (i)}$ is the stochastic growth in the level of the ith seasonal component that is needed to describe the change in the seasonal component over time. $k_i$ is also the number of harmonics required for the ith seasonal component, whose approach is equivalent to index seasonal approaches when $k_i=m_i/2$ for even values of $m_i$, and when $k_i=(m_i-1)/2$ for odd values of $m_i$. The transition parameter is bounded by $0<\varphi \le 1$ to prevent a negative coefficient being applied to $b_t$. In case $\varphi =0$, it would indicate the absence of the trend in time series.

2.2.1. State-space representation

The linear state-space models can be extended to incorporate the fixed-effects regression. Such regression effects can be included in one or two different ways: (i) the first approach is including exogenous or predetermined variables in the signal equation; (ii) the second approach is including the exogenous or predetermined variables in the state equation. The above model with fixed-effects regression may be written in a state-space formulation. We adopt $z{z}_t$ to represent the vector containing any control inputs (meteorological variables, for example, air temperature, wind-speed, relative humidity or predetermined variables, it may also contain the indicator variables) and $\Gamma \Gamma$ to represent the control input matrix (coefficient matrix formed by regression coefficients ${\mathit{\beta }}_{\kappa }^{\ast }$ which applies the effect of each control input parameter in $z{z}_t$ on the observation vector, for example, applies the effect of temperature on the electricity consumption. The vector $\Gamma \Gamma$ contains unknown parameters but these do not affect the stochastic properties of the model, only enter the model in a deterministic way, that is, the parameters appearing in $\Gamma \Gamma$ only affect the expected value of the observations $y{y}_t$ in a deterministic way. This distinction can become blurred, for example, if $\Gamma \Gamma$ is a function of a lagged value of $y{y}_t$. If $\Gamma \Gamma$ is a linear function of unknown parameters, these parameters can be treated as state variables [21]. So, we wrote the Gaussian linear model in state-space representation as

$$\begin{array}{rl}{\mathit{y}}_t^{(\omega )}& ={\mathit{A}}_t{\mathit{x}}_t+\mathit{\Gamma }{\mathit{z}}_t+{\mathit{\nu }}_{t}t=1,2,\dots ,n\end{array}$$ (5a)

$$\begin{array}{rl}{\mathit{x}}_t& =\mathit{\Phi }{\mathit{x}}_{t-1}+\eta {\eta }_{t}t=1,2,\dots ,n\end{array}$$ (5b)

where $A{A}_t$ is a $q\times p$ measurement or observation matrix; (5a) is called the observation equation. The observed data vector, ${\mathit{y}}_t$, is q-dimensional, which can be larger than or smaller than p, the state dimension. (5b) is called the state equation, $\mathit{\Phi }$ is a $p\times p$ transition matrix. We suppose we have an $r\times 1$ vector of inputs ${\mathit{z}}_t$ and $\Gamma \Gamma$ is $q\times r$ matrix. ${\mathit{\nu }}_t$ and ${\mathit{\eta }}_t$ are white noises, being that:

$$\begin{array}{rl}E({\mathit{\nu }}_t{\mathit{\nu }}_t^{\mathrm{\prime }})& =\mathit{R}\end{array}$$ (6a)

$$\begin{array}{rl}E({\mathit{\eta }}_t{\mathit{\eta }}_t^{\mathrm{\prime }})& =\mathit{Q}\end{array}$$ (6b)

Furthermore, ${\mathit{\nu }}_t$ and ${\mathit{\eta }}_t$ are assumed to be uncorrelated:

$$E({\mathit{\nu }}_t{\mathit{\eta }}_t)=0$$ (7)

${\mathit{x}}_t$ represents the unobserved state vector. The observation and transition models are represented by ${\mathit{A}}_t$ and $\mathit{\Phi }$ matrices, respectively. Given (7), Equation  (5b) is typically used to describe a finite time series of observations $y_1,y_2,\dots ,y_n$ and for which assumptions about the initial value of the state vector are necessary [20]. It is assumed that ${\mathit{x}}_t$ is uncorrelated with any realization of ${\mathit{\nu }}_t$ or ${\mathit{\eta }}_t$:

$$\begin{array}{rl}E({\mathit{\nu }}_t{\mathit{x}}_t^{\mathrm{\prime }})& =0\mathrm{for}t=1,2,\dots ,n\end{array}$$ (8a)

$$\begin{array}{rl}E({\mathit{\eta }}_t{\mathit{x}}_t^{\mathrm{\prime }})& =0\mathrm{for}t=1,2,\dots ,n\end{array}$$ (8b)

The statement that ${\mathit{z}}_t$ is exogenous or predetermined means that ${\mathit{z}}_t$ does not provide information about the ${\mathit{x}}_{t+h}$ or ${\mathit{\eta }}_{t+h}$ for $h=1,2,\dots$ beyond the information contained in $y_{t-1},y_{t-2},\dots ,y_1$. So, the system matrices (5) for TSCov model can be obtained by defining: (i) the state vector, ${\mathit{x}}_t=\{{\ell }_t,b_t,s_{1,t}^{(i)},s_{2,t}^{(i)},\dots ,s_{k_i,t}^{(i)},s_{1,t}^{\ast (i)},s_{2,t}^{\ast (i)},\dots ,s_{k_i,t}^{\ast (i)}\}$, (ii) the replica vector of 0 and 1 (which depends on the seasonal component) defined as $\mathit{a}=({\mathit{a}}^{(1)},\dots ,{\mathit{a}}^{(T)})$ with ${\mathit{a}}^{(i)}=({\mathbf{1}}_{k_i},{\mathbf{0}}_{k_i})$ and $\tau =2\sum _{i=1}^T{k}_i$. (iii) we also need to define the block matrix, $\mathit{B}$, resulting from the direct sum, $⨁$, of the matrices ${\mathit{B}}_i$, that is, $\mathit{B}=\underset{i=1}{\overset{T}{⨁}}{\mathit{B}}_i$,

$${\mathit{B}}_i=\left[\begin{array}{cc}{\mathit{C}}^{(i)}& {\mathit{S}}^{(i)}\\ -{\mathit{S}}^{(i)}& {\mathit{C}}^{(i)}\end{array}\right]$$

where, ${\mathit{C}}^{(i)}$ and ${\mathit{S}}^{(i)}$ are diagonal matrices of size $k_i\times k_i$ with elements $\cos ({\lambda }_j^{(i)})$ and $\sin ({\lambda }_j^{(i)})$, for $j=1,2,\dots ,k_i$.

$$\mathit{B}=\underset{i=1}{\overset{T}{⨁}}{\mathit{B}}_i=\left[\begin{array}{cccc}{\mathit{B}}_1& \cdots & & \mathbf{0}\\ & {\mathit{B}}_2& \\ ⋮& ⋮& \ddots & ⋮\\ \mathbf{0}& \cdots & & {\mathit{B}}_T\end{array}\right]=\left[\begin{array}{cccc}\cos {\lambda }_j^{(1)}& \sin {\lambda }_j^{(1)}& \cdots & \mathbf{0}\\ -\sin {\lambda }_j^{(1)}& \cos {\lambda }_j^{(1)}& \\ ⋮& ⋮& \ddots & ⋮\\ & & \cos {\lambda }_j^{(T)}& \sin {\lambda }_j^{(T)}\\ \mathbf{0}& \cdots & -\sin {\lambda }_j^{(T)}& \cos {\lambda }_j^{(T)}\end{array}\right]$$

The covariance matrices for measurement and the state are given by

$$\mathit{R}={\sigma }_{\epsilon }^2\mathrm{and}{\mathit{Q}}^{(i)}=\left[\begin{array}{ccc}{\sigma }_{\xi }^2& 0& 0\\ 0& {\sigma }_{\zeta }^2& 0\\ 0& 0& {\tilde{\mathit{q}}}^{(i)}\end{array}\right]$$

where ${\sigma }_{\epsilon }^2$ is the noise variance in the measurement equation, ${\sigma }_{\xi }^2$ and ${\sigma }_{\zeta }^2$, the model variances corresponding to the level and trend components, ${\tilde{\mathit{q}}}^{(i)}$ given in (9) is the model variance corresponding the seasonal component.

$$\begin{array}{rl}{\tilde{\mathit{q}}}^{(i)}& =\{{\tilde{\mathit{\sigma }}}_e^{2(1)},\dots ,{\tilde{\mathit{\sigma }}}_e^{2(T)}\},\mathrm{where}{\tilde{\mathit{\sigma }}}_e^{2(i)}=\{{\mathit{\sigma }}_e^{2(i)},{\mathit{\sigma }}_{e^{\ast }}^{2(i)}\}\mathrm{and}\end{array}$$ (9a)

$$\begin{array}{rl}{\mathit{\sigma }}_e^{2(i)}& ={\sigma }_e^{2(i)}{\mathbf{1}}_{k_i}\end{array}$$ (9b)

$$\begin{array}{rl}{\mathit{\sigma }}_{e^{\ast }}^{2(i)}& ={\sigma }_{e^{\ast }}^{2(i)}{\mathbf{1}}_{k_i}\end{array}$$ (9c)

By modeling in this way, we allow the seasonal component noise to have a dual function: (i) to be the source of randomness for seasonal component; (ii) propagate the randomness effect on the stochastically variant trigonometric coefficients terms over time. This way of modeling the seasonal component variances it is similar to the methodology used by De Livera et al. [1] to model the smoothing parameter of seasonal component.

The homoscedastic exponential smoothing model (BATS) with covariates can be obtained by letting ${\mathit{x}}_t=\{l_t,b_t,s_t^{(i)},s_{t-1}^{(i)},\dots ,s_{t-(m_i-1)}^{(i)}\}$, ${\mathit{a}}^{(i)}=({\mathbf{0}}_{m_i-1},1)$, $\mathit{B}=\underset{i=1}{\overset{T}{⨁}}{\tilde{\mathit{D}}}_i$, and by replacing $2k_i$ with $m_i$ in the matrices presented above for the TSCov model, that is, $\tau =\sum _{i=1}^T{m}_i$,

$${\tilde{\mathit{D}}}_i=\left[\begin{array}{c}{\mathbf{0}}_{m_i-1}\mathbf{1}\\ {\mathit{I}}_{m_i-1}{\mathbf{0}}_{m_i-1}^{\mathrm{\prime }}\end{array}\right]$$

where ${\tilde{\mathit{q}}}^{(i)}=({{\tilde{\mathit{\sigma }}}_{\mathit{w}}}^{2(i)},{\mathbf{0}}_{m_i-1})$ and ${\tilde{\mathit{\sigma }}}_{\mathit{w}}^{2(i)}={\sigma }_{\mathit{w}}^{2(i)}{\mathbf{1}}_{m_i}$.

2.2.2. Kalman filter with recursively computed covariances

Given (5), according to the principles featuring a state-space model, the Kalman predictor used when t>s and the Kalman filter applied when t = s are given by (10). For more details, see [37].

$$\begin{array}{rl}{\hat{\mathit{x}}}_{t|t-1}& =\mathit{\Phi }{\mathit{x}}_{t-1|t-1}\end{array}$$ (10a)

$$\begin{array}{rl}P{P}_{t|t-1}& =\mathit{\Phi }{\mathit{P}}_{t-1|t-1}{\mathit{\Phi }}^{\mathrm{\prime }}+\mathit{Q}\end{array}$$ (10b)

$$\begin{array}{rl}{\mathit{x}}_{t|t}& ={\hat{\mathit{x}}}_{t|t-1}+{\mathit{K}}_t{\mathit{ϵ}}_t\end{array}$$ (10c)

$$\begin{array}{rl}{\mathit{P}}_{t|t}& ={\mathit{P}}_{t|t-1}-{\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }}{\mathit{\Sigma }}_t^{-1}{\mathit{A}}_t{\mathit{P}}_{t|t-1}\end{array}$$ (10d)

where

$$\begin{array}{rl}{\mathit{K}}_t& ={\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }}[{\mathit{A}}_t{\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }}+{\mathit{R}}_t{]}^{-1}\end{array}$$ (11a)

$$\begin{array}{rl}{\mathit{ϵ}}_t& ={\mathit{y}}_t^{(\omega )}-{\mathit{A}}_t{\hat{\mathit{x}}}_{t|t-1}-\mathit{\Gamma }{\mathit{z}}_t\end{array}$$ (11b)

$$\begin{array}{rl}{\mathit{\Sigma }}_t& =\mathrm{Var}({\mathit{ϵ}}_t)={\mathit{A}}_t{\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }}+\mathit{R}\end{array}$$ (11c)

When performing a Kalman filter, the correct setting of the covariance matrices is critical, since the Kalman filter performance is highly affected by the system covariance matrices. Its inadequate choice can significantly degrade the performance of Kalman filter and even make the filter divergent [29]. It is quite common to use ad-hoc procedures to determine the system of covariance matrices, such as conventional filters [8,14,37], among others, in which $\mathit{Q}$ and $\mathit{R}$ are constant matrices and adjusted manually by trial and error. To address this challenge, a Kalman filter with recursively computed covariance is constructed with reference to [3] and [43] approaches. Now we denote the covariance matrices of measurement and state equations variants over time as $R_t$ and $Q_t$. The procedure that we apply is based on the innovations (a priori and a posteriori) of the model; these will influence the adjustment of covariance matrices recursively to improve the accuracy of state estimate. Given (10c), first, set up the a posteriori innovations as

$${\mathit{\varsigma }}_t={\mathit{y}}_t^{(\omega )}-{\mathit{A}}_t{\mathit{x}}_{t|t}-\mathit{\Gamma }{\mathit{z}}_t,$$ (12)

then, define the covariance estimates ${\mathit{R}}_t$ and ${\mathit{Q}}_t$ that participate in the recursive process by synchronizing them with a priori ( ${\mathit{ϵ}}_t$) and a posteriori ( ${\mathit{\varsigma }}_t$) innovations.

For measurement covariance estimation, ${\mathit{R}}_t$. Given (11c), the measurement covariance can be given by

$${\mathit{R}}_t={\mathit{\Sigma }}_t-{\mathit{A}}_t{\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }}$$ (13)

Theoretically ${\mathit{\Sigma }}_t$ should be set positive. However, the Equation (13) does not guarantee the positivity of the estimated matrix, ${\mathit{R}}_t$, because it results from the subtraction of two positive definite matrices. According to (6a), it is ensured that ${\mathit{R}}_t$ is a positive definite matrix by combining covariance with a posteriori innovations, ${\mathit{\varsigma }}_t$, as in (14).

$$\begin{array}{rl}{\mathit{\Sigma }}_t^{\ast }& =E[{\mathit{\varsigma }}_t{\mathit{\varsigma }}_t^{\mathrm{\prime }}]=E[{\mathit{\nu }}_t{\mathit{\nu }}_t^{\mathrm{\prime }}]-{\mathit{A}}_t{\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }}\\ {\mathit{R}}_t& =E[{\mathit{\varsigma }}_t{\mathit{\varsigma }}_t^{\mathrm{\prime }}]+{\mathit{A}}_t{\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }}\end{array}$$ (14)

where, the operation on ${\mathit{\varsigma }}_t{\mathit{\varsigma }}_t^{\mathrm{\prime }}$ is usually approximated by averaging ${\mathit{\varsigma }}_t{\mathit{\varsigma }}_t^{\mathrm{\prime }}$ over time t [29]. Instead, we apply the procedure used by Akhlaghi et al. [3], which consists of applying a forgetting factor, $0<\delta \le 1$, to estimate the covariance adaptively, as defined in (15).

$${\mathit{R}}_t=\delta {\mathit{R}}_{t-1}+(1-\delta )({\mathit{\varsigma }}_t{\mathit{\varsigma }}_t^{\mathrm{\prime }}+{\mathit{A}}_t{\mathit{P}}_{t|t-1}{\mathit{A}}_t^{\mathrm{\prime }})$$ (15)

For state covariance estimation, ${\mathit{Q}}_t$. Given (5b), the state covariance estimate, ${\mathit{Q}}_t$ can be obtained by doing ${\mathit{\eta }}_t={\mathit{x}}_t-\mathit{\Phi }{\mathit{x}}_{t-1}$. Since ${\mathit{x}}_{t|t}$ is the estimator of ${\mathit{x}}_{t|t-1}$, given (10c), the estimated state error can be given by ${\hat{\mathit{\eta }}}_t={\mathit{x}}_{t|t}-\mathit{\Phi }{\hat{\mathit{x}}}_{t|t-1}={\mathit{K}}_t{\mathit{ϵ}}_t$. Its covariance is

$$E({\hat{\mathit{\eta }}}_t{\hat{\mathit{\eta }}}_t^{\mathrm{\prime }})=E[{\mathit{K}}_t({\mathit{ϵ}}_t{\mathit{ϵ}}_t^{\mathrm{\prime }}){\mathit{K}}_t^{\mathrm{\prime }}]={\mathit{K}}_{t}E({\mathit{ϵ}}_t{\mathit{ϵ}}_t^{\mathrm{\prime }}){\mathit{K}}_t^{\mathrm{\prime }}$$ (16)

From (16), given (11c), the covariance estimate of state is given by ${\hat{\mathit{Q}}}_t={\mathit{K}}_t{\mathit{\Sigma }}_t{\mathit{K}}_t^{\mathrm{\prime }}$. Using the same procedure, the estimate of ${\mathit{Q}}_t$ over time is given by (17)

$${\mathit{Q}}_t=\delta {\mathit{Q}}_{t-1}+(1-\delta )({\mathit{K}}_t{\mathit{ϵ}}_t{\mathit{ϵ}}_t^{\mathrm{\prime }}{\mathit{K}}_t^{\mathrm{\prime }})$$ (17)

3. Empirical fitting of TSCov model

3.1. Maximum likelihood estimation

The approach used was obtaining the conditional distribution $p({\mathit{x}}_t|y_t^{(\omega )})$ of the state ${\mathit{x}}_t$ for a set of observations ${\mathit{Y}}_{t-1}$, [26,47,48]. We calculate the conditional densities and the classical maximum likelihood theory based on the situation by which the n transformed observations, $y_1^{(\omega )},y_2^{(\omega )},\dots ,y_n^{(\omega )}$, are independent and identically distributed with $\mathbf{\Omega }$ a vector of unknown parameters, allowed us defining the joint density function as

$$\mathcal{L}({\mathit{y}}_t^{(\omega )},\mathbf{\Omega })=\prod _{t=1}^{n}p({\mathit{y}}_t^{(\omega )}|{\mathit{Y}}_{t-1})$$ (18)

where $p({\mathit{y}}_t^{(\omega )}|{\mathit{Y}}_{t-1})$ describes the distribution of ${\mathit{y}}_t^{(\omega )}$ conditioned on the established information in the period t−1, that is, ${\mathit{Y}}_{t-1}=\{y_{t-1},y_{t-2},\dots ,y_1\}$ [21]. Since ${\mathit{x}}_0\sim iid\mathcal{N}({\mu }_0,{\mathit{\Sigma }}_0)$, the distribution of ${\mathit{y}}_t^{(\omega )}$ conditioned to ${\mathit{Y}}_{t-1}$ is in itself normal. The mean and covariance of this distribution are given by Kalman filter from the derivations and the likelihood is calculated using (11b) and (11c). Then, the likelihood of the model (5) at time t for time series possibly transformed¹ is given as

$$\begin{array}{rl}\mathcal{L}(\Omega \Omega )& =\prod _{t=1}^n{g}_t(y_t^{(\omega )}|y_1,\dots ,y_{t-1},\Omega \Omega )=\prod _{t=1}^n{g}_t(y_t^{(\omega )}|{\mathbf{y}}_{1:t-1},\Omega \Omega )\mathrm{where}\\ g_t(y_t^{(\omega )}|{\mathbf{y}}_{1:t-1},\Omega \Omega )& ={\left(\frac{1}{\sqrt{2\pi }}\right)}^{\kappa }|{\Sigma }_t{|}^{-(1/2)}exp\{-\frac{1}{2}ϵ{ϵ}_t^{\mathrm{\prime }}\Sigma {\Sigma }_t^{-1}ϵ{ϵ}_t\},\mathrm{then}\\ g_t(y_t|{\mathbf{y}}_{1:t-1},\Omega \Omega )& =g_t(y_t^{(\omega )}|{\mathbf{y}}_{1:t-1},\Omega \Omega )|\det \left(\frac{\mathrm{\partial }{y}_t^{(\omega )}}{\mathrm{\partial }{y}_t}\right)|=g_t(y_t^{(\omega )}|{\mathbf{y}}_{1:t-1},\Omega \Omega )\prod _{t=1}^n{y}_t^{\omega -1}\\ & ={\left(\frac{1}{\sqrt{2\pi }}\right)}^{\kappa }|{\Sigma }_t{|}^{-(1/2)}\exp \{-\frac{1}{2}ϵ{ϵ}_t^{\mathrm{\prime }}\Sigma {\Sigma }_t^{-1}ϵ{ϵ}_t\}\prod _{t=1}^n{y}_t^{\omega -1}\end{array}$$

The log-likelihood is given by

$$\mathcal{L}(\Omega \Omega )=-\frac{\kappa n}{2}\log (2\pi )-\frac{1}{2}\log |\Sigma {\Sigma }_t|-\frac{1}{2}\sum _{t=1}^nϵ{ϵ}_t^{\mathrm{\prime }}\Sigma {\Sigma }_t^{-1}ϵ{ϵ}_t+(\omega -1)\sum _{t=1}^n\log y_t$$

Multiplying this expression by $-1$ and omitting constant terms, we get

$$-\mathcal{L}(\Omega \Omega )=\frac{1}{2}\log |\Sigma {\Sigma }_t|+\frac{1}{2}\sum _{t=1}^nϵ{ϵ}_t^{\mathrm{\prime }}\Sigma {\Sigma }_t^{-1}ϵ{ϵ}_t-(\omega -1)\sum _{t=1}^n\log y_t$$ (19)

If there is no need the Box–Cox transformation, the log-likelihood is given by

$$-\mathcal{L}(\Omega \Omega )=\frac{1}{2}\log |\Sigma {\Sigma }_t|+\frac{1}{2}\sum _{t=1}^nϵ{ϵ}_t^{\mathrm{\prime }}\Sigma {\Sigma }_t^{-1}ϵ{ϵ}_t$$ (20)

3.2. Computational procedure

The functions (19) and (20) are highly nonlinear and complicated functions of the unknown parameters. The procedure is to fix the initial state vector ${\mathit{x}}_0$ and develop a recursive process for log-likelihood function and successively apply the Newton–Raphson algorithm to update the parameter estimates until the log-likelihood is minimized. The optimization process is combined with the Kalman filter and conditioned according to the necessity or not of the Box–Cox transformation. The adopted seasonal formulation for TSCov model requires the estimation of seasonal initial values $2(k_1,k_2,\dots ,k_T)$. In this work, we apply the method described by De Livera et al. [1], based on the multiple linear regression for selecting the appropriate number of harmonics in the trigonometric terms.

The parameters are incorporated in the Kalman filter by the following procedure: (i) construct a vector of unknown parameters, $\mathit{\Omega }=\{{\sigma }_{\epsilon },{\sigma }_{\xi },{\sigma }_{\zeta },\varphi ,{\mathit{\sigma }}_e^{2(i)},{\mathit{\sigma }}_{e^{\ast }}^{2(i)}\}$ conditioned to seasonal patterns, Box–Cox transformation (indicate whether to use the Box-Cox transformation or no) and damping parameter (indicate whether to include a damping parameter in the trend or not) and incorporate it into the log-likelihood function of the model, (ii) incorporate the log-likelihood function in the Kalman filter to ensure step 2 described below. According to the steps 3 to 5, the optimal values of parameters are determined by minimizing the MSE (Mean Squared Error) of the one-step-ahead forecasting errors, through the Newton–Rapson method using the optim r function under the L-BFGS-B method. We formed a unique, recursive and systematic process combining the Kalman filter and the multiple linear regression to the Newton–Rapson method. We summarize the iterative procedure as follows:

1. Select the initial values of the parameters, ${\mathbf{\Omega }}^{(0)}$. In this step, the transition parameter is configured with TRUE/FALSE to indicate whether the final model should or not include damping in the trend. Being set to NULL, the previous two cases are tried and from AIC the best fitted is selected;

2. Run the Kalman filter using the initial values of the parameters, ${\mathbf{\Omega }}^{(0)}$, to obtain the set of innovations and covariance, that is, $\{{\mathit{ϵ}}_t^{(0)};t=1,\dots ,n\}$ and $\{{\mathit{\Sigma }}_t^{(0)};t=1,\dots ,n\}$;

3. Perform one iteration of the Newton-Rapson procedure by taking $-\ln [\mathcal{L}(\mathbf{\Omega })]$ as a criterion function to obtain the new set of estimates, therefore, ${\mathbf{\Omega }}^{(1)}$. In this step, the selection procedure of harmonics for seasonal component comes into play;

4. At iteration j, $(j=1,2,\dots )$, repeat step $(ii)$ using ${\mathbf{\Omega }}^{(j)}$ in place of ${\mathbf{\Omega }}^{(j-1)}$ to obtain a new set of innovations values $\{{\mathit{ϵ}}_t^{(j)}\}$ and $\{{\mathit{\Sigma }}_t^{(j)}\}$. Then, step $(iii)$ is repeated to obtain new estimates, ${\mathbf{\Omega }}^{(j+1)}$.

5. While run step $(iii)$ in $(iv)$, the Kalman filter is updated with new estimates ${\mathbf{\Omega }}^{(j+1)}$. The process ends when the estimates or the likelihood stabilize.

Standard errors of parameters estimates (St.Error). Since we use Newton's procedure, the Hessian matrix at the time of convergence may be used as an estimate to obtain the standard errors estimates, that is, we included a numerical evaluation of the Hessian matrix of $-\ln [\mathcal{L}(\hat{\mathbf{\Omega }})]$, where $\widehat{\Omega \Omega }$ is the vector of estimated parameters at the time of convergence.

3.3. Model selection

Let ${\mathit{x}}_0$ be the initial state vector and $\mathbf{\Omega }$ the vector of unknown parameters. The Information Criterion

$$AIC={\mathcal{L}}^{\ast }(\hat{\mathbf{\Omega }},{\hat{\mathit{x}}}_0)+2(\mathrm{\Delta }+\varrho )$$ (21)

is used for choosing between the models, where Δ is the number of parameters in $\mathbf{\Omega }$ and ϱ is the number of estimated states, and $\hat{\mathit{\Omega }}$ and ${\hat{\mathit{x}}}_0$ denote the estimates of $\mathbf{\Omega }$ and ${\mathit{x}}_0$.

4. Forecasting

4.1. Empirical forecasting under TSCov model without bootstrap

The Kalman filter equations can deal with missing observations in a natural way. We used as first forecast strategy the so-called Increasing Horizon Prediction of the State [26]. By extending the sample data $y_1^{(\omega )},\dots ,y_n^{(\omega )}$ as missing values for ${\mathit{y}}_t^{(\omega )}$ with $t=n+1,n+2,\dots$, and applying the Kalman filter to the extended sample, the predictions are produced. Since ${\mathit{z}}_t$ does not contain information about ${\mathit{x}}_t$ beyond that contained in ${\mathbf{y}}_{1:t-1}$, $E({\mathit{x}}_t|{\mathit{z}}_t,{\mathbf{y}}_{1:t-1})=E({\mathit{x}}_t|{\mathbf{y}}_{1:t-1})={\hat{\mathit{x}}}_{t|t-1}$.

Let ${\widehat{yy}}_{t|t-1}^{(\omega )}\equiv E({\widehat{yy}}_t^{(\omega )}|x{x}_t,{\mathbf{y}}_{1:t-1})$ be the forecast of $y{y}_t$. From (5a), $E({\mathit{y}}_t|{\mathit{x}}_t,{\mathit{z}}_t)={\mathit{A}}_t{\mathit{x}}_t+\mathit{\Gamma }{\mathit{z}}_t$ and applying the law of iterated projections, one obtains

$${\hat{\mathit{y}}}_{t|t-1}^{(\omega )}={\mathit{A}}_{t}E({\mathit{x}}_t|{\mathbf{y}}_{1:t-1},{\mathit{z}}_t)+\mathit{\Gamma }z{z}_t={\mathit{A}}_t{\hat{\mathit{x}}}_{t|t-1}+\mathit{\Gamma }{\mathit{z}}_t$$

and its MSE is given as 11c.

Covariates forecast strategy. According to Hyndman et al. [25], if the covariates consist of indicator variables, their values are known to a certain future point in time. In addition, if such indicator variables reflect the effect of known future interventions that have also occurred in the past, then these values are also known. However, when they are unknown, predictions of future values of covariates are needed. In this work, we adopt the exponentially weighted moving average approach to predict covariates. The covariates are recursively smoothed by calculating the exponentially weighted moving average. In this way, the forecast of ${\mathit{z}}_t$ in time t + 1 is equal to a weighted average of the most recent observation ${\mathit{z}}_t$ and the previous forecast ${\hat{\mathit{z}}}_{t|t-1}$, that is

$${\hat{\mathit{z}}}_{t+1|t}=\rho {\mathit{z}}_t+(1-\rho ){\hat{\mathit{z}}}_{t|t-1}$$ (22)

where $0\le \rho \le 1$ is the smoothing parameter that is typically close to 1. This strategy is similar to that applied by Dordonnat et al. [13].

Due to Markov structure in the state dynamics and assumptions about conditional independence of observations, the predictive distributions can be recursively calculated. The Kalman filter update in time t provide ${\mathit{x}}_{t+1|t}$ and ${\mathit{P}}_{t+1|t}$, used to obtain the one-step-ahead forecast of ${\mathit{y}}_{t+1}^{(\omega )}$. The h-steps-ahead forecast is given by

$$\begin{array}{rl}{\hat{\mathit{y}}}_{t+h|t}^{(\omega )}& \equiv E({\mathit{y}}_{t+1}^{(\omega )}|{\mathbf{y}}_{1:t-1},{\mathit{z}}_t)={\mathit{A}}_{t+1}{\hat{\mathit{x}}}_{t+1|t}+\mathit{\Gamma }{\hat{\mathit{z}}}_{t+h|t}\end{array}$$ (23a)

$$\begin{array}{rl}MSE({\hat{\mathit{y}}}_{t+h|t}^{(\omega )})& ={\mathit{A}}_{t+h}{\mathit{P}}_{t+h|t}{\mathit{A}}_{t+h}^{\mathrm{\prime }}+{\mathit{R}}_{t+h}\end{array}$$ (23b)

where

$$\begin{array}{rl}{\hat{\mathit{x}}}_{t+h|t}& =\mathit{\Phi }{\hat{\mathit{x}}}_{t+h-1|t}\end{array}$$ (24a)

$$\begin{array}{rl}MSE({\hat{\mathit{x}}}_{t+h|t})& ={\mathit{P}}_{t+h|t}=\mathit{\Phi }{\mathit{P}}_{t+h-1|t}{\mathit{\Phi }}^{\mathrm{\prime }}+{\mathit{Q}}_{t+h}\end{array}$$ (24b)

The prediction intervals can be obtained directly. Since the h-steps-ahead forecast errors are Gaussian, we generate the prediction intervals (PI) of the nominal coverage rate of $95\mathrm{\%}$ as,

$$PI={\hat{\mathit{y}}}_{t+h}^{(\omega )}\pm 1.96\sqrt{MSE({\hat{\mathit{y}}}_{t+h|t}^{(\omega )})},$$ (25)

4.2. Bootstrap procedure

The combination of state-space models and bootstrap methods currently has received much attention from researchers [11,12,19,28,37,38]. The results of these works shows that the combination of bootstrap methods with state-space models is valuable for time series forecasting, once it can provide more accurate forecasts than individual methods. The basic approaches of existing bootstrap methods, one of these approaches is the residual bootstrap in state-space models. Theoretically, if the model is correctly fitted, the residuals of the model would be independent and identically distributed. So, it is possible to sample these residuals with replacement to obtain a replica of the original sample. Then fit the model to the original sample replica and repeat the process, see [10].

Our proposed bootstrap procedure, which we call henceforward Boot.TSCov, is inspired by Cordeiro and Neves [12] and Rodrigues and Ruiz [38]. The first goal of our bootstrap procedure is to improve the forecasting method described in section 4.1. The second goal is to provide contributions for short-term forecasting of time series with complex seasonal patterns using bootstrap method, once it is a paradigm that in the scope for forecasting of time series with complex seasonal patterns still presents a void in the scant existing literature.

4.2.1. Boot.TSCov general procedure

First, use the procedure presented in section 3.2 to estimate the initial model TSCov given in (4) to obtain the sequence of innovations, ${\mathit{ϵ}}_t$ (which must be uncorrelated), and the fitted values $\{{\hat{y}}_1,\dots ,{\hat{y}}_n\}$ of the initial estimated model. Second, the standardized innovations, (26a), with the assurance that these innovations have, at least, the same first two moments, are resampled with replacement $b$ times to obtain a bootstrap sample of standardized innovations, ${\mathit{\upsilon }}_t^{\ast s}$ (step 3.1). Then, according to Cordeiro and Neves [12] the bootstrap replica of original time series can be obtained using (26b).

$$\begin{array}{rl}{\mathit{\upsilon }}_t^s& ={\mathit{\Sigma }}_t^{-1/2}{\mathit{ϵ}}_t\end{array}$$ (26a)

$$\begin{array}{rl}{\mathit{y}}_t^{\ast }& ={\hat{\mathit{y}}}_t+{\mathit{\Sigma }}_t^{1/2}{\mathit{\upsilon }}_t^{\ast s}\end{array}$$ (26b)

Then, use bootstrap replica, ${\mathit{y}}_t^{\ast }$, estimate the bootstrap model to obtain bootstrap estimates, ${\hat{\mathbf{\Omega }}}^{\ast }$, a priori ( ${\widehat{\mathit{ϵ}}}_t^{\ast }$) and a posteriori ( ${\hat{\mathit{\varsigma }}}_t^{\ast }$) innovations, the state vector and other Kalman filter derivatives. The forecast up to h-steps-ahead is obtained using the Kalman filter recursions with bootstrap estimates. Next, we summarize the main steps of our proposed bootstrap procedure.

4.2.2. Procedure steps

1. Use the iterative procedure described in section 3.2 and estimate the model defined by (5) to obtain the sequence of innovations, ${\mathit{ϵ}}_t$;

2. Compute the standardized innovations using (26a);

3. For each replica B,

   1. Resample with replacement b times the standardized innovations $\{{\mathit{\upsilon }}_1^s,{\mathit{\upsilon }}_2^s,\dots ,{\mathit{\upsilon }}_n^s\}$ to obtain the bootstrap sample of standardized innovations $\{{\mathit{\upsilon }}_1^{\ast s},{\mathit{\upsilon }}_2^{\ast s},\dots ,{\mathit{\upsilon }}_n^{\ast s}\}$;
   2. Compute a bootstrap replicate, ${\mathit{y}}_t^{\ast }$, by Equation (26b) using ${\mathit{\upsilon }}_t^{\ast s}$. From the iterative procedure described in section 3.2, estimate the corresponding bootstrap parameters, ${\hat{\mathit{\Omega }}}^{\ast }$, and the Kalman filter derivatives, such as a priori ( ${\widehat{\mathit{ϵ}}}_t^{\ast }$) and a posteriori ( ${\hat{\mathit{\varsigma }}}_t^{\ast }$) innovations, the state vector at time t and others;
   3. Obtain conditional bootstrap h-step-ahead forecast, ${\hat{\mathit{y}}}_{t+h|t}^{\ast }$, from the following expressions:
      $$\begin{array}{rl}& {\hat{\mathit{x}}}_{t+h|t}^{\ast }={\hat{\mathit{\Phi }}}^{\ast }{\hat{\mathit{x}}}_{t+h-1|t}^{\ast }\\ & {\hat{\mathit{P}}}_{t+h|t}^{\ast }={\hat{\mathit{\Phi }}}^{\ast }{\hat{\mathit{P}}}_{t+h-1|t}^{\ast }{\hat{\mathit{\Phi }}}^{{\ast }^{\prime }}+{\hat{\mathit{Q}}}_{t+h|t}^{\ast }\\ & {\hat{\mathit{y}}}_{t+h|t}^{\ast }={\hat{\mathit{A}}}_{t+h}^{\ast }{\hat{\mathit{x}}}_{t+h|t}^{\ast }+{\hat{\mathit{\Gamma }}}^{\ast }{\mathit{z}}_{t+h}\\ & {\hat{\mathit{\Sigma }}}_{t+h|t}^{\ast }={\hat{\mathit{A}}}_{t+h}^{\ast }{\hat{\mathit{P}}}_{t+h|t}^{\ast }{\hat{\mathit{A}}}_{t+h}^{{\ast }^{\prime }}+{\hat{\mathit{R}}}_{t+h|t}^{\ast }\\ & {\hat{\mathit{Q}}}_{t+h|t}^{\ast }=\delta {\hat{\mathit{Q}}}_{t+h-1}^{\ast }+(1-\delta )({\hat{\mathit{K}}}_t^{\ast }{\widehat{\mathit{ϵ}}}_t^{\ast }{\widehat{\mathit{ϵ}}}_t^{{\ast }^{\prime }}{\hat{\mathit{K}}}_t^{{\ast }^{\prime }})\\ & {\hat{\mathit{R}}}_{t+h|t}^{\ast }=\delta {\hat{\mathit{R}}}_{t+h-1}^{\ast }+(1-\delta )({\hat{\mathit{\varsigma }}}_t^{\ast }{\hat{\mathit{\varsigma }}}_t^{{\ast }^{\prime }}+{\hat{\mathit{A}}}_{t+h}^{\ast }{\hat{\mathit{P}}}_{t+h-1|t}^{\ast }{\hat{\mathit{A}}}_{t+h}^{\ast })\end{array}$$

where, ${\hat{\mathit{\varsigma }}}_t^{\ast }={\mathit{y}}_t-{\hat{\mathit{A}}}_t^{\ast }{\hat{\mathit{x}}}_{t|t}^{\ast }-{\hat{\mathit{\Gamma }}}_t^{\ast }{\mathit{z}}_t$ and ${\widehat{\mathit{ϵ}}}_t^{\ast }={\mathit{y}}_t-{\hat{\mathit{A}}}_t^{\ast }{\hat{\mathit{x}}}_{t|t-1}^{\ast }-{\hat{\mathit{\Gamma }}}_t^{\ast }{\mathit{z}}_t$. The hat on the matrices means that they are matrices estimated by bootstrap and used for forecast. By taking the average ( ${\hat{\mathit{y}}}_H$) of the empirical distribution of ${\hat{\mathit{y}}}_{t+h|t}^{\ast }$, the prediction intervals are generated as (25).

5. Applications to real time series

The results obtained from the application of TSCov and TBATS to the two complex time series in Figure 1 are reported in this Section. Each one of time series is splitted into two parts: fitting set and validation set.

$$\underset{\mathrm{Fitting}\mathrm{set}}{\underbrace{y_1,y_2,\dots ,y_{n-h}}},\stackrel{\mathrm{Validation}\mathrm{set}}{\overbrace{y_{n-h+1},\dots ,y_n}}$$

The TSCov model is configured to run with or without covariates. If there is no need to include covariates in the model, we only need to set input = 0 and $\mathit{\Gamma }=0$ from the generic function which projects the Kalman filter. The initialization of parameters is not automatic, it depends on the features of each dataset used, except the transition parameter, φ. Details about the initialization for each estimated models are presented in subsections 5.1 and 5.2.

Figure 1.

Complex seasonality showing: (a) Hourly $N{O}_2$ concentrations levels (with multiple seasonal periods) measured from October 1 ${}^{st}$ and December 31 ${}^{st}$ in 2014 in Paredes/Portugal, including Temperature, Relative humidity and Wind-speed as covariates also observed at hourly intervals; (b) Weekly US finished motor gasoline products in thousands (with non-integer seasonal periods), from February 1991 to July 2005.

We reported some results of TBATS model to compare with the TSCov model and Boot. TSCov procedure, for example, using the same nominal coverage rate to generate the prediction intervals and compare the performance of each model. To assess forecasting performance, we used the mean absolute percentage forecast error (MAPE) and the root mean squared forecast error (RMSE).

5.1. Application to multiple seasonal patterns data

Figure 1(a) shows the hourly $N{O}_2$ concentrations levels, obtained from the online database on air quality [34]. The dataset under analysis concern 49 stations located over Portugal (mainland) from October 1 ${}^{st}$ and December 31 ${}^{st}$ in 2014. The selected period corresponds to the highest $N{O}_2$ levels along the year, according to Andreia et al. [5]. The time series of $N{O}_2$ denoted by $N_t$ has a daily pattern with period 24 and a weekly seasonal pattern with period 168. The covariates considered are Temperature, $T_t$, Relative humidity, $H_t$, and Wind-speed, $W_t$. We estimated two TSCov models: one model with real covariates and other with predicted covariates. The series, which consists of 2208 observations, was split into two segments: an estimation sample period (1488 observations) and a test sample (720 observations). The point forecasts are obtained only using the Increasing Horizon Prediction of the State strategy.

A preliminary analysis suggests the following considerations: fit a TBATS model to $N{O}_2$ to determine the cross-correlation function (CCF) between the $N{O}_2$ residuals, temperature series, humidity series and wind-speed series. The results (no output is shown) indicates that the strongest correlations occur at 12-h lag with air temperature ( $T_{t-12}$), 2-h lag with relative humidity ( $H_{t-2}$) and current Wind-speed ( $W_t$).

Initialization of TSCov model with real covariates. The initial mean was fixed at ${\mathit{x}}_0=0.7$ with uncertainty modeled by the diagonal covariance matrix ${\mathit{\Sigma }}_{0ii}=4$, for $i=1,\dots ,16$. Initial state covariance values were taken as $Q{Q}_0=\mathrm{diag}\{{\sigma }_{\xi }^2,{\sigma }_{\zeta }^2,{\sigma }_e^{2(i)}\}=\{0.004,0.17,0.0001,0.0001\}$ (i = 4). The measurement error covariance was started at $R{R}_0={\sigma }_{\epsilon }^2={10}^{-8}$. Initial regression coefficients values was fixed at ${\beta }_1^{\ast }={\beta }_2^{\ast }={\beta }_3^{\ast }=0.1$, and the forgetting factor, $\delta =0.999$.

Initialization of TSCov model with predicted covariates. The initial mean and its uncertainty (modeled by the diagonal covariance matrix) was fixed at $x{x}_0=0.7$ and ${\mathit{\Sigma }}_{0ii}=3.1$ ( $i=1,\dots ,17$), respectively. Initial state covariance values were taken as $Q{Q}_0=\mathrm{diag}\{{\sigma }_{\xi }^2,{\sigma }_{\zeta }^2,{\sigma }_e^{2(i)}\}=\{0.004,0.045,0.001,0.001\}$ (i = 4). The measurement error covariance was started at $R{R}_0={\sigma }_{\epsilon }^2={10}^{-6}$. Initial regression coefficients values was fixed at $\{{\beta }_1^{\ast },{\beta }_2^{\ast },{\beta }_3^{\ast }\}=0.01$, and the forgetting factor, $\delta =0.8$.

The TBATS models are implemented in the forecast r package [45]. Therefore, its initialization process is automatic.

Figure 2 shows the residual analysis. For TSCov model with real covariates, Figure 2(a), the lag 18 is outside the interval, but we view this value as acceptable, since we do not find any pattern that leads us to believe that we have missed a structural dynamic feature in the time series. The Ljung-Box test provides a Chi-square value equal to 29.154, with 19 degrees of freedom and p-value = .164, allowing not to reject the null hypothesis that the residuals are independent. The required number of significant harmonics for the trigonometric terms of daily seasonal pattern with periodicity 24 was $k_1=5$ and $k_2=3$ for weekly seasonal pattern with periodicity 168. For TBATS model, Figure 2(b), the correlation in lags 23 and 24 shows that the model does not capture all the dynamics well. However, the Ljung-Box test provides a Chi-square value equal to 19.443 with 18 degrees of freedom and p-value = .265, we also do not reject the null hypothesis.

Figure 2.

Autocorrelation plots of the one-step-ahead forecasting errors under the: (a) TSCov model (with real covariates) and (b) TBATS model for $N{O}_2$ levels data. Dashed: confidence intervals of level $95\mathrm{\%}$.

Table 1 reports the estimated parameters of TSCov model with real covariates and TBATS model. Almost all of the estimates are significant. The estimated $\hat{\varphi }$ from TBATS model does not suggest the damping effect in the trend component, unlike TSCov model that suggests a damping effect in the trend component. The measurement uncertainty was, in general, large at ${\sigma }_{\epsilon }^2=2.294$, compared with the model uncertainties of the level, trend and seasonal.

Table 1. Estimated parameters and their standard errors obtained from TSCov model (with real covariates): $N{O}_2$ levels in Paredes/Portugal.

	Estimated models for $N{O}_2$ concentrations
Parameter	MLE (TSCov)	St.Error	MLE(TBATS)
${\beta }_1^{\ast }$	$-0.724$	0.437	–
${\beta }_2^{\ast }$	$-0.007$	0.013	–
${\beta }_3^{\ast }$	0.257	0.035	–
ω	–	–	–
α	–	–	0.039
β	–	–	0.0002
φ	0.947	0.175	1
${\sigma }_{\epsilon }^2$	2.294	0.161	–
${\sigma }_{\xi }^2$	1.596	0.073	–
${\sigma }_{\zeta }^2$	0.091	0.036	–
${\sigma }_e^2$	$\{0.002;0.019\}$	$\{0.012;0.022\}$	–
${\sigma }_{e^{\ast }}^2$	$\{0.012;-0.048\}$	$\{0.164;0.011\}$	–
${\gamma }_1$	–	–	$\{-{10}^{-4};-{2.10}^{-6}\}$
${\gamma }_2$	–	–	$\{-{3.10}^{-5};{3.10}^{-5}\}$

Note: The table also shows the parameters obtained from TBATS model.

We may also consider forecasting the $N{O}_2$ series, and the result of a 24-steps-ahead forecast is shown in Figure 3(a). Using the same nominal coverage rate, $95\mathrm{\%}$, we generate the prediction intervals for both models, Figure 3(b). As it can be seen the prediction intervals based on the proposed TSCov model are narrower and more regular over the forecast horizon and cover all future values than those obtained with TBATS model.

Figure 3.

Forecasts up to 24-steps-ahead of $N{O}_2$ levels in Paredes/Portugal under the: (a) TSCov model (with real covariates) and TBATS model; (b) $95\mathrm{\%}$ prediction intervals obtained from TSCov and TBATS models.

Table 2 shows the forecasts accuracy computed using real and predicted covariates for TSCov model. It should be noted that, for TSCov model (with real or predicted covariates), the RMSE provided values that we consider small in almost all forecast horizons, unlike the TBATS model that provided higher values for higher forecast horizons. In addition, both graphically and in terms of accuracy measures, both are in agreement that the TSCov model got a good rating.

Table 2. Forecasting accuracy up to 24-steps-ahead of $N{O}_2$ levels in Paredes/Portugal.

	TSCova		TSCovb		TBATS
Horizon	RMSE	MAPE	RMSE	MAPE	RMSE	MAPE
1 – 3	4.127	3.643	4.717	5.217	2.179	6.819
1 – 6	4.067	3.646	4.727	5.387	2.111	6.699
1 – 9	4.206	3.787	4.833	5.287	3.866	9.326
1 – 12	4.246	3.607	5.303	6.513	5.973	11.213
1 – 15	4.293	3.827	5.081	6.571	6.962	12.404
1 – 18	4.361	3.827	5.356	6.581	6.969	15.243
1 – 21	4.388	3.848	5.369	7.748	9.840	15.657
1 – 24	4.495	3.911	5.903	7.804	11.448	17.574

^aForecast with real covariates.

^bForecast with predicted covariates.

5.2. Application to non-integer seasonal periods data

Figure 1(b) shows the number of barrels of motor gasoline product supplied in the United States, in thousands of barrels per day, from February 1991 to July 2005. This dataset was also used by De liver et al. [1] (see https://robjhyndman.com/publications/complex-seasonality/). The data are observed weekly and show a strong annual seasonal pattern, where the length of seasonality of the time series is $365.25/7\approx 52.179$. According to De Livera et al. [1], the time series exhibits an upward additive trend and an additive seasonal pattern, that is, a pattern for which the variation does not change with the level of the time series.

For this case, the TSCov model and Boot.TSCov procedure are applied without the inclusion of covariates. The time series consists of 745 observations and was split into two segments: an estimation sample period (520 observations) and a test sample (225 observations). The initial model for Boot.TSCov procedure is fitted using the same procedure described in section 3.2. Thus, the initial mean was fixed at ${\widehat{xx}}_0=0$ with uncertainty modeled by the diagonal covariance matrix ${\mathit{\Sigma }}_{0ii}=6.5$ ( $i=1,\dots ,18$), the forgetting factor was fixed at $\delta =0.999$. The measurement error covariance was started at $R{R}_0={\sigma }_{\epsilon }^2={10}^{-8}$. The initial state covariance values were taken as $Q{Q}_0=\mathrm{diag}\{{\sigma }_{\xi }^2,{\sigma }_{\zeta }^2,{\sigma }_{\mathit{w}}^2\}=\{0.0005,0.0005,0,0\}$.

The point forecasts are obtained using the Increasing Horizon Prediction of the State and Bootstrap strategies.

Figure 4 shows the residual analysis. We noted that, although the Box–Cox transformation, the empirical autocorrelation of estimated TBATS model shows a negative correlation for lag 1, Figure 4(b), which is unlikely to be due to random sampling variation. However, the Ljung-Box test provides a Chi-square value equal to 19,392, with 24 degrees of freedom and p-value = .731, allowing not reject the null hypothesis. For TSCov model, there is no significant correlations, Figure 4(a). They are all within the $95\mathrm{\%}$ confidence interval, which is satisfactory. In addition, the Ljung-Box test provides a Chi-square value equal to 11,485, with 24 degrees of freedom and p-value = .985; we also do not reject the null hypothesis.

Figure 4.

Autocorrelation plots of the one-step-ahead forecasting errors under the: (a) TSCov model without covariates for weekly U.S. Gasoline data; (b) TBATS model for weekly U.S. Gasoline data. Dashed: confidence intervals of level $95\mathrm{\%}$

Table 3 shows the estimated parameters and both models including Boot.TSCov procedure suggests a damping effect in the trend component. Figure 5(a) shows the 52-steps-ahead forecasts obtained from TSCov model using Increasing State Horizon strategy and Figure 6(a) shows 52-step-ahead forecasts obtained from Boot.TSCov procedure. The forecasts accuracy are shown in Table 4. As it can be seen, TSCov model competes with TBATS model, but Boot.TSCov procedure performs better for all lead times than TBATS model. The fitted values of the two models indicate that the forecasts generated by the TSCov model are closer to the validation series and TBATS model offers smoother fitted values and forecasts than those obtained from the TSCov model.

Table 3. Estimated parameters and their standard errors of the TSCov model for weekly U.S. Gasoline data.

	Estimated models for weekly U.S. Gasoline data
Parameter	MLE(TSCov)	St.Error	Boot.TSCov	St.Error	MLE(TBATS)
ω	–	–	–	–	0.709
α	–	–	–	–	−0.063
β	–	–	–	–	0.031
φ	0.829	0.154	0.801	0.457	0.834
${\sigma }_{\epsilon }^2$	220.69	1.751	222.25	1.916	–
${\sigma }_{\xi }^2$	1094.29	9.597	1143.19	9.947	–
${\sigma }_{\zeta }^2$	860.39	3.819	991.51	4.022	–
${\sigma }_{\mathit{w}}^2$	7.055	0.489	1.249	0.604	–
${\sigma }_{{\mathit{w}}^{\mathit{\ast }}}^2$	2.892	0.216	0.029	0.263	–
${\gamma }_1$	–	–	–	–	−0.003
${\gamma }_2$	–	–	–	–	0.002

Note: The table also shows the parameters obtained from TBATS model.

Table 4. Forecasting accuracy up to 52-steps-ahead of weekly U.S. Gasoline data.

	TSCova		Boot.TSCovb		TBATS
Horizon	RMSE	MAPE	RMSE	MAPE	RMSE	MAPE
1 – 7	204.524	2.834	213.342	2.451	269.031	2.446
1 – 14	253.443	3.116	219.713	2.643	269.543	2.691
1 – 21	273.329	3.203	226.368	2.669	272.326	2.687
1 – 28	270.323	3.250	255.145	2.725	275.282	2.711
1 – 35	291.233	3.314	263.405	2.776	278.056	2.745
1 – 42	297.547	3.425	267.096	2.780	281.356	2.760
1 – 49	323.651	3.671	293.612	2.860	296.847	2.936
1 – 52	358.524	3.993	311.634	2.973	359.863	3.906

^aForecast (without covariates) obtained using the Increasing Horizon Prediction of the State strategy.

^b Forecast (without covariates) obtained using Bootstrap strategy.

Figure 5.

Increasing State Horizon Strategy -- Forecast up to 52-steps-ahead of weekly U.S. Gasoline data under: (a) TSCov and TBATS models; (b) $95\mathrm{\%}$ prediction intervals obtained from TSCov and TBATS models.

Figure 6.

Bootstrap Strategy–Forecast up to 52-steps-ahead of weekly production of weekly U.S. Gasoline data under: the (a) Boot.TSCov and TBATS models; (b) $95\mathrm{\%}$ prediction intervals obtained from Boot.TSCov and TBATS models.

We may also consider the prediction intervals, see Figures 5(b) and 6(b). Using the same nominal coverage rate, $95\mathrm{\%}$, the prediction intervals based on the Boot.TSCov procedure got a good rating compared to those obtained with the TSCov and TBATS models. These results allow to conclude that the TSCov model and Boot.TSCov procedure are also pertinent in handling the non-integer seasonality.

6. Conclusion and future direction

The main contribution of this work is to explore the use of covariates in short-term forecasting of time series with complex seasonal patterns, as an extension of the TBATS model. Our proposed framework responds two interesting problems: (i) in the field of forecasting, the proposed framework deals with covariates that may be important for the short-term forecasting; (ii) from the point of view of formulation, the proposed framework is a new tool for structural models for modeling and forecasting of time series with complex seasonal patterns. The answer to these ”two” problems is a valuable contribution to limited existing literature on structural models for predicting time series with complex seasonal patterns. The formulated bootstrap procedure has as main objective to improve the short-term forecasting obtained from the usual procedure with Kalman filter recursions. The procedure was applied satisfactorily and the results show that the proposed bootstrap procedure provides good results for the used dataset.

The empirical study shows the potential of our proposals as promising methodologies for short-term forecasting of time series with complex seasonal patterns. The used covariates had a significant impact on the forecast and, as expected, the forecasts obtained were more accurate under the use of covariables. Our estimation procedure not only obtains point forecasts and prediction intervals, but also allows to obtain the standard errors of each estimated parameter. However, our proposed methodologies can be improved in several ways and we can see some lines of future work listed below:

• Automatic selection of covariates. Some study can be done in this line for the automatic selection of candidate covariates to the estimated final model;

• Estimator of the coefficient matrix, $\mathit{\Gamma }$. The state space model given in (5) involves covariates in the measurement equation. However, the Kakman filter constructed for this model does not provide the estimator of the coefficient matrix $\mathit{\Gamma }$. An updated estimate of this matrix can be obtained by applying the Expectation Maximization algorithm (EM) [6];

• Multivariate analysis. When significant dependencies between individual time series can not be ignored, multivariate time series need to be introduced. A projection of the proposed framework to deal with multivariate time series is necessary. Our study was done for the univariate case. But the proposed framework can ”easily” be reformulated to the multivariate case.

Finally, with this work, a question can be made here: which models are preferred to use, TSCov and TBATS? It should, however, be noted from the results that there is little distinction between the two models, therefore, the immediate response is, the TSCov approach which is preferable if there are covariates that are useful predictors since they can be added as regressors and improve the forecasts.

All computational results of this work were obtained with the R software environment [35].

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Funding Statement

This work was partially supported by the Center for Research and Development in Mathematics and Applications (CIDMA) through the Portuguese Foundation for Science and Technology (FCT – Fundação para a Ciência e a Tecnologia), references UIDB/04106/2020 and UIDP/04106/2020 and ENAGBE (National Institute of Managment of Scholarships) – Angola.

Disclosure statement

No potential conflict of interest was reported by the author(s).