Deep Reinforcement Learning Assisted Beam Tracking and Data Transmission for 5G V2X Networks

Abstract

Beam tracking is a core issue in 5G vehicle-to-everything (V2X) networks. Specifically, higher beamforming gain is required to compensate for the path loss at higher frequencies, e.g., 5G FR2, to realize high data rate vehicle-toinfrastructure (V2I) communications. However, shorter time slots at higher frequencies, high velocity of vehicles, and unpredictable localization errors make this problem more challenging. Under these circumstances, wider beams can lead to higher beam tracking accuracy. Bear in mind that wider beams mean lower beamforming gain, which cannot compensate for high path loss at high frequencies and would further influence the data rate of V2I communications. Thus, there exists a trade-off between tracking accuracy and data rate in V2I communications. Furthermore, this problem needs to be solved within an extremely short time slot according to the high transmission frequency. To solve this problem, we propose a reinforcement learning (RL) assisted, high-resolution codebook-based beam tracking method. By comparing several different RL frameworks, we found that the twin delayed deep deterministic policy gradient (TD3) framework can help the roadside infrastructure (RSI) determine a proper beam pattern within a short duration. Moreover, according to the Hurst exponent analysis, recurrent neural networks (RNNs) are selected to improve the performance of the RL framework. The simulation results show that the proposed method performs well in tracking accuracy, data rate, and temporal efficiency.

I. Introduction

Aa a core part of the fifth generation (5G) standard, V2X communications have achieved worldwide attention. With the research on technologies such as autonomous driving becoming more profound, industry and academia have also put forward higher requirements for the transmission capability of a vehicular network [1], [2], [3]. However, the transmission for vehicles requires both high reliability and low latency, together with a high data rate [4]. In order to solve this problem, a series of technologies are suggested to be used in the V2X network, such as massive multiple inputs and multiple outputs (MIMO), millimeter wave communications, hybrid precoding, and so on [5], [6], [7].

To achieve high data rate transmission in vehicular communication scenarios, transmission on a higher frequency is a potential solution [8]. The massive MIMO technology with large antenna arrays is necessary for roadside infrastructures (RSIs) to generate pencil-like beams to compensate for the high path loss on corresponding frequencies [9]. So far, there have been considerable studies focusing on this area [8], [10], [11], [12], [13], [14], [15]. The authors of [10] propose an improved beam sweeping scheme, which aims to guarantee the occurrence of beam alignment and optimizes the latency to achieve beam alignment. By using a deep learning (DL) estimator, the authors of [11] propose an angle of arrival (AoA) and angle of departure (AoD) estimation algorithm, called the two-step angle estimation (TSAE) algorithm, to handle the high mobility vehicular communications. By using a realistic power consumption model, the authors of [8] investigate the performance of the generalized hybrid beamforming array structure in a cellular infrastructure-to-everything (C-I2X) communication scenario using a single-path terahertz (THz) channel model. By analyzing the capacity and deriving closed-form upper bounds on the capacity of the massive MIMO vehicle-to-vehicle (V2V) non-orthogonal multiple access (NOMA) channels, a pair of power allocation optimization schemes are formulated to improve the bit error rate (BER) in [12]. A packet-level channel model in the form of a finite-state Markov chain (FSMC) is proposed in [13] for vehicle-to-infrastructure (V2I) elevation multiplexing uplinks. The proposed channel model provides an enabling tool for analyzing and simulating the V2I elevation multiplexing links in 5G networks. A three-dimensional (3D) geometry-based stochastic model for V2V massive MIMO channels is proposed in [14] to study the influence of vehicle density on channel statistics. The authors of [15] derive asymptotic spectral efficiency to explore the performance of large antenna arrays used in V2X networks. A bio-inspired algorithm was used in [16] to improve the latency and throughput performance of V2I communications.

Despite the above studies, fast and accurate beam tracking under high mobility conditions still faces many challenging issues. For instance, under the pencil-like beam patterns requirement, RSIs need to know the vehicle’s precise location information in order to achieve highly accurate beam tracking during the data transmission phase [17]. However, acquiring accurate location information cannot be easily achieved [18].

Furthermore, acquiring accurate location information is not the only problem. Spectral efficiency is also an important key performance metric of V2I communications [19]. To improve the spectral efficiency, the authors of [20] propose a radar-assisted algorithm to perform the beam alignment task in a V2I scenario. By applying the analytical results with the traffic and data information, a distributed multisource scheduling algorithm is proposed in [21] to enhance the data dissemination efficiency in V2I communications. There is a trade-off between tracking accuracy and channel capacity when a vehicle communicates with its associated RSI. The authors of [22] propose a V2I data offloading scheme with QoS provisioning. The performance evaluation shows that their method can offload more high-priority data compared to traditional V2I data offloading schemes. In [23], the authors propose a QoS-aware data offloading scheme for V2I data offloading with QoS provisioning (DOVEQ), demonstrating that DOVEQ outperforms other schemes by offloading more data with less offloading delay and running time. A method to improve the data packet reception rate of optical camera communication based V2I using a deep learning based region-of-interest (ROI) detector is introduced in [24]. In [25], the authors adopt regression models to estimate traffic states. They show that their approach improves the performance in terms of decoding error rate in time-varying traffic. A new data dissemination algorithm, named the offline algorithm for hybrid data dissemination (OFDD), which chooses feasible V2I disseminations based on V2V broadcasting, is proposed in [26]. Reference [27] investigates the minimum RSI height required in multi-lane highways to guarantee all-time 60GHz line of sight (LOS) connectivity for different lanes. Also, the numerical results in [27] analyze how both signal strength and overall system data rate decrease with RSI height.

While the above studies have made solid progress in this area, little attention has been paid to the impact of the beam width on the performance. For instance, an RSI can achieve very high tracking accuracy by using a wide beam, but this would be at the expense of compromising the channel capacity due to the low beamforming gain. Moreover, for a high fading channel, a wide beam with low beamforming gain may also cause a disconnection between a vehicle and an RSI due to poor SNR. Under these conditions, a suitable AI-driven method can potentially assist overcoming this problem. Currently, there is considerable research activity mainly focusing on AI applications in vehicular networks [28], [29], [30], [31], [32], [33], [34]. For example, a DL-based channel prediction method is considered to estimate channel responses for V2I communications [28]. The authors of [29] propose a channel estimation algorithm based on deep learning and show that their algorithm can enhance channel estimation accuracy, hence improving the bit error rate and robustness. Modeling the resource sharing of V2V and V2I communications as a multi-agent reinforcement learning (RL) problem is investigated by the authors in [30]. They offer a solution using a fingerprint-based deep Q-network method that is amenable to a distributed implementation. Furthermore, in [31] a modelfree Q-learning technique is proposed to predict the channel adaptively by selecting the best impulse response predictor at the current orthogonal frequency division multiplexing block. Using deep RL, the authors of [32] propose a Fast Reflection (LFR) algorithm, which autonomously learns from the observable traffic pattern to select desirable reflector angles. An AI-based model is also proposed in [33] to perform an efficient blind channel state information (CSI) prediction. To accelerate handovers, a method that is based on historical handover data and K-nearest neighbor (KNN) machine learning (ML) algorithms is investigated in [34] to predict handover decisions without involving time-consuming target selection and beam training processes. A novel integrated ML and coordinated beamforming solution is developed to overcome the coverage, reliability, and latency challenges of mmWave applications under high mobility conditions [35]. The authors of [36] propose an ML approach to achieve an efficient and fast analog beam selection for mmWave V2V communications. In [37] the authors study the temporal effects of dynamic blockage in vehicular networks and propose a deep RL framework to overcome dynamic blockage. To improve network capacity whilst suppressing the additional beam search overhead, a partitioned search method is designed with an ML framework in [38]. Despite significant research progress in this area, the trade-off between beamforming gain and tracking accuracy, still remains unresolved.

Therefore, to investigate this issue, we consider a V2I network for relatively high velocity vehicular communications in a highway or sub-urban scenario. The RSIs are assumed to be situated along a straight road where the mobility of VUs is restricted within two lanes. Downlink beam tracking and data transmissions of the V2I network are supported by RL, i.e., an RL agent determines the beam pattern of each RSI. This would require overcoming the following challenges: 1), Vehicles are moving with high velocity, hence their locations are fast changing. Under these conditions, the location information achieved by RSI via uplink feedback may not be accurate enough for downlink beam tracking and data transmission. This is mainly due to the movement of the vehicle during the time gap between uplink feedback and downlink beamforming. 2), Not highly accurate location information provided by vehicles to assist the beam tracking may cause unpredictable errors. 3), The trade-off issue between tracking accuracy and spectral efficiency in V2I communications. The contributions of this paper to address these shortcomings are summarized as follows:

1. An RL assisted beam tracking and data transmission method is proposed to improve the performance of V2I communications. We formulate the spectral efficiency and tracking accuracy optimization as an RL problem to improve the performance of V2I communications. The RL-assisted method basically requires defining the state, action, and reward of the RL framework. Thus, to solve the optimization problem, we map the parameters that influence the performance of V2I communications into state, action, and reward forms.

2. We firstly use a regular deep deterministic policy gradient (DDPG) framework to do the optimization. To improve the performance of the RL framework, we make a Hurst exponent analysis on the training data and found that there exists a strong temporal dependency. Then, we change the fully connected neural networks (NNs) used in the DDPG framework to hybrid NNs that contain fully connected NNs and recurrent neural networks (RNN).

3. To increase the tracking accuracy and spectral efficiency of V2I communications, we revise the DDPG framework with hybrid NNs to a twin delayed DDPG (TD3) framework. Moreover, through simulations, we find that the TD3 framework with hybrid NNs has the best performance as compared with the regular DDPG framework and DDPG framework with hybrid NNs.

The rest of the article is organized as follows: The system model is described in detail in Section II and Section III presents our optimization approach. Simulations of the proposed algorithms are carried out in Section IV, followed by the conclusion in Section V.

Notation:

Throughout this paper, $J$ is the imaginary unit, $\min (\cdot )/\max (\cdot )$ is the minimum/maximum operator, ${[\cdot ]}_{i,j}$ represents the corresponding element of a vector or matrix, $\text{Pr}(\cdot )$ is the probability operation, $\mathbb{E}(\cdot )$ is the expectation operator, $\lfloor \cdot \rfloor$ is the floor function, and $\det (\cdot )$ is the determinant operator, ${\varphi }_{\text{E}}$ is the empty set.

II. System Model

We consider a V2I network where RSIs are assumed to be situated along a straight road (e.g., under a highway or sub-urban scenario). The road consists of two lanes and the mobility of Vehicle Units (VUs) is restricted to these two lanes. In this paper, the downlink beam tracking and data transmissions of the V2I network are supported by RL technology., i.e., an RL agent determines the beam pattern of each RSI.

A. Network Architecture

At the beginning of each training episode, RSIs’ locations are configured to be governed by a uniform distribution within the range $\left[0,l_R\right]$, where $l_R$ is the length of the road. Notice that this configuration is only used in the training process of the AI-agent to enhance the diversity of the training data. This is due to the fact that since the locations of the RSIs are fixed among all training episodes, the trained AI-agent can only be applied to the same RSIs locations. If we use a random distribution to generate the locations of the RSIs, the trained AI-agent can be used in other cases, regardless of the RSIs locations. Without loss of generality, we assume the road consists of two lanes, ${\mathbb{L}}_{\text{up}}$ and ${\mathbb{L}}_{\text{down}}$. The movement of the vehicles is configured to be restricted to these two lanes. The width of a lane is 4 meters and the length of each lane is 500 meters. A set of vehicles is defined by $\left\{V{u}_k\mid k\in \left[1,N_{\text{V}}\right]\right\}$, where $N_{\text{V}}$ is the number of vehicles in the V2I network. Similarly, a set of the RSIs is defined by $\left\{R{s}_k\mid k\in \left[1,N_{\text{R}}\right]\right\}$, where $N_{\text{R}}$ is the number of RSIs in the network.

In this paper, the network is assumed to operate in a time-slotted manner. Based on the configuration in [1], [39], and [40], we choose OFDM for V2I downlink transmissions. Also, based on [1], [39], and [40], the V2I link frame has a duration of 10 ms, and a frame is divided into ten subframes, each with a duration of 1 ms. Depending on the transmission frequency, the number of slots per subframe and the subcarrier spacing (SCS) for the OFDM waveform can be flexible for NR V2X. To realize high data rate downlink transmission we choose FR2 (i.e., 24.25 GHz and 52.6 GHz) as the V2I downlink transmission frequency. The number of time slots per subframe is defined by $2^{{\mu }_{\text{s}}}$, and the corresponding duration of a time slot should be $2^{-{\mu }_{\text{s}}}$. Based on [40], when FR2 (from 24.25GHz to 52.6GHz) is used, the value of ${\mu }_{\text{s}}$ can be 1, 2, or 3 in the absence of any gap between each time slot. The velocity of $V{u}_n$ at time slot $t$ is defined as $v_n(t)$. Furthermore, the movement of a typical vehicle, $V{u}_n$, is governed by three moving patterns,

1. Regular pattern: If there is no other vehicle that exists within a given distance ($D_{\text{p}}$ in front of $V{u}_n$) then $V{u}_n$ will keep moving in the same lane.

2. Merging pattern: If there exists another vehicle $V{u}_o$ within a given distance, $D_{\text{p}}$, in front of $V{u}_n$ in the same lane and the velocity satisfies the condition $v_n(t)-v_o(t)\ge v_{\text{thr}}$ (where $v_{\text{thr}}$ is the pre-defined threshold), then $V{u}_n$ will check its surroundings to see whether there are vehicles on the other lane within a distance, $D_{\text{M}}$, in front of or behind $V{u}_n$. If not, then $V{u}_n$ will merge into the other lane.

3. Following pattern: If $V{u}_n$ fails to merge into the other lane (e.g., there are vehicles within a certain range in the other lane), to avoid an accident the $V{u}_n$ will keep moving in the same lane and reduce its velocity to $v_n^{\ast }(t)=v_n(t)-\text{Δ}{v}_n<v_o(t)$. Also, $V{u}_n$ will keep observing the surroundings. Once the surrounding environment satisfies the condition for the merging pattern, it will follow that pattern to avoid potential traffic congestion caused by low-velocity vehicles. Once it completes the merging pattern, it will change to a regular pattern.

While in a regular pattern, the velocity of the vehicle will follow,

$$v_n(t)={\tau }_n{v}_n(t-1)+\left(1-{\tau }_n\right){\overline{v}}_n+\sqrt{1-{\tau }_n^2}{e}_{\text{v}},$$ (1)

where ${\overline{v}}_n$ is the average velocity of $V{u}_n$, and $e_{\text{V}}$ is a uniformly distributed random variable, i.e., $e_{\text{v}}\sim N\left(0,{\sigma }_{\text{v}}\right)$. Also, to make the mobility model as realistic as possible, the value of the average velocity ${\overline{v}}_n$ is set to range from 20 m/s to 30 m/s. ${\tau }_n$ is a parameter that reflects the degree of randomness of the VU’s mobility, e. g., ${\tau }_n=1$ and ${\tau }_n=0$ indicate the lowest and highest degree of randomness, respectively. Notice that the duration of a time slot is very short, thus, we can ignore the changes in VUs’ velocity and acceleration during a time slot without loss of generality. The network architecture can be shown as Fig. 1.

Fig. 1.

Network architecture.

B. V2I Beam Tracking and Communication

The beam tracking and communication process is shown in Fig. 2. Recall that the network operates in a time-slotted manner. The duration of each time slot is defined by $t_{\text{Ts}}=2^{-{\mu }_{\text{s}}}$. Here we define the nearest RSI of $V{u}_n$ as $R{s}_m$, and $V{u}_n$ will be associated to $R{s}_m$ before it moves out of the coverage of $R{s}_m$. As shown in Fig. 2, the V2I communication process can be summarized into two phases;

1. Pre-access and beam training phase: In this phase, a typical RSI, $R{s}_m$ will use a sub-6GHz band to sweep all directions with a wide beam pattern to capture $V{u}_n$. Once it finishes beam sweeping, feedback will be sent from $V{u}_n$ to $R{s}_m$ through the uplink with sub-6GHz and omnidirectional antennas. Then, $R{s}_m$ will perform beam refining to narrow the beam for downlink transmission. To achieve this, the $R{s}_m$ will use the channel state information, which is provided by $V{u}_n$ through the uplink feedback. Once $V{u}_n$ moves out of the coverage of $R{s}_m$, or $R{s}_m$ fails to track $V{u}_n$ in the following beam tracking phase the communication between $V{u}_n$ and $R{s}_m$ will be turned to pre-access phase.

2. Beam tracking and data transmission phase: In this phase, $R{s}_m$ will keep adjusting the beam direction and width to maintain the link quality between $R{s}_m$ and $V{u}_n$. At each time slot, $R{s}_m$ will first determine a proper beam pattern based on the previous feedback information from $V{u}_n$ through uplink transmission. Then, if the link quality is high enough, e.g., the channel capacity is higher than a given threshold, $R{s}_m$ will maintain the beam tracking phase and keep transmitting data to $V{u}_n$. However, if the link quality drops (mostly because of an unpredicted large movement of $V{u}_n$), $R{s}_m$ will start a new pre-access phase to re-capture $V{u}_n$.

Fig. 2.

Communication process within a subframe.

Since there is no data transmitted during the pre-access phase, $R{s}_m$ needs to avoid this case to improve the long-term average channel capacity. However, the movement of $V{u}_n$ cannot be easily predicted since $R{s}_m$ only has the feedback information (including location, velocity, and so on) of the previous time slots. In most cases however, even such information is not accurate enough as the location information provided by GPS (e.g., inaccuracies in meters level). In practice there are some scenarios where GPS signals cannot be utilized (e.g., driving through tunnels). Under these conditions, the received signal strength indication (RSSI)-based positioning method can be used to obtain the location information of VUs through the pre-access phase [41]. Thus, similar to [42], in this paper we assume the location information provided by $V{u}_n$ always has a small normally distributed error $e_{\text{lo}}$, i.e., $e_{\text{lo}}\sim N\left(0,{\sigma }_{\text{lo}}\right)$ where the value of $e_{\text{lo}}$ ranges from several centimeters to decimeters. Moreover, since RSI uses the massive MIMO technology to enable high data rate downlink transmission, the beam width will be pencil-like to compensate for the high channel fading. In this case, even if the value of the error is as small as several centimeters, it still may influence the performance of the link and force the communication to repeat the pre-access phase. Furthermore, we model the failure of uplink transmission/decoding as a stationary stochastic process ${\text{Θ}}_{\text{F}}=\left\{{𝓝}_i=0.01\mid i=1,2,\cdots ,\infty \right\}$, where ${𝓝}_i$ is the probability that an uplink transmission/decoding failure occurs at $f_i^{\text{sub}}$. Under these conditions, no feedback information will be received by the RSI. Moreover, since blockage effects caused by other vehicles can be significant at mmWave frequencies [37], [43], [44], we model the blockage failure as a stationary stochastic process ${\text{Θ}}_{\text{B}}=\left\{{𝓜}_i=0.01\mid i=1,2,\cdots ,\infty \right\}$, where ${𝓜}_i$ is the probability that there is a blockage failure at $f_i^{\text{sub}}$. To solve these problems, we propose an AI-assisted beam tracking method as described next.

C. AI-Assisted V2I Beam Tracking and Data Transmission

1). codebook Based Beamforming:

As we mentioned in the last section, in the pre-access phase $R{s}_m$ and $V{u}_n$ will set up an initial link through beam training. However, since $V{u}_n$ is probably moving with a high velocity, the link is not stable. Thus, $R{s}_m$ needs to keep adjusting the beam pattern to maintain an acceptable link quality. In this paper, the link quality is measured by the channel capacity, which is contained in the feedback message that $V{u}_n$ sends to $R{s}_m$ through uplink. We assume that $R{s}_m$ is equipped with a transmit uniform planar array (UPA) of $M_x\times M_y$ antenna elements and $V{u}_n$ is equipped with a receiver UPA of $N_x\times N_y$ antenna elements. Since the duration of a time slot is very short, the channel condition during each time slot is supposed to be stable, i.e., the channel matrix between $R{s}_m$ and $V{u}_n$ will not change during each time slot. The channel between the transmit array and receiver array at time slot $t$ can be expressed by (2), shown at the bottom of the next page, [45], where $G_{\text{TR}}$ and $G_{\text{AR}}$ are the transmit and receive antenna gains; ${\mathit{\alpha }}_{\text{TR}}^{m,n}(\cdot )$ and ${\mathit{\alpha }}_{\text{AR}}^{m,n}(\cdot )$ represent the array steering vectors of $R{s}_m$ and $V{u}_n$, respectively; ${\alpha }_{\text{LoS}}^{m,n}\left(f_{\text{c}},d_t^{m,n}\right)$ and ${\alpha }_{\text{NL}}^{k,m,n}\left(f_{\text{c}},d_t^{k,m,n}\right)$ are the path losses of the LoS path and NLoS path, respectively; $f_{\text{c}}$ is the carrier frequency; $d_t^{m,n}$ is the length of the LoS path at time slot $t$; $d_t^{k,m,n}$ is the length of the $k\text{th}$ NLoS path at time slot $t$; $N_{\text{pa}}$ represents the number of NLoS paths; ${\theta }_{\text{DP}}^{m,n}(t)$ and ${\theta }_{\text{AR}}^{m,n}(t)$ are the azimuth AoD and AoA of the LoS path at time slot $t$, respectively; ${\theta }_{\text{AR}}^{k,m,n}(t)$ and ${\theta }_{\text{DP}}^{k,m,n}(t)$ are the azimuth AoD and AoA of the $k\text{th}$ NLoS path at time slot $t$, respectively. Similarly, ${\phi }_{\text{DP}}^{m,n}(t)/{\phi }_{\text{AR}}^{m,n}(t)$ is the elevation AoD/AoA of the LoS path at time slot $t$, and ${\phi }_{\text{AR}}^{k,m,n}(t)/{\phi }_{\text{DP}}^{k,m,n}(t)$ is the elevation AoD/AoA of the the $k\text{th}$ NLoS path at time slot $t$. More specifically, the number of multipath components, $N_{\text{pa}}$, is a uniform distributed variable within a range of [1] and [5]. For the azimuth AoD and AoA of the $k\text{th}$ NLoS path at time slot $t$, i.e., ${\theta }_{\text{DP}}^{k,m,n}(t)$ and ${\theta }_{\text{AR}}^{k,m,n}(t)$, we have

$$\{\begin{array}{l}{\theta }_{\text{DP}}^{k,m,n}(t)={\theta }_{\text{DP}}^{m,n}(t)+{\vartheta }_{\text{DP}}^{k,m,n}(t)\hfill \\ {\theta }_{\text{AR}}^{k,m,n}(t)={\theta }_{\text{AR}}^{m,n}(t)+{\vartheta }_{\text{AR}}^{k,m,n}(t)\hfill \end{array}$$ (3a)

where ${\vartheta }_{\text{DP}}^{k,m,n}(t)$ and ${\vartheta }_{\text{AR}}^{k,m,n}(t)$ follow two independent uniform distributions on $[-\pi ,\pi ]$ (i.e., −180°, 180°). Similarly, for the elevation AoD and AoA of the $k\text{th}$ NLoS path at time slot $t$, i.e., ${\phi }_{\text{DP}}^{k,m,n}(t)$ and ${\phi }_{\text{AR}}^{k,m,n}(t)$, we have

$$\{\begin{array}{l}{\phi }_{\text{DP}}^{k,m,n}(t)={\phi }_{\text{DP}}^{m,n}(t)+{\psi }_{\text{DP}}^{k,m,n}(t)\hfill \\ {\phi }_{\text{AR}}^{k,m,n}(t)={\phi }_{\text{AR}}^{m,n}(t)+{\psi }_{\text{AR}}^{k,m,n}(t)\hfill \end{array}$$ (3b)

where ${\psi }_{\text{DP}}^{k,m,n}(t)$ and ${\psi }_{\text{AR}}^{k,m,n}(t)$ follow two independent uniform distributions on $[-\pi /4,\pi /4]$ (i.e., −45°, 45°). For $M_x\times M_y$-elements UPA, the array steering vector can be expressed by

$$\begin{gathered} {\mathbf{a}}_{\text{UPA}}(\theta ,\phi )=[1,\cdots ,e^{J\pi \left(m_x\sin \phi \sin \theta +m_y\cos \theta \right)}, \\ {\cdots ,e^{J\pi \left(\left(M_x-1\right)\sin \phi \sin \theta +\left(M_y-1\right)\cos \theta \right)}]}^T, \end{gathered}$$ (3c)

where $m_x$ and $m_y$ are the antenna elements index with $0\le m_x\le M_x$ and $0\le m_y\le M_y$, respectively; $r_{\text{A}}={\lambda }_{\text{C}}/2$ is the antenna element spacing, $J$ is the imaginary unit, ${\lambda }_{\text{C}}$ is the wavelength; $\theta$ and $\phi$ are variables of the function ${\mathbf{a}}_{\text{UPA}}(\theta ,\phi )$. Since the duration of a time slot is short, here we assume that the beam pattern, i.e., precoding vector, is reselected in every subframe instead of every time slot. Denoting $N_{\text{TR}}=M_x\times M_y$, $N_{\text{RC}}=N_x\times N_y$, based on (3), the channel capacity of the link at time slot $t$ can be expressed by,

$$C_t^{m,n}={\log }_2\det \left({\mathbf{I}}_{N_{\text{RC}}}+\frac{P_{\text{TR}}}{{\sigma }_{\text{s}}^2}{\mathbf{H}}_t^{m,n}{\mathbf{F}}_i^{m,n}{\left({\mathbf{F}}_i^{m,n}\right)}^H{\left({\mathbf{H}}_t^{m,n}\right)}^H\right),$$ (4)

where ${\sigma }_{\text{s}}^2$ is the power of additive white Gaussian noise (AWGN) of the channel between the RSI and UE; ${\mathbf{I}}_{N_{\text{RC}}}$ is an $N_{\text{RC}}\times N_{\text{RC}}$ identity matrix; $\det (\cdot )$ is the determinant of the given matrix; ${\mathbf{F}}_i^{m,n}$ is the precoding vector for subframe $f_i^{\text{sub}}$ and $P_{\text{TR}}$ is the transmit power of the UPA.

A regular mechanism to drive $R{s}_m$ to change the beam pattern is based on defining a threshold for the SNR of the current link. If the SNR of the current link is beyond the threshold, then $R{s}_m$ will keep the current beam pattern. Otherwise, it will choose another beam pattern to improve the link quality. However, this method is not suitable for V2I communication because the quality of the link may dramatically change when the vehicle’s velocity is very high. To solve this problem, ${\mathbf{F}}_i^{m,n}$ is selected from a pre-defined codebook by an AI agent, which controls the actions of $R{s}_m$. The pre-defined codebook is shown as follows;

$$\begin{gathered} {\mathbf{H}}_t^{m,n}={\alpha }_{\text{LoS}}^{m,n}\left(f_{\text{c}},d_t^{m,n}\right)G_{\text{TR}}{G}_{\text{AR}}{\alpha }_{\text{AR}}^{m,n}\left({\theta }_{\text{AR}}^{m,n}(t),{\phi }_{\text{AR}}^{m,n}(t)\right){\alpha }_{\text{TR}}^{m,n}{\left({\theta }_{\text{DP}}^{m,n}(t),{\phi }_{\text{DP}}^{m,n}(t)\right)}^H \\ +\sum _{k=1}^{N_{\text{pa}}}{\alpha }_{\text{NL}}^{k,m,n}\left(f_{\text{c}},d_t^{k,m,n}\right)G_{\text{TR}}{G}_{\text{AR}}\cdot {\alpha }_{\text{AR}}^{k,m,n}\left({\theta }_{\text{AR}}^{k,m,n}(t),{\phi }_{\text{AR}}^{k,m,n}(t)\right){\alpha }_{\text{TR}}^{k,m,n}{\left({\theta }_{\text{DP}}^{k,m,n}(t),{\phi }_{\text{DP}}^{k,m,n}(t)\right)}^H. \end{gathered}$$ (2)

The construction of the codebook can be found in [46]. Since the movement of $V{u}_n$ is hard to predict, a codebook with the same beam width is not robust enough to handle all the situations that may occur during beam tracking. This is why we have used a multi-level codebook. For example, a codeword ${\mathbf{F}}_{\text{P}}^{\{a,b\}}$ in the codebook will be selected by the RL agent to do beam tracking and data transmission in each subframe $f_i^{\text{sub}}$, i.e., ${\mathbf{F}}_{\text{P}}^{\{a,b\}}\to {\mathbf{F}}_i^{m,n}$ in (4). The parameter $a$ denotes the level of the codeword and $b$ is the location of the codeword at level $a$. Since the vehicles’ dynamics on elevation is relatively small compared with the azimuth cases, we do not consider the variation of elevation AoD and AoA in this paper. Then, the corresponding beam width and beam direction of ${\mathbf{F}}_{\text{P}}^{\{a,b\}}$ can be expressed by

$$\{\begin{array}{l}{B}_{\text{W}}^{a,b}=\frac{\pi }{2^a}\hfill \\ B_{\text{D}}^{a,b}=\frac{\pi }{2}-\frac{b\pi }{2^a}.\hfill \end{array}$$ (5)

To improve the beamforming gain and compensate for excessive path losses at higher frequencies, a codeword in the lower level of the codebook is not considered for beam tracking and data transmission. Thus, in this paper, RSIs only use the codeword from levels 5 to 7 of the codebook to do beam tracking and data transmission. Based on (5), the corresponding beam width is from $\pi /32={5.625}^{\circ }$ to $\pi /128={1.40625}^{\circ }$ [47].

2). RL Framework:

The communication process between $R{s}_m$ and $V{u}_n$ can be summarized as follows:

1. At the beginning of each subframe $f_i^{\text{sub}}$ (i.e., the first time slot of $f_i^{\text{sub}}$), $V{u}_n$ will send a feedback message (including information of channel capacity, velocity, and location) through the uplink.

2. After $R{s}_m$ successfully receives and decodes the feedback message, the AI agent uses this information and previous experiences to determine a codeword (i.e., beam pattern) for $R{s}_m$ to track $V{u}_n$. If $R{s}_m$ cannot determine a codeword ${\mathbf{F}}_i^{m,n}$ in a given time slot due to the computation complexity, the next time slot in $f_i^{\text{sub}}$ will be used to select the codeword. If $R{s}_m$ fails to determine a beam pattern within a subframe, then its data rate will be 0. We define this type of failure as overtime failure. If the beam tracking at subframe $f_{i-1}^{\text{sub}}$ succeeded, then the codeword ${\mathbf{F}}_{i-1}^{m,n}$ for $f_{i-1}^{\text{sub}}$ will be used for tracking and data transmission until ${\mathbf{F}}_i^{m,n}$ is selected.

3. Subsequently, $R{s}_m$ will use ${\mathbf{F}}_i^{m,n}$ as the codeword for precoding to track $V{u}_n$. If the tracking process succeeds, then $R{s}_m$ will use ${\mathbf{F}}_i^{m,n}$ to do data transmission as well as for the following time slots until the next codeword ${\mathbf{F}}_{i+1}^{m,n}$ in the next subframe $f_{i+1}^{\text{sub}}$. If the tracking process fails, $V{u}_n$ will also send a feedback message to $R{s}_m$. Then the communication process will be turned to the pre-access phase, and no downlink data transmission will take place by $V{u}_n$ before ${\mathbf{F}}_{i+1}^{m,n}$ is selected. We assume that the pre-access phase can be done before the next subframe $f_{i+1}^{\text{sub}}$ starts. Once $f_{i+1}^{\text{sub}}$ begins, the communication process will go back to step 1).

The following figure shows the communication process within a subframe.

Since $R{s}_m$ needs to determine a beam pattern, i.e., codeword, for beam tracking and data transmission after it receives the feedback from $V{u}_n$ at each subframe $f_i^{\text{sub}}$ in a complicated environment, here we use a deep deterministic policy gradient (DDPG) as a basic RL framework, to assist the beam tracking and data transmission.

Details of the DDPG agent can be found in [48]. Here we define $Ex{p}_i=\left\{{\mathbf{s}}_i^t,{\mathbf{a}}_i^t,r_i^t,{\mathbf{s}}_{i+1}^{t+\text{Δ}t}\right\}$ to as the agent experience gained from the environment at subframe $f_i^{\text{sub}}$, time slot $t$, where ${\mathbf{s}}_i^t$ is the state of the environment at subframe $f_i^{\text{sub}}$, time slot $t$; ${\mathbf{a}}_i^t$ is the action taken by the agent at subframe $f_i^{\text{sub}}$, time slot $t$; $r_i^t$ is the current reward that the AI agent has achieved through the action ${\mathbf{a}}_i^t$, and ${\mathbf{s}}_{i+1}^{t+\text{Δ}t}$ is the state of the environment at subframe $f_{i+1}^{\text{sub}}$, time slot $t+\text{Δ}t$ (here $\text{Δ}t$ denotes the duration of a subframe), respectively. $N_{\text{ba}}$ is the capacity of the minibatch, and ${\pi }_{\theta }(\cdot )$ is the policy that the DDPG agent uses to determine an action, i.e., codeword. Notice that there exists a relationship between $t$ and $i$, i.e., $\lfloor t/4\rfloor =i$, where $\lfloor \cdot \rfloor$ is the floor function. We extend $v_n(t)$ to $v_n(i,t)$, because the AI agent is updated in each subframe. Moreover, if a failure of uplink transmission/decoding occurs, the feedback information will be an empty set, i.e., $Ex{p}_i={\varphi }_{\text{E}}$; once a blockage failure happens, the reward of $f_i^{\text{sub}}$ will be 0. Let us define ${\mathbf{s}}_i^t$ by

$${\mathbf{s}}_i^t={\left[v_n(i,t),x_n^{\text{Vu}}(i,t),y_n^{\text{Vu}}(i,t),x_m^{\text{Rs}}(i,t),y_m^{\text{Rs}}(i,t)\right]}^T.$$ (6)

where $v_n(i,t)$ is the velocity of $V{u}_n$ at subframe $f_i^{\text{sub}}$, time slot $t$; $v_n(i,t)$, $x_n^{\text{Vu}}(i,t)$, $y_n^{\text{Vu}}(i,t)$, $x_m^{\text{Rs}}(i,t)$, $y_m^{\text{Rs}}(i,t)$ are the coordinates of $V{u}_n$ and the associated RSI at subframe $f_i^{\text{sub}}$, time slot $t$, respectively. Normally, $x_n^{\text{Vu}}(i,t)$ and $y_n^{\text{Vu}}(i,t)$ are achieved by GPS or other localization algorithms, such as the RSSI-based method, which may have an estimation error. Thus, we add a normally distributed random variable $e_{\text{lo}}$ to the exact location to make the training data more realistic and the AI agent more robust. $r_i^t$ is the corresponding channel capacity when the selected codeword is used for data transmission at subframe $f_i^{\text{sub}}$, time slot $t$. ${\mathbf{a}}_i^t$ is the RSI selected codeword at subframe $f_i^{\text{sub}}$, time slot $t$. ${\mathbf{a}}_i^t$ can be expressed by,

$${\mathbf{a}}_i^t=\left[{\mathbf{F}}_{i,t}^{R{s}_1^{\text{V}},1},{\mathbf{F}}_{i,t}^{R{s}_2^{\text{V}},2},\cdots ,{\mathbf{F}}_{i,t}^{R{s}_n^{\text{V}},n},\cdots ,{\mathbf{F}}_{i,t}^{R{s}_{N_{\text{V}}}^{\text{V}},N_{\text{V}}}\right].$$ (7)

where $R{s}_n^{\text{V}}$ is the associated RSI of $V{u}_n$ during subframe $f_i^{\text{sub}}$. Notice that, since the network is operating at a high frequency (i.e.,24.25 GHz and 52.6 GHz), the duration of a timeslot is very short. Thus, the AI agent is configured to update in each subframe instead of each time slot. The flowchart of the beam tracking process is shown as follows.

III. Problem Formulation

In this paper, the system is operated in a time-slotted manner. Based on the discussion in Section-B, for a given subframe $f_i^{\text{sub}}$, time slot $t$, the communication status between a VU $V{u}_n$ and the associated RSI; $R{s}_m$, may either be in the pre-access phase or beam tracking phase, depending on the result of the beam tracking at the corresponding subframe. We define the beam width and direction that the $R{s}_m$ uses to track $V{u}_n$ at subframe $f_i^{\text{sub}}$, time slot $t$ as $B{w}_{i,t}^{m,n}$ and $B{d}_{i,t}^{m,n}$, respectively. Then, only in these two cases, the beam tracking phase will be turned to the pre-access phase at subframe $f_i^{\text{sub}}$, time slot $t$:

1. The SNR of the link between $R{s}_m$ and $V{u}_n$ at subframe $f_i^{\text{sub}}$, and time slot $t$, is lower than a given threshold ${\gamma }_{\text{th}}$, i.e.,

   $$\begin{gathered} {\rho }_{i,t}^{m,n}\le {\gamma }_{\text{th}}, \\ {\rho }_{i,t}^{m,n}=\det \left(\frac{P_{\text{TR}}}{{\sigma }_{\text{s}}^2}{\mathbf{H}}_{i,t}^{m,n}{\mathbf{F}}_{i,t}^{m,n}\times {\left({\mathbf{F}}_{i,t}^{m,n}\right)}^H{\left({\mathbf{H}}_{i,t}^{m,n}\right)}^H\right). \end{gathered}$$ (8)

   Where ${\mathbf{F}}_{i,t}^{m,n}$ is the codeword (i.e., beam pattern) used by $R{s}_m$ and $V{u}_n$ at subframe $f_i^{\text{sub}}$, time slot $t$ (i.e., $m=R{s}_n^{\text{V}}$); ${\rho }_{i,t}^{m,n}$ and ${\mathbf{H}}_{i,t}^{m,n}$ are the SNR and channel matrix of the link between $R{s}_m$ and $V{u}_n$ at subframe $f_i^{\text{sub}}$, time slot $t$, respectively.
2. The location of $V{u}_n$ at subframe $f_i^{\text{sub}}$, time slot $t$ is out of the coverage of ${\mathbf{F}}_{i,t}^{m,n}$, i.e.,

   $$\begin{gathered} {\kappa }_{i,t}^{m,n}>\frac{B{w}_{i,t}^{m,n}}{2}, \\ {\kappa }_{i,t}^{m,n}=\arctan \left(\frac{y_m^{\text{Rs}}(i,t)-y_n^{\text{Vu}}(i,t)}{x_m^{\text{Rs}}(i,t)-x_n^{\text{Vu}}(i,t)}\right)+\frac{\pi }{2}-B{d}_{i,t}^{m,n}, \end{gathered}$$ (9)

   where $y_m^{\text{Rs}}(i,t)$, $x_m^{\text{Rs}}(i,t)$, $y_n^{\text{Vu}}(i,t)$ and $x_n^{\text{Vu}}(i,t)$ are the coordinates of $R{s}_m$ and $V{u}_n$ at subframe $f_i^{\text{sub}}$, time slot $t$, respectively; ${\kappa }_{i,t}^{m,n}$ is the deviation angle from $V{u}_n$’s location to $B{d}_{i,t}^{m,n}$.

Suppose all the above conditions are satisfied, then the communication between $R{s}_m$ and $V{u}_n$ will be turned to pre-access phase, and no data, except the control signal will be sent in this phase. Otherwise, the communication between $R{s}_m$ and $V{u}_n$ will maintain in the beam tracking and data transmission phase. In this case, the AI agent ${\text{Θ}}_{\text{A}}$ will pick up a codeword, ${\mathbf{F}}_{\text{P}}^{\{a,b\}}$(i.e., ${\mathbf{F}}_{i,t}^{m,n}={\mathbf{F}}_{\text{P}}^{\{a,b\}}$), from a pre-defined codebook CB to track $V{u}_n$ and transmit data. If conditions (8) and (9) are satisfied, then the $V{u}_n$ is successfully tracked by $R{s}_m$. Consequently, the corresponding channel capacity can be expressed as,

$$C_{i,t}^{m,n}={\log }_2\det \left({\mathbf{I}}_{N_{\text{RC}}}+\frac{P_{\text{TR}}}{{\sigma }_{\text{S}}^2}{\mathbf{H}}_{i,t}^{m,n}{\mathbf{F}}_{i,t}^{m,n}{\left({\mathbf{F}}_{i,t}^{m,n}\right)}^H{\left({\mathbf{H}}_{i,t}^{m,n}\right)}^H\right).$$ (10)

We also assume that each VU can only be associated with the nearest RSI, So the average channel capacity at subframe $f_i^{\text{sub}}$, time slot $t$, can be expressed by,

$$\overline{C_{i,t}}=\frac{1}{N_{\text{V}}}\sum _{m=1}^{N_{\text{V}}}{C}_{i,t}^{m,n}\mathbf{1}\left({\rho }_{i,t}^{m,n}>{\gamma }_{\text{th}}\right)\mathbf{1}\left({\kappa }_{i,t}^{m,n}\le \frac{B{w}_{i,t}^{m,n}}{2}\right)).$$ (11)

Obviously, the value of $\overline{C_{i,t}}$ can be achieved by the uplink feedback of each VU at subframe $f_i^{\text{sub}}$, time slot $t$. The problem can be formulated by,

$$\begin{gathered} \text{maximize}{C}_{\text{AV}}=\frac{1}{t}\sum _{k=1}^t\overline{C_{i,t}} \\ \text{subject to}\overline{C_{i,t}}=\frac{1}{N_{\text{V}}}\sum _{m=1}^{N_{\text{V}}}{C}_{i,t}^{m,n}\mathbf{1}\left({\rho }_{i,t}^{m,n}>{\gamma }_{\text{th}}\right) \\ \times \mathbf{1}\left({\kappa }_{i,t}^{m,n}\le \frac{B{w}_{i,t}^{m,n}}{2}\right) \\ {C}_{i,t}^{m,n}={\log }_2\det ({\mathbf{I}}_{N_{\text{RC}}}+\frac{P_{\text{TR}}}{{\sigma }_{\text{s}}^2} \\ \times {\mathbf{H}}_{i,t}^{m,n}{\mathbf{F}}_{i,t}^{m,n}{\left({\mathbf{F}}_{i,t}^{m,n}\right)}^H{\left({\mathbf{H}}_{i,t}^{m,n}\right)}^H) \\ {\mathbf{F}}_{i,t}^{m,n}={\mathbf{F}}_{\text{P}}^{a,b}\in \mathbf{CB}. \end{gathered}$$ (12)

Now, we can extend the action defined in (7) to

$${\mathbf{a}}_i^t=\left\{a_{i,t}^1,a_{i,t}^2,\cdots ,a_{i,t}^{N_{\text{V}}},b_{i,t}^1,b_{i,t}^2,\cdots ,b_{i,t}^{N_{\text{V}}}\right\},$$ (14)

where $a_{i,t}^m$ and $b_{i,t}^m$ are equal to the $a$ and $b$ in ${\mathbf{F}}_{i,t}^{m,n}={\mathbf{F}}_{\text{P}}^{\{a,b\}}$, respectively.

The state of the environment at subframe $f_i^{\text{sub}}$ time slot $t$ can be expressed by (14), where $x_n^{\text{Vu}}(i,t)$, $y_n^{\text{Vu}}(i,t)$, $x_m^{\text{Rs}}(i,t)$, $y_m^{\text{Rs}}(i,t)$ are the coordinates of $V{u}_n$ and the associated RSI at subframe $f_i^{\text{sub}}$, and time slot $t$, respectively. Also, by defining $r_i^t=\overline{C_{i,t}}$, (12) can be solved as an RL problem.

The (target) actor-network and (target) critic-network used in the DDPG framework consist of three hidden layers; each having 300 neurons. The activation functions used in the NNs are Relu functions. The capacity of the replay memory is 10000, and the batch size is 256. The learning rate is set as 0.0001, the updating rate of the target networks is 0.005, the variance of the normally distributed exploration noise is 0.05, and the discounted factor is set at 0.99. The optimizer used in the DDPG framework is the Adam optimizer. The number of iterations in each episode is 2000. In this paper, we adopt an offline training phase. Thus, when applying the trained model to the actual network, the RL agent only needs to do forward propagation at each subframe. After 200 episodes’ training, the performance of the RL framework is shown in Fig. 6 where the transmission frequency is denoted by $F_{\text{T}}$. For the sake of comparison, we consider the well-known Extended Kalman Filter (EKF) based method [49]. More specifically, the EKF is used to track the VUs with a $\pi /64={2.8125}^{\circ }$ beam width. Once a VU has been tracked, the associated RSI will use a beam pattern with the same beam width for data transmissions.

Fig. 6.

Average spectral efficiency and tracking accuracy of DDPG framework.

As we can see, the corresponding tracking accuracy is only around 10% and the spectral efficiency is as low as 2bits/s/Hz, while the tracking accuracy and spectral efficiency of the EKF-based method are as high as 60% and 11bits/s/Hz. Obviously, the performance of the DDPG framework is not acceptable for V2I beam tracking and data transmissions. This may be caused by the following possibilities: 1) inability to accurately predict VUs’ locations; 2) the NNs used in the DDPG framework fails to capture the characteristics of VU’s mobility; 3) the task is beyond the capabilities of the DDPG framework.

$${\mathbf{s}}_i^t={\left[\begin{array}{c}{v}_1(i,t),x_1^{\text{Vu}}(i,t),y_1^{\text{Vu}}(i,t),x_1^{\text{Rs}}(i,t),y_1^{\text{Rs}}(i,t)\\ v_2(i,t),x_2^{\text{Vu}}(i,t),y_2^{\text{Vu}}(i,t),x_2^{\text{Rs}}(i,t),y_2^{\text{Rs}}(i,t)\\ ⋮\\ v_{N_{\text{V}}}(i,t),x_{N_{\text{V}}}^{\text{Vu}}(i,t),y_{N_{\text{V}}}^{\text{Vu}}(i,t),x_{N_{\text{V}}}^{\text{Rs}}(i,t),y_{N_{\text{V}}}^{\text{Rs}}(i,t)\end{array}\right]}^T,$$ (13)

A. Dependency Analysis

For such a challenging task, the performance of the regular DDPG framework is not acceptable. The problem may have been caused by the RL framework itself or the NN used in the RL framework. 1), The relationship between state ${\mathbf{s}}_i^t$, action ${\mathbf{a}}_i^t$, reward $r_i^t$, and next state ${\mathbf{s}}_{i+1}^{t+\text{Δ}t}$, is too complicated, which makes the DDPG framework with a simple structure incapable of handling it.; 2), the structure of the training data is too complex to handle for a regular fully connected NN. To improve the performance of the beam tracking accuracy and data transmission capacity, we analyze the structure of the training data. Since the action at each subframe is selected based on the current state ${\mathbf{s}}_i^t$ and previous experiences (also including previous states), we choose state data to make the analysis. If the data has a strong temporal dependence, then a recurrent neural network (RNN) may be better than a fully connected NN to solve the problem in (12).

The Hurst exponent is commonly used to analyze the dependency of a given data set. It is a measure of long-term memory of time series. Studies involving the Hurst exponent were originally developed in hydrology for practical matters of determining optimum dam sizing for the Nile river’s volatile rain and drought conditions that had been observed over a long period of time [50].

Based on [51] and [52], the Hurst exponent can be estimated using three typical methods: 1) the Periodogram Method; 2) the Variance-Time Analysis Method; 3) and the Rescaled Adjusted Range Statistic (R/S) Method. Here we choose the R/S method to evaluate the Hurst exponent of the training data. For a given series ${𝓧}_i$, we define the partial sum of ${𝓧}_i$ as,

$$𝓨(k)=\sum _{i=1}^k{𝓧}_i,$$ (15)

and sample variance is denoted by,

$${𝓢}^2(k)=\frac{\sum _{i=1}^k{𝓧}_i-\frac{𝓨{(k)}^2}{k}}{k},k\ge 1.$$ (16)

Furthermore, the R/S statistic is defined as,

$$\begin{gathered} \frac{𝓡(k)}{𝓢(k)}=\frac{\underset{0\le h\le k}{\max }\left(0,𝓨(h)-\frac{h}{k}𝓨(k)\right)}{𝓢(k)} \\ -\frac{\underset{0\le h\le k}{\min }\left(0,𝓨(h)-\frac{h}{k}𝓨(k)\right)}{𝓢(k)},k\ge 1. \end{gathered}$$ (17)

A log-log plot of the R/S statistic versus the number of points of the aggregated series should be a straight line with the slope being an estimate of the Hurst parameter. A value H in the range (0.5, 1.0) indicates that ${𝓧}_i$ has long-term positive autocorrelation, meaning that more high values are expected in the series. A value in the range (0, 0.5) indicates that ${𝓧}_i$ has a long-term switching behavior between high and low values in adjacent pairs. This indicates that a single high value is likely to be followed by a low value. Also, this tendency to switch between high and low values will continue over a long period of time.

By this method, we evaluate the Hurst exponent of position and velocity data of $V{u}_n$. The Hurst exponent of the velocity data and position data are 0.14 and 0.92, respectively. The Hurst exponent for velocity is close to 0 because when a vehicle’s velocity is very high, it’s more likely that it will reduce the velocity, and vice versa. The Hurst exponent for position is close to 1 because the vehicle keeps moving towards the same direction in our simulation model (also, in most of the application scenarios). This result indicates that the training data series has a longer-term dependency, which justifies a use of RNN to make the prediction. Nonetheless, the main drawback is that it takes a much longer time to train an RNN, such as LSTM, than a fully connected NN.

B. LSTM/GRU Network

Since the mobility of vehicles has a temporal dependency, we use an RNN, instead of a regular fully connected NN to enhance the performance of the AI agent. We choose LSTM and GRU to assist the RL framework. For instance, by adopting gate functions into the cell structure, LSTM can handle the problem of long-term dependencies [53].

Gated recurrent units (GRUs) are a gating mechanism in RNNs and were introduced in [54]. Compared with LSTM, GRU has fewer parameters as it lacks an output gate. Although GRU has a simpler structure, the performance of the GRU was found to be similar to that of LSTM [55]. Furthermore, GRUs have been shown to exhibit better performance on smaller datasets [55]. The structures of NNs used in the actor network, target actor network, critic network, and target critic network will be evaluated next,

The performance of the configured DDPG framework is shown in Fig. 8, where LSTM/GRU is used in the DDPG framework, which has an input size and output sizes of 128, and a step size of 32. We use a hyperbolic tangent (tanh) function as the activation function for the LSTM/GRU. Other hyperparameters are the same as a regular DDPG framework. As we can observe from Figs 6 and 8, the performance of the configured DDPG framework is higher than the regular DDPG framework. However, the improved performance is still not sufficient to handle the V2I beam tracking and data transmissions.

Fig. 8.

Average spectral efficiency and tracking accuracy of configured DDPG framework.

C. TD3 Framework With LSTM/GRU

As we can see, sometimes DDPG can achieve a great performance, but is frequently brittle with respect to hyperparameters and other types of tuning. A common issue for DDPG is that the learned Q-function begins to dramatically overestimate Q-values, which can lead to policy breaking because it exploits errors in the Q-function. A Twin Delayed DDPG (TD3) algorithm can solve this issue by using the following three important configurations:

1. double critics: TD3 learns two Q-functions (estimated by critic networks) instead of one and uses the smaller one of the corresponding two Q-values to evaluate the targets in the Bellman error loss functions.

2. delayed policy updates: TD3 updates the policy (and target networks) less frequently than the Q-function. Paper [56] recommends one policy update for every two Q-function updates.

3. target policy noise: TD3 adds noise to the target action (also called policy noise). Thus, it’s harder for the policy to exploit Q-function errors by smoothing out Q-values.

Together, these three tricks result in substantially improved performance over baseline DDPG, which only contains a single critic, updates the policy and the Q-function simultaneously, and has no policy noise [48]. Although the TD3 has a higher learning capability, it also has a more complex structure, which requires more training time. Thus, as mentioned before, we adopt an offline training phase. Notice that for the regular reinforcement learning framework, a common approach to construct a minibatch is to randomly pick up $N_{\text{ba}}$ experiences from the replay buffer. However, this process will not work when the RNN is used. This is mainly because the RNN requires the training data to be temporally related and randomly picked $N_{\text{ba}}$ experiences will not fit this requirement. Thus, we need to make a change with the regular construction process of the minibatch, such as; 1), randomly pick up an experience $Ex{p}_k$ from the replay buffer; 2), pick up the following $\frac{N_{\text{ba}}}{N_{\text{sba}}}-1$ experiences from the replay buffer to construct a sub-batch, i.e., $\left\{Ex{p}_k,Ex{p}_{k+1},\cdots ,Ex{p}_{k+\frac{N_{\text{ba}}}{N_{\text{sba}}}-1},Ex{p}_{k+\frac{N_{\text{ba}}}{N_{\text{sba}}}}\right\}$; 3), repeat steps 1) and 2) for $N_{\text{sba}}$ times, to construct a mismatch with $N_{\text{ba}}$ experiences. With this process, the TD3 framework can make full use of the GRU network and further improve the performance of the RL agent.

In this paper, the normally distributed noise with a mean value of 0 and variance of 0.1 is added to the target action. In addition, no dropout and batch normalization are used. This is because both are not suitable for the RL framework with RNN. Also, based on [55], the policy is updated for every two Q-function updates. Other hyperparameters are the same as the regular DDPG framework with GRUs.

IV. Simulation Results

We carried out our simulations according to the parameters defined in Table-I.

TABLE I.

Values Of Symbols Used in Simulation

Symbol	Definition/explanation	Value
$l_{\text{R}}$	Length of the road	500 m
$W_{\text{la}}$	Width of the lane	4 m
$N_{\text{V}}$	Number of VUs	10
$N_{\text{S}}$	Number of RSIs	10
${\mu }_{\text{s}}$	SCS exponent	2
$D_{\text{p}}$	Protecting distance of the regular pattern	30 m
$D_{\text{M}}$	Protecting distance of the merging pattern	30 m
$v_{\text{thr}}$	Threshold of velocity gap	5 m/s
${\tau }_n$	Degree of randomness	0.9
${\sigma }_{\text{v}}$	Variance of $e_{\text{v}}$	3 m/s
${\sigma }_{\text{lo}}$	Variance of $e_{\text{lo}}$	0.1 m
$P_{\text{TR}}$	Transmit power of $R{s}_m$	0.1 W
$N_{\text{TR}}$	Number of transmit antennas	64 × 16
$N_{\text{RC}}$	Number of receive antennas	4 × 4
${\gamma }_{\text{th}}$	SNR threshold	10 dB
$N_{\text{ba}}$	Capacity of minibatch	256
$N_{\text{sba}}$	Capacity of sub-minibatch	32

Based on [40], the maximum carrier bandwidth of FR2 when ${\mu }_{\text{s}}=2$ is 200 MHz. Thus, during the training process, the spectrum of the network is randomly chosen as 200 $N_{\text{V}}$ MHz bandwidth (i.e, 24.25 GHz, 52.6 GHz: 200 MHz bandwidth per VU). Also, in the simulation, we add a normally distributed noise with each coordinate to represent estimation errors on the location information. The average velocity of each VU ranges from 20 m/s to 30 m/s (72 km/h to 108 km/h, or 45 mile/h to 67.5 mile/h), and instant velocity, $v_n(i,t)$, changes over time and is governed by the mobility model. Furthermore, we choose ${\mu }_{\text{s}}=2$ so that a time slot will not be too short for the RL agent to complete beam selection.

The average spectral efficiency of configured/regular TD3 assisted V2I communications is shown in Fig. 9. Compared with Figs. 6 and 8, the spectral efficiency of data transmissions is significantly improved. Also, the performance of TD3 with GRU/LSTM is better than that of the regular TD3 framework with the same transmission frequency and training episode. The performance of all RL methods increases with the training episode. For the same RL framework, when the training episode is fixed, the spectral efficiency of the 52.6 GHz transmission frequency is lower than that of the 24.25 GHz transmission frequency. This is because the path loss increases at higher carrier frequencies (i.e., 52.6 GHz). Also, as we can see, the performance of TD3 with GRU is better than TD3 with LSTM. This is because LSTM is more complicated than GRU. Thus, LSTM requires more time for training compared with GRU. Notice that the performance of the EKF-based method is still better than our proposed method. This is based on the fact that the TD3 framework adds noise to both actions and target actions, so the performance under training will be much lower than the performance while testing. The corresponding testing performance is displayed in Fig. 11.

Fig. 9.

Average spectral efficiency of configured/regular TD3 framework.

Fig. 11.

Average spectral efficiency and tracking accuracy of TD3 with GRU (testing).

Similarly, the tracking accuracy is significantly improved by changing the RL framework from regular DDPG to TD3 with GRU/LSTM. Furthermore, the performance of TD3 with GRU is superior to that of the regular TD3 framework and the LSTM-assisted TD3 framework, with the same transmission frequency and training episode. An interesting observation is that the tracking accuracy of TD3 with GRU for different transmission frequencies, i.e., 24.25 GHz and 52.6 GHz, are almost the same. This is because the GRUs help the TD3 framework to better predict the mobility of VUs, which makes the RSI select a better codeword, i.e., beam width and direction, to improve the tracking accuracy while considering the higher path loss on higher frequencies. Also, the corresponding testing performance is depicted in Fig. 11.

The average spectral efficiency and tracking accuracy of TD3 with GRU/LSTM while testing is shown in Fig. 11. As we can see, the testing performances of TD3 with GRU/LSTM are much higher than those while training. As mentioned before, this is due to the characteristics of the TD3 framework. As shown in Fig. 11, the average tracking accuracy is high enough, i.e., over 90%, to handle the V2I communication even under a 52.6 GHz transmission frequency with an average velocity (30 m/s). Also, the average spectral efficiency of TD3 with GRU/LSTM is much higher than that of the regular DDPG framework. Furthermore, all the RL based methods outperform the EKF-based method in the testing phase, which is opposite in Fig. 9 and Fig. 10. This is because Figs. 9 and 10 show the training performance while Fig. 11 depicts the testing performance. RL-based methods, such as DDPG and TD3, are configured to add noise to action ${\mathbf{a}}_i^t$ during the training phase. Therefore, to encourage the RL agent to explore the action space sufficiently (i.e., the action ${\mathbf{a}}_i^t$ during training phase may not be the action that can achieve the best performance due to noise). When the fully trained RL agent is used in testing, it does not need to explore the action space any longer hence, the action ${\mathbf{a}}_i^t$ during the testing phase is the most suitable for the best performance. On the other hand, the EKF-based method does not use exploration noise to make decisions. Thus, the performance of the EKF-based method is better than RL-based methods during the training phase, but worse than the RL-based methods while testing. These observations indicate that the TD3 framework with GRU/LSTM can effectively capture the mobility of VUs and help RSIs to make a better codeword, i.e., beam pattern selection. Moreover, the performance of TD3 with GRU is better than TD3 with LSTM because of the higher requirement for training an LSTM network.

Fig. 10.

Average tracking accuracy of configured/regular TD3 framework.

The selecting probability of different beam widths is shown in Fig. 12. As we can observe, for the lower carrier frequency (24.25 GHz), the AI agent supported by TD3 with GRU is more likely to select a beam pattern with a wider beam width. When the transmission frequency is 52.6 GHz, the AI agent supported by TD3 with GRU starts to choose narrower beam width, e.g., $\pi /128={1.40625}^{\circ }$, to achieve a higher beamforming gain to compensate for the higher path loss. For the AI agent supported by TD3 with LSTM, the selection probability varies significantly with the transmission frequencies and beam widths. This is because the LSTM requires more training time compared to GRU, which is less effective in handling the tradeoff between spectral efficiency and tracking accuracy during the training phase. This observation indicates that the TD3 framework with GRU can successfully help the RSI to determine a more robust beam pattern while considering VU mobility and channel conditions.

Fig. 12.

Selecting probability of different beam width.

The tracking latency of the proposed TD3 with the GRU/LSTM method and baseline method is shown in Fig. 13. The baseline method uses a pruned codebook to support the fast beam tracking in V2I communications [57]. Based on [57], this approach can reduce tracking latency by two orders of magnitude. As we can observe from Fig. 13, our method outperforms the baseline method, even when the size of the codebook is as small as 256. Furthermore, the tracking latency increases dramatically with the scale of the codebook when the baseline method is used but remains stable with our method. This is because our method uses the RL framework to determine the codeword directly without searching the codebook. Therefore, our method can use a larger codebook to assist beamforming to obtain higher beamforming gain and beam tracking accuracy. Moreover, the tracking latency for TD3 with LSTM is a little higher than that of TD3 with GRU. This is because LSTM has a more complicated structure than GRU.

Fig. 13.

Tracking latency of proposed method and baseline method.

Fig. 14 shows the average spectral efficiency and tracking accuracy of the proposed method when tested with the Intelligent Driving Model (IDM) [58]. IDM is a widely used model of vehicle group movement that can simulate well the following behavior of moving VUs. Here we choose TD3 with GRU as an example. As shown in Fig. 14, the test performance of our method is still good despite changing the mobility model to IDM. Compared with Fig. 11, the average spectral efficiency and tracking accuracy are both higher. This is because the mobility model in this paper considers more frequent changes in velocity and direction of moving VUs. Therefore, when using IDM to model VUs’ mobility, the RL agent can achieve a better prediction of the trajectories of moving VUs.

Fig. 14.

Average spectral efficiency and tracking accuracy of proposed method with IDM (testing).

The standard deviation of spectral efficiency and tracking accuracy of TD3 with GRU while training is shown in Fig. 15. For TD3 with GRU, the standard deviation of spectral efficiency ranges from [0.41,0.97] with $F_{\text{T}}=24.25\text{GHz}$, and [0.36,0.78] with $F_{\text{T}}=52.6\text{GHz}$, respectively. The standard deviation of tracking accuracy ranges from [0.023, 0.048] with $F_{\text{T}}=24.25\text{GHz}$, and [0.024,0.047] with $F_{\text{T}}=52.6\text{GHz}$, respectively. Also, as we can see, the standard deviation of spectral efficiency/tracking accuracy increases with training episodes. This is because the average spectral efficiency/tracking accuracy increases gradually with the number of training episodes.

Fig. 15.

Standard deviation of spectral efficiency and tracking accuracy of proposed method (training).

V. Conclusion

This paper mainly focuses on downlink beam tracking and data transmission for V2I communications. Due to the short duration of each time slot, high velocity of vehicles, and the estimation errors of vehicles’ locations, vehicles’ mobility is difficult to predict. Even more, the higher transmission frequency results in more issues, such as higher path loss and following trade-off between spectral efficiency and tracking accuracy. Thus, an RL-assisted method is proposed to address these issues while considering all the aspects of vehicles’ mobility and transmission frequency together. Furthermore, we make an analysis of the mobility of vehicles and find a high temporal dependency. Based on the analysis, we modify the regular DDPG framework to the TD3 framework with LSTM/GRU. The simulation results verify that the TD3 framework with GRU outperforms the regular DDPG framework, the DDPG with GRU, regular TD3 framework, and TD3 framework with LSTM. Furthermore, the proposed TD3 framework with GRU achieves high spectral efficiency whilst maintaining high tracking accuracy.

Fig. 3.

Codebook used for precoding.

Fig. 4.

DDPG-assisted beam tracking process.

Fig. 5.

Beam tracking status.

Fig. 7.

Structure of NNs used in the AI agent.

Biographies

Junliang Ye (Member, IEEE) received the B.Sc. degree in communication engineering from the China University of Geosciences, Wuhan, China, in 2011, and the Ph.D. degree from the Huazhong University of Science and Technology, Wuhan, in 2018. He is currently a Guest Researcher with the National Institute of Standards and Technology (NIST), U.S. Department of Commerce, Gaithersburg, MD, USA. His research interests include heterogeneous networks, stochastic geometry, mobility-based access models of cellular networks, millimeter communication. Wave communications, and next-generation wireless communication.

Hamid Gharavi (Life Fellow, IEEE) received the Ph.D. degree from Loughborough University, U.K., in 1980. He joined the Visual Communication Research Department, AT&T Bell Laboratories, Holmdel, NJ, USA, in 1982. He was then transferred to Bell Communications Research (Bellcore) after the AT&T-Bell divestiture, where he became a Consultant on video technology and a Distinguished Member of Research Staff. In 1993, he joined Loughborough University, as a Professor and the Chair of communication engineering. Since September 1998, he has been with the National Institute of Standards and Technology (NIST), U.S. Department of Commerce, Gaithersburg, MD, USA. His research interests include smart grids, wireless multimedia, mobile communications, wireless systems, mobile ad hoc networks, and visual communications. He was a Core Member of the Study Group XV (Specialist Group on Coding for Visual Telephony), International Communications Standardization Body CCITT (ITU-T) and a member of the IEEE 2030 standard working group. He received the Charles Babbage Premium Award from the Institute of Electronics and Radio Engineering in 1986 and the IEEE Circuits and Systems Society Darlington Best Paper Award in 1989. He was a recipient of the Washington Academy of Science Distinguished Career in Science Award in 2017. He was a TPC Co-Chair of IEEE SmartGridComm in 2010 and 2012. He served as a member for the Editorial Board of proceedings of the IEEE from January 2003 to December 2008. He has been a Guest Editor of a number of special issues of the proceedings of IEEE, including Smart Grid, Sensor Networks and Applications, Wireless Multimedia Communications, Advanced Automobile Technologies, and Grid Resilience. He was the Editorin-Chief of IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY and IEEE WIRELESS COMMUNICATIONS. He served as a Distinguished Lecturer of the IEEE Communication Society.