DOA Estimation of Coherent Signals on Coprime Arrays Exploiting Fourth-Order Cumulants

Abstract

This paper considers the problem of direction-of-arrival (DOA) estimation of coherent signals on passive coprime arrays, where we resort to the fourth-order cumulants of the received signal to explore more information. A fourth-order cumulant matrix (FCM) is introduced for the coprime array. The special structure of the FCM is combined with the array configuration to resolve the coherent signals. Since each sparse array of a coprime array is uniform, a series of overlapping identical subarrays can be extracted. Using this property, we propose a generalized spatial smoothing scheme applied to the FCM. From the smoothed FCM, the DOAs of both the coherent and independent signals can be successfully estimated on the pseudo-spectrum generated by the fourth-order MUSIC algorithm. To overcome the problem of occasional false peaks appearing on the pseudo-spectrum, we use a supplementary sparse array whose inter-sensor spacing is coprime to that of either existing sparse array. From the combined spectrum aided by the supplementary sensors, the false peaks are removed while the true peaks remain. The effectiveness of the proposed methods is demonstrated by simulation examples.

1. Introduction

A coprime array consists of two uniform sparse arrays, from which a virtual uniform linear array (ULA) can be constructed from the spatial differences between any two sensors [1,2]. The spatial autocorrelations at all lags are estimated on the virtual ULA. The increased degrees of freedom has been used to identify $\mathcal{O}\left(MN\right)$ sources from only $\mathcal{O}(M+N)$ physical sensors [3,4]. Due to the simplicity of the array configuration, and the ability to resolve many more signals than the number of sensors, coprime arrays have attracted considerable interest in the DOA estimation applications [5,6,7]. In real scenarios, due to multi-path propagation or smart jammers, signals from different DOAs may become partially correlated, or coherent (fully correlated) in the extreme case [8]. The correlated/coherent signals pose a great challenge to the DOA estimation on coprime arrays. Since the spatial autocorrelations are estimated from the sample mean of the sensor-to-sensor signal multiplications, the presence of coherent signals indicates that the spatial autocorrelations contain cross-terms, which strongly affects the structure of the signal subspace. Incorrect extraction of the signal subspace brings about a failed DOA estimation.

The spatial smoothing preprocessing scheme was developed for a physical ULA to resolve coherent signals [9]. On coprime arrays, such scheme was employed to construct a correlation matrix for the virtual ULA [2]. However, the scheme cannot eliminate the cross-terms and hence the coherent signal problem is not solved. Recently, BouDaher et.al. proposed an algorithm to locate coherent targets using an active sensing approach on the coprime multiple-input multiple-output radar [10]. However, their method cannot be used for the DOA estimation on passive coprime arrays.

The fourth-order (FO) array processing methods were developed for the DOA estimation of non-Gaussian signals [11,12]. The main interests in using the FO cumulants relies on the increased degrees-of-freedom provided by the virtual coarray, and the higher resolution brought by the larger effective aperture [13,14,15]. Currently, the FO processing methods are used in coprime arrays [16] or nested arrays [17] to increase the virtual aperture. However, as the authors stated, their algorithms cannot handle coherent signals.

In our work, the scenario where the independent and coherent signals coexist is considered. We first formulate an FO cumulant matrix (FCM) with a special form, from which the DOA estimation can be carried out by the fourth-order MUSIC (4-MUSIC) algorithm [11]. Unfortunately, the FCM cannot be used for DOA estimation of the coherent signals directly. The particular form of the FCM is combined with the array configuration to resolve coherent signals. Since each sparse array is uniform, a series of overlapping identical subarrays can be extracted. Taking one such subarray from each of the sparse arrays, we can build a coprime subarray. An FCM is inherently defined on such coprime subarray, whose size is determined by the subarray sensor numbers. On two similar coprime subarrays, the FCMs share the same structure. Analogous to the spatial smoothing scheme applied to the correlation matrix of a ULA, we propose a generalized spatial smoothing scheme applied to the FCM. When the smoothed FCM is adopted by the 4-MUSIC algorithm, both the independent and coherent signals can be successfully estimated.

Occasionally, the pseudo-spectrum generated from the smoothed FCM encounters a false-peak problem. Some false peaks may appear at the directions where none of the true signals reside, interfering with the extraction of the true signals. We analyzed the causation of this phenomenon. To overcome this challenge, a supplementary sparse array can be added, whose inter-sensor spacing is respectively coprime to each of the existing sparse arrays. On the combined pseudo-spectrum aided by the supplementary sensors, the false peaks are removed.

This paper is organized as follows. In Section 2, we briefly review the coprime array configuration and then formulate the signal model. In Section 3, the FO cumulants as well as the form of the FCM are detailed, and the impact of coherent signals on the FCM is investigated. In Section 4, a generalized spatial smoothing scheme on the FCM is proposed to resolve coherent signals. Section 5 provides a method to remove the false peaks on the pseudo-spectrum. The effectiveness of the new approach is demonstrated in Section 6. Section 7 concludes the paper.

Notations: We use lower-case (upper-case) bold characters to denote vectors (matrices). $E\{·\}$ represents the statistical expectation. ${\left(·\right)}^T$ and ${(·)}^H$, respectively, denote the transposition and conjugate transposition of a vector or a matrix. ${\left(·\right)}^{*}$ is the element-wise complex conjugate. ⊗ denotes the Kronecker product. $\mathrm{rank}(·)$ denotes the rank of a matrix. $\parallel \mathit{x}\parallel$ is the 2-norm of the vector $\mathit{x}$. We use $\mathrm{diag}\left(\mathit{x}\right)$ to denote a diagonal matrix that uses the elements of $\mathit{x}$ as its diagonal elements.

2. Signal Model

As illustrated in Figure 1, a typical coprime array consists of two sparse uniform arrays, denoted by Array $\mathbb{A}$ and Array $\mathbb{B}$, respectively. Let M and N be coprime integers, the sensors of the two sparse arrays are located at (with common sensors for both sparse arrays):

$$\begin{array}{cccc}\hfill {\mathit{x}}_A& =& \left\{M{i}_{A}d|i_A=0,1,\dots ,L_A-1\right\}\hfill & ,\end{array}$$ (1)

$$\begin{array}{cccc}\hfill {\mathit{x}}_B& =& \left\{N{i}_{B}d|i_B=0,1,\dots ,L_B-1\right\}\hfill & .\end{array}$$ (2)

Figure 1.

The coprime array configuration.

In (1), $i_A$ and $i_B$ are the indices of the sensors, the unit inter-sensor spacing $d=\lambda /2$ with $\lambda$ as the half-wavelength, and $L_A$ and $L_B$ are the number of sensors of Array $\mathbb{A}$ and $\mathbb{B}$, respectively. Typically, $L_A\ge N$ and $L_B\ge M$. Denote this coprime array by Coarray $\mathbb{AB}$.

Suppose a narrowband signal from the DOA $\overline{\theta }$ impinges on the coprime array. The normalized DOA of the signal $\theta$ is defined as $\theta =2\pi dsin\overline{\theta }/\lambda$, which indicates the phase difference of the planar wave at the unit spacing d. The steering vectors of for the individual sparse arrays are

$$\begin{array}{ccc}\hfill \mathit{a}\left(\theta \right)& =& {\left[1,e^{jM\theta },\dots ,e^{jM\left(L_A-1\right)\theta }\right]}^T,\hfill \end{array}$$ (3)

$$\begin{array}{ccc}\hfill \mathit{b}\left(\theta \right)& =& {\left[1,e^{jN\theta },\dots ,e^{jN\left(L_B-1\right)\theta }\right]}^T.\hfill \end{array}$$ (4)

Let Q narrowband signals impinge on the array from the distinct DOAs ${\left\{{\theta }_q\right\}}_{q=1}^Q$, and the complex amplitude of the qth signal at snapshot time t is $s_q\left(t\right)$. The noise-corrupted measurement vectors on the two sparse arrays are

$$\begin{array}{ccccc}\hfill {\mathit{y}}_A\left(t\right)& =& \sum _{q=1}^Q\mathit{a}\left({\theta }_q\right)s_q\left(t\right)+{\mathit{w}}_A\left(t\right)\hfill & =& \mathit{A}\mathit{s}\left(t\right)+{\mathit{w}}_A\left(t\right),\end{array}$$ (5)

$$\begin{array}{ccccc}\hfill {\mathit{y}}_B\left(t\right)& =& \sum _{q=1}^Q\mathit{b}\left({\theta }_q\right)s_q\left(t\right)+{\mathit{w}}_B\left(t\right)\hfill & =& \mathit{B}\mathit{s}\left(t\right)+{\mathit{w}}_B\left(t\right).\end{array}$$ (6)

In (5), ${\mathit{w}}_A$ and ${\mathit{w}}_B$ are additive noise, $\mathit{s}\left(t\right)={\left[s_1\left(t\right),\dots ,s_Q\left(t\right)\right]}^T$ is the vector of complex amplitudes, and the matrices $\mathit{A}$ and $\mathit{B}$ are the collections of steering vectors of Array $\mathbb{A}$ and $\mathbb{B}$, respectively

$$\begin{array}{ccc}\hfill \mathit{A}& =& \left[\mathit{a}\left({\theta }_1\right),\dots ,\mathit{a}\left({\theta }_Q\right)\right],\hfill \end{array}$$ (7)

$$\begin{array}{ccc}\hfill \mathit{B}& =& \left[\mathit{b}\left({\theta }_1\right),\dots ,\mathit{b}\left({\theta }_Q\right)\right].\hfill \end{array}$$ (8)

The assumptions on the signals and noises are listed below.

1. The noise vectors ${\mathit{w}}_A$ and ${\mathit{w}}_B$ in (5) are zero-mean complex white Gaussian, with arbitrary correlation matrices. The noises are statistically independent of $\mathit{s}\left(t\right)$.

2. The complex amplitude of each signal follows a circularly-symmetric complex non-Gaussian distribution. A variety of modulations like quadrature amplitude modulation (QAM) or phase-shift keying (PSK) meet this assumption [11]. Under this assumption, the FO cumulants of the signal are non-zero [14].

3. Both statistically independent and coherent signals exist. We divide the signals into G groups. The signals in the same group are coherent, and the signals belonging to different groups are statistically independent.

Suppose that there are $Q_g$ coherent signals in the gth group ($Q_g=1$ for the independent signal case and ${\sum }_{g=1}^G{Q}_g=Q$) with the DOAs ${\mathit{\theta }}_g={[{\theta }_{g_1},\dots ,{\theta }_{g_{Q_g}}]}^T$. Since the complex amplitudes of coherent signals are linearly dependent [9], we can write the group signal vector by

$$\begin{array}{c}\hfill {\mathit{s}}_g\left(t\right)={\mathit{\eta }}_g{\sigma }_g\left(t\right).\end{array}$$ (9)

In (9), ${\mathit{\eta }}_g={[{\eta }_{g_1},\dots ,{\eta }_{g_{Q_g}}]}^T$ represents the complex coefficients along the respective propagation paths and hence the elements are non-zero. ${\sigma }_g\left(t\right)$ is a scalar representing the complex amplitude of the source of the gth group at snapshot time t.

Because Array $\mathbb{A}$ and $\mathbb{B}$ are both sparse and uniform, direction ambiguity exists on the sparse arrays. If a collection of F signals with the DOAs ${\left\{{\theta }_f\right\}}_{f=1}^F$ satisfies ${\theta }_f={\theta }_1+2\pi m_f/M$ for the distinct non-zero integers ${\left\{m_f\right\}}_{f=2}^F$, these DOAs are ambiguous on Array $\mathbb{A}$ because their steering vectors are identical

$$\begin{array}{c}\hfill \mathit{a}\left({\theta }_1\right)=\mathit{a}\left({\theta }_2\right)=\dots =\mathit{a}\left({\theta }_F\right).\end{array}$$ (10)

If F signals are coherent, and their propagation coefficients are ${\left\{{\eta }_f\right\}}_{f=1}^F$, it is necessary to assume that

$$\begin{array}{c}\hfill {\eta }_1+{\eta }_2+\dots +{\eta }_F\ne 0.\end{array}$$ (11)

This assumption guarantees that the ambiguous signals do not vanish on the individual sparse arrays; otherwise, the collection of coherent signals are cancelled out on Array $\mathbb{A}$ since ${\eta }_1\mathit{a}\left({\theta }_1\right)+\dots +{\eta }_F\mathit{a}\left({\theta }_F\right)=\mathit{0}$ holds for every snapshot. In real cases, the probability for a collection of signals to be vanishing is extremely low. We assume that the signals are non-vanishing on both Array $\mathbb{A}$ and $\mathbb{B}$.

In the DOA estimation using coprime array, one needs to estimate ${\left\{{\theta }_q\right\}}_{q=1}^Q$ from $T_s$ snapshots of the measurements ${\left\{{\mathit{y}}_A\left(t\right),{\mathit{y}}_B\left(t\right)\right\}}_{t=1}^{T_s}$. The existing methods rely on the sensor-by-sensor correlations of the received signal [1]—for example, the signals on the $i_A$th sensor of Array $\mathbb{A}$ and the $i_B$th sensor of Array $\mathbb{B}$. Suppose the samples on the two sensors (omitting the additive noises) are, respectively,

$$\begin{array}{ccc}\hfill y_A(i_A,t)& =& \sum _{q=1}^Q{s}_q\left(t\right)e^{jM{i}_A{\theta }_q},\hfill \end{array}$$ (12)

$$\begin{array}{ccc}\hfill y_B(i_B,t)& =& \sum _{q=1}^Q{s}_q\left(t\right)e^{jN{i}_B{\theta }_q}.\hfill \end{array}$$ (13)

When the Q signals are statistically independent, $E\left\{s_p\left(t\right)s_q^{*}\left(t\right)\right\}=0$ for $p\ne q$. The correlation becomes

$$\begin{array}{c}\hfill E\left\{y_A(i_A,t)y_B^{*}(i_B,t)\right\}=\sum _{q=1}^{Q}E{\left|s_q\left(t\right)\right|}^2{e}^{j\left(M{i}_A-N{i}_B\right){\theta }_q}.\end{array}$$ (14)

Taking all the integer combinations $(i_A,i_B)$, $M{i}_A-N{i}_B$ traverses all the integers between $-MN$ and $MN$. The rearranged spatial autocorrelations are therefore a superposition of Q sinusoids on an virtual ULA of size $\mathcal{O}\left(MN\right)$. A much larger correlation matrix can be constructed to resolve $\mathcal{O}\left(MN\right)$ signals by the subspace-based methods like MUSIC [1,2].

However, the presence of coherent signals indicates that $E\left\{y_A(i_A,t)y_B^{*}(i_B,t)\right\}$ contains cross-terms. For example, if two signals from ${\theta }_p$ and ${\theta }_q$ are coherent, the following component included in the correlation is non-zero:

$$\begin{array}{c}\hfill E\left\{s_p\left(t\right)s_q^{*}\left(t\right)\right\}{e}^{j\left(M{i}_A{\theta }_p-N{i}_B{\theta }_q\right)}.\end{array}$$ (15)

Since ${\theta }_p\ne {\theta }_q$, the cross-term is not corresponding to any sinusoid component on the virtual ULA. When the rearranged spatial autocorrelations are used to form a correlation matrix as in [2] or [18], the signal subspace structure is strongly contaminated, leading to a failed DOA estimation. A demonstrative example of a failed DOA estimation is given in Figure 4a in the simulations.

Figure 4.

The pseudo-spectra produced by 4-MUSIC are the blue curves. The true DOA of independent signals are marked by dashed vertical lines, and the true DOA of coherent signals are marked by dotted vertical lines.

Based on the above signal model, in Section 3, we formulate an FCM of the coprime array signal that can be adopted for the fourth-order DOA estimation. Then, a generalized spatial smoothing scheme, which is crucial for resolving coherent signals from the FCM, is introduced in Section 4.

3. Formulation of FO Cumulant Matrix

In this section, we begin by revising the FO cumulants of a random vector. Next, we formulate an FCM for the coprime array signal. The subspace structure of the FCM is carefully analyzed.

3.1. FCM of Complex Amplitude Vector

Under the assumption that the complex amplitudes of signals are symmetrically distributed, the FCM of the vector $\mathit{s}\left(t\right)$, denoted by $\mathsf{\Psi }\left(\mathit{s}\right)$, is well defined and given in [11]

$$\begin{array}{ccc}\hfill \mathsf{\Psi }\left(\mathit{s}\right)& =& E\left\{\left(\mathit{s}\left(t\right)\otimes {\mathit{s}}^{*}\left(t\right)\right){\left(\mathit{s}\left(t\right)\otimes {\mathit{s}}^{*}\left(t\right)\right)}^H\right\}\hfill \\ & & -E\left\{\mathit{s}\left(t\right)\otimes {\mathit{s}}^{*}\left(t\right)\right\}E{\left\{\mathit{s}\left(t\right)\otimes {\mathit{s}}^{*}\left(t\right)\right\}}^H\hfill \\ & & -E\left\{\mathit{s}\left(t\right){\mathit{s}}^H\left(t\right)\right\}\otimes E{\left\{\mathit{s}\left(t\right){\mathit{s}}^H\left(t\right)\right\}}^{*}.\hfill \end{array}$$ (16)

In the gth coherent group, the FCM of the group complex amplitude vector ${\mathit{s}}_g\left(t\right)$ is similarly formulated as in (16), and is denoted by $\mathsf{\Psi }\left({\mathit{s}}_g\right)$. Since the elements in ${\mathit{s}}_g\left(t\right)$ are linearly dependent, substituting (9) into (16), the FCM of ${\mathit{s}}_g\left(t\right)$ becomes

$$\begin{array}{c}\hfill \mathsf{\Psi }\left({\mathit{s}}_g\right)=\left({\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}\right)\mathsf{\Psi }\left({\sigma }_g\right){\left({\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}\right)}^H.\end{array}$$ (17)

In (17), ${\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}$ is a vector of length $Q_g^2$, and $\mathsf{\Psi }\left({\sigma }_g\right)$ is a scalar, in detail

$$\begin{array}{c}\hfill \mathsf{\Psi }\left({\sigma }_g\right)=E\left\{\sigma \left(t\right){\sigma }^{*}\left(t\right){\sigma }^{*}\left(t\right)\sigma \left(t\right)\right\}-2E\left\{\sigma \left(t\right){\sigma }^{*}\left(t\right)\right\}E\left\{{\sigma }^{*}\left(t\right)\sigma \left(t\right)\right\}.\end{array}$$ (18)

$\mathsf{\Psi }\left({\sigma }_g\right)$ represents the FO cumulant of the source ${\sigma }_g\left(t\right)$. When the source follows a circularly symmetric non-Gaussian distribution, $\mathsf{\Psi }\left({\sigma }_g\right)$ is non-zero. To sum up, $\mathsf{\Psi }\left({\mathit{s}}_g\right)$ is a $Q_g^2\times Q_g^2$ matrix with rank one, which describes the FO cumulants of the vector ${\mathit{s}}_g$.

3.2. FCM of Coprime Array Signal

Defining an auxiliary vector

$$\begin{array}{c}\hfill \mathit{z}\left(t\right)={\mathit{y}}_A\left(t\right)\otimes {\mathit{y}}_B^{*}\left(t\right),\end{array}$$ (19)

the following matrix is the FO moments of the array signal:

$$\begin{array}{ccc}\hfill {\mathbf{\Gamma }}_4\left(\mathit{z}\right)& =& E\left\{\mathit{z}\left(t\right){\mathit{z}}^H\left(t\right)\right\}\hfill \\ & =& E\left\{\left({\mathit{y}}_A\left(t\right)\otimes {\mathit{y}}_B^{*}\left(t\right)\right){\left({\mathit{y}}_A\left(t\right)\otimes {\mathit{y}}_B^{*}\left(t\right)\right)}^H\right\}.\hfill \end{array}$$ (20)

The autocorrelation matrices of the received signal on Array $\mathbb{A}$ and Array $\mathbb{B}$, and the cross-correlation vector between sparse arrays are the second-order moments of the array signal, respectively formulated as

$$\begin{array}{ccc}\hfill {\mathbf{\Gamma }}_2\left({\mathit{y}}_A\right)& =& E\left\{{\mathit{y}}_A\left(t\right){\mathit{y}}_A^H\left(t\right)\right\},\hfill \end{array}$$ (21)

$$\begin{array}{ccc}\hfill {\mathbf{\Gamma }}_2\left({\mathit{y}}_B\right)& =& E\left\{{\mathit{y}}_B\left(t\right){\mathit{y}}_B^H\left(t\right)\right\},\hfill \end{array}$$ (22)

$$\begin{array}{ccc}\hfill {\mathit{\mu }}_2\left(\mathit{z}\right)& =& E\left\{{\mathit{y}}_A\left(t\right)\otimes {\mathit{y}}_B^{*}\left(t\right)\right\}.\hfill \end{array}$$ (23)

We now introduce a matrix consisting of the FO cumulants of the received signal. The matrix is a combination of the FO and second-order moments defined above, formulated as

$$\begin{array}{c}\hfill \mathsf{\Phi }={\mathbf{\Gamma }}_4\left(\mathit{z}\right)-{\mathit{\mu }}_2\left(\mathit{z}\right){\mathit{\mu }}_2^H\left(\mathit{z}\right)-{\mathbf{\Gamma }}_2\left({\mathit{y}}_A\right)\otimes {\mathbf{\Gamma }}_2^{*}\left({\mathit{y}}_B\right).\end{array}$$ (24)

Each element in $\mathsf{\Phi }$ is a FO cumulant of the received signal. Since the additive noises are statistically independent of the signal, and the FO cumulants of the Gaussian noise are identically zero [19], substituting (5), (16), (20) and (21) into (24) yields

$$\begin{array}{ccc}\hfill \mathsf{\Phi }& =& E\left\{\left(\left(\mathit{A}\mathit{s}\left(t\right)\right)\otimes {\left(\mathit{B}\mathit{s}\left(t\right)\right)}^{*}\right){\left(\left(\mathit{A}\mathit{s}\left(t\right)\right)\otimes {\left(\mathit{B}\mathit{s}\left(t\right)\right)}^{*}\right)}^H\right\}\hfill \\ & & -E\left\{\left(\mathit{A}\mathit{s}\left(t\right)\right)\otimes {\left(\mathit{B}\mathit{s}\left(t\right)\right)}^{*}\right\}E{\left\{\left(\mathit{A}\mathit{s}\left(t\right)\right)\otimes {\left(\mathit{B}\mathit{s}\left(t\right)\right)}^{*}\right\}}^H\hfill \\ & & -E\left\{\mathit{A}\mathit{s}\left(t\right){\mathit{s}}^H\left(t\right){\mathit{A}}^H\right\}\otimes E{\left\{\mathit{B}\mathit{s}\left(t\right){\mathit{s}}^H\left(t\right){\mathit{B}}^H\right\}}^{*}\hfill \\ & =& \left(\mathit{A}\otimes {\mathit{B}}^{*}\right)\mathsf{\Psi }\left(\mathit{s}\right){\left(\mathit{A}\otimes {\mathit{B}}^{*}\right)}^H.\hfill \end{array}$$ (25)

The special structure of $\mathsf{\Phi }$ informs the array configuration and the FO property of the sources. Most importantly, $\mathit{A}\otimes {\mathit{B}}^{*}$ is built up by the steering vectors of impinging signals, which implies the DOAs. In the remainder of this paper, $\mathsf{\Phi }$ is referred to as the FCM of Coarray $\mathbb{AB}$.

In practical situations, the theoretical FCM is unknown and has to be estimated. If the signal is second-order and fourth-order ergodic, the empirical estimates of the moments in (20) and (21) can be derived by the sample second-order statistics and the sample fourth-order statistics [14]

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{\Gamma }}}_4\left(\mathit{z}\right)& =& \frac{1}{T_s}\sum _{t=1}^{T_s}\left({\mathit{y}}_A\left(t\right)\otimes {\mathit{y}}_B^{*}\left(t\right)\right){\left({\mathit{y}}_A\left(t\right)\otimes {\mathit{y}}_B^{*}\left(t\right)\right)}^H,\hfill \end{array}$$ (26)

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{\Gamma }}}_2\left({\mathit{y}}_A\right)& =& \frac{1}{T_s}\sum _{t=1}^{T_s}{\mathit{y}}_A\left(t\right){\mathit{y}}_A^H\left(t\right),\hfill \end{array}$$ (27)

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{\Gamma }}}_2\left({\mathit{y}}_B\right)& =& \frac{1}{T_s}\sum _{t=1}^{T_s}{\mathit{y}}_B\left(t\right){\mathit{y}}_B^H\left(t\right),\hfill \end{array}$$ (28)

$$\begin{array}{ccc}\hfill {\widehat{\mathit{\mu }}}_2\left(\mathit{z}\right)& =& \frac{1}{T_s}\sum _{t=1}^{T_s}{\mathit{y}}_A\left(t\right)\otimes {\mathit{y}}_B^{*}\left(t\right).\hfill \end{array}$$ (29)

The empirical estimation of the FCM $\widehat{\mathsf{\Phi }}$ [11] can be constructed with the sample moments in (26), matching the structure in (24)

$$\begin{array}{c}\hfill \widehat{\mathsf{\Phi }}={\widehat{\mathbf{\Gamma }}}_4\left(\mathit{z}\right)-{\widehat{\mathit{\mu }}}_2\left(\mathit{z}\right){\widehat{\mathit{\mu }}}_2^H\left(\mathit{z}\right)-{\widehat{\mathbf{\Gamma }}}_2\left({\mathit{y}}_A\right)\otimes {\widehat{\mathbf{\Gamma }}}_2^{*}\left({\mathit{y}}_B\right).\end{array}$$ (30)

3.3. Subspace Structure of FCM

In (25), since the cumulants of sums of independent processes are the sums of the individual cumulants [11], the FCM $\mathsf{\Phi }$ is the sum of FCMs of the individual coherent groups

$$\begin{array}{ccc}\hfill \mathsf{\Phi }& =& \sum _{g=1}^G\mathsf{\Phi }\left(g\right),\hfill \\ \hfill \mathsf{\Phi }\left(g\right)& =& \left(\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)\right)\mathsf{\Psi }\left({\mathit{s}}_g\right){\left(\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)\right)}^H,\hfill \end{array}$$ (31)

where $\mathsf{\Phi }\left(g\right)$ is the FCM of the gth group, $\mathit{A}\left(g\right)$ and $\mathit{B}\left(g\right)$ are the steering vector matrices for the gth group on Array $\mathbb{A}$ and $\mathbb{B}$, respectively:

$$\begin{array}{ccc}\hfill \mathit{A}\left(g\right)& =& \left[\mathit{a}\left({\theta }_{g_1}\right),\dots ,\mathit{a}\left({\theta }_{g_{Q_g}}\right)\right],\hfill \end{array}$$ (32)

$$\begin{array}{ccc}\hfill \mathit{B}\left(g\right)& =& \left[\mathit{b}\left({\theta }_{g_1}\right),\dots ,\mathit{b}\left({\theta }_{g_{Q_g}}\right)\right].\hfill \end{array}$$ (33)

When $\mathsf{\Phi }$ is eigen-decomposed, it forms a signal-subspace (with the projection operator $\mathbf{\Pi }$) spanned by the eigenvectors corresponding to the large eigenvalues, and a noise-subspace (with the projection operator ${\mathbf{\Pi }}^{\perp }$) spanned by the eigenvectors corresponding to the small eigenvalues. The two subspaces are orthogonal.

From (31), the signal subspace of $\mathsf{\Phi }\left(g\right)$ is spanned by the column vectors in the matrix $\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)$. The signal subspace of $\mathsf{\Phi }$ is the direct sum of the signal subspaces of each $\mathsf{\Phi }\left(g\right)$. Obviously, $\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)$ includes the column vectors ${\{\mathit{a}({\theta }_{g_q})\otimes {\mathit{b}}^{*}({\theta }_{g_q})\}}_{q=1}^{Q_g}$. We can resort to the 4-MUSIC algorithm [11] to estimate the DOAs in the following manner. For a signal from the DOA $\theta$, the vector $\mathit{a}\left(\theta \right)\otimes {\mathit{b}}^{*}\left(\theta \right)$ is in the signal subspace of $\mathsf{\Phi }$, then there exists a null at $\theta$ on the so-called null-spectrum:

$$\begin{array}{c}\hfill h\left(\theta \right)={∥{\mathbf{\Pi }}^{\perp }\left(\mathit{a}\left(\theta \right)\otimes {\mathit{b}}^{*}\left(\theta \right)\right)∥}^2.\end{array}$$ (34)

The pseudo-spectrum, or $1/h\left(\theta \right)$, is used to locate the DOAs by searching for the maxima on it.

3.4. Impact of Coherent Signals on FCM

For an independent signal, itself constitutes a group, say the gth group, with $Q_g=1$. As in (17), $\mathsf{\Psi }\left({\mathit{s}}_g\right)$ becomes a non-zero scalar. The contribution of the independent signal to the FCM as in (31) is

$$\begin{array}{c}\hfill \mathsf{\Phi }\left(g\right)=\left(\mathit{a}\left({\theta }_{g_1}\right)\otimes {\mathit{b}}^{*}\left({\theta }_{g_1}\right)\right)\mathsf{\Psi }\left({\mathit{s}}_g\right){\left(\mathit{a}\left({\theta }_{g_1}\right)\otimes {\mathit{b}}^{*}\left({\theta }_{g_1}\right)\right)}^H.\end{array}$$ (35)

Clearly, the signal subspace of $\mathsf{\Phi }\left(g\right)$ is spanned by the vector $\mathit{a}\left({\theta }_{g_1}\right)\otimes {\mathit{b}}^{*}\left({\theta }_{g_1}\right)$. Then, a peak appears at ${\theta }_{g_1}$ on the 4-MUSIC pseudo-spectrum.

For a group with multiple coherent signals that $Q_g\ge 2$, $\mathsf{\Psi }\left({\mathit{s}}_g\right)$ is a $Q_g^2\times Q_g^2$ matrix with rank one. Then, the signal subspace of $\mathsf{\Phi }\left(g\right)$ has only one dimension. Substituting (17) into (31), we derive

$$\begin{array}{c}\hfill \mathsf{\Phi }\left(g\right)=\left(\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)\right)\left({\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}\right)\mathsf{\Psi }\left({\sigma }_g\right){\left({\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}\right)}^H{\left(\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)\right)}^H.\end{array}$$ (36)

In (36), the one-dimensional signal subspace of $\mathsf{\Phi }\left(g\right)$ is spanned by the vector

$$\begin{array}{c}\hfill \left(\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)\right)\left({\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}\right).\end{array}$$ (37)

The vector in (37) is a linear combination of the $Q_g^2$ column vectors ${\left\{\mathit{a}\left({\theta }_{g_p}\right)\otimes {\mathit{b}}^{*}\left({\theta }_{g_q}\right)\right\}}_{p,q=1}^{Q_g}$. In the coherent signal case, the column vectors in $\mathit{A}\left(g\right)\otimes {\mathit{B}}^{*}\left(g\right)$ are merged. Hence, the signal subspace of $\mathsf{\Phi }\left(g\right)$ has only one dimension. The 4-MUSIC algorithm therefore fails to locate the DOAs of coherent signals.

4. Generalized Spatial Smoothing on FCM

In this section, we introduce a generalized spatial smoothing scheme applied to the FCM $\mathsf{\Phi }$. The scheme leads to a successful estimation of the coherent signals.

Since the two sparse arrays are both uniform, we can divide each of them, for example, Array $\mathbb{A}$ into overlapping subarrays. Every subarray contains $K_A$ continuous sensors, with sensors $\left\{0,1,\dots ,K_A-1\right\}$ forming the 0th subarray, sensors $\left\{1,\dots ,K_A\right\}$ forming the 1st subarray, etc. Similarly, Array $\mathbb{B}$ is divided into overlapping subarrays of size $K_B$. Choosing the uth subarray of Array $\mathbb{A}$ and the vth subarray of Array $\mathbb{B}$, a coprime subarray can be formed, and is denoted by the $(u,v)$ sub-coarray. The subarray indices can be chosen from $u=0,1,\dots ,L_A-K_A$ and $v=0,1,\dots ,L_B-K_B$. An illustration of the $(u,v)$ sub-coarray is in Figure 2.

Figure 2.

$(u,v)$ sub-coarray formed by two sparse subarrays.

For a DOA $\theta$, the partial steering vectors on the uth subarray of Array $\mathbb{A}$, and the vth subarray of Array $\mathbb{B}$ are denoted by

$$\begin{array}{ccc}\hfill {\mathit{a}}_u\left(\theta \right)& =& {\left[e^{jMu\theta },\dots ,e^{jM(u+K_A-1)\theta }\right]}^T,\hfill \end{array}$$ (38)

$$\begin{array}{ccc}\hfill {\mathit{b}}_v\left(\theta \right)& =& {\left[e^{jNv\theta },\dots ,e^{jN(v+K_B-1)\theta }\right]}^T.\hfill \end{array}$$ (39)

The matrices of the collection of partial steering vectors in the gth group are denoted by

$$\begin{array}{ccc}\hfill {\mathit{A}}_u\left(g\right)& =& \left[{\mathit{a}}_u\left({\theta }_{g_1}\right),\dots ,{\mathit{a}}_u\left({\theta }_{g_{Q_g}}\right)\right],\hfill \end{array}$$ (40)

$$\begin{array}{ccc}\hfill {\mathit{B}}_v\left(g\right)& =& \left[{\mathit{b}}_v\left({\theta }_{g_1}\right),\dots ,{\mathit{b}}_v\left({\theta }_{g_{Q_g}}\right)\right].\hfill \end{array}$$ (41)

On the $(u,v)$ sub-coarray, an FCM is inherently defined analogous to (25). The size of the sub-coarray FCM is determined by the sub-coarray size $K_A,K_B$. As in (31), the sub-coarray FCM can be decomposed into the sum of contributions from each group

$$\begin{array}{c}\hfill \begin{array}{c}{\mathsf{\Phi }}_{u,v}=\sum _{g=1}^G{\mathsf{\Phi }}_{u,v}\left(g\right),\\ {\mathsf{\Phi }}_{u,v}\left(g\right)=\left({\mathit{A}}_u\left(g\right)\otimes {\mathit{B}}_v^{*}\left(g\right)\right)\mathsf{\Psi }\left({\mathit{s}}_g\right){\left({\mathit{A}}_u\left(g\right)\otimes {\mathit{B}}_v^{*}\left(g\right)\right)}^H.\end{array}\end{array}$$ (42)

Comparing the uth subarray and the 0th subarray on Array $\mathbb{A}$, and comparing the vth subarray and the 0th subarray on Array $\mathbb{B}$, a relationship exists between the following matrices:

$$\begin{array}{ccc}\hfill {\mathit{A}}_u\left(g\right)& =& {\mathit{A}}_0\left(g\right){\mathsf{\Omega }}_A^u\left(g\right),\hfill \end{array}$$ (43)

$$\begin{array}{ccc}\hfill {\mathit{B}}_v\left(g\right)& =& {\mathit{B}}_0\left(g\right){\mathsf{\Omega }}_B^v\left(g\right).\hfill \end{array}$$ (44)

In (43), ${\mathsf{\Omega }}_A^u\left(g\right)$ and ${\mathsf{\Omega }}_B^v\left(g\right)$ are the uth and vth power of the following $Q_g\times Q_g$ diagonal matrices, respectively:

$$\begin{array}{ccc}\hfill {\mathsf{\Omega }}_A\left(g\right)& =& \mathrm{diag}\left[e^{jM{\theta }_{g_1}},\dots ,e^{jM{\theta }_{g_{Q_g}}}\right],\hfill \end{array}$$ (45)

$$\begin{array}{ccc}\hfill {\mathsf{\Omega }}_B\left(g\right)& =& \mathrm{diag}\left[e^{jN{\theta }_{g_1}},\dots ,e^{jN{\theta }_{g_{Q_g}}}\right].\hfill \end{array}$$ (46)

Comparing the $(u,v)$ sub-coarray with the $(0,0)$ sub-coarray, the following relation exists

$$\begin{array}{c}\begin{array}{cc}{\mathit{A}}_u\left(g\right)\otimes {\mathit{B}}_v^{*}\left(g\right)& =\left({\mathit{A}}_0\left(g\right){\mathsf{\Omega }}_A^u\left(g\right)\right)\otimes \left({\mathit{B}}_0^{*}\left(g\right){\mathsf{\Omega }}_B^{-v}\left(g\right)\right)\hfill \\ & =\left({\mathit{A}}_0\left(g\right)\otimes {\mathit{B}}_0^{*}\left(g\right)\right){\mathsf{\Omega }}^{u,-v}\left(g\right),\hfill \end{array}\hfill \end{array}$$ (47)

where ${\mathsf{\Omega }}^{u,-v}\left(g\right)={\mathsf{\Omega }}_A^u\left(g\right)\otimes {\mathsf{\Omega }}_B^{-v}\left(g\right)$ is a $Q_g^2\times Q_g^2$ diagonal matrix. Therefore, ${\mathsf{\Phi }}_{u,v}\left(g\right)$ can be written as

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{u,v}\left(g\right)=\left({\mathit{A}}_0\left(g\right)\otimes {\mathit{B}}_0^{*}\left(g\right)\right)\left[{\mathsf{\Omega }}^{u,-v}\left(g\right)\mathsf{\Psi }\left({\mathit{s}}_g\right){\mathsf{\Omega }}^{-u,v}\left(g\right)\right]{\left({\mathit{A}}_0\left(g\right)\otimes {\mathit{B}}_0^{*}\left(g\right)\right)}^H.\end{array}$$ (48)

We observe that ${\mathsf{\Phi }}_{u,v}\left(g\right)$ and ${\mathsf{\Phi }}_{0,0}\left(g\right)$ share the same signal subspace, and are related by a ‘rotation’ of the matrix $\mathsf{\Psi }\left({\mathit{s}}_g\right)$.

The generalized spatial smoothed FCM of coprime array is defined as the sum of FCMs on all of the similar sub-coarray:

$$\begin{array}{c}\hfill \overline{\mathsf{\Phi }}=\sum _{u=0}^{L_A-K_A}\sum _{v=0}^{L_B-K_B}{\mathsf{\Phi }}_{u,v}.\end{array}$$ (49)

Denote $\overline{\mathsf{\Psi }}\left({\mathit{s}}_g\right)$ as the smoothed FCM of the group complex amplitude vector ${\mathit{s}}_g\left(t\right)$, written as

$$\begin{array}{c}\hfill \overline{\mathsf{\Psi }}\left({\mathit{s}}_g\right)=\sum _{u=0}^{L_A-K_A}\sum _{v=0}^{L_B-K_B}{\mathsf{\Omega }}^{u,-v}\left(g\right)\mathsf{\Psi }\left({\mathit{s}}_g\right){\mathsf{\Omega }}^{-u,v}\left(g\right).\end{array}$$ (50)

Substituting (42), (48) and (50) into (49), the smoothed FCM $\overline{\mathsf{\Phi }}$ of the coprime array signal is

$$\begin{array}{c}\hfill \begin{array}{c}\overline{\mathsf{\Phi }}=\sum _{g=1}^G\overline{\mathsf{\Phi }}\left(g\right),\\ \overline{\mathsf{\Phi }}\left(g\right)=\left({\mathit{A}}_0\left(g\right)\otimes {\mathit{B}}_0^{*}\left(g\right)\right)\overline{\mathsf{\Psi }}\left({\mathit{s}}_g\right){\left({\mathit{A}}_0\left(g\right)\otimes {\mathit{B}}_0^{*}\left(g\right)\right)}^H.\end{array}\end{array}$$ (51)

In (50), $\overline{\mathsf{\Psi }}\left({\mathit{s}}_g\right)$ is rank-enhanced after spatial smoothing compared to $\mathsf{\Psi }\left({\mathit{s}}_g\right)$. Consequently in (51), the smoothed FCM $\overline{\mathsf{\Phi }}$ is rank-enhanced. The effect of the generalized spatial smoothing scheme on the FCM is analogous to the spatial smoothing scheme on the correlation matrix of a ULA [9].

An important theorem is in place here. We show that with some restrictions, the vectors ${\left\{{\mathit{a}}_0\left({\theta }_q\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)\right\}}_{q=1}^Q$ for all the signals are in the signal subspace of $\overline{\mathsf{\Phi }}$.

Theorem 1.

In the gth group, if $L_A-K_A+1\ge Q_g$ and $L_B-K_B+1\ge Q_g$, the vector ${\mathit{a}}_0({\theta }_{g_q})\otimes {\mathit{b}}_0^{*}({\theta }_{g_q})$ for any one of the DOAs ${\{{\theta }_{g_q}\}}_{q=1}^{Q_g}$ is in the signal subspace of $\overline{\mathsf{\Phi }}\left(g\right)$.

Proof. 

See Appendix A. ☐

Corollary 1.

Let $Q_{max}=\max (Q_1,\dots ,Q_G)$. If $L_A-K_A+1\ge Q_{max}$, $L_B-K_B+1\ge Q_{max}$, then the vector ${\mathit{a}}_0\left({\theta }_q\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)$ for any one of the DOAs ${\left\{{\theta }_q\right\}}_{q=1}^Q$ is in the signal subspace of the smoothed FCM $\overline{\mathsf{\Phi }}$.

Proof. 

Following Theorem 1 and that $\overline{\mathsf{\Phi }}={\sum }_{g=1}^G\overline{\mathsf{\Phi }}\left(g\right)$, the signal subspace of $\overline{\mathsf{\Phi }}$ is the direct sum of all the signal subspaces of $\overline{\mathsf{\Phi }}\left(g\right),g=1,\dots ,G$. ☐

Corollary 1 indicates that, if the numbers of overlapping subarrays on both sparse arrays are no less than the largest number of the coherent signals, the vectors ${\left\{{\mathit{a}}_0\left({\theta }_q\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)\right\}}_{q=1}^Q$ for both the independent and coherent signals are in the signal subspace of $\overline{\mathsf{\Phi }}$. Then, one can eigen-decompose $\overline{\mathsf{\Phi }}$ to acquire a noise subspace with the projection operator ${\overline{\mathbf{\Pi }}}^{\perp }$. The vector ${\mathit{a}}_0\left(\theta \right)\otimes {\mathit{b}}_0^{*}\left(\theta \right)$ for any signal with a DOA $\theta$ is orthogonal to the noise subspace. From the smoothed FCM, the null-spectrum produced by 4-MUSIC is defined as

$$\begin{array}{c}\hfill \overline{h}\left(\theta \right)={∥{\overline{\mathbf{\Pi }}}^{\perp }\left({\mathit{a}}_0\left(\theta \right)\otimes {\mathit{b}}_0^{*}\left(\theta \right)\right)∥}^2.\end{array}$$ (52)

On the pseudo-spectrum $1/\overline{h}\left(\theta \right)$, both the independent and coherent signals create peaks at their respective directions.

Remark: the sub-coarray FCM ${\mathsf{\Phi }}_{u,v}$ in (42) is a principle sub-matrix of the full-coarray FCM $\mathsf{\Phi }$. The indices of columns (and rows) of the principle sub-matrix ${\mathsf{\Phi }}_{u,v}$ in $\mathsf{\Phi }$ are

$$\begin{array}{c}\hfill (u:u+K_A-1)·L_B+(v:v+K_B-1).\end{array}$$ (53)

The generalized smoothing process can be accomplished by summing all the proper principle sub-matrices with the indices in (53) from $\mathsf{\Phi }$.

The generalized spatial smoothing scheme is obviously at the expense of a reduced effective array aperture. In fact, the size of the FCM $\mathsf{\Phi }$ is $L_A{L}_B\times L_A{L}_B$, while the smoothed FCM $\overline{\mathsf{\Phi }}$ is only $K_A{K}_B\times K_A{K}_B$. Considering the algorithmic complexity, the eigen-decomposition of the FCM $\overline{\mathsf{\Phi }}$ takes $\mathcal{O}\left(K_A^3{K}_B^3\right)$ operations. The complexity is equivalent to MUSIC on a ULA with a similar extent to the coprime array.

5. Removing False Peaks

On the pseudo-spectrum from the smoothed FCM, some false peaks occasionally arise at the directions where none of the true signals resides. An example of the false peaks is in Figure 5 in the simulations. In this section, we explain the false peak phenomenon and provide a technique to remove them.

Figure 5.

The pseudo-spectra using the smoothed FCM of two sparse arrays have false peaks as in (a,b). Combining the null-spectra of the three coprime arrays following (65) can remove the false peaks as in (c).

5.1. Explanation of the False Peaks

When the generalized spatial smoothing scheme is applied to enhance the rank of $\overline{\mathsf{\Psi }}\left({\mathit{s}}_g\right)$ in (51), not only the vector ${\mathit{a}}_0({\theta }_{g_q})\otimes {\mathit{b}}_0^{*}({\theta }_{g_q})$ for the signal DOAs ${\{{\theta }_{g_q}\}}_{q=1}^{Q_g}$, but also the cross-terms ${\left\{{\mathit{a}}_0({\theta }_{g_p})\otimes {\mathit{b}}_0^{*}({\theta }_{g_q})\right\}}_{p\ne q}$ appear in the signal subspace of $\overline{\mathsf{\Phi }}(g)$. These cross-terms are not corresponding to any signal component and should not create peaks on the pseudo-spectrum. However, the steering vectors ${\mathit{a}}_0\left(\theta \right)$ and ${\mathit{b}}_0\left(\theta \right)$ are ambiguous due to the sparsity of Array $\mathbb{A}$ and $\mathbb{B}$. We show that a false peak may appear as a result of the direction ambiguity. Because two independent signals will not create such cross-terms, in the remainder of this section, the discussion is limited to a single coherent group. For simplicity, we omit the group index g.

Since Array $\mathbb{A}$ is M-sparse and Array $\mathbb{B}$ is N-sparse, the steering vectors are ambiguous. In particular, for ${\theta }_p$ on Array $\mathbb{A}$ and ${\theta }_q$ on Array $\mathbb{B}$,

$$\begin{array}{ccc}\hfill {\mathit{a}}_0\left({\theta }_p\right)& =& {\mathit{a}}_0\left({\theta }_p+2\pi m/M\right),\hfill \end{array}$$ (54)

$$\begin{array}{ccc}\hfill {\mathit{b}}_0\left({\theta }_q\right)& =& {\mathit{b}}_0\left({\theta }_q+2\pi n/N\right),\hfill \end{array}$$ (55)

where $m,n$ are arbitrary integers. The ambiguity can be illustrated more clearly by the array beam patterns. In Figure 3, since Array $\mathbb{A}$ and $\mathbb{B}$ are both sparse and uniform, their beam patterns have multiple grating lobes.

Figure 3.

An illustration of the beam patterns on Array $\mathbb{A}$ and $\mathbb{B}$. The solid line represents the grating lobe of Array $\mathbb{A}$, the dotted line represents the grating lobe of Array $\mathbb{B}$. The true directions ${\theta }_p$ and ${\theta }_q$ are marked by the highest lobes, while the lower lobes are the grating lobes. Theoretically, the lobes of each array should have the equal height.

If it happens that a grating lobe of the Array $\mathbb{A}$ beam pattern and a grating lobe of the Array $\mathbb{B}$ beam pattern overlap at the direction $\phi$, i.e., for a pair of non-zero integers $(m,n)$,

$$\begin{array}{ccc}\hfill \phi & =& {\theta }_p+2\pi m/M,\hfill \end{array}$$ (56)

$$\begin{array}{ccc}\hfill \phi & =& {\theta }_q+2\pi n/N,\hfill \end{array}$$ (57)

the cross-term of the steering vectors becomes

$$\begin{array}{c}\hfill {\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)={\mathit{a}}_0\left(\phi \right)\otimes {\mathit{b}}_0^{*}\left(\phi \right).\end{array}$$ (58)

When the smoothed FCM ${\overline{\mathsf{\Phi }}}_{AB}$ of Coarray $\mathbb{AB}$ is eigen-decomposed with the noise subspace projection operator ${\overline{\mathbf{\Pi }}}_{AB}$, the null-spectrum at $\phi$ becomes

$$\begin{array}{c}\begin{array}{ll}{\overline{h}}_{AB}\left(\phi \right)& ={∥{\overline{\mathbf{\Pi }}}_{AB}^{\perp }\left({\mathit{a}}_0\left(\phi \right)\otimes {\mathit{b}}_0^{*}\left(\phi \right)\right)∥}^2\\ & ={∥{\overline{\mathbf{\Pi }}}_{AB}^{\perp }\left({\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)\right)∥}^2=0,\end{array}\hfill \end{array}$$ (59)

since the cross-term ${\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)$ is in the signal subspace of ${\overline{\mathsf{\Phi }}}_{AB}$. A false peak at $\phi$ will appear on the pseudo-spectrum.

One remark is in place here. In (56), the direction $\phi$ needs not to be strictly equal to ${\theta }_p+2\pi m/M$ or ${\theta }_q+2\pi n/N$. Once a grating lobe of the beam pattern ${\mathit{a}}_0\left({\theta }_p\right)$ and a grating lobe of the beam pattern ${\mathit{b}}_0\left({\theta }_q\right)$ overlap around $\phi$, a false peak still appears.

Furthermore, the grating lobe beamwidths of the beam patterns of ${\mathit{a}}_0\left({\theta }_p\right)$ and ${\mathit{b}}_0\left({\theta }_q\right)$ are $2\pi /M{K}_A$ and $2\pi /N{K}_B$, respectively. If $K_A\ge N$ and $K_B\ge M$, the beamwidths of the grating lobes are narrower than $2\pi /MN$. From the observation that $M,N$ are coprime numbers, the grating lobes of the beam patterns ${\mathit{a}}_0\left({\theta }_p\right)$ on Array $\mathbb{A}$ and ${\mathit{b}}_0\left({\theta }_q\right)$ on Array $\mathbb{B}$ may overlap at one direction within $[-\pi ,\pi ]$ at most, which means that the cross-term ${\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)$ may produce at most one false peak.

5.2. Supplementary Sparse Array

Suppose that a supplementary sparse array, namely Array $\mathbb{C}$, is deployed with its sensors positioned at

$$\begin{array}{c}\hfill {\mathit{x}}_C=\left\{R{i}_{C}d|i_C=0,1,\dots ,L_C-1\right\}.\end{array}$$ (60)

In (60), R is an integer, which is respectively coprime to M and N, and $L_C$ is the number of sensors in Array $\mathbb{C}$. Now, Array $\mathbb{A}$ and Array $\mathbb{C}$ can form a new coprime array, denoted by Coarray $\mathbb{AC}$. An FCM ${\mathsf{\Phi }}_{AC}$ can be derived for the array signal. Dividing Array $\mathbb{C}$ into identical subarrays of size $K_C$, we can perform the generalized spatial smoothing scheme on ${\mathsf{\Phi }}_{AC}$ to obtain the smoothed FCM ${\overline{\mathsf{\Phi }}}_{AC}$. Denoting the noise subspace projection operator by ${\overline{\mathbf{\Pi }}}_{AC}^{\perp }$, the null-spectrum of Coarray $\mathbb{AC}$ is

$$\begin{array}{c}\hfill {\overline{h}}_{AC}\left(\theta \right)={∥{\overline{\mathbf{\Pi }}}_{AC}^{\perp }\left({\mathit{a}}_0\left(\theta \right)\otimes {\mathit{c}}_0^{*}\left(\theta \right)\right)∥}^2,\end{array}$$ (61)

where ${\mathit{c}}_0\left(\theta \right)={\left[e^{jR0\theta },\dots ,e^{jR(K_C-1)\theta }\right]}^T$.

If $K_A\ge R$ and $K_C\ge M$ both hold, on the pseudo-spectrum $1/{\overline{h}}_{AC}\left(\theta \right)$ derived from Coarray $\mathbb{AC}$, the cross-term ${\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{c}}_0^{*}\left({\theta }_q\right)$ may produce at most one false peak, denoted by ${\phi }^{\prime }$. The false peak is induced by the following cross-term:

$$\begin{array}{c}\hfill {\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{c}}_0^{*}\left({\theta }_q\right)={\mathit{a}}_0\left({\phi }^{\prime }\right)\otimes {\mathit{c}}_0^{*}\left({\phi }^{\prime }\right).\end{array}$$ (62)

We can derive an important theorem for the position of the false peaks.

Theorem 2.

From the same pair of coherent signals ${\theta }_p,{\theta }_q$, if the cross-term ${\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)$ produces a false peak at $\phi$ on the pseudo-spectrum of Coarray $\mathbb{AB}$, and the cross-term ${\mathit{a}}_0\left({\theta }_p\right)\otimes {\mathit{c}}_0^{*}\left({\theta }_q\right)$ produces a false peak at ${\phi }^{\prime }$ on the pseudo-spectrum of Coarray $\mathbb{AC}$, then $\phi \ne {\phi }^{\prime }$.

Proof. 

The proof is by contradiction. On Coarray $\mathbb{AC}$, the direction ambiguity indicates the following relations for a pair of non-zero integers $(m,r)$:

$$\begin{array}{ccc}\hfill {\phi }^{\prime }& =& {\theta }_p+2\pi m/M,\hfill \end{array}$$ (63)

$$\begin{array}{ccc}\hfill {\phi }^{\prime }& =& {\theta }_q+2\pi r/R.\hfill \end{array}$$ (64)

If $\phi ={\phi }^{\prime }$, combining (56) and (63), we can deduce that $n/N=r/R$. Since N and R are coprime numbers, $(m,n,r)$ are within a range such that $\phi ,{\phi }^{\prime }$ are between $[-\pi ,\pi ]$, the equation holds only when $n=r=0$, which is contradictory to the non-zero assumption on the integers $n,r$. ☐

Theorem 2 indicates that, on two different coprime array configurations, the false peaks created by the same pair of coherent signals do not overlap on the pseudo-spectrum. This property can be used for removing the false peaks.

5.3. Removing False Peaks by Combined Spectrum

To remove the false peaks induced by the direction ambiguity, we use the property for which the false peaks do not overlap. From the three sparse arrays $\mathbb{A}$, $\mathbb{B}$ and $\mathbb{C}$, any two sparse arrays constitute a coprime array. Therefore, we may derive three null-spectra: ${\overline{h}}_{AB}\left(\theta \right)$ from Coarray $\mathbb{AB}$, ${\overline{h}}_{AC}\left(\theta \right)$ from Coarray $\mathbb{AC}$ and ${\overline{h}}_{BC}\left(\theta \right)$ from Coarray $\mathbb{BC}$. A combined null-spectrum can be generated from the individual null-spectra:

$$\begin{array}{c}\hfill {\overline{h}}_{ABC}\left(\theta \right)={\overline{h}}_{AB}\left(\theta \right)+{\overline{h}}_{AC}\left(\theta \right)+{\overline{h}}_{BC}\left(\theta \right).\end{array}$$ (65)

A necessary condition for the existence of a peak at $\theta$ on $1/{\overline{h}}_{ABC}\left(\theta \right)$ is that $\theta$ is corresponding to a null at any one of the three null-spectra. In fact, if $\theta$ is the DOA of a true signal, there always exists a null at $\theta$ on any one of the three null-spectra, ${\overline{h}}_{AB}\left(\theta \right)={\overline{h}}_{AC}\left(\theta \right)={\overline{h}}_{BC}\left(\theta \right)=0$. However, from Theorem 2, the false peak positions are different. Consequently, the false peaks are removed on the combined pseudo-spectrum $\theta$ on $1/{\overline{h}}_{ABC}\left(\theta \right)$.

6. Simulation Results

In this section, we present some simulations that demonstrate the DOA estimation of coherent signals using the generalized spatial smoothing scheme. We also exhibit the false peak phenomenon, and the removal of false peaks by the combined spectrum. The ability of estimating more signals than the number of sensors is verified, and the performances under various signal-to-noise ratio (SNR) and number of snapshots are studied as well.

6.1. Effectiveness of Generalized Spatial Smoothing

A coprime array is used to verify the generalized spatial smoothing scheme. Array $\mathbb{A}$ ($M=6$) has $L_A=9$ sensors, and Array $\mathbb{B}$ ($N=5$) has $L_B=10$ sensors. The unit inter-sensor spacing is half-wavelength. We consider ten narrowband signals impinging on the coprime arrays, in which four of them are independent signals, and the other six signals are divided into three coherent groups, with two signals in each group. The signal types are either Quadrature PSK or quaternary QAM, the powers of all sources are equal, and the coefficients along all the propagation paths have the equal amplitude, but with random complex phases. The noise is additive complex Gaussian, and the SNR on the sensors is set to be 0 dB (SNR is defined as the power of one signal to the noise power). In the estimation of the FCM $\mathsf{\Phi }$ in (25), 2000 snapshots are used. In the generalized spatial smoothing scheme, Array $\mathbb{A}$ is divided into subarrays of size $K_A=6$, and Array $\mathbb{B}$ is divided into subarrays of size $K_B=7$.

Figure 4a is the spectrum derived from the method in [2], which is most widely used on coprime arrays currently. The cross-terms induced by the coherent signals contaminate the structure of the signal subspace, and the DOA estimation consequently fails. In Figure 4b, the 4-MUSIC algorithm is applied to the FCM $\mathsf{\Phi }$. We can see that the four peaks corresponding to the independent signals appear, but the DOA of coherent signals are not resolved. On the contrary, if the generalized spatial smoothing scheme is used on the FCM, the peaks for both independent and coherent signals are clearly present in Figure 4c. It is exemplified that the DOA of coherent signals can be estimated from the smoothed FCM.

6.2. Removing False Peaks

In the second simulation, there exists a coherent group with three signals at the normalized DOAs $\left[-0.6\pi ,0.2\pi ,0.3\pi \right]$, another coherent group with two signals at $\left[-0.5\pi ,-0.3\pi \right]$, and four other independent signals at $\left[-0.4\pi ,-0.1\pi ,0.1\pi ,0.25\pi \right]$ in the environment. The signal types, source amplitudes and propagation coefficients are set up as in the first simulation. We first derive the pseudo-spectrum using the smoothed FCM of Coarray $\mathbb{AB}$ in Figure 5a. Both independent and coherent signals are estimated on the spectrum. However, there also exist multiple false peaks, which seriously affect the extraction of the true signals.

A supplementary sparse array, namely Array $\mathbb{C}$, with sparsity $R=7$ and $L_C=8$ sensors is deployed, and we can construct a new coprime array with Array $\mathbb{A}$ and Array $\mathbb{C}$. Let Array $\mathbb{C}$ be divided into subarrays of size $K_C=5$, and the pseudo-spectrum using the smoothed FCM of Coarray $\mathbb{AC}$ is displayed in Figure 5b. We can see that the peaks corresponding to the true signals are still there, while the directions of the false peaks are different from that in Figure 5a. Combining the three null-spectrum using (65), the pseudo-spectrum is displayed in Figure 5c. We can see that all the signals are found on the spectrum, while the false peaks are removed. Thus, we have shown the effectiveness of removing false peaks by the combined spectrum.

6.3. Estimating More Signals Than Sensor Numbers

In the third simulation, we demonstrate the ability of estimating more signals than the number of sensors, which still holds using the FCM-based approach. The array configuration, parameters and the estimation of FCM follow the second simulation. The number of signals is thirty-five, in which six signals are coherent. In Figure 6, we can see that the DOAs of the individual signals are correctly estimated, and even the number of signals exceeds the number of sensors. In the newly proposed FCM-based method, the size of the spatial smoothed FCM is $\mathcal{O}\left(MN\right)$; therefore, 4-MUSIC can process $\mathcal{O}\left(MN\right)$ signals using $\mathcal{O}(M+N)$ sensors.

Figure 6.

Twenty-seven sensors are used in the array configuration. Thirty-five signals (more than the number of sensors) exist in the environment, where six of them can be divided into three coherent pairs.

6.4. Performance Study

In the fourth simulation, we briefly study the root-mean-square error (RMSE) of the DOA estimates using the new method. The array configuration follows the second simulation. Suppose that ten signals with the equal power arrive at the coprime array, in which four signals are coherent, and the other six signals are statistically independent. We consider the cases that the number of snapshots varies from 16 to 4096. The noise follows a complex Gaussian distribution and SNR varies from −14 dB to 4 dB. Under each case, 500 trials were taken to calculate the average RMSE of all the DOA estimates. The result is shown in Figure 7.

Figure 7.

Average RMSE of all the DOA estimates. Each line indicates a fixed snapshot number, and the SNR varies within a line.

We can conclude from the figure that the performance drops when the SNR decreases, as well as when the number of snapshots reduces. If the snapshot number is sufficiently high, the requirement for SNR can be relaxed. This is due to the fact that Gaussian noise vanishes in the FO cumulants if the number of snapshots is large.

7. Conclusions

In this paper, the problem of direction-of-arrival (DOA) estimation of coherent signals on passive coprime arrays is investigated. We resort to the fourth-order cumulants to explore more information about the received signal. Formulating a fourth-order cumulant matrix (FCM) for the signal on a coprime array, a new estimation scheme based on the fourth-order MUSIC algorithm is developed.

The special structure of the FCM is combined with the array configuration to resolve the coherent signals. Using the property that the individual sparse arrays are uniform, on either of the sparse arrays, a series of overlapping identical subarrays can be extracted. Then, taking individually one subarray from each of the sparse arrays, a coprime subarray is constructed. We revealed that the FCMs of any two similar coprime subarrays share the same structure. Analogous to the spatial smoothing scheme applied to the correlation matrix on a uniform linear array, we propose a generalized spatial smoothing scheme applied to the FCM. The scheme yields a smoothed FCM with rank-enhancement. The DOAs of both the independent and coherent signals can be estimated using the smoothed FCM.

To remove the false peaks induced by the direction ambiguity, we use a supplementary sparse array for assistance. On the combined spectrum aided by the supplementary array, the false peaks are removed while the true peaks remain. Simulation examples are given to demonstrate the effectiveness and performance of the proposed approach. Future work includes considering the case of mixed independent, partially correlated and coherent signals. A few simulations were executed to verify the ability of the proposed method to handle this case. However, theoretical guarantees are not provided yet.

Appendix A. Proof of Theorem 1

Substituting (17) into (50), we derive

$$\begin{array}{c}\hfill \overline{\mathsf{\Psi }}\left({\mathit{s}}_g\right)=\sum _{u=0}^{L_A-K_A}\sum _{v=0}^{L_B-K_B}{\mathsf{\Omega }}^{u,-v}\left(g\right)\left({\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}\right)\mathsf{\Psi }\left({\sigma }_g\right){\left({\mathit{\eta }}_g\otimes {\mathit{\eta }}_g^{*}\right)}^H{\mathsf{\Omega }}^{-u,v}\left(g\right).\end{array}$$ (A1)

Since $\mathsf{\Psi }\left({\sigma }_g\right)$ is a non-zero scalar, we can omit it without affecting the rank of $\overline{\mathsf{\Psi }}\left({\mathit{s}}_g\right)$. The following discussion is limited to the gth group. For simplicity, the group index g is omitted.

Equation (A1) can be written as

$$\begin{array}{c}\begin{array}{ll}\overline{\mathsf{\Psi }}\left(\mathit{s}\right)& =\sum _{u=0}^{L_A-K_A}\sum _{v=0}^{L_B-K_B}{\mathsf{\Omega }}^{u,-v}\left(\mathit{\eta }\otimes {\mathit{\eta }}^{*}\right){\left(\mathit{\eta }\otimes {\mathit{\eta }}^{*}\right)}^H{\mathsf{\Omega }}^{-u,v}\\ & =\sum _{u=0}^{L_A-K_A}\sum _{v=0}^{L_B-K_B}{\mathit{\zeta }}_{u,v}{\mathit{\zeta }}_{u,v}^H,\end{array}\hfill \end{array}$$ (A2)

where ${\mathit{\zeta }}_{u,v}$ is a $Q^2\times 1$ vector, written as

$$\begin{array}{c}\hfill {\mathit{\zeta }}_{u,v}={\mathsf{\Omega }}^{u,-v}\left(\mathit{\eta }\otimes {\mathit{\eta }}^{*}\right)=\left({\mathsf{\Omega }}_A^u\mathit{\eta }\right)\otimes {\left({\mathsf{\Omega }}_B^v\mathit{\eta }\right)}^{*}.\end{array}$$ (A3)

The summation in (A2) can be written in the vector inner-product form

$$\begin{array}{c}\hfill \overline{\mathsf{\Psi }}\left(\mathit{s}\right)=\sum _{u=0}^{L_A-K_A}\sum _{v=0}^{L_B-K_B}{\mathit{\zeta }}_{u,v}{\mathit{\zeta }}_{u,v}^H=\mathbf{W}{\mathbf{W}}^H,\end{array}$$ (A4)

where $\mathbf{W}$ is a matrix with $Q^2$ column vectors

$$\begin{array}{c}\begin{array}{ll}\mathbf{W}& =\left[{\mathit{\zeta }}_{0,0},{\mathit{\zeta }}_{0,1},\dots ,{\mathit{\zeta }}_{0,L_B-K_B},\dots ,{\mathit{\zeta }}_{L_A-K_A,L_B-K_B}\right]\\ & =\left[{\mathsf{\Omega }}_A^0\mathit{\eta },\dots ,{\mathsf{\Omega }}_A^{L_A-K_A}\mathit{\eta }\right]\otimes {\left[{\mathsf{\Omega }}_B^0\mathit{\eta },\dots ,{\mathsf{\Omega }}_B^{L_B-K_B}\mathit{\eta }\right]}^{*}={\mathbf{W}}_A\otimes {\mathbf{W}}_B^{*}.\end{array}\hfill \end{array}$$ (A5)

In (A5), ${\mathbf{W}}_A$ and ${\mathbf{W}}_B$ are both $Q\times Q$ matrices. Each of them can also be written as the multiplication of a diagonal matrix and a Vandermonde matrix:

$$\begin{array}{ccccc}\hfill {\mathbf{W}}_A& =& \left[{\mathsf{\Omega }}_A^0\mathit{\eta },\dots ,{\mathsf{\Omega }}_A^{L_A-K_A}\mathit{\eta }\right]\hfill & =& \mathrm{diag}\left(\mathit{\eta }\right)·{\mathbf{V}}_A,\end{array}$$ (A6)

$$\begin{array}{ccccc}\hfill {\mathbf{W}}_B& =& \left[{\mathsf{\Omega }}_B^0\mathit{\eta },\dots ,{\mathsf{\Omega }}_B^{L_B-K_B}\mathit{\eta }\right]\hfill & =& \mathrm{diag}\left(\mathit{\eta }\right)·{\mathbf{V}}_B,\end{array}$$ (A7)

where the Vandermonde matrices ${\mathbf{V}}_A$ and ${\mathbf{V}}_B$ are

$$\begin{array}{ccc}\hfill {\mathbf{V}}_A& =& \left[\begin{array}{ccc}{e}^{jM0{\theta }_1}& \dots & e^{jM(L_A-K_A){\theta }_1}\\ ⋮& \ddots & ⋮\\ e^{jM0{\theta }_Q}& \dots & e^{jM(L_A-K_A){\theta }_Q}\end{array}\right],\hfill \end{array}$$ (A8)

$$\begin{array}{ccc}\hfill {\mathbf{V}}_B& =& \left[\begin{array}{ccc}{e}^{jN0{\theta }_1}& \dots & e^{jN(L_B-K_B){\theta }_1}\\ ⋮& \ddots & ⋮\\ e^{jN0{\theta }_Q}& \dots & e^{jN(L_B-K_B){\theta }_Q}\end{array}\right].\hfill \end{array}$$ (A9)

Substituting (A4) and (A5) into (51) yields

$$\begin{array}{c}\begin{array}{ll}\overline{\mathsf{\Phi }}& =\left({\mathit{A}}_0\otimes {\mathit{B}}_0^{*}\right)\mathbf{W}{\mathbf{W}}^H{\left({\mathit{A}}_0\otimes {\mathit{B}}_0^{*}\right)}^H\\ & =\left(\left({\mathit{A}}_0{\mathbf{W}}_A\right)\otimes \left({\mathit{B}}_0^{*}{\mathbf{W}}_B^{*}\right)\right){\left(\left({\mathit{A}}_0{\mathbf{W}}_A\right)\otimes \left({\mathit{B}}_0^{*}{\mathbf{W}}_B^{*}\right)\right)}^H=\mathbf{Y}{\mathbf{Y}}^H,\end{array}\hfill \end{array}$$ (A10)

where

$$\begin{array}{c}\begin{array}{ll}\mathbf{Y}& =\left({\mathit{A}}_0{\mathbf{W}}_A\right)\otimes \left({\mathit{B}}_0^{*}{\mathbf{W}}_B^{*}\right)\\ & =\left({\mathit{A}}_0·\mathrm{diag}\left(\mathit{\eta }\right)·{\mathbf{V}}_A\right)\otimes \left({\mathit{B}}_0^{*}·\mathrm{diag}\left({\mathit{\eta }}^{*}\right)·{\mathbf{V}}_B^{*}\right).\end{array}\hfill \end{array}$$ (A11)

In (A10), the structure of the signal subspace of $\overline{\mathsf{\Phi }}$ is determined by the the rank of $\mathbf{W}$, and the column vectors in ${\mathit{A}}_0\otimes {\mathit{B}}_0^{*}$. From (A5), $\mathrm{rank}\left(\mathbf{W}\right)=\mathrm{rank}\left({\mathbf{W}}_A\right)·\mathrm{rank}\left({\mathbf{W}}_B\right)$. In order to determine the rank of $\mathbf{W}$, the ranks of ${\mathbf{W}}_A$ and ${\mathbf{W}}_B$ are carefully discussed in the following two cases.

Case I: The DOAs are non-ambiguous on the individual sparse arrays, which means that the steering vectors on Array $\mathbb{A}$ ${\left\{{\mathit{a}}_0\left({\theta }_q\right)\right\}}_{q=1}^Q$ are distinct, and the steering vectors on Array $\mathbb{B}$ ${\left\{{\mathit{b}}_0\left({\theta }_q\right)\right\}}_{q=1}^Q$ are also distinct.

In (A6), since the elements of $\mathit{\eta }$ are non-zero, $\mathrm{diag}\left(\mathit{\eta }\right)$ is a full rank diagonal matrix. In the Vandermonde matrix ${\mathbf{V}}_A$, each row vector is the transposition of a steering vector of length $L_A-K_A+1$ and is distinct to one another. When $L_A-K_A+1\ge Q$, the Vandermonde matrix has full row rank. Hence, $\mathrm{rank}\left({\mathbf{W}}_A\right)=Q$. Similarly, $\mathrm{rank}\left({\mathbf{W}}_B\right)=Q$ when $L_B-K_B+1\ge Q$. $\mathbf{W}$ is henceforth a full rank matrix with rank $Q^2$.

In (A10), the vectors ${\{{\mathit{a}}_0\left({\theta }_q\right)\otimes {\mathit{b}}_0^{*}\left({\theta }_q\right)\}}_{q=1}^Q$ are Q columns in the matrix ${\mathit{A}}_0\otimes {\mathit{B}}_0^{*}$. When $\mathbf{W}{\mathbf{W}}^H$ has full rank, they are clearly in the signal subspace of $\overline{\mathsf{\Phi }}$.

Case II: The DOAs are ambiguous. For demonstrating purpose, we assume that two DOAs ${\theta }_1$ and ${\theta }_2$ satisfy ${\theta }_1={\theta }_2+2\pi m/M$ for a non-zero integer m. In this case, ${\mathit{a}}_0\left({\theta }_1\right)={\mathit{a}}_0\left({\theta }_2\right)$ on Array $\mathbb{A}$. We also assume that the other steering vectors ${\left\{{\mathit{a}}_0\left({\theta }_q\right)\right\}}_{q=3}^Q$ are distinct. In (A11), we can delete the repeated column vector ${\mathit{a}}_0\left({\theta }_2\right)$ in ${\mathit{A}}_0$ and the repeated row vector in ${\mathbf{V}}_A$, at the same time, combining the coefficients ${\eta }_1,{\eta }_2$ to obtain

$$\begin{array}{ll}{\mathit{A}}_0·\mathrm{diag}\left(\mathit{\eta }\right)·{\mathbf{V}}_A& =\left[{\mathit{a}}_0\left({\theta }_1\right),{\mathit{a}}_0\left({\theta }_3\right),\dots ,{\mathit{a}}_0\left({\theta }_Q\right)\right]\mathrm{diag}\left[{\eta }_1+{\eta }_2,{\eta }_3,\dots ,{\eta }_Q\right]\left[\begin{array}{ccc}{e}^{jM0{\theta }_1}& \dots & e^{jM(L_A-K_A){\theta }_1}\\ e^{jM0{\theta }_3}& \dots & e^{jM(L_A-K_A){\theta }_3}\\ ⋮& \ddots & ⋮\\ e^{jM0{\theta }_Q}& \dots & e^{jM(L_A-K_A){\theta }_Q}\end{array}\right]\\ & ={\tilde{\mathit{A}}}_0·\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)·{\tilde{\mathbf{V}}}_A.\end{array}$$ (A12)

In (A12), ${\tilde{\mathit{A}}}_0$ is constructed by deleting the column vector ${\mathit{a}}_0\left({\theta }_2\right)$ from ${\mathit{A}}_0$, the Vandermonde matrix ${\tilde{\mathbf{V}}}_A$ is deleting the row vector $\left[e^{jM0{\theta }_2},\dots ,e^{jM(L_A-K_A){\theta }_2}\right]$ from the Vandermonde matrix ${\mathbf{V}}_A$. The remaining row vectors are distinct. The diagonal matrix $\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)=\mathrm{diag}\left[{\eta }_1+{\eta }_2,{\eta }_3,\dots ,{\eta }_Q\right]$ is still full rank since ${\eta }_1+{\eta }_2\ne 0$ from the non-vanishing assumption. Hence, the matrix $\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)·{\tilde{\mathbf{V}}}_A$ has full rank.

Similarly, when the DOAs are ambiguous on Array $\mathbb{B}$, we can delete the repeated column vectors in ${\mathit{B}}_0$ to obtain a full rank matrix ${\tilde{\mathit{B}}}_0$, and delete the repeated row vectors in ${\mathbf{V}}_B$ to obtain a full rank matrix $\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)·{\tilde{\mathbf{V}}}_B$. Equation (A11) becomes

$$\begin{array}{c}\begin{array}{ll}\mathbf{Y}& =\left({\tilde{\mathit{A}}}_0·\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)·{\tilde{\mathbf{V}}}_A\right)\otimes \left({\tilde{\mathit{B}}}_0^{*}·\mathrm{diag}\left({\tilde{\mathit{\eta }}}^{*}\right)·{\tilde{\mathbf{V}}}_B^{*}\right)\\ & =\left({\tilde{\mathit{A}}}_0\otimes {\tilde{\mathit{B}}}_0^{*}\right)\left(\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)·{\tilde{\mathbf{V}}}_A\right)\otimes \left(\mathrm{diag}\left({\tilde{\mathit{\eta }}}^{*}\right)·{\tilde{\mathbf{V}}}_B^{*}\right).\end{array}\hfill \end{array}$$ (A13)

Analogous to Case I, the matrices $\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)·{\tilde{\mathbf{V}}}_A$ and $\mathrm{diag}\left(\tilde{\mathit{\eta }}\right)·{\tilde{\mathbf{V}}}_B$ are of full rank. Since ${\tilde{\mathit{A}}}_0$ and ${\tilde{\mathit{B}}}_0$ are removing only the repeated steering vectors, for any DOA $\theta$, the vector ${\mathit{a}}_0\left(\theta \right)\otimes {\mathit{b}}_0^{*}\left(\theta \right)$ is still in the matrix ${\tilde{\mathit{A}}}_0\otimes {\tilde{\mathit{B}}}_0^{*}$. Hence, the vectors ${\{{\mathit{a}}_0\left({\theta }_q\right)\otimes {\mathit{b}}_0\left({\theta }_q\right)\}}_{q=1}^Q$ are in the signal subspace of $\overline{\mathsf{\Phi }}$.

Author Contributions

Y. Hu, Y. Liu and X. Wang conceived and designed the schemes; Y. Hu and Y. Liu performed the experiments and analyzed the data; Y. Hu, Y. Liu and X. Wang wrote the paper.

Conflicts of Interest

The authors declare no conflict of interest.