Low complexity sphere decoding with probabilistic radius prediction and hybrid modulation for low latency wireless MIMO systems

Abstract

Sphere Decoding is an important technique for detecting signals in MIMO systems. It is renowned for its near-optimal performance but is restricted by high computational complexity. This work proposes an improved framework, integrating Gaussian Approximation based radius prediction with hybrid modulation to overcome this limitation. To balance performance and complexity, the adaptive Gaussian model dynamically estimates the decoding radius based on channel conditions. This reduces the number of visited nodes and enhances computational efficiency. Building on this, a hybrid modulation scheme integrating BPSK and 16-QAM is used to further enhance error performance by adjusting modulation levels to varying channel qualities. The proposed method is simulated on an 88 MIMO system using 16-QAM and hybrid modulation using BPSK and 16-QAM modulations. This Hybrid modulated MIMO system with Gaussian Approximation technique attains a drastic enhancement in system performance, reducing BER by up to 99.5%, and node complexity by 77.6% at 5 dB SNR compared to conventional Chi-Square approach. These results highlight the effectiveness of the two-tiered approach as a feasible, low-complexity, and low-latency solution for efficient detection, with promising applications in low-SNR and disaster-affected scenarios. Computationally efficient and low-energy algorithms are also crucial in 6G and IoT applications including massive machine-type communications and ultra-reliable low-latency communications which conserves battery power while supporting real-time functionality.

Introduction

Multiple input multiple output (MIMO) technology has a significant role in modern wireless communication systems by using multiple antennas to enhance data throughput and improve signal reliability. As wireless networks progress toward 5G and beyond, the importance of efficient MIMO detection becomes even more critical to meet the demands of high-speed, low-latency communication^1–5. However, detecting transmitted signals at the receiver becomes a difficult task, especially when higher-order modulation schemes like 16-QAM or 64-QAM are used³. The most reliable detection approach is Maximum Likelihood (ML) detection, but it tends posing significant computational burden in large MIMO systems⁶.

Sphere Decoding (SD) is a widely recognized technique that ensures near ML performance⁷ while achieving computational efficiency by employing a bounded search area, inside a sphere. Over time, several techniques have been developed to make Sphere Decoding faster and simpler, each facing its own set of challenges. The K-best algorithm, while reducing complexity, struggles with optimal path selection, especially in massive MIMO systems^8,9. Fixed-Complexity Sphere Decoding (FSD) offers reliability, but with compromised performance in comparison to full ML decoding^10,11. Lattice Reduction Aided Sphere Decoding (LR-SD)^12,13 helps to reduce the computational burden, but can result in suboptimal detection and demands specialized hardware for effective execution. While iterative decoding techniques have the potential to enhance performance, they continue to face challenges with complexity and delay in large-scale systems¹⁴. Configurable quasi-optimal Sphere Decoding balances performance and cost with adjustable complexity¹⁵, but it may reduce accuracy when complexity is adjusted, affecting performance in difficult MIMO scenarios. SVD-based Sphere Decoding enhances BER but imposes a significant computational overhead¹⁶. The regularized SD (RSD) technique¹⁷ and reordering schemes such as sorting QR decomposition and V-BLAST¹⁸ are two methods for improving computational efficiency but suffereing from lower detection accuracy.

Recent developments in machine learning-driven MIMO detection techniques, including deep unfolding¹⁹, neural-assisted sphere decoding, probabilistic graphical models²⁰, and methods based on variational inference²¹, have substantially improved the efficiency and scalability of MIMO systems. These approaches leverage deep learning and probabilistic methods to enhance decoding accuracy and minimize computational complexity, especially in high-order MIMO systems. By employing these strategies, MIMO detection can be enhanced to process more challenging environments and attain higher efficiency in high-dimensional systems. Finally, the inclusion of machine learning techniques demonstrate significant potential, although it encounters scalability challenges and demands significant training overhead^22,23. Consequently, every technique comes with its own difficulties related to performance, complexity, and scalability, which must be avoided for practical application in MIMO systems²⁴.

Thus Sphere Decoding in MIMO systems has made considerable progress over the years, but many obstacles still remain to be overcome. The primary concern is the computational complexity associated with the method, which increases dramatically when the number of antennas and the modulation level increase^25–27. Another crucial issue is the estimation of the accurate starting radius, since an inaccurate radius estimate can result in excessive computing load or poor bit error rate performance. The Sphere Decoding methods discussed above encounter challenges including complex processing, long execution time, and system limitations in handling large MIMO systems. Such approaches tend to overestimate the radius causing the decoder to check too many nodes. This results in slower processing and more power usage. These techniques struggle to adjust dynamically to changing channel conditions also. This work integrates Gaussian Approximation based Sphere Decoding with hybrid modulation to enhance the efficiency and accuracy of MIMO system detection. The Gaussian method^28,29 predicts the radius more precisely, reduces the number of checks, and keeps the performance close to that of maximum likelihood detection. This approach offers a more efficient and accurate way to manage computational complexity and optimize performance in MIMO systems. Gaussian Approximation method is adopted because it provides a good trade off between computational efficiency and performance, decreasing the number of nodes visited during decoding. Hybrid modulation is introduced to increase the overall efficiency and performance of the Sphere Decoding process. Hybrid modulation is chosen because it integrates the strengths of various modulation techniques such as BPSK and 16-QAM, making the system adaptable to diverse channel conditions. The proposed methods are evaluated on 88 MIMO systems. The simulation results show that Gaussian-based radius prediction considerably decreases the number of computations, while hybrid modulation demonstrates superior error performance. Together, these approaches constitute a practical and scalable solution for real-time MIMO detection.

The structure of the paper is as follows: Section 2 focuses on the Sphere Decoding algorithm, discussing its application in MIMO systems to enhance decoding accuracy. Section 3 presents the Gaussian approximation method and hybrid modulation in Sphere Decoding. Section 4 describes the system model, outlining the MIMO configuration and the modulation schemes employed. Finally, Section 5 provides simulation results and analysis of key performance parameters, and highlights its implications for future wireless communication systems.

Sphere decoding

By utilizing antenna diversity approaches, spatially multiplexed MIMO (SM-MIMO) systems may send information faster than MIMO systems. Take into account a MIMO configuration illustrated in Fig. 1, where stands for the matrix of channels and let be the gain in channel arising from the th receiving antenna and the th transmitting antenna in its th order. In this research, the phased scattering model is used to simulate multipath propagation. Phased scattering model describes the scattering due to environmental interferences creating several propagation channels with varying delays, phase shifts, and attenuation and is very much useful in MIMO modeling³⁰. Each of the multi path components contributes with a distinct amplitude and phase shift, and the received signal is represented as the sum of these components.

Fig. 1.

Sphere Decoding. a Principle Underlying Sphere Decoder. b Sample tree generated for a 4-dimensional case.

With regard to wireless communication systems with multiple antenna, information is conveyed among channels that are frequently represented as both time-invariant and linear. For such systems, the data symbols that were broadcast are combined linearly to produce the received signal, tainted by additive Gaussian noise given by

1

where s,x and v are given by

and the NtXNr channel matrix is given by

The model with real value equivalent can be expressed as

2

At the receiver, a detector estimates the signal that is being sent, and an ideal detector aims to reduce the average likelihood of an error. The maximum-likelihood design does this by carrying out the nonlinear optimisation^31–33.

3

where represents the -dimensional square lattice which is the complete collection of vector symbols that could be communicated. Although the symbol error rate of the ML detector is ideal⁶, implementing it by exhaustively searching over , can result in an unimaginably high computational complexity.

The solution to issue (3), is sometimes known as an integer least-squares problem, is straightforward geometric in nature. The challenge lies in determining integer values that reduce the total squared errors between the observed and predicted values³⁴. Sphere Decoding effectively resolves the Integer Least Squares (ILS) issue in MIMO communication systems to provide the best possible signal detection. Identifying the integer vector x that has the lowest Euclidean distance is the first step in the procedure. That is, for any matrix that defines a lattice spans a skewed lattice for any specified lattice-generating matrix ²⁴. Consequently, the objective is to identify the “nearest” lattice point. The solution of the integer least-squares equation to considering the skewed lattice and the vector as shown in Fig. 1a.

According to the aforementioned, every lattice point within a sphere with dimensions and radius d can be found by first locating every lattice point within a sphere having dimensions and a uniform distance d. A tree is fundamentally built by an algorithm that identifies the points on the lattice inside a sphere with a dimension of . Also the branches of -th level tree map to points on the lattice within a radius of d. and in dimension , as shown in Fig. 1b³⁵. We can now pinpoint the issue at hand more precisely as

4

To divide the problem into the sub problems outlined above, we can use matrix’s QR factorisation.

5

Here is an orthogonal matrix and is an is a matrix which is upper triangular. QR decomposition reduces computational complexity and improves numerical stability by breaking down the channel matrix. The total performance and throughput of contemporary wireless communication are enhanced by this application, which is crucial in enhancing the precision and dependability of signal demodulation in MIMO transmission. So the condition (4) can be rearranged as

6

where represents Hermitian matrix transpose. Or we can write

7

Defining and we can reformulate the above equation as

8

The above procedure can be formalized as an algorithm and can be described by Algorithm 1. Here m represents diamensional of signal space and l is the initial level of the lattice.

Algorithm 1.

Algorithm for Sphere Decoding in MIMO

Gaussian approximation method

In MIMO systems, optimizing decoding efficiency relies on accurately predicting the sphere’s radius,^26,27,36,37 which defines the decision zone around each received signal vector. The Chi-Square technique provides a simple way to estimate this radius by analyzing signal and noise dispersion, but further refinement is needed to improve BER. It computes the radius based on the chi-square distribution of the noise energy, thus ensuring simplicity, predictability, and channel independence. This technique is not highly effective in low SNR conditions. Because of its conservative characteristics, it often results in more node visits, but it serves as a foundational benchmark in the evaluation process for newer methods. Compared to the proposed Gaussian Approximation method, which adaptively tightens the radius to reduce complexity, the Chi-Square method provides a more conservative yet less efficient option making it a strong baseline for performance comparison. This approach optimizes the initial search radius and adjusts it dynamically during decoding, significantly reducing errors by enhancing the decoder’s ability to differentiate between signals and noise. This tighter radius enables early pruning by restricting the search to a smaller sphere, thereby reducing the number of nodes visited. By utilizing the Gaussian Approximation, Sphere Decoding can more accurately represent noise characteristics, allowing the decoding process to concentrate on signal points most likely to match the transmitted signal.

The Gaussian noise Approximation, using Additive White Gaussian Noise (AWGN), simplifies the noise model and its impact on signal processing, making it essential for improving Sphere Decoding in MIMO systems. This approach uses the mean and variance of noise to simplify metric computations, such as the noise-adjusted Euclidean distance, thereby significantly reducing the computational complexity of decoding while improving speed and accuracy. This targeted approach enhances BER and SER by improving symbol detection accuracy, while also considerably reducing the number of visited nodes during decoding, leading to more efficient and reliable signal recovery.

The Gaussian approximation is used in Sphere Decoding mainly to predict the preliminary search radius in a way that is efficient and statistically grounded. This enhancement leads to a reduction in computational complexity and an acceleration of decoding speed, all while maintaining a best performance on bit error rate performance. These are the Key Performance Indicators for the MIMO system. By leveraging Gaussian noise properties to determine noise power, the size of the search hypersphere is optimized, focusing on the most likely signal vectors and eliminating improbable ones, thereby simplifying the decoding process. This method also reduces computational load by pruning low-probability paths early in the search. The radius of the search sphere can be initially set based on the variance of the noise and adjusted during the search, commonly starting with a factor of the noise standard deviation determined by simulations or analytical limits. For a partial solution at depth p, the conditional Euclidean distance, based on the received vector and the partial channel matrix, can be used to decide whether to prune the branch. The equation is given by:

9

If this condition is not met, the branch can be pruned, as it is unlikely to lead to the minimum Euclidian distance. The likelihood of a candidate vector x under Gaussian noise for a complex model is given by the conditional probability.

10

This expression, while not directly to prune the search space, underlies the rationale for using Euclidian distance as the metric for Sphere Decoding.

Algorithm 2 describes the step by step procedure for determining BER, SER, Execution time and the Average Number of Visiting Nodes using Gaussian and Chi-Square methods.

Algorithm 2.

Algorithm for Obtaining BER, SER and Average Number of Visiting Nodes

System model

Figure 2 shows the system model of a typical MIMO communication system with improved Sphere Decoding for efficient decoding of transmitted signals. In this study number of transmitting antennas and number of receiving antennas are employed. The system begins with the distinct signals which are the transmitted symbols. The symbols are fed to the modulator, where hybrid modulation(a combination of BPSK and 16QAM )is utilized. This hybrid modulation technique reduces computational complexity and leads to a significant enhancement in system performance. Modulated signals are transmitted through transmitting antennas and are received in several receive antennas after traveling through a complicated channel with many paths representing the effects of fading, interference, and noise³⁸.

Fig. 2.

System model.

The received signals are decoded by the sphere decoder, which reconstructs the original transmitted symbols (). To enhance the accuracy of the Sphere Decoding process further, a Gaussian approximation is utilized for radius prediction, enhancing the decision boundary and minimizing computational complexity. The combination of hybrid modulation and Gaussian approximation for radius prediction yields a substantial decrease in computational complexity, while also enhancing detection accuracy. This improves both the efficiency and robustness of the system, especially in high-noise environments.

The Phased Scattering model was selected in preference to Rayleigh and Rician due to its superior ability to accurately depict actual MIMO operating conditions characterized by angular dispersion and phase correlation, both of which are critical in hybrid-modulated systems³⁰. In contrast to Rayleigh, which is based on non-line-of-sight conditions and Rician, which is based on the existence of a predominant line-of-sight trajectory, the Phased Scattering model effectively captures both multipath effects and phase variations thereby facilitating a more valid evaluation of antenna performance.

The Phased Scattering Model represents a MIMO channel as a sum of multiple scattered paths, each characterized by its own angle of departure and arrival, complex gain, and phase shift. It accounts for spatial correlation and angular dispersion, making it more accurate for realistic propagation environments.

The channel matrix is expressed as:

11

where represents the complex gain associated with the -th component of the multipath, and denote the response vectors of the array at the receiver and transmitter for the angles of arrival () and departure (), respectively, and denotes the overall count of the multipath components which denotes the unique signal pathways from the transmitter to the receiver. Each unique path is defined by its own gain and direction of arrival (AoA) and departure (AoD). These angles follow specific distribution models, such as uniform or Gaussian distributions, according to the physical surrounding of the communication system. The array response vectors and are calculated based on the antenna geometry. The channel model is normalized to account for the overall path loss and fading in the system. The gain accounts for the effects of fading and the power of each multipath component.

This model uses ULAs at both transmitter and receiver, using an inter-element spacing of with normalization in baseband . Each realization comprises L=10 independent paths, with AoD/AoA drawn i.i.d. The array responses are

12

13

Ensuring SNR is comparable across settings by performing Frobenius normalization on every realization.

14

resulting in and . These settings fully specify the geometric configuration, spacing, normalization and angle statistics for consistent, repeatable results.

In this study, an 8x8 MIMO system is used. The confidence level () selected for the radius prediction using Gaussian Approximation is 0.95, and the degrees of freedom are estimated as . The radius is then evaluated using the noise variance and other system parameters, ensuring an efficient search boundary in Sphere Decoding. The fading variance, which models multipath fading, is taken as 1, and the error threshold is set to 0.000001. The Signal-to-Noise Ratio is specified on a per-bit basis and Monte Carlo simulations are conducted with 100,000 frames to calculate the average bit error rate across different SNR values. The transmitted symbols are normalized so that the average power is unity. The phased scattering model is constructed using 10 scattering elements, where phase shifts are randomly assigned to each scatterer. The antenna configuration is simplified, assuming arbitrary configurations for both transmit and receive antennas. The simulation is executed for a fixed number of frames without a clear termination condition. Sphere decoding employs an exhaustive recursive search, without using a specialized enumeration technique such as Schnorrer-Euchner. The simulations were carried out using MATLAB R2024a with 100,000 Monte Carlo trials per SNR to ensure reliability. The signals, channels and noise of the transmission were drawn at random with the MATLAB pseudo random generator with a fixed seed to ensure reproducibility. This research also utilises a recursive brute-force search that examines every candidate state.

Effect of Gaussian approximation on key Performance Indicators (KPI)

BER and SER optimization

Through careful selection of the initial sphere radius d based on noise characteristics such as variance, the algorithm avoids exploring nodes that are unlikely to yield the optimal solution. Using Gaussian noise statistics, a tighter initial radius ensures that only the most probable subset of lattice points are considered. Enhanced by Gaussian noise approximation, dynamic radius adjustment, and probabilistic pruning improve both the efficiency and accuracy of Sphere Decoding in MIMO systems. This focused search not only reduces computational complexity but also improves detection accuracy, resulting in a lower Bit Error Rate. The radius of the sphere dynamically contracts during decoding, focusing more precisely the search on high-probability regions and reducing decoding errors across a wide range of SNR levels.

Gaussian approximation method also enhances the symbol error rate performance in SD by guiding the selection of an optimal decoding radius. If the radius is too small, the decoder may incorrectly filter out the correct transmitted vector, leading to symbol detection errors. In contrast, an overestimated radius increases complexity in decoding without contributing to SER improvement. By dynamically predicting a well-calibrated radius, the GA method guarantees that the correct symbol vector is contained within the search space while avoiding unnecessary node visits. As a result, SER is considerably reduced across varying noise and channel conditions.

Reduction in number of visiting nodes

By targeting the most promising regions of the solution space, Sphere Decoding with Gaussian noise approximation enhances efficiency by reducing the number of visited nodes. Initially, the sphere’s radius d can be set according to the variance in noise . The decision to visit a node depends on whether the squared Euclidean distance is within the sphere, i.e., less than d. To understand the reduction in visited nodes, consider the probability of the squared Euclidean distance being less than a certain threshold given the Gaussian noise distribution. This probability informs the likelihood of a candidate vector being the correct one.

For a given candidate , the squared distance normalized by noise variance, , obeys chi-squared distribution if Gaussian noise is present. Cumulative Distribution Function (CDF) of the chi-squared distribution offers a probabilistic basis for pruning by determining the likelihood that a random draw from the distribution is below a certain value³⁹. Without loss of generality, consider a simplification where the decision to prune or not is based on comparing against . The probability of a node being within the sphere is

15

where refers to the CDF of Chi-square distribution. Probability of this decreases as decreases, guiding the dynamic adjustment of based on the closest vectors found and effectively pruning the search space. Thus, Gaussian Approximation enables the decoding process to focus on signal points most likely to be the transmitted signal, further reducing complexity. The Gaussian Approximation facilitates precise estimation of the initial sphere radius and this more constrained radius permits the decoder to eliminate unfavorable symbol pathways at an earlier stage. Consequently, the quantity of nodes traversed is markedly diminished, thereby enhancing computational efficiency.

Reduction in computational complexity and execution time

The principal advantage of employing Gaussian Approximation in Sphere Decoding lies in the significant reduction in computational complexity, which directly contributes to faster execution. In conventional Sphere Decoding methodologies, the algorithm exhaustively evaluates the Euclidean distance for all possible transmitted symbol vectors, making the process computationally intensive as the MIMO system dimensions increase.

By contrast, Gaussian Approximation allows the decoder to focus on a reduced subset of likely candidates, avoiding unnecessary computations. Instead of searching over symbols, the time complexity of Sphere Decoding with Gaussian Approximation can be represented as , where , reflecting a substantial reduction. This reduction in complexity not only reduces the algorithm’s computational burden but also significantly decreases the overall decoding time.

Hybrid modulation

In traditional MIMO systems, standardized modulation techniques such as pure QAM or BPSK are generally utilized across all transmitting antennas to facilitate system design. Higher-order modulation schemes, such as 16-QAM or 64-QAM, provide enhanced data throughput, but exhibit an increased susceptibility to noise and interference. Furthermore, the adoption of uniform high-order modulation markedly enlarges the constellation size, consequently amplifying the computational complexity of detection algorithms, including Sphere Decoding. The sphere decoder tries different symbol combinations to find the one with the smallest Euclidean distance, but the number of possibilities grows very quickly as the modulation order and number of transmit antennas increase. This phenomenon results in increased decoding latency and computational demands, thereby complicating the feasibility of real-time implementation. Hybrid modulation schemes effectively address these challenges by integrating various modulation orders within a single MIMO framework. Each transmit antenna is assigned a specific modulation format based on its channel quality or the system’s performance requirements. For instance, while certain antennas may implement high-order QAM to optimize spectral efficiency, others may utilize lower-order schemes such as BPSK to enhance resistance against noise perturbations. From the standpoint of Sphere Decoding, hybrid modulation markedly diminishes the overall search space. Rather than engaging with uniformly high-order constellations, the decoder works through a mixed set of constellations, where lower-order modulated symbols simplify decoding and allow early pruning of the search tree. In this research, the BPSK-16 QAM combination is used as a hybrid modulation strategy.

In fifth-generation (5G) wireless communication systems, enabling multiple types of service requirements, ranging from ultra-reliable low-latency communications (URLLC) to enhanced mobile broadband (eMBB), requires the use of flexible modulation schemes. The hybrid modulation technique integrating BPSK and 16 QAM is especially significant in this regard. BPSK gives enhanced levels of consistency and strength necessary for important control signals and URLLC applications, where reducing error rates under challenging channel conditions is critical. On the other hand, 16 QAM provides superior spectral efficiency, which is necessary for eMBB services, and provides increased data throughput. Using these two modulation techniques together in a hybrid system enables 5G networks to select the best modulation depending on the state of the channel and type of service, enhancing both reliability and data rates.

Complexity of the algorithm is primarily influenced by the number of candidate symbol vectors evaluated in Sphere Decoding, the number of nodes visited in the search tree, and the decoding operations performed by the decoder. In a conventional MIMO system employing uniform modulation of order , the total number of possible transmitted symbol vectors is . Accordingly, the worst-case complexity of Sphere Decoding grows exponentially with :

In the proposed hybrid-modulated MIMO system, BPSK is assigned to antennas and 16-QAM to the remaining antennas. The worst-case complexity of Sphere Decoding in this setup is reduced to , compared to for full 16-QAM, enabling a significant reduction in computational load while preserving performance. Hybrid modulation offers benefits in Sphere Decoding that go beyond just reducing complexity. Simulation results consistently indicate that hybrid modulation frameworks yield better BER and SER in comparison to their uniform high-order counterparts, alongside a marked reduction in decoding complexity. Additionally, the average quantity of nodes visited within the Sphere Decoding tree is considerably decreased, thereby resulting in enhanced energy efficiency and greater applicability for practical real-time scenarios.

Algorithm 3 describes the step-by-step procedure for determining BER, SER, Execution time, and Average Number of Visited nodes using 16QAM and Hybrid modulation.

Algorithm 3.

Algorithm for Obtaining BER,SER,Execution time and Average Number of Visiting Nodes using 16QAM and Hybrid modulation

Impact of hybrid modulation on KPIs

Reduction in number of visiting nodes

In Sphere Decoding algorithms for MIMO systems, the number of nodes visited during the search process is directly influenced by the size and structure of the modulation constellation. When uniform high-order modulation schemes such as 16-QAM is employed across all transmit antennas, the search tree becomes extremely dense, as every transmit stream contributes to an exponentially growing number of candidate symbol combinations. Specifically, for a system with transmit antennas and modulation order M, the total number of possible transmitted symbol vectors is . Consequently, the sphere decoder must explore a large portion of this expanded search space, especially in scenarios with poor initial radius estimation or under noisy conditions, leading to a high number of node visits.

Hybrid modulation reduces complexity by strategically combining different modulation orders across the transmit streams. For instance, a subset of antennas may employ lower-order modulation schemes such as BPSK, while the remaining antennas utilize higher-order constellations like 16-QAM. This intentional reduction in modulation order for some streams effectively prunes the search tree, as lower-order constellations contribute fewer branches to the decoding process. Mathematically, if antennas use BPSK () and antennas use 16-QAM (), the total constellation size becomes:

16

Furthermore, the incorporation of hybrid modulation essentially enhances the accuracy of the preliminary radius estimation within the context of Sphere Decoding. Given that symbols with lower modulation orders exhibit superior noise resilience, their influence on the initial radius estimation is typically diminished and more precise. This more constrained initial radius enables the decoder to efficiently restrict its search domain, thereby eliminating unlikely candidate paths at an early stage of the traversal. Moreover, during the execution of the tree search, the branches corresponding to lower-order modulated streams can frequently be comprehensively examined or eliminated more rapidly due to their less complex decision boundaries, which accelerates the overall decoding procedure. In conclusion, hybrid modulation effectively diminishes the quantity of nodes accessed during Sphere Decoding and facilitating a more efficient pruning process within the search tree.

Improvement in BER and SER

In MIMO systems, BER is highly affected by channel conditions, modulation schemes, and the presence of noise or interference. Modulation techniques like 16-QAM or 64-QAM tend to experience increased BER at low to moderate SNR levels due to closely spaced constellation points, causing them to be more prone to errors. Techniques such as hybrid modulation can lead to a reduction in BER by adjusting the modulation scheme in response to channel variations and boosting detection accuracy. Specifically, lower-order modulations like BPSK, when integrated with higher-order modulations, enhance the reliability of the system through improved symbol differentiation and decreasing the likelihood of errors in noisy environments.

In the proposed hybrid modulated system the transmit antennas are allocated in equal proportion, with half configured for BPSK and the other half operate with 16-QAM. This antenna arrangement allows the system to benefit from the inherent reliability of BPSK while maintaining the modulation efficiency offered by 16QAM at elevated SNR levels. If and indicate the bit error rates of BPSK and 16-QAM then the overall bit error rate of the hybrid modulated system is given by:

17

Here, P(BPSK) and P(16QAM) represent the fraction of time/bits transmitted using each modulation type. This expression guarantees a fair performance comparison between modulation schemes and it remains unbiased by power disparities, since both constellations are maintained under uniform spectral power normalization. Under an Additive White Gaussian Channel(AWGN) environment for a particular SNR of BER of BPSK can be formulated as

18

and for 16QAM:

19

As BER exhibits a nonlinear relationship with SNR, even a small number of BPSK symbols leads to a significant reduction in the combined BER, particularly in mid-SNR ranges where 16QAM errors are higher. The weighted average is greatly affected by the lower probability of error of BPSK, and the Gaussian approximation method further lowers BER by avoiding unnecessary branches during sphere decoding, enhancing the efficiency of decoding.

In traditional MIMO systems that utilize uniform high-order modulation techniques, such as 16-QAM or 64-QAM, the symbol error rate also exhibits a pronounced rise under conditions characterized by moderate to low SNRs. This phenomenon is due to the dense symbol mappings inherent in high-order constellations, which contribute to the decrease in Euclidean distances among constellation points. As noise and channel impairments adversely affect the received signals, the probability of erroneous symbol detection increases due to the overlap of symbols within the constellation. Hybrid modulation addresses this issue by allocating varying modulation orders across MIMO transmission antennas. In this method, lower-order modulation schemes such as BPSK is utilized on a specific subset of transmit antennas, whereas higher-order modulation schemes, including 16-QAM, are designated for alternative streams in order to sustain throughput levels. The incorporation of lower-order modulated streams facilitates enhanced symbol separation within the constellation diagram, thereby rendering these streams inherently more resilient to noise and fading phenomena.

Reduction in execution time

In Sphere Decoding algorithms, the cumulative execution duration exhibits a direct correlation with the quantity of nodes traversed throughout the tree search procedure. Each node specifies either a partial or complete candidate solution, and the decoder allocates computational resources to assess the distance metrics and execute pruning decisions at each node. In uniform high-order modulation schemes, the dense constellation generates a significant amount of symbol candidates, thereby exponentially enlarging the search tree and augmenting the time necessary to investigate valid paths.

Hybrid modulation reduces execution time by markedly narrowing the effective search space. By designating lower-order modulation schemes, such as BPSK, to a subset of transmit antennas, the number of branches at each tier of the decoding tree decreases. Moreover, the implementation of hybrid modulation significantly enhances the accuracy of the preliminary radius estimation in Sphere Decoding. The utilization of lower-order streams, characterized by their superior adaptability to noise and increased symbol differentiation, facilitates the establishment of a more constrained initial search radius. This preliminary pruning process results in a diminished number of nodes that are investigated further within the tree structure. The combined effect of a diminished initial candidate pool and effective pruning mechanisms results in a substantial reduction of overall execution duration.

As the execution time is influenced by the hardware configuration and its implementation method, in this work complexity analysis is performed that is independent of hardware. In SD, complexity in computation is typically measured by the mean number of nodes visited during tree search.

Results and discussions

Comprehensive simulations were conducted to evaluate the efficiency of the proposed integration of Gaussian Approximation and hybrid modulation in MIMO Sphere Decoding, in terms of BER, SER, execution time, and complexity reduction. This work uses an 88 MIMO configuration and analyzes the performance impact of integrating Gaussian Approximation with hybrid modulation. In the hybrid modulation scheme, half of the transmit antennas use BPSK and the other half use 16-QAM to balance reliability and spectral efficiency. The Gaussian approximation improves performance by effectively predicting the Sphere Decoding radius, minimizing computational load, while preserving detection precision in the presence of noise. Hybrid modulation is utilized to optimize system performance by combining the advantages of BPSK and 16-QAM leading to enhanced overall efficiency and reliability. The results show the trade-off between performance (BER and SER) and computational efficiency (nodes visited and execution time) for Gaussian approximation and hybrid modulation, demonstrating its effectiveness in optimizing performance and reducing computational load in the 8x8 MIMO configuration. In this analysis confidence level () selected for the radius prediction is 0.95, and the degrees of freedom are estimated as . The fading variance, which models multipath fading, is taken as 1, and the error threshold is set to 0.000001.

This study first analyzes the performance of Gaussian Approximation in Sphere Decoding, and then evaluates the impact of a hybrid-modulated MIMO system employing Gaussian Approximation based Sphere Decoding on key performance indicators.

Analysis of KPIs with Gaussian approximation

Figure 3 shows the comparative performance of Gaussian Approximation and Chi-Square methods for radius prediction in SD across four different parameters.

Fig. 3.

Comparison of Performance Parameters of Gaussian Approximation method and Chi-Square method using 16 QAM. a BER vs SNR, b SER vs SNR, c Number of nodes visited vs SNR, and d Execution time vs SNR.

Figure 3a illustrates the BER performance of Sphere Decoding using different radius estimation methods. It presents a comparison of the K-Best SD approach, Fixed Sphere Decoding (FSD), the Gaussian-based method, the chi-square approach, the linear zero forcing method, and the linear minimum mean square error (MMSE) detectors. With an increase in SNR, the BER reduces for all these methods, which is expected. However, the Gaussian approximation method regularly shows lower BER across all SNR values, indicating enhanced decoding accuracy. This shows that Gaussian-based radius estimation gives better symbol detection and decreased error probability compared to the chi-square-based approach. At low SNRs, the BER exhibits a significant reduction, progressively decreasing as the SNR is varied from 0 to 20 dB. The BER is observed to be reduced by approximately 33. 77%, decreasing from 0.459968 (Chi square) to 0.30466 (Gaussian method), at 5dB, indicating a computationally more efficient alternative while maintaining comparable performance at higher SNRs. This enhancement is due to the Gaussian method’s capability to predict the initial search radius by modeling the noise-induced distance metric using statistical properties of Gaussian distribution. Thus, it adapts more accurately to varying channel conditions, ensuring that the correct decoded symbols are included within the search space while minimizing unwanted computations. However, ZF and MMSE perform worse, especially at lower SNRs, due to noise amplification and residual interference.

Figure 3b shows the variation of SER with respect to SNR values for an 8x8 MIMO system for different detection techniques. The studied detection methods cover K-Best SD approach, FSD, the Gaussian-based method, the chi-square approach, the linear zero forcing method, and the linear MMSE detectors. As shown in the figure, the Gaussian approximation methodology consistently yields a lower SER compared to other techniques, especially in the low to moderate SNR values. The SER is reduced by approximately 34.94%, decreasing from 0.6342(for Chi-Square) to 0.3823(Gaussian method) at 5 dB. This significant enhancement in performance indicates the efficacy of the Gaussian Approximation in deriving a more accurate decoding radius, thereby facilitating superior symbol detection. At increased SNR levels, both k best SD and Gaussian method techniques show analogous SER characteristics, converging due to the decreasing influence of noise at higher SNRs. The Gaussian method evaluates a tighter initial radius by adapting to the noise distribution, which enables the SD to focus on the most likely symbol vectors. Thus, the probability of wrong symbol detection is decreased, thereby reducing the SER.

Figure 3c shows the mean number of nodes visited against SNR values as part of the Sphere Decoding process in a 88 MIMO configuration using 16-QAM, analyzed under two discussed radius prediction methodologies. With an increase in SNR, both methodologies provide a consistent reduction in node visits, but in Gaussian method it is reduced, from 67(for Chi-Square method) to 17 at 5 dB, indicating reduced search complexity. This represents a reduction of approximately 74.6 %. The Gaussian Approximation method reduces the number of visiting nodes in Sphere Decoding by predicting a tighter initial radius, thereby narrowing the search space and lowering computational complexity. This marked reduction in complexity, especially at lower SNR levels, emphasizes its proficiency in lowering the decoding workload, consequently positioning it as a resource efficient and scalable approach for real-time MIMO decoding.

Figure 3d shows the execution time comparison between SD methods using Chi-Square and Gaussian Approximation techniques across different SNR levels. As the SNR increases, both methods exhibit a notable reduction in execution time due to lesser candidate nodes being explored during decoding. At lower SNRs, especially up to 5dB execution time shows a significant reduction, gradually reducing as the SNR is increased. However, the Gaussian Approximation method consistently outperforms the Chi-Square approach, especially at low SNR values where the computational complexity is highest. At 5 dB, the execution time is reduced by approximately 47.69%, from 536.18 s to 280.45 s. Gaussian radius prediction reduces this by effectively narrowing the initial search space, thereby avoiding unnecessary node visits. This highlights the efficiency of this approach in reducing decoding time while maintaining performance, demonstrating it a more suitable choice for real-time MIMO detection in noisy environments.

The listed microsecond values represent the time taken for each detection for each frame in an MIMO system employing sphere decoding, where the detection cycle encompasses signal preprocessing, initializing the radius, then enumerating the tree with pruning, and final hard decision on symbols. Times were measured with high-resolution timing functions, averaged across frames at each SNR level. The measurements reflect the overall computational workload that covers the detection of 16-QAM and the combined BPSK–16-QAM mode.

Analysis of KPIs for hybrid modulated MIMO system with Gaussian approximation

Figure 4 shows an overview of the performance metrics achieved with hybrid modulation using BPSK and 16QAM. Figure 4a shows the performance of BER as a function of SNR for an 8x8 MIMO system using two distinct modulation schemes: 16-QAM (represented by the blue line) and hybrid modulation (indicated by the red line). For a justifiable performance analysis, the comparison between the hybrid scheme and 16QAM was normalized via code rate adjustment to maintain equal spectral efficiency. The graph demonstrates that Hybrid modulation attains a markedly lower BER in comparison to 16-QAM across the entire spectrum of SNR values. As the SNR increases, both modulation schemes exhibit a decline in BER, signifying enhanced performance with elevated SNR levels. However, hybrid modulation consistently exceeds 16-QAM, particularly at lower SNR thresholds, emphasizing its superiority in minimizing errors and increasing reliability, particularly in adverse communication scenarios. In high-order QAM systems, the dense constellation structure leads to increased susceptibility to noise, especially at lower and medium SNR ranges. Even a slight portion of BPSK symbols in the hybrid modulation scheme considerably reduces the overall BER, due to BPSK’s high robustness against noise and fading. It also affects the weighted BER performance under similar SNR conditions. Additionally, the use of Gaussian based probabilistic radius prediction in sphere decoding significantly improves detection accuracy while reducing BER.

Fig. 4.

Comparison of Performance Parameters of the Proposed Hybrid Modulated System with 16 QAM System: a BER vs SNR, b SER vs SNR, c Number of nodes visited vs SNR, and d Execution time vs SNR.

Figure 4b illustrates the SER performance as a function of SNR for an 8x8 MIMO system utilizing two distinct modulation methodologies: basic 16-QAM (depicted by the blue line) and Hybrid modulation (comprising BPSK and 16-QAM) (illustrated by the red line). The graph shows that Hybrid modulation markedly surpasses the performance of basic 16-QAM at diminished SNRs, thereby yielding a reduced SER. As the SNR increases, both modulation techniques exhibit enhanced performance; however, the hybrid modulation consistently retains a pronounced advantage, especially in adverse low SNR scenarios. This underscores the resilience of hybrid modulation in improving system error rate performance, particularly in suboptimal channel conditions. Simple QAM show increased symbol error rates caused by the efficient arrangement of symbols within the constellation space, thereby boosting the probability of erroneous symbol interpretation in the presence of noise. The hybrid modulation framework, by changing the modulation order across various transmit streams, effectively reduces the average SER. Lower-order modulation streams play a critical role in this enhancement, as their stable constellation structure inherently decreases the undesirable impacts of channel imperfections. Thus, the hybrid modulation method achieves superior SER performance over a broad spectrum of SNR values in comparison to the conventional QAM system.

The reduction in the quantity of nodes traversed during the Sphere Decoding procedure is intrinsically linked to execution time and is shown in Fig. 4c. At low SNRs,there is a considerable reduction in the number of visiting nodes, demonstrating its significant role in reducing complexity. As the SNR increases, the reduction in the number of visiting nodes becomes less noticeable. Uniform Quadrature Amplitude Modulation systems, attributable to their expansive and densely configured constellation architectures, yield a search tree that is both intricate and densely populated. Hybrid modulation methodologies effectively induce sparsity within the search tree by integrating lower-order modulation streams, which contribute a reduced number of branches at each level. This structural optimization enables the decoder to eliminate improbable paths at earlier stages of the search procedure, consequently decreasing the overall count of nodes that are visited.

One of the greatest advantage of hybrid modulation is the reduction in execution time throughout the process of Sphere Decoding and is presented in Fig. 4d. At low SNRs, there is a substantial reduction in execution time, resulting in a significant decrease in computational time. In uniform high-order modulation frameworks, the decoder is needed to navigate a large search space, leading to a elevated computational cost and increased decoding latency. Hybrid modulation efficiently reduces the constellation size by integrating both lower and higher order streams, reducing the number of candidate symbol vectors for the decoder to assess. In addition, improved initial radius estimation, boosted by the resilience of lower-order streams, facilitates early pruning of the decoding tree. Collectively, these factors significantly reduce the mean decoding time, making hybrid modulation more suitable for real-time and low-latency applications.

The performance parameters for three different Sphere Decoding techniques at an SNR of 5 dB–(1) SD using the chi-square method, (2) SD using the Gaussian Approximation method, and (3) SD using Gaussian Approximation followed by hybrid modulation–are shown in Table 1. Four important parameters are analyzed: BER, SER, execution time and number of node visits.

Table 1.

Comparison of performance parameters for different methods at 5 dB SNR.

Performance Parameter	SD using Chi-Square Method	Gaussian Approx Method	GA+Hybrid Modulation
BER	0.459968	0.30466	0.002234
SER	0.6342	0.3823	0.2452
Execution Time(in microsec)	536.18	280.45	185.70
No of Visiting Nodes	67	17	15

A gradual enhancement is reflected in all aspects of all performance metrics as the process moves from the Chi-Square method to Gaussian Approximation and then to hybrid modulated MIMO system with Gaussian Approximation based SD. The performance measures shows a clear enhancement across all parameters during the transition from the Chi-Square method to the Gaussian Approximation method, and further to the GA combined with Hybrid Modulation. Particularly, the Bit Error Rate is reduced from 0.45996 to 0.30466 with GA (an 33.7% reduction), and drastically to 0.002234 with hybrid modulated MIMO system with Gaussian Approximation based SD (a 99.5% reduction). Similarly, the Symbol Error Rate reduces from 0.6342 to 0.3823 using GA, and again to 0.2452 with hybrid modulated MIMO system with Gaussian Approximation based SD, showing reductions of 39.72% and 61.31%, respectively. Execution Time also shows significant improvement, dropping from 536.18 ms to 280.45 ms with GA (47.7% reduction), and further to 185.70 ms with hybrid modulated MIMO system with Gaussian Approximation based SD (65.4% reduction). Finally, the Number of Visiting Nodes decreases from 67 to 17 with GA (a 74.6% reduction), and then to just 15 with hybrid modulated MIMO system with Gaussian Approximation based SD (77.6% reduction). These findings demonstrate the capability of using Gaussian Approximation and Hybrid Modulation in improving decoding performance in terms of accuracy, efficiency, and computational complexity making them suitable for low-latency applications.

Conclusion

In this research, a two-tier enhancement to conventional Sphere Decoding was developed, integrating both Gaussian Approximation based radius prediction and the hybrid modulation scheme. It is observed that dynamic radius adjustment using Gaussian Approximation considerably decreased decoding complexity, while hybrid modulation based on channel information improved BER and SER performance. For an 88 MIMO system with Gaussian Approximation based SD and hybrid modulated MIMO system with Gaussian Approximation based SD produced noticeable gains in accuracy, efficiency, and computational cost. These results show that the proposed method provides a reliable and scalable solution for practical MIMO detection. It is suitable for real-time systems with high data rate requirements, low-latency applications in advanced wireless networks such as 5G and beyond, and low-SNR conditions commonly found in disaster-affected areas. Furthermore, the simple architecture and power-efficient design make it suitable for IoT devices and applications in smart cities. Building on this, further research can be extended to higher-order MIMO configurations and modulation schemes, enabling even more robust and efficient solutions for future wireless communication systems. The introduced Gaussian-supported Sphere Decoding with hybrid modulation, while achieving BER gains for 88 MIMO, is limited by higher computational complexity, scalability issues to massive MIMO, and dependence on precise channel estimation.

Author contributions

Girija M G. conceived the study, designed and implemented the simulation framework, conducted all experiments, and analyzed the results. Sudha T. supervised the work and reviewed the manuscript.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Data availability

No data were generated or analysed during this study.

Code availability

The complete implementation code used to generate the results in this research is available in Zenodo with DOI: https://doi.org/10.5281/zenodo.18656698.

Declarations

Competing interests

The authors declare no competing interests.