Safe image transmission with hybrid quantum key-genetic encryption via spatial modulation assisted NOMA

Abstract

This manuscript presents a secure and spectrally efficient Spatial Modulation (SM)-based Non-Orthogonal Multiple Access (NOMA) system for two-user downlink 5G wireless communication. The proposed SM-NOMA shows significant improvements in spectral efficiency and energy savings by activating only one transmit antenna per symbol duration and all other remaining antennas remain silent, reducing hardware complexity and RF chain usage compared to traditional Multiple Input-Multiple Output (MIMO) systems. In terms of detection strategy, the weak user employs Maximum Ratio Combining (MRC) to directly decode its signal, as it receives higher power and does not need interference cancellation. Conversely, the strong user first applies MRC to detect the weak user’s signal, performs Successive Interference Cancellation (SIC) to remove the weak user’s contribution, and then applies MRC again to decode its own signal. This approach balances complexity and performance, offering practical implementation benefits. Additionally, the system includes optimized power allocation to manage interference and maintain user fairness. The proposed SM-NOMA system performance is evaluated using Bit Error Rate (BER), outage probability, and the spectral efficiency under imperfect CSI and imperfect SIC conditions. SM-NOMA outperforms conventional NOMA by significantly reducing BER, lowering outage probability and enhancing spectral efficiency. At a transmit power of 25 dBm, in the case of User 1 the BER is improved from 10⁻² (NOMA) to 10⁻³ (SM-NOMA). Similarly, for User 2, there is an improvement in BER from 10⁻² (NOMA) to 10⁻⁴ (SM-NOMA). Also, the outage probability values for both the users improved significantly with User 1 dropping from the previous value of 10⁻¹ (NOMA) to 10⁻⁵ (SM-NOMA) at 20 dBm transmit power and for User 2 it drops from 10⁻⁴ to 8.39233 × 10⁻⁵. The overall spectral efficiency improved from 2.57 bps/Hz (NOMA) to 2.74 bps/Hz (SM-NOMA). Wireless systems are easily prone to attacks by intruders. To enhance transmission security, the system integrates Quantum-Enhanced Genetic Encryption (QGE²) a hybrid encryption framework combining quantum gate-generated keys and genetic algorithm-based encryption. The sequence generated using quantum gates is used as a seed value for the genetic algorithm. It is used to encrypt a 256 × 256 8-bit grayscale image. The randomness of the proposed quantum key is tested through the NIST test. The proposed encryption scheme passed all 17 NIST randomness evaluations (15 core + 2 subsections) with p-values ≥ 0.01. Additional metrics such as correlation, entropy, and histogram analysis validate the strength of the encryption against both classical and quantum threats. The integration of spatial modulation NOMA with QGE² ensures the safe and secure transfer of information through the Rayleigh faded wireless channel.

Introduction

Fifth-generation (5G) wireless systems are expected to provide new types of enhanced user connectivity services such as huge data rates, enhanced capacity, improved security, higher reliability, lower latency, increased quality of service and Spectral Efficiency (SE)¹. Unlike previous generations, which utilized orthogonal multiple access (OMA) technique to eliminate inter-user interference, the new generations of wireless networks are striving for improved spectral efficiency. 1G used Frequency Division Multiple Access (FDMA) in which the same channel is shared by many users, 2G used Time Division Multiple Access (TDMA) where the users are allocated the bandwidth based on time slots, 3G used Code Division Multiple Access (CDMA) where the data is covered with the Pseudo noise sequence, and 4G used Orthogonal Frequency Division Multiple Access (OFDMA) where the high data rate symbols is divided into low data rate symbols. All of these OMA techniques in a network assign different time slots, frequency bands or codes to users keeping them orthogonal to avoid collisions². As far as 5G is considered one of the prominent solutions is Non-Orthogonal Multiple Access (NOMA) that allows numerous users to share the same time-frequency resource block improving system capacity, spectrum efficiency and meeting huge connection requirements^3–5. MIMO (Multiple-Input Multiple-Output) technology stands as a breakthrough innovation in wireless communications, enabling more efficient and higher-capacity data transmissions⁶. MIMO systems⁷ use multiple antennas at the transmitter and receiver to achieve substantial spectrum gain, making it a key application of emerging technologies. However, issues like various antenna designs and high route loss make it impossible to assume perfect channel state information (CSI)⁸ and in a typical cellular network baseband processor, radio-frequency (RF) chains and transmit antennas use energy mostly from power amplifiers in each RF chain. As a result, conventional MIMO methods require the same number of RF chains as transmit antennas face challenges due to the consumption of energy⁹.

Conventional modes of MIMO such as V-BLAST and OSTBC, have a number of shortcomings. These types of MIMO require that multiple RF chains be active at the same time which adds complexity, power consumption and hardware requirements. These systems create significant inter-antenna interference when transmitting from multiple antennas, which interferes with other transmitted signals from multiple RF chains. This limitation becomes particularly strong in use cases with unbalanced antennas where the number of Tx antennas exceeds the number of Rx antennas for downlink transmissions. This limits the scalability and efficiency of conventional MIMO systems which render them inappropriate for next-generation wireless systems such as 5G.

To overcome these challenges, a unique expansion of the multiple-input multiple output (MIMO) technique is proposed, which is Spatial Modulation (SM) that has gained significant attention in academia and industry.

In spatial modulation, a single transmitting antenna is engaged for each symbol transmission and information is provided by both active antenna indices and transmitted amplitude-phase modulation symbols¹⁰. It reduces the hardware cost and complexity¹¹. Compared to traditional MIMO techniques such as point-to-point SM requires only one RF chain. SM can drastically reduce design and implementation complexity while improving spectrum, energy efficiency, reliability and greater resistance to inter channel interference¹². This approach simplifies and lowers the cost of multiple-antenna systems while maintaining high data rates. Through SM the transceiver design achieves both low complexity and good spectrum efficiency through simple modulation and coding processes¹³. Additional benefits include improved energy efficiency, reduced detection complexity with fewer receive antennas and RF circuits also eliminated inter-channel interference and no need for inter-antenna synchronization and MIMO compatibility¹⁴.

The NOMA principle involves superposition coding (SC) at the transmitter, followed by successive interference cancellation (SIC) at the receiver and to recover the original signal. Furthermore, the NOMA user may employ many power domain resources at the same time, frequency or code domain. As the number of users increases the system gets more sophisticated resulting in lower decoding success rates. To address this issue spatial modulation (SM) in MIMO systems was integrated with NOMA. This reduces system complexity and prevents interference between antennas. Thus SM-NOMA outperforms traditional orthogonal multiple access schemes and also the conventional NOMA providing greater spectral efficiency.

Encryption prevents unauthorised access and protects sensitive data. Encryption has evolved significantly from traditional approaches to cutting-edge solutions that use current computation. Advancements in information security are crucial for sustaining trust and integrity in an interconnected society¹⁵. The broadcast aspect of wireless transmission and successive interference cancellation (SIC)-based decoding at receivers provides important security challenges for NOMA, particularly addressing user privacy. Especially the NOMA systems users farther from the transmitter are in a greater level of risk of being intercepted, especially if an eavesdropper is located closer to the transmitter, thus reducing the secrecy capacity for the users further away. Multi-antenna schemes, cooperative relaying and machine learning based intelligent resource allocation schemes are standard mechanisms in diminishing those risks¹⁶. Physical layer security (PLS) is a popular method for ensuring secure wireless communications¹⁷. NOMA requires safeguarding confidential messages against unauthorised users¹⁸.

Encryption is the process of transforming plaintext into ciphertext, rendering information unreadable to unauthorised individuals. Encryption algorithms transform data from plaintext to ciphertext. There are two types of encryptions named symmetric (private key) and asymmetric (public key) encryption. Traditional encryption techniques such as DES, 3DES, AES and RSA are effective for general data but are unsuitable for images due to high redundancy, big data size and strong pixel correlation. To solve these issues, researchers investigated chaos-based image encryption. This method generates secret keys using chaotic maps, which take the advantage of traits including sensitivity to initial conditions, pseudo-randomness, ergodicity, and non-periodicity. These properties make chaos-based image processing algorithms faster, more secure, and more efficient than previous methods. The challenge is Chaotic map-based encryption exhibits extreme sensitivity to initial parameters and improper selection of these values may lead to weak encryption, increasing the risk of key exposure and unauthorized data access. To address this vulnerability, researchers have adopted metaheuristic optimization methods in image encryption to fine-tune the initial conditions and enhance overall security. From the survey, it is understood that metaheuristic methods are employed to optimize initial parameters, which prevents weak encryption outcomes¹⁹. A novel beamforming strategy based on a hybrid Grey Wolf Optimizer (GWO) combined with L2-norm is proposed in²⁰. In addition, when compared with the traditional encryption methods rapid advancement of quantum computing offers a new area of research called Quantum encryption.

Quantum computers use unitary gates to operate qubits (nlike classical computers that use logic gates such as AND, NOT, OR, and XOR (IBM, 2024). This innovation utilizes principles such as superposition and entanglement to provide computing capabilities considerably beyond those of traditional computers. QKD allows parties to exchange keys securely and detect eavesdropping. Rather than processing bits sequentially, quantum computers use qubits, which can be in several states simultaneously due to superposition (IBM, 2024). Increasing the number of key bits in a QKD system increases the possibility of detecting an eavesdropper. Thus, to improve security, higher key sizes are necessary²¹.

Traditional methods generate Pseudo-Random Number Generators (PRNGs) that operate using a defined series of mathematical operations to create sequences that resemble randomness. Despite their seemingly random output, these generators follow a fixed logic, making them predictable when their internal configuration is exposed. They require a starting input known as a seed value which plays a crucial role in determining the sequence they produce. This reliance on a seed limits their ability to generate truly random values. On the other hand, Quantum Random Number Generators (QRNGs) derive unpredictability from quantum mechanics. Measurement outcomes are inherently uncertain, as the system collapses unpredictably into one of its eigenstates, producing non-deterministic results²².

Thus, Quantum computing is expected to disrupt present encryption technologies over the next 20 years requiring immediate preparation. Quantum algorithms like Shor’s and Grover’s could break the encryption methods putting secure communications at risk^23,24.

Motivated by the existing surveys, the proposed work uses quantum-generated key that is used as a seed for encryption. 128-bit quantum-generated key is used, which is the same length as the standard encryption to have a baseline level of security for protecting sensitive information across sectors²⁵.

The proposed NOMA-SM system superimposes signals from multiple users in the power domain and transmits them over shared time-frequency and spatial resources, improving spectral efficiency. At the receiver, a Maximum Ratio Combining (MRC) technique is employed to combine the received signals across multiple antennas enhancing signal strength before applying Successive Interference Cancellation (SIC). The model accounts for imperfect SIC and imperfect CSI, acknowledging real-world limitations such as residual interference and channel estimation inaccuracies.

The simulation results confirm that the proposed secure NOMA-SM system with MRC-SIC detection offers superior performance in terms of spectral efficiency, error rates and security robustness compared to conventional NOMA even under imperfect conditions.

Related works

Reference²⁶ introduced NOMA-GSM to enhance NOMA-SM by selecting a transmit antenna combination (TAC) instead of a single antenna. Novel user grouping and power allocation methods have also been proposed, offering a favourable trade-off between performance and complexity. When compared with²⁶, which employed NOMA-GSM and achieved a minimum BER of 10⁻⁵ at SNR of 20 dB through more than one transmits antenna selection and dynamic user grouping and power allocation schemes, the proposed system has yielded almost the same BER at transmit power of 30 dBm with simple SM-NOMA under imperfect CSI and imperfect SIC. This shows that the proposed system achieves performance close to NOMA-GSM with considerable reduction in system complexity. In addition, the inclusion of quantum-generated keys as the part of the security mechanism offers an additional layer of security for communications when the channels provided by the NOMA systems are practical and real-world. In²⁷ a theoretical analysis of CRS-SM-NOMA system was made, deriving closed-form BER and rate expressions for both users using M-array PAM/QAM. The study extends existing CRS-NOMA models by incorporating multiple receive antennas and provides a more general BEP analysis considering superposition coding and SIC. All the simulations done in²⁷ assume Rayleigh flat-faded channels perfectly known to users A and B. The proposed SM-NOMA system is designed under imperfect CSI and imperfect SIC, achieves Bit Error Rate (BER) of roughly 10^−4.7 at high transmit power of 30 dBm which is close to CRS-SM-NOMA BER of 10⁻⁶ at high SNR of 27 dB. Additionally, the proposed SM-NOMA integrates quantum-enhanced encryption enhancing overall communication security something that is lacking in²⁷ making it more robust and practical for real-life 5G settings. In²⁸ Spatial Modulation (SM), which is an emerging MIMO technique where only selected antennas transmit, reducing complexity and inter-antenna interference, has been implemented. In²⁹ the error performance of MRC detection in Spatial Modulation systems is analysed and it is compared with optimal ML detection. In³⁰ Visible-Light Communication (VLC) with intensity modulation and direct detection offers high-speed data for 5G/6G indoor networks. This paper introduces Pulse Amplitude Width Modulation (PAWM), a hybrid scheme that lowers bit error rates and boosts data rates up to 210.8 Mb/s. Future integration with Non-Orthogonal Multiple Access (NOMA) can further improve multi-user capacity and spectral efficiency. In³¹ a Wavelet Packet Transform-based NOMA (WPT-NOMA) with a Complex-Valued Convolutional Neural Network (CVNN)–based Sequential Interference Cancellation (SIC) receiver is proposed, assuming perfect Channel State Information (CSI) at the Base Station (BS) and users. CNN and CVNN SIC receivers are trained on simulated data to enhance detection across varying Signal-to-Noise Ratios (SNRs), outperforming conventional NOMA schemes. In³² Quantum Permutation Pad (QPP) using IBM’s Qiskit framework is implemented on real quantum hardware. The scheme employs 28 two-qubit permutation gates to provide 128-bit entropy. Plaintext is split into 2-bit blocks then encrypted individually via dynamically generated quantum circuits. Thus, the proposed work demonstrates real quantum encryption implementation IBM Qiskit, showing feasibility of dynamic quantum circuits for secure classical communication. This supports proposed hybrid encryption architecture. In³³ Quantum logic gates are used. These gates manipulate qubits to perform computations, whether individually (single-qubit) or jointly (multi-qubit). Recent interest in quantum-based randomness has grown across fields like cryptography and physics. Thus overall, the paper reviews quantum gates (single and multi-qubit) foundational to quantum logic operations crucial for building secure quantum circuits using superposition and entanglement principle using quantum gates like Hadamard gate, Pauli gates, CNOT gate. Ref³⁴ work proposes a quantum random number generator using SX and CNOT gates on IBM’s quantum platform. By applying SX gates to all qubits and measuring the outcomes, a large sample of random bits were generated. Recent research has shown that quantum systems can benefit not only from superposition of states but also from superpositions of gate orders. By using a control qubit to determine the order of quantum operations, tasks such as identifying gate commutation can be performed more efficiently than with traditional fixed-order circuits. Thus, the quantum random number generator with proven NIST randomness validates the use of quantum randomness in QGE² for secure key generation. In³⁵ experimental demonstrations in photonic systems highlighted the potential of this approach for enhancing quantum algorithm performance. The work proves that quantum systems can benefit not only from superposition of states but also from superpositions of gate orders. By using a control qubit to determine the order of quantum operations, tasks such as identifying gate commutation can be performed more efficiently than with traditional fixed-order circuits. In³⁶ quantum logic gates are used which originated from reversible Boolean gates, with the quantum XOR (controlled NOT) gate emerging as a key two-bit gate for quantum computing. This gate is essential for quantum error correction networks. Thus, the paper discusses the origin and role of the CNOT gate essential for constructing quantum circuit for generating true random key. In³⁷ The Hadamard gate, though simple in appearance, plays a crucial role in quantum computing has been implemented. When applied to n qubits in the state |0>, it generates a superposition of all integers from 0 to 2^{n − 1}, enabling the fundamental properties of quantum systems. The paper highlights Hadamard gate’s role and its fundamental properties in quantum systems which is used in the proposed encryption system for the generation of key. In³⁸ chaos-based encryption strengthens data protection by utilizing the sensitivity and randomness of chaotic maps, making it particularly suitable for integration with NOMA systems. In³⁹ a secure and privacy-preserving NOMA-based visible light communication (VLC) framework is proposed, where OFDM modulation was combined with a two-level chaotic encryption scheme. Notably, it represents the first attempt to jointly enhance both security and privacy in NOMA-VLC systems. In⁴⁰ an OFDM-PD-NOMA-based physical layer encryption using a 3D chaotic model for dynamic key hiding and multi-dimensional signal encryption scheme is proposed. The scheme securely transmits encrypted data and keys with high reliability, achieving near-zero BER and a key space of 10⁴⁶² over multi-core fiber, demonstrating strong potential for future optical communication systems. In⁴¹ Quantum key distribution (QKD) enables secure key generation between users, which is combined with NOMA-based wireless communication to enhance security. A low-complexity iterative algorithm was proposed to maximize the minimum secure rate, demonstrating improved fairness and system performance. In⁴² Quantum wireless communication (QWC) is attracting attention for secure key transmission in free-space channels. A multi-user QKD system combined with NOMA over Poisson atmospheric channels was proposed, employing a parallel interference cancellation (PIC) approach to mitigate multi-user interference.

The key features of the existing literature and the limitations found in the existing literature are tabulated in Table 1. From this, the research gap is identified, and the proposed work is carried out in response to the identified research gap.

Table 1.

Research gap in existing works.

Ref	Key features	Constraints	Gap
7	NOMA with Precoded Spatial Modulation (PSM) with low-complexity transceiver	Complexity with precoding and in this system need accurate CSI	The Bit Error Rate (BER) performance is moderate No security and application-layer protection
8	SMBM-NOMA supports SIC and non-SIC detection under imperfect CSI	BER performance degrades as CSI becomes imperfect	Requires trade-off between imperfect SIC and overall system performance Lacks encryption
9	NOMA combined with spatial modulation (SM) improves spectral efficiency and allows multiple users to share transmit antennas	The analysis assumes perfect SIC	Performance under imperfect SIC is not evaluated Security aspects are not considered
25	GSM with multiple TA selection and dynamic user grouping	Increased number of activating antennas complicate hardware architecture	Even though the GSM added, the performance like BER is still moreover near to conventional SM-NOMA in a maximum 1 order of magnitude and Lack of security
26	Cooperative relaying with SM-NOMA and closed form BER deductions Relays and ML detection	High reliance on relays and ML detection; higher latency	To be analysed in both perfect and imperfect CSI as well SIC
30	Implements QPP using IBM Qiskit with 28 two-qubit permutation gates for 128-bit entropy for encryption	Applies classical channels for qubit transmissions; limited statistical attack resistance	Current version does not have improved randomization and full quantum-secure communication scalability
31	Describes quantum gates like H, Y, and CNOT, essential for logic operations and quantum computation	Limited actual implementations; quantum decoherence and energy inefficiency are critical concerns	Lack of system level integration
32	Generates high entropy random bitstring implementation using SX and CNOT gates and passes NIST random tests	Generates random bits only; cannot do encryption or communication	Lacks systematic integration of secure key distribution systems
38	Chaotic downlink NOMA (C-DL-NOMA) using hybrid chaotic maps (logistic, Hénon, Arnold’s cat, hyperchaotic DNA)	Depends on fixed power-allocation control and perfect SIC demodulation	While enhancing security and BER, system performance is still dependent on channel conditions and assumes perfect SIC
39	Secure and private NOMA-VLC with OFDM and two-level chaotic encryption: (1) chaotic constellation transformation (CCT), (2) chaotic subcarrier scrambling (CSS). Ensures both user security and inter-user privacy. Experimentally demonstrated with two users + one eavesdroppers	Expanding the NOMA-VLC system to accommodate additional users is primarily constrained by the need for low-complexity yet effective user pairing/grouping and power allocation mechanisms, which are not addressed in this study	May be vulnerable due to sensitivity to initial conditions and imperfect SIC
40	Multi-dimensional disturbance secure system with dynamic key hiding transmission using OFDM-PD-NOMA and 3D chaotic model. Achieves multi-point mutual-coupling, key masking, bit/symbol/carrier-level encryption	In this work, perfect SIC is considered at the receiver side for the sake of simplicity, while the imperfect SIC case needs further investigation	May be vulnerable due to sensitivity to initial conditions and imperfect SIC
41	Quantum-assisted NOMA downlink system where QKD is used to generate secure keys before data transmission, with a low-complexity iterative algorithm designed to maximize the minimum secure rate among users	For the considered system, perfect SIC is assumed, while the imperfect SIC case will be left for future work	Further analysis is required to evaluate secure rate performance under imperfect SIC conditions
42	Multi-user QKD-NOMA system over Poisson atmospheric channels with a parallel interference cancellation (PIC) receiver and nonlinear Poisson shot-noise limited interference cancellation algorithm	QBER performance deteriorates with increasing atmospheric attenuation and polarization mismatch	System reliability reduces under severe atmospheric conditions; performance at longer distances remains a challenge

State of the art

NOMA combined with spatial modulation (SM) has been studied to improve spectral efficiency, throughput, and multi-user performance^7–9,25,26. These studies typically assume perfect channel state information (CSI) and ideal successive interference cancellation (SIC), focusing primarily on communication performance while largely neglecting data security.

Chaos- and quantum-based encryption methods^{30–32,38–42} provide high randomness and theoretical resistance to classical and quantum attacks. However, these approaches are rarely integrated with SM-NOMA under practical wireless conditions, and many assume perfect SIC or ideal channels. Limitations include sensitivity to initial parameters, lack of systematic end-to-end encryption, and limited evaluation under imperfect SIC and CSI.

The works summarized in Table 1 illustrate these limitations, highlighting the need for a comprehensive framework that combines secure, high-performance communication with robust encryption.

Key contributions:

• The proposed method combined quantum gate-based seed values (validated via NIST randomness tests) with genetic encryption, providing strong protection against both classical and quantum attacks.

• Integration with spatial modulation-based NOMA and quantum encryption provides a complete 5G solution capable of achieving competitive BER and spectral efficiency under realistic network conditions, ensuring end-to-end security.

• The proposed work also benchmarked against existing chaos-, quantum-, and metaheuristic-based encryption schemes using metrics such as entropy, NPCR, UACI, and PSNR, demonstrating superior performance as highlighted in the literature summarized in Table 1.

System model

A downlink two-user NOMA, spatial modulation, MIMO with imperfect SIC is considered in this manuscript. The base station (BS) is equipped with M_t = 4 transmit antennas and each user has M_r = 4 receive antennas. The BS simultaneously transmits the encrypted image to two users using spatial modulation over a shared wireless channel. The image is encrypted using genetic algorithm for which the seed value is generated using quantum technique.

Transceiver of SM-NOMA

The transceiver architecture of the SM-NOMA downlink system with integrated security is depicted in Fig. 1a, and the corresponding simulation parameters are provided in Table 2.

Fig. 1.

(a) Transceiver model of the QGE²-enhanced downlink SIC-based SM-NOMA system with imperfect SIC: Encrypted data enters NOMA superposition enhanced by Spatial Modulation propagates through a Rayleigh fading channel and is decoded via MRC-based spatial demodulation and imperfect SIC. (b) Block diagram of 4-QAM Spatial Modulation system, where each transmission carries four bits: two bits select the active antenna index, and two bits are mapped to the 4-QAM symbol.

Table 2.

System simulation parameters.

Parameter	Value
Number of transmit antennas (Mt)	4
Number of receive antennas (Mr)	4
Modulation scheme	4-QAM
Image size	256 × 256 pixels
Path loss exponent (η\eta)	3
Distance of user 1 (d1)	1900 m
Distance of user 2 (d2)	1000 m
Optimized power allocation factors	α1 = 0.6731, α2​=0.3269
SIC imperfection factor (δ)	0.4
CSI error variance (σε2​)	10− 5
Bandwidth (BW)	100 MHz
Noise power spectral density	−174 dBm/Hz
Transmit power range	0 to 30 dBm
Target data rates (Rktarget)	0.1 Gbps (both users)

Figure 1 presents the transceiver architecture of the proposed downlink SIC-based SM-NOMA system implemented with QGE² security. The system involves two users receiving image data over a common wireless communication channel. The base station (BS) has four transmit antennas with each user having four receive antennas. For both users, input data is grayscale images with the resolution of 256 × 256 pixels. The input images were encrypted with QGE² encryption, a combination of quantum-generated keys with genetic algorithm. The bitstreams of the encrypted images are then processed independently between the two users. For both User 1 and User 2, each block of bits is divided into two parts one part is used to select the index of the active transmit antenna, and the other part is modulated using 4-QAM to determine the complex symbol to transmit. This ensures that both users independently implement spatial modulation, where only one antenna transmits a modulated symbol per user per transmission slot. The spatially modulated signals from both users are then summed together through superposition coding with optimized power allocation factors (α₁ and α₂) forming a composite transmit vector. This signal is transmitted over a MIMO channel which is modelled as a Rayleigh fading channel to accurately represent the multipath propagation characteristics commonly found in wireless environments. Next, the BS transmits the combined signal over a channel modelled under Rayleigh fading conditions and imperfect channel state information (CSI). At the receiver both users use MRC to improve their signal quality by using their multiple antennas. User 1 is the weaker user who is located further from the BS. User 1 uses MRC to directly detect its signal from the received superimposed signal. User 2, the stronger user first estimates and subtracts User 1’s signal from the received signal using SIC. User 2 then applies MRC to detect User 2’s data. At the end of the user’s antenna system both received signals are demodulated and decrypted to recover the original images. This architecture is an effective way to implement spatial modulation with NOMA and encryption together to provide spectral efficiency, secure transmission and to accommodate multiple users in a power-domain multiplexing scenario.

Parameter selection

According to 3GPP Technical Report TR 38.900⁴³, in downlink NOMA user equipment (UE) may have 2 to 4 receive antennas. Therefore, this study utilized a 4 × 4 MIMO configuration with 4 transmit antennas (Mt = 4) and 4 receive antennas (Mr = 4). The transmitted symbols used 4-QAM, a widely used scheme balancing complexity and performance in multiuser systems. To model practical channel estimation limitations, an imperfect Channel State Information (CSI) model with error variance σε² = 10⁻⁵ is used. This variance reflects high-fidelity yet non-ideal CSI, comparable to values reported in the literature where acceptable estimation error levels range up to 10^{− 444}. The lower variance demonstrates improved signal processing while accounting for residual imperfections. A path loss exponent η = 3 is used, commonly accepted one for urban macrocell Non-Line-of-Sight (NLOS) scenarios⁴⁵. It captures signal attenuation caused by dense buildings and environmental obstacles in urban areas. User distances are set to 1900 m for User 1 and 1000 m for User 2, within the 10–5000 m range defined in 3GPP TR 38.901 for Urban Macrocell (UMa) environments. These provide realistic mid- and far-cell coverage in dense urban 5G deployments.

To represent non-ideal interference cancellation, a SIC imperfection factor δ = 0.4 is used. Prior studies^3,46 report values from 0.1 to 1.0, where 0.1 yields better performance and 1.0 degrades the system. The moderate value of 0.4 allows evaluation of SIC degradation without assuming extreme cancellation failure. The system bandwidth is 100 MHz, the maximum per-component carrier bandwidth in FR1 (Frequency Range 1) per 5G NR⁴⁷. The carrier frequency is 3.5 GHz, within the mid-band spectrum (3300–3800 MHz), offering a balance of coverage and capacity suitable for urban 5G deployments. Noise power spectral density is − 174 dBm/Hz, reflecting typical thermal noise. Transmit power ranges from 0 to 30 dBm to evaluate low- and high-power performance. The target data rate per user is 0.1 Gbps (100 Mbps), consistent with reported throughputs in urban 5G at 3.5 GHz. For reference⁴⁸, reports ~ 100 Mbps in both uplink and downlink. This target supports eMBB services and meets cell-edge throughput expectations.

The input data for each user is grayscale image of size 256 × 256. Each pixel is converted into 8 bits and reshaped into a serial bitstream. Let b1 and b2 represent the bitstreams for User 1 and User 2, respectively. The bitstreams are mapped using SM with 4-QAM modulation.

Each SM symbol carries 4 bits per symbol and is formulated as,

1

The transmit antennas are chosen from the M_t = 4 antennas. For each user, a 4-bit group is mapped. The first two significant bits are used to select the transmit antenna index while the remaining 2 bits select the corresponding 4-QAM symbol. Each 4-bit group is mapped to a specific antenna and a modulation symbol. The symbols s1 and s2​ are the spatially modulated symbols for User 1 and User 2, respectively.

Signal transmission

Let s1, s2 ∈ C ^Mt×1 be the spatially modulated signal vectors for User 1 and User 2 respectively. Each has only one non-zero entry corresponding to the active antenna, holding the selected modulation symbol. This structure defines Spatial Modulation (SM), where both the antenna index and the modulation symbol carry information. The remaining Mt−1 entries in the vector is zeros. The base station uses superposition coding to transmit both users’ signals simultaneously, allowing both users to share the time-frequency resource. Let x to be the signal transmitted from the base-station, which is a linear combination of both users’ signals s1, s2 with scaling by the respective power allocation factor α1 and α2.

The superposition coded signal transmitted from the BS is,

2

where,

• x∈C^Mt×1 is the transmitted signal vector.

• α1 + α2 = 1,

• P_t is the total transmit power in Watts.

Channel and noise model

H_k ∈ C ^Mr×Mt for k∈ {1,2} is the Rayleigh fading MIMO channel matrix with imperfect CSI modelled as,

3

where,

• is the estimated channel including imperfect CSI,

• ε_k ∼ CN (0, σε²I) represents the channel estimation error, modelled as a complex Gaussian noise with zero mean and covariance σ_ε²I.

Signal reception

The received signals at each user k ∈ {1,2} are,

4

where,

• n_k ∼ CN (0, NoI_Mr) is AWGN,

• N_o​ is computed based on noise spectral density and bandwidth,

The path loss is modelled as d^−η with a path-loss exponent η = 3. The distances of User 1 and User 2 from the BS are 1900 m and 1000 m.

Maximum Ratio Combining (MRC) is a low complexity detector utilized in wireless systems to increase the quality of the signal received by maximizing the signal-to-noise ratio (SNR). As MRC applies to Spatial Modulation (SM) systems, it separates the detection process of finding the active transmit antenna (the spatial symbol) and the modulated data symbol (e.g., QAM or PSK) into two separate stages of detection instead of jointly detecting as in traditional maximum likelihood (ML) methods. Thus, to enhance signal reliability, MRC was applied to merge the received signals captured by two spatially separated detectors assigned to the users⁴⁹. The combined result was then forwarded as the final output.

User 1 detection

User 1, being farther from the BS detects its signal directly from the superimposed received signal without performing interference cancellation.

Let y1∈C^Mr×1 be the received signal vector at User 1, and ∈C^Mr×Mt be the estimated channel matrix between the BS and User 1. Each column h_1,i∈C^Mr×1 of corresponds to the channel from the i-th transmit antenna to all receive antennas.

To detect the active, transmit antenna and the modulated symbol, User 1 performs Maximum Ratio Combining (MRC) on the received signal to retrieve its symbol. For each transmit antenna i∈{1, 2,…,M_t}, it computes the projection as,

5

where,

•

• z₁​(i) is the soft estimate for antenna i.

The index of the antenna estimated by,

6

Then the estimated modulated symbol detection is,

7

where,

•

•

This projection effectively filters the received signal based on each transmit antenna’s channel direction, thereby maximizing signal energy while minimizing noise. Together a1 and s1 can be considered as the entire spatial modulation symbol for User 1.

Demodulation and de-mapping are applied to to recover the original bit sequence for User 1.

User 2 detection

User 2 applies Successive Interference Cancellation (SIC) to decode its signal. First, it attempts to decode User 1’s signal using MRC then subtracts it from the received signal and then detecting its own signal.

Let y₂∈C^Mr×1 be the received signal vector at User 2, and ∈C^Mr×Mt be the estimated channel.

For each antenna i, the projection for estimating User 1’s signal a User 2 computes as,

8

where,

The estimated antenna and symbol of user1 are

9

User 1’s signal is reconstructed as,

10

where,

•

•

User 2 reconstructs the user 1’s SM vector corresponding to and removes it from The residual interference term will be present due to imperfect SIC is given as,

11

The residual interference added signal becomes,

12

User 2 then estimates its own symbol using MRC is given as,

13

where,

•

•

The pair a₂ and s₂ constitutes the spatial modulation symbol for User 2. Thus, the demodulation and demapping of s₂ and a₂​ yields the original bit sequence for User 2.

Performance metrics

The BER is calculated after demodulating and decoding the received signal. The demodulated bitstreams for User 1 and User 2 are compared against the original transmitted bitstreams to count the number of bit errors.

BER is calculated as,

14

The outage probability for user k is defined as the probability that the instantaneous channel capacity falls below the required target data rate R_k^target ,

15

B is the system bandwidth in MHz.

Impact of SINR with imperfect SIC, CSI and imperfect antenna detection

In the proposed NOMA system, User 1 (the weak user) decodes its signal directly without applying SIC, as its received power is relatively low. Due to imperfect channel state information (CSI), the estimated channel for User 1 contains errors, which reduce the effective channel gain and degrade the signal-to-interference-plus-noise ratio (SINR). The SINR for User 1 is given by,

16

User 2 (the strong user) performs successive interference cancellation (SIC) to decode and remove User 1’s signal before decoding its own data. However, imperfect SIC causes residual interference, modelled by a factor δ, which along with imperfect CSI reduces the effective SINR. The SINR for User 2 is expressed as,

17

where,

• and are the estimated channel vectors (columns) corresponding to the active antennas for User 1 and User 2, respectively.

• corresponds to the antenna index of User 1 as seen by User 2 (used in SIC).

• α1 and α2\alpha_2α2 are the power allocation coefficients,

• P_t is the total transmit power,

• N_o is the noise power,

• δ is the residual interference factor modelling imperfect SIC.

The residual interference increases with and combined with imperfect CSI, it causes degradation in User 2’s SINR, thereby negatively impacting its bit error rate and spectral efficiency.

The normalized data rate or spectral efficiency (SE) for user k is defined as the expected achievable rate,

18

The total system spectral efficiency is the sum of the individual SEs,

19

Hybrid quantum-genetic encryption

In the Quantum Key Distribution (QKD) process, quantum keys are generated using quantum gates specifically Hadamard (H) and Y gates. These gates manipulate qubits into superposition and complex quantum states introducing randomness and quantum uncertainty. The properties of quantum mechanics ensure any interception attempt is detectable, safeguarding the integrity of the key exchange.

Generation of quantum key

In classical computing, bits are the fundamental units of information processed sequentially via logic gates, whereas in quantum computers, it leverages quantum bits (qubits). Classical bits are restricted to binary states (0 or 1), whereas qubit can exist in superposition of the two states, ∣0 and ∣1> represented as a linear combination of these states⁵⁰.

20

where α and β are complex amplitudes and the sum of their squared magnitudes satisfies the normalization condition ∣α∣²+∣β∣²=1|.

Quantum gates are the building blocks of quantum circuits, similar to classical logic gates. They manipulate qubit states and are represented by unitary matrices that operate on the state of one or more qubits and are inherently reversible, meaning their effects can be undone by applying the inverse operation.

Various types of quantum gates perform distinct transformations on qubit states. For instance, the Pauli-X, Pauli-Y, and Pauli-Z gates correspond to the quantum analogy of classical NOT, CNOT, and CZ gates, respectively. Additional fundamental gates include the Hadamard gate, which creates superposition; the phase gate, which introduces a phase shift; and the controlled-phase (CZ) gate, which applies a phase shift conditionally based on the state of another qubit. By combining different quantum gates within the circuit, complex quantum algorithms can be implemented to perform advanced computations.

Quantum True Random Number Generators (QTRNGs) exploit the probabilistic nature of quantum observables such as the collapse of a qubit in superposition upon measurement to produce non-deterministic and unbiased random numbers⁵¹. QTRNGs allow for the generation of secure, non-reproducible encryption keys that are resilient to both classical and quantum attacks⁵².

In the proposed QTRNG, each one of the 24 qubits applies a Hadamard gate followed by a Pauli-Y gate, then the qubit is measured. This means that there are two quantum gate operations and one measurement per qubit per shot. Since both the Hadamard (H) and Pauli-Y (Y) gates are single-qubit gates, the total number of single-qubit gates per qubit is two. Therefore, for each of the 50,000 shots, the QTRNG system performs 2.4 million quantum gate operations (48 operations per shot × 50,000 shots) and 1.2 million measurement operations (24 measurements per shot × 50,000 shots). As a result, the number of gate operations scales linearly with the number of qubits i.e., O(n). This shows that the circuit remains computationally feasible, validating its practical implementation capability.

QTRNG with 24 qubits and 50,000 shots

In this model, a quantum circuit is designed using IBM Quantum Composer and then executed on the backend quantum processing unit (IBM Quantum Backend) “ibm_brisbane”. The circuit utilizes 24 qubits, with each qubit undergoing a Hadamard gate followed by a Pauli-Y gate. These quantum gates enable the system to achieve superposition, which is crucial for generating random numbers based on quantum mechanics. The output sequence of random bits is then validated against NIST SP 800-22 and SP 800-90 standards for randomness.

The quantum states evolve through the following steps, and the proposed quantum circuit is shown in Fig. 2.

Fig. 2.

Implementation of Pauli-Y and Hadamard gates for true random number generation (a) Quantum circuit showing sequential gates (b) Q-sphere visualization (c) Measurement outcome probabilities (d) State vector.

1. Initialization: All qubits are initially set to the state ∣0>|.

21

• b.Hadamard gate (H): The Hadamard gate is applied to each qubit. It places each qubit into an equal superposition of ∣0>and ∣1>.

22a

22b

• c.Pauli-Y gate (Y): After the Hadamard gate, the Pauli-Y gate is applied. It induces both a bit flip and a phase flip on the qubits.

23a

23b

• d.After applying the Hadamard and Pauli-Y gates, the final quantum state of each qubit is.

24

• e.Measurement: The final quantum state is measured, collapsing the qubit states into one of the computational basis states ∣0> or ∣1>. This measurement generates a random bit.

25

After applying the Hadamard and Y gates the quantum state is visualized on the Q-sphere. The Hadamard gate creates an equal superposition of states while the Y gate introduces a phase shift. The resulting quantum states are depicted on the Q-sphere, with the state vector and probabilistic graphs illustrating the collapse of the superposition into a definite state after measurement. Figure 2a shows the circuit of Hadamard-Pauli Y gate and the measurement that caused the qubit to collapse into a definite state. The state of the qubit is shown clearly in Fig. 2b, where the Q-sphere shows that the state vector is aligned with one of the poles, ∣0> or ∣1>. The measurement statistics shown in Fig. 2c show the probability of both outcomes occurring, which emphasizes the intrinsic randomness of the measurement outcome. Figure 2d supports that the post-measurement state points directly to a pole on the Q-sphere, affirming that the quantum state has been reduced to a classical basis state. Figure 2a–d together show that the Hadamard-Y gate arrangement can produce true quantum randomness.

Quantum-genetic concatenation

To further enhance the security QTRNG is combined with optimization schemes like Meta-heuristic techniques. Quantum Key (QK) can handle the key exchange process with a high degree of security, while classical encryption algorithms take care of encrypting the actual data. This hybrid setup ensures that even if one layer is compromised, the other one continues to protect the system, offering a more resilient approach to securing wireless communications.

QTRNG is used in conjunction with a Genetic Algorithm (GA). The quantum key generated utilizing QTRNG acts as the seed for the GA, which is then used to optimize the encryption parameters, offering a more secure and efficient encryption scheme.

In the proposed system, quantum key of size 128 bits is used as seeds for initializing the random number generator in the genetic algorithm which generates a permutation sequence to encrypt and decrypt data. The quantum key ensures that the encryption and decryption are symmetric and it involves permutation of bits based on an evolved sequence generated by GA. The GA performs optimization over generations to create permutation of bits (using selection, crossover, and mutation), which is used to shuffle the bits of the plaintext to obtain the ciphertext. The same process with the same quantum key is used to reverse the permutation for decryption. The core of the GA process is that it initializes a population of random bit permutations, evaluates their fitness based on diversity, and applies crossover and mutation operators to evolve the population over several generations. The best permutation is selected as the final result. To achieve this, the 128-bit quantum key is divided into 32 blocks of 4 bits each, and an XOR operation is performed on each block to generate a 32-bit binary seed. This seed is then converted to a decimal value and used to initialize random number generator, ensuring a highly diverse and unpredictable initial population for the genetic algorithm.

Figure 3 shows the flow of the proposed algorithm. It explains about the key generated using quantum gates and genetic algorithm.

Fig. 3.

Flowchart of the proposed algorithm, illustrating the steps from key generation using Quantum Gates to encryption using Genetic Algorithm.

The flow of operation is explained as follows,

Step 1: Quantum key generation:

• A 128-bit quantum key Q is generated using a quantum random number generator (QRNG). The Quantum Key is used as a seed for the Random Number Generator (RNG).

Step 2: Generate RNG seed.

• The 128-bit quantum key is divided into 32 blocks of 4 bits each.

• XOR each block to generate the 32-bit RNG seed.

• The XOR operation ensures that the bits are mixed to provide randomness.

Step 3: Initialize population for genetic algorithm (GA).

• Create an initial population of chromosomes (permutations) of image pixels.

• The image I(input image) has L(256*256) pixels, where each pixel is represented by a single bit in an 8-bit grayscale image.

• A chromosome is a permutation of L indices that represents where each bit of the image will be relocated.

Step 4: Evaluate fitness.

• Measure how well each chromosome (permutation) performs by comparing it with a reference permutation.

• A reference permutation R= [1,2, 3…., L] is used for comparison.

• The fitness of each chromosome is evaluated based on the number of mismatches (diversity from the reference permutation).

Step 5: Selection of parents.

• Select the top 50% of chromosomes based on their fitness scores.

• Sort the population based on the fitness scores in descending order.

• Select the top 50% chromosomes as parents for the next generation.

Step 6: Crossover.

• Combine two selected parents to create offspring.

• For each pair of parents P1​ and P2​ select a random crossover point cp.

• Perform a crossover to create two offspring c1​ and c2.

Step 7: Mutation.

• Introduce small random changes (30%) to the population for genetic diversity.

• With a probability Pm randomly swap two indices in a chromosome.

• Mutation ensures that the algorithm does not converge prematurely.

Step 8: Encryption of the image.

• Rearrange the pixels of the image according to the best permutation after several generations.

• After several generations, the best permutation P_best is selected.

• The original image I is encrypted by applying the best permutation P_best​.

Decryption process

The decryption process reverses the encryption by applying the inverse of the permutation generated during encryption using the same key.

Algorithm: GA-based encryption and decryption.

The encryption scheme is shown in Fig. 4 and its system parameters are given in Table 3 for two users.

Fig. 4.

Quantum True Random Number Generator (QTRNG) based key generation with Genetic Algorithm (GA) operations (selection, crossover, and mutation) for robust image encryption and secure decryption across multiple users with 256 × 256 grayscale Pepper and Cat images as test inputs.

Table 3.

System parameters used in GA-based image encryption and decryption.

Parameter	Symbol	Value/range	Description
Quantum key length	Q	128 bits	Binary key used to derive the initial seed
Reduced seed length	S	32 bits	Obtained by XOR-ing each 4-bit group of the key
Image size	M×N	256 × 256	Size of input grayscale image
Total bits in image	L	M×N×8	Number of bits in the image
Population size	P	100	Number of chromosomes in the genetic algorithm
Number of generations	G	50	Number of generations the GA evolves through
Chromosome length	L	524,288	Each chromosome is a permutation of image bit indices
Crossover probability	–	100% (Top 50% pairs)	Always applied between selected parents
Mutation probability	–	30%	Random swap of two indices within a chromosome
Crossover point	cp.	1 ≤ cp. ≤ L − 1	Random crossover point for two parent chromosomes
Fitness metric	F(Ci)	Hamming distance	Measures diversity from the identity permutation

The Hamming distance-based fitness function used in the proposed encryption framework quantitatively measures the amount of bit-level mismatch between the known plain image and the encrypted image, as well as between the decrypted and the encrypted image, by the use of a candidate key, as clarified in⁵³. This is a computationally low-cost operation and an attractive method for cases where perfect key recovery is the key response, rather than approximating or identifying entropy-based fitness. Unlike statistical or component-wise similarity measures the goal of the encryption here is to identify the exact permutation key. Hamming distance follows bit-for-bit matching, whereas statistical methods used in piracy mitigation (i.e., entropy or correlation) promote estimates of matching keys. Though other fitness functions may offer certain advantages, the use of Hamming distance provides accuracy which is particularly important in permutation-only encryption schemes where even a minor mismatch results in complete decryption failure. This decisive nature makes Hamming distance a reliable and effective choice for assessing key correctness in the proposed system.

The crossover and mutation probabilities utilized in this work were chosen from the recommended ranges proposed in other studies⁵⁴. The mutation probabilities generally range from 0.001 to 1, and probabilities for crossover range from 0.1 to 1. These parameters offer a balance between a high exploratory search (crossover) and an exploitation search (mutation) within the genetic algorithm. The use of full crossover allows strong genetic mixing and the mutation rate permits continued diversity in the population and prevents premature convergence. Future work may consider tuned or adapted values that help to achieve optimal performance.

Results and discussions

The proposed system compares the performance of the proposed SM-based NOMA with SIC and SIC-based NOMA under practical conditions, including both imperfect SIC and imperfect CSI. The system was simulated via MATLAB with two users; The input images chosen were a 256 × 256 8-bit grayscale image of a pepper for user 1 and a cat image of the same size for User 2.

Figure 5 is the graph plotted between BER and transmit power for two users. It is plotted between SM NOMA and conventional NOMA.

Fig. 5.

BER versus transmit power comparison between conventional NOMA and the proposed SM NOMA. SM-NOMA reduces BER by 83.6% for User 1 and 96.6% for User 2 under 25 dBm transmit power.

In the SIC-based NOMA system at 25dBm transmit power, User 1 experiences a BER of 4.98 × 10⁻² and User 2 a BER of 2.33 × 10⁻². These error rates arise from the combined effects of path loss, residual interference and error propagation inherent in imperfect SIC. User 1 being farther away from the base station at 1900 m suffers more from signal attenuation and residual interference which limits the effectiveness of successive interference cancellation. To mitigate this, a power domain approach is adopted where the weaker user (User 1) is allocated higher transmission power and the stronger user (User 2) being closer to the base station is assigned with lower power. This power imbalance enables the weak user to decode its signal with higher reliability while the strong user performs SIC to decode its intended data. In contrast, the proposed SM-NOMA with SIC achieves reduction in BER with User 1 BER falling to 8.15 × 10⁻³ and User 2 BER dropping to 7.87 × 10⁻⁴. This over-order-of-magnitude improvement stems from spatial modulation’s single-active-antenna transmission, which inherently suppresses inter-user interference by mapping information onto both antenna indices and symbol constellations. Even with only one antenna active per symbol, the diversity gain provided by MRC detection further strengthens resilience to fading and CSI inaccuracies. As a result, SM-NOMA effectively mitigates the impact of imperfect SIC and CSI eliminating the error floor observed in the SIC-based NOMA scheme.

In the analysis of outage probability at 20dBm transmit power User 1 is farther from the base station (1900 m) and has higher power allocation compared to User 2 (1000 m), the outage probabilities for both users in the proposed SM-NOMA system are significantly lower than in the SIC-based NOMA system, which is plotted in Fig. 6. The higher power allocated to User 1, despite its greater distance, compensates for the increased path loss but the SIC-based NOMA system still suffers from imperfect interference cancellation, resulting in higher outage probabilities particularly for User 2 (0.180303). On the other hand, SM-NOMA leverages spatial modulation effectively reducing interference and leading to a much lower outage probability for both users with values of 1.52588 × 10^− 5 for User 1 and 8.39233 × 10^− 5 for User 2.

Fig. 6.

Outage probability versus transmit power for User 1 and User 2 of NOMA and SM-NOMA system. The proposed SM-NOMA reduces outage probability by 99.9% for User 1 and 99.87% for User 2 showing strong robustness across 20 dBm transmit power.

Figure 7 is the graph plotted between transmit power and spectral efficiency. In the SIC-based NOMA system spectral efficiency for user 1 is 1.46788 bps/Hz and for User 2 it is 1.10287 bps/Hz, resulting in a total spectral efficiency of 2.57075 bps/Hz. In contrast, for SM-NOMA the spectral efficiency is 1.59729 bps/Hz for user 1 and 1.14325 bps/Hz for user 2, yielding a higher total spectral efficiency of 2.74054 bps/Hz. This improvement is primarily due to the spatial modulation, which efficiently utilizes the available spatial resources by encoding information onto the active transmit antenna index, thereby reducing inter-user interference.

Fig. 7.

Spectral efficiency comparison between conventional NOMA and SM-NOMA system. SM-NOMA improves spectral efficiency by up to 6.6% at 25 dBm transmit power compared to NOMA.

Table 4 provides a comparison between NOMA and spatial modulation-based NOMA in terms of bit error rate, outage probability and spectral efficiency. The values are tabulated for various values of transmit power. It is observed that the bit error rate is less in the case of SM-NOMA when compared with NOMA. Similarly, outage probability is also less for SM-NOMA. At the same time, spectral efficiency increases for the proposed system.

Table 4.

Comparison between BER, outage probability and spectral efficiency of conventional NOMA and SM-NOMA.

Transmit power (dBm)		BER (NOMA)	BER (SM-NOMA)	Outage probability (NOMA)	Outage probability (SM-NOMA)	Spectral efficiency (bps/Hz) (NOMA)	Spectral efficiency (bps/Hz) (SM-NOMA)
0	User 1	0.380611	0.396713	1	0.999687	0.409247	1.10943
	User 2	0.53581	0.343025	1	1
5	User 1	0.312447	0.293415	0.990746	0.686859	0.849193	1.7824
	User 2	0.411369	0.249247	0.998065	0.871399
10	User 1	0.238003	0.179499	0.771409	0.0629272	1.42191	2.30683
	User 2	0.267136	0.142069	0.862539	0.140404
15	User 1	0.16877	0.100266	0.373486	0.0014801	1.96246	2.58639
	User 2	0.145649	0.0522232	0.466263	0.00399017
20	User 1	0.102949	0.0403576	0.137069	1.52388 × 10− 5	2.34886	2.70053
	User 2	0.0650826	0.00867844	0.180303	8.39233 × 10− 5
25	User 1	0.0498562	0.00815773	–	–	2.57075	2.74054
	User 2	0.0233765	0.000787735	–	–
30	User 1	0.0191593	0.000705719	–	–
	User 2	0.00767708	2.28882*10− 5	–	–

Figure 8a shows the graph plotted between BER and transmit power under imperfect CSI conditions. BER increases with increasing channel estimation errors, which lowers SINR. Figure 8b illustrates the decline in spectral efficiency as CSI errors increase. This means imperfect CSI reduces SINR, causing worse performance.

Fig. 8.

Effect of imperfect CSI on system performance: (a) BER rises with increasing channel estimation errors, (b) Spectral efficiency declines as SINR reduces due to imperfect CSI.

Figure 9 shows how imperfect SIC affects the system. In Fig. 9a, BER increases as residual interference increases. For this system, a moderate residual interference value of 0.4 is used. Figure 9b shows that spectral efficiency decreases as SINR becomes worse with more imperfect SIC. This means residual interference harms both error rates and throughput.

Fig. 9.

Effect of imperfect SIC on system performance: (a) BER rises with increasing residual interference, (b) Spectral efficiency declines as SINR reduces due to imperfect SIC.

Key testing and validation of QTRNG using NIST statistical test suites

To ensure the reliability and unpredictability of the generated quantum key, the output from the proposed Quantum True Random Number Generator (QTRNG) was rigorously tested and validated using two statistical test suites provided by the National Institute of Standards and Technology (NIST): SP800-22 and SP800-90B. These tests serve as industry benchmarks to assess both the randomness and entropy quality of random number generators used in cryptographic systems.

NIST SP 800-22 (statistical test suite)

This test suite comprises 15 core tests and 2 sub-tests (Serial Test and Cumulative Sums Test), total 17 statistical tests. These tests evaluate the statistical randomness of binary sequences generated by a source. Each test computes a p-value and for a sequence to pass the p-value must be greater than or equal to 0.01. The tests assess various aspects of randomness including frequency, pattern complexity, linear complexity and periodic features⁵⁵. All tests returned p-values > 0.01, indicating that the sequence passes the SP800-22 randomness criteria.

NIST 800-90B (entropy source evaluation)

This suite is designed to evaluate the entropy source that underlies a random number generator. It employs ten entropy estimators to measure the worst-case min-entropy, ensuring that the entropy source provides sufficient unpredictability. The tests include methods such as the T-Tuple test, compression test, Markov model prediction tests, and LZ78-based predictors. A raw data sequence of 100 Mb was collected from the QTRNG for this evaluation. The T-Tuple test, which estimates entropy based on tuple frequency, yielded a minimum entropy of 0.9263 per bit, confirming that even under worst-case conditions, the entropy source maintains randomness. As illustrated in Fig. 10, the QTRNG output probability distribution reflects a uniform and unbiased random behaviour. Figure 10a shows the SP800-22 test results, where all tests meet or exceed the required threshold (p-value ≥ 0.01), confirming high statistical randomness, and Fig. 10b shows the SP800-90B test results, where the min-entropy values across different estimators validate that the entropy source is robust and unpredictable.

Fig. 10.

(a) NIST SP 800-22 randomness evaluation for quantum-generated key, confirming high statistical randomness. (b) NIST 800-90B randomness evaluation for quantum-generated key, showing high-quality entropy source for encryption.

Histogram

A grayscale histogram illustrates the frequency of pixel intensities in an image and is commonly used in evaluating the strength of image encryption schemes. A secure encryption process should produce an output image whose histogram does not reveal any recognizable patterns or correlations with the original image, thus minimizing vulnerability to statistical analysis. Figure 11 shows the histogram comparison for two input images: Pepper (used for User 1) and Car (used for User 2). From the figure, it’s clear that the histograms of the encrypted images exhibit nearly uniform distributions. Despite the distinct differences in content and histogram structure between the original Pepper and Car images, their encrypted counterparts show highly similar and randomized histogram patterns⁵⁶. This observation confirms that the proposed encryption method effectively conceals statistical features of the input images, rendering them secure against histogram-based and statistical attacks.

Fig. 11.

Histogram of User 1’s original/encrypted Pepper image and User 2’s original/encrypted Cat image, demonstrating uniform distribution.

Chi-square test

In order to test for randomness and its effectiveness of encryption Chi-square statistical test if performed on the histogram of the image. This test checks if the pixel intensity values are uniformly distributed. Uniform distribution is an important indication of the strength of encryption. The Chi-square value is determined using the following Eqs⁵⁷. ,

26

Here, represents the actual observed) frequency of pixel intensity level , whereas represents the expected frequency, which has been calculated as 256, or the total number of pixels to 256. It can be observed that a smaller value for Chi-square corresponds with more uniform distribution, which illustrates statistical security in the encrypted image, as shown in Table 5.

Table 5.

Chi-square analysis of pixel intensity uniformity in original and encrypted images.

Image	Original image	Encrypted image
User 1	60,961.16	290.48
User 2	7760.84	282.50

The encrypted images show significantly reduced Chi-square values, indicating improved histogram uniformity.

Mean square error (MSE)

An effective image encryption algorithm should ensure that the encrypted output bears minimal resemblance to the original image. One widely used metric for quantifying this dissimilarity is the Mean Square Error (MSE), which measures the average squared difference between corresponding pixels of two images.

27

For images of size 2ⁿ × 2ⁿ, MSE is calculated based on the pixel intensities of the original image I(i, j) and the encrypted image E(i, j) at position (i, j). A higher MSE value indicates a greater level of distortion which in turn reflects a stronger encryption performance. In the proposed system the MSE of user 1 and 2 is 7991 and 9686 confirms the strongest encryption.

PSNR

The Peak Signal-to-Noise Ratio (PSNR) is a key metric used to assess how closely a decrypted image resembles the original one. It quantifies the reconstruction accuracy by measuring the level of distortion introduced through the encryption and decryption processes. PSNR is mathematically expressed as⁵⁸

28

where,

• X(i, j) and Y(i, j) represent the pixel intensities of the original and decrypted images at location (i, j),

• M and N denote the height and width of the image and.

• 255 is the highest possible pixel value for 8-bit grayscale images.

In general, higher PSNR values correspond to lower distortion and indicate better image reconstruction. Conversely lower PSNR values suggest more significant degradation during processing. For the proposed encryption scheme, the PSNR measured for User 1’s is 9.1045 dB while that for User 2 is 8.2689 dB. These values reflect that the decryption process retains a reasonable level of visual quality, affirming the effectiveness of the encryption algorithm in preserving image fidelity.

Entropy

Information entropy serves as a statistical measure to capture the randomness in the grayscale distribution of an image. It is derived based on the likelihood of occurrence of each pixel intensity level. For a grayscale image, let P(i) represent the probability of a pixel having the intensity value i. The entropy is then calculated as,

29

A higher entropy value suggests a more uniform and unpredictable distribution of pixel values, indicating a stronger level of encryption or randomness. Ideally, for an 8-bit grayscale image, the maximum entropy can reach 8 bits which signifies a perfectly uniform pixel distribution.

Correlation analysis

To assess the effectiveness of the encryption method, the correlation between adjacent pixels is analysed in horizontal, vertical, and diagonal orientations. In unencrypted (plain) images, adjacent pixels often have strong correlation, resulting in clustered scatter plots near the diagonal line. After encryption, however, a secure algorithm should eliminate this correlation, leading to a uniform scatter across the plot. The adjacent pixel correlation is measured using the formula⁵⁹

30

where:

• U and v represent adjacent pixel values,

• E(⋅) and D(⋅) are the expectation and variance functions.

Figure 12. Correlation analysis of User 1’s original/encrypted Pepper image and User 2’s original/encrypted Cat image, demonstrating low correlation between the input and the encrypted images.

Fig. 12.

shows the correlation analysis and Table 6 provides the correlation coefficients for each direction before and after encryption.

Table 6.

Correlation coefficient analysis of input and encrypted image of two users.

Input image	Original image correlation			Encrypted image correlation
	Horizontal	Vertical	Diagonal	Horizontal	Vertical	Diagonal
User 1	0.9825	0.9867	0.9654	− 0.0011	− 0.0009	0.0003
User 2	0.9622	0.9503	0.9125	− 0.0007	− 0.0105	− 0.0073

NPCR & UACI

NPCR (Number of Pixels Change Rate) measures the percentage of differing pixels between two cipher images generated from slightly different plain images. A high NPCR value indicates strong sensitivity⁶⁰.

31

UACI (Unified Average Changing Intensity) evaluates the average intensity variation between the two encrypted images:

32

where P_ij and C_ij are pixel values from the plain and encrypted images, and L is the grayscale level (typically 255).

Figure 13 gives the NPCR and UACI values for user 1 and user 2. The ideal value of NPCR lies between 99.54 and 99.72 and UACI is 33.25-33.362⁶¹. For user 1 the obtained NPCR is 99.64 and user2 it is 99.60. Similarly, the UACI of user1 is 33.30 and user2 is 33.49.

Fig. 13.

UACI and NPCR of the user 1 pepper image and user 2 cat image, verifying strong diffusion and resistance to differential attacks.

Table 7 presents a comparison of MSE, PSNR, entropy, UACI, NPCR, and correlation coefficients for User 1 and User 2 with methods from the existing literature, including chaos-based techniques as well as other classical encryption methods.

Table 7.

Performance metrics comparison.

Method	Size	MSE	PSNR	Entropy	UACI	NPCR	Correlation coefficient
							V	H	D
Proposed user1 (pepper)	256 × 256	7991.5733	9.1045	7.9967	33.30	99.64	− 0.0011	− 0.0009	0.0003
Proposed user 2 (cat)	256 × 256	9686.9381	8.2689	7.9957	33.4	99.60	− 0.0007	− 0.0105	− 0.0073
[63]	256 × 256	8708	8.73	7.99	33.46	99.61	0.0016	− 0.0031	− 0.0081
[64]	256 × 256	10,561	8.45	8.0214	33.45	99.6	− 0.0081	0.0030	0.0168
[65]	256 × 256	277.74	45.6	5.5549	50.172	94.426	0.12283	0.11034	0.2208
[66]	256 × 256	–	–	7.99	33.4	99.6	0.0062	0.0063	0.0069
[67]	256 × 256	12,101	7.3	7.9973	33.53	99.63	− 0.018	− 0.019	0.0013
[68]	256 × 256	–	–	7.9974	33.46	99.60	− 0.0008	− 0.0027	0.0027
[69]	256 × 256	–	–	7.99	33.44	99.52	0.1179	0.1482	0.0900

Figure 14 demonstrates the key sensitivity performance of the proposed encryption method. In chaotic systems, even a minimal change in the initial key leads to drastically different encrypted outputs. This unpredictability ensures decryption fails when using an incorrect key, confirming strong cryptographic resilience. The results analysed using NPCR and UACI metrics, validate the system’s high sensitivity to key variation.

Fig. 14.

Key sensitivity analysis of QGE²-encrypted images for User 1 and User 2, highlighting the encryption scheme’s robustness to slight key variations.

Thus, the proposed method has good encryption properties as demonstrated by strong entropy values (almost equal to 7.99), high UACI (roughly 33.3–33.4%), NPCR (around 99.6%) and almost zero correlation coefficients in all directions. This indicates that the encrypted images possess strong statistical randomness and are very robust against differential attacks.

Figure 15. SSIM comparison between the original and decrypted images, illustrating the balance between encryption effectiveness and image reconstruction quality. Higher SSIM values suggest minimal distortion during decryption, highlighting that strong encryption can still retain perceptual image integrity.

Fig. 15.

shows the Structural Similarity Index (SSIM) plot. The decrypted images have SSIM value of 1, indicating perfect structural similarity to the original image and this refers to high quality reconstruction of image.

Robustness against attacks

An effective image encryption method must resist multiple adversarial models, including statistical, brute-force, ciphertext-only, known-plaintext, chosen-plaintext, differential, and noise/data-loss attacks⁶⁹. The proposed GA-based permutation combined with SIC-SM-NOMA was evaluated against each of these attack types.

Statistical attack

Natural images exhibit strong correlations among adjacent pixels, which can be exploited in statistical attacks. A secure encryption algorithm should eliminate these correlations to prevent leakage of structural information. The correlation coefficient, ranging from − 1 to 1, measures the dependency between two pixels, where 0 indicates no correlation. For the proposed system, the encrypted images showed near-zero correlations in all directions. For User 1 (Pepper), the horizontal correlation was − 0.0011, vertical − 0.0009, and diagonal 0.0003, while for User 2 (Cat), horizontal was − 0.0007, vertical − 0.0105, and diagonal − 0.0073. These results confirm that adjacent pixels in the cipher images reveal no statistical patterns, demonstrating strong resistance to statistical attacks.

Brute-force attack

The key space represents the set of all possible keys that an encryption algorithm can use. To ensure a stronger level of security against brute-force attacks, the key space of an image encryption algorithm should be larger than 2¹⁰⁰⁷⁰. In the proposed system, a 128-bit master key was used, from which a 32-bit GA seed was derived by XORing 4-bit blocks of the master key. The key space, determined by the 128-bit master key, corresponds to 2¹²⁸ possible keys, making exhaustive search attacks computationally infeasible with current technology. Additionally, the encrypted images exhibited very high cipher entropy, with User 1 (Pepper) having 7.9967 and User 2 (Cat) having 7.9957. Values near the ideal 8 indicate maximum randomness and a uniform pixel distribution, further strengthening resistance to brute-force attacks.

Ciphertext-only attack

Ciphertext-only attacks assume that the adversary has access only to the encrypted images⁷¹. The proposed system resists such attacks because of the near-zero correlation among adjacent pixels and high cipher entropy. These properties prevent meaningful information from being inferred from the ciphertext.

Known-plaintext attack

Even when some plaintext–ciphertext pairs are available, recovering the key or decrypting other ciphertexts is infeasible. This is because the GA-based permutation is key-dependent, and each encryption is uniquely tied to the 128-bit master key and the evolution of the GA, leaving no exploitable mapping for attackers.

Chosen-plaintext attack

Chosen-plaintext attacks allow an adversary to select specific plaintexts and obtain their corresponding ciphertexts⁷². Resistance is ensured by the nonlinear GA-based permutation in combination with the secret 128-bit master key. Experimental diffusion metrics further validate resistance, with NPCR values of 99.64% and 99.60% and UACI values of 33.30% and 33.40% for Users 1 and 2, respectively. These high values indicate that even minor changes in the plaintext lead to drastically different ciphertexts, preventing predictable mappings⁷³.

Differential attack resistance

Differential attacks exploit relationships between small modifications in plaintext and the resulting changes in ciphertext. Robust encryption ensures that a single-pixel change propagates widely across the encrypted image. The proposed system demonstrates this quantitatively through NPCR and UACI metrics, confirming strong diffusion and resilience against differential attacks.

Noise and occulsion resistance

The robustness of the proposed system was evaluated by introducing Gaussian noise with zero mean and variance of 0.0004, salt-and-pepper noise with 5%, and 25% partial cropping. The results confirm that the encrypted images can be reliably decrypted even under noise or occlusion, demonstrating strong resilience against data corruption. Figure 16a shows the robustness of QGE²-encrypted images for User 1, and Fig. 16b shows the robustness for User 2 under noise and data-loss conditions. Furthermore, the obtained results show that even when encrypted images are affected by such attacks, successful decryption can still be achieved without significant perceptual distortion.

Fig. 16.

Robustness of QGE²-encrypted images under noise and occlusion attacks: (a) User 1, (b) User 2.

Quantum attack resistance

In the proposed system, the 128-bit key is derived from a quantum random number generator using quantum gates, making the key inherently unpredictable and suitable for post-quantum security. This design provides strong resilience against potential quantum attacks. Specifically, Grover’s algorithm, which accelerates brute-force searches, is still ineffective due to the large key space and dynamic GA-based permutation. Similarly, Shor’s algorithm, which targets number-theoretic cryptosystems, is irrelevant because the encryption does not rely on factorization or discrete logarithms. Combining quantum-generated keys with GA-based permutation ensures robust security while maintaining practical feasibility for real-world deployment.

Practical challenges

QTRNG implementation

The QTRNG (Quantum True Random Number Generator) module has been implemented with IBM’s Quantum platform which has cloud-based access to quantum processors, allowing key generation through real quantum circuits that utilize gates. There is no need for local quantum devices since IBM Quantum Platform supports simulated and real backends. In the proposed system, IBM Quantum Composer is used to design and run the quantum circuits, which are executed on the ibmbrisbane backend. However, if the system is implemented in real-time hardware, challenges such as high hardware cost⁷⁴, integration complexity with classical systems⁷⁵, network incompatibility, qubit decoherence and error correction demands⁷⁶, high energy requirements for ultra-low-temperature maintenance², rigidity and cost of entangled photon-based QKD systems, and lack of built-in authentication mechanisms would need to be addressed.

To generate 1,200,000 key bits, the system must perform 2,400,000 quantum gate operations and 1,200,000 measurements, which may introduce delays incompatible with the ultra-low latency requirements of 5G. Scaling the system is also difficult due to the high costs of quantum transmitters and receivers, estimated at approximately $26,935 for deployment and $6,528 for operational expenses, resulting in a total of roughly $33,463 per unit⁷⁷.

SM-NOMA implementation

As real-time testing, the proposed work utilized Universal Software Defined Radio Peripheral (USRP B210 SDR) hardware shown in Fig. 17a, which is a universally accepted testbed for wireless systems. 4-Quadrature Amplitude Modulated (QAM) NOMA signals with spatial modulation is transmitted and received. The constellation diagrams obtained from the real-time transmission and reception setup added to show the feasibility and integrity of the SM-NOMA transmission and reception in the real time over air. It is shown in Fig. 17b.

Fig. 17.

(a) USRP Testbed. (b) Constellation diagram of 4 QAM.

Complete SM-NOMA transmission via USRP will be implemented in future work to enable comprehensive evaluation under real-time conditions. The current USRP setup primarily validates hardware functionality and does not incorporate advanced features such as complete SM-NOMA or security mechanisms.

• Implementation challenges of QTRNG with SM-NOMA—The integration of quantum true random number generators into SM-NOMA systems faces limitations due to unreliable quantum devices, limited qubit coherence duration, and alignment of classical USRP units.

• Cost considerations with SM-NOMA—Quantum devices and high-performance USRPs will raise the cost of any system, both in terms of isolation requirements for cryogenic cooling and synchronization of the RF front-ends at millisecond scales.

• Scalability issues of quantum-assisted NOMA systems—Any possible real-time scalability is limited by quantum measurement speed, multi-user interference, and practical difficulties in extending QTRNG-assisted security across larger networks.

Conclusion

This work presents a secure and spectrally efficient Spatial Modulation (SM)-assisted Non-Orthogonal Multiple Access (NOMA) system designed for two-user downlink. The system is built using 4-transmit and 4-receive antennas. However, unlike traditional MIMO-based NOMA, where all antennas are active simultaneously, the proposed SM-NOMA system activates only one transmit antenna per symbol duration, effectively reducing the need for multiple RF chains. In the proposed scheme, the input bitstream is split into two parts: one part selects the active transmit antenna, while the other part maps the data to a conventional modulation symbol (e.g., 4-QAM) for transmission through the selected antenna. This approach drastically reduces the hardware complexity by limiting the number of RF chains needed for transmission, resulting in a significant power saving and reduced system cost while maintaining high communication performance.

The Simulation results confirm that SM-NOMA significantly reduces BER, lowers outage probability, and improves spectral efficiency compared to conventional NOMA.At 25 dBm transmit power, User 1 and User 2 exhibit BER improvements of 83.6% and 96.6%, respectively. Outage probability is reduced by over 99.99% for User 1 and 99.87% for User 2 at 20 dBm. Spectral efficiency increases by 6.6%, from 2.57 bps/Hz to 2.74 bps/Hz.

Security of the information (image) is achieved through the integration of Quantum-Enhanced Genetic Encryption, a hybrid encryption method combining quantum gate-generated keys and genetic algorithm-based encryption. Utilizing it the image is encrypted and transmitted. The randomness of the key generated has been validated through NIST randomness test. Additional metrics such as entropy, Mean square error, correlation, PSNR and histogram analysis confirm the robustness.

Limitations and future work

The bottleneck to this work is the lack of real-time quantum hardware, GA convergence delays and constraints in SDR-based implementations. Future work will focus on deploying the full SM-NOMA SDR system with real-time quantum integration to validate the proposed framework’s practical adaptability and robustness in Sub-6 GHz 5G applications.

Abbreviations

5G

Fifth generation

NOMA

Non-orthogonal multiple access

SE

Spectral efficiency

MIMO

Multiple-input multiple-output

mm-wave

Millimetre-wave

RF

Radio frequency

TA

Transmit antenna

PA

Power amplifier

CSI

Channel state information

SM

Spatial modulation

QAM

Quadrature amplitude modulation

SC

Superposition coding

SIC

Successive interference cancellation

OMA

Orthogonal multiple access

PLS

Physical layer security

QKD

Quantum key distribution

QTRNG

Quantum true random number generator

QGE²

Quantum-enhanced genetic encryption

SM-NOMA

Spatial modulation non-orthogonal multiple access

BER

Bit error rate

MRC

Maximum ratio combining

RNG

Random number generator

SNR

Signal-to-noise ratio

SINR

Signal-to-interference-plus-noise ratio

RF Chain

Radio frequency chain

AES

Advanced encryption standard

MSE

Mean squared error

PSNR

Peak signal-to-noise ratio

SSIM

Structural similarity index

Qubit

Quantum bit (basic unit of quantum information)

QRNG

Quantum random number generator

QK

Quantum key

QPP

Quantum permutation pad

NIST

National Institute of Standards and Technology

NIST SP 800-22

NIST Special Publication 800-2 (statistical test suite)

NPCR

Number of pixels change rate

UACI

Unified average changing intensity

IoT

Internet of things

Author contributions

S.T. carried out the study, including conceptualization, design, experiments and data analysis. A.J. supervised the study, including data analysis, and drafted the manuscript. All authors have read and approved the final manuscript.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.