High-speed FSO-5G wireless communication system with enhanced loss compensation using high-power EDFA

Abstract

In this paper, we demonstrated a novel bidirectional high-speed transmission system integrating a free-space optical (FSO) communication with a 5G wireless link, utilizing a high-power erbium-doped fibre amplifier (EDFA) for enhanced loss compensation. The system supports downlink rates of 1-Gb/s/4.5-GHz and 10-Gb/s at 24-GHz and 39-GHz, and an uplink rate of 10-Gb/s/28-GHz. The high-power EDFA boosts signal strength, facilitating reliable long-distance transmission through the FSO transmission link while compensating for losses and reducing bit error rates (BER) and error vector magnitude (EVM). Fibre Bragg grating sensors are employed as wavelength selectors for both downlink and uplink, offering a simpler, cost-effective solution compared to previously utilized reflective semiconductor optical amplifiers and multiple laser sources. The system successfully transmits 16-quadrature amplitude modulation orthogonal frequency division multiplexing signals across various carrier frequencies, achieving total data rates of 21-Gb/s for downlink and 30-Gb/s for uplink across a 1.6-km FSO transmission link integrated with 10/15-m 5G links. Performance metrics, including low EVM and BER, and well-defined constellation patterns, indicate the reliability and effectiveness of the system. This bidirectional FSO-5G wireless communication system offers a high-speed and cost-effective solution for extending 5G coverage in both densely and sparsely populated areas.

Introduction

With the increasing demand for faster communication, higher data rates, and ultra-low latency in next-generation networks, particularly with the emergence of 5G, the development of innovative technologies has accelerated^1,2. 5G is expected to enable emerging technologies such as connected autonomous vehicles and augmented reality, which require simultaneous support for ultra-reliable, low-latency communication and enhanced mobile broadband^3,4. To meet these demands, 5G operates across both sub-6 GHz and millimeter-wave (MMW) bands^4,5. Sub-6 GHz bands offer wider coverage and better performance in sparsely populated areas, with the ability to penetrate obstacles such as walls and buildings⁶. However, the limited bandwidth of these bands is insufficient to meet the growing demand for higher data transmission rates. In contrast, MMW bands offer significantly faster data speeds and higher capacity, making them ideal for densely populated urban environments, although they are constrained by limited coverage and high susceptibility to attenuation from obstacles such as walls and buildings^5,7. These MMW bands, operating within the 24-GHz to 40-GHz range, offer substantial benefits but also present distinct challenges⁸. These limitations can be strategically exploited to enhance communication security, as their restricted coverage and susceptibility to obstacles reduce vulnerability to hacking, while enabling more efficient frequency reuse. To address the challenges of both sub-6 GHz and MMW bands, the integration of these systems has emerged as a promising solution. This integration combines the broad coverage and reliability of sub-6 GHz bands with the high capacity and speed of MMW band⁹. Deploying MMW base stations within sub-6 GHz coverage areas reduces transmission latency and enhances the system’s ability to support high-speed access and low-latency services. Furthermore, the use of multiple carrier frequencies improves the performance and scalability of 5G systems. By optimizing spectrum utilization, this approach facilitates the support of a wide range of services, from low-data-rate IoT applications to high-data-rate mobile broadband. Distributing traffic across different frequency bands allows for load balancing, reduces congestion, and enhances overall network stability. Additionally, operating across a broad spectrum of frequencies helps minimize interference and increases the system resilience to environmental factors, as lower frequencies can compensate for attenuation caused by adverse weather conditions. By integrating these advanced strategies, 5G communication systems can effectively allocate resources while maintaining an optimal coverage, capacity, and quality of service. This comprehensive framework is essential for supporting next-generation applications, including ultra-high-definition video streaming, augmented and virtual reality, and smart city technologies^7–10. However, despite the advantages of sub-6 GHz and MMW bands, these frequencies are still susceptible to atmospheric absorption and attenuation, which limits their transmission range¹⁰. Therefore, 5G sub-6 GHz/MMW communications may not be the optimal solution for providing long-range coverage.

Integrating free space optical (FSO) communication with short-distance 5G sub-6 GHz/MMW can effectively address these limitations, particularly concerning transmission coverage and reduce number of base stations of 5G communication^10–12. FSO employs modulated laser beams to transfer data across free space, enabling high-speed communication over long distances^13,14. Thus, integrating FSO with 5G sub-6 GHz and MMW communication can greatly expand the reach of 5G networks and improve both speed and coverage. FSO has rapidly emerged as a key foundation for the widespread connectivity required by 5G communication systems and the growing space/satellite communication^8,13. In 5G communication systems, FSO communication technology provides numerous benefits compared to radio frequency and microwave communication systems, including high bandwidth, long transmission range, low deployment costs, low power requirements, use license-free spectrum, easy to install, secured links, and reduced susceptibility to interference with low bit error rate (BER)¹⁵. Its ability to deliver point-to-point wireless communication over long distances, even in challenging environments, makes it a practical substitute for fibre-based communication systems, especially in regions where fibre installation is either impractical or prohibitively expensive such as rocks, mountains and rivers^16,17. Unlike fibre-optic communication, FSO does not require the installation of buried fibre cables for long-distance communication, making it a more cost-effective solution¹⁸. Despite these benefits, FSO is susceptible to atmospheric conditions including pointing errors, turbulence, and atmospheric attenuation, which can limit its transmission range¹⁷. Moreover, the equipment used in the FSO transmission link setup may introduce performance errors. However, these can be mitigated by integrating FSO with a 5G MMW link and through equipment upgrades¹⁹. For instance, the utilization of triplet lenses in place of doublet lenses can significantly enhance the transmission coverage of FSO transmission links. This advancement is essential for increasing the transmission coverage of FSO communication systems and addressing the inherent limitations of traditional short-range optical wireless links. Transmitting laser light over long distances presents challenges in fully coupling it into the fibre ferrule input. At the reception end, the triplet lens acts as a laser beam reducer, effectively shrinking the laser beam size to ensure optimal coupling into the fibre ferrule input compared to doublet lenses¹⁹. By mitigating challenges related to the dispersion of the laser beam and enhancing coupling efficiency, triplet lenses extend the range of FSO transmission links and facilitate reliable communication over longer distances. Consequently, the FSO integrated with 5G sub-6 GHz/MMW communication systems address the growing demand for high-capacity, high-speed wireless communication across numerous applications by combining the advantages of both technologies. As shown in Fig. 1, FSO-5G wireless communication systems integrate FSO and 5G sub-6 GHz/MMW wireless links for data transmission, providing reliable communication even in adverse weather conditions and through obstacles by utilizing alternative paths, while ensuring high-speed and secure transmission. The integration of high-speed FSO communication with 5G technology holds significant potential to revolutionize various industries by providing ultra-fast, reliable, and scalable connectivity. This convergence is poised to drive innovation across several applications, including enhanced mobile broadband, ultra-reliable low-latency communication, and massive machine-type communication. In the transportation sector, FSO-5G communication technology is expected to enhance autonomous vehicle systems by supporting vehicle-to-everything communication with ultra-low latency²⁰. This development will improve operational safety, traffic management, and overall driving efficiency. Furthermore, it will enable the growth of intelligent transportation systems, facilitating real-time traffic routing and autonomous driving, thereby reducing congestion and enhancing safety. Additionally, the IoT is set to benefit substantially from the FSO-5G communication system. This integration will support the expansion of smart cities, precision agriculture, and advanced smart grids by enabling massive device connectivity and real-time data exchange²¹. In the healthcare sector, FSO-5G will enable remote patient monitoring and facilitate the use of augmented and virtual reality for medical training and surgery^22,23. The ultra-low latency and high-speed capabilities of FSO-5G will ensure that real-time patient data can be transmitted swiftly, enhancing diagnosis and treatment decisions. Thus, the integration of FSO with 5G technology is anticipated to revolutionize critical sectors, including autonomous systems, immersive technologies, the IoT, smart city and healthcare^22,23.

Fig. 1.

Integration of FSO with 5G wireless links for high-speed bidirectional communication systems in densely and sparsely populated areas.

In our previous investigation, a WDM-FSO communication system has been demonstrated employing a 0.1-km FSO transmission link using doublet lenses²⁴. Besides, a dense WDM-FSO system utilizing an 8-wavelength configuration has also been reported to achieve a transmission distance of 0.05-km over a FSO transmission link using fibre collimators and convex lenses²⁵. However, the transmission ranges of these systems are limited to 0.1-km and 0.05-km due to the use of doublet lenses and fibre collimators, which are significantly shorter than the proposed 1.61-km achieved using triplet lenses. Furthermore, these systems do not establish a direct connection to 5G signals through FSO transmission links, which is essential for combining 5G communications with FSO communication systems. A distinctive feature of a 5G FSO system is its direct correlation with 5G networks. Therefore, in practical applications, the development of a 5G FSO communication system is essential, as opposed to a FSO communication system that lacks direct connectivity to 5G communications. In another of previous study, the four-level pulse amplitude modulation (PAM-4) signals transmission through a 0.0012-km FSO transmission link utilizing collimator lenses was proposed²⁶. However, this system transmission range is significantly shorter than the proposed 1.6-km achieved with triplet lenses. Besides, direct integration with 5G is not practical because the 5G physical layer relies on OFDM signals. To create a feasible connection with 5G communication, the high-speed FSO-5G wireless communication system should utilize OFDM signals instead of PAM-4. Additionally, the proposed FSO communication systems are unidirectional^24–26, allowing communication in only one direction, whereas a bidirectional FSO communication system facilitates transmission of signals in both downlink and uplink directions at the same time. This feature increases the reuse of free space and maximizes spectrum efficiency, ultimately contributing to higher transmission capacity and data rates. To overcome the limitations of these proposed systems, the researcher demonstrated a phase modulation bidirectional FSO communication system over a 0.6-km FSO transmission link, employing remotely injection-locked distributed feedback (DFB) laser diode (LD)²⁷. For this system, the transmission coverage was limited to 0.6-km due to the use of doublet lenses in the system. Additionally, the conversion from phase modulation to intensity modulation posed challenges due to the substantial costs involved with DFB LD. Similar to the FSO communication system that utilized PAM-4 modulation²⁶, this system also employed non-return-to-zero and PAM-4 signals for data transmission. However, it is important to note that the specifications set by the Third Generation Partnership Project currently exclude these types of signals. To overcome the limitations of the systems^24–27, we demonstrated a bidirectional FSO communication system using two reflective semiconductor optical amplifiers (RSOAs)¹. However, its transmission coverage was limited to 0.1-km due to use of a broadband light source (BLS), which is susceptible to noise and Rayleigh scattering, making it the more suitable for shorter distances²⁸. The proposed system overcomes this limitation by incorporating a high-power erbium-doped fibre amplifier (EDFA) to enhance signal power and by utilizing triplet lenses in place of doublet lenses, which together extend the transmission coverage of the FSO to 1.6-km. Additionally, the previous system relied solely on MMW frequencies for end user applications, which despite offering faster data speeds and higher capacity, face challenges associated to limited coverage and vulnerability to obstacles^8,9. To address these issues, integrating sub-6 GHz and MMW frequencies presents a promising solution. This approach utilizes the sub-6 GHz for broader coverage and better performance in sub-urban and rural areas while utilizing the high capacity of MMW for fast access in dense urban areas, thus reducing latency and ensuring seamless service. Furthermore, the proposed system is more cost-effective by utilizing fibre Bragg grating (FBG) sensors instead of the two RSOAs used in the previously proposed system for securing uplink transmission. Consequently, the proposed system enhances transmission capacity and distance coverage compared to previous bidirectional FSO communication systems. The key differences between our study and prior research are summarized in Table 1.

Table 1.

Comparison of our study with previously reported systems

Ref.	Bidirectional/ Unidirectional	5G signal	Medium/s	Transmission coverage (km)	Uplink wavelength reused mechanism
1	Bidirectional	5G MMW (Downlink) 5G MMW (Uplink)	FSO	0.1 (Downlink/ uplink)	Two cascaded RSOAs
24	Unidirectional	-	FSO	0.1	-
25	Unidirectional	-	FSO	0.05	-
26	Unidirectional		FSO	0.0012	-
27	Bidirectional	-	FSO	0.6 (Downlink/ uplink)	DFB LD
This study	Bidirectional	5G MMW/sub-6 GHz (Downlink) 5G MMW (Uplink)	Integrated FSO with 5G wireless links	1.61/1.612/1.615 (Downlink) 1.612 (Uplink)	FBG sensors

This paper presents a high-speed bidirectional FSO-5G wireless communication system, practically demonstrated with a high-power EDFA. The system utilizes three central wavelengths from FBG sensors for both downlink and uplink transmissions. As narrowband filters, the FBGs reflect light within a specific spectral range centred around the Bragg wavelength, enabling the transmission of various data rates corresponding to each Bragg wavelength. For downlink transmission, wavelengths of 1525 nm (FBG1), 1530 nm (FBG2), and 1534.5 nm (FBG3) are used to deliver 1-Gb/s/4.5-GHz and 10-Gb/s at both 24-GHz and 39-GHz 5G sub-6 GHz/MMW signals, utilizing 16-quadrature amplitude modulation (QAM) orthogonal frequency division multiplexing (OFDM) modulation. In this system, the FBG sensors act as specific wavelength selectors, replacing the need for tunable filters and thereby reducing system costs. Implementing a bidirectional communication system is essential for better spectrum utilization, leading to higher transmission capacities and data rates compared to a unidirectional system. To achieve this, WDM filters were employed. These filters split the light generated from a BLS into two parts to secure both downlink and uplink transmission. In the proposed system, a single broadband light source is employed, instead of multiple laser sources, for both optical signal transmission directions, which significantly reduces cost and complexity. After splitting, the three-downlink modulated and unmodulated signals are combined utilizing a 4 × 1 optical combiner and transmitted across a 1.6-km FSO transmission link. Another WDM filter is then used to separate the downlink modulated signals from the unmodulated signal. The unmodulated signal is directed to three serially cascaded FBG sensors. Each FBG sensor with centre wavelengths of 1545.6 nm (FBG4), 1546.9 nm (FBG5), and 1548.6 nm (FBG6) delivers 10-Gb/s/28-GHz (5G MMW) signals utilizing 16-QAM OFDM modulation, ensuring secure and cost-effective uplink transmission. Conversely, two cascaded RSOAs or three laser sources are typically required to secure uplink transmission in bidirectional FSO-5G wireless communication systems. However, the use of FBG sensors instead of two RSOAs and multiple laser sources offers a promising solution for cost reduction in practical system implementations. Moreover, signals transmitted over the 1.6-km FSO transmission link experience attenuation. However, the incorporated high-power EDFA in the proposed system mitigates signal degradation. The high-power EDFA amplifies the signal for transmission through the 1.6-km FSO transmission link, significantly improving system performance by boosting signal strength, reducing BER and error vector magnitude (EVM), and ensuring reliable extended-range transmission. A bidirectional FSO-5G wireless communication system capable of transmitting 16-QAM OFDM signals with an aggregated data rate of 21-Gb/s for downlink and 30-Gb/s for uplink. The system integrated a 1.6-km FSO transmission link with 10/15-m 5G wireless links, and exhibited good performance, as proved by low EVM and BER, and clear constellation patterns. This bidirectional FSO-5G wireless communication system offers a robust, scalable, and cost-effective solution for extending high-speed 5G coverage across both densely populated/sparsely populated areas.

Results

Impact of high power EDFA and triplet lens on the performance of a system

Figure 2illustrates the high-power EDFA configuration, designed for efficient amplification and transmission of optical signals through a 1.6-km FSO transmission link. The amplifier utilizes a two-stage structure with Erbium-doped fibre (EDF) in each stage. Each stage incorporates EDF, WDM couplers, and pump lasers. In the first stage, the input signal is combined with a forward 980 nm pump LD using a WDM coupler, directing the combined signal into the EDF for initial amplification. In the second stage, the amplified signal is coupled with backward 1480 nm pump LD via another WDM coupler, further enhancing the signal in the second EDF. To ensure stable and high-performance amplification, optical isolators are positioned at both the input and output of the amplifier to mitigate unwanted reflections. The EDFA operates by employing stimulated emission in the EDF to amplify optical signals, with EDF serving as the amplification medium. An EDF is an optical fibre doped with erbium ions, allowing for efficient signal amplification²⁹. When combined with WDM couplers, pump light sources, and optical isolators, the EDF enables stable, unidirectional operation by effectively blocking reflected light²⁹. This configuration is crucial for long-distance optical communication, particularly within WDM systems³⁰, where high signal integrity is needed. The most effective pump wavelengths currently used in EDFA are 980 nm and 1480 nm³¹. The 980 nm pump is favored for its low noise and high gain, making it particularly suitable for shorter EDF^31,32. The 1480 nm pump offers enhanced power efficiency, improving gain for longer EDFs while reducing amplified spontaneous emission (ASE)³¹. In two-stage EDFA amplifiers (high-power EDFAs), both 980 nm and 1480 nm pump wavelengths are utilized together. The 980 nm pump aids in noise reduction, while the 1480 nm pump enhances power conversion³². The performance of the EDFA is influenced by the direction of pumping. Forward pumping minimizes ASE noise, resulting in quieter and more stable amplification, while backward pumping increases output power but may also amplify noise. Bi-directional pumping, combining both forward and backward directions, achieves optimal performance by maximizing population inversion and balancing noise reduction with high output power³².High-power EDFAs offer significant advantages for modern optical networks³⁰. With their ability to provide high gain, they enable long-distance signal amplification without frequent regeneration, simplifying the system and improving efficiency. These amplifiers support wide optical bandwidth, allowing for the simultaneous amplification of multiple WDM channels, which is essential for high-capacity communication systems. The low-noise characteristics help preserve signal integrity over extended distances, while the 1480 nm pump improves power efficiency and reduces energy consumption. Additionally, their polarization-insensitive design eliminates the need for complex polarization management^32,33. One key benefit of high-power EDFAs is their capacity to extend transmission distances, especially in long-haul WDM systems, where maintaining high data rates is crucial³⁴. The pump laser power in high EDFAs typically ranges from 500 mW to 2000 mW, compared to 100 mW in standard EDFAs, providing greater amplification. The higher erbium-doped fiber concentration and two-stage design further enhance gain and output power. This combination of features ensures high-power EDFAs are essential for expanding optical networks and mitigating signal loss over long distances. These capabilities make high-power EDFAs a critical component in high-speed, long-distance optical communication systems, such as the 1.6 km FSO transmission link, facilitating stable and efficient data transmission across optical networks.

Fig. 2.

Configuration of two-stage high power EDFA amplifier.

The performance of high power EDFA depends on several key parameters, such as gain, noise figure, saturation output power, and ASE noise. These parameters are crucial in determining the efficiency and signal integrity of the amplifier at different operating wavelengths. A summary of these parameters, based on 10 Gb/s / 24 GHz at different wavelengths, is provided in Table 2.

Table 2.

Values of different parameters for high-power EDFA using 10-Gb/s/24-GHz at various wavelengths.

Parameter	1530 nm	1550 nm	1560 nm
Gain	38	37	35
Noise figure (dB)	4.5	5.0	5.5
Saturation output power (dBm)	36	35	34
ASE noise (mW)	0.1	0.2	0.3

FSO communication systems: challenges and solutions

FSO communication systems encounter significant challenges due to environmental factors such as rain, fog, snow, and atmospheric turbulence. These adverse conditions cause attenuation, scattering, and absorption of optical signals, leading to a reduction in signal strength and an increase in transmission errors. Atmospheric attenuation can vary considerably, from approximately 2.7 dB under clear weather conditions to as high as 50 dB in severe weather conditions over a 1.6-km FSO link²¹. Moreover, fluctuations in atmospheric pressure and refractive indices can result in optical beam divergence or convergence, complicating signal alignment and focus³⁵. In this demonstration, the 1.6-km FSO link experiences a total loss of 5.25 dB, comprising 2.74 dB from atmospheric attenuation and turbulence, and 2.51 dB from atmospheric link loss due to background light noise. To mitigate performance degradation under adverse weather conditions, advanced techniques such as zero-cross correlation coding³⁶and FSO/MMW hybrid backup systems³⁷are essential to ensure reliable operation. Our previous research involved a bidirectional convergent system designed to assess resilience under various environmental conditions, integrating multiple transmission media: a 20-km SMF, a 500-m FSO link, and a 1/4-m RF wireless link using doublet lenses³⁶. A laboratory-designed weather simulator was employed to replicate adverse conditions such as heavy rain, fog, and snow. Under clear weather, the system demonstrated excellent performance, characterized by low BER, clear eye diagrams, and well-defined constellation patterns, indicating high reliability. However, adverse weather conditions caused significant performance degradation, evidenced by increased BER, blurred eye diagrams, and distorted constellation patterns. The coverage of the FSO link was notably limited when doublet lenses were used, with the maximum free-space transmission distance reaching only 595.4 m. In contrast, triplet lenses extended the coverage to 1.6-km by effectively minimizing signal dispersion and preserving beam quality The lens design played a pivotal role in maintaining the reliability of the FSO link. Triplet lenses, comprising two convex lenses and one concave lens, were employed at the transmitter and receiver ends. These lenses exhibited superior aberration correction compared to doublet lenses³⁸, enhancing signal focus and minimizing dispersion. This improvement extended the maximum free-space transmission coverage from 595.4 m (using doublet lenses) to 1.6-km (using triplet lenses). Precise alignment of the laser beam into the fibre ferrule input was critical, as any misalignment resulted in significant coupling losses, increased BER, and EVM³⁹. Misalignment was mitigated by positioning the fibre ferrule input accurately at the focal point of the triplet lens. To enhance the FSO communication system reliability, various advanced strategies were considered. These included adaptive optics, multi-beam transmission, and hybrid FSO/MMW systems³⁵. Additionally, multiple-input multiple-output (MIMO) optical channel technology was explored to address signal degradation by leveraging multiple optical paths^25,40,41. However, MIMO technique comes with increased complexity and higher deployment cost. Despite these challenges, FSO communication systems remain attractive due to their high data rates, flexibility, and potential for seamless integration with 5G wireless networks. By combining advanced lens designs, hybrid transmission strategies, and resilient system architectures, FSO technology offers reliable performance and emerges as a promising solution for next-generation communication networks. In previous studies, we demonstrated the feasibility of constructing fibre-FSO communication systems using cascaded MZM-optoelectronic oscillators⁴². In these systems, a 1.6-km FSO link experienced atmospheric attenuation of approximately 2.7 dB and a 2.4 dB loss due to background light noise, resulting in a total loss of 5.1 dB. Despite these losses, the implementation of triplet lenses ensured sufficient link accessibility. The use of triplet lenses, combined with precise alignment techniques and hybrid transmission strategies, significantly enhances system reliability by extending the coverage and performance of FSO links. Additionally, integrating triplet lenses into FSO systems improves light-focusing capabilities, reduces signal dispersion, and increases the maximum free-space transmission range, all while optimizing energy efficiency. These advancements make FSO systems more cost-effective and reliable for next-generation communication networks.

The BERs and EVMs of the bidirectional FSO-5G wireless communication system using different data rates

Figure 3a shows the BERs performance of 16-QAM OFDM signals at different data rates, measured under various received sub-6 GHz/MMW power. In this experiment, the BERs of a bidirectional FSO-5G wireless communication system were analysed at different received sub-6 GHz/MMW power using an integrated 1.6-km FSO transmission link and 10/15-m 5G wireless links. Both downlink and uplink transmission were evaluated, considering various data rates and the impact of using a high-power EDFA in the systems. Additionally, FBG sensors were used as wavelength selectors for both downlink and uplink transmission. In the downlink, the selected wavelengths were 1525 nm (FBG1), 1530 nm (FBG2), and 1534.5 nm (FBG3), while for the uplink, the selected wavelengths were 1545.6 nm (FBG4), 1546.9 nm (FBG5), and 1548.6 nm (FBG6). The use of FBG sensors for wavelength selection offers a promising solution by combining precise wavelength filtering with cost-effectiveness. The experimental results show no occurrence of crosstalk between the selected wavelengths, indicating that wavelength crosstalk did not adversely affect system performance. This demonstrates the effectiveness of the wavelength separation and filtering mechanisms, which successfully prevented interference between the different wavelength channels. In the downlink, the 1-Gb/s/4.5-GHz and 10-Gb/s at both 24-GHz and 39-GHz 5G sub-6 GHz/MMW 16-QAM OFDM signals achieved BERs of 3.8 × 10⁻³, 3.8 × 10⁻³, and 3.7 × 10⁻³ at received sub-6 GHz/MMW power of 2.19 dBm, 1.91 dBm, and 1.78 dBm, respectively. The BERs for the 1-Gb/s/4.5-GHz and 10-Gb/s at 24-GHz and 39-GHz signals were significantly reduced to optimal BERs of 8.32 × 10⁻⁶, 1.78 × 10⁻⁵, and 6.76 × 10⁻⁵ at received sub-6 GHz/MMW power of 2.46 dBm, 2.52 dBm, and 2.8 dBm, respectively. These very low BERs indicate the significant impact of the high-power EDFA on system performance. For uplink transmission, BERs of 3.8 × 10⁻³, 3.7 × 10⁻³, and 3.9 × 10⁻³ were achieved at received sub-6 GHz/MMW power of 1.99 dBm, 2.03 dBm, and 2.09 dBm for the 10-Gb/s/28-GHz signal using FBG4 (1545.6 nm), FBG5 (1546.9 nm), and FBG6 (1548.6 nm), respectively. The optimal BERs of 2.14 × 10⁻⁵, 3.16 × 10⁻⁵, and 3.39 × 10⁻⁵were achieved at received sub-6 GHz/MMW power of 2.56 dBm, 2.58 dBm, and 2.69 dBm, respectively. These findings indicate that the optical sensitivity to received sub-6 GHz/MMW power progressively diminishes as the wavelength move toward longer wavelengths⁴³.

Fig. 3.

Obtained BERs and EVMs of the FSO-5G wireless communication system in various scenarios. (a) BERs for downlink and uplink transmissions. (b) EVMs for downlink and uplink transmissions.

Moreover, Fig. 3b presents the EVMs for downlink signals of 1-Gb/s/4.5-GHz and 10-Gb/s at both 24-GHz and 39-GHz 16-QAM OFDM signals, and the uplink performance of 10-Gb/s/28-GHz 16-QAM OFDM signals, all evaluated at various received sub-6 GHz/MMW power. In the downlink scenario, the signals of 1-Gb/s/4.5-GHz and 10-Gb/s at both 24-GHz and 39-GHz were transmitted over a 1.6-km FSO transmission link integrated with 10/15-m 5G wireless links. This configuration achieved EVMs of 9.24%, 9.6%, and 11%, corresponding to received sub-6 GHz/MMW power of 2.51 dBm (1525 nm), 2.55 dBm (1530 nm), and 2.65 dBm (1534.5 nm), respectively. Additionally, for downlink signals transmitted at EVM values of approximately 12.25%, 12.32%, and 12.64%, the received sub-6 GHz/MMW power were 0.31 dBm (1525 nm), 0.44 dBm (1530 nm), and 0.94 dBm (1534.5 nm). This resulted in power penalties of 0.13 dB and 0.63 dB for the 1-Gb/s/4.5-GHz signal compared to the 10-Gb/s/24-GHz and 10-Gb/s/39-GHz signals, respectively. Furthermore, a power penalty of 0.50 dB was observed for the 10-Gb/s/24-GHz signal compared to the 10-Gb/s/39-GHz signal. In the uplink scenario, 10-Gb/s/28-GHz signal was transmitted across a 1.6-km FSO transmission link hybrid with a 15-m 5G wireless link, achieving EVMs of 9.8%, 10%, and 10.4% at received sub-6 GHz/MMW power of 2.57 dBm (1545.6 nm), 2.58 dBm (1546.9 nm), and 2.6 dBm (1548.6 nm), respectively. When the uplink signals were transmitted, EVM values of 12.38%, 12.44%, and 12.5% were achieved, the received sub-6 GHz/MMW power were 0.5 dBm (1545.6 nm), 0.63 dBm (1546.9 nm), and 0.78 dBm (1548.6 nm). This resulted in power penalties of 0.13 dB and 0.28 dB were obtained for the 10-Gb/s/28-GHz signals at 1545.6 nm compared to10-Gb/s/28-GHz signals at 1548.6 nm, while a 0.15 dB penalty was noted between the 10-Gb/s/28-GHz at 1546.9 nm and10-Gb/s/28-GHz signals at 1548.6 nm. These power penalty differences arise from the transmission of 10-Gb/s/28-GHz signals at distinct wavelengths. The findings indicate a gradual decline in optical sensitivity to received sub-6 GHz/MMW power as the wavelength shift toward longer wavelengths. Factors that contribute to these optical power penalties arise from atmospheric attenuation over the 1.6-km FSO transmission link and connection losses occurring when the laser beam enters the fibre ferrule. Similar to the obtained BERs, the findings confirmed that optical sensitivity to received sub-6 GHz/MMW power decreases as the wavelength shifts to longer wavelengths. The overall performance across both downlink and uplink transmissions demonstrates the viability of FBG sensors as effective devices for wavelength selection and for reducing the costs associated with uplink wavelength reuse mechanisms.

The relationship between the displacement of the fibre ferrule centre from the triplet lens and the resulting BER for the 10-Gb/s/39-GHz downlink signal highlights the critical importance of precise laser light alignment in optical wireless systems, particularly in FSO transmission links. As shown in Fig. 4, displacements of less than 1.5 mm ensure that the BER remains below the forward error correction (FEC) threshold, which is crucial for maintaining reliable transmission quality. The sensitivity of the system to displacement becomes more pronounced with increasing data rates. For high data rates like 10-Gb/s/39-GHz, even slight misalignments greater than 1.5 mm led to significant performance degradation, pushing the BER above the FEC threshold. This confirms that the alignment of laser light is crucial for the performance and integrity of FSO transmission links, emphasizing the critical role of alignment in achieving optimal system performance.

Fig. 4.

BER performance of 10-Gb/s/39-GHz downlink signal with displacement of the fibre ferrule centre variations from 0 to 3 mm.

The performance of the system was analysed by examining constellation patterns and electrical spectra in different scenarios

Figure 5 presents the constellation patterns for the transmission of 1-Gb/s/4.5-GHz and 10-Gb/s at both 24-GHz and 39-GHz for the downlink and a 10-Gb/s/28-GHz (1545.6 nm) for the uplink utilizing 16-QAM OFDM modulation. These signal transmissions were evaluated at a BER of 3.8 × 10⁻³ over 1.6-km FSO transmission link integrated with 10/15-m 5G wireless links. The downlink signal of 1-Gb/s/4.5-GHz (Fig. 5a) exhibits the clearest constellation pattern, indicating minimal errors due to lower noise and distortion. In contrast, the 10-Gb/s/24-GHz signal (Fig. 5b) shows a slightly clearer constellation pattern than the 10-Gb/s/39-GHz signal (Fig. 5d). The 10-Gb/s/39-GHz signal exhibits progressively less defined constellations due to increased noise and distortion. For the uplink, the 10-Gb/s/28-GHz signal (Fig. 5c) shows a reasonably well-defined constellation pattern. However, it is less clear compared to the downlink signals at lower frequencies, such as 1-Gb/s/4.5-GHz and 10-Gb/s/24-GHz. Notably, the 10-Gb/s/39-GHz signal (Fig. 5d) demonstrates the most degraded pattern among the tested frequencies, emphasizing a general trend were higher frequencies result in poorer constellation clarity due to increased phase noise and distortion. These findings highlight the trade-off between achieving higher data rates and mitigating noise and distortion at higher frequencies. While higher frequencies are critical for supporting high-capacity transmission, they introduce greater phase noise, which negatively impacts signal quality and constellation clarity^44,45. To address these challenges, deep learning techniques⁴⁶and residual carrier modulation-enabled carrier phase recovery⁴⁷ are employed to enhance the clarity of the constellation patterns. Conversely, lower-frequency signals maintain better-defined constellations at the same BER of 3.8 × 10⁻³ because they introduce lower phase noise.

Fig. 5.

The constellation patterns of 16-QAM OFDM signals (a) 1-Gb/s/4.5-GHz (downlink), (b) 10-Gb/s/24-GHz (downlink), (c) 10-Gb/s/28-GHz (uplink), and (d) 10-Gb/s/39-GHz (downlink).

Figure 6 demonstrates the electrical spectra of 1-Gb/s/4.5-GHz (1525 nm, downlink), 10-Gb/s/28-GHz (1545.6 nm, uplink), and 10-Gb/s/39-GHz (1534.5 nm, downlink) 16-QAM OFDM signals over a 1.6-km FSO transmission link integrated with 10/15-m 5G wireless link with received sub-6 GHz/MMW power of 2.46 dBm, 2.56 dBm, and 2.8 dBm, respectively. The 1-Gb/s/4.5-GHz downlink signal (Fig. 6a) shows minimal amplitude fluctuations of ± 2.53 dB. Additionally, the 10-Gb/s/28-GHz uplink signal (Fig. 6b) exhibits amplitude fluctuations of ± 4.41 dB, which are larger compared to the 1-Gb/s/4.5-GHz downlink signal but smaller compared to the 10-Gb/s/39-GHz downlink signal. Additionally, due to increased transmission losses at higher frequencies, the 10-Gb/s/39-GHz downlink signal (Fig. 6c) exhibits slightly greater power within the 34–44-GHz range compared to the 0–5-GHz and 23–33-GHz ranges, respectively. When using the 39-GHz frequency, the downlink signal experiences large amplitude fluctuations within ± 5.24 dB. The results confirm that the performance of system deteriorated at higher carrier frequencies due to increased phase noise, with larger amplitude fluctuations observed as the frequency increased, even when different data rates transmitting through the same transmission medium.

Fig. 6.

Electrical spectra of various data rates using 16-QAM OFDM modulation (a) 1-Gb/s/4.5-GHz (downlink), (b) 10-Gb/s/28-GHz (uplink), and (c) 10 Gb/s/39-GHz (downlink).

Discussion

In our configuration, we used triplet lenses at both the receiver and transmitter ends of the FSO transmission link with a focal length of 146-mm and a diameter of 88.6-mm. These triplet lenses contain three components such as a concave lens and two convex lenses, combined together to optimize performance. With a fibre numerical aperture of 0.14, the diameter (Y) of the laser beam is determined by:

1

The laser beam diameter (YR) is 40.88-mm, which is less than the 88.6-mm diameter of the first triplet lens. For the successful deployment of an FSO transmission link using the second triplet lens, it is important that YR remains less than the diameter of the second triplet lens for the distance coverage (R).

2

The divergence angle (θ) describes how much the laser beam expands as it transmits through free space. The computed value for R is 1.604-km, demonstrating that the highest possible distance for the FSO transmission link is 1.604-km.

The computed value for the maximum FSO transmission distance, 1.604-km, represents the optimal range based on the current system configuration, including the specifications of the lenses and associated parameters. This distance is derived from factors such as the laser beam diameter, divergence angle, and the efficiency of the optical system. However, it is important to note that this value does not imply a strict limit but rather denotes the maximum achievable distance under the given system conditions. As the transmission distance increases, the power loss also tends to increase. This is due to the natural divergence of the laser beam, which causes a reduction in intensity, and the additional attenuation introduced by atmospheric conditions, such as scattering and absorption. Therefore, a longer transmission distance generally results in greater signal degradation, impacting the overall performance of the FSO link. While increasing the transmission power may help mitigate some of these losses and extend the range, it is not a straightforward solution. Although higher power can compensate for attenuation over longer distances, the divergence of the laser beam and the persistence of atmospheric losses remain significant challenges. Thus, simply boosting the transmission power is unlikely to achieve substantial distance increases without addressing other factors such as beam divergence and environmental interference. To extend the FSO transmission distance beyond the calculated 1.604-km, several strategies could be considered. These include optimizing the system design, such as improving lens configurations or focusing mechanisms, or implementing advanced techniques like adaptive optics to counteract atmospheric distortions. Additionally, increasing transmitter power and enhancing receiver sensitivity could provide further improvements. However, the ultimate transmission distance will always depend on a combination of system parameters, environmental conditions, and the effectiveness of mitigating strategies. Therefore, while 1.604 km represents the theoretical maximum transmission coverage under optimal conditions, the actual achievable range may vary due to real-world factors such as weather conditions, system alignment, and atmospheric disturbances.

Methods

Experimental demonstration of high-speed bidirectional FSO-5G wireless communication system

Figure 7a demonstrates the experimental configuration of high-speed bidirectional FSO-5G wireless communication system utilizing high-power EDFA, we employed the BLS to generate light. Figure 7b shows the output spectrum of the BLS. The light generated by the BLS is split into two parts using a 1 × 2 wavelength division multiplexing filter (F), separating it into the 1520–1539 nm and 1540–1560 nm wavelength ranges. This filter is used for wavelength separation and does not function as a power divider. In the experimental setup, the WDM filter is employed to split signals for downlink and uplink transmission purposes. This filter offers advantages such as compact size, a wide operating wavelength range, excellent stability, and low insertion loss, enabling the system to transmit both signals simultaneously. The light within the wavelength range of 1540–1560 nm is transmitted through the upper path and enters a polarization controller (PC) to adjust the polarization of the light. Additionally, the light within the wavelength range of 1520–1539 nm is transmitted through the lower path. For downlink signal transmission, the signal in the lower path is first amplified by an EDFA to support further propagation. It then passes through a 1 × 4 power splitter, which divides the signal into four paths. The signal power is approximately equal across each path. Then, the signal propagates through path 1, path 2, and path 3, entering FBG1 (with a centre wavelength of 1525 nm), FBG2 (with a cntere wavelength of 1530 nm), and FBG3 (with a centre wavelength of 1534.5 nm), respectively. The wavelength differences between FBG1 and FBG2, as well as between FBG2 and FBG3, were 5 nm and 4.5 nm, respectively, as illustrated in Fig. 7b. This means that the crosstalk of the wavelengths has no impact on the system performance during the transmission of different data signals through the system. Hence, each FBG, along with the optical circulator (OC) is used to select a single wavelength within the range of 1520–1539 nm with small optical power loss. After selecting the single wavelength using each FBG with OC, the signal transmitted through each path enters to PC. Then, the signal transmitted through each path modulated by MZM. The 10-Gb/s/39-GHz, 10-Gb/s/24-GHz, and 1-Gb/s/4.5-GHz drives an MZM via a modulator driver. Then, the unmodulated signal transmitted through path1 after transmitted through PC combined with the modulated signal transmitted through another three paths using 41 power combiner. Figure 7c illustrates the combination of modulated and unmodulated optical signals after a 4 × 1 optical combiner. As observed in the figure, the modulated and unmodulated signals do not interfere with each other because the WDM filter effectively separates them. This allows simultaneous downlink and uplink transmissions using a single BLS without mutual interference. Then, the signal is amplified by using high power EDFA for further transmission of signal through an OC4 and 1.6-km FSO transmission link using a triplet lens and a parabolic optical disk antenna. In this setup, the atmospheric attenuation for a 1.6-km FSO transmission link is about 2.74 dB. After passing through a triplet lens, the signals transmitted through OC5 and are split into two paths using 1 × 2 F. Then, the signal transmitted through the upper port of F path again split into four paths using 1 × 4 power splitter. Subsequently, the signal from the four paths is transmitted through variable optical attenuators (VOAs) to regulate its power before being detected by three photodiodes (PD1 (Optilab PD-40), PD2 (Optilab PR-30-A), and PD3 (Thorlabs DET08CFC)). These photodiodes play a critical role in converting optical signals into electrical signals, ensuring precise signal processing. PD1, with a bandwidth of 0 to 40-GHz, detects 10-Gb/s at 39-GHz and 24-GHz signals, as well as a 1-Gb/s/4.5-GHz signal. PD2, with a bandwidth of 0 to 30-GHz, detects a 10-Gb/s/24-GHz signal and a 1-Gb/s/4.5-GHz signal while effectively filtering out the 10-Gb/s/39-GHz signal. PD3, with an operational bandwidth of 1 to 5-GHz, detects the 1-Gb/s/4.5-GHz signal while filtering out 10-Gb/s at both 39-GHz and 24-GHz signals. The electrical signals obtained from the photodiodes are subsequently amplified by power amplifiers (PAs). PA1, operating within the 37.5 to 42.5-GHz range, amplifies the 10-Gb/s/39-GHz signal while filtering out 10-Gb/s/24-GHz signals and 1-Gb/s/4.5-GHz signals. Similarly, PA2, with an operational range of 17 to 24-GHz, amplifies the 10-Gb/s/24-GHz signal while eliminating the 1-Gb/s/4.5-GHz signal. Finally, PA3, with an operational range of 4.4 to 5-GHz, amplifies the 1-Gb/s/4.5-GHz signal. Through lower path, after the signal is amplified by PA1, the 10-Gb/s/39-GHz 5G MMW signal is wirelessly transmitted using a pair of Ka-Band HA with a range of 26.5 to 40-GHz. Then, a 10-Gb/s/39-GHz signal is down-converted into a 10-Gb/s/5-GHz signal using a mixer with a 17-GHz local oscillator signal multiplied by 2 and then amplified using low noise amplifier (LNA1) with a range of 6 to 20-GHz. After amplifying the power by using LNA1, an OFDM receiver receives the signal. Similarly, the signal after amplified by PA2, the 5G-MMW 10-Gb/s/24-GHz signal is transported wirelessly through the 15-m 5G MMW wireless link using a pair of K-band horn antennas (HAs) with a range of 18–26.5-GHz. Through a 15-m MMW wireless transport, a 10-Gb/s/24-GHz signal is down-converted into a 10-Gb/s/5-GHz signal using a mixer with a 19-GHz local oscillator signal and then amplified using LNA2 with a range of 6 to 20-GHz. Moreover, after amplifying the signal by using PA3, 1-Gb/s/4.5-GHz signal is transmitted wirelessly using a pair of flat panel antennas with an operational range of 4.4 to 5.1-GHz over a 10-m 5G sub-6 GHz wireless link. Then, the signal amplified by a LNA3 with an operational range of 3.8 to 5-GHz. After amplifying the power by using LNA1, LNA2, and LNA3, a signal transmitted through each path received by OFDM receiver. Finally, using MATLAB to evaluate the BER, EVM and corresponding constellations of the OFDM receiver should provide valuable insights into the effectiveness of the system.

Fig. 7.

(a) Experimental setup of the high-speed bidirectional FSO-5G wireless communication system. (b) Optical spectrum of the BLS, (c) optical spectrum after combining modulated and unmodulated signals, (d) unmodulated optical spectrum of three FBG sensors (uplink), and (e) modulated optical spectrum of three FBG sensors (uplink).

For uplink signal transmission, the unmodulated signal transmitting through the lower port of the F path is filtered by the three FBGs (FBG4, FBG5, and FBG6), with centre Bragg wavelengths of 1545.6, 1546.9, and 1548.6 nm, respectively, as shown in Fig. 7d. Here, the wavelength differences between FBG4 and FBG5, and between FBG5 and FBG6, are 1.3 nm and 1.7 nm, respectively. Figure 7e shows the optical spectrum of the three FBG-filtered signals after modulation. The filtered signal is then fed into a MZM for modulation, driven by a 10-Gb/s/28-GHz 16-QAM OFDM signal via a modulator driver. The modulated optical signal is subsequently amplified using an EDFA, and a VOA is employed to regulate the amplified signal. After passing through the VOA, the signal circulates through OC5 and is transmitted across a 1.6-km FSO transmission link. The optical signal then circulates via OC4 and is directed to 1 × 4 optical power splitter, which divides the signal into four channels before being detected by a 30-GHz PD4 (Optilab PR-30-A). Then, the optical signal transmitted through one of the four paths is captured by PD4, which converts the optical signal into an electrical signal. After conversion to electrical signal, the 10-Gb/s/28-GHz signal is amplified by the PA4 with an operational range of 27 to 31-GHz. Then, the 10-Gb/s/28-GHz 5G MMW signal is wirelessly transmitted using a pair of Ka-Band HA with a range of 26.5–40-GHz. Through a 15-m MMW wireless transport, a 10-Gb/s/28-GHz signal is down-converted into a 10-Gb/s/14-GHz signal using a mixer with a 14-GHz local oscillator signal, and then amplified using LNA4 with a range of 6 to 20-GHz. After amplifying the power by using LNA4, a signal transmitted through path received by OFDM receiver. Finally, OFDM receiver is being used to analyse the system performance using MATLAB in terms BER, EVM, and the corresponding constellation patterns.

Author contributions

Stotaw Talbachew Hayle, Hua-Yi Hsu, Chia-Peng Wang, and Hai-Han Lu contributed to the experiment design.Jia-Ming Lu, Wei-Wen Hsu, Yu-Chen Chung, Yu-Yao Bai, and Kelper Okram contributed to the experimental construction and measurement.Stotaw Talbachew Hayle, Hua-Yi Hsu, Chia-Peng Wang, Hai-Han Lu, and Jia-Ming Lu contributed to the data analysis.Stotaw Talbachew Hayle, Hua-Yi Hsu, Chia-Peng Wang, and Hai-Han Lu contributed to the manuscript writing.

Data availability

The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.

Declarations

Competing interests

The authors declare no competing interests.