Brute-force attack mitigation on remote access services via software-defined perimeter

Abstract

Remote Access Services (RAS)—including protocols such as Remote Desktop Protocol (RDP), Secure Shell (SSH), Virtual Network Computing (VNC), Telnet, File Transfer Protocol (FTP), and Secure File Transfer Protocol (SFTP)—are essential to modern network infrastructures, particularly with the rise of remote work and cloud adoption. However, their exposure significantly increases the risk of brute-force attacks (BFA), where adversaries systematically guess credentials to gain unauthorized access. Traditional defenses like IP blocklisting and multifactor authentication (MFA) often struggle with scalability and adaptability to distributed attacks. This study introduces a zero-trust-aligned Software-Defined Perimeter (SDP) architecture that integrates Single Packet Authorization (SPA) for service cloaking and Connection Tracking (ConnTrack) for real-time session analysis. A Docker-based prototype was developed and tested, demonstrating no successful BFA attempts observed, latency reduction by above 10% across all evaluated RAS protocols, and the system CPU utilization reduction by 48.7% under attack conditions without impacting normal throughput. It also proved effective against connection-oriented attacks, including port scanning and distributed denial of service (DDoS) attacks. The proposed architecture offers a scalable and efficient security framework by embedding proactive defense at the authentication layer. This work advances zero-trust implementations and delivers practical, low-overhead protection for securing RAS against evolving cyber threats.

Introduction

Remote Access Services (RAS) have become indispensable tools for managing network resources securely from remote locations. The RAS encompasses protocols and applications such as Remote Desktop Protocol (RDP), Secure Shell (SSH), Virtual Network Computing (VNC), Telnet, File Transfer Protocol (FTP), and Secure File Transfer Protocol (SFTP). Primarily, RAS facilitates vital remote work, system administration, and collaboration within modern distributed organizations^1–3. Due to the global transition towards remote working environments and widespread cloud adoption, the reliance on these services has dramatically increased, resulting in heightened exposure to cybersecurity threats, notably brute-force attacks (BFA)^1,2. BFA involves attackers systematically employing automated trial-and-error techniques to compromise sensitive authentication credentials^4–8. A simplified Fig. 1 illustrates a typical RAS system architecture and how a typical BFA works. Recent studies indicate a dramatic surge of up to 400% since 2023 in BFA campaigns targeting RAS⁹, contributing significantly to ransomware incidents, data breaches, and financial losses exceeding $2.5 billion annually¹⁰. High-profile breaches have targeted organizations, including GitHub, Alibaba, Firefox, and the Northern Irish Parliament¹¹. In light of such threats’ increased complexity and persistence, there is a pressing need to tighten cybersecurity protocols and governance strategies tailored to secure RAS infrastructures against evolving BFA vectors.

Fig. 1.

Workflow of a BFA within the architecture of a RAS system.

Existing defense strategies have important vulnerabilities. Known methods like account lockout policy can inhibit repeated guessing on individual accounts, but not if attackers randomly switch accounts or even source IP addresses. IP blocking (for example, Fail2Ban^12,13), rate limiting⁶, and multifactor authentication (MFA)^14,15 mainly use reactive tactics based on defensive action on multiple failure occurrences. Accordingly, it allows attackers to make initial attempts, enabling distributed attack evasion. Static limits have a problem with the configuration, where being too tightly configured ends up inconvenient for honest users, and being too loose grants attackers enough attempts. Although MFA strengthens security orders of magnitude, it still faces problems such as poor adoption and vulnerability to network-level attacks such as resource exhaustion attacks. These limitations highlight the need for more proactive and context-based cybersecurity strategies.

As a solid/resilient alternative to plug these significant gaps, SDP¹⁶, built on the zero-trust security model¹⁷, steps in. The SDP was originally put forward by the Cloud Security Alliance (CSA) in 2013; where SDP follows the principle of ‘authenticate first, connect later’, dynamically wrapping protected services with cryptographically enforced micro-perimeters not visible to unauthorized users^18–20. The authenticated users authenticate with an SDP controller that dynamically establishes gateways per access rights. Unauthorized access is disregarded (silently being dropped without feedback/acknowledgment), reducing the attack surface area. Henceforth, it facilitates mitigation of port scanning, denial of service (DoS), distributed denial of service (DDoS), man-in-the-middle (MitM), and BFA.

Despite these SDP’s great strengths, its convectional deployment has inherent challenges. Initially, the authentication mechanisms (for instance, SPA and mutual TLS mTLS) are cryptographically sound. Yet, they are vulnerable to consistent malicious connection attempts and can induce resource exhaustion or DoS/DDoS scenarios²¹. Furthermore, enriching SDP controllers with legacy username/password authentication for users’ convenience creates vulnerabilities to credential guessing at the SDP controller layer^22–24. On the other hand, traditional SDP realizations often do not provide sufficient and sophisticated intrusion detection and response capabilities, usually limiting the available defenses to binary allow/deny decisions, which use static, not dynamic, analyses of attacks^25,26. As a result, persistent malicious traffic—especially insider threats or continuous attack vectors—can still create significant network noise or overload the SDP infrastructure. Lastly, while deploying SDP in legacy RAS, particularly in the setup stage, for instance, RDP, transitional attack scenarios/vulnerabilities or configuration errors are likely to happen that can expose services to BFA^27,28.

Motivated by these critical gaps, this paper introduces an Enhanced SDP Architecture optimized for proactively securing RAS against BFA. The proposed architecture uniquely integrates SPA for cryptographic invisibility of services, Connection Tracking (ConnTrack) for real-time session monitoring, and leveraging the Snort Intrusion Detection and Prevention System (IDPS)²⁹ for dynamic anomaly detection. This multi-layered approach proactively detects and mitigates credential stuffing, BFA attempts, and distributed attacks by continuously verifying authentication and rapidly responding to emerging threats. A lightweight, Docker-based SDP prototype was implemented and empirically evaluated against realistic BFA and DoS/DDoS scenarios to validate this framework. The experiments demonstrated complete mitigation of BFA targeting prevalent protocols (RDP, SSH, VNC, Telnet, FTP, and SFTP), significantly reduced resource utilization, and minimal latency impact during active threat conditions, preserving seamless performance and user experience under typical usage.

In summary, this work makes the following core contributions:

• Designing and implementing a scalable multi-layered SDP Architecture leveraging cryptographic invisibility (via SPA), real-time session monitoring (ConnTrack), and dynamic threat detection (Snort IDPS) prototyped as a lightweight docker container explicitly tailored for proactive BFA mitigation on RAS (across six (6) protocols (RDP, SSH, VNC, Telnet, FTP, SFTP). This architecture eliminates protocol exposure to unauthorized entities and detects brute-force attempts in real-time.

• Performance and security empirical validation of the enhanced SDP architecture against realistic BFA/DDoS scenarios on RAS (across six (6) protocols (RDP, SSH, VNC, Telnet, FTP, SFTP). To evaluate and analyze the enhanced SDP’s security efficacy and resource efficiency comparatively with the conventional BFA reactive defenses on RAS.

• Providing practical insights and guidance for effective real-world implementation and adoption of zero-trust principles (via lightweight and proactive approaches such as SDP dockers templates and playbooks) in operational network environments to reduce incident response and deployability time.

The remainder of the paper is structured as follows: Section “2” presents related work summarizing previous research; Section “3” outlines the detailed design of the proposed SDP architecture and its security components; Section “4” describes the experimental setup, procedures, and comprehensive evaluations; Section “5” discusses findings, implications, and limitations; finally, Section “6” concludes the paper while highlighting the identified future research directions.

Related work

Existing brute-force attack mitigation techniques

BFA systematically attempts numerous password combinations or encryption keys to compromise user authentication, posing a significant threat to RAS. Several techniques have been proposed for mitigation, each presenting unique strengths and associated limitations as described hereafter.

Password policies, enforcing complexity and frequent updates, are commonly employed due to ease of implementation but often inconvenience users and remain vulnerable to sophisticated dictionary attacks^30–32. Account lockout mechanisms prevent repeated attempts by temporarily deactivating accounts after multiple failures, effectively hindering BFA efforts but risking DoS scenarios and requiring significant administrative management^33–35.

MFA significantly improves security by requiring multiple credentials (for example, tamper-resistant applications³⁶, hardware tokens, or SMS verification). Despite reducing credential compromise risks, MFA introduces usability challenges and susceptibility to phishing and social engineering^37–40.

Network-based solutions like rate limiting and IP blocklisting/allowlisting reduce attack frequency. However, rate limiting can inconvenience legitimate users and is bypassed by distributed attacks^4,41,42. IP-based mechanisms (for example, Fail2Ban) effectively block known malicious addresses but are limited against IP spoofing, dynamically allocated addresses, and distributed botnet attacks, creating extensive administrative overhead^43,44

Recent advancements leverage machine learning (ML) and deep learning (DL), analyzing network telemetry^7,8,45,46, or behavioral biometrics such as typing patterns^7,47. These methods offer adaptive attack detection capabilities but entail significant data processing overhead and raise privacy concerns due to extensive user behavior monitoring^39,48,49. Table 1 summarizes the advantages and limitations of these techniques.

Table 1.

Advantages and limitations of BFA mitigation techniques.

Technique	Advantages	Limitations
Password policies	Simple to implement	User inconvenience, vulnerable to sophisticated attacks32
Account lockout	Prevents continuous attempts	Risk of DoS, management overhead34,35
Multifactor authentication	High-security level	Additional user steps, potential phishing risks37–40
Rate limiting	Reduces attack speed	Circumvented by distributed attacks, it affects legitimate users4,41,42
IP blocklisting/Allowlisting	Effective against known IPs	Limited against spoofing, dynamic IPs, management overhead43,44
Behavioral biometrics	Non-intrusive, challenging to mimic	Data collection and privacy concerns39,48,49

Addressing RAS security gaps with software-defined perimeter (SDP) architecture

Traditional BFA mitigation strategies motivate adopting SDP, a zero-trust security architecture that dynamically enforces context-aware access controls, continuously authenticating users, devices, and applications^16,17,50. Unlike traditional static approaches, SDP minimizes the attack surface through micro-segmentation, isolating critical resources within secure micro-perimeters and enabling granular security enforcement and dynamic threat adaptation^51,52

SDP integrates seamlessly with MFA, complementing and enhancing protection by continuously verifying multiple authentication factors and monitoring session contexts, thereby mitigating risks associated with compromised credentials and phishing^17,23,24. Prior studies validated SDP’s efficacy against volumetric attacks, demonstrating improved resilience to DDoS and port-scanning while incurring modest latency overhead (approximately an average of 1s connection setup)^18,19,53.

However, prior research has not explicitly integrated dedicated BFA mitigation within SDP’s authentication layer. Our research bridges this gap by embedding proactive BFA detection directly within SDP, enabling adaptive threat response and minimizing exposure to attackers. Moreover, formal verification of SDP resilience against BFA remains unexplored; thus, our contribution extends zero-trust principles to active authentication-layer intrusion prevention and real-time threat adaptation. This study addresses critical limitations in existing approaches, delivering robust, proactive SDP-based protection for RAS against evolving BFA.

Proposed software-defined perimeter architecture

SDP system architecture

In this work, the design and implementation of the SDP system architecture for mitigating BFA on RAS starts by identifying prominent RAS assets and their risk level. Then, understanding their operation anatomy through identifying and recording each RAS asset’s connections, including sources, destinations, protocols, and ports. To this end, selecting an appropriate SDP architecture that fits the number of RAS assets and the required security level is critical. Zero-trust security policies for each RAS asset can be established based on the tailor-made SDP architecture for RAS, specifying which connections are allowed or denied based on strict authentication and authorization rules. Lastly, continuously monitor system events for signs of BFA attempts, log these events, and promptly address any detected issues while reassessing each RAS asset’s criticality and updating the flow mappings to maintain adequate protection against evolving threats.

Figure 2 illustrates the SDP architecture’s primary components for securing RAS. The primary architecture for essential SDP operations comprises the system components, including the SDP Client Initiating Host (SCIH), SDP Service Accepting Host (SSAH), SDP Gateway (SG), and SDP Controller (SC). The SCIH, i.e., the user and device, and SSAH (protected resource/application) are connected to SC, the authentication center, and the policy engine.

Fig. 2.

The SDP architecture and the primary system components.

The SC assigns the appropriate policy to the SCIH, determining/resolving its allowed resource access. The SC provides the control layer and enables authentication of SCIH and SSAH before allowing any communication. The SPA protects the SC and SSAH by limiting their accessibility and visibility to unauthorized network entities, i.e., users, devices, and applications. As a result, every network entity is cloaked/blackened/hidden from users. The SG sits before the resources to protect them, allowing only authorized network entities, i.e., devices, users, and applications, to access the secured services and processes. Once the SC grants the policy to the CISH, the SG opens the specific port to allow only the specified service and blocks any other communication, i.e., it is a default-deny-all firewall. Figure 3 shows the SDP architecture’s connection setup. It details how SC connects to services for authentication and authorization and how SG manages secure communications with encrypted connections and enhanced authentication.

Fig. 3.

SDP architecture connection setup, showing SC service connections for authentication and authorization and SG management of secure, encrypted communications with enhanced authentication.

The SDP services access, Hash-based Message Authentication Code (HMAC), and encryption keys management operations involve several steps and processes. The main objective is establishing a secure tunnel to access services only to legitimate clients, i.e., verified and assigned dynamic, least-privilege, and fine-granular policies, via the SDP Gateway. The established tunnel between the legitimate client and the SDP gateway resets service access by closing the firewall, i.e., removing all rules and returning to the default, denying all settings every 30 s. As shown by the flowchart in Fig. 4, after the introduction and initiation of the SDP primary components, i.e., the SDP controller, the SDP gateway, and the client, the operation starts with the client application sending an HTTP request every 25 s to the SDP controller to get HMAC and encryption keys. Thus, this operation allows the legitimate client to have continuous access to the requested services upon successful connection with the gateway.

Fig. 4.

A flowchart for HMAC and encryption keys management operation for continuously securing RAS in the proposed SDP architecture.

When the SDP controller receives the HTTP request from the legitimate client, the fwknop tool generates the HMAC and encryption keys incorporating multiple attributes, including a unique random number (such as timestamps and nonce), protocols, port numbers, and device identity. The SDP controller sends the generated HMAC and encryption keys to the gateway server before responding to the client. The gateway server receives the generated keys for the user and updates the respective stanza file for successful key verification at a later stage. After updating the gateway, the client who initiated the access request receives the updated HMAC and encryption keys from the SDP controller and updates its local fwknoprc stanza file, which matches the updated gateway’s fwknopd stanza file. Then, the client sends a fwknop knock request to the gateway to access services. The fwknopd does not reply; if the keys match, access is given; otherwise, the client lacks access. This repetitive and ongoing process allows services access to only legitimate entities. Algorithm 1 also depicts the SDP access, HMAC, and encryption key management operations.

Algorithm 1.

SDP Services Access and Maintenance Operation

1. Features and Techniques

SPA: SPA is a technique used within SDP architecture to enhance security by ensuring that network services are hidden and inaccessible to unauthorized users. SPA requires an authorized user to send a specially crafted packet (authorization packet) to the SDP gateway. This packet serves as a “knock” to reveal the hidden service.

ConnTrack: ConnTrack is a feature in network firewalls that tracks the state of network connections. It maintains a state table of active connections, allowing the firewall to decide based on the traffic state.

Snort IDS: Network Intrusion Detection & Prevention System (Snort) is an open-source Intrusion Detection and Prevention System (IDS/IPS) that monitors network traffic for suspicious activity. In our setup, Snort IDS/IPS is integrated into the SDP architecture for validation and extensibility. Snort provides an extra security layer by monitoring traffic and identifying intrusions that bypass SDP, helping us continuously improve our security measures. Additionally, Snort demonstrates the flexibility of our SDP architecture, showing how it can work seamlessly with existing security tools. Combining SDP and Snort creates a more robust, layered defense against threats, enhancing traditional security methods.

• 2.Effectiveness of SPA and ConnTrack Synergy in Mitigating BFA and Associated Threat Vectors on RAS

Hiding RAS: By hiding services behind the SDP gateway and only revealing them to authorized users through SPA, BFA attackers cannot even detect the existence of the RAS to initiate attacks.

Limited Exposure: SPA ensures that only authenticated requests reach the SDP gateway, significantly reducing the attack surface and making it difficult for attackers to perform BFA on RAS.

Reduced Noise: SPA helps filter out unauthorized traffic, reducing the chances of associated threat vectors such as DoS/DDoS attacks overwhelming the gateway and protecting RAS resources.

Stateful Inspection: ConnTrack performs stateful inspection of packets, ensuring that only legitimate, stateful connections are allowed through the SDP gateway. This functionality/capability helps block unauthorized connection attempts and mitigate BFA on RAS.

Rate Limiting: By monitoring and managing connection states, ConnTrack can enforce rate limiting on connection attempts, making it difficult for attackers to perform rapid, successive connection attempts typical in BFA and DoS/DDoS attacks on RAS.

Anomaly Detection: ConnTrack can detect anomalies in network traffic patterns, such as unusual spikes in connection attempts, indicative of DoS/DDoS attacks. It can then trigger appropriate countermeasures to protect RAS and the network.

SPA and ConnTrack, when integrated into the SDP architecture, provide a robust mechanism for securing RAS against BFA and DoS/DDoS attacks. SPA effectively hides services from unauthorized users, while ConnTrack ensures that only legitimate connections are maintained and monitored for anomalies. Synergetically, SPA and ConnTrack create a secure, dynamic, and resilient perimeter that adapts to evolving threats, providing enhanced security for RAS.

Formal system model

To rigorously assess the security of the enhanced SDP architecture, we define a formal system model capturing essential entities, states, protocols, and attacker capabilities, enabling formal verification of critical security properties. The system is represented as a tuple:

where

1. Sets

: Each is an SDP controller for managing authentication.

: Each is an authentication server verifying credentials.

: Each is a policy engine evaluating access rules.

: Each is an SDP gateway enforcing secure access.

: Each is a user attempting access.

: Each d_i is a user device.

: Each is a network resource protected by SDP.

: Each is an access policy defining authorization rules.

: Each is a BFA event.

: Each ids_i is a Snort IDS/IPS instance.

: Set of network metrics, where the subscript p indicates the RAS protocol for which the metric is measured.

: Attack state indicator (0: no attack, 1: attack active)

: Set of discrete time points.

• 2.Functions

The system evolves through clearly defined dynamics, including authentication, authorization, connection establishment, and security event detection and mitigation:

Authentication: A user is considered authenticated on a device at a given time if and only if both the user’s credentials and the device’s certificate are successfully verified by an authentication server, along with a nonce exchange to prevent replay attacks:

1 if a user “u” is authenticated on device “d” at time “t,” 0 otherwise.

Authorization: A user is authorized to access a resource from a device at a given time if and only if the user is authenticated on the device and the policy evaluation for the access request is successful:

if user “u” is authorized to access resource “r” from device “d” at a time “t,” 0 otherwise.

Where, if the policy evaluation of user’s “u” request from device “d” to access resource “r” under policies “P” a time “t,” is successful, “0” otherwise.

Connection Establishment: A device is considered connected to a gateway at a given time if and only if the user is authenticated on the device and a secure tunnel is successfully established between the device and the gateway:

if device “d” is connected to gateway “g” at a time “t,” 0 otherwise.

Where, if a secure tunnel is established between device “d” and gateway “g” at a time “t,” 0 otherwise.

BFA Detection and Mitigation: if a BFA by user “u” on device “d” is detected at a time “t,” 0 otherwise.

Intrusion Detection: The function is evaluated at each time “t,” for each Snort IDS/IPS instance . If an intrusion is detected , an alert is generated and sent to the SDP Controller.

if an intrusion is detected by Snort instance ids at a time “t,” 0 otherwise.

• 3.Quantitative Security Analysis

This subsection presents a security analysis that uses mathematical models and statistical methods to quantify security properties and evaluate the effectiveness of security mechanisms. The critical elements of the proposed SDP architecture under this quantitative security analysis are Mutual Authentication Security, SPA Security, and Dynamic Firewall Effectiveness.

Mutual Authentication Security: The probability of a successful authentication attempt given a BFA can be modeled as:

where is a small probability representing the chance of the attacker guessing the correct credentials and nonce.

SPA Security: The probability of successful unauthorized access to a resource using an intercepted token is:

where is a small probability due to the short-lived and resource-bound nature of the tokens.

Dynamic Firewall Effectiveness: The effectiveness of dynamic firewall rules in mitigating attacks can be quantified by comparing the average number of successful attacks with and without SDP being enabled:

where

• SDP = 0 indicates SDP is disabled.

• SDP = 1 indicates SDP is enabled.

• A = 1 indicates the scenario is under attack conditions.

• 4.Algorithms

The algorithms designed to mitigate BFA and network intrusion attacks using the SDP architecture in the context of RAS combine authentication, authorization, connection establishment, and security monitoring to enhance network security. Specifically, after initialization, load system configurations. Initialize authentication servers (AS), policy engines (PE), SDP controllers (C), and Snort IDS/IPS instances (IDS). Algorithms 2, 3, 4, 5, and 6 provide a high-level overview of the earlier described SDP architecture’s operation tailored to fulfill the authentication and authorization, connection establishment, and security monitoring and mitigation, respectively.

Algorithm 2: This procedure handles user authentication and authorization on a device trying to access a respective RAS resource.

Algorithm 2.

Authentication and Authorization

Algorithm 3: This procedure establishes a secure tunnel between a device and the SDP gateway upon successful authentication and updates the ConnTrack table.

Algorithm 3.

Connection Establishment

Algorithm 4: This function detects BFA based on the number of failed login attempts within a time window.

Algorithm 4.

BFA Detection

Algorithm 5: This function is implemented within the Snort IDS/IPS and analyzes network traffic.

Algorithm 5.

Intrusion Detection

Algorithm 6: This procedure encapsulates the main loop of the SDP architecture, continuously monitoring and responding to security events.

Algorithm 6.

Security Monitoring and Mitigation

Experimental evaluation

Experimental setup

The proposed enhanced SDP architecture was implemented and evaluated using a containerized test environment deployed on a physical host machine running Ubuntu Linux 20.04 LTS with Docker Engine. This setup consists of four Docker containers, each simulating distinct roles within the architecture to facilitate performance and security evaluation under realistic attack scenarios.

The SDP Controller component was deployed in Container 1, running a custom implementation of the Waverley SDP controller module. Container 2 served as the SDP Gateway, implementing a deny-all firewall policy using iptables and a customized configuration of the fwknop module to represent the Accepting Host (AH). This gateway was responsible for cloaking protected RAS and managing access enforcement. The attacker environment was emulated in Container 3 using Kali Linux, equipped with BFA scripts to simulate unauthorized access attempts on the RAS protected by the proposed system. Lastly, Container 4 acted as a legitimate client, configured with the Initiating Host (IH) version of the fwknop module to generate valid access requests.

All containers were deployed on a single physical host machine with sufficient hardware resources to ensure realistic emulation of network operations. Table 2 provides the detailed specifications of the host machine and each container, including allocated hardware resources, operating systems, and component roles.

Table 2.

Experimental testbed specifications and descriptions.

Component	Software stack	Hardware allocation	Role description
Host Machine	Ubuntu Linux 20.04 LTS, Docker Engine	Intel Core i7-7700HQ @ 2.80 GHz (8 MB Cache), 16 GB DDR4 RAM, NVIDIA GTX 1050 4 GB GDDR5, 1.0 Gbps Ethernet	Host platform for all containers
Container 1	Ubuntu Linux 20.04 LTS, Waverley SDP Controller Module	1 vCPU, 2 GB RAM, 2 NICs	Runs SDP Controller
Container 2	Ubuntu Linux 20.04 LTS, fwknop (AH configuration), RAS Services	1 vCPU, 2 GB RAM, 3 NICs	Acts as SDP Gateway with iptables firewall
Container 3	Kali Linux, Attack Scripts	1 vCPU, 1 GB RAM, 1 NIC	Simulates BFA and DoS attack scenarios
Container 4	Ubuntu Linux 20.04 LTS, fwknop (IH configuration)	1 vCPU, 1 GB RAM, 1 NIC	Emulates legitimate client using SPA authentication

The implemented SDP prototype comprises multiple components integrated to achieve the main objective of the system development. We have incorporated usability and other related metrics to meet the intended tasks. The SDP infrastructure implementation setup in this experimentation is the customized version of the open-source SDP architecture prototype from the Waverly Labs⁵⁴. It is also available in a dockerized setup via Aifrruis Labs^55,56. The infrastructure comprises the SDP controller, which manages clients’ identities and trust score computation (Based on context-aware metrics such as username and credential, MAC address, Geolocation, and Face recognition) for client authentication and verification.

The developed SDP controller technologies are Vue JS for the front end, Laravel for the backend, i.e., API, swagger for API documentation, and MySQL version 8.0.27 for the database. Also, the SDP infrastructure possesses an SDP gateway with allocated services operating with the deny-all policy by default, which accepts access only to authenticated and verified clients by the SDP controller. The port mappings via the iptable tool facilitate the filtering and routing for only allowed services as per predefined business policies with the connection status tracking capability. Table 3 shows the port mappings for the service configurations in this SDP prototype implementation. The desktop client is another component of the implemented SDP infrastructure, the lightweight software application facilitating authentication, verification, and maintaining client access to the protected gateway services. The detailed SDP controller interface and the desktop client of the developed SDP prototype framework are shown in the Appendix and Figs. 14 and 15 are shown, respectively.

Table 3.

Ports mapping for services running on dockerized implemented SDP architecture prototype.

Remote access service type	Port inside docker	Port outside docker
HTTP, Apache2, Phpmyadmin, and Related Services	80	11800
HTTPS	443	11801
SSH and SFTP	22	11802
FTP	21	11803
Mysql Service	3306	11804
Iperf	5001	11805
Telnet	23	11806
RDP	3389	11809
VNC	5901	11810

Fig. 14.

SDP controller interface main components.

Fig. 15.

Desktop SDP client application.

Experimental procedure

The experimentation procedure involves three phases: phase one validates and demonstrates the efficacy of the proposed SDP framework against BFA and DDoS attacks targeting critical RAS—namely RDP, SSH, VNC, Telnet, FTP, and SFTP. The testing employs widely recognized cybersecurity tools: THC Hydra v9.0⁵⁷for automating BFA via credential guessing, Nmap v7.80⁵⁸ for service discovery via port scanning, and hping3 for realistic DDoS simulation. These specific tools were selected due to their widespread adoption, versatility, and realism, enabling accurate emulation of genuine attacker behaviors^6,7,59,60. Initially, Nmap scans were conducted multiple times to identify available RAS services within two experimental scenarios—SDP-enabled and SDP-disabled architectures. Following this reconnaissance phase, THC Hydra was utilized to execute BFA on the detected RAS endpoints systematically. Each BFA is terminated upon successful credential validation or after reaching a predefined maximum threshold of login attempts. Simultaneously, hping3 simulated realistic network-layer DDoS attacks to assess the SDP framework’s resilience further. Figure 5 illustrates the flowchart depicting the detailed procedure of these BFA tests.

Fig. 5.

A flowchart of a typical BFA operation.

In phase two, we conducted a performance evaluation on connection setup, latency, throughput, packet loss, and CPU usage as the performance metrics upon comparison with the proposed SDP framework performance against the convectional RAS setup, i.e., when SDP is disabled. We measure the connection setup time for the developed SDP architecture prototype when a client with permission connects to the SDP gateway by using “Postman”(a widely-used API development and testing tool that simplifies the process of creating, sending, and managing HTTP requests to APIs)to send a request to the SDP gateway. After receiving the response, the “Timings” section in the “Tests” tab was checked, revealing details like “DNS Lookup,” “Initial Connection,” “SSL Handshake,” and “Time to First Byte,” which were used to assess the connection setup time. Regarding the throughput, we use the Iperf tool to measure the throughput between the SDP gateway enclaving RAS and the client and record the results. Tools like “ping” (for latency and packet loss) and “netcat” (for real-time latency) were used to understand network response times and data loss. Also, via the devised prototype’s SDP controller interface, we recorded and gauged real-time network traffic and CPU usage statistics collected from the system hardware in both scenarios when SDP is enabled and when it is disabled with attacks and without attacks for the RAS. These measurements were repeated several times, and additional diagnostic tools like “my traceroute” (MTR) were used to understand better the SDP gateway’s performance and the whole SDP architecture under similar scenarios. Through this thorough approach, we ensured a comprehensive and reliable analysis.

In phase three, we conducted a comparative performance and security analysis of the enhanced SDP architecture versus the existing BFA mitigation approaches. We conducted a comparative study by iteratively integrating conventional security methods for BFA mitigation: VPN, MFA, ML, Rate Limiting, and Fail2Ban. Towards this comparative performance and security analysis study, the VPN approach leveraged OpenVPN’s open-source solution to provide encrypted and secure communication channels to obscure endpoints from brute-force attacks and network reconnaissance. A robust multi-layered verification was implemented for MFA, combining device identification and username-password credentials. The rationale was to assess the effectiveness of strong identity validation layers against credential-based attacks. The ML approach was implemented via selected pre-trained models within SnortML, enabling adaptive anomaly detection based on historical traffic patterns to identify and mitigate brute-force attempts proactively.

Additionally, rate limiting and Fail2Ban methods were implemented via customized SnortML rules that monitor login attempt frequencies and automate IP blocking upon exceeding predefined thresholds, thereby reducing attack surface exposure. This unified integration provided a clear comparative perspective on each method’s effectiveness, efficiency, and practical overhead relative to the proposed SDP architecture.

Security and performance analysis

Security analysis

Table 4 summarizes the Nmap port scanning and BFA results in the experimental setup with SDP architecture enabled and disabled. With the SDP-enabled architecture, the targeted RAS, i.e., RDP, VNC, SSH, Telnet, SFTP, and FTP, are hidden behind the SDP gateway with the default-deny all firewall, making them invisible by unverified users and applications. Hence, the Nmap tool unsuccessfully discovers the protected RAS during port scanning. Conversely, with the SDP-disabled architecture, RAS and related services are discoverable and thus susceptible to port scanning via the Nmap tool, leading to BFA. Figure 16a and b (in the Appendix) clearly illustrate the Nmap tool’s port scanning outcomes with the SDP architecture in its enabled and disabled states, respectively.

Table 4.

Security analysis results of RAS against port scanning and BFA.

Attacking tool	Remote access service	With SDP disabled	With SDP enabled
Nmap for port scanning	RDP, VNC, SSH, Telnet, SFTP and FTP	Discoverable as Open ports 3389, 5901, 21, 23, 21, and 22, respectively	All ports are filtered and not discoverable. RAS is cloaked
Hydra for BFA	RDP, VNC, SSH, Telnet, SFTP and FTP	Susceptible to BFA	RAS never accepts requests from unverified users/applications

Fig. 16.

(a) Nmap port scanning and identifying RAS with SDP-enabled architecture (b) Nmap port scanning and identifying RAS with SDP-disabled architecture.

Likewise, the RAS within the SDP-enabled architecture, RDP, VNC, SSH, Telnet, SFTP, and FTP, for example, are intrinsically secure against BFA. Since services in the SDP-enabled architecture are provisioned and accessed based on a need-to-know fashion and in the least privileged fashion after thorough verification, only verified users and applications can send requests and access them upon qualifying access resource policies. In contrast, the RASs within the SDP-disabled architecture are susceptible to BFA. They allow trial and error attempts for login and credentials before verifying (following the connecting first, then verifying the next paradigm). For a detailed demonstration, see Appendix, Figs. 17–22 (denoted with “a” for the SDP-enabled architecture and “b” for the SDP-disabled architecture) illustrate the BFA results conducted against RDP, VNC, SSH, Telnet, SFTP, and FTP, respectively.

Fig. 17.

(a) BFA on RDP service with SDP architecture enabled (b) BFA on RDP service with SDP architecture disabled.

Fig. 22.

(a) BFA on FTP service with SDP architecture enabled (b) BFA on FTP service with SDP architecture disabled.

Performance analysis

1. Connection Setup Time

In SDP-enabled environments, the SDP components experience trivial startup time overhead compared to the environments without the application of the SDP. The overhead is primarily due to two additional steps: a client has to send encryption and HMAC requests, and a server has to send a request to the gateway to update Stanza. Specifically, the initiating host authentication takes 595 ms for encryption and HMAC requests. Also, the accepting host authentication time incurs an extra 455 ms for a server to request the gateway to update Stanza. According to our experimentation setup, the aforementioned additional steps are not of concern within the regular network infrastructure, which needs only a 0.12-ms connection setup, as shown in Table 5. Similarly, previous research works^18,53, and⁶⁰ reported such overhead, which varies in size based on the system components’ computation capability and the link speed between participated entities/components.

• 2.Network Throughput

Table 5.

Time for connection setup.

SDP status	With SDP disabled	With SDP enabled
Time (in seconds)	0.12	1.05

The recorded throughput in our experiment justifies that the proposed SDP architecture deployment does not significantly affect the infrastructure throughput. It does not affect either peer’s connection within the infrastructure. Instead, the bandwidth bottlenecks are the network interfaces, connection links, and processing capabilities of the connected peers. For a real depiction of this, see Appendix, Fig. 23a and b, with SDP enabled and disabled, respectively.

Fig. 23.

(a) Throughput measured by iperf with SDP architecture enabled (b) Throughput measured by iperf with SDP architecture disabled.

However, across RAS protocols, the throughput performance evaluation with the proposed SDP architecture shows significant changes from the setup without SDP architecture. The throughput is significantly improved when the SDP architecture is enabled during the attack scenarios across the RAS protocols. The boxplots in Fig. 6 illustrate the Mbps throughput when the SDP is enabled and disabled for all four scenarios: under normal (without attacks) conditions and under attack conditions. On the one hand, with the SDP-disabled mode under normal conditions, the RDP throughput has a median of 10.19 Mbps (Mean = 10.15 Mbps, Standard deviation (STD) = 1.94 Mbps, and Variance = 3.78 Mbps). On the other hand, with the SDP-enabled mode under normal conditions, the RDP throughput had a median of 9.89 Mbps (Mean = 10.03 Mbps, Standard deviation (STD) = 2.20 Mbps, and Variance = 4.83 Mbps). Meanwhile, SDP generally improves throughput during attacks, especially for RDP, SSH, VNC, and FTP. For precise clarification via statistical significance confirmation, the descriptive statistical information of the gateway throughput results for respective RAS protocol under the four scenarios is given in the Appendix section by Fig. 24a (Normal with SDP disabled mode), b (Normal with SDP enabled mode), c (under attack with SDP disabled mode), and d (under attack with SDP enabled mode). During attacks, SDP maintained 4.0 times higher RDP throughput (7.91 Mbps vs. 1.99 Mbps without SDP) and 3.1 times higher FTP throughput (9.90 Mbps vs. 3.10 Mbps).

• 3.Latency

Fig. 6.

Measured throughput for all RAS protocols when the SDP is enabled and disabled under attack and normal conditions.

Fig. 24.

(a–d) The descriptive statistical information of the RAS protocols and Resource Consumption of the gateway CPU.

The performance evaluation demonstrates that the proposed SDP architecture significantly reduces latency across all RAS protocols. Under normal conditions, RDP latency decreased by 11% (median: 50.09–45.02 ms), attributed to SDP’s optimization of UDP communication and elimination of ad-hoc security checks. During attacks, SDP achieved even greater reductions: RDP latency improved by 48.2% (median: 194.39–100.76 ms; 95% CI [98.64, 102.89 ms]), and VNC by 50.6% (median: 297.48–147.33 ms). Also, FTP latency decreased by 22% (median: 248.11–125.45 ms). These improvements correlate with reduced variance (RDP’s standard deviation dropped 22.3%), indicating more predictable network conditions. For precise clarification via statistical significance confirmation, the descriptive statistical information of the gateway latency results is given in the Appendix section by Fig. 24a (Normal with SDP disabled mode), b (Normal with SDP enabled mode), c (under attack with SDP disabled mode), and d (under attack with SDP enabled mode). Under Attack: SDP reduced median latency by 48.2% for RDP (194.39–100.76 ms) and 50.6% for VNC (297.48–147.33 ms) (Fig. 7).

• 4.Packet Loss

Fig. 7.

Recorded latency for all RAS protocols when the SDP is enabled and disabled under attack and normal conditions.

The performance evaluation of the packet loss with the proposed SDP architecture shows significant changes from the setup without SDP architecture. The packet loss is significantly reduced when the SDP architecture is enabled, especially during attack scenarios. Specifically, SDP effectively reduces packet loss across all protocols during attacks. Without SDP, the average packet loss is about 10%, but with SDP, it’s reduced to around 2%. Notably, there’s no FTP packet loss with SDP during attacks. Figure 8 illustrates the packet loss percentage when the SDP is enabled and disabled under attack and normal conditions for respective RAS protocols. Another notable recorded observation is the reduction in RDP packet loss by 79.0% during attacks (0.0989–0.0208) and SSH packet loss by 80.0% (0.1002–0.0200). For precise clarification via statistical significance confirmation, the descriptive statistical information of the gateway packet loss results is given in the Appendix section by Fig. 24a (Normal with SDP disabled mode), b (Normal with SDP enabled mode), c (under attack with SDP disabled mode), and d (under attack with SDP enabled mode).

• 5.CPU Usage

Fig. 8.

Recorded packet loss for all RAS protocols when the SDP is enabled and disabled under attack and normal conditions.

The performance evaluation of the CPU usage with the proposed SDP architecture shows insignificant changes from the setup without SDP architecture. Generally, the CPU usage varies insignificantly and thus proves the viability of the proposed SDP architecture in an actual deployment environment. Figure 9 shows the CPU usage percentage when the SDP is enabled and disabled. On the one hand, with the SDP-disabled mode, the CPU usage has a median of 16.78 (Mean = 17.01, Standard deviation (STD) = 4.60, and Variance = 21.13). On the other hand, with the SDP-enabled mode, the CPU usage had a median of 14.67 (Mean = 14.71, Standard deviation (STD) = 3.93, and Variance = 15.43) under normal conditions. In this regard, in the context of resource consumption, SDP doesn’t significantly change the average amount of data received and transmitted. Meanwhile, SDP reduces CPU usage during attacks, lowering it from an average of 98.25% to 50.41%, a reduction of around 48.7%, henceforth enabling sustained service availability. For precise clarification via statistical significance confirmation, the descriptive statistical information of the gateway CPU percentage usage results is given in the Appendix section by Fig. 24a (Normal with SDP disabled mode), b (Normal with SDP enabled mode), c (under attack with SDP disabled mode), and d (under attack with SDP enabled mode).

Fig. 9.

CPU percentage usage of the SDP gateway when SDP is enabled and disabled under attack and normal conditions.

Without SDP architecture, the CPU usage of the gateway increased during BFA. At the same time, it was stable, and insignificant changes were recorded in the setup with the SDP-enabled architecture. Figures 10a and 11b show the CPU usage of the gateway during BFA in the SDP-enabled and disabled architecture, respectively. In the SDP-disabled architecture, high gateway CPU usage is due to a lack of ability to differentiate between legitimate and malicious requests. In this case, normal and malicious traffic are indistinguishable, leading to continuous high gateway CPU usage.

Fig. 10.

(a) The Gateway’s CPU usage during BFA with SDP architecture enabled (b) The Gateway’s CPU usage during BFA with SDP architecture disabled.

Fig. 11.

SDP Gateway is enabled with Snort IDS to enhance real-time malicious network activity detection on RAS.

Overall, statistical analysis shows that SDP significantly mitigates attack impacts in the SDP architecture, reducing CPU usage, latency, and packet loss while improving or maintaining throughput. Under normal conditions, SDP primarily reduces CPU usage, latency, and packet loss, with a less pronounced effect on throughput. The confidence intervals in the data support these findings, highlighting the statistical significance of the differences observed under the four scenarios subjected to the SDP architecture for securing RAS.

Regarding DDoS on SDP architecture, Our proposed SDP model for securing RAS against DDoS attacks uses SPA combined with a deny-all firewall at the gateway. This approach blocks malicious traffic, making DoS/DDoS attacks largely ineffective. SPA filters out unauthorized packets, allowing only legitimate traffic through, and forwards suspicious activity to Snort IDS for validation. This proactive rejection of DDoS traffic by the SDP Gateways places a minimal computational load on the network infrastructure.

In this regard, the proposed SDP model effectively protects RAS from DDoS attacks as the RAS access undergoes thorough authentication before granting access. The model counters well-known DDoS attacks by hiding services behind an SDP Gateway and employing a dynamic firewall mechanism to drop malicious packets quickly. Furthermore, the sessions are tracked by the ConnTack and Snort IDS in real-time, as shown in Fig. 11, analyzing and alerting insights of traffic behavior and enabling informed decision-making by the firewall.

Figure 12a and b illustrate the effectiveness of the proposed SDP architecture. Figure 12a shows a network overwhelmed by DDoS attacks with SDP disabled mode, leading to resource unavailability for legitimate users. In contrast, 12b demonstrates how, with SDP enabled, the network remains stable as the SDP Gateways proactively reject DDoS traffic with minimal strain on the infrastructure. These results highlight the proposed SDP architecture’s capability to mitigate DDoS attacks and maintain service availability.

• 6.Comparative Performance and Security with Existing BFA Mitigation Approaches

Fig. 12.

(a) The SDP Gateway’s CPU usage during DDoS attacks with SDP architecture enabled (b) The SDP Gateway’s CPU usage during DDoS attacks with SDP architecture disabled.

To compare the proposed SDP architecture with the existing BFA mitigation approaches (covered in the related work Section), we conducted a comparative performance and security analysis of the enhanced SDP architecture versus conventional security methods for BFA mitigation: VPN, MFA, ML, Rate Limiting, and Fail2Ban.

Figure 13 demonstrates the SDP architecture’s superiority over existing protection methods (VPN, MFA, ML, Rate Limiting, and Fail2Ban).

Fig. 13.

Comparative performance and security analysis of the enhanced SDP versus existing BFA mitigation mechanisms.

Specifically, the enhanced SDP achieved an exceptional average block rate of 99.9% against brute-force attacks, substantially surpassing VPN (33.23%), MFA (66.57%), ML (76.97%), rate Limiting (54.5%), and Fail2Ban (78.3%). Furthermore, SDP significantly reduced CPU utilization to 18%, compared to higher usage by traditional methods, such as VPN (45%), MFA (41%), ML (45%), rate Limiting (40%), and Fail2Ban (43%), indicating enhanced operational efficiency. SDP also showed the lowest latency at 52 ms, significantly better than VPN (88.3 ms), MFA (73.2 ms), ML (85.6 ms), rate limiting (75.0 ms), and Fail2Ban (80.4 ms), highlighting its superior responsiveness. Throughput remained consistently high (10 Mbps) across all evaluated methods, confirming that SDP effectively maintains robust security without compromising network performance. These results confirm SDP’s effectiveness as a balanced, high-performance security solution ideal for enterprise deployment.

Discussion and limitation

This study presents an enhanced SDP framework tailored and designed to secure better RAS akin to BFA. It explicitly focuses on mitigating BFA, targeting widely used protocols like RDP, SSH, VNC, Telnet, FTP, and SFTP. The approach combines several security layers to limit unauthorized access attempts significantly. The combined approach includes service obfuscation via cloaking/hiding capability, SPA, and adaptive threat response mechanisms via snort IDPS. Tests conducted during the evaluation phase revealed notable gains. These gains include significantly reducing 48.7% CPU usage during ongoing attacks and enabling cost-effective scaling for enterprises while maintaining a seamless experience for legitimate users during attacks. One trade-off observed was a minor authentication delay, primarily attributed to the overhead from cryptographic verification and establishing secure communication tunnels. While this delay is generally tolerable in most corporate settings, optimizing the system further could improve its performance in latency-critical applications.

Despite its apparent benefits, the SDP solution has certain limitations. Firstly, the centralized nature of the controller could become a bottleneck in large-scale implementations. This insight suggests the need for further research into distributed or clustered controller designs. Additionally, integration complexities arise when incorporating this solution with legacy systems such as Active Directory, potentially hindering smooth organizational adoption. The current security framework presumes SPA token confidentiality and does not explicitly cover scenarios involving compromised internal credentials or token exposure, indicating the necessity for supplemental security measures. Thus far, the formal security analyses have simplified several assumptions, notably omitting a detailed examination of side-channel vulnerabilities. Consequently, future investigations should involve more rigorous verification using sophisticated analytical tools (such as ProVerif or Tamarin).

Furthermore, given the ever-evolving nature of cyber threats—such as AI-assisted credential cracking and coordinated, slow-paced brute-force attacks by distributed botnets—it becomes imperative to incorporate advanced behavioral analytics, anomaly detection systems, and reputation-based IP filtering technologies. Additionally, the extensive logging required for threat detection poses compliance challenges concerning privacy regulations like GDPR. Future implementations must thus carefully balance detailed forensic logging with privacy safeguards such as anonymization and data retention policies to maintain user trust and regulatory adherence.

The findings from this study provide practical advice for implementing zero-trust security principles effectively within operational networks. The SDP model aligns with zero-trust frameworks by proactively hiding services using SPA, effectively preventing BFA with minimal latency and reducing computational overhead. In contrast to conventional reactive security measures (like fail2ban and rate limiting) that address attacks only after initiation, SDP preemptively blocks threats, significantly lowering false positive rates and user inconvenience. Moreover, the SDP design surpasses the capabilities of traditional static defenses such as firewalls and VPNs, which are susceptible to encrypted attacks and incur notable latency and processing overhead.

A recommendation building upon these understanding and insights is to achieve end-to-end zero-trust security by integrating SDP architecture with other supporting technologies, such as MFA for robust identity authentication and ML-based anomaly detection systems for real-time response against emerging attacks. The combined solution provides security layers, thus overcoming the scalability and operational performance limitations normally inherent in traditional protective strategies. These real-world insights aim to enlighten practitioners on deploying robust, scalable, and privacy-friendly zero-trust security architectures for diversified and dynamic enterprise networks.

Conclusion and future work

Preventing BFA on RAS (RDP, SSH, VNC, Telnet, FTP, and SFTP) continues to be challenging in light of the increased adoption of remote work and the evolving cybersecurity threats. This paper introduces an enhanced SDP architecture intending to actively secure RAS by leveraging SPA, dynamic access control, and active threat mitigation capabilities. We validated the proposed practical, lightweight prototype (implemented in Docker) to effectively thwart BFA and DDoS attacks (as realistic testing scenarios targeting RAS) using rigorous formal analysis and thorough empirical evaluations. Cryptographically obscuring services from unauthorized visibility and dynamically blocking malicious access attempts, the architecture maintains strong security without significantly impacting legitimate user experience or operational performance. However, further research is needed to address outstanding challenges related to scalability, seamless integration with legacy infrastructure, and continuously adapting to new cyber threats. Future work should focus on expanding SDP compatibility to additional protocols and post-quantum schemes/algorithms across a myriad of verticals, such as those used in IoT and web-based systems; employing adaptive security measures, such as AI-assisted techniques to catch up with emerging threats and improve user experience without compromising security and privacy; conducting comprehensive formal verification of cryptographic protocols using advanced tools like Tamarin and ProVerif; enhancing user experience by introducing streamlined recovery mechanisms for legitimate users inadvertently affected by security measures; and rigorously evaluating system scalability and robustness through simulation across a myriad of advanced attacks. Ultimately, the enhanced SDP architecture introduced here provides a solid foundation for securing RAS, with ongoing improvements in intelligent authentication strategies expected to further strengthen cybersecurity practices in diverse and evolving environments.

Appendix

See Appendix Figs. 14, 15, 16, 17, 18, 19, 20, 21, 22, 23 and 24.

Fig. 18.

(a) BFA on VNC service with SDP architecture enabled (b) BFA on VNC service with SDP architecture disabled.

Fig. 19.

(a) BFA on SSH service with SDP architecture enabled (b) BFA on SSH service with SDP architecture disabled.

Fig. 20.

(a) BFA on Telnet service with SDP architecture enabled (b) BFA on Telnet service with SDP architecture disabled.

Fig. 21.

(a) BFA on SFTP service with SDP architecture enabled (b) BFA on SFTP service with SDP architecture disabled.

Author contributions

F.A.R.: Conceptualization, Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, Writing—Original Draft Preparation, Writing—Review & Editing, Visualization, Project Administration, Funding Acquisition. E.E.M.: Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, Writing—Review & Editing. B.L.: Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, Writing—Review & Editing. A.A.A.: Conceptualization, Validation, Data Curation, and Writing—Review & Editing. A.A.A.E.: Conceptualization, Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, Writing—Review & Editing, Supervision, Project Administration, Funding Acquisition. M.A.: Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, Writing—Review & Editing. B.A.: Conceptualization, Methodology, Software, Validation, Formal Analysis, Investigation, Data Curation, and Writing—Review & Editing.

Data availability

All data will be made available upon reasonable request to the corresponding author.

Declarations

Competing interests

The authors declare no competing interests.