A ten-year intrusion that replaced the authentication stack entirely, and what it tells you about the shortfalls in network security architecture. 

By Alan DeKok, CEO, InkBridge Networks 


When forensic investigators at Sygnia began piecing together an intrusion for a recent incident response engagement, the earliest artefacts they found dated to 2016. The attackers had been inside the network for nearly a decade, undetected and in complete control of the authentication stack. 

The operation, documented by Sygnia as Operation Highland and attributed to the threat actor Velvet Ant, is a neat illustration of a cautionary axiom: authentication does not matter if the authentication software has been replaced.  

Once an attacker controls the components that verify identity (on Linux, the PAM modules, the SSH binaries, the authorised keys files) the attacker no longer needs credentials. They are the authentication system. 

Sygnia have published an excellent technical breakdown if you want details on the forensics and incident response. I want to look at what the attack reveals about network security architecture - specifically the design assumptions that allowed it to persist undisturbed for so long, and what ISPs and enterprises running critical infrastructure should do differently. 

Authentication is infrastructure, not just a control 

Most discussions of network security architecture treat authentication as one layer among many: deploy it, configure it, move on. The mental model is a checklist: 

• Firewall 

• Segmentation 

• Access control 

• Authentication 

and once each box is ticked, the architecture is considered sound. 

Operation Highland exposes the problem with this model. Rather than attacking through the authentication layer, Velvet Ant attacked the authentication layer itself, replacing core system binaries with custom-compiled backdoored versions.  

The PAM module that verifies passwords was modified to accept a hardcoded backdoor password and harvest legitimate credentials in parallel. The SSH binaries were replaced with versions that logged every keystroke and connection. Nine distinct variants of the backdoored PAM module were identified, each built in a separate compile environment. The level of effort involved points to a well-resourced, deliberate operation. 

Once the authentication stack is compromised, conventional containment measures stop working. Password resets do nothing because the modified PAM module harvests new credentials as they are entered. Credential rotation is actively counterproductive - you are handing the attacker your new passwords. The persistence survives reboots because the backdoored binaries are the system binaries. From the network's perspective, everything looks normal because authentication is functioning exactly as the modified code intends. 

The lesson for network security architecture is that your authentication infrastructure deserves the same security discipline as the services it protects.  

Treat RADIUS servers, PAM configurations, SSH binaries, and directory service connections as critical infrastructure components with their own integrity requirements. You can’t just configure once and forget it like background plumbing. 

What the absence of change control costs you 

One of the most striking aspects of Operation Highland is what the attack required in terms of the target environment. They replaced binaries, modified the system configurations, and established outbound connections to command-and-control infrastructure.  

That points to an absence of some very basic operational controls:  

• No file integrity monitoring on critical system binaries 

• No change control processes that would have flagged modified authentication libraries 

• No network monitoring that would have alerted on unexpected outbound connections from systems that had no business making them 

• No reprovisioning of critical machines from secure sources 

The outbound connection point is worth dwelling on. Velvet Ant used a modified Nginx configuration to proxy requests through to backend systems and ultimately to their command-and-control infrastructure.  

A web server making unexpected outbound connections to arbitrary internet endpoints should be an immediate alert. If the internal web server does not need to make outbound SSH connections to the public internet, those connections should be blocked at the network layer - and if they occur anyway, someone should know about it within minutes. 

This is what is meant by least privilege network access. It applies to what your users and infrastructure components are permitted to do. Every service should have a defined, minimal set of network behaviours. Anything outside that set should be blocked or alerted. This is documented in more detail in our post on making RADIUS more secure, but the principle extends well beyond RADIUS to every component of your authentication infrastructure. 

The reprovisioning discipline that most organisations skip 

I want to push back on something that appears in the Sygnia writeup and in most incident response guidance: the idea that remediation of a compromised system is a matter of removing the attacker's persistence while keeping the service running. Delete the malicious binary, remove the rogue cron job, patch the modified configuration. Done. 

This framing is wrong. When an attacker has had access to a system for an extended period and has modified system-level binaries, you cannot trust the system. You do not know what else has been changed. You don’t know whether the backup of that system is clean. You don’t know whether the tools you are using to investigate the system are themselves compromised. The only reliable remediation is to build a new system. 

1. Build a new virtual machine image.  

2. Restore the data (not binaries!) from a known-clean backup.  

3. Swap the running instance for the clean one.  

4. Then compare the old instance against the new one to understand the scope of modification.  

This is standard practice for any organisation that has thought seriously about operational resilience. 

The reason most organisations resist this is the "zero-downtime" argument: the service is too critical to take down, even briefly. I understand the concern, but it reflects a confusion between two different things. The requirement is that the service stays operational. There is no requirement that any particular machine provide that service. Those are different constraints; conflating them leads organisations to keep using systems they cannot trust because they are afraid of the maintenance window. 

Our network designs separate critical pieces of functionality into different systems, precisely so that individual components can be upgraded, replaced, or restored from backup without affecting overall service continuity. A RADIUS cluster where one node can be reprovisioned while others continue handling authentication is table stakes for critical network infrastructure. This approach has worked for Tier 1 ISPs the world over for more than 25 years. 

Beyond recovery, regular reprovisioning is also a detection mechanism. When you rebuild critical systems on a scheduled cadence and compare the new image against the running instance, you will find any modifications that have been made. The comparison also reveals any running configuration that has drifted from the approved one. Some organisations run this cycle daily for their most critical systems. The discipline costs operational effort up front, and pays for itself the first time it surfaces something unexpected. 

Worth subscribing to.
Worth reading.

Our weekly newsletter covers network authentication tips, how-tos, security vulnerabilities, free resources, standards updates, and industry news. (All stuff you should stay up to date on!)

Thanks for registering!

SIGN UP Générer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel aliasGénérer un nouvel alias

Separating services to support continuous operation 

The operational resilience point extends to how critical services are architected in the first place. Operation Highland succeeded in part because the target environment appears to have been relatively flat. Once inside, lateral movement was possible because systems that should have been isolated were not. 

Our approach to network security architecture for critical infrastructure often separates RADIUS functionality into multiple distinct systems. This serves multiple purposes.  

• It limits the impact of any single compromise.  

• It allows each component to be upgraded or replaced independently.  

• It means that a problem with the accounting system does not affect authentication availability, and vice versa.  

• It creates natural monitoring boundaries: traffic that crosses service boundaries unexpectedly is visible and alertable. 

The same principle applies to data. For any system we build, the base operating system and executables live separately from the persistent data. Any database mounts its data from a separate volume. This means you can wipe and rebuild the OS layer (replacing potentially compromised binaries) without touching the data, and you can take independent backups of each. It sounds obvious, but the default configuration of most systems does not enforce this separation, and organisations that have not made deliberate architectural choices about it are running mixed environments where remediation is far more disruptive than it needs to be. 

The supply chain dimension 

One other aspect of Operation Highland that deserves attention is the supply chain implication. Velvet Ant compiled custom versions of system libraries. These were purpose-built modifications to core authentication components, compiled in separate environments for different target systems. 

This is, in effect, a software supply chain attack against the authentication stack. There’s a direct parallel to open-source supply chain risk: in both cases, the attack vector is the integrity of the software running on your systems rather than a vulnerability in that software. If you cannot verify the integrity of the binaries running on your critical systems, you cannot trust the outputs of those systems. 

This is another argument for regular reprovisioning from verified base images. If you know that your running instance was built from a known-good image at a specific point in time and has not been modified since, you have a basis for trust. If you have no record of when the system was last rebuilt and no comparison baseline, you have no way to assess whether what is running is what you intended to deploy. 

PGP-signed releases, verified build pipelines, and hash comparison of critical binaries are the authentication infrastructure equivalent of certificate validation: the mechanism by which you establish that what you are trusting is what it claims to be. 

What this means for ISPs specifically 

ISPs running RADIUS authentication for millions of subscribers are operating authentication infrastructure that is, by any reasonable definition, critical national infrastructure. The authentication system is the mechanism by which the network knows who is on it, what they are permitted to do, and what they have done. If that system is compromised, you lose the ability to make any of those determinations. 

The threat calculus has also changed. Operation Highland was almost certainly a nation-state operation; historically, only a nation-state has the patience, funding, and operational discipline to maintain undetected access for a decade.  

The cost of sophisticated attacks is dropping rapidly. AI-assisted vulnerability discovery and exploit development is making capabilities that previously required nation-state resources accessible to smaller, less disciplined actors.  

The practical implication for ISPs is that authentication infrastructure hygiene needs to be treated as an ongoing operational discipline. That means: 

• Regular reprovisioning cycles 

• File integrity monitoring on critical authentication components 

• Strict network-layer controls on what authentication systems are permitted to communicate with 

• Architecture that separates functions so that individual components can be replaced without service interruption 

None of this is technically difficult. Most of it is operationally uncomfortable - it requires committing to processes that feel like unnecessary overhead when nothing is wrong, and that reveal their value only when something is. The organisations that have this discipline in place discover compromises early. The ones that do not discover them when forensic investigators find artefacts from 2016. 

Need more help? 

If your team is wrestling with network configuration, a troubleshooting problem you cannot resolve, or a system that needs to be more resilient, we can help. InkBridge Networks has 25 years of expertise: we wrote the standards, maintain FreeRADIUS, and have seen every failure mode there is. Reach out to request a quote. 

Related Articles

RADIUS security best practices: How to harden your deployment

RADIUS has several well-known security limitations, most of which are easy to mitigate once you know what to do. This guide walks through five common weaknesses in the RADIUS protocol and the practical steps you can take to harden your deployment against each. 

AAA information security: building secure network architecture beyond authentication

Authentication, Authorisation, and Accounting (AAA) is the security framework that controls who gets onto your network, what they can do once they're there, and what gets logged while they are. Authentication verifies identity. Is this person or device who they claim to be? It's the check at the door.