1.5 Million Lessons from the Infamous SingHealth Hack
On January 10th, 2019, the Committee of Inquiry (COI) published its public report detailing the now-infamous SingHealth hack.
This 453-page report is already a difficult read for most security practitioners and, as long and exhaustive as it looks, the most determined readers may still feel unsatisfied by its lack of details on some of the attack’s critical steps.
We provide here, in layman’s words, a description of the incident. We also try to bridge the information gaps to provide a clearer description of the attackers’ moves. While these deductions are not proven, cybercriminals’ modus operandi are predictable enough to consider them probable.
We follow up with an analysis of what organizations can do to prevent this disaster from happening in the first place. And in the event it’s too late, we also discuss the most common options for remediation.
While the aftermath of the attack is still uncertain, its sheer size is sufficient to cause anxiety for Singapore’s citizens: 1.5 million personal records plus the prescription details of 160,000 patients, including those of Prime Minister Lee Hsien Loong, have leaked to an unknown criminal.
As for attribution, the COI seems convinced the perpetrators are state-sponsored operators. As a matter of fact, some of the attack techniques used, as well as the persistent approach, clearly distinguish the attackers from your usual low-tech, hit-and-run hacker crowd.
Considering potential targets, and despite the fact that millions of records have effectively leaked, it seems probable that this operation specifically targeted the private medical records of Prime Minister Lee Hsien Loong. It should be noted that in 2015 the Prime Minister’s Office publicly released a statement on PM Lee Hsien Loong being diagnosed with prostate cancer. Whether the attackers were looking for embarrassing details to be used as leverage is unknown but cannot be excluded.
After their initial reconnaissance, on which we naturally have no information, the operatives sent booby-trapped emails to SingHealth employees. The trap consisted of a malicious attachment capable of exploiting a known Outlook vulnerability. When opened, the attachment dropped a rogue program on the victims’ systems, matter-of-factly providing full control to the attackers over the targeted computers.
There is not much to be said on this initial approach. While all organizations do their best to detect patient-zero infections, this will remain a cat-and-mouse game for the foreseeable future and SingHealth is not to be blamed for suffering such a breach, particularly considering the nature of its opponent.
This initial effort aimed at providing the attackers with the ability to explore SingHealth’s IT infrastructure from within. And as with the vast majority of sophisticated attacks, the actual IT target was SingHealth’s Active Directory infrastructure. Owning your victim’s AD systematically signals the end of the game: with full control over all IT resources, there is not much an attacker cannot access.
Using Active Directory as a transport for destructive malware
Destructive malware is not rocket science. Highly-sophisticated payloads such as Stuxnet are the exceptions, while today’s consumer-level ransomwares are good enough to deal destruction effectively. The only challenge in those attacks is distribution: getting malware installed on a sufficiently large number of endpoints so that recovery at scale becomes unrealistic. In this regard, exploiting Active Directory weaknesses is the only practical option for hackers to move laterally within the infrastructure. Every large-scale, infrastructure-wide attack that has crippled production capabilities in recent years has had an Active Directory exploit at its core. In the SingHealth case, it is not entirely clear how the Active Directory was compromised. Though considering common industry practices and the details provided in the report, we can safely assume they followed a course of action similar to the one proposed here:
They used the current user’s credentials (supposedly unprivileged) to propagate, but did not get elevated privileges from it. As the report suggests though, they may have used weak or cleartext passwords found in network shares to directly authenticate into more-privileged accounts—which is in itself quite the vulnerability. Re-using those accounts, they may have ended up on a computer holding the credentials of a privileged user (e.g. a domain administrator), thus gaining effective control over the whole Active Directory. The rest of the attack simply consisted of using legitimate accounts to query databases and scout resources, like you would on your own systems, until they found what they were looking for.
From Active Directory, with love
This attack sheds a somber light on the state of insecurity among Active Directory infrastructures. There is no doubt that AD was the primary IT target of the attackers.
The SingHealth incident is only the latest sorry example of a long list of operations that succeeded once when they gained access to their victim’s Active Directory: Aurora, Target, Sony, Carbanak, NotPetya… the list goes on.
Active Directory infrastructures remain the nexus of everything that electronically matters in organizations but is still dangerously ignored by security operators. In their defense, the size, complexity, and volatility of a given AD makes it a singular security challenge. Still, it remains concerning to witness how our industry underestimates the risks it poses to global security.
Now it’s time to analyze what organizations can do to prevent this type of disaster in the first place, notably regarding the protection and monitoring of their AD infrastructure. We will constructively criticize the most common security practices for Active Directory and will review potential remediation approaches for those organizations in the unfortunate position of having to recover from such an incident.
“Active Directory infrastructures remain the nexus of everything that electronically matters in organizations but is still dangerously ignored by security operators.”
A deep dive into the 453-page report released by the COI on January 10th leads to a safe assumption: the attack reached a turning point when it compromised SingHealth’s Active Directory. But what exactly went wrong with this critical infrastructure?
Six months exploiting known Active Directory vulnerabilities
You read that right: for a whopping six months—from December 2017 to June 2018—SingHealth’s tormentors raided their victim’s Active Directory unchallenged. That’s more than sufficient to achieve lateral movement on the organization’s machines and to reach a critical mass of compromised systems. Beyond this point, probabilities work for you: it is very likely you’ll end up on a machine that either holds AD admin credentials or has direct access to the data or resources you were looking for.
There are 2 clear takeaways here:
- The audit methods used in anticipation of those threats did not translate into adequate hardenings of the infrastructure.
- The detection mechanisms SingHealth had in place were not sufficient.
Audits without remediations are a waste of time and money
COI’s report shows that SingHealth routinely performed technical audits that covered its Active Directory. Unfortunately, the recommended remediations weren’t fully applied. This observation is not a SingHealth exclusive: audits tend to become an administrative requirement, a box that needs checking, rather than an actual security improvement tool.
If a given organization does not enforce its audit’s recommended remediations as a process, then it boils down to the particular willingness of its technical staff to implement them… which usually ends up in no remediation applied at all. Not judging those individuals here: manually dabbling in Active Directory is neither risk-free nor quick, and it often competes with the gazillion other duties that fall on these professionals.
Behold, a few more glaring security holes:
- The Citrix servers that held access to the attackers’ targeted resources were in an organizational unit (OU) blocked for Group Policy Object (GPO) propagation. As a result, the central password policy for AD accounts wasn’t applied, allowing attackers to exploit weak passwords.
- Beyond those AD accounts, local admin accounts also seem to have gone unmanaged when they should have been. Using Microsoft’s Local Administrator Password Solution (LAPS) could have prevented hackers from exploiting these.
- Finally, the report reveals the existence of a dormant privileged (service) account. Privileged and dormant: two words we don’t like to see next to each other! But hackers do. This account was exploited by attackers to access the data repositories they were targeting. Following standard AD security best practices would have required the deactivation of this account, therefore closing a pathway to SingHealth’s critical assets.
Although these items look like evident flaws, Active Directory remains a complex, moving infrastructure made of thousands of different entities. Looking for weaknesses in this gigantic haystack is a time-consuming and error-prone process. As it so happens, we know of a robust tool that can help, but that’s another story.
Beyond the SingHealth incident, let’s have a quick look at other weaknesses commonly exploited by attackers.
“Audits tend to become an administrative requirement, a box that needs checking, rather than an actual security improvement tool.”
Active Directory: vulnerability central
Here, vulnerability is understood in its broader sense. While software vulnerabilities exist, exploiting misconfigurations or gaping holes is far more common.
There are thousands of configuration “atoms” that, if badly configured, can cascade into major infrastructure-level vulnerabilities. This profusion is a nightmare for security professionals and a blessing for their opponents. Let’s go through some common exploits:
- Dangerous credentials exposure is the #1 threat Active Directory infrastructures face. Due to some legacy administration practices dating back to Windows 2000, accounts with complete access to their organization’s most private resources (like user passwords) have proliferated. This situation dramatically increases the opportunities for an attacker to leverage process injection techniques (available as ready-to-use packages in various open-source tools like Mimikatz).
- Finding privileged accounts running Kerberos services is a common occurrence. Kerberoasting is a widely known technique that allows for stealthily extracting those accounts’ credentials through offline brute-force attacks.
- Failing to restrict delegation on sensitive accounts allows for their impersonation. This Kerberos delegation exploit is as simple as running a command line but remains a major hit to many large infrastructures.
- Dangerous access control on GPO consists of modifying a legit GPO to inject malicious commands, allowing attackers to gain control over privileged accounts and servers.
Attackers are not short on options when it comes to exploiting bad configurations, and new, clever ways emerge regularly.
How SingHealth detected intruders in their AD
Well, there’s a catch. They didn’t.
Imagine: we just went through a dozen reasons that prove it’s tough to manually harden an Active Directory. Now, what about monitoring those thousands of “atoms” continuously in real time, seeking odd-looking modifications and unexpected behaviors? This is humanly impossible.
In fact, SingHealth’s incident detection came from observing the last near-death symptoms of the attack: the eventual queries run against the medical database. Admittedly, later is better than never; though let’s agree that detecting an attack after it’s burned its way through your infrastructure and has exfiltrated your most sensitive assets is not a win.
In this regard, the lack of AD-centered detection technology was catastrophic: instead of running emergency mitigation tactics to contain the attackers’ movements and stop them from reaching their targets, SingHealth was condemned to a reactive approach.
Let’s take a closer look at the remediation tactics SingHealth implemented.
There is no denying that SingHealth’s reaction after detection was swift. It came in two waves: the first, immediate one was orchestrated by SingHealth’s teams, while the second stage was led by the Cyber Security Agency of Singapore (CSA).
First, SingHealth’s remediation team created a new set of domain administrators and revoked the access of the former ones. This is a must-do that’s never sufficient but always necessary.
Then incident responders enforced the deployment of a GPO preventing domain administrators from logging into servers. The intent was to avoid a DA logging in to a compromised system, thus exposing their new immaculate credentials. It’s a widely known best practice to limit privileged accounts to particular use cases.
Antiviruses were next used to scan domain controllers. Although the effectiveness of those scans is debatable, it would have been risky not to do so.
At this point, CSA took over the rest of the incident response efforts.
Notably, they started with two consecutive resets of the KRBTGT account. This KRBTGT account is a unique account used to encrypt the access key of every domain user. If an attacker owns it, it can easily create legit authentication material to impersonate any other account. Obviously, not what you want. On a side note, it’s a best practice to renew the KRBTGT’s password regularly (e.g. on a yearly basis) since it’s not an automatic process.
Unfortunately, the operational steps to get a potential attacker out of KRBTGT’s reach are not as trivial as they seem. Because the last two KRBTGT passwords are valid (that’s normal behavior), CSA performed two consecutive resets. The tricky part here is that you absolutely must ensure with 100% certainty
that your attackers don’t get the chance to compromise the KRBTGT account between those two consecutive resets. Here, CSA performed those two rotations within a 24-hour window. We have no insight into the measures they applied to make sure no breach happened during that time frame.
Then CSA went into a password-changing spree on accounts and applications. Although operationally painful, this step was necessary to invalidate stolen credentials. Of course, this measure is only as good as the hardening implemented to block the offensive tactics used to steal those creds in the first place. Again, we have no insight into whether CSA hardened SingHealth’s Active Directory as a prerequisite to changing passwords at scale.
One last thing
Let’s assume—and there’s no reason not to—that all those remediation measures were conducted thoroughly and adequately. Is there any other way an attacker could have persisted in SingHealth’s systems?
Of course. The COI purposefully omitted publishing some of the incidents’ details in its report, so that there is no way to know whether those vulnerabilities were actioned upon or not. However, they do theoretically exist. Here are a couple of examples:
- It is possible to ensure persistency by adding malicious permissions on the adminSDHolder object. This object is used as a template for the permissions applied to newly-created privileged accounts. By tampering with it, an attacker can maintain permissions on every privileged account, even those created after the remediation mentioned above.
- Another classic consists of injecting a privileged Security Identifier (SID… ring a bell?) into the sIDHistory attribute of an otherwise harmless-looking account. This would provide attackers with de facto elevated privileges, without the caveat of utilizing an account that’s explicitly in a privileged group.
- SingHealth’s threat actors could also have associated their own certificates with existing privileged users. Those certificates’ validity does not expire when their related account’s password is reset. And because they can be used as you would a password to authenticate, they represent a stealthy way to maintain control over victim’s accounts.
We don’t intend to be exhaustive here, but it should now be evident that Active Directory is complex machinery with hundreds of tracks, branches, and hidden pathways leading to its core. After a successful attack, cleaning and sanitizing can be complex and time-consuming. Regaining trust in this system is not a given.
It’s no surprise that AD teams everywhere, and not only at SingHealth, struggle to audit, harden, and monitor it as a whole.
It’s time to care (about AD)
As an industry, we have spent billions on endpoint, data, and network protections. At a time when all major attacks have a strong AD component, we have no excuse for our collective carelessness. Whether it’s auditing, software, or processes, there are ways to make a difference. None is perfect, but having none makes us part of the problem.