Continuing my discussion of common classes of attacks, this time I’ll be covering rootkits and rootkit detection.
What is a rootkit? You can see it right in the etymology of the word itself; it’s a combination (kit) of software that, once root access is achieved, can carry out stealthy activity of a sort that is usually, but not always, malicious in nature. Detection therefore can be fairly difficult.
The advantage of having root access is that, as the initial (or root) user you have superuser privileges, giving you full rights to all files and programs on the system. When the host is compromised via this level of access, detection of the rootkit can be thwarted by sophisticated malware, because the tools an analyst might use to detect or resolve the problem might be manipulated by the malware, causing it to yield bogus or incomplete information.
For instance, consider the infamous case of the 2005 Sony BMG rootkit. Initially, Sony authored a package of software for Windows operating systems that had the relatively benign goal of preventing users from copying CD content. To escape detection, the rootkit modified the operating system in such a way as to prevent all files beginning with a particular prefix from being revealed in searches. Its own files then, of course, were given that prefix.
Today, rootkits are typically combined with malware and, as a rule, are much more sophisticated and much less benign than anything Sony imagined. You can classify them by the level of abstraction they occupy on a system:
- Rootkit detection:
- User mode. These operate as user-level tasks, usually by invading existing processes, overwriting application memory with their own contents, or both.
- Kernel mode. Moving down in abstraction, these rootkits modify the operating system itself (kernel and/or drivers) and are thus substantially harder to detect and eradicate because they can conceal themselves more comprehensively. If you’re using a compromised operating system to look for rootkits, it’s a tool whose results you can’t trust.
- Hypervisor. Still lower in abstraction are rootkits which modify or replace a hypervisor used to govern virtual machines, each of which is running its own operating system on a shared host. At present, rootkits of this type are not present in the wild, but proof-of-concept examples have been developed.
- Firmware. These are extraordinarily difficult to address because they are, in a practical sense, embedded in the hardware itself — for instance, a computer BIOS or router firmware — and hence cannot be eliminated even by replacing the operating system completely. In some cases, replacing hardware may be the only plausible solution.
What rootkits all have in common is that they are used to disguise classic malware activity (logging keystrokes, sending credit card numbers to an offsite server, collecting and uploading banking credentials, establishing hidden backdoors for subsequent access by the attacker, etc.) to prevent rootkit detection. And once installed and running, rootkits can lead to disaster as attackers escalate from one application or system to another.
A variety of investigative detection techniques
Fortunately, as usual in security, it’s more of an arms race than a one-sided victory. While rootkits have gotten more sophisticated and diverse in nature, so have the tactics and tools available to deal with them. Rootkit detection methods, for instance, include:
- A trusted analysis host. To bypass the problem of a compromised OS that may have been modified to conceal a rootkit, simply use a different host, with an OS known to be secure, to do your analysis. A CD, for instance, can’t be overwritten or compromised, and can be booted from.
- Signature-based. Much as antiviral solutions look for predictable signatures, byte-level rootkit detection can as well — but this approach is usually only effective against older threats.
- Behavior-based. Once installed, rootkits modify system performance in subtle ways that are sometimes detectable. For instance, the timing of API calls sometimes slows, and CPU utilization sometimes climbs. A known clean system with otherwise identical hardware and software can be used to establish baselines for comparison to aid in rootkit detection.
- Integrity checking. The idea here is to compare key files or Windows registry entries on a suspect host with clean examples to see if they’ve been changed in any way.
- Difference-based. Are installed binaries on a drive identical to their RAM-resident counterparts in a working system? If not, that’s a bad sign (though false positives are also possible).
- Memory dumps. Rootkit detection can also be accomplished by analyzing virtual memory dumps because the rootkit hasn’t got a chance to detect and block the analysis — though getting a proper dump may also require separate hardware.
As you can see, it’s apparent that rootkits pose a formidable threat, and not just because they’re sometimes installed by vendors (like Sony) or created by bad actors (like hackers). They are swiftly evolving in incredible ways as researchers break new ground.
For instance, the Jellyfish rootkit was developed last year to show that it’s possible to install a rootkit in a graphic processing unit! This way, it inherits the processing strengths of GPU hardware while also achieving a new and impressive stealth via a logical location in the host that is beyond the purview of most OS-based rootkit detection tools.
And researchers showed last August that it was possible to develop rootkits that exploit a previously undetected flaw in the venerable x86 processor architecture — specifically the System Management Mode, which was added nearly twenty years ago. Staying on top of threats like those, should they be released in the wild, will require security professionals to stay current and may also mandate a new class of security solutions for rootkit detection.
