Hunting cyber threats can be likened quite a bit to a real hunting engagement out in the wild. It requires patience and a keen eye. It requires having the appropriate arsenal at your disposal in order to be effective when taking down your prey. Hunting can also be a means for pest control, much like the eradication of a malware-infested environment that requires a good ol’ cleansing. Skillful tracking and acquisition of an elusive target can be very exhilarating and rewarding, but in order to do so, one needs to be well-equipped with the right skills and tools.
The first item you’ll need, of course, is the data. Having a centralized SIEM (Security Information & Event Management) with all of your data-generating devices feeding the monster would of course be preferred, but even just access to your proxy logs, anti-virus Logs, or even your Windows Domain Controller logs will benefit your hunt. It doesn’t matter if you have hundreds or even billions of events, the hunting process whittles away the noise like a digital wood carver chipping away to reveal his masterpiece. Your masterpiece will be the identification of an unknown threat to your company.
The second item you’ll need is a list of suspicious-type activities that would be generated out of the logs you’re collecting. It’s imperative to have an intelligence-driven approach to this process, otherwise you’ll feel like a gerbil caged in his spherical plastic ball just bouncing into dead end walls. See below for some examples of what to look for when hunting for threats in your proxy logs, anti-virus logs, Windows logs, and Digital Guardian logs.
Proxy Logs
- Is there traffic being sent out port 22 through your proxy servers or even firewalls? Of course it’s good practice to source restrict this clear-text protocol, but if it’s not locked down, look for any exfiltration patterns in the data. Low and slow connections.
- Are there any network connections that exhibit the same pattern of bytes in and bytes out each day? This was more prevalent back like five years ago, but malware today still leverages this technique of beaconing out to its master to let them know they’ve implanted successfully. So looking for the same amount of bytes up and bytes down on a frequent basis could be a sign of badness.
- Identify a listing of all dynamic DNS sites that are visited by endpoints and look specifically at the outliers across your organization. If only 3 machines out of 20,000 visit turtle.dyndns.com once a day, either they have an affinity to looking at the cartilaginous shelled creature, or Mr. Threat Actor has a command & control infrastructure. Of course there could be other explanations as well here, but you get the point!
- Other items to look at:
- Unique user agent strings
- Base64 encoded strings within the URL
- Executables being downloaded
Windows Logs
- Profile your “A logon was attempted using explicit credentials” event logs and whitelist out normal activity. This would be for Event ID 4648 and/or 552. This log kicks off when a user connects to a system or runs a program locally using alternate creds. Did someone say ‘Lateral Movement’? Threat actors love to move laterally like a slithering snake looking to munch on any data it finds appealing. Intellectual property, trade secrets, internal security policies, etc = lunch.
- “User Added to Privileged Group” – Event ID 4728, 4732, 4756. Escalation of privileges will often occur once a foothold has been achieved within an environment. These logs may assist in the identification of such activity. It’s good to profile your IT admin’s legitimate activities as well since they’ll more often than not cause a bit of noise themselves.
- I know this is an obvious one, but failed logon attempts. This can be difficult to filter through, depending on the size of your organization, but looking for successive failed access attempts using multiple accounts could indicate a brute force. Focusing on one failed attempt per account may signify a threat actor trying to log in with passwords they’ve previously dumped from the environment in hopes that one may still work. Windows Event 4625, 529-539.
- Low Hanging Fruit:
- Log clearing – Windows Events 104 & 1102
- EMET crash logs – 1 & 2
- Application crashes and hangs – Windows Events 1000 & 1002
- Windows Defender errors – Windows Events 1005, 1006, 1008, 1010, 2001, 2003, 2004, 3002, 5008
Anti-Virus Logs
So why would one hunt through anti-virus logs? Won’t they just show that I’m covered and AV did its job remediating the infection? Well one would hope that AV could be super-effective at identifying malware all the time, but the truth of the matter is, it’s not. AV scanners and software are all primarily signature-based, meaning they detect malware by identifying a segment of code within a file that matches their internal database of malicious code. It’s actually fairly elementary to identify this code segment that AV will trigger on, make changes, re-compile, and bammmm! Your code will no longer be detected as malicious! However, threat actors do make mistakes when they’ve made a successful intrusion. Here are some of the items to look for and reasons to look for them:
- Signs of any password dumping programs executing within the environment. This would include pwdump, gsecdump, mimikatz, etc. Research what your AV provider flags as a password dumping program and go searching! For example, one of McAfee’s password dumping detection tools is called HTool—GSECDump. I’ve seen countless times where an actor ran a password dumper, AV detected and removed it, and the attacker then successfully executed another dumper that wasn’t detected. So although they’ve achieved their initial objective, they’ve left behind a clue of evidentiary value.
- This comes with time, but knowing your adversary/ies is ultimately the goal here. Then you can begin to profile their tactics, techniques, and procedures. You’ll know the tools they most commonly use and the types of backdoors they may leverage. Some common advanced threat backdoors include PlugX, 9002 RAT, Nettraveler, Derusbi, Winnti, Pirpi, etc. If you come across names like these within your AV logs, be ‘skurrred’! So once again, I’d research what your AV vendor calls these detections and go hunt for them. The odds may be slim if it’s a targeted piece of malware whose primary purpose is to not only establish a command and control connection, but to NOT be detected in doing so.
- Identify any detections with the name ‘dropper’ in it. A dropper program is intended to download/install a backdoor or virus. It’s kind of like sending your younger sibling into the upstairs hallway late at night to see if the parents are still up so you can power on your Playstation for some gaming. The dropper is the ‘coast is clear’ guy which then initiates the second stage of action. If a dropper has been detected, it’s possible there is still something lurking in the depths of the OS it was detected on.
- Some AV solutions also have the ability to create custom detections. McAfee’s AV has a section within their ePolicy Orchestrator to configure ‘Access Protection Policies’. So for example, if you want to log executions of binaries from a user’s APPDATA directory, you can create an alert with the following syntax “C:\Users\*\AppData\Local\Temp\*.exe”. Now any time a program launches from that directory, a log will be generated and sent to your console.
Digital Guardian Logs
I try to be as vendor-agnostic as I can in my blogs, but I may reference a product I’ve used before and DG is a perfect example. I was a customer of Digital Guardian for over 8 years in my prior job and had incredible success hunting threat actors leveraging their logs, so I figure I could give some tips using DG as a source.
Process execution logs are registered any time an application launches on the host and contain a wealth of metadata about the binary that has spawned. Drilling down into binaries with the following traits is a great starting point for finding evil:
- No company name or version assigned to an executable launching from a temporary location. See below, sure looks like malware to me.
- Svchost.exe launching without Services.exe being its parent process. Bad.
- Processes launching from extremely odd directory locations: %windows\fonts, %windows\help, %windows\wbem, %windows\addins, %windows\debut, %windows\system32\tasks, etc.
- Rar.exe being executed to compress files. I mean c’mon, how often is that legitimately going to happen besides an IT guy maybe running some backups? In that case, have him rename it to rar-it.exe and whitelist so you can spot any potential staging activity to exfiltrate data. That goes for lateral movement tools as well such as psexec.exe. Come up with a standard naming convention for those types of tools and anything that deviates outside of that, flag it.
- Analyze your rare process executions within the environment. Over a 24 hour period or up to a 7 day window, look at the processes that only launched once or twice on a single machine or a few machines. Those events always yield something interesting.
- Whoami.exe being executed. Yes this program is completely legitimate to show the name of the currently logged-on user, but as you’re hunting through your whoami.exe execution logs, look for any entries that stand out over a 30 day time period. This is one of the commands that threat actors love to execute to see what account they’re logged in as after compromising a system.
- Navigate through your scheduled AT jobs (at.exe) and review command line entries. Adding registry keys, executing processes, transferring executables, etc = Not good.
Now I know what you’re thinking. Looks cool and all, but maybe right now my company can’t afford an enterprise solution and I’m just a single incident responder with a budget the size of a microbe. Well, fear not dear reader. If you’re operating on a shoe string budget, there are some pretty good open source tools out there with the ability to cover some of the examples above. Grab a copy of Sysmon from the Windows SysInternals site. This freeware solution has the ability to log all process creations with full command line, hash, etc, along with the ability to collect network-based activity and is stored within C:\Windows\System32\winevt\Logs\Microsoft-Windows-Sysmon%4Operational.evtx. The install process is ridiculously simple, however you’ll also need to configure WEF (Windows Event Forwarding) to send these logs to a centralized log management solution. Setting up WEF with Sysmon was no walk in the park though, I must admit. You have to make manual changes to the registry and it becomes a Nightmare on Elm Street. If you’re open to having an additional agent, NxLog is an open source tool that can forward those logs with ease.
Finally, you’ll need a solution that will now ingest this data and provide an intuitive interface to hunt through. If you haven’t heard of the ELK stack (Elasticsearch, Logstash, & Kibana), it’s the perfect free solution to meet those needs. To simply test it out quickly, there’s an ELK Vagrant box that leverages VirtualBox for a virtual machine and is available on GitHub that can have you up and running in under 10 minutes! No joke, I timed it. Refer to my Reference section below on how to configure the conf file for both Logstash and NxLog. There’s literally nothing out on the web that defines how to do this correctly so I’ve just saved you an inordinate amount of time. You’re welcome!
Conclusion: Happy Hunting!
I could probably go on forever here with what to specifically look for during a cyber hunt, but those are the types of events you want to be sifting through day in and day out. Threat actors do everything in their power to blend in and attempt to become a ghost in your network. My recommendation? Become a Ghostbuster. Load up your ectoplasmic gun of cyber threat knowledge, centralize all of your critical logging data, and pivot, pivot, pivot until you find your Slimer. Then down a box of Ecto Cooler Hi-C and do it again.
I’m the Director of Cybersecurity here at Digital Guardian and run our ATAC (Advanced Threat & Analysis Center). We have a team of cyber experts that conduct hunting engagements every day across our customers’ DG Agent data to identify malicious activity and unknown threats. We have hundreds of rules and pivoting points that we leverage to locate the presence of attackers within an environment and assist on mitigating and neutralizing the threat upon discovery. So if your organization is unable or ill-equipped to conduct this imperative activity, we’re here to help.
For more threat hunting tips, check out Stopping Cyber Threats: Your Field Guide to Threat Hunting.
