(light, airy music)
(echoing note)
(swish)
(applause)
- Thank you for having me.
I'm sorry my co-worker couldn't make it.
So, I'm going to try to go through it.
We had it all separated so we'll see how it goes.
Just a quick About Us.
Andrea, she's the one who can't be here, obviously,
she's been working at Cisco for about
two or three years, and a lot of her research is
in this presentation, of course.
And there's me, as mentioned, I worked with NASA
about five years; I took a year off, and went to Mandiant,
went back; and was their threat researcher.
One of two threat researchers for
the whole agency, which is crazy.
I've done consulting for nonprofits in San Francisco,
where I'm currently based, and I worked
for Cisco Umbrella, so previously OpenDNS,
and get to play with a lot of DNS data every day.
I'll give a quick summary of what I'll be talking about.
We're gonna talk about the kind of research that we do,
and how we find maliciousness, why we build visualizations,
and I'll show some visualizations and some findings.
So, we research a lot of criminal
activity, like ransomware, spam,
Remote Access Trojans, infected websites.
When you get spam, you might get something like
this email, where you have Collete87 really wants
you to click on this Russian website,
so you can meet her, and if you do, you get ransomware.
We look at website compromises where iframes are injected
into a website, and it sends you off to other locations
to install malware, do all sorts of things.
Look at Remote Access Trojans, so malware on a system
that allows an attacker to connect in
and control the system, exfiltrate data,
drop new malware, things like that.
There's a lot of data to visualize.
So, we can put, look right at an IP or a domain
or a registrant, it builds up quickly.
It depends on the kind of attack,
when you're looking at what you want to visualize.
So, compromised domains, it depends on the scope of it.
Maybe it's just a command shell, it's doing something
bigger, or maybe it's just a simple compromised domain.
Maybe it's just phishing, so it's domain
that's set up specifically for malicious activity,
or it's a compromised website with phishing
deep in the directory structure.
Or DGAs, so you may have one complex domain name,
and that doesn't give you much information,
but the more you have, the better information you have.
It depends on the infection method.
So, you know, if it's the spam infecting users
that's one thing we're looking at.
If it's just a simple compromise of a website.
Maybe you've got one indicator of compromise
that doesn't give you a lot to go on.
And exploit kits, maybe what you're looking at sites
that are compromised via exploit kits.
So, this is a service where you can pay to compromise
sites, and try to compromise users, depending
on their vulnerabilities in their browser and their system.
It also depends on the size and scale.
So if, like I said, if it's just a defacement
of a website, you don't have a lot,
or if it's one DGA domain, one
dynamically generated algorithm domain,
it's not a lot to go with, but you can build off that.
There's a lot of features you can get from even one IOC,
one domain, in this case magicpharmacyinc.su
See, there's this big spike in traffic?
Compared to another domain, when you have domain queries
like this, where you see a regular traffic
pattern that doesn't look suspicious.
So, finding maliciousness, you have to find
all this stuff, and then do stuff to it.
But, it's kinda difficult to find.
But, if you have a lot of DGAs, you can analyze
a whole bunch of them; a lot of them won't be going
anywhere, they won't be registered, they're NX domains,
but you might be able to find one that is pointed
to the command and control server,
and then get additional information on that;
and the more DGAs you have, the more likely
you can find out what it's attributed to, as well.
Here's an example of an infection where you
might have a command and control contact,
and what you can get from that.
So, at Layer 1 at the top, it's a system's infected,
and then it contacts the C&C server;
and if you have visibility into this,
you get the C&C server, and you have the victim,
and occasionally on obfuscation, there will be
multiple proxy nodes that it will contact,
so you can get those nodes as well,
and then you can build a dataset on that,
and you can keep on going, as you'll see.
So, you need a large amount of these changing DGA domains.
You can get nonexistent domains;
they're not all registered, as I mentioned.
One of these domains, maybe a few of them,
will be pointed to an IP address.
So, a lot of this is using passive DNS,
a lot of the stuff I do.
So, I'm looking, like I said, at domain names,
I'm looking at IP addresses, name servers, which is,
we can get historical name server and WHOIS information,
which is important; which EDPR, so you can see changes
in that, and if you tie in other things, as I'll show,
like VirusTotal or ThreatGrid, you can get add-in
malicious binaries, and the relationships
between IPs and domains and stuff.
So, using one example, is just tracking Hailstorm spam.
Hailstorm spam is just a lot of spam sent out,
a spam campaign done really fast before spam, anti-spam
mechanisms can catch up to it.
Just looking at this one IP address,
what can you get from this, this one thing?
Using passive DNS, you can see that there are
a lot of domains hosted at this IP: 857.
There's also Level Two domains, there's 422,
that's subdomain, .domain.com, and you can keep on going
down the chain of subdomains; and then you can get
basically a list of that, and do something with it.
Just looking at a select example of that,
you can see that there's a lot of domains
that look like medical, luckypillmall, firstdrugmall,
so they're usually, they're probably pharma-spam,
which is attributed to one kind
of ransomware that we've seen a bunch.
Pivoting off that, you can take that IP address,
which is on the lower right, and you could see
that there's another domain that
has been pointed to other IPs.
So, you can go, look at those IPs,
and see where they've been pointing to.
Basically, you're adding to your dataset,
and it's getting probably pretty overwhelming.
You keep on going, you find more domains,
more domains, more.
Just to go on what you can get with one specific domain,
the kind of things you can, before I get into the big stuff.
You can see query volume, if you have this
kind of visibility; so you could do this in
your environment, looking at your DNS queries,
or if you have access logs; you could also
do it if you have something bigger, like where I work.
If you could see the spike in traffic, all of sudden
when there was very little, almost zero to one queries
per day, and then it drops down.
You can get other things, like the WHOIS data
I was talking about, IP addresses.
The TTLs, so you can see if it's changing IP addresses
often, or if it's set for a while.
You can pick up the popularity based on requester
geolocation; so how many people are visiting
from this country versus that country.
And what's really interesting, which is something
you could also do inside your own environment,
without third-party tools, but is more powerful
with third-party tools, is co-occurring domains.
So, these are domains that, when our DNS query was made,
right, at the time time, or just before,
or after the domain was queried.
So, then you can use that to branch out.
Of course, this adds to your overall data that you're
looking at, still so, there's still a problem here.
Collecting all that stuff, you still have to,
I mean, when you have one domain, you still
have to get this data somewhere.
So, we use a lot of Open Source Intelligence.
Things like MALWARE-TRAFFIC-ANALYSIS.net
A blog by a security researcher who publishes
PCAPs, live malware, and a quick blog post
on what he's seeing; and you can
download this and play with it.
It has things like, there's a PCAP right there,
and I'll get into this shortly.
We actually, well, actually I guess
I'm getting into this now.
You can download those PCAPs, you can run them,
and you can look at the domains; but it's a lot easier
if you program this in Python.
You can process, batch process,
through tons of them at one time.
So, you end up with a list of domains,
which is problematic in itself,
because it's a list of domains and lot of third parties
will send you lists of domains,
and there's legitimate things in there,
and there's non-legitimate things,
so you still have to weed out things and clean your data.
There are other sources, like My Online Security is one.
I mean, there's a million sources out there,
but you can scrape these, or you can go download things.
If you're, as a security researcher you get access
to places like this, DGA Archive,
which are closed-off to the public,
so that attackers can't see what you're working on.
And of course, there's VirusTotal, which is a really
great repository for malware and other viruses.
If you are doing it for free, you can use an API,
you can call it four times a minute, I believe.
But if you're paying, you can do a lot more.
So, you can set up your RI rolls,
and hunt for specific kind of activity,
and then download all that stuff
programatically, which is what we do.
We download hundreds of samples a day,
and maybe run them in a Cuckoo Sandbox,
or run them through various systems to get
additional information to look at.
We also look at feeds.
So, there are some feeds that are
providing constantly changing viruses.
They're usually a little bit more up-to-date
than the historical data that we're finding
on VirusTotal and Malware Traffic Analysis.
One example of a really great feed is John Bambenek.
He runs a security company; I'm sure
some of you have heard of him.
He just runs all these malware samples, every day.
I think he actually might be stopping the feed pretty soon.
I heard, but I'm not sure about that.
But, he publishes them as the malware families
that they belong to; so, you can
quickly ingest that and work with it.
So, why visuals?
I feel dumb asking this question.
I mean, I know why visuals, I feel like
it's extremely important to be able
to see what you're looking at.
But, as you can see, just to go over it again,
it's a lot of data that you're looking at.
A ton of data, and it's really hard
to keep it in your mind, what you're
looking at; and you wanna work fast.
Threat hunting is, it moves really fast.
You can't just sit and wait for something to change.
'Cause, as you can see, a domain
spikes up in a campaign, and it's gone.
So, you need to turn that messy data
into meaningful information that you can
work with really fast, and move on to the next thing,
and also go home and have a life.
It also helps us quickly analyze threat data.
This is a quote from Edward Tufte.
It's basically kind of long-winded.
He's a data visualization expert;
a professor at Yale, I think.
But, he's basically saying assessing the change,
and the dynamics, and the cause of something,
helps you really understand it.
So, some visuals we like to use as we, like I said,
we program in Python, we've had some coworkers
that work with R, but we convert them.
I mean, R is fine, but we use Python.
We throw it in PANDAs for data analysis.
Sometimes, use other tools, and then
we have been playing a lot with D3,
which is a JavaScript library,
and it's a pain for me to work with.
I'm always struggling with it, but the things
I've gotten working have been satisfying, at least.
So, we will do some specific kinds of graphs, like
force-directed graph; this is really great for networks.
We'll do timelines of first seen
queries versus domain queries.
So, this is the first time a system in your network
contacts a domain versus the time it was registered,
also versus other things, other components of the domain.
Timelines of domain queries, this is just the volume
that you've seen before in some of the previous slides.
Timelines of queries just from
network captures, and log files too.
And then, signature patterns built from queries
in network capture files; so I'm gonna get into that,
that's a little bit different, it's something
I'm exploring, I've been exploring for about two weeks.
An example of force-directed graphs, how you can
take a list of domains and IPs,
and get something maybe useful?
I mean, this is a mess to look at,
but it gives you an idea of maybe where to go next.
You have this IP address, and it's got
a whole bunch of domains on it.
So, if this is something bad you saw happening,
maybe you know where to go next.
You have all these one offs on the left side
that maybe aren't as pressing, so...
And while I'm talking about force-directed graphs,
this isn't about this, this is a fantastic product
that's actually a free thing called OpenGraphiti,
made by one of my coworkers that used to work at NVIDIA.
So, he's made this beautiful thing.
This is an example of what Zeus, Game Over Zeus,
looks like in one environment.
So, obviously you can really visualize
things, and just look at it all day.
Here's a video of what Mirai looks like,
like a section of Mirai botnet.
So, you can do some really beautiful things.
I'm sorry to show you that, 'cause we're
gonna go back to really boring-looking visuals, but...
We can do, as I mentioned, on the query timeline,
so if you look at Google you can see there's a pattern
here, there's a Monday through Friday pattern.
It's kinda normalized for North America, in this graph,
but it goes high up in the day, and down
in the evening, and the weekend is lower.
So, you can see there's nothing, basically nothing
malicious here, I mean, you know it's Google.com
But, you can see this kinda pattern, or other patterns
that you recognize as not bad, then you can move on.
And in doing pattern matching, so I'm gonna
talk about this towards the end,
but I'm looking at PCAPs with specific
kinds of activity and counting the times in-between,
trying to find if this pattern is involved.
A different way of building signatures.
So, what makes a visual useful?
We have some questions that we ask ourselves.
What will make it clear?
This is pretty obvious, it has a count, if it's
a timeline, you know, how many, and when did it happen.
So, in this case we've got this graph here.
The line, the blue line, is the query volume.
So, we can see that this domain, humoronoff.top
had a big spike kind of, a relatively big spike,
and then it went down, it's still being used,
but so, if it was in a malware campaign, or something
then it's probably something to look at.
Am I putting it in the right context?
So, force-directed graphs, those are great
for networks, terrible for timelines.
So, you gotta make sure you put it in the right place.
I was gonna put some really ugly graphs here,
but there wasn't a lot of time.
Also, how can we reduce the visual clutter?
So, maybe we can combine domain names.
So, here we're looking at Google and the humoronoff.top
domain, in comparison can be useful.
And looking at, we can assess change by looking
at spikes, and changes, and traffic volume.
So, some visual examples of attack campaigns.
Just to give you an idea of where we're going.
I'm going to talk about three different malware variations.
EMOTET, Hancitor, and TrickBot.
EMOTET is a mal-spam malware.
Spam is sent to victims and it usually includes
an attachment, like a Word document, or something like that,
or a link to a Word document; and if you run it,
and have macros enabled, it'll do all sorts of stuff.
So, grabbing just from Malware Traffic Analysis,
just from a bunch of PCAPs, for a selected time period,
we ended up with what you're commonly,
probably see, in your daily job.
Domain lists and IPs, and it's the job of some
poor analyst, which I've been before, to look at these
IPs and domains, and figure out which is bad.
Someone higher up says, "Block this."
And you're like, I don't know if I should block all this,
but you should definitely look at it.
But there are domains in here that shouldn't be blocked,
obviously, like ipinfo.io; these are just IP checks
from the malware, maybe, trying to find out
if it should infect you, or find out where you are.
So, you need to do something else with this.
If you put all these IPs and these domains
that you just saw into a force-directed graph,
you get this cool-looking graph that looks kinda like candy.
I don't know.
You can see that there's not a lot that sparks interest,
except for, there's one section where there's a few
more connections, so maybe you could start there.
You can check out what's there, and you could,
in the stuff that we've written,
that we use, we can highlight the nodes.
The red is is the IP; the blue is the domains.
And we can get some more information on that.
We can click on that and go to
further resources that we use.
If we run those same domains through to create
that kind of messy graph, and we enrich
it with passive DNS data, so we include
a whole bunch more stuff that wasn't in the initial file,
and we turn the labels on, we can see that there
are a couple IP addresses that
are hosting domains associated with this,
or have hosted domains; so, there are some more places
that are more interesting to look at.
This mykeeptake.xyz, we'll see that again in
a few moments, but just keep that in mind.
We can do some timelines on this kinda thing.
So, if I take just the PCAPs from one EMOTET infection,
just these two files right here, and I pull out
their domains; I've got those akademia, I'm not gonna
try to say the names, this one here.
I pull them out with my Python script, and I've
got these four domains here that are interesting.
I didn't pull the DNS queries, just the GETs in the post.
I can put these on a timeline, which is
a different way to look at a timeline.
I put the dates the emails were received
by victims, according to this blog post.
So, there's probably a lot more emails than this.
But, put those on a timeline, I put the domains,
and you can see that there's three domains
on the far left, back in October,
and then there's this one domain right there.
So, this gives me a quick image of
psychedelicsociety.org.au is probably really
associated with this, and the other ones, I mean,
it's probably compromised or it's been setup
just for this campaign, right here.
Those old ones are probably compromised, or, I mean,
they're not as important, but they are important.
Looking at that psychedelicsociety page,
you can see there's a spike in traffic,
all of a sudden, where as before,
around it there's pretty much nothing.
I can see the First Seen date is
the date that was on my graph.
Looking at those other domains, you can see
that there's kind of consistent traffic, it's not as big.
I can look using other tools, like VirusTotal,
I can find that there's some interesting URLs,
and I can scrape these or pull
them programmatically, which I do.
Looking at, just at the site, sometimes I do this a lot
in threat analysis, I see what's up.
It's a site, whatever, but it's
WordPress, which I am suspicious of.
I could tell by those previous domain names
there's a word, WP Admin Directory,
but I didn't take screenshot of it;
but going into one of those directories,
you can see there's some spam inside those sites.
So, the site's compromised, and it's
being used in this campaign; as you can see,
'cause of the PCAP, I mean, that's proof.
But looking at all the EMOTET domains,
I pulled like all of them for a huge amount of time,
I can't remember the exact time,
but you can see that there's a pattern here.
And it's a mess, but you don't have to read all the domains,
but you can see October had a huge spike in domains
being used, kinda dipped down and then up again.
So, this isn't relative of the entire internet,
this is some PCAP files, but it's place to start
if you're hunting for something.
So, there's some patterns we can look at,
and I've been talking about this a little bit,
this is stuff I'm doing that's brand new.
I'm just trying to play around a little bit,
which is a big part of my job,
and that's my favorite part, to play around.
So, I can get those PCAPs and I can see
that there's GET requests, and there's posts,
and there's all sorts of cool data to play with.
This is, right here, that you're seeing...
There's GET posting to different IPs, different domains.
I thought about maybe, if I look
at the time between events.
So, Malware Traffic Analysis is real great,
because he provides these PCAPs that it's just
a specific moment in time where EMOTET, or Hancitor,
or something is, or some other malware,
is infecting a system; so, it's pretty consistent.
So, I was thinking, if you can count the times in-between,
maybe there's a way to look at patterns,
and then research further on will be more interesting,
'cause I hope to be able to run it on live traffic,
and maybe catch something interesting.
But, I'll keep working on that.
But, looking at it, just, I put it on a graph with
with MatPLotLib, just to see what it looks like.
These dots are just the, they represent...
Well, what I did, was I took the times
and I took away the time information,
and I made it a unique number, not to get too weird with it.
But, just to try to do a constant thing.
So, as I did that, I would plot it on this line.
So, the numbers don't really mean anything;
it's just a visual for me to kind of to start with.
I'm looking to see if there are patterns.
So, if I run a bunch of EMOTET PCAPs,
you can see that there's, there's sort of a pattern.
I mean, maybe, I haven't yet compared it
to what's it like when you're viewing YouTube,
or viewing news websites, but you'll see
when I get to TrickBot, there's
a little bit more of a unique pattern.
And I'm at TrickBot.
So, TrickBot is like a mal-banking Trojan.
Sent usually phishing links, or phishing emails
are sent to people to login to their fake bank.
They think it's the real bank.
And it's changed a little bit to have kind of a
ransomware component too, but I won't go into that.
So, let's look at some graphs with this.
So, putting it in a force-directed graph,
we can see there are a few interesting things.
These are the same kind of, we took a list,
just like we did for EMOTET; we can
look at the nodes of this, we can expand upon that.
This is really pretty, it doesn't tell you a lot
of information, if you don't know the context, but you
can see that there's something interesting to look at.
So, we turn the labels on and we enrich it even more.
Actually, that last graph we enriched already,
but with additional passive DNS information,
but you enrich it even further and add the labels.
You can get a lot more useful information.
You can see that there's something; there are a couple
probably bad hosting providers that are hosting
a lot of these domains, so you can take action on that.
Whatever that action may be.
Looking at timelines of this, just
to give you what it looks like.
Same thing, October, big, huge thing, comparatively.
And then some smaller bits.
And then patterns, of course, I wanna look at patterns.
So, this is running through a bunch.
This is actually a video, it's showing
a bunch of different TrickBots.
You got these two, two times that things happened.
So, it's pretty unique.
As I work on it, maybe I'll be able to catch TrickBot live.
I mean, I'm not building the next Snort, or anything,
but, you know, I'm just playing around.
Hancitor, which is another kind of mal-spam.
Look at this and see what we can find.
We've got timelines, that kind of have
the same pattern again, so I don't know what's going on.
Maybe this is a general, a larger trend.
Maybe if I look back at what I've been doing,
for the last year, if I can even remember,
'cause you're always on a million things,
I would maybe be able to see that there's, like,
maybe I would assume that October was a hard month.
So, putting this in a force-directed graph.
Same kinda thing, you get a lot more connections.
This is a lot more interesting, right here.
And if we enrich that with extra data,
the past domains and whatnot, we get some more stuff.
And as we were researching this, we found that there
were a lot of relationships, as you can probably guess,
based on those timelines I showed you.
A lot of relationships between the different campaigns.
But, we can see things like this IP is hosting
a whole bunch of these iffy, or bad, or known bad domains.
There's that mykeeptake.xyz, which was in EMOTET,
which is also being used in this Hancitor campaign.
So, now that's been raised up a little bit
as something important that we need to focus on.
Looking at patterns, just to kinda give you an idea,
it's definitely, definitely different than EMOTET,
definitely different than TrickBot, the PCAP pattern,
but it's kind of interesting to see.
And then the relationships between attack
and infrastructure, I just mentioned this,
but this is everything all tied together.
All the static domains and IPs from TrickBot, EMOTET,
and Hancitor that we looked at.
You got these outliers on the side,
and then those are interesting, for sure,
but you've got this huge amount of just working together,
the domains are working together,
and IPs are working together, it's the same infrastructure.
And as, when you're trying to find attribution
for something, generally you find it based on
the sharing of infrastructure and the reusing
of domains and registration information.
But, it's just nice to see that.
If you enrich that further, you can find a place to attack.
And this is historical data, but there are ways to
be up to speed with the present moment.
And then I put these on, or just relayed them,
this is not a programmatic thing, but it will be,
when I'm continuing to work on this, and so is Andrea.
So, if you put the Hancitor over EMOTET, you can see
there's a definite relationship between them.
On TrickBot, it's not as much the same,
but it's, I mean, EMOTET and Hancitor are mal-spam,
and TrickBot behaves a little differently.
So, you expect some different results with it.
This probably should have been in back a little bit further,
but you can, when I was talking about bringing data
together, to give you better visuals, but you could,
I didn't want to put all those domains on
one big, huge timeline, but I bet I would see some
query volume patterns, which would be very useful to see.
Cool, thank you.
(applause)
(piano music)
(deep, booming piano chord)
(piano music)
(metallic echo)