(dramatic music)
(audience applauds)
- Welcome to Quantify your Hunt
and if you saw this at Besides Charm,
I am very pleased to remind you,
this is not that talk.
This is about,
about half of this is gonna be new content
and it's gonna end in a very different place.
So without further ado, let's get started.
This is the agenda.
We're gonna cover all of these topics
at some point in this talk.
We may not have time for questions
but we will stick around this evening,
so if you've got anything you wanna follow up with,
please let us know.
And so next, who we are.
We're not as interesting as this topic.
Yeah, my name's Devon.
I spend a lotta time doing IR
and more recently working on endgame solutions,
basically trying to make this easier for threat hunters.
Particularly things like,
make it easier for you to get to initial compromise
so you can really contain very quickly.
I run our SOC, I run our IR team
and I do a bunch of other stuff in the public.
I'm not the original author of Red Team Automation
which is one of the simulation frameworks
that we kinda talk about,
but I am its biggest spokesperson.
Roberto take it away.
- All right,
thank you very much everybody for coming today also.
And it's my first time here in the Threat Hunting Summit.
So, I'm very excited about this opportunity.
And I work actually at SpecterOps
and my, what's it called, Twitter handle is Cyb3rWard0g.
I think you guys have seen some of the projects out there.
I'm the author of the ThreatHunter-Playbook,
the HELK, you know, OSSEM which is more like
documenting datasets and things like that.
And the primary two goals I guess that I have
with my projects is always to learn something new,
because that's how I learn actually little bit
about malware attack,
a little bit elastic and things like that
is just to learn the application
but at the same time also to share
pretty much everything that I do with the community.
I think that that's what keeps me going
and every time I have an idea,
I'm like, you know what,
let's make it an open source project
and then just share it with everybody.
And I'm the former Capital One Senior Threat Hunter.
That's where I actually had my first experience
to understand data at scale,
'cause I had never worked with more then 500,000 endpoints
and things like that.
And that was actually really interesting to see
how things change from a 20,000
to 500,000 input environment.
All right, so why this talk?
And I've heard a lot about different talks that happened
today and yesterday
and there's some great ideas about how you can start
being effective hunting and things like that,
but something that we like to share today
is to start kinda also focusing on the things we have seen
companies still struggle with.
And one of the things could be just as basic as mapping
your hunting engagements to business use cases
and how you do start sharing the effectiveness
and how you're going over with your hunting engagement
but at the same time also how you can start
showing transparency to your senior leadership.
Right, 'cause you gotta show 'em
how you're doing your progress over time
and things like that,
how you map your tools back to,
for example techniques and things like that.
That will show also the effectiveness of hunting engagement.
And of course at the end for people
that don't know where to start and things like that,
I believe that this talk also helps you to understand
how you can prioritize what you do you
before you want to start hunting engagement
and things like that.
And we're gonna start using also
some type of understanding of using simulation for
measuring your hunting engagement
and things like that as well.
But first it's very important to understand
exactly what effective hunting is.
And I think that that's pretty much what drives exactly
what you do in a hunting engagement, how you prepare,
and things like that.
And the problem that I see a lot in the industry
is that there is this buzzword that you can use,
you know, everything is effective.
Right and you can use a product,
you can just use anything
and can just be an effective threat hunting tool.
- [Devon] You could dance up on a robot with a dress on.
All those things work.
- The idea is to start, make sure that we understand
why that's so important to define
this specific word, like effectiveness.
And I think I've heard people using efficiency,
efficacy, and effectiveness like if it was the same thing.
Problem is that actually, efficiency is
when you start making the most
out of the resources that you have,
and then efficacy focuses more on achieving the objective.
And I think that there is where we start thinking
from a hunting perspective,
we say we gotta find evil
and we gotta uncover an incident.
You start just creating this mentality
that that's all you have to do when you hunt.
Well the problem is that if you wanna be effective,
you gotta take pretty much both in consideration, right?
You have to understand how you can achieve your objective,
but how you can do it using the resources that you have.
We've been talking about data a lot in this past couple
of presentations before ours,
and I'm so excited that actually
that's being also shared with everybody, nowadays,
'cause last year I didn't hear that much about
basing your hunting engagement
on the data that you have and things like that.
So effectiveness is huge.
Understand like why, how you're gonna start mapping
that to your engagements.
Now in this, I can tell you
that from an efficiency perspective,
that's when you start thinking about,
MITRE ATT&CK, for example, like it's a number one,
where you have to be able to categorize
your techniques and things that you're doing.
And I think that that's pretty much where it fits.
It's that it helps you to be a little bit more efficient
than just saying, let's hunt for,
I don't know, like this new thing
that I don't know what it is, but let's just figure it out.
At least the data allows you to create some type of like
order in what you do,
understanding what you have
because you wanna make sure that it's not just,
I would say like, normal just to say
let's enable processes monitoring,
but then at the end you don't understand exactly
if the data's complete and things like that.
So you gotta be more efficient into what you do
and what you enable in your environment.
And at the same time
understand if you have the right people,
right, the right skills, right?
Because as we were hearing also Rob Lee talking about
that if you even apply a machine learning model
and things like that,
if you don't have the right people that can understand that,
the results and how you can even make it better
with the data that you have,
then you might not be as successful
with that specific model.
And efficacy of course in this part,
is where we starting thinking more like,
if you measure your hunting engagements
by just how many incidents you uncover,
let me tell you, you're gonna definitely have
a really bad time telling your senior leadership
what you do everyday
and how that's effecting your organization.
'Cause you might find something today,
but in probably in three months you don't find anything.
And then it turns back into like,
hey you're actually not doing anything,
from a hunting perspective.
So that's where you show it more from,
hey I'm being efficient,
I'm actually validation the detection
of this technique rather than just saying
I'm gonna uncover an incident today and things like that.
- [Devon] So where do we start?
This is the question we often get
and we're gonna give you a couple of ideas
for places that we think are reasonable.
You know obviously if we just kind of run off
and randomly do this,
we're gonna wind up getting nowhere.
So our suggestion is to be a little bit more targeted
and to think about the type of attack model that's helpful.
This really started,
I think with some of the research that Roberto did.
So How Hot is Your Hunt Team,
I think is one of the first blog posts I read
that got me really excited about threat hunting again.
I felt like it kind of had fallen off a cliff
so to speak, there wasn't a lotta innovation.
And then as soon as this was conceptualized for me,
actually I think that's when Roberto
and I started collaborating a lot more often.
And then much more recently,
Ready to hunt? Show me your data.
Basically assessing data quality
as an input in this process
which I think is really foundational.
- And the idea with these two blog posts,
'cause I wanna make sure that everybody understands
the value that I was trying to share with this.
Like the first one with the heat map,
was just the idea that if you wanna show
some type of either coverage or how good you are,
or how you can pretty much track your progress,
visualization goes ways.
I actually talked to senior leadership
when they were trying to buy a tool
and they explain to you
how you can tell them how effective you can be
with that tool and things like that.
If you start going through like five, six slides,
they just start looking at their phones
and they don't even care,
but if you're at least showing a visualization,
then the conversation kinda like starts going through
with your nice heat map and you can talk about it
a little bit where your weaknesses are and things like that.
That's a transparency that you gotta start going forward.
But that was just an idea.
That was a how you can start at least showing something.
Then how you, you know,
build on the top of it and how you use it,
that depends on your organization
and exactly what you wanna show with it.
And the second one was of course
just focusing more into, hey,
it's not as easy as just saying,
I'm gonna hunt today.
Let's go on a hunt for this weekend.
No, first show me the data that you wanna hunt with.
I think that also was mentioned before,
that that would avoid probably spending 30 days
of a hunting engagement that you might not even have
the data to work with.
And how you gonna start measuring that data,
like the quality of your data.
So there's where we start.
So what are you potentially measuring already?
This is also very interesting,
because you might not even think about it,
if you're not doing some type of numbers
around your hunting engagement,
but the conversations that I've had before,
are hey can we, uh,
if this happens next week, in this environment,
can we detect it?
And you start thinking, maybe?
I mean like probability could be like 20%, 30%.
That to me sounds a little bit like
when you want to forecast the weather and you say,
eh give this a probability of 50% it might rain next week.
That actually, it's a thing.
And it's actually was explained by,
I can't see that well Ryan McGeehan,
which goes under Magoo in Twitter.
And he talks about risk forecasting.
And this is great, because now I can start thinking
how I can provide some type of numbers with what I do.
And in this case,
he's showing how you can choose a scenario,
then you decompose a scenario,
then define threats.
Then you start pretty gathering supporting data
to reduce the bias.
And then you can forecast
and say hey this is the probability.
I start with a baseline of like 20%
and then, as a I go and execute the controls,
not necessarily hunting engagements,
but you can start feeling like, you know what,
yeah 10%, 15%, and at the end you can see for example
you can increase confidence again it's a 15% of difference
than when you start, where you're doing your forecasting.
Very interesting because
this would definitely start mapping to
concepts that you might be familiar with,
which is just basic threat modeling, for example.
You know, where you model the system,
you pick the system, you model and identify the threats,
understand how those threats works,
and then you address the threats,
validate the detection of this.
You measure again and you keep going that way.
This is very interesting because
now in this scenarios for example,
you start identifying the connections
between this crown jewel, for example?
How the data flows.
What are the external connections?
Trust boundaries around those things.
Very interesting, because when you start thinking about
what can I measure from this perspective,
there's a lot of data that gets generated
across this model for example.
Okay, so where do you fit hunt with this?
And I think that this is also an interesting question
because threat hunting we all see it as this
nice program where you have different steps,
identify a technique, hypothesis,
but at the end of the day,
if you don't understand
like exactly how you can map that to a business use case
or to how your systems actually work,
then to me it's not actually that valuable, okay?
So when you start now going back
to the other frameworks that I was talking about,
like threat modeling or risk forecasting,
there are some similarities going on in here, for example.
Some of you might agree, some of you might disagree that
for example threat hunting to me
could be also part of when you try to mitigate risk,
when you wanna reduce the risk of the attacker
achieving an objective.
Threat hunting could be part of that,
but at the same time you could start identifying
that some steps might be similar,
and how you're can start then mapping them
to how your organization works,
your systems and your crown jewels.
In this case, if you model what happens
around your most important systems,
you might probably then start identifying,
for example MITRE ATT&CK techniques
that would fit into exactly what the,
let me go back in here,
exactly where the data flows goes.
I focused on some,
for example lateral movement between systems
that might get to this crown jewel that you have.
You give it a meaning.
It's not anymore this isolated process
that you want to just hunt for the weekend.
You actually have some type of connection
across what you're doing in your organization most likely.
So what can we measure from then hunt,
or in this case I like when Devon actually crossed hunt,
and say it's just basically detection.
Then when you start understanding that
if you modeling your systems
and you're hunting across that model,
then I wanna understand what happened
across my trust boundaries, how my data flows.
Then I started then staying,
you know what, to me it's very important to know
what percentage of data I have for detection
or for any type of preventative measure in my model.
What tools are helping me the most?
What data's helping me the most across all my data sets,
because we also know that even though we say that
it's cheaper now to store data,
when we start going with solutions that
charge a lotta money for data,
you have to understand what data is,
what is the most value in your organization.
How much you can cover with that data
and things like that.
And at the end you're gonna start asking,
I'm sorry, answering questions,
are how are we reducing the probability
of an attacker to achieve an objective.
You can provide data,
numbers and metrics to answer those questions.
- So this kinda all brings us to ATT&CK.
You know Roberto alluded to it before.
You know, Enterprise ATT&CK is really
the practitioners knowledge base.
I'm gonna try and trademark that.
But I think MITRE has got a pretty good
description for what it is.
My description is really the most complete source
of knowledge about adversary tactics
loosely categorized into 11 different types
of adversary objectives with groups, software.
And again if you haven't expressed your love for ATT&CK
and you do love ATT&CK, the MITRE ATT&CK team
is here in the back row, second from the rear.
We're big fans and there's a lot to like.
And particularly the fact that
all of these techniques are cross-referenced.
They all contain meta-data that useful to analysts,
who don't necessarily understand
what these techniques look like in the wild.
And even better, they contain reference material,
so links to published documentation,
whether that's reports, blog post, GitHub repos.
It is a tremendous asset to teams
especially if you have a knowledge curation problem.
This is a place where knowledge is curated.
So this is just some statistics
as of the most recent update to ATT&CK's content.
Again, we broke it down by OS
and we'll show some further statistics
from our analysis of ATT&CK
that we think you could weaponize.
- [Roberto] So, how can you measure against ATT&CK?
I think that, also, right,
cause we see ATT&CK going this whole conference,
which is great, right,
so how can you start pretty much getting your strategy
of what you have in your environment
and just mapping it back to ATT&CK?
I think it's important that you understand
what ATT&CK actually gives to you,
besides just the technique name,
besides just the tactic name, and the ID,
there is actually some data
that you can start using to probably mapping it to
what I was referring earlier as your threat model,
for example, where you have in this case
what are the permissions an attacker needs
in order to accomplish
in a specific technique for example?
And you can start mapping that to your,
for example, trust boundaries.
The users that have access to that specific system,
like, how does that look like from an attack perspective?
What are the techniques
that probably you're going to start using
from a user being an admin as well, for example?
Later, what is the permissions the attacker will obtain,
in this case for example, SYSTEM.
What are the consequences of this happening?
And that's when you start mapping what you're doing, okay?
If an adversary becomes SYSTEM here or an Admin here,
this is what we can do next,
and, you know, things like that.
The other one that I love the most, of course,
is the data sources there are recommended for the detection
and the validation for the detection of each technique,
for example.
Very, very important
and we had a presentation yesterday by Hacker Here Again,
uh, Michael, right?
And then he was talking about how you are going to start
mapping data sources to techniques,
and something that I'm also doing,
which we're gonna probably hopefully
be collaborating on that piece
and the one that I like also the most is
something that I didn't know,
it actually, I asked Blake directly, and I was like,
hey, what is this supports remote,
I see that in some techniques.
He said, oh that's actually for execution,
you know, techniques.
You know, yes or no,
which technique can be used for the remote event
or something,
you can executive something remotely.
Very interesting,
because when you start mapping that to your, let's say,
to your model into how your system operates
and how the data flows go from system to system,
then you can start mapping which techniques
will actually happen across two systems,
for example.
What can you execute remotely and things like that.
When you start understanding this,
then you can start then saying,
you know what from a data perspective,
process monitoring, for example,
covers 70% of the current 219 techniques in ATT&CK,
for example,
and if you go a little further and you say,
okay, now map techniques of data sources,
I can map into a tactic but all the way to a platform,
then you can say,
okay, you know what, based out of this 70%,
now of course Mac OS, LINUX, and Windows,
from a data perspective, Mac, it's kind of like,
it kind of overlaps, that's the word, sorry.
It kind of overlaps, that's why if you add,
for example, the numbers in there,
it's not going to give you 149,
but it actually tells you from an attack perspective,
from a Windows side, 60% is still,
process monitoring is going to help you across
all these techniques.
If you go a little bit further then,
and you start getting information from
an adversarial perspective,
and you can map it from adversary uses this software,
software is used for this technique,
and things like that,
then you can start getting some numbers such as,
based on all this information,
I can tell that 65% of the groups identified
in MITRE ATT&CK,
they like a shell, or in this case,
would be a reverse shell, using a standard protocol.
For example, application layer protocol,
such as DNS and stuff like that.
So, you can get information
and then you can pretty much map to
what you're trying to accomplish with
your hunting engagement.
And, in this specific scenario,
from an adversarial perspective,
Katie Nickels has done a great job in talking about
all those difference presentations,
MITRE has been pushing that a lot, which is great,
and those are two presentations that you can actually see
and you can get information such as
challenges you face when you try to, you know,
use an attack for cyber threat intelligence
and things like that, so pretty useful.
So, now that we understand how we can use ATT&CK
and start mapping it to a detection strategy,
then let's go back to the data sources.
In a data source piece,
it's when you start understanding that
they give you a lot of data,
and this is just for reference,
'cause you can go back to the slides
and you can see all the data sources
that MITRE maps their techniques.
You can then just pick, let's say these two.
Process command-line parameters and process monitoring.
To me, that pretty much maps to a process.
What a process is, right?
And I define what I need from that process.
When you start doing this exercise,
you're technically modeling your data,
because you're finding this structure of your data,
and you're finding relationships
across what you're identifying with ATT&CK.
MITRE did a great job also by creating,
starting a model, like a couple years ago,
with the first versions of Sysmon, and there,
we're using the Cyber Observable Objects from STIX 2
to define what the pretty much the data objects are.
In this case, I'm defining my own two in my project OSSEM,
when I say an IP, it's supposed to have this information.
A process supposed to have this, and then a file as well.
When I starting doing this,
I find that there are relationships across these objects
because a process creates a file,
a process creates another process,
and a process can connect to an IP.
When I started doing that, then I understand,
I start building relationships,
source destination, my relationship,
like a basic probably graphing,
but you're defining the model,
so in this case, if we go back to process use of network,
then I can tell you that's basically
a process connecting to an IP to a host name to a URL.
Then I can say, you know what,
the information that I need from a process,
the information I need from an IP,
will give me that information,
so I can start saying Sysmon can give me process,
at the same Windows Security event logs can give me
a process information,
it's on the, that is being done already.
But, at the same time, you have to understand that
when you start looking into the data that you need,
versus what you have,
you're gonna start measuring that you can tell that
4688 is missing some fields that might be valuable to you
that Sysmon provides.
Now, this is not the only thing.
If you look into this specific relationship,
we're just talking about it from a process perspective.
Sysmon event ID one, and then 4688
but our relationship was a process connecting to an IP,
so for that,
you will pretty much need Sysmon event ID three,
for example.
So, that changes how your data connects to
what MITRE pretty much asks you to, or not ask you,
but request, no, recommend to use as a data set
for the detection of that technique.
So, that's when you start saying
I shouldn't just be mapping out process,
data set that MITRE recommends to Sysmon Event ID One,
because then if you require process use of network,
you might not get it with 4688 by itself.
That's when you start pretty much
putting numbers into what you have
in order to go and start hunting.
In this case, I was able to create just a basic model
also from what Sysmon provides
so depending on the events,
a process creates a process creates a file,
creates a registry, it renames a registry,
it updates a registry,
you have a lot of information that they provide to you.
That's what you should be doing, modeling your data sources,
so you can tell what data sources will help you
for specific techniques, and not just say,
process this what I have.
No, if a process talks to an IP,
I might need a different data source.
So, in this case,
what we're trying to accomplish is
that we don't just wanna do a mapping of a tactic
straight to Sysmon,
we want to start creating a connection that will
make sense where metrics can be applied.
I can just start measuring exactly what I need for Sysmon,
what I have, how it helps me to create also
analytics based on the data model,
that to me is more valuable than just saying,
I bought this tool and detects power shell.
No, right?
So that gives me extra context that would allow me to
be a little bit more efficient.
We come back to efficiency.
I don't want to just detect something.
I want to be efficient at detecting that specific technique.
And, of course, remember,
as Matt Graver and also Lee Christenson in Blackhat
they were showing how also
an attacker can influence the data,
and in this case, in this specific event,
these are the data fields that
it's high attacker influence rating.
So, when you understand that as well,
then you can start weighting
when you build your signatures for example,
if you have to build a signature at the end
because your hunting engagement might not be
just needed to happen every day,
you have to be mindful that you gotta understand
what an attacker can also influence as well,
and that might affect your metrics as well.
- [Devon] So this kind of brings us to the question of,
what can I measure?
Because we talk about measuring techniques,
do we finally have what we need?
And, you know, data quality is an important point.
I mean, we can express availability of evidence as a ratio,
we can attempt to assess data just by eyeballing it,
but there actually are ways for us to measure
the quality of our data.
This is a quote that basically just gives
an all purpose description of what data quality is.
The thing to take away is,
it's only of high quality if it serves the purpose
you need it to work for,
and the DOD, fortunately,
has a standard that gives us six attributes of data quality,
and each one is represented as a ratio or a percentage,
which, again,
fits right into our plan of being able to quantify
every aspect of this.
We're gonna focus on three of these aspects
that we think are most relevant to our purposes,
just be aware those other attributes do still exist
and can still be measured.
- And the reason why also decided to pick these three,
is because I heard this a lot,
why do I have to focus on doing some like
data governance and management.
I'm a threat hunter,
what am I supposed to be doing that stuff?
Well, you as a hunter, based on my experience as well,
you can influence these areas of data quality.
You can influence completeness, consistency, and timeliness
in your engagements.
And I think that's very important to understand,
so, also before that,
it's very important to do this,
because it's not just as let's collect more data,
I think I need this,
let's collect this and use it.
There should be a process behind that,
and data quality accomplishes that,
and just as data scientists,
they spend the longest time making sure
that their data's consistent, their data's complete,
that's a similar process that you should be doing.
So, you as a hunter should be also focusing on that.
'Cause at the end,
you're gonna come up with the best analytic
that you could use based on the data you have,
and you have to create the process
that every data set you ingest something new
has to go through the same process.
Is this consistent, and is this complete,
or do I have enough of that data?
From a completeness perspective,
you know, we went over this in Besides Charm
and this is basically is just saying,
if I have Sysmon event ID One and Windows Security 4688
sure, am I collecting everything?
Well make sure that you are enabling Process Command Line.
So, understand what you really need from a data perspective.
Also, completeness can translate into coverage as well,
from a hunting engagement, measure
What is the data available for you?
Let's say, your scope touches a hundred computers,
and at the end of the day you say, you know what,
sorry, we couldn't reach, you know, thirty computers.
Measure that.
Because, that would actually drive the change
to enabling probably logging,
enabling controls that would give you that data next.
Consistency, very interesting, as you can see,
like Sysmon calls the process Whole Path Image
and then 4688 gives you New Process as the data field name.
If I run a query just saying image equal something,
I might not hit those devices that
probably don't have Sysmon but that have, you know,
Windows Security event logs.
Very, very important, as it was said before,
not just because you run a query on your console
and nothing comes back,
doesn't mean that something happened.
So, if you're not aware of
the consistency of your data sets,
you're gonna have a hard time proving that that's something,
probably something needs to happen.
- In my old life, I had a customer
who really struggled with this,
spending six figures on data normalization
just so that they could run one query
and not multiple queries
'cause of all these different field names.
So, it does matter.
It does, there are dollars attached to those things.
- So, that's actually,
I'm actually trying to open a company in doing that
- [Devon] Exactly
- So, in the timeliness perspective,
it goes in my opinion both ways.
It could be the time that it takes data to pretty much get
to your database all the way from the ETL procedures,
Extraction, transformation, and loads of the data.
That's important,
because that might define reality in your environment.
I was in an environment that was taking minutes,
up to an hour to actually show
that that event happened in an environment.
That's very interesting,
because when you do some type of simulation,
you might say, uh, that didn't happen,
I might not have detection, or the data sources.
No, it's just saying so much time to get to your stack,
to your L stack or anything, and then at the end of course,
the data retention, because, believe me,
you might not find a pattern in seven days,
but if you extend it to a month,
then you might have a better understanding
of your environment.
I've done hunts in seven days, they are the worst,
because, unless you export the data out of every week
and then kind of like put it together, it's really hard,
because by the time you get to the next week
and the third week, the data that you wanted to test later,
then it's gone and then you're testing new data
and your whole pattern changes.
It's really hard, so measure that.
Every engagement, how much data to we have,
and believe me, that drives also changes to your,
you know, to your resources that you have.
- [Devon] So, there's lots of things you could measure,
you could talk about,
like hunt metrics as part of your organization,
we provide some examples right here
of things that might be relevant.
But, we thought we would give you a couple of alternatives
to this that might be a little bit more effective,
and I think, next slide Roberto.
- [Roberto] Yup.
- [Devon] And I think that this is a graphic
that we've shown before.
This is how some folks attempt to represent coverage,
and I mean,
how do you derive any meaning from something like this?
- [Roberto] Yeah, like, I believe that, for example,
I started doing something similar before as well,
and this is what I like to keep doing research,
'cause also improves the way how I try to look at things,
and I learn from other people,
and I think that we all should be doing that.
In this case, even though now we have a better understanding
that a color now might be tied
to percentages of how much data I have
for that detection of that technique and things like that,
you might have that understanding now, after this talk,
but still, hard to visualize.
You might have a legend with like ten colors,
and you would be going back to your legend and saying,
wait, red meant this, purple meant, wait, green,
no, green to me actually means something better than--
- [Devon] Green with black text, green with white text.
- [Roberto] Yeah, (laughs)
so, to me,
you have to start understanding that your visualizations,
you gotta tell a story with your visualizations,
and, in this case, for example,
I'm a fan of going now from the lightest color
to the darkest color because it tells you something.
This one tells me a lot of things,
but I don't know what they are.
This tells me at least that
there is some type of progress might be happening,
something that probably, a weakness,
to something more like stronger and things like that,
so it gives me something to think about,
and then you apply of course now what they mean
and things like that, but this should tell you something.
Now, what I like about using a heat map, also,
was that after talking about data against attack,
if somebody now asks me,
what is my detection for Powershell,
if I go to the data sources that I'm actually collecting,
to validate the detection of a technique
that uses a variant, that uses Powershell, then to me,
it's easy to say, hey, all these techniques in here,
depending on the variant that you're using,
I have data sources that map to Powershell,
so my detection about Powershell
is across this fifty techniques, or a hundred techniques.
Then saying, Powershell is just one square in the framework.
So, once again,
the heat map to you at the beginning might not be as useful,
'cause you start identifying those things,
just say wait, but Powershell is at one,
so I don't like the heat map anymore,
I don't like the idea of having this whole thing.
It depends how you use the heat map,
and depends how you are approaching
the detection of techniques.
In this case, if I look at it from a data perspective,
then to me it makes more sense.
I can tell you what techniques require specific data sources
that might map to something like Powershell, for example.
And then, I would love to do also,
hunts depending on the data.
I like to do that because it touches several techniques,
it doesn't just touch one thing.
So, I think that keep that in mind
when you start applying this type of scenarios.
- And what you're seeing here is ATT&CK Navigator,
another wonderful thing
that MITRE has released to the community,
and it's just the proportion of data sources covered,
versus those that exist as recommended data sources.
So, it's a simple ratio, easy to get this map.
- [Roberto] And when you start doing these heat maps also,
we recommend adding, um,
you were saying that End Game does that as well,
which is a, starts dividing something
like coverage from a data perspective,
from an analytics perspective.
You can start showing all these different scenarios,
and then have a better understanding
of what you're trying to accomplish with each heat map.
In this case, we have data sources,
in the other case we have analytics,
which is something that Devon then will be talking about.
- [Devon] Yeah, so we're gonna jump right into this.
This is not gonna be a big data science
or machine learning talk,
but those principles are definitely gonna play a role here,
if we can advance,
and we're gonna go right past this slide.
So, we're gonna start with statistics.
In our last talk, we showed this slide from Chris Gerritz.
It is a great slide that illustrates
kind of the trade offs between threat hunting
and traditional like alert based products,
and basically the difference is,
hunt products need to be resistant to things
like false negatives, and false positives,
but you don't want that from a platform
that's supposed to alert you,
and where this usually comes into conflict is
if you use a product that does both things,
so a multi purpose device.
So, what are we measuring now?
Well, when alerts fire, when we run hunts,
we collect meta data about true positives,
which is just malicious things we correctly call bad,
and false positives,
which are benign things we incorrectly call bad.
And, fortunately,
statistics gives us a tool for this called precision,
which is just a simple division.
All the stuff you label in the denominator,
the true positive count in the numerator,
and this winds up giving you a confidence score,
which is basically how much you tolerate false positives,
and here is just an example.
100 events, 74 true positives, a .74 precision,
so 74% tolerant of true and false positives.
But, that's not a good measurement
of how well the analytic works.
So, for that,
statistics gives us a different measurement called recall,
and for this, we need false negatives.
These are the things that are bad
but which we don't characterize as bad.
These are the things we miss.
So, when leadership asks you, what have you missed?
These are gonna be the things that you wanna count up.
And, again, you simply divide your true positives
by all the stuff that is actually bad,
whether you found it or not.
In this example, does give you that metadata.
But you have to be wary of going too far one way or another.
If you happen to have perfect recall,
it basically means that everything that you find,
basically is bad, but all the other stuff is bad too.
You have so many false positives
that it becomes a meaningless measurement.
And, with perfect recall,
you wind up with a similar challenge.
So, going too far one way or another does not help you,
and you do have to find a middle ground.
So, there was a step I skipped,
just for those folks that might be concerned,
how do you determine false negatives in these contexts?
And it's really the place where adversary simulation
plays a role.
Adversary simulation gives you counts
of the techniques that you want to find,
so whether you're using something like Red Team Automation,
whether you're using Ubers Meta, MITRE's Caldera,
any of the other frame works that you can package,
you can get those numbers.
But, that doesn't mean you can't use a red team
or a third party service provider,
because they will give you reports.
So, harvest that data, count those numbers,
and suddenly you have the ability to measure recall,
which is a much more meaningful measurement
of how good you are at detecting a thing.
It also has a strong relationship between data quality,
because if you don't have the sources of evidence,
if you can't run analytics against them,
if it's in a different time frame,
suddenly this whole thing falls apart.
All of your numbers go sideways on you.
- [Roberto] And the reason why we're approaching it
this way, also, is
and keeping it separate from the data quality measurements
and this and that from a hunting perspective,
is that there might be 100, 10, or 5 variations
of a technique, so you cannot just say,
if I have five analytics, I'm great,
'cause you cannot measure it against something
that might be infinite or it's just unknown.
It's something you cannot see,
so that's why we like to keep it separate
and approach a secured analytics development
by applying the concept of a true positive
or if a false positive, false negatives
and things like that, so that's actually--
- [Devon] And this works whether you're talking about rules,
signatures, human processes, or threat hunting.
Those are all detections,
and they can all be measured in the same way
using adversary simulation as one input,
again noting that adversary simulation might be
your red team or other human beings.
And, getting closer to precision, getting closer to recall,
also lets you figure out how good you are
at assessing badness.
And so, data science gives us this thing called an F1 score.
You can see the formula kinda up here,
and using the examples from before,
those analytics basically gave us an F1 score of .71
Now, this isn't an arithmetic average,
it's a harmonic average, which makes it a little bit easier
to express averages of rates,
which is what we are talking about,
rates of true/false positives, etc.
and we can even map these to visualizations
that help us understand what that looks like.
So, here's a notion of what F1 scores look like in a table.
This .71 basically means that we're finding about 60%
of all the bad stuff,
which is the true positives and the false negatives.
We're just calculating that ratio.
But, if we advance, you'll see that we might have
an organizational need for a different number.
Slide ahead.
So, what if you're organization says,
everything we do must be 80% successful?
Well, we could just figure out our threshold is a .4
and work backwards.
We can figure out how tolerant we are,
and then use available data and human processes
to tweak these approaches, whether it's a threat hunt,
an analytic, a signature, we control this.
And this is a data science based approach,
not a machine learning based approach,
we do have to know what we miss
in order to create these thresholds.
However, from there you could go on it
and you could look at finding ideal thresholds,
and this is a graph that basically shows
what's called a Receiver Operator Characteristic curve
or a ROC curve.
This is a tool that a lot of data scientists use
to express how tolerant they are of these thresholds,
basically the area underneath is how well you're doing,
and you can measure yourself against this,
even plot out hypothetical curves,
to see if maybe those give you better
true and false positive averages.
And all of this can be done with common scripting
and just a little bit of visibility into your data.
So, that's a lot - [Roberto] That's a lot
- That's a lot of stuff, we could probably keep going
with more aspects of this topic,
I think measuring how well we are at finding evil
is very important to the entire industry
- Something I wanted to add in there,
just a closing thought, is that if you thought
that threat hunting was just going to a console
and run a query and then hunt like that for the month,
that's not what threat hunting would be.
As you can tell,
there is a lot of stuff that happens besides that.
You know, it's interesting when I talk to people that does,
you know, data science, I'm like,
where do you spend your time the most?
I dunno, man, I do a lot of this, like transformation,
make sure that everything's good, the quality of my data,
and then when I apply my model, yeah,
it might not be as the same time as I spend doing the rest.
I see that as a similar thing happening here
because we're trying to develop, right, all these analytics,
trying to do all these crazy stuff that is really cool,
but threat hunting, in my opinion,
should be also looked at, as there are procedures,
things that should happen before you do that,
and also after you do your hunting engagement,
that involve the whole process of threat hunting.
And I think that's what I want you to take away
from this presentation, is to understand
what needs to happen when you want to do threat hunting,
right?
Going to the console, go crazy,
do those analytics and stuff, it's beyond that,
as you can tell.
You gotta focus a lot on what you have, what you need,
how you can get better, how can you be efficient,
how can you be organized, right?
You know, from an attack perspective, for example,
MITRE ATT&CK frame work,
and then how you can start to measure your analytics,
in a way that also would impact your organization.
- So we've got three last things, if we advance,
we do want to thank
- Oh - Oh, so we've got some links.
We also have an appendix,
so it's stuff that we didn't include in the talk
but we wanted you to have in the slides,
but we also wanted to just thank some folks, you know,
basically everybody who contributed,
the Beyond Science Data blog, MITRE of course,
and all of the defenders who have given us feedback
since Besides Charm
- Thank you, thank you very much
(Applause) (tinkling piano music)
(dramatic music)