(electronic music)
(applause)
- Thank you for being here, thank you for taking time.
I understand exactly how much time, like, our time is worth.
So we're gonna talk about that today.
We're gonna talk about shortening some cycles,
gonna talk about finding evil faster.
We're gonna have a whole discussion around normal.
I love this motto.
It's four simple words, but how hard is it
to actually know normal?
How hard is it to actually find evil?
A lot of us in this room, understand
how hard that equation actually becomes.
We spend a whole lot of time chasing normal,
lookin' for evil, only to, in the end,
find something that looks exactly like an APP, but it's not.
And we're gonna talk about a bunch of those examples today,
and how we can start, as a community,
put this intelligence together
in a way that we can all use.
Sound good?
Awesome.
Look, the journey at Carbon Black
really started with one premise,
centrally collect the right data,
to rewind the tape on attacks.
But as we started this journey, we started to realize
that collecting all of that date doesn't necessarily mean
you can determine normal faster.
In fact, when you start to collect all of this data,
the more you actually start to look at it,
the more abnormal everything becomes.
I'm gonna talk about some of those examples today,
and I'll give you, one of our brief hypothesis
that we started with that was quickly
myth-busted along the journey.
We thought when we started, that binaries
on an endpoint would stabilize over time.
Actually not true, they continue to linerally increase.
Like, why does a user continue to need executables
when they can do their job on a thing?
That still continues to happen.
We wasted a lot of time.
During our engagements, we found a lot of normal
at the tail end of our chain.
And it turns out none of it was evil,
but we wasted a whole lot of time.
So what we're gonna talk about today,
is how we can apply those lessons
back to what you're doing in your program,
and hopefully inspire this community to get together
to help fix some of this stuff.
'Cause I'm tired of wasting time
chasing good, known applications.
Anybody else tired of chasing good, known applications?
At CCM it was a pain, so we'll talk a little bit about that.
But now we're gonna talk about evil.
When we start, we generally ask ourselves, is this abnormal?
I.e. a thing happened.
Maybe it was an alert, maybe we started an instant
response engagement, maybe we're out hunting.
But we start with that hypothesis, is this abnormal?
Then we start to say to ourselves, is this normal?
The intersection of both becomes a very interesting place.
And we start to pay attention to these things,
we start to realize that abnormal and normal overlap
all the time, and the buckets we use to define those,
are probably a little incorrect.
This year, some great work has been done
to help actually find evil, and that work,
thank you MITRE, by the way.
Carbon Black is a huge fan of what happened
with MITRE ATT&CK.
We actually have a taxonomy, that's understandable,
updatable, and relatable back to the kill chain.
So now we understand the TTPs
mapped out to the entire kill chain.
This helps us start to focus on evil faster.
But what happens when you start to overlay this data,
is you get a massive amount of false positives.
We can tell you this, because we created a watch list
which maps back to ATT&CK MITRE,
so if you're a Carbon Black customer,
that'll be available in, like, a week.
In GA we have a beta program right now,
there's a number of false positives go through the roof.
And this is the problem, while it allows us
to focus on attacker behavior,
we are still left with this giant amount of noise.
And it becomes, what I like to refer to it as
finding the needle in the needle stack.
So when we take all of that away,
we're left with all this noise, still.
Is it a print queue, or is it the Russians?
I don't know, we've gotta figure that out faster, right?
Can't find evil, unless we can get through
the cycle of knowing normal, faster.
Abnormal and normal are intertwined,
they're in a relationship with each other.
It depends on how you analyze data,
and I'll give you an example.
Is notepad.exe at 10am,
like, if you observe the behavior
of that binary at 10am, it may look one way.
But if you do it at 10am and 10pm, it's gonna change.
If you do it every 10 seconds, it's gonna change,
if you do it every one second, it's gonna change.
Abnormal is based on your observations.
So now we're gonna talk a little bit about
the levels of abnormal that are out there.
They exist all over the place.
When you start looking at endpoint behavior,
you start to quickly realize that
there is process abnormalities.
There are system abnormalities,
memory based abnormalities, and certainly users
have abnormalities all the time, right.
Some of you guys can probably relate to this.
What does this all mean?
It really means that you have to understand
each one of these to actually understand normal.
And if you can get to know that normal,
all of these get removed, then the focus
becomes just finding evil.
If we can spend our time just finding evil,
if we can get through the cycle of knowing normal faster,
I think we have a chance of getting down
to a reasonable time to actually defend our systems.
Like, if we don't, and we continue to do
what we've done as an industry,
we are gonna continue to have a problem.
I mean look, Gardner says it's 180 days meantime to detect.
I think Verizon last year, said it was 211.
When I started in 1998, the number was 369 days.
So, at best, in 20 years, we've gotten six months better.
It's not a good statistic, like,
that's just not a good look, as an industry, right.
So now, let's talk about how we bucket normal and evil.
Today, we generally use two buckets.
We say something is normal, therefore it's benign.
Or something is abnormal, therefore it's evil.
But as an industry, how do we figure that out?
We start to use frequency analysis, right.
'Cause frequency analysis tells us,
if it happens frequently, it's probably good.
If it happens infrequently, it's probably bad,
because that's your outlier.
We think that's bad, we think we should actually
focus on infrequent and malicious.
Not frequency being the indicator.
And we think the table's gonna look
a little bit more like this.
Normal things act evil all the time.
Look, if you watch any of the cool talks
on evading antivirus, what you'll quickly see
is none of the attackers are actually
dropping binaries anymore, at least the good ones.
They're all using good, known processes.
Whether that's PowerShell, WMI, all of these things.
So, normal stuff acts evil.
Why, because I approved it.
I approved that application.
When somebody submitted that, this application
come up my network, and I was the CCO, I approve that.
So guess what, it's a normal evil.
In a Dungeons & Dragons sense,
it looks a little bit more like this.
And apps operate like that all the time.
There's chaotic good apps, like CCM,
that just stray things all over the place
and network connections like crazy, right.
You have abnormal benign things, like an increase in x,
where something changes in your environment,
but it's completely benign and you chased it down.
It's a lot of time wasted.
So look, here's what we're gonna do.
I've never seen a Keynote do this,
we're gonna play a game today.
Everybody ready to play a game?
- [Audience Member] Yeah.
- It should be pretty awesome.
The game's gonna be called, evil or not evil.
I'm gonna give you some examples of things that happen,
give you the front end of the chain.
I want you to think about it,
you don't have to shout answers, we're gonna think about it,
we're gonna walk these in our head.
We'll give you an example of where we won,
and were actually right, and where we failed in our hunt.
Alright, evil or not evil.
You get an alert, or maybe a watch list hit that says
your SAM file was accessed and scraped.
Is that evil, or not evil?
Well if you go to MITRE ATT&CK, oh look pwdump does that,
mimikatz does that, so it must be evil, right.
I like that answer.
(muffled talking)
Yeah, what's the context of that, right?
What's the context, like, what are
the parent things that kicked that off?
In this particular case, we were assuming evil.
That shouldn't happen right?
But it turns out, we were wrong.
And it was completely normal.
You know what does this all the time?
Adobe updater.
You know what Adobe updater does?
It like, watches all of these processes
so it can stay up and running.
And from Adobe's perspective,
that probably makes perfect sense.
I mean, after all, they've sold billions of dollars
of software, and their software stays up all the time.
From our perspective, we're like,
hey Adobe, maybe you don't need to do that.
Maybe you can create two processes, an A and a B,
with shared view texts and just watch each other,
and if it fails, get out of what the adversaries do, right.
So now in this particular case,
the red balloon is not evil.
A child just lost his red balloon.
But that wasted a ton of time.
Alright, here we go, next up.
Notepad.exe makes a netcom, good or bad?
Supposed to happen, 'cause I was told for years
Notepad never makes network connections unless it's evil.
Oh, but wait, if I use file shares, and I'm a user,
and I open a text file from a file share, well guess what.
Notepad makin' netcoms all over the place.
Context becomes everything to determine
whether it's evil or not.
In this particular case, when you start to trace it,
you start to see that the parent process
is actually cpuchecker.exe.
And our tell team discovered
a brand new piece of monera mining software, a used Notepad
to connect back, and start the mining process.
So is it bad that notepad.exe makes connections?
No, that's gonna happen everyday.
I'm guess unless you get rid of the Notepad
in that environment, right?
But injection's happening all the time,
and the context becomes everything.
If you don't have the visibility and you can
put that picture together, you're just gonna miss it
and assume things, and honestly waste a lot of time.
I'm gonna reiterate that a few times.
What's my part in here.
Oh, this is a good one.
MSBuild, I hate MSBuild right now.
MSBuild got put on every Windows 10 box.
And MSBuild, and I'll tell you right now,
if you work in a company that doesn't
have a lot of engineers, and you see MSBuild activity,
like, immediately look into that.
So, we'll just skip the part where we play the game,
and say, if you don't have engineers building things
you shouldn't have MSBuild running.
What's the problem with MSBuild?
Is it evil, is it not evil?
Like, how often should that happen,
when the parent process isn't Visual Studio?
These are questions we all have to answer
when we're not hunting, right,
when we're doing IR engagements.
So, what's that answer?
The answer becomes, context becomes everything.
MSBuild will make all kinds of network connections,
because most of the time, developers are
compiling things on the network, right.
They're so scored repositories are getting a lot of network,
so guess what, I might as well go, netcon, netcon,
it's gonna look like lateral movement all over the place.
So, is it evil?
In this particular case, there's a whole lot of activity,
but it was abnormal and benign.
We throw a whole bunch of red balloons
when it turns out it wasn't, however,
Joe Casazza from Red Canary is presenting at lunch,
he's gonna deep dive into MSBuild and why it's bad,
but just to tee up his talk a little bit,
I can make anything run, I can make mimikatz run
and it's gonna launch, that's MSBuild, right.
But that activity that it does, the behaviors
that are expressed by MSBuild post that happening,
we should have visibility to, and be able to stop.
That's how you're gonna determine
whether it's actually evil or not.
Alright, one of my favorite ones.
Powershell.exe starts making network connections
that are boxes.
Evil, not evil?
You gotta make a call, come on, we got like,
two minutes, when we triage this stuff, right.
Alright, he's in the camp of evil.
I'm in the camp of evil too, 'cause PowerShell's
pretty awesome.
I mean look, we've seen it rise ever since
PowerShell empire came out, right.
There's droppers every where.
It's pretty awesome the stuff you can do with PowerShell.
My system administrators, however, would tell me
don't be on PowerShell.
Like, we can't do our job without PowerShell.
I can't do data center automation,
I can't do rebuilds of endpoints.
I can't do any of that stuff because
we've built our entire operations around PowerShell.
So good luck banning that, right.
How do I determine whether PowerShell doin' something,
making a network connection to a new box, is good or bad?
That's the life we live, those are
the questions we have to answer.
But if I had that, what I'm gonna refer to
as normal intelligence, and we treated it like we do
with adversarial threat intelligence, as a community,
like, we exchanged it, we talk about it,
we've got a place to go find this stuff together,
that's where we need to be.
Because this cycle becomes shorter then.
I certainly know what it was like
when I actually had a real job,
and this would happen, we had trouble.
Spreadsheets, I had text messages to people
who were way smarter than me.
You know, hey Ben, have you seen this before,
Mike, have you seen this before?
How many of us live in that world, where it's like,
is this, I gotta make the call,
is it actually evil, or is it good, right.
And we err on the side of not
interrupting the user all the time.
So we have to walk that razor line,
as responders and threat hunters all the time.
'Cause I can tell you what it's like to be
in an engineering company, when you developer
from actually being able to push out a binary
that sells millions of dollars worth of stuff,
it's gonna be a bad day for you in that seat.
So, in this particular case, it's abnormal, benign,
because uh oh, it was my system administrators
orchestrating things on endpoints, right.
You have to see the behaviors of something like PowerShell.
And I was gonna swap this out, and I'm sure a couple people
in my company would be upset that I didn't with WMI.
And I'm gonna briefly add one to this list, which is WMI,
and it's gonna fall in the same category.
If you look at WMI, and how it actually works,
SCCM is gonna spray that out.
So if you have SCCM in your environment,
you're gonna have to know that.
You're gonna have to know how WMI actually looks
to find it when it's being evil.
So, just know that the bad guys know that we're after
PowerShell, and they've already moved on to WMI.
So, expect a lot of lateral movement to be that.
Alright, I'm gonna go to the last one.
Dllhost.exe.
When is it okay for dllhost.exe to kick off
something like a command interpreter, like,
I'll just use cmd.exe (mumbles).
This one, I didn't come here to give answers.
'Cause, ultimately, no one has all the answers
when it comes to this game, but as a group,
we're really smart.
We can start to put this data together,
and start to talk to each other.
And vendors can stop competing
on things like false positive rates.
I think we're gonna get better on that.
So in this particular case, I don't know.
We've got a whole bunch more investigating to do.
But how much time did we waste until we actually get
an answer on whether or not this is normal behavior
for an approved application in my environment.
This is extremely hard.
We wanna help.
We wanna help all of you in this room,
because we've seen what happens
when a community gets together.
If you're a customer of Carbon Black,
you're part of the user exchange,
and this happens on a daily basis.
Defenders exchanging information with defenders,
and actually getting through these cycles faster.
As a community, we need that, it cannot be exclusive
to one organization, and one set of customers.
'Cause no vendor out there is gonna sell a hundred percent
of the market, it's just not gonna happen, right.
So, if we continue to hoard that information,
we continue to not give it away as a community,
we are doing a disservice to ourselves as a company.
Imagine a world without all of this noise,
and third party applications.
Imagine a world, where we could actually
help to educate major software vendors,
that their applications look exactly like an APT.
Imagine a world, and some of you may actually work
for a major software manufacturer.
You may actually be able to influence
this problem in your company.
But imagine a world, where you have to approve
a new application comin' in, and you can
look the vendor in the face, and say
do you trigger any of MITRE ATT&CK?
So, now I know, right at the beginning of it.
They don't actually know that what they're doing
is causing us pain and wasting time.
What they're doing, from their perspective,
makes perfect sense, 'cause their binaries work.
Look, Microsoft has sold a lot of binaries, right.
We have to help educate them.
We really do, like, that's a big part of our job.
They don't know that all this noise lands on us
and we've got to cut through it.
So we all have work to do as a community.
What we need to start doing, is linking
application software development
with threat hunting and incident response,
and feed that back over to them.
Here's what Carbon Black would like to do about all of this.
We would like to provide MITRE with good, known binaries
and false positives for every single TTP in MITRE ATT&CK.
So, imagine having a repository where you can go
and quickly rule out whether it's a false positive.
Where you can start to manage
normal intelligence in your environment.
We wanna help developers do the right thing,
stop yelling at them.
We need to get into their world,
and they need to get into our world.
We gotta get together, and get a little weird for a while.
We wanna reduce false positive rates
for everybody in the industry, not just ourselves.
Look, I'll stay on the stage and say
I've never worked for another vendor.
I came here because of Carbon Black's vision,
I came here, because of a mission,
but more importantly, I came here,
it's fundamental to the company, we are defenders.
Like, we do this on a daily basis, and we live with the pain
on a daily basis, so we understand it.
We want to save everybody time.
You know, Ben has talked about this,
JJ talked about this last year.
One of the most precious commodities we have, is time.
They're giving us more money, right.
We're at 98 billion dollar market cap now, in this period.
That has grown extremely, since 1998.
They're giving us more money.
They are giving us more people.
Now granted, everyone I've talked to says
six to nine months to actually get someone
in the seat, trained to be effective.
We need to start cutting that down,
'cause our army ain't gettin' big fast enough.
Look, I work with Universities all the time,
we're not graduating people fast enough, right.
So we have to do something different.
This idea that we can continue to make linear progress
and get .1% better on efficacy or detection
is just not the world we can live with.
Most of the people in this room
are quite aware of what's happening across the world.
Geopolitically, cyberworld-wise, all of those things.
We now have nation state tools makin' the common ransomware.
That's the world we live in, and we gotta change it.
We continue to do this thing, where we make linear progress,
but it's not gonna get better.
Ultimately, we're gonna be the ones stuck with it.
We're gonna be the ones at 3am triagin' it,
and trying to figure it out for our company,
whether or not we just lost millions out the front door.
So I'm gonna call this normal intelligence for now,
you can pick on me all you want.
We can come up with a different name.
I really don't care, as long as the effort gets done.
What I wanna challenge everyone with in this room,
vendors, competitors, defenders,
let's go fix this as a community.
We have seen the power of community at Carbon Black.
We want that for all of us, not just our customers.
So let's stop competing on this stuff, it's intelligence.
We have spent so much time on threat intelligence
and looking at adversarial behaviors,
that we have ignored our home-field advantage for too long.
For far too long, we've given up.
I've sat in the room with major CISOs who've said,
asset managements is never gonna happen.
Not if you give up.
Look, I'm a cynic too, I don't think you can
work in security and not be a cynic.
But if you give up from the start
and say something can never happen, it will never happen.
We in this room are this community.
Like, you know I spent time talking to Phil this morning,
and I asked him about the growth
of this particular event.
Like, have we seen as threat hunters,
and incident responders, more people
wanting to get in this game.
The answer is yes, it's one of the most
rapidly growing summits that SANS has.
That's awesome.
So guess what, we are the leaders.
Like, we are the community.
We need to remember that, especially
when it comes to vendors out there.
We can drive change.
So imagine the world without the noise.
How much easier would it be if we could
rule out a false positive, faster.
How much easier would it be to say,
oh, this is normal behavior for WMI net ccm.
And oh, by the way, I'm gonna go
share that out in the community.
So, Carbon Black is committed to
setting up a slack channel around this,
so that you can ask questions for normal.
Now, I couldn't commit every resource in the company
to manning that, but Mike Mascuzza, the CTO of the company
has guaranteed that we will man that
for an hour, one day a week.
What we're asking is for anybody else
that wants to participate, man another hour during the week.
It's just an hour, put a couple of your experts in there
so we can exchange normal, start talking about it,
rule the stuff out fast in the community.
'Cause the conversations are happening
in our UEX, and it's awesome.
But that's only gonna impact Carbon Black customers.
We want to impact the entire industry.
I didn't come into this industry, to like,
make incremental change, and I certainly
didn't come to a vendor like Carbon Black,
who has the ability to lead, and the willingness
to actually stand up and lead.
We could hoard this information.
We certainly could, and keep our false positive rates low.
We know that some of this information
is probably gonna help our competitors
like, we're fully aware of that.
We don't care, why?
'Cause it makes your life better,
and we have to have that commitment.
The leaders in this industry, have got to
have that commitment, and as a community,
we've got to stand up and demand that from our vendor.
So again, imagine your world without this noise.
How much faster can we find evil?
Oh, this all sounds great, Rick, sounds awesome.
But, but, but, we've heard things like this before.
Yeah, you probably have.
Hey, but wait, wouldn't this give adversaries
a roadmap for how to evade technology?
Yeah, but guess what, they're right now, how.
I mean look, some of us in the room
can't actually go and do it, but if you go look at things
like marble and grasshopper, and you know,
some of these vault seven and vault, they already know.
They already know.
I don't think we should be held up by but.
It's a terrible way to hold yourself up.
But, but, but, but, but.
Arguing about edge cases.
Can we do something about this problem.
I mean, for once as a community, agree this is a problem,
and actually do something about it, and get together.
I sit on a whole lot of committees with other
vendors and competitors in a space and it gets weird.
And let me tell you something, the adversary,
they don't care about our politics.
They don't care about who owns the market space,
or who's the best vendor, like
they don't care about any of that.
That is not a consideration to them, no.
But here we sit, as a community, arguing about edge cases.
Not getting together to actually,
drastically change this game, and we have got to.
I just wrote a blog, to kind of tee this up,
and I said, you know, we're hurdling
toward something as a community.
And I can't figure out if that thing
is disaster or safety.
Like, to be honest with you, some days I wake up
and I go, oh, this is the day, like, we're doomed.
I'm also really trying to stay optimistic
about the problem, because we are making a dent.
There are good things that are happening.
Our community has got to get together
and demand that we're actually gonna
be a community around this stuff.
Spent some time with Carl yesterday,
he said the same thing.
Yeah, we waste time and companies are paying for it.
Like, cool, $300, $400 an hour, awesome.
You just spent six hours, find an SCCM.
It's not good, right.
So we all have these internal sources of intelligence,
we have to treat it like external threat intel.
Yeah, I'm a poker player, and I've worked
a long time to actually go play
in the World Series if Poker.
What I can tell you about this,
is poker is very interestingly, a whole lot
like incident response and threat handling.
You have a constant changing set of statistics.
Well, you have a baseline set of statistics,
It changes per hand.
As an example, is that player aggressive?
Have you seen hands from him before?
Context in poker becomes everything, as well,
because per hand statistics change.
Aces aren't always the best hand to play, trust me.
Normal and abnormal gets really weird.
Is it normal for someone to just go all in
when they pay $10, 000 on the first hand
in the World Series of Poker?
Yeah, I can tell you that happens on a regular basis,
so it's not abnormal, but I have to
factor that in to how I play.
To actually have a chance of playing a hand and winning.
I think for us, we have to do the same, okay.
Look, we can't choose when the gaze of Sowron
comes upon our systems, we can't choose
when evil's gonna hit us.
But we can choose what to do once they're
inside our environments.
We can choose what to do to set ourselves up
to be successful when that does happen
and get through those cycles faster.
That's what we can do first.
I can't control an adversary when they're doing their thing.
Once they get on my network, I can probably control 'em.
But we have no control over them, what we can control
is who we are as a community, and how we learn
and it's gotta change.
Knowing normal is everything.
We spend far too long focused on finding evil,
we need to get through the cycle faster.
We need to actually act like a community.
We need massive leaps in progress,
not linear changes.
I wanted to say thank you to the entire Carbon Black team.
This was a true team effort.
I also wanted to thank SANS, it was a goal of mine
to Keynote a SANS event for, probably the last 15 years,
so thank you for the opportunity,
and thank each and every one of you
for being in the room, and for doing what you do.
It's extremely important, what we do.
And we don't get told that enough,
so as a defender, as someone who has wasted a lot of time
alongside all the rest of us, I would just like
to personally thank you.
Keep up the good fight, let's talk more,
let's get together as a community.
We are Carbon Black, and I am proud to (mumbles).
Thank you.
(applause)
(electronic music)