My name is Ben Johnson I'll get to my back in a second I think Rob kind of covered it I am on Pacific time
This is my first conference wearing glasses I have 47 slides and I got in at 1:00 a.m. So let's see how this goes
But I'm really excited to be here throughout. Hunting incident response summits the challenge is
What do I talk about we have a whole slew of great technical talks deep dives
vendors
The keynotes meant to be a little bit more high-level, which is good because then
I can just say stuff that's neither right nor wrong and you can figure it out later
So let's just dive in mmm and thanks to san putting up with my late
Late slide submission and and all that they can you know kick me later
so
agenda real simple
state of cyber I'll give you a spoiler it's not great and
Then maybe we can look a little bit into entrepreneurship
Are, there some lessons we can take from entrepreneurship and apply that to our efforts to hunt and then, we'll wrap up
my general approach to
Presentations is to throw as much at you as I can and just hope a couple things stick
you can take all the pictures you want I will happily give you all these slides so you don't have to take any pictures if
you don't want to
So as Rob mentioned
got my start in the Intel community and I say went to man Tech worked with Rob Phil
Henry I think I see mark you know a bunch other people intrusion operations division it was unbelievable
Stuff, I would have done for free
I think we all would have done for free and we got paid for it so so there we go
Ultimately ended up leaving government world did a couple years in finance actually went back
Helped found carbon black helped write the first version and then ended up
Jumping on planes for several years to talk to people like yourselves so up to 600 organization and my goal
excuse me was quite simply to absorb learn and then hopefully I can
Contribute something back to the discussion and then I get the start again so I moved to Southern California
Started obsidian we're about 35 people last year, we were about 5 so it's been kind of a wild year and being in Southern California
you'll spots of slides with beaches and stuff just so I can rub it in a little bit
but if anyone wants to come visit please come visit
Speaking about entrepreneurship I really love the startup journey so I'm on all other boards different stages different types of companies and I taught
Entrepreneurship at University of Chicago to the Masters in computer science program, so that's where some of this stuff comes from
So today's goal and I always throw this slide in there you'll see if you've seen me present before you'll see a few slides that
maybe you've seen multiple times
Reuse right why not but my goal for a presentation is always just to get you to think I just want you to contemplate
Hopefully the alerts are shut off the phones on airplane mode don't have meetings you just hear all in one room
absorb think contemplate and hopefully you walk away with something
Another reason my
Being up here is a little crazy is
I'm in the middle of a physical world Incident Response where I have to in my Chicago house that refused to leave and
I've been talking to all of these yes including SWAT team
over the last week I was in Chicago till last night
Mmm, they're all their stuff still there they finally left
What their stuff is still there and they have a lot of stuff
So yeah kinda like a cyber incident response right there's all these different
Groups not necessarily having the same information different incentives so I move fast so I move incredibly slow
But anybody snuff about that just just thought that was interesting
The state of cyber
You probably all sick of the headlines
Breach fatigue headline fatigue we in this room care that's why you're here
But to be honest I don't even up this slide
Because you don't really care what the headlines are the point is it's not pretty and
What's maybe a little bit sobering is getting better
Now we're becoming more and more connected more digital so just in pure
quantity there's going to be more, breaches but it's just exploded and
Even the cloud which I truly believe is an opportunity for us to be way more secure is
Leaky is causing more problems
This is also probably outdated by now but there's just a lot of leaks whether it's compromised, whether it's an open s3 bucket
it's too easy to to leak data and
A variety of adversaries and what would a talk be without a threat scape slide so you get minerals hacktivists
Nation-states
insiders you get financial crimes usually maybe making a statement
Espionage maybe election interference other aspects of nation states and insiders but the only reason I put this slide up here
Is the insiders
people forget about
Insiders and I'm not talking the person sitting next to you
They might be an insider sure but I'm talking about their access
because what are the first three after the
First thing they're after is access
then they
Try to focus on their objectives they need that maintained persistent access and a lot of times, that's through
employee access credential access blending in
So, I want us to keep that in mind and there's a challenges
so
to me there's no debate that there's a skills gap the reasons for the skills gap maybe people have different opinions HR has bad
Descriptions or the pool is too small, whatever it is there's not enough qualified butts and seats
Then you set up your tech stack your security stack and you
Have deploy and decay it gets worse over time you don't have enough people giving it the care and feeding
Sort of brushing its teeth all the time so environments change adversaries change
maybe your detection rules prevention rules your configs stay the same
You're going to get decay
then attackers bad guys, whomever they're having success
So they get encouraged to do more or their friends like any market it drives competition
they dive in and
Then finally huge data I don't even know if that's a real term but
It's so easy you click a button you copy terabytes of information to the cloud everything syncing all over the place
every copy of data is a liability
And we're copying it all over the place so
essentially
the summation of that is this lack of cyber self-esteem
now people in this room actually probably have the most
Cyber self-esteem you actually think you, can make a difference and I love that I think we can too but
When you travel the world you talk to lots of organizations whether it's at sea so?
having little faith in their team whether it's the team having little faith in the executives or the culture or whatever
It's tough, we need mr. Rogers to come out and give us an after-school special
So hunting rob
Was just talking about what's the definition of hunting to me it's pretty simple and I'm not saying this is the end-all be-all definition
You can detect all of the things
tools don't
So there's a gap there's a gap where we need human minds critical thought human time manual efforts
to me that's hunting finding the stuff that lives in that gap and
Ideally we all have the equivalent of the predator drone
fly it to the target
take these nests
But reality is never ideal
We might not even have a weapon we might not even have time to hunt and in fact that's a lot of this talk is
Just trying to do maybe a little bit even if you don't have much
So ken hunting be formulaic can we come up with a formula do we just you know add X headcount throw some tools in there
Get some buy-in and you're done
No, there's no formula
Sure maybe you can find the Rambo or the Wonder Woman or that sort of badass technical hunter
Cool those are not scalable to find they're probably mostly in this room
there's not many of them in the world
So entrepreneurship sort of set in the stage entrepreneurship who here thinks they're an entrepreneur
Come on I get I get a couple hands few come on all right we'll come back to that
So what's the formula for startups
Do you just have an idea throw some work at but maybe raise some money and you're done your profit
It's incredibly hard incredibly hard there's no formula
But maybe we can start thinking about how startups have a little bit more guidance
So Toyota in the 70s or 80s invented something called lean manufacturing and some people, even think it was in the 30s
Quite simply they said we're gonna reduce waste eliminate waste and what is waste waste is anything that doesn't directly contribute to
value for the customer and
there's lots of ways you might have waste
You might ship things and an inefficient route you might have workers on a factory floor taking too
Many, steps you might do too many quality control X it doesn't really matter for this purpose but
What matters is there's a lot of waste and we all have waste
we all do things that don't Altima add value at the end
and
so
thinking about that thinking about reducing waste and focus I just want to throw out there one of my favorite books and
Actually have a book section at the end because if you take nothing else away other than a book recommendation
I think it's still a successful day
Essentialism is about finding your greatest point of contribution
Basically what's the biggest ROI of your time and?
I've done lots of presentations on this kind of topic because we all have the same amount of time
86,000 and change seconds per day
How do we spend it?
How do you get the biggest ROI and we talk about throw our instant response how are you getting the biggest bang for your buck
and
these kinds of ideas
Led, to something called the Lean Startup methodology and a lot of you have probably heard some of this before maybe you know it way
better than me that's okay, but Lean Startup is about how can you get your product and service into the hands of
Customers faster and how do you reduce uncertainty know that you're building the right thing
Only like four people raised their hands when I asked where entrepreneurs were
Entrepreneurs everywhere it's a mindset
even if you work for a big company you can be an entrepreneur and
sometimes they close intrapreneurs
But it's just about identify
And trying to solve it trying to build something to solve it doesn't mean you have to go raise money or something like that
but a lot of it is you got to think big you got to think about solving something worthwhile
but then you start small you take that first step and
Then ideally move fast scale fast and I love quotes you'll see quotes throughout mouth we'll talk
The day before something's a breakthrough it's a crazy idea
and
So hopefully in your heads and we'll start to tie this a little bit more directly to hunting but hopefully in your heads you can
See, where some of this is going in terms of things like hunting incident response security in general
Another topic in another principle in
Lean Startup methodology is
validated learning
Basically how fast can you learn there's another question you might ask which is when you raised five million bucks 10 million
Bucks, what did you learn from spending that money
It's not just about product output, or whatever it's about learning as a company as a team as a business, whatever and same here
how quickly can you learn
create hypothesis we're an experiment analyze the results repeat to me that sounds a lot like hunting and
All I do is basically work family and then read books
Or watch TED Talks and stuff
Reading a book right now called rocket men amazing book haven't finished it yet
But they had this quote they said are you learning in gulps or sips it's a very simple way to remember it
Are you learning in gulps or?
Sips in your work are you learning in gulps or sips during instant response and you're learning in gulps or sips
So a big part of Lean Startup methodology is how fast can you learn is this the right thing
the right feature the right product the right color the right price and
So what Lean Startup methodology did was then create a cycle because everyone likes cycles and
it's really simple to remember build measure learn
Build something go deploy it run it it whatever but you have to be able to measure it
Then learn from it if you can't measure it you can't optimize it
So you go through this cycle and this is very common in startup and this is really what agile software development and agile
Methodology tries to follow, which is very iterative can I do something every sprint or every every day even that improves things and
if we're talking about loops doodle loops
You probably have uh de loop fatigue if you come to a lot of these
But the whole point is in the US Air Force they came out with
Something called odo loop, which is in military dogfighting whoever can observe?
Orient, decide and act the fastest as a pilot
Welwyn
Can you observe your environment orient yourself to the environment
Decide and then act
Go through that loop and I have to say decide is tough sometimes you've got to be comfortable making decisions just as like a tip
right
The thing I still have to do all the time is make decisions and sometimes very quickly without information
MVP Minimum Viable Product
part of
entrepreneurship part of Lean Startup
This is what they're trying to get across what we're trying to talk about
Which is what's the smallest thing you can build it adds value?
get that out there get that in the hands of people and
See if it's working see if it's valuable
So start to think about what MVP you thinks may be necessary so
Maybe slightly slightly more interesting I hope I'm keeping everyone awake. I know it's early
So
talking about applied lean hunting
and I call it just hunting
Because I think we should be hunting more than threats we hunt a risk and a risk is an overloaded term but
Aspects of your environment to contribute to a higher likelihood of compromise
So sitting here can you all think about what can I build
To start helping hunting in my environment maybe you have a great hunting program. I'm sure it can be better
Maybe you have zero what can you do to start
So what is your pain point
when you sit down and you say you know what I want to hunt what is your pain point and
Start to think about what would you build is a product
service
piece of information whatever it is who is this for
what is this for it's
a pretty common
Example, which is are you building a painkiller or a vitamin I
Like vitamins but they're not that necessary when you need a painkiller you need a painkiller and you will pay for it
And
Einstein said this or at least people think he did that's up for debate like most things
If I had to spend an hour solving a problem that's been the first 55 minutes thinking about the problem
Do you understand your pain points when it comes to hunting what's truly preventing you from doing more hunting or doing better hunting
can you think about that
So
When you think about hunting or at least when I think about hunting
you're kind of choosing left or right and
Then you need to figure out did you make the right choice or not
But if you made the wrong choice you need to figure that out very quickly we all pull those threads
during analysis during investigation and go down a rat hole and lose days weeks whatever
And find nothing or learn very little
so can you very quickly fail and realize hey, I went down the wrong path
let me
course-correct pivot go the other direction
And you kind of do that over and over again until you get to what you're looking for
So how quickly can you do that and can you learn from that can you start to realize why did you make that decision
was it just sort of gut flip a coin whatever
Why'd you run this query why'd you decide to look over here can you start to understand maybe a little bit more data or science
Behind that so that next time you get better and better and better and better
And
You might have seen this slide before
but a lot of it starts with visibility a lot of hunting is you have tons of logs tons of data
If you have a data like a sim bunch of tools whatever and you're querying you're searching you're looking at stuff cool
But if you don't have the information what are you doing how are you hunting not, saying it's impossible
But it's gonna be a lot more
retrieving and waiting for things to come back so if you can collect information the better and
Can you collect the best information can you collect things that are already enriched or can you enrich information before you start using it
What is this IP address who is this user what are the relationships between these people are these events that's super valuable information
You've got to try to start with that and
Then utilize open source and ap is
Hunting isn't necessarily
Programming software engineering but I have to say the best security teams out there that I see are basically all software engineers at this point
they try to automate the crap out of everything and then they write code for anything new and
a lot of teams do a great job of combining commercial and open source
some products maybe it makes way more sense to do the commercial version but there are some pretty cool open-source tools out there and
Guess what maybe you can prove out your point with an open source tool that doesn't require a procurement
Because I know that's a pain point that's a block to hunting sometimes and
just to call out a specific example
One of the coolest things i've i've seen
It's just a sort of slightly thinking outside the box is like honey cards or honey credit cards kind of thing where teams
just spend like a thousand bucks on some
prepaid credit cards sprinkle those credit card numbers throughout your environment and documents and
then if you ever get a charge in any of those cards
it's a very high signal-to-noise ratio
You might never get a hit so you can't just count on that but very high signal-to-noise ratio
Those are the kinds of things you can start to deploy without a lot of effort without a lot of cost
And
In thinking about this you know my original thought for the talk was because I wanted to get you riled up
Our threat hunting an instant response the same thing
And I still asked myself that and I still think maybe they are
We hunted, bin Laden but that
was really Incident Response if if we're InfoSec they're sort of hunting my tenants in Chicago but
sort of Incident Response there too
so
You know I don't know but the point here
Is when you start thinking about your pain points maybe how you can contribute to improving hunting
Improving some aspects of maybe Incident Response triage there are different stages you know hunting may be a little bit more exploratory
Discoverable events and then usually you still have to do some triage before it's a full-blown investigation or maybe what people call true incident response
Where are you helping? Where can you provide value can you take capabilities from different parts of the spectrum and
help and use tools for different things
Can you use a hunting tool for?
investigation probably can use an investigation tool for hunting probably
So just be thinking about that how do I use stuff. I already have people already have
And
then entrepreneurial journey you have to sell
How many people in the room are salespeople?
How many people who didn't raise their hand consider themselves like they sell stuff
Well I'll give you a
Spoiler here when I was teaching my entrepreneurship class and the class
was pretty cool is 10 weeks you come in with nothing you make teams and at 10 weeks it's like Shark Tank you pitch to
A whole panel and you know here's my idea. Here's my MVP there's you know customer feedback etc. Is awesome I really enjoyed that
But I asked one of the teams what are you selling they said oh we're not selling anything
I'm like no you're always selling. I'm selling you guys right now on these topics on
me on my company on everything can you sell
your company on new spending
can you sell your
Organization on freeing up time to hunt
Can you sell the culture on?
helping you improve
efficacy or other aspects of hunting and it's in response for example if everyone would just listen to music on their phone instead of installing
Spotify and everything else on their laptop you'll have fewer events to look at
the culture can truly make a difference in how easy you can hunt or how effective you can be
So you have to learn how to sell
I'm a huge introvert
Technical geek guy and I have to do all sorts of stuff and go ask for millions of dollars and things like that
it just helps if you learn how to sell and
Then competition you know you don't want to completely focus on competition when you're building a product
but if you're a practitioner
Your competition is other
activities
Everything is grabbing at your time trying to steal your time
And some of its waste like we talked about so can you look at comput mission hunting competition and
Figure out ways to create more leverage involve other people
Automates that kind of thing so then you free up. Time the other thing I want you to do is
Beat up your vendors and I'm a multi vendor time person so I can say that
When you figure out something that should be done in your environment or something that would make your life easier
build it yourself but at the same time can currently ask your vendor join a customer advisory board
just provide some feedback go to user groups whatever it is
It doesn't hurt you to ask
Could they add this one field could they output it in JSON instead of syslog
whatever it is there's a whole bunch of stuff just ask
Ask as a vendor we crave
Feedback, we want to hear what makes your lives easier?
So, yes build stuff yourself the best teams out there are building things themselves but they
also push hard on their vendors
So wrapping up and ranting even more than I already am
So one of the TED Talks I was watching I can't member even which one but the absence of disease does not mean health
And I think that describes
InfoSec very well
the absence of a PT or the absence of compromise or pick your term
Does not mean you have a great environment. I think everyone would pretty pretty quickly
Agree with that so when I talked about hey
I consider hunting not just thread hunting but like other stuff
Other things you can go look for is a human to try to improve things
Can you figure out where your entropy is or where different aspects of risk are
Can you when you hunt a successful hunting outcome why not be finding
apt, or pick your terminology
Sophisticated thread actors or even run-of-the-mill, malware it might be finding misconfigurations or lots of configuration drift
that kind of stuff
and
You know us thinking about this and I was like you know what I'm gonna you know we talked about upstream downstream
I'm gonna make a chart or a graph that says hey risk is a slope and the more risk you have the steeper the slope
and then your environment is sitting at the top and it's just kind of sliding down into
Compromise
So can you lower the slope decrease the slope
Reduce risk reduce the slope
and
So one thing that's really
really common is something called identity creep
privilege creep pick your latest buzzword of the year kind of thing but everyone has access in this room everyone has access they don't need
100% guaranteed I've bet a lot of money on that our percent we have 30 people
We have people that haven't used AWS for 238 days now that's my CEO who is busy doing other stuff
But even we at 3030
have
accounts we don't need and
Then it is hard I dumped our AWS
schema they have a way to dump it your security settings and policy settings in JSON
30. People not doing anything crazy
20,000 lines of JSON so it's a hard problem figure out who has what permissions I am roles etc
The point is when you're hunting you might find some of this stuff can you go find accounts that don't need to exist
It's not just about malware we're gonna get a lot of great talks on worse engineering and network packets and stuff, that's awesome cool
We need to know that stuff but it's not just about that
Can you right size your surface area?
can you find either it could be
Accountants be accounts it could be systems we've all heard of the server that everyone thought was unplugged and that
Was the one that got compromised no one even knew it was still in the network
Can you shrink your surface area reduce your risk
and guess what
it gives you a more focused area to hunt in
It's like if you shrink the size of a football field and you're playing defense you're probably going to have a better time
Because you don't have to run all over
Defending the wide receivers and stuff
and
Look we're talking about lean we're trying to do more with less and and just you know reduce waste
talk to a 600 person company
Just in three services they're wasting 35 K a month
They can't hire headcount the the C
So has actually just left because he couldn't get any headcount that's like 400 grand that's plenty of headcount there
Or at least that's a start will say
So in your hunting there might be different outcomes the point of all this is it's not just about things like malware or compromised
find things that contribute to waste
find waste and
To rant a little bit more I think Phil and and Rob mentioned there are 60
60 talk submissions I was on the I'm on the advisory board for this so
Thank you for all the submissions I really enjoyed reading them and voting on which ones I thought
could get picked and and if you didn't get picked to come come yell at me or whatever
It wasn't just me it was a group of like ten of us or whatever?
probably blamed Phil or Rob the most though
but
60 submissions I
Think three or four were on cloud I
Was blown away we got to think more about cloud cloud is here
cloud is here
IT is going from zero to 100 and leaving cloud security in the dust when it comes to cloud
We're blind to all these SAS accounts
50% of IR
For rapid seven or at least this team at rapid seven is
office 365
50% that's a lot
And we have 300 AWS accounts and no governance sure no one has that problem in this room
And I just like this quote hackers don't break and they log in but the point is I hope next year
We talk a little bit more about cloud
AWS forensics hunting and slack I don't know pick your pick your topic but it's here it's here
even the big
Conservative financial institutions they've been around a hundred years are
Moving I've talked to the intelligence community they're like we want to use slack but
We don't know how to use it because we can't really monitor it
It's here we need more talks on cloud
So wrapping up here, where's the waste where's the extra risk where's the entropy
when you hunt
Can you find that kind of stuff
where can you spend your time when you actually do get time to hunt
Can you find time to do that?
Identify what's providing value and?
focus on that and try to reduce
The other time you spend we all have to experiment and try things but as soon as you realize this isn't actually
increasing our capabilities to haunt do instant response overall cyber defense whatever it is
get rid of it or Park it for a bit
Build measure learn
I think everyone will remember that but go build something even if it's just tying a couple api's together
See if that helps you write a little bit of python
Python, never killed anybody as far as I know maybe I'm wrong
think big start small act fast
Be the hunter that your environment needs
There's a quote from Guy Kawasaki a legend in Silicon Valley a chief evangelist at Apple be an entrepreneur being an entrepreneur is a state
of mind not a job title and I'm pretty certain
I'm not the first person to say this but I don't know who who
Did so I just put my name there being a hunter is a state of mind. It's not a job title
So I told you I like a lot of books I could give you tons and tons of recommendations I
love these books I read them multiple times
Rocket men is one I'm reading right now about Apollo 8, which is really exciting space race between Russia and the u.s
Essentialism team of teams about how
JSOC had to reform themselves to fight al-qaeda in Iraq it
Was just really interesting on extreme ownership all about some some lessons learned from
really high combat areas really cool stuff
So the goal sparked contemplation
And I know I threw a lot at you and some of it you're like why the hell, did we learn about that
well
keynotes are like that
But Reid Hoffman founder of LinkedIn Silicon Valley God
If you're not embarrassed by your first version your first product, you've shipped too late the point here is you can take a small step
Towards better hunting capabilities without a lot without a lot you have some data already you have some api's already you have some skills
Go connect them in the right way think about it more like an
Entrepreneurship how can I build my case how can I build my case to get funding even if that funding is just time
So what can you do today to upgrade your hunting
Thanks
You