(applause)
- So I'm Ryan Nolette, I've been doing
IR forensic-ey fun stuff for about 15 years now.
There's my GitHub page if you're interested
in any of the public stuff I've done.
As part of this talk I'm actually
giving away an Amazon detonation lab.
So that URL at the bottom is a quick easy way to set up
a fully configured detonation lab in AWS
takes about five minutes, builds a cool topology,
I'll expand on that a little bit more.
And of course, Time Person of the year 2006, come on.
(person cheering)
Three people got the reference, and I like you.
All right, so the things that we're gonna go over
are pretty much the overview of
why cloud really isn't that scary, right.
If you guys are all used to trying to protect
a perimeter that's on your on-prem network,
it's really the same thing just different words now.
So I'm gonna go through some of the logging examples
that you guys can do, and then go over an example
showing how you could actually do the exact same
investigation that you're doing
or the same kind of hunts right now.
The who did what and when.
These are the questions that I like to try and go through
for every single time I have to work through
in any kind of threat, hunt, right.
I want to bucket things into two groups,
visibility and accountability.
Because if you can't see it you can't account for it.
Therefore, I need to keep increasing my visibility
until I can account for everything.
Which is a nice end state.
So to get to that end state,
we do the same typical things, right.
Use OS hardening, you guys subscribe to NIST or BIS,
or any of those benchmarks, right.
You're still gonna apply those to your templates in AWS.
Config management, you can run Ansible, Chef,
you could use CloudFormation.
There's a ton of tools that you can use to maintain that.
Identity management and process monitoring,
those are a little bit more interesting.
And that's because of things like server-less computing.
So bucketing those back into the visibility
and accountability buckets, what you're really trying to do
is answer those three questions
of who did what and when they did it, right.
Because if you can answer any of those questions
you can cycle through and answer the other two, right.
That's where you want to be.
I want to be able to say I know when this happened,
now who did what.
And until you're able to answer those questions
you haven't added enough visibility
to have that accountability.
What are some common tools that we all use?
We use IDSes, we have EDR tools,
we do different things for Netflow or Identity Management.
Well these are the equivalents that are there.
All right, instead of Netflow it's called VPCFlow.
Do we know why?
Because it's the Netflow in your VPC,
which is your Virtual Private Cloud.
When you set things up inside your Amazon account
you're putting them in either one or a zillion
different VPCs that you create.
It's your own little virtual private space to play.
So increasing that visibility,
what we want to do
is we want to use these different groupings of controls.
Up here I've broken into about six different groups,
there's more of course, you guys have all seen
different compliance charts.
So, this is going through more of what are the tools
that are either native to the AWS realm,
or are able to be used either free
or just open source in general.
So, authentication launcher using tools like IAM,
which allows you to control the identities
that log into your account
and your different Amazon resources.
But that doesn't say who logs in
on the actual instance, right.
So you still need to collect the logs
off the instances that you spin up,
to see did the person sign in as root,
did they sign in as Apache for some reason.
And those kinds of normal system configs.
But, the end to end example,
this is going to be kind of fun,
because this is my detonation lab that I set up
with these scripts that I'm giving away,
and what's kind of cool is this is the typical instance
for people that have extensions to their on-prem networks.
So, right now show of hands whose boss has told them
that they're going to the cloud, right.
All right.
Who actually wants to do that? (laughs)
Right?
It sounds scary, it has a whole lot of
different controls and things,
but what I kinda want to bring out is
cloud's not gonna rain on your parade. (laughs)
Yes, cloud reference, dad joke, thank you.
The thing is, a lot of people are just looking at things
as can I bring my current technologies up with me.
And that's often no because those products
were designed for static systems,
in traditional networks.
Well, that doesn't always work.
So, what I've done here is I've built this
CloudFormation template which I'll talk more about
CloudFormation in the lightning talk,
to give you an idea of what it does.
But you basically run this script
and it goes in and configures your entire topology for you.
This is the topology that it builds.
It has a private VPC that I stuck four systems in,
and that's guarded by a Bastion host.
So I can do all kinds of fun stuff in there,
but nothing really leaks out.
Inside I have a red team server,
which is where I'm gonna be doing my attacks from
for this demonstration.
I have a Windows victim, and I have a Linux victim,
and then I have a Wazuh server.
Has anyone here ever heard of Wazuh or ever used it?
No?
All right, how about OSSEC.
Do people know what that is?
All right.
Oh, Wazuh is a more commercial version of that.
It's basically an elastic stack with a app that does
OSSEC stuff on top of a lot of other new features.
But the reason why I'm using them
is because I worked with their team
over the last three months to build in
ways to ingest logs from AWS.
All the different logs that I normally use
to do detections they now can all ingest.
So super big shout out to the developer, Marta,
because she's the one that actually developed all of this
(laughs) for the Wazuh side.
The pipelines that this template builds
look kind of complicated, and that's because
there's a different methodology with things
in AWS and other cloud providers.
They're no longer talking about your application,
your instance, or terms like that.
Now things are being called workloads
because people are moving off of,
I'm gonna stand up an instance of E2,
I'm gonna install Apache, NGX, whatever,
and run my application off that instance.
Well that's pretty expensive
because there's a lot of overhead
that's unnecessary to do your end goal
which is run that one application
or your one workload.
That's where people start moving into things like Lambda
or Kubernetes, I never can pronounce it. (laughs)
You know, docker style virtualization systems.
That's why people are moving their stuff there
'cause it's significantly cheaper to run the same workloads.
So the first tool that we went over is CloudFormation.
This is what I use for automation.
If I want to stand up a stack to quickly poke at
some new attacks that I just learned,
I'll run this script and somewhere around
five minutes from there I have
that full detonation lab set up.
Right, the same kind of idea.
It's quick, it's repeatable, it's free.
Big, big fan.
S3, that's just where I dump all my logs, that's it.
It's this giant bucket,
and they're actually called buckets. (laughs)
That allows me to store the data that I'm pulling
from all these different logging sources.
VPC we quickly went over, that's just the virtual network
that all my instances are gonna be installed into.
VPC Flow is what monitors inter-instance
and extra-instance network traffic.
That allows me to monitor any connections
between the systems in the VPC
as well as external to the VPC.
Lambda, this is a piece of code that I'm using
to add in enrichment metadata to my VPC flow.
Firehose, this is what collects those logs.
So I'm sending all my logs to Firehose
which can actually handle that kind of volume.
Because Firehose scales natively.
So, if today I only have 20,000 logs,
sorry 20,000 traffic events,
but tomorrow I have 20 million,
it will still scale for that.
IAM, this is that identity management
that we've been messing around with.
This is how I create all the users
that are allowed to access my S3 buckets,
as well as are allowed to run different privileged items
from inside my network.
CloudTrail, CloudTrail monitors API calls.
So anytime you interact with the AWS console,
or use AWSCLI, or any of those native tools,
you're interacting with the AWS APIs.
Every single time you do that
you are generating events in CloudTrail.
So we use CloudTrail for all kinds of fun detection stuff.
Did somebody just XFIL your temporary credentials
from your instance and are now using them
to access your S3 buckets?
There's CloudTrail events for that,
and we can use them to simply say this bad thing happened.
Really, really useful for correlation.
Macie, this is an actual security tool that people use.
This monitors and categorizes data in your S3 buckets,
so if Bill from accounting uploaded one of his spreadsheets
and it has Social Security numbers in it,
and stuff, that gets flagged.
So it does cool stuff like that.
And you can create your own different triggers
if you want something to flag on if the name Ben is in there
you can write something like that.
Kind of think of it like, it's based on, what it's called,
but the data, it's not DEP,
it's a compliance one that I can't think of right now.
So Inspector, this is my vulnerability scanner.
This allows me to scan against my instances
for things like CIS compliance, common vulnerabilities,
Oauths, Pop10, stuff like that.
And CloudWatch Event Rules and Logs.
This is the main thing that I use to capture
events from these services, so I can forward those logs
and write them to my S3 bucket.
Without CloudWatch monitoring all those different services
nothing's actually grabbing the logs
out of the service when they happen.
So you can log everything but the only way you'll see it
is in the console of that tool.
This allows me to take it out of that tool
and drop it into S3 so then I can ingest it
into an elastic stack kind of setup.
And then GuardDuty.
This is another tool that works kind of like,
it's kind of like an IDS/behavioral analysis tool.
And it has a bunch of pre-configured stuff
that it looks for.
This one is just looking for a cryptocurrency.
This is traffic going from one of my instances
to a known destination that's been categorized
as cryptocurrency.
And then this is Wazuh.
Once again big shout out to them for all
of the stuff that they did for me, huge, huge help.
This is what I'm using for an EDR tool in here.
This basically installs OSSEC on the different instances,
as well as it's gonna pull stuff into my log stash
and eventually feed my Kibana dashboards,
which is extremely, extremely helpful
'cause now I can do full stack analysis.
I have an EDR tool,
I have a tool to look at my network traffic,
I have a tool to look at my identity management,
my config, vulnerability scanning,
and categorization of compliance stuff,
and it all gets dropped into one place for me.
This is what the detonation lab CloudFormation script does.
So you guys can go in there and play around,
and it'll tell you what breaks and where.
To give you an idea of some of the mapping stuff
that I've done, this is just for CloudTrail events.
I wanted to give an idea of how many things overlap
between these logs that allow you to map between them
in order to do further kinds of detection, right.
If you're hunting through data
and none of the data is communicating or connected together,
how do you end up connecting
those relationships between the data?
Well, what I ended up doing is I took
common overlapping things, for instance for the AWS account
we have something called an ARN which is an ID.
You have a principal ID, you have an access key ID,
those are overlapping fields under account management.
So those are what identify accounts.
By using that I can now connect every event
from every AWS service I have
that uses one of those four identifiers,
and I can map it to the same account now.
So that's huge for the attack that I'm gonna be showing.
So an example of a couple of the dashboards.
Here's some simple, easy, pretty pictures.
Management loves these.
These are not that helpful, but management loves them,
so keep them in mind.
These idea for these are top external destination IPs.
Now, not always helpful.
I have the accounts for GuardDuty alerts.
So these are the accounts that set up GuardDuty alerts.
I already have one account in the test lab
that gets destroyed so I only have one account there.
But in a real production environment
you could have a couple hundred IDs.
And this tells you which one of your users
is constantly doing things that are weird.
Next up are talkative, the most talkative instances
based on their interface ID, right.
When you have these virtual instances
you can put as many nicks in them as you want,
so how do we know if a instance is being talkative
or if their attacker is moving the nick
to different systems so they can run traffic
outside of your monitoring, cool attacks like that.
This ends up capturing things like that.
So, finding what matters.
This is what I did with the Kibana boards,
that come in this,
I feel like the voice of god just happened there. (laughs)
The Kibana boards, these are gonna use all these dashboards
and they're gonna show you what's happening.
So this is the threat hunt
that I had a junior analyst do, right.
They went in and they started looking at
places where there's a ton of traffic.
Now this is a test at Nation Lab,
this is not production data,
so it looks an awful lot like China (laughs)
has a lot of traffic going to it.
Maybe they're outsourcing their math,
but most likely what's happening
is something is sending out and talking.
So I dig into that country.
From there I can see a massive spike of traffic
that happened yesterday, but all the other days before it,
for 30 days, nothing has happened.
I can dig into that and see that the top IP address
is this 58.218 address.
Well, from there I can go and see if any alerts happened
for that destination.
And lo and behold, I have some alerts from this IP,
magically for this demo, everything worked out perfectly.
So, this now allows me to find what internal instance
has this fun traffic to it, and now I want to see
why this thinks that I have something talking out
to a cryptocurrency exchange,
and I can use those logs for the EDR tool
to find out that my netstat table changed,
and it shows a binary called pip3.7,
that is using that destination IP address.
And when I go into that I can see that it was installed
by the user Apache, and then I can look at the hashes
for that binary and VirusTotal
happens to have that as a known crypto miner.
Yes, there's a lot of conspiracies
about how well this worked out,
but the overall idea is all of this information
was able to be related together,
and we're able to do a full stack (laughs)
investigation of something that happened, right.
All this was visual, I didn't have to go and dig through
a couple million lines of text.
I was able to pull these together very quickly
and easily, to show this full picture.
This is something I would end up presenting
to management of a great example of this junior analyst
who's ready to take on more responsibility.
Right, they were able to go start to finish
without any issues and were able to connect
all the different disparate data sets
that they ended up having to work with
all within free and open source products.
So, how did they get in?
Now, this one I actually have a feature request out for
because it's not currently baked in,
but I went ahead and I used inspector to do some scanning
of my instances, and I found that my web server
on the Linux victim has not been patched.
So, (laughs and imitates sad trombone)
I wonder if it was a web shell,
but most likely it was just a really bad web server
that ran Netcat to an attacker.
Now, to do kind of a recap, right.
Does cloud seem as scary now?
I hope not because every tool that I just used
is available on prem, and most likely
if you have a sock, hell if you have something simple
like free and open source Security Onion running with OSSEC.
That's all free, yes the cost is really an effort
for setting stuff up, but you could do everything
that I just did without spending any money
outside of hardware, right.
So I wanted to go through that stuff showing,
these are common tools,
they just have a different name right now,
and we're working on naming conventions
to make things a little clearer for people.
But, with the last couple minutes that I have
I want to get to the most serious question,
the attacker lifestyle.
Now, this is in honor of squirrel,
because if anybody knows my corgi attacker life cycle,
this is quite similar, though this is from the idea
of the cryptocurrency miner.
We have script kiddie scanning around the internet
looking for open addresses,
they end up popping one and run a cryptocurrency miner,
all of a sudden they have some cash,
so they buy a fancy car.
They own the only Lamborghini in Eastern Europe.
Somehow that stands out for some reason.
They get investigated for tax evasion,
they get arrested, they flip on people,
and now they're in the witness protection agency,
and they want to start it all over again.
I'd watch that movie. (laughs)
So kind of my motto that I like to have
coming from a blue team perspective
is Flag it, Tag it, and Bag it, AKA the double tap.
If you find something, flag it, right.
It's either the user manually looking for things
in the threat hunt, your static defenses
that are looking for common signatures
or anything else, right.
Show what happened and then tag it.
Even if it doesn't look malicious,
if it looks anomalous, tag it.
And eventually you build enough information to it
to bag it and get that out of your network.
So, I ended up right on time for minutes,
do we want to do questions or push them to the end,
so that way we can keep on schedule?
(laughs)
Play Free Bird!
All right, and while they're talking.
That's just so much fun. (laughs)
- All right.
- It makes me happy.
- You good?
- Oh yeah.
- All right, round of applause. (cheers)
(applause)
(light piano music)