Splunk: Building a Secure Monitoring Solution (Part 1)

During the last several weeks of my Cybersecurity boot camp, one of our final projects was to build a secure monitoring environment for a fictitious organization called VSI (Virtual Space Industries) using Splunk Enterprise, which for those who may not know, is a SIEM (Security Information and Event Manager). SIEMs are essential tools that companies can use to detect, analyze, and respond to potential threats against their organization.

Since this was a big project with a lot of steps, I'll be breaking it up into 2 parts:

Part 1:
Creating Reports, Alerts, and Dashboards for Windows server log data as well as Apache webserver log data that can help point out any abnormal activity.

Part 2:
Checking to see if the solutions created in part 1 were effective against a fictitious attack by uploading the attack log data and seeing if our reports, alerts, and dashboards, picked up anything that would have helped the organization take the appropriate action as quickly as possible.

Part 1
I started by launching Splunk, which had been pre-installed in my ubuntu VM. I logged into the application and uploaded the files I would be using to create reports, alerts and Dashboards for.

Once the logs were uploaded, I briefly took notice of and analyzed the following fields:
o signature
o signature_id
o user
o status
o severity

Apologies that some images might be a bit small and difficult to read. Luckily, I also took some screenshots of the data inside each individual field as well:

signature:

signature_id:

user:

status:

severity:

These would be the main points of interest we will be using to create our reports, alerts, and dashboard. Let's start with the reports!

Report 1: A report with a table of signatures and their associated signature_id. This would allow VSI to view reports that show the ID number associated with each specific signature for a Windows activity.

Report 2: A report that displays severity levels and the count and percentage of each. This would allow VSI to quickly understand the severity levels of Windows logs being viewed.

Report 3: A report that provides a comparison between the success and failure of Windows activities. This would show VSI is there is any suspicious level of failed activities on their Windows server.

Now that those were done, it was time to move onto creating the Alerts!

Alerts would all trigger an email to be sent to the fictitious company at SOC@VSI-company.com

Alert 1: An alert that is triggered when a threshold for hourly failed Windows activities has been reached. This would help VSI see if any failed logins or any other activities occurred an excessive amount of times within an hour, which could be indicative of someone trying to do something they shouldn't be able to do, such as trying to login and failing repeatedly.

The threshold for this alert I chose was > 18 per hour.

Alert 2: An alert that is triggered when a threshold has been reached for the amount of successful logins per hour.

The threshold I chose for this alert was anything > 26 per hour.

Alert 3: An alert that is triggered when a threshold is met for the signature count when a user's account has been deleted, once again in an hourly window.

For this alert, I chose anything > 35 within one hour.

Now for the fun part, creating Dashboards to monitor Windows Server Activity at a quick glance!

I always have a lot of fun creating dashboards in Splunk, this project being no exception. I made:

1. A line chart that displays account deletion signatures over time within the span of 1h.
2. A line char that displays the different user field values over time.
3. A pie chart of the different signatures based on windows activities.
4. Another pie chart showing the different users who are active.
5. A final pie chart that tracks the source domain.

Line charts 1 & 2:

Pie Charts:

Full-view Windows Server Monitoring Dashboard:

Next, we needed to repeat this process, but for the Apache log data. I went ahead and uploaded the log files and got to work on the reports first.

This time, the important fields we wanted to pay special attention to were:
o method
o referrer_domain
o status
o clientip
o useragent

Report 1: A report that shows a table of the different HTTP Methods (GET, POST, HEAD, etc.). This would show VSI the types of HTTP requests being made to the VSI webserver.

Report 2: A report that displays the top 10 domains that refer to VSI's website, to help VSI identify any suspicious referrers.

Note: I found it kind of funny that 'referer' was spelled incorrectly (should be 'referrer' in the fields list), but knew it had to match the data, so spelled it incorrectly to match when I had to.

Report 3: A report that shows the count of each HTTP response code. This will help VSI to quickly gauge the overall health of their webserver and activities taking place on it.

With that, it was time to create some Alerts!

Alert 1: The project called for an alert that triggers whenever a connection is made from any IP address outside of the United States. I chose France for this example, which had an IP Address of 176.31.39.30 (Roubaix, France).

Alert 2: An alert that triggers whenever a threshold was met for the count of HTTP POST methods within the span of 1 Hour.

The threshold I chose for HTTP POST requests was anything > 10 within 1 hour.

And now back to the fun of creating visuals for our Dashboard!

HTTP GET Method requests per hour:

HTTP POST Method requests per hour:

HTTP Methods by type per hour:

Top Countries connecting to the server:

Top User agents:

Top URI:

Full-view Apache Server Monitoring Dashboard:

So there you have it! I created Reports, Alerts, and Monitoring Dashboards for VSI's Windows and Apache servers.

In Part 2 we will see whether or not the solutions I made protected VSI.

Part 2: https://dev.to/r33keeper/splunk-building-a-secure-monitoring-solution-part-2-208m