loading...
Cover image for Analyzing the Attacks on my Website
Pluralsight

Analyzing the Attacks on my Website

jeremycmorgan profile image Jeremy Morgan Updated on ・5 min read

I was casually doing a security audit on my blog (JeremyMorgan.com) recently and decided to look a little deeper into my security logs. With a bit of Linux command line kung fu, some Golang, and Google sheets, I was able to get a pretty good idea of where the attacks are coming from.

To start, I'm using CentOS to host my site, so I checked out /var/log/secure. This log is where authentication logs are stored on my server.

This is what the log file looks like:

Analyzing attacks on my website

and with 301,327 lines it's not likely I'm going to manually look around much. Let's automate this a bit.

Getting the IP Address of attackers

I wanted to extract the IP address of attackers from this file. That way I can block them.

I started to mess around with Linux commands until I came up with this script.

What it does is pretty simple, it's going to look for these strings:

declare -a badstrings=("Failed password for invalid user"
                "input_userauth_request: invalid user"
                "pam_unix(sshd:auth): check pass; user unknown"
                "input_userauth_request: invalid user"
                "does not map back to the address"
                "pam_unix(sshd:auth): authentication failure"
                "input_userauth_request: invalid user"
                "reverse mapping checking getaddrinfo for"
                "input_userauth_request: invalid user"
                )

These are strings that identify logs of failed attacks. If they put in the wrong username or tried some other form of attack, it would have one of these strings.

So we loop through that list and search for these strings, then extract an IP address from the line the string exists in.

cat /var/log/secure | grep "$i" | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | awk '{print $0}' | sort | uniq >> "temp.txt"

It then dumps the IP into a temp.txt file. It will do this for all of the messages I have in my "badstrings" list.

That text file had a ton of duplicates in it, so I removed the duplicates and put only the unique IPs into a file:

# grab unique ips from temp and put them in a file
cat "temp.txt" | sort | uniq > "badguyips.txt"
# remove the temp file
rm "temp.txt"

Cool, now I have a list of IP addresses ready to go.

Analyzing attacks on my website

Yikes, I have 1,141 IP addresses here.

Blocking Them

Now I want to block these IP addresses. Since I'm running iptables, I can just drop them with this simple script:

#!/bin/bash
input="badguyips.txt"
while IFS= read -r line
do
  iptables -A INPUT -s $line -j DROP
done < "$input"

service iptables save

Cool. Now the attackers blocked from my server.

Then I got curious. Where the heck are these attacks coming from?

Getting their Location Data

Since I have a list of IP addresses, I thought I'd run them against a database like Maxmind to find some location information. So I did just that.

I wrote this Golang program called "find the bad guys" that would go through the text file of IP addresses, and look up their location information, then write it to a series of text files.

I wrote out locations based on:

  • Continent
  • Countries
  • Cities
  • Subdivisions of Cities

I wanted to see where the attacks are coming from and share that information. so I ran the program I built, and now have some helpful lists of location information:

Analyzing attacks on my website

Continents

So now I want to take a look at continents.txt.

Analyzing attacks on my website

Well, that's going to be a problem, there are some duplicates.

I can run a quick command and get unique values:

cat continents | sort | uniq

The results should come as no surprise if you've ever looked at a globe:

Analyzing attacks on my website

But I want to see how many attacks from each continent. So I call on my old friend uniq for that:

awk -F '\n' '{print $0}' continents.txt | sort | uniq -c

Analyzing attacks on my website

Pretty sweet, right? So I'll remove the leading spaces, insert a comma after the count and drop it into a text file.

awk -F '\n' '{print $0}' continents.txt | sort | uniq -c | awk '{$1=$1};1' | sed -r 's/\s+/,/'  > contintent-totals.txt

Analyzing attacks on my website

Now I can drop it into Google sheets.

Analyzing attacks on my website

and get this nice chart:

Analyzing attacks on my website

This is the process I repeat for the other locations (country, city, subdivision), so I won't repeat it. So here are my results:

Countries

Analyzing attacks on my website

Here are the top 10 countries attackers are coming from:

  • China (304)
  • United States (138)
  • France (95)
  • India (46)
  • Singapore (43)
  • South Korea (38)
  • Germany (37)
  • Russia (37)
  • Brazil (35)
  • United Kingdom (29)

Cities

Analyzing attacks on my website

Attacks per city are a little more aggregated.

  • Beijing (57)
  • Shanghai (53)
  • Hefei (25)
  • Amsterdam (21)
  • Bengaluru (16)
  • London (14)
  • Xinpu (14)
  • Clifton (10)
  • North Bergen (9)

But still pretty interesting.

Subdivisions

Analyzing attacks on my website

This one is aggregated even more. But it drills down a bit more. Here are the top 10 subdivisions attackers are coming from:

  • Beijing (145)
  • Shanghai (61)
  • Anhui (26)
  • England (22)
  • Jiangsu (22)
  • New Jersey (22)
  • North Holland (22)
  • California (18)
  • Sao Paulo (18)
  • Karnataka (16)

Conclusion

Great things always come from curiosity. I'm curious about what other kinds of patterns and data I can extract from this, so I'm going to keep experimenting and playing with it.

If you decide you want to do this for your website, try it these steps, and Let me know if you need any help with it.

Posted on by:

jeremycmorgan profile

Jeremy Morgan

@jeremycmorgan

Silicon Forest Developer/hacker. I write about .NET, DevOps, and Linux mostly. Once held the world record for being the youngest person alive.

Pluralsight

With our technology skills platform, companies can upskill teams and increase engineering impact.

Discussion

markdown guide
 

You should look into two things:

  1. fail2ban, for automatically banning IP addresses. github.com/fail2ban/fail2ban
  2. firewalls above your individual VM. Banning SSH connections from all but the IP addresses you personally access your VM from will make this a non-issue.
 

Seconding the fail2ban tooling - excellent support for just about anything you wish to expose to the random Internet, and you can still pull many stats from it's logs for graphing.

Curious as to how many of the IPs you are seeing attacks from are exit nodes for commercial VPNs or Tor? How many are cloud hosted VMs, or zombie personal machines? IP allocation / reverse DNS should help find this stuff out, as would services like this: ipvoid.com/ip-blacklist-check/ :)

In addition to local controls (such as fail2ban), I would also strongly encourage use of a CDN/anti-DDoS/Web App Firewall product (such as Cloudflare, Azure CDN, AWS CDN, Imperva Cloud WAF, etc.) for your outer layer or protection - they will keep up with threats waay faster than one human!

 

You didn't mention SSH keys and maybe you already do this, but for the benefit of others trying to secure a public-facing machine, it's best to disable password-based authentication and use ssh keys to authenticate instead. Combined with fail2ban (mentioned by Ben) it's a good way to prevent brute force attacks.

DigitalOcean has a good writeup of how to generate SSH keys and configure a Linux machine to use them.

 

Cool writeup. There are millions of ways to do this ofcourse.
May be you want to look at google sheets API so you can mostly automate all tasks in the above list and can analyze stuff over time directly in google sheets ;)
or you can just use prometheus and grafana...haha

 

First off, very nicely done. Love the breakdown and that IPTables loop script is awesomely useful too.

Don't they often spoof IPs so banning that stops them right now, but they're back in an hour on another IP?

Also, using the auth log, is that a guarantee these are all attacks, or is it possible some were "good guys"? [just playing devil's advocate here]

Gonna do something like this on my servers, see what I get out of it :)

 

I was just thinking this. They could definitely use a VPN or some other proxy to mask their IP address.

 

Please note that using

iptables -A INPUT -s $line -j DROP

is appending your rule to the end of INPUT rules. This can be a problem if the INPUT chain has any other rules that have been applied before.

To ban an IP is better to prepend the rule at the beginning:

iptables -I INPUT 1 -s $line -j DROP

Althoug I prefer to create a custom chain for them, so you can prepend your custom chain to the beginning of INPUT, and append ips safely to it:

iptables -N bannejats
iptables -A bannejats -j RETURN
# No line specified because it defaults to 1 :)
iptables -I INPUT -p tcp -j bannejats

input="badguyips.txt"
while IFS= read -r line
do
  iptables -I bannejats 1 -s $line -j REJECT --reject-with icmp-port-unreachable
  # Or also safe now because bannejats is on head
  # iptables -A bannejats -s $line -j DROP
done < "$input"

(A note on Drop versus Reject chiark.greenend.org.uk/~peterb/net...)

 

You don't need to pipe the log to grep with cat since grep will take a file input as a second parameter.

grep "$i" /var/log/secure | ...

Thanks for the great write up.

 

sshguard, fail2ban will automate this. You should try ipset instead of blocking via iptables.

 

One of the easiest steps (next to the already mentioned) is using a different port for ssh. This way these automated attacks won't happen anymore.
Additionally fail2ban and you're set.

Disclaimer: I know this won't stop a dedicated attacker, but it will stop the automated attacks and scans.

 

Blocking IP addresses of attackers is a lost battle. It's incredibly easy for attackers to get a different IP...

I wrote (in 2016) an article with 3 simple tricks (that don't require installing any other app, like fail2ban) that have been working great for me to stop tentative brute force attacks: withblue.ink/2016/07/15/stop-ssh-b... TL;DR: use SSH keys, rate-limit connections to port 22 (using iptables), and/or change the SSH default port.

 

would make a great grafana dashboard.

 

I'd never heard of Grafana until now, but I'd have to agree! Thanks for the recommendation

 

What kind of attack are your website getting? Login attempt? DDOS?

 

Hello,

Very nice job. Great script and ideas. I know the point of your text was to illustrate the script and the automation behind it, but since you used a real life situation, I'd like to offer my two cents on that.

Instead of adding those IP addresses to an iptables script like that, you could try to take advantage of cloudflare's free plan to defend your vps. I'm not their employee, just a happy user. I manage small cloud servers as a side job and their firewall allows you to add a small captcha to validate a connection instead of simply blocking it. And you can create rules for specific countries. If you block an IP, there's no way for it to reach your website, even if it's a legitimate connection. A captcha is boring for the users, but it costs them only one click ;)

 

Thanks. It was interesting to learn about such a way to stop web attacks.

 
 
 

These are some GREAT tips! I'm saving this for the future. Thanks!

 
 

Nice write up, thanks for sharing the scripts!
I see similar attacks from China on all the servers I run.

 

Yeah, these are automated attacks, likely someone scanning a whole IP range. I haven't locked up this server as tight as others in the past, but in my next move, I'll probably see it a lot less.