Linux Firewall: Blocking a lot with a little

I have a need to block a large list of ever changing IP addresses from servers and systems I operate.

The list of IP's I'm using can change hour by hour based on certain signals obtained from different logging and monitoring systems that are in place and some times the list of IP's can belong to an entire country!

Basic Blocking

I can block single things easily with iptables by using this on the command line:

iptables -A INPUT -s 1.1.1.1 -j REJECT

This will REJECT incoming connections from 1.1.1.1 ( Cloudflaire's public DNS resolver )

You probably don't want to do this as its just an example of a simple block command but the IP address listed there can be changed for something different or it can be changed for a network prefix for example 10.0.0.0/16

Grouping things

I can create an additional Chain in iptables to group a bunch of things together.

for example, I may want to filter things heading to my mail services
I'd create a new chain with the command:

iptables -N mail-services

Then I would point all of my mail services in that direction:

iptables -A INPUT -p tcp -m multiport --dports 25,465,587,143,993,110,995 -j mail-services

As before I can then add in specific rules for servers to the mail-services chain:

iptables -A mail-services -s 1.1.1.1 -j REJECT
iptables -A mail-services -s 8.8.8.8 -j ACCEPT

In this example I am rejecting connections from 1.1.1.1 but ACCEPTing connections from 8.8.8.8

Extending further

Now I have a chain setup for my mail services I can now start blocking ( or allowing ) certain sources attempting to connect.

I have a handy selection of IP addresses gathered from my monitoring systems of IP addresses that are stored safely within a database which I can access by running a local script.

The script returns a list of IP addresses from the database on each line.

The output of the script would look something like this:

1.1.1.1
1.1.2.2
192.168.1.0/24

etc

I make use of this script output by wrapping it in another shell script like so:

#!/bin/bash
## Flush the mail-services chain
iptables -F mailservices
for ipaddr in $(./spot-block-ips); do iptables -A mail-services -s ${ipaddr} -j REJECT; done

This will add in all IP addresses flagged to be blocked to the servers firewall after all the old ones have been cleaned out ("flushed")

CPU Overhead

This works really well when there's a nice handful of IP's to block from the mail services.

However, now the list has grown quite large there is a slight overhead on CPU time to do this now.

IPset's

Turns out there's a better way of doing firewalling with much bigger lists and that's by using ipset

From the man page for the command:

ipset is used to set up, maintain and inspect so called IP sets in the Linux kernel. Depending on the type of the set, an IP set may store IP(v4/v6) addresses, (TCP/UDP) port numbers, IP and MAC address pairs, IP address and port number pairs, etc. See the set type definitions below.

You may need to install that tool to use it. On Debian based distributions that will be apt install ipset

To simplify this process a little; if an IP address has broken one of our rules we'll now block it outright from all services rather than just the services it has attacked.

Create a new set

Run the following command to create a new ipset called blocklist:

ipset create blocklist hash:net counters

Note:
Its best to try and plan ahead here when you create a new set.
hash:net will define how the IP addresses are stored in the set inside the kernel

by specifying ip here, in place of net, it will allow for single IP's and network blocks of IP addresses.
be aware however; adding a netblock of 192.168.1.0/24 will add in 192.168.1.0, 192.168.1.1, 192.168.1.2 etc individually.
You may opt to select ip here in place of net depending on your needs.

The counters option here allows us to keep an eye on the number of hits we've had against an IP address in a similar fashion to how we would see this in iptables list entries ( iptables -L INPUT -v -n )

If items are present in the list but are getting no hits then the IP address could perhaps be flagged for removal from the block list.

Populating the list

We can very easily adapt the wrapper script from earlier to use this new set.

here is an updated version:

#!/bin/bash
for ipaddr in $(./spot-block-ips); do ipset add -exist blocklist ${ipaddr}; done

Depending on the number of IP address you have to hand this may take a moment to complete.

Using the list

Now the list is populated we have to make use of it.
As mentioned earlier we're not paying attention to the service the IP is trying to access anymore, we're just blocking all services for that IP instead.

Run the following to enable the ipset list:

iptables -A INPUT -m set --match-set blocklist src -j REJECT

Done !

The list is now in use on your system and you should be blocking the things you've decided to block! 🎉

Depending on the frequency of a connection attempt you should be able to see any new hits by running:

ipset list blocklist |less

Updating the list

From time to time you may want to add or remove items to the list

you can use the ipset add blocklist <ip> and ipset del blocklist <ip> to add and remove single items on the list.

You can also clean out the list with the flush option

The wrapper script in use here will add IP addresses to the list regardless of if they are present or not but will not remove them.

A fix for this would be to use a temporary list and then switch the lists round.

Example:

#!/bin/bash

## End the script if there's an issue
set -e

## Create a new temp blocklist
ipset create blocklist-tmp

## Add IP addresses to it
for ipaddr in $(./spot-block-ips); do ipset add -exist blocklist-tmp ${ipaddr}; done

## Swap the tmp list in place of the old list 
ipset swap blocklist-tmp blocklist

## Destroy the now old list
ipset destroy blocklist-tmp

This is a better way of performing updates to our lists as there is now no gap between the iptables flush command and the new IP addresses making an appearance.

Dedicated firewalling

You'll correctly ask why dedicated firewalls, Hardware or otherwise, are not in use for this.

While they are already in use for more static rules some service providers some times struggle with the number of entries and frequency of updates to the IP addresses in the system that are put through.

At time of writing there's about 7741 IP entries in the database with would mean multiple entries with cloud providers or many hours of work with others.

Summary

By switching the list of blocked IP addresses out of a multi rule iptables chain in to a hash list stored in the kernel we saved about 1% cpu overhead on the systems its deployed on.

🎉