Binding privileged port 514 to Logstash 7.10.0

Sometimes there comes a scenario where you need to receive syslog traffic in the default port 514 in your logstash server. But as we know ports in range 1 to 1024 are privileged ports and only the root user can listen to them. By default, logstash daemon starts with its own user, so to resolve our problem we need to follow any below option. My production logstash server is on CentOS 7.

To bind logstash below 1024 ports we have 3 options.

1. run logstash as root.
2. use iptables to forward port 514 to an unprivileged port.
3. use setcap to grant java permission to use privileged ports.

We could run the Logstash daemon through the root user but this process is not recommended and the process is also more complicated in the newer version of logstash so we gonna escape that option. You may prefer the 2nd option which is through iptables but their is limitation. If our logstash server is receiving millions of packets then every packet going through iptables rule will definitely increase our server CPU usages and we may face a performance bottleneck. So, therefore, we will choose the 3rd option in our case.

I will install logstash-7.10.0 through YUM repo which is bundled with JDK 11. That means I don't have to install java manually for my CentOS 7 anymore before logstash installation. Logstash offers architecture-specific downloads that include AdoptOpenJDK 11, the latest long term support (LTS) release of JDK.

Install logstash 7.10.0

Download and install the public signing key:

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo

vim /etc/yum.repos.d/logstash.repo

[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

And your repository is ready for use. You can install it with:

yum install logstash

Now check bundled java version.

/usr/share/logstash/jdk/bin/java -version

openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.8+10, mixed mode)

Add capabilities to the java binary

What is capabilities?
(According to it's man page)
For the purpose of performing permission checks, traditional UNIX implementations distinguish two categories of processes: privileged processes (whose effective user ID is 0, referred to as superuser or root), and unprivileged processes (whose effective UID is nonzero). Privileged processes bypass all kernel permission checks, while unprivileged processes are subject to full permission checking based on the process's credentials (usually: effective UID, effective GID, and supplementary group list).

Starting with kernel 2.2, Linux divides the privileges traditionally associated with superuser into distinct units, known as capabilities, which can be independently enabled and disabled. Capabilities are a per-thread attribute.

CAP_NET_BIND_SERVICE: Bind a socket to Internet domain privileged ports (port numbers less than 1024).

The capabilities are added per file. This is why we need to modify the java binary itself. The capability we need to add is CAP_NET_BIND_SERVICE, which is explicitly defined as the capacity for an executable to bind to a port less than 1024.

First, locate the java path in logstash. It's on /usr/share/logstash/jdk/bin/java

setcap CAP_NET_BIND_SERVICE=+eip /usr/share/logstash/jdk/bin/java

Now check that the capability is added:

getcap /usr/share/logstash/jdk/bin/java

java = cap_net_bind_service+eip

ldd will show shared library dependencies of an executable binary java

ldd /usr/share/logstash/jdk/bin/java

linux-vdso.so.1 =>  (0x00007ffd85fcf000)
libz.so.1 => /lib64/libz.so.1 (0x00007faa3d2c5000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007faa3d0a9000)
libjli.so => /usr/share/logstash/jdk/bin/../lib/jli/libjli.so (0x00007faa3ce98000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007faa3cc94000)
libc.so.6 => /lib64/libc.so.6 (0x00007faa3c8c6000)
/lib64/ld-linux-x86-64.so.2 (0x00007faa3d6de000)

systemctl start logstash

Now you will see error message in /var/log/message saying libjli.so missing library. Java couldn't find that library.

logstash: /usr/share/logstash/jdk/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory.

Above setcap command breaks how java looks for it's library to run. To fix this, we need to symlink the library it’s looking for into /usr/lib and then run ldconfig

ln -s /usr/share/logstash/jdk/lib/jli/libjli.so /usr/lib/

Execute command ldconfig.

ldconfig

Now create a test config file /etc/logstash/conf.d/test.conf listening on port 514.

input {
    syslog {
        port => "514"
    }
}

output {
    stdout { codec => rubydebug }
}

systemctl restart logstash

netstat -plnu

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
udp        0      0 127.0.0.1:323           0.0.0.0:*                           676/chronyd
udp        0      0 0.0.0.0:514             0.0.0.0:*                           26999/java
udp6       0      0 ::1:323                 :::*                                676/chronyd

That's all. We have now successfully bind privileged port 514 to logstash.

In case if you need to remove capability then use setcap command again with the option -ep

setcap cap_net_bind_service=-ep /usr/share/logstash/jdk/bin/java