How to build NGINX from the source with modules

I recently encountered an issue with service discovery while using Consul and NGINX.

Specifically, I was trying to route traffic to services deployed via Nomad using Consul DNS (service-name.service.consul), but my setup wasn't working as expected.

What was happening? When I tried to DIG into the SRV records, the IP and port were coming.

ubuntu@master:~/nomad$ dig @127.0.0.1 -p 8600 echo-server.com.service.consul SRV

; <<>> DiG 9.18.28-0ubuntu0.20.04.1-Ubuntu <<>> @127.0.0.1 -p 8600 echo-server.com.service.consul SRV
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3407
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 7

;; ANSWER SECTION:
echo-server.com.service.consul. 0 IN SRV    1 1 25268 ac1f1647.addr.dc1.consul.
echo-server.com.service.consul. 0 IN SRV    1 1 24717 ac1f1647.addr.dc1.consul.

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1) (UDP)
;; WHEN: Mon Sep 09 18:53:03 UTC 2024
;; MSG SIZE  rcvd: 352

When I tried redirecting the traffic from nginx to DNS it was not reaching the server. I just got a welcome nginx page.

But when i tried to curl the service that aswell failed. Only when the port was specified it worked.

ubuntu@master:~/nomad$ curl echo-server-com.service.consul:25268
Accessing port 25268, version 65

After some investigation, I realized that service discovery in NGINX Open Source doesn't natively support DNS-based service discovery without additional modules—this functionality is part of NGINX Plus, a paid version.

To enable the same functionality without upgrading to NGINX Plus, I found the nginx-upstream-dynamic-servers module, which allows dynamic upstream management using DNS with Consul. This module, however, is a static module, meaning I needed to rebuild NGINX from source to install it.

This module allows services deployed with Nomad (using dynamic port allocation) to be accessed via Consul DNS (e.g., echo-server.com.consul). Since this module is a static one, it required me to rebuild NGINX from scratch.

Here's how I added the static module to NGINX to achieve DNS-based service discovery, overcoming the limitation of not having NGINX Plus.

Here’s a guide on how I added a static module to NGINX.

NGINX Module Types

• Dynamic Modules: Can be easily added to an existing NGINX installation.
• Static Modules: Require rebuilding NGINX from source.

Since the upstream module is static, I had to rebuild NGINX. I’m currently using NGINX 1.19 because this version is tested with the upstream module.

Step-by-Step Guide

1. Check Existing NGINX Build Files

Before rebuilding, check if you already have NGINX and module files in place:

cd /usr/local/src/nginx-1.19.10
ls

If the necessary files are already present, the process becomes easier. If not, proceed to clone the required source files.

2. Clone the NGINX and Module Repositories

If NGINX or the upstream module isn’t present, download them:

cd /usr/local/src

sudo wget http://nginx.org/download/nginx-1.19.10.tar.gz

sudo tar -zxvf nginx-1.19.10.tar.gz

cd nginx-1.19.10

sudo git clone https://github.com/GUI/nginx-upstream-dynamic-servers.git

Now clone your new module in the same folder so that all the modules are at one place in case the next person wants to add a new module.

After adding adding your module we need to configure the module.

Note: Don't forget to add the existing installed modules else it will break the running infra.

3. Configure NGINX with the Static Module

To add a module during the NGINX build process, use the following configuration command:

sudo ./configure \
    --prefix=/etc/nginx \
    --sbin-path=/usr/sbin/nginx \
    --conf-path=/etc/nginx/nginx.conf \
    --pid-path=/var/run/nginx.pid \
    --lock-path=/var/run/nginx.lock \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_realip_module \
    --with-http_auth_request_module \
    --with-http_v2_module \
    --with-http_dav_module \
    --with-http_slice_module \
    --with-http_addition_module \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_sub_module \
    --with-mail_ssl_module \
    --with-stream_ssl_module \
    --add-module=/usr/local/src/nginx-1.19.10/nginx-upstream-dynamic-servers

Common Configuration Options:

• --prefix=/etc/nginx: Installation directory for Nginx binaries and configuration files.
• --sbin-path=/usr/sbin/nginx: Path to the Nginx binary executable.
• --conf-path=/etc/nginx/nginx.conf: Path to the main Nginx configuration file.
• --pid-path=/var/run/nginx.pid: Path to the Nginx process ID file.
• --lock-path=/var/run/nginx.lock: Path to the Nginx lock file.
• --error-log-path=/var/log/nginx/error.log: Path to the Nginx error log file.
• --http-log-path=/var/log/nginx/access.log: Path to the Nginx access log file.
• --with-http_ssl_module: Enable SSL support.
• --with-http_stub_status_module: Enable the stub_status module for server status monitoring.
• (Add other desired modules with --with-modulename_module)
• Make sure to include the --add-module option for each static module you want to build with NGINX.

Once the configuration is complete, you can proceed to remove the existing NGINX installation and rebuild NGINX with the new module.

4. Backup Your Current NGINX Configuration

Before purging the current NGINX installation, you must back up the existing configuration:

sudo cp -r /etc/nginx ~/nginx-backup-1

5. Remove Existing NGINX Installation

If NGINX is already installed, remove it completely:

# Stop nginx service if running
sudo systemctl stop nginx

# Remove NGINX installed from packages (if any)
sudo apt-get remove --purge nginx nginx-common nginx-full

# Remove NGINX binaries from /usr/local if built from source
# Remove any remaining nginx configuration or directories
sudo rm -rf /usr/local/nginx /etc/nginx /var/log/nginx /var/run/nginx.pid

6. Install Required Dependencies

To compile NGINX from source, install the necessary dependencies:

sudo apt-get update
sudo apt-get install build-essential libpcre3 libpcre3-dev zlib1g zlib1g-dev libssl-dev

7. Build and Install NGINX

Once dependencies are installed, build and install NGINX:

sudo make
sudo make install

8. Verify the Installed Modules

Check if the modules have been installed correctly:

/usr/sbin/nginx -V

This should display the version of NGINX along with all configured modules.

ubuntu@master:~$ s nginx -V
nginx version: nginx/1.19.10
built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
built with OpenSSL 1.1.1f 31 Mar 2020
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-http_slice_module --with-http_addition_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_sub_module --with-mail_ssl_module --with-stream_ssl_module --add-module=/usr/local/src/nginx-1.19.10/nginx-upstream-dynamic-servers

9. Restore NGINX Configuration

Copy your backed-up configuration to the new installation:

# sudo cp -r ~/nginx-backup/* /etc/nginx/

# Create new directory for sites-enabled
sudo mkdir /etc/nginx/sites-enabled

# Below command will create a symbolic link for each site in the sites-available directory to the sites-enabled directory.
for site in /etc/nginx/sites-available/*; do
    sudo ln -s "$site" /etc/nginx/sites-enabled/
done

Nginx conf will be available at:

sudo cat /etc/nginx/nginx.conf

Logs will be available at:

# Error log
sudo tail -f /var/log/nginx/error.log

# Access log
sudo tail -f /var/log/nginx/access.log

10. Test and Reload NGINX

After restoring the configuration, test and reload NGINX:

# Test configuration
sudo nginx -t

# Reload NGINX
sudo systemctl reload nginx

11. Set NGINX as a System Service (Optional)

If NGINX is not running as a service, create a systemd service file:

sudo vim /lib/systemd/system/nginx.service

Add the following content:

[Unit]
Description=The NGINX HTTP and reverse proxy server
After=network.target

[Service]
Type=forking
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/usr/sbin/nginx -s reload
ExecStop=/usr/sbin/nginx -s stop
PIDFile=/var/run/nginx.pid
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Enable and start the NGINX service:

sudo systemctl enable nginx
sudo systemctl start nginx

Conclusion

That’s how you add a static module to NGINX!

It’s important to back up your configurations, purge the old installation, and build from source when using static modules like the upstream dynamic servers module.

Once NGINX is built, you can verify the installation and restore your configuration seamlessly.

Happy NGINXing!