Unlocking 100% of NGINX Ingress Controller

Hey folks 🙋🏻‍♂️

Ingress Controller is that to the Kubernetes pods what NGINX is to backend servers.

One of the most common uses of an NGINX web server is load balancing. NGINX decides which server processes the external request based on configurations written in nginx configuration file. Similarly, an Ingress Controller does the routing of external requests to pods in a Kubernetes namespace by reading the routing rules mentioned in the ingress yaml file.

Introduction 🦄

With the ever-growing product and technical requirements, one may need to do complicated stuff or add some custom logic in NGINX configuration. There might be a use case where such custom configuration is required to be done at the Ingress level. We can tune Ingress behavior using NGINX directives that are applied via Kubernetes annotations for that Ingress resource.

But not all NGINX directives/configurations have a respective annotation in NGINX Ingress Controller.
I have done some following NGINX configurations that were not available as annotations in NGINX Ingress Controller.

1. Manipulate request or response headers
2. Add if conditions / Comparison of a variable
3. Write multiple rewrite rules for request URI
4. Set/Read variables
5. Use map module
6. Implement request authorization via sub-request (auth-request)

This blog will discuss how we can use almost all NGINX directives to heavily customize the behavior of the Ingress.

Back to Basics ⚡️

A typical NGINX configuration looks like,

http {
  # http block
  server {
    # server block
    location {
      # location block
    }
  }
}

There are 3 blocks in NGINX,

1. http block: The http block includes directives for web traffic handling, which are generally known as universal.
2. server block: The server block sets the configuration for a virtual server. A single http block can house multiple server blocks helping us to encapsulate configuration detail and host more than one domain on a single NGINX server.
3. location block: The location block lives within a server block and is used to define how NGINX should handle requests for different resources and URIs for the parent server.

Snippet Annotation 🖋

Advanced Configuration can be done using snippet annotations. There are some disadvantages of using the snippet explained here. There are 4 kinds of snippet annotation,

Ignoring location-snippet since it can be replaced by configuration-sippet.

1. server snippet: As we discussed above location block lives within a server block in NGINX. Similarly, we can write location blocks inside the server snippet. nginx.ingress.kubernetes.io/server-snippet Sets a custom snippet in the server context.

Some scenarios we can use server-snippet are,

Restricting URIs

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: nginx-configuration-snippet
    annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /$2
        nginx.ingress.kubernetes.io/server-snippet: |
          location ~* "^/base/path/v1/api/update" {
              deny all;
              return 403;
            }
spec:
  rules:
    - http:
        paths:
          - path: /base/path(/|$)(.*)
            backend:
              serviceName: foo-service
              servicePort: 80

Multiple rewrite rules

We can have multiple paths but only one rewrite rule in an Ingress yaml file.
The regular expression that will be used to match against the incoming request URI is written against the path key whereas the replacement string to be used to change the requested URI is mentioned in the rewrite-target annotation.
In order to have multiple, we have to create multiple ingress resources or we can take advantage of server-snippet.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: nginx-configuration-snippet
    annotations:
        nginx.ingress.kubernetes.io/server-snippet: |
          location ~* "^/base/path/.*" {
              rewrite /base/path/(v4|v5)(/|$)(.*) /v5/$2;
              rewrite /base/path/(v2|v3)(/|$)(.*) /v3/$2;
              rewrite /base/path(/|$)(.*) /v1/$2;
            }
spec:
  rules:
    - http:
        paths:
          - path: /base/path(/|$)(.*)
            backend:
              serviceName: foo-service
              servicePort: 80

In this case,

• /base/path/v1/foo-api → /foo-api?_version=v1
• /base/path/foo-api → /foo-api

Play with variables

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: nginx-configuration-snippet
    annotations:
        nginx.ingress.kubernetes.io/server-snippet: |
          location ~* "^/base/path/v1/api" {
              set $var $request_method;
              if ($request_method =  "POST") {
                return 403;
              } 
          }
spec:
  rules:
    - http:
        paths:
          - path: /base/path(/|$)(.*)
            backend:
              serviceName: foo-service
              servicePort: 80

2. configuration-snippet: The configuration-snippet adds custom configurations to locations. It is used when location block directives that can't be used in the server block need to be configured for all location blocks. Some scenarios we can use configuration-snippet are,

Add, delete or modify request and response headers

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
    name: nginx-configuration-snippet
    annotations:
        nginx.ingress.kubernetes.io/server-snippet: |
          location ~* "^/base/path/v1/api" {
              more_set_headers 'Request-Id: $req_id';
              more_clear_headers 'X-Hidden-*';
              if($http_custom_header = "foo") {
                more_set_headers '$http_custom_header: bar';
              }
          }
spec:
  rules:
    - http:
        paths:
          - path: /base/path(/|$)(.*)
            backend:
              serviceName: foo-service
              servicePort: 80

Edit 1: We have realized that any header manipulation should be done in configuration-snippet.

nginx.ingress.kubernetes.io/configuration-snippet: |
      if ($request_uri ~* "^/base/path/v1/api") {
          more_set_headers 'Request-Id: $req_id';
              more_clear_headers 'X-Hidden-*';
              if($http_custom_header = "foo") {
                more_set_headers '$http_custom_header: bar';
              }
      }

3. auth-snippet: Though we have an auth-url annotation to send a sub-request to the authentication service to authenticate an external request it takes an auth-snippet to customize this sub-request.

auth block is a location block that proxies the auth sub-request.

We can add headers or rewrite sub request URIs or do other custom configuration before proxying the sub-request to auth server.

4. http-snippet: Adds custom configuration to the http section of the N configuration. What makes this snippet an exception is that we need to inject it via Configmap for it to work properly You may get errors like,

Error: exit status 1
2018/10/16 07:45:49 [emerg] 470#470: "tcp_nopush" directive is duplicate in 
/tmp/nginx-cfg468835321:245
nginx: [emerg] "tcp_nopush" directive is duplicate in /tmp/nginx-cfg468835321:245
nginx: configuration file /tmp/nginx-cfg468835321 test failed

Error: exit status 1
2022/08/04 11:28:25 [emerg] 263#263: unknown "name" variable
nginx: [emerg] unknown "name" variable
nginx: configuration file /tmp/nginx/nginx-cfg3947851792 test failed

Some scenarios we can use configuration-snippet are,

Use map module

As discussed above to configure http-snippet in ConfigMap yaml,

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  http-snippet: |
    map $http_custom_header $custom_var {
      "foo" 0;
      "bar" 1;
    }

Make sure namespace: ingress-nginx since it's the default namespace for the Ingress Controller in k8s.

And that was the blog! I will keep adding new use cases if I encounter any in future.

Thanks for reading my blog.

👋🏻👋🏻👋🏻👋🏻👋🏻👋🏻👋🏻👋🏻