Add Custom Headers to Outgoing request from applications deployed on Kubernetes

Introduction

Kubernetes in todays world is the defacto platform for deploying containerized applications including microservices because of the various benefits this platform offers including automated deployment, scaling security, resiliency, load balancing and self healing along with many others.
This article will explain how we can automate the insertion of custom headers in outgoing requests from the applications that are deployed in Kubernetes with Istio Service Mesh.

Communication between Pods using Kubernetes Service

Each Pod deployed gets its onw unique internal IP address in the cluster. When applications running inside pods wants to connect to other pods, they can connect using the podname or pod ipaddress.
In case there are multiple replicas of an application running leading to multiple pods then kubernetes service is used to reach the destination.
Below diagram shows the pod to pod communication through kubernetes service.

Istio Service Mesh

Service Mesh is an infrastructure layer that handles communication and networking between services in an application. It handles traffic, monitors performance and enforces different policies.
Some of the benefits of using service mesh includes Service Discovery, Traffic control, Observability, Security and Compliance
Istio is an open source service mesh which offers all of the above mentioned features of a typical service mesh. You can read about it more on this link Istio / Architecture
Istio uses an extended version of the Envoy proxy. Envoy is a high-performance proxy developed in C++ to mediate all inbound and outbound traffic for all services in the service mesh. Envoy proxies are the only Istio components that interact with data plane traffic. Istio deploys Envoy proxies as side car containers to workloads logically augmenting the services with many built in features of Envoy.

Below diagram shows the communication across applications when Istio Service Mesh is used.

Insert Custom Headers to Outgoing Requests

Envoy proxy intercepts all the incoming and outgoing requests within the pod to/from all the containers that are running in the pod. This configuration can be overridden to implement various functionalities like logging, auditing, authentication, modifying requests etc. Envoy filters are additively applied to the proxy configuration. We can add multiple filters with different functionalities.
Here we will see how can we extend this functionality to add custom headers to outgoing requests from the pods.

We need to create the EnvoyFilter kubernetes Istio CRD and override the function which is invoked when a request is intercepted by the proxy.
The following code enables Envoys Lua filter for all outbound HTTP calls on service port 8080 of the myapplication service pod with labels "app: myapplication". The Lua filter adds a custom header key and value to the request handle headers object.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: customheaders-filter
spec:
  workloadSelector:
    labels:
      app: myapplication
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        portNumber: 8080
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
          inlineCode: |

              function envoy_on_request(request_handle)
                request_handle:headers():add( "my-custom-header", "my-custom-header-value )
              end

We can also extend this function to add headers only if the destination host is expected one.

  function envoy_on_request(request_handle)
    local destinationHost = request_handle:headers():get(":authority")
    if destinationHost ~= nil and destinationHost ~= '' then
      request_handle:logInfo(" found authority  :"..destinationHost)
    else
      request_handle:logInfo(" no authority found")
      destinationHost = ''
    end

    request_handle:logInfo("destination host -> "..destinationHost) 
    request_handle:logInfo("configured host -> {{ $customHeadersByHost.host }}")

    if destinationHost:find({{ $customHeadersByHost.host | quote }}, 0, true) ~= nil then
      {{- range $header := $customHeadersByHost.headers }}
      request_handle:headers():add({{ $header.name | quote }} , {{ $header.value | quote }} )
      {{- end }}
      request_handle:logInfo(" added headers for host - {{ $customHeadersByHost.host }}")
    else
      request_handle:logInfo(" destination does not match configured host ")
    end

  end

Inorder to view the envoy-filter proxy logs from the pods we need to add the below annotations to the Pod definition if logging is restricted at istio system level.

apiVersion: apps/v1
kind: Deployment
metadata:
  sidecar.istio.io/logLevel: info
spec:
  ..
  ..

To view the logs that are added in the on request function above, we can view them using kubectl command

kubectl logs -n mynamespace myappliation-pod -c istio-proxy