Configure Nginx Ingress Controller to use JSON log format

#kubernetes #nginx #devops #logging

Problem

I'm running Nginx Ingress Controller deployed on GKE and I want to structure my web server logs so I can create metrics and alerts in Google Cloud Stackdriver.

References

Solution

If you are using GKE, there is already a Fluentd DaemonSet that forwards all container logs to Stackdriver.

All you need to do is ensure that Nginx Ingress Controller's ConfigMap has the following log-format-upstream settings:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress-controller
  labels:
    app: nginx-ingress
data:
  log-format-escape-json: "true"
  log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
    "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
    "$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
    "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
    "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
    "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
    "protocol":"$server_protocol"}}'

If you are deploying Nginx using the Helm Chart, the values file will look something like this:

controller:
  config:
    log-format-escape-json: "true"
    log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
    "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
    "$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
    "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
    "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
    "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
    "protocol":"$server_protocol"}}'

Top comments (4)

Ian Ker-Seymer • Feb 27 '20 • Edited

Is this properly indexed by Stackdriver? From reading the documentation, it seems that Stackdriver wants the HTTP info nested in the httpRequest key, and IIRC, JSON logs end up in the jsonPayload key for k8s containers in Stackdriver. Or is Stackdriver smart enough to figure out it should be in the httpRequest key?

UPDATE:

So I think I figured this out, the JSON should be nested in an httpRequest key. Here's the working version:

log-format-escape-json: "true"
log-format-upstream: log-format-upstream: '{"timestamp": "$time_iso8601", "requestID": "$req_id", "proxyUpstreamName":
    "$proxy_upstream_name", "proxyAlternativeUpstreamName": "$proxy_alternative_upstream_name","upstreamStatus":
    "$upstream_status", "upstreamAddr": "$upstream_addr","httpRequest":{"requestMethod":
    "$request_method", "requestUrl": "$host$request_uri", "status": $status,"requestSize":
    "$request_length", "responseSize": "$upstream_response_length", "userAgent": "$http_user_agent",
    "remoteIp": "$remote_addr", "referer": "$http_referer", "latency": "$upstream_response_time s",
    "protocol":"$server_protocol"}}'