Have you ever tried to access Jaeger programmatically in OpenShift? It can be a little bit difficult. Sometimes, it is needed because you want, for instance, access to the REST API which is not documented because it is for internal usage. Or you want (as it is my case) to test something from the UI.
Before reading this article
This article is about how to get access to the Jaeger query service when using the “production” deployment strategy and deploying via the Jaeger Operator. Some steps can be the same if you are using other deployment methods. Even, these steps can be not needed if you disabled the security:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: disable-oauth-proxy
spec:
ingress:
security: none
Also, take into account this is for OpenShift. So, if you are using Kubernetes without any security proxy or something, you’ll not need to follow these steps to access the Jaeger query endpoints. Or (if you are using another proxy) you’ll need to configure your credentials to access the API. I’ll assume you’re using the OpenShift Oauth Proxy.
Getting access to the Jaeger query service
Let’s imagine I deployed the following CRD (kubectl create -f simple-prod.yaml) in the israel namespace:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
strategy: production
storage:
type: elasticsearch
elasticsearch:
nodeCount: 1
resources:
requests:
cpu: 200m
memory: 1Gi
limits:
memory: 1Gi
The result of running curl against the created OpenShift route will be a 403 error:
$ curl https://$(kubectl get route simple-prod -o jsonpath='{.spec.host}') -I
HTTP/1.1 403 Forbidden
Set-Cookie: _oauth_proxy=; Path=/; Domain=simple-prod-israel.apps.mycluster.iblancasa.com; Expires=Thu, 05 May 2022 11:09:34 GMT; HttpOnly; Secure; SameSite
Date: Thu, 05 May 2022 12:09:34 GMT
Content-Type: text/html; charset=utf-8
Set-Cookie: 3f978cf9c2d3ee8d70cad1d11aef9dcd=8d2f11be8db90157d1e73d19b515f187; path=/; HttpOnly; Secure; SameSite=None
If you didn’t specify the -I option to receive just the headers, you will receive the full HTML code from the logging webpage. Let’s see how to get the “correct” answer from the server.
The first step is to create a Service Account. From the OpenShift documentation:
A service account is an OpenShift Container Platform account that allows a component to directly access the API.
I’ll name my Service Account as “automation-access”. So, my file sa.yaml contains:
apiVersion: v1
kind: ServiceAccount
metadata:
name: automation-access
And create it in OpenShift using kubectl create -f sa.yaml:
$ kubectl create -f sa.yaml
serviceaccount/automation-access created
Now, we need to create a Cluster Role Binding. From the Kubernetes documentation:
A role binding grants the permissions defined in a role to a user or set of users. It holds a list of subjects (users, groups, or service accounts), and a reference to the role being granted. A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding grants that access cluster-wide.
As a simplification, we can use Cluster Role Bindings to establish relationships between, for instance, a Service Account and Cluster Role. In our case, it would be enough with system:auth-delegator for the jaeger-operator Service Account from the observability namespace and cluster-reader for the Service Account we created before. Our crb.yaml file will be similar to this one:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jaeger-operator-with-auth-delegator
namespace: observability
subjects:
- kind: ServiceAccount
name: jaeger-operator
namespace: observability
roleRef:
kind: ClusterRole
name: system:auth-delegator
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: automation-access-bind
subjects:
- kind: ServiceAccount
name: automation-access
namespace: israel
roleRef:
kind: ClusterRole
name: cluster-reader
apiGroup: rbac.authorization.k8s.io
And apply it to our OpenShift cluster using: kubectl create -f crb.yaml:
$ kubectl create -f crb.yaml
clusterrolebinding.rbac.authorization.k8s.io/jaeger-operator-with-auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/automation-access-bind created
Finally, we need to set some extra options to our Jaeger deployment (jaeger-sar.yaml):
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-prod
spec:
query:
options:
query.bearer-token-propagation: true
ingress:
openshift:
sar: '{"namespace": "israel", "resource": "pods", "verb": "get"}'
delegateUrls: '{"/":{"namespace": "israel", "resource": "pods", "verb": "get"}}'
options:
pass-access-token: true
pass-user-bearer-token: true
scope: "user:info user:check-access"
pass-basic-auth: false
Then, we apply it with kubectl apply -f jaeger-sa.yaml:
$ kubectl apply -f jaeger-sar.yaml
Warning: resource jaegers/simple-prod is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
jaeger.jaegertracing.io/simple-prod configured
Ok! We’re almost there. The next step is to get the token that will allow us to access the Jaeger query service programmatically. To get the token, we need to get the name of the Kubernetes secret to use. We can get that information from the following command:
$ kubectl get sa automation-access -o jsonpath='{.secrets}'
[{"name":"automation-access-token-q2p2x"},{"name":"automation-access-dockercfg-xvzsq"}]
The result is the name of 2 secrets. We’ll use the one called auomation-access-token-<ID>. If you use yq, the command can be:
$ kubectl get sa automation-access -o yaml | yq eval '.secrets[] | select( .name == "*-token-*")'.name
automation-access-token-q2p2x
And, finally, we extract the token from the secret:
$ kubectl get secret automation-access-token-q2p2x -o jsonpath='{.data.token}' | base64 -d
eyJhbGciOiJSUzI1NiIsImtpZCI6Iks3TXcwUWhyb3BVdUVJblhkQmVQT25Fek1KVVlfa2RPV3dDZWpJVXpmekUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJpc3JhZWwiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiYXV0b21hdGlvbi1hY2Nlc3MtdG9rZW4tcTJwMngiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiYXV0b21hdGlvbi1hY2Nlc3MiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI1NGY1ZDE0MC1lMDEwLTQ5NWYtYWE4Yy0wNmQ3ZjIwNjg3OGIiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6aXNyYWVsOmF1dG9tYXRpb24tYWNjZXNzIn0.J5USv-olD1vU0A5tL8TT1SDoH-jf5f58hSIEPnkkiZzTiOjHHx2Vj2v1p0bm26epfTQwLV3TfIsgcsV7tnpC0cR64MmDQe9EuBtNonJqV45nTkSoboWuBfeSxXqjk7Tuj_bcZcutcA0vsVHzAglb-Z0wavx9ETfQJueDWTzF6Cry5HxVxYn6ytogHfbaEgiLN9HNeZBWW4xi7JAzY-2viECrFq1yEIJKzWDihZScFkyX2fu1UDpFLH2ewA8N6bdqSd5FTEna0JS0e8yxQcGo-oXykDW2G_YcLDBSrCpQaji390FgMp_6dEl1yHSazGgCaKwulwOU4Rr6usf807mdBQ
This command gets the token from the secret and decodes it. We need to provide this token in the Authorization header when doing a curl call:
$ export TOKEN=$(kubectl get secret automation-access-token-q2p2x -o jsonpath='{.data.token}' | base64 -d)
$ curl https://$(kubectl get route simple-prod -o jsonpath='{.spec.host}') -I -H "Authorization: Bearer $TOKEN"
HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Date: Thu, 05 May 2022 12:44:06 GMT
Gap-Auth: system:serviceaccount:israel:automation-sa@cluster.local
Gap-Upstream-Address: localhost:16686
Vary: Accept-Encoding
Set-Cookie: 3f978cf9c2d3ee8d70cad1d11aef9dcd=2e6b6fcf2050268c74e1aabcf71e9bab; path=/; HttpOnly; Secure; SameSite=None
Cache-control: private
Thank you for reading! I hope this article helped you! :)
Top comments (0)