Once your app is ready, you excitedly deploy it to your favorite Kubernetes cluster, such as Red Hat OpenShift. Inputting the URL into the browser, you hit enter and are greeted with...
Application is not available
The application is currently not serving requests at this endpoint. It may not have been started or is still starting.
Well, that's a disappointment!
Fortunately, there are some steps that we can take to determine why your app isn't available to the world.
This post makes a few assumptions about your app's deployment. In particular:
- You deployed to a Red Hat OpenShift cluster (although this is mostly applicable to any Kubernetes cluster)
- You created a service resource
- You created a route resource, which is specific to OpenShift and provides ingress for HTTP traffic
- You are not using any network policies within your project
Our app deployment
By way of example, let's troubleshoot the OpenShift Hello app, deployed with the following manifest. The key aspects of this manifest are as follows:
- A deployment that creates two pods that we can load balance traffic between
- A service that manages our service endpoints, the two pods the deployment creates
- A route that load balances incoming HTTP requests to our service endpoints
It is essential that the TCP ports are specified correctly. The app uses 8080 for HTTP traffic.
apiVersion: apps/v1
kind: Deployment
metadata:
name: openshift-hello
labels:
app: ocp
spec:
replicas: 2
selector:
matchLabels:
app: ocp
template:
metadata:
labels:
app: ocp
spec:
containers:
- name: openshift-hello
image: openshift/hello-openshift:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
name: openshift-hello-svc
labels:
app: ocp
spec:
selector:
app: ocp
ports:
- port: 8080
protocol: TCP
targetPort: 8080
type: ClusterIP
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
name: hello-openshift
labels:
app: ocp
spec:
host: openshift-hello.apps-crc.testing
port:
targetPort: 8080
tls:
termination: edge
to:
kind: Service
name: openshift-hello-svc
weight: 100
wildcardPolicy: None
And once executed, we see the following resources in our project:
$ oc get all -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/openshift-hello-589f9c7749-rhb2p 1/1 Running 0 11h 10.217.0.232 crc-lgph7-master-0 <none> <none>
pod/openshift-hello-589f9c7749-wbn9l 1/1 Running 0 11h 10.217.0.236 crc-lgph7-master-0 <none> <none>
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/openshift-hello-svc ClusterIP 10.217.5.72 <none> 8080/TCP 11h app=ocp
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/openshift-hello 2/2 2 2 11h openshift-hello openshift/hello-openshift:latest app=ocp
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/openshift-hello-589f9c7749 2 2 2 11h openshift-hello openshift/hello-openshift:latest app=ocp,pod-template-hash=589f9c7749
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/hello-openshift openshift-hello.apps-crc.testing openshift-hello-svc 8080 edge None
Our troubleshooting steps
Given the deployment described in the previous section, now we can dive into troubleshooting. We want to check for connectivity issues between each resource. In particular, we're going to look whether the app responds to an HTTP request.
We're going to answer the following questions:
- Did the app start successfully?
- Can I connect to an app pod through a port forward?
- Can I connect to the service?
- Can I connect to the route?
In all cases, oc commands are executed within the hello-openshift namespace.
Did the app start successfully?
For this, let's look at the logs for our app pods:
$ oc get pods -l app=ocp \
-o jsonpath='{range .items[*]}pods/{.metadata.name}{"\n"}{end}' | \
xargs -L1 -I% bash -c 'echo % && oc logs %'
pods/openshift-hello-589f9c7749-rhb2p
serving on 8888
serving on 8080
Servicing request.
Servicing request.
Servicing request.
Servicing request.
Servicing request.
pods/openshift-hello-589f9c7749-wbn9l
serving on 8888
serving on 8080
Servicing request.
If the app log includes any errors, these are worth investigating further.
Can I connect to an app pod through a port forward?
For this, we can setup a port forward to an app pod and attempt to connect directly to the application, bypassing our ingress configuration.
$ oc port-forward pods/openshift-hello-589f9c7749-rhb2p 8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080
With the forward successfully running, in a different terminal window, use the curl command to test connectivity. (In this example, we're using an http:// URL because our app does not use TLS. Instead, we rely on the OpenShift ingress controller to manage TLS encryption at the edge. This configuration is referred to as edge terminated.)
$ curl -v http://localhost:8080/
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 07 Dec 2022 23:17:43 GMT
< Content-Length: 17
< Content-Type: text/plain; charset=utf-8
<
Hello OpenShift!
* Connection #0 to host localhost left intact
And our app is reachable through the port forward.
Can I connect to the service?
Because our original manifest created a service for this, each app pod with the label app=ocp is included as an endpoint for the openshift-hello-svc service.
By describing the service, we learn its endpoints: 10.217.0.232:8080 and 10.217.0.236:8080. And these endpoints match the IP addresses for our app pods.
$ oc describe services/openshift-hello-svc
Name: openshift-hello-svc
Namespace: hello-openshift
Labels: app=ocp
Annotations: <none>
Selector: app=ocp
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.217.5.72
IPs: 10.217.5.72
Port: <unset> 8080/TCP
TargetPort: 8080/TCP
Endpoints: 10.217.0.232:8080,10.217.0.236:8080
Session Affinity: None
Events: <none>
Or we can look at the endpoint resources in the project. The IP addresses and ports match our openshift-hello-svc service:
$ oc get endpoints
NAME ENDPOINTS AGE
openshift-hello-svc 10.217.0.232:8080,10.217.0.236:8080 12h
For this connection test, let's use a Netshoot container.
kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot
dP dP dP
88 88 88
88d888b. .d8888b. d8888P .d8888b. 88d888b. .d8888b. .d8888b. d8888P
88' `88 88ooood8 88 Y8ooooo. 88' `88 88' `88 88' `88 88
88 88 88. ... 88 88 88 88 88. .88 88. .88 88
dP dP `88888P' dP `88888P' dP dP `88888P' `88888P' dP
Welcome to Netshoot! (github.com/nicolaka/netshoot)
From the Netshoot container, let's confirm that we can access the cluster IP of the service, which internally routes the traffic to one of our app pods.
$ curl -v http://10.217.5.72:8080/
* Trying 10.217.5.72:8080...
* Connected to 10.217.5.72 (10.217.5.72) port 8080 (#0)
> GET / HTTP/1.1
> Host: 10.217.5.72:8080
> User-Agent: curl/7.86.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 09 Dec 2022 03:39:28 GMT
< Content-Length: 17
< Content-Type: text/plain; charset=utf-8
<
Hello OpenShift!
* Connection #0 to host 10.217.5.72 left intact
#
Exit the shell to terminate the Netshoot pod.
Can I connect to the route?
Finally, let's check whether we can connect to the exposed public route of our app. We can describe the route and confirm what the hostname for our app is: openshift-hello.apps-crc.testing.
$ oc describe routes/hello-openshift
Name: hello-openshift
Namespace: hello-openshift
Created: 20 hours ago
Labels: app=ocp
...
Requested Host: openshift-hello.apps-crc.testing
exposed on router default (host router-default.apps-crc.testing) 20 hours ago
Path: <none>
TLS Termination: edge
Insecure Policy: <none>
Endpoint Port: 8080
Service: openshift-hello-svc
Weight: 100 (100%)
Endpoints: 10.217.0.232:8080, 10.217.0.236:8080
In a browser, we can visit https://openshift-hello.apps-crc.testing/ to see if the app loads. Behind the scenes, the ingress controller is handling the TLS termination and load balancing across app endpoints.
If we cannot connect successfully, on OpenShift it's possible to enable logging on the ingress controller to investigate further.
As a cluster administrator, edit the ingress controller configuration:
$ oc -n openshift-ingress-operator edit ingresscontrollers/default
Add the spec.logging.access.* fields as described in the following YAML:
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
name: default
namespace: openshift-ingress-operator
spec:
logging:
access:
destination:
type: Container
httpLogFormat: log_source="haproxy-default" log_type="http" c_ip="%ci" c_port="%cp" req_date="%tr" fe_name_transport="%ft" be_name="%b" server_name="%s" res_time="%TR" tot_wait_q="%Tw" Tc="%Tc" Tr="%Tr" Ta="%Ta" status_code="%ST" bytes_read="%B" bytes_uploaded="%U" captrd_req_cookie="%CC" captrd_res_cookie="%CS" term_state="%tsc" actconn="%ac" feconn="%fc" beconn="%bc" srv_conn="%sc" retries="%rc" srv_queue="%sq" backend_queue="%bq" captrd_req_headers="%hr" captrd_res_headers="%hs" http_request="%r"
logEmptyRequests: Log
After updating the ingress controller configuration, you can tail the haproxy logs on a router pod. To list the available router pods, run the oc get -n openshift-ingress pods command.
$ oc logs -f -n openshift-ingress router-default-<id> log
Ideally, if your app is accessible on the pods that it is running on and through the service, the problem lies with the ingress configuration. Troubleshooting issues with ingress is beyond the scope of this post.
For more information on the OpenShift Ingress Operator, see the official documentation.
While in these examples, the app is reachable at every step, you can use this approach to reveal whether a connectivity issue prevents you from connecting to your app successfully from your exposed route if you are facing an Application is not available error page.
Top comments (0)