I was facing issues when I performed kubectl with exec or port-forward option on my Rancher clusters that used EKS and ALB, it was giving me this error
kubectl exec -it app -- bash
Error from server (BadRequest): Upgrade request required
exec and port forward are using SPDY protocol and ALB does not support it.
The HTTPS request is going from the user to ALB, then SSL is terminated on the ALB, and the request is forwarded to the Nginx controller service after that forward to the rancher service.
See the part 1 to setup Rancher on EKS and ALB
After that, you need to do the following:
1- Install Nginx Ingress Controller
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
Kubernetes v1.16+ use version 3.x.x
helm upgrade --install nginx ingress-nginx/ingress-nginx --namespace cattle-system --set controller.service.type=NodePort --set controller.service.targetPorts.https=http --set-string controller.config.use-forwarded-headers="true" --version 3.12.0
Kubernetes v1.19+ use version 4.x.x
helm upgrade --install nginx ingress-nginx/ingress-nginx --namespace cattle-system --set controller.service.type=NodePort --set controller.service.targetPorts.https=http --set-string controller.config.use-forwarded-headers="true" --version 4.2.3
2- Edit the rancher Ingress
kubectl edit ingress -n cattle-system rancher
Change the host and name inside spec with the following:
spec:
rules:
- host: '*.example.com'
http:
paths:
- backend:
service:
name: nginx-ingress-nginx-controller
port:
number: 80
pathType: ImplementationSpecific
3- Create a new ingress with the following:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
name: rancher-exec
namespace: cattle-system
spec:
rules:
- host: 'rancher.example.com'
http:
paths:
- backend:
service:
name: rancher
port:
number: 80
pathType: ImplementationSpecific
You need to add ingressClassName: nginx only for rancher-exec
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
name: rancher-exec
namespace: cattle-system
spec:
ingressClassName: nginx
rules:
- host: 'rancher.example.com'
http:
paths:
- backend:
service:
name: rancher
port:
number: 80
pathType: ImplementationSpecific
Note: It will remove the ALB if you add ingressClassName: nginx to rancher ingress
Test the kubectl exec and port-forward
kubectl create deployment nginx2 --image nginx:alpine
kubectl expose deployment nginx2 --port=80
kubectl exec -it nginx2-XXXXX -- sh
kubectl port-forward service/nginx2 --address 0.0.0.0 80:80
Top comments (1)
Great article!
I encountered another problem after implementing your solution, the terminal session in Rancher was closed after about 60 seconds.
The reason was that the nginx controller config was set to the default of 60 seconds.
to solve this add to the nginx helm command:
--set-string controller.config.proxy-connect-timeout: "1800" --set-string controller.config.proxy-read-timeout="1800"
This will increase the terminal session to half an hour (1800 seconds).