Anthos is a software offering from Google, using which we can build kubernetes clusters on nodes both on and off cloud. In this post, we would launch an EC2 instance in AWS and build a single node kubernetes cluster on it with Anthos.
Let's get started.
AWS cloud shell
Login to the AWS console, and access the cloud shell from the top bar. You should see a prompt like below.
[cloudshell-user@ip-10-0-89-211 ~]$
EC2
Create an EC2 instance with other relevant components with these commands. Please refer to this post for explanation on what each of these commands does.
$ mkdir ~/.aws
$ cat ~/.aws/config <<EOF
[default]
region=ap-south-1
EOF
$ export CIDR_BLOCK="10.10.10.0/28"
$ aws ec2 create-vpc --cidr-block $CIDR_BLOCK
$ export ANTHOS_VPC_ID=$(aws ec2 describe-vpcs | jq -r '.Vpcs[] | select(.CidrBlock == env.CIDR_BLOCK) | .VpcId')
$ aws ec2 create-internet-gateway --tag-specifications 'ResourceType=internet-gateway,Tags=[{Key=Name,Value=anthos-igw}]'
$ export ANTHOS_IGW_ID=$(aws ec2 describe-internet-gateways --filters Name=tag:Name,Values=anthos-igw --query "InternetGateways[*].InternetGatewayId" --output text)
$ aws ec2 attach-internet-gateway --internet-gateway-id $ANTHOS_IGW_ID --vpc-id $ANTHOS_VPC_ID
$ export ANTHOS_RTB_ID=$(aws ec2 describe-route-tables | jq -r '.RouteTables[] | select(.VpcId == env.ANTHOS_VPC_ID) | .RouteTableId')
$ aws ec2 create-route --route-table-id $ANTHOS_RTB_ID --destination-cidr-block 0.0.0.0/0 --gateway-id $ANTHOS_IGW_ID
$ aws ec2 create-subnet --cidr-block $CIDR_BLOCK --vpc-id $ANTHOS_VPC_ID
$ export ANTHOS_SUBNET_ID=$(aws ec2 describe-subnets | jq -r '.Subnets[] | select(.CidrBlock == env.CIDR_BLOCK) | .SubnetId')
$ export ANTHOS_AVAILABILITY_ZONE=$(aws ec2 describe-subnets | jq -r '.Subnets[] | select(.CidrBlock == env.CIDR_BLOCK) | .AvailabilityZone')
$ aws ec2 create-security-group --group-name anthos-sg --description "anthos security group" --vpc-id $ANTHOS_VPC_ID
$ export ANTHOS_SG_ID=$(aws ec2 describe-security-groups | jq -r '.SecurityGroups[] | select(.GroupName == "anthos-sg") | .GroupId')
$ aws ec2 describe-instance-types | jq '.InstanceTypes[] | select(.MemoryInfo.SizeInMiB == 7680) | (.InstanceType, .VCpuInfo.DefaultVCpus)'
"c4.xlarge"
4
$ aws ec2 describe-instance-types | jq '.InstanceTypes[] | select(.MemoryInfo.SizeInMiB == 8192) | select (.VCpuInfo.DefaultVCpus == 2) | .InstanceType' | sort
"m4.large"
"m5ad.large"
"m5a.large"
"m5d.large"
"m5.large"
"m6a.large"
"m6gd.large"
"m6g.large"
"m6i.large"
"t2.large"
"t3a.large"
"t3.large"
"t4g.large"
$ aws ec2 describe-instance-type-offerings --location-type availability-zone | jq '.InstanceTypeOfferings[] | select(.Location == env.ANTHOS_AVAILABILITY_ZONE) | .InstanceType' | grep t2.large
"t2.large"
$ aws ec2 create-key-pair --key-name anthosKeyPair --query 'KeyMaterial' --output text > anthosKeyPair.pem
$ mkdir .ssh
$ mv anthosKeyPair.pem ~/.ssh/
$ aws ec2 run-instances --image-id ami-0bba4b75264ecbfbd --count 1 --instance-type t2.large --key-name anthosKeyPair --security-group-ids $ANTHOS_SG_ID --subnet-id $ANTHOS_SUBNET_ID --associate-public-ip-address --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=200}'
The instance is now created. We can give it a name.
$ ANTHOS_INSTANCE_ID=$(aws ec2 describe-instances | jq -r '.Reservations[] | .Instances[] | select(.SubnetId==env.ANTHOS_SUBNET_ID) | .InstanceId')
$ aws ec2 create-tags --resources $ANTHOS_INSTANCE_ID --tags Key=Name,Value=anthos-node
SSH
In order to SSH from the cloud shell to the Anthos instance, we first need to obtain the public IP of the cloudshell and add a rule in the security group to allow SSH access from that.
$ export MY_PUBLIC_IP=$(curl ifconfig.me --silent)
$ aws ec2 authorize-security-group-ingress --group-id $ANTHOS_SG_ID --protocol tcp --port 22 --cidr $MY_PUBLIC_IP/32
Copy the SSH key pair to the instance as it's needed in the Anthos cluster config.
$ export ANTHOS_INSTANCE_IP=$(aws ec2 describe-instances --filter Name=tag:Name,Values=anthos-node --query "Reservations[*].Instances[*].PublicIpAddress" --output text)
$ scp -i ~/.ssh/anthosKeyPair.pem ~/.ssh/anthosKeyPair.pem ubuntu@$ANTHOS_INSTANCE_IP:~/.ssh/anthosKeyPair.pem
SSH into the instance.
$ ssh -i ~/.ssh/anthosKeyPair.pem ubuntu@$ANTHOS_INSTANCE_IP
Install gcloud
Install the gcloud cli on the Anthos instance.
sudo apt-get install apt-transport-https ca-certificates gnupg -y
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -
sudo apt-get update -y && sudo apt-get install google-cloud-cli -y
Login to gcloud
This step is optional. Login to the gcloud cli with your account if you are going to create a service account yourself from the CLI.
gcloud auth login
Authenticaion
Create a service account in GCP and give it the following roles.
roles/gkehub.connect
roles/gkehub.admin
roles/logging.logWriter
roles/monitoring.metricWriter
roles/monitoring.dashboardEditor
roles/stackdriver.resourceMetadata.writer
roles/opsconfigmonitoring.resourceMetadata.writer
If you are using the gcloud CLI, you can create a service account and bind the roles with the following command.
gcloud iam service-accounts create <service-account-name>
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member=<service-account-client-email> \
--role=<role> \
--no-user-output-enabled
Create a key for the service account and copy it's credentials.
gcloud iam service-accounts keys create <key-file-path> \
--iam-account=${service-account-name}@${PROJECT_ID}.iam.gserviceaccount.com
Setup the credentials in the instance, activate the service account and set the project id.
$ mkdir .gcloud
$ export PROJECT_ID=<project_id>
$ cat > .gcloud/keyfile.json << EOF
{
"type": "service_account",
"project_id": $PROJECT_ID,
"private_key_id": "<private_key_id>",
"private_key": <private_key>,
"client_email": <client_email>,
"client_id": <client_id>,
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": <client_x509_cert_url>
}
EOF
$ gcloud auth activate-service-account <client_email> --key-file .gcloud/keyfile.json
$ export GOOGLE_APPLICATION_CREDENTIALS='/home/ubuntu/.gcloud/keyfile.json'
$ gloud config set project $PROJECT_ID
APIs
Enable the services.
gcloud services enable \
anthos.googleapis.com \
anthosaudit.googleapis.com \
anthosgke.googleapis.com \
cloudresourcemanager.googleapis.com \
container.googleapis.com \
gkeconnect.googleapis.com \
gkehub.googleapis.com \
serviceusage.googleapis.com \
stackdriver.googleapis.com \
monitoring.googleapis.com \
logging.googleapis.com \
opsconfigmonitoring.googleapis.com
Other tools
Install kubectl, bmctl and docker.
curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
chmod +x kubectl
sudo mv kubectl /usr/local/sbin/
gsutil cp gs://anthos-baremetal-release/bmctl/1.13.0/linux-amd64/bmctl .
chmod a+x bmctl
sudo mv bmctl /usr/local/sbin/
curl -O https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz
tar xzvf docker-20.10.9.tgz
sudo cp docker/* /usr/bin/
rm -rf docker*
sudo groupadd docker
sudo usermod -aG docker $USER
sudo dockerd&
newgrp docker
VxLAN
Set up vxlan with the IP 10.200.2/24.
sudo ip link add vxlan0 type vxlan id 42 dev eth0 dstport 0
sudo ip addr add 10.200.0.2/24 dev vxlan0
sudo ip link set up dev vxlan0
Cluster config
Create the Anthos cluster config with bmctl.
export CLUSTER_ID=anthos-aws
bmctl create config -c $CLUSTER_ID
Change the config.
$ cat > bmctl-workspace/${CLUSTER_ID}/${CLUSTER_ID}.yaml << EOF
---
gcrKeyPath: /home/ubuntu/.gcloud/keyfile.json
sshPrivateKeyPath: /home/ubuntu/.ssh/anthosKeyPair.pem
gkeConnectAgentServiceAccountKeyPath: /home/ubuntu/.gcloud/keyfile.json
gkeConnectRegisterServiceAccountKeyPath: /home/ubuntu/.gcloud/keyfile.json
cloudOperationsServiceAccountKeyPath: /home/ubuntu/.gcloud/keyfile.json
---
apiVersion: v1
kind: Namespace
metadata:
name: cluster-${CLUSTER_ID}
---
apiVersion: baremetal.cluster.gke.io/v1
kind: Cluster
metadata:
name: ${CLUSTER_ID}
namespace: cluster-${CLUSTER_ID}
spec:
profile: edge
type: standalone
anthosBareMetalVersion: 1.13.0
gkeConnect:
projectID: $PROJECT_ID
controlPlane:
nodePoolSpec:
clusterName: ${CLUSTER_ID}
nodes:
- address: 10.200.0.2
clusterNetwork:
pods:
cidrBlocks:
- 192.168.0.0/16
services:
cidrBlocks:
- 172.26.232.0/24
loadBalancer:
mode: bundled
ports:
controlPlaneLBPort: 443
vips:
controlPlaneVIP: 10.200.0.49
ingressVIP: 10.200.0.50
addressPools:
- name: pool1
addresses:
- 10.200.0.50-10.200.0.70
clusterOperations:
location: asia-south1
projectID: $PROJECT_ID
storage:
lvpNodeMounts:
path: /mnt/localpv-disk
storageClassName: node-disk
lvpShare:
numPVUnderSharedPath: 5
path: /mnt/localpv-share
storageClassName: local-shared
nodeConfig:
podDensity:
maxPodsPerNode: 64
nodeAccess:
loginUser: ubuntu
EOF
Cluster creation
Create the cluster.
$ bmctl create cluster -c ${CLUSTER_ID}
The above command should take some time and once it's successful, the kubernetes cluster should be ready.
$ export KUBECONFIG=bmctl-workspace/${CLUSTER_ID}/${CLUSTER_ID}-kubeconfig
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-0-9 Ready control-plane,master 153m v1.24.2-gke.1900
The cluster should appear on the Anthos clusters plage on GCP.
Run workloads
Test with a sample nginx deployment.
$ cat > deploy.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
selector:
matchLabels:
app: nginx
EOF
$ kubectl create -f deploy.yaml
$ kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx 1/1 1 1 65s
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-8f458dc5b-jvrbb 1/1 Running 0 68s
Expose this deployment with a service.
$ cat > svc.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
app: nginx
ports:
- port: 8080
targetPort: 80
name: web-server
EOF
$ kubectl create -f svc.yaml
$ kubectl get svc nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx ClusterIP 172.26.232.21 <none> 8080/TCP 106s
$ kubectl get ep nginx
NAME ENDPOINTS AGE
nginx 192.168.0.48:80 2m3s
Try to curl the service IP and see if it works.
$ curl 172.26.232.21:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Reset the cluster
Finally reset the cluster when you no longer need it.
bmctl reset cluster -c ${CLUSTER_ID}
Thus, we have launched a single node kubernetes cluster with Anthos on AWS, and tested it by running an nginx service.
Thanks for reading!!!
Latest comments (0)