DEV Community

timtsoitt
timtsoitt

Posted on • Updated on

Crossplane is better than Terraform in K8S world

TL;DR

Crossplane is a solution you should consider when your infrastructure is to serve your k8s applications.

Story

I have been using Terraform for a very long time. It is simple to use with a very huge community support. However every solution has its shortcomings. Terraform is not really a nice solution when you have to work with k8s.

My AWS infrastructure is closely related to my K8S applications. Maybe I have to upload objects to S3, or store my data in RDS. If you are using AWS EKS, you should be very familiar with IRSA (IAM roles for service accounts) feature, it grants your service accounts with AWS permissions. Then you pods can reference these service accounts and interact with AWS APIs.

So IAM roles are AWS stuffs, and service accounts are K8S stuffs. You have to create both IAM roles and service accounts using some methods.

I can use Terraform to create both IAM roles and service accounts but it is very operational unfriendly. K8s deployments not just only have service accounts need to be applied. How about using Terraform to deploy all AWS infrastructure and k8s manifests? IMO, DO NOT DO IT. Argo CD, Flux CD or any GitOps tools are much better than Terraform.

Never solve a small problem by bringing another big trouble.

Now back to my question, how I should integrate IAM roles with service accounts? How about doing the opposite, use k8s to provision IAM roles?

And then Crossplane comes into my eyes. Simply speaking, Crossplane is a k8s style of Terraform.

Tutorial

Now I am going to show you how to use Crossplane.

Prerequisite

  • You need to have an admin privilege in your AWS account.
  • A k8s cluster

1. Define your variables

Specify any value you like.

EKS_CLUSTER_NAME=""
AWS_REGION=""
AWS_IAM_ROLE_NAME="${EKS_CLUSTER_NAME}-crossplane-controller"
Enter fullscreen mode Exit fullscreen mode

2. Install Crossplane using helm charts

Everyone loves helm chart :)

helm repo add crossplane https://charts.crossplane.io/master/
helm install --create-namespace --namespace crossplane-system crossplane crossplane/crossplane --version 1.9.0-rc.0.9.g243f1f47 
Enter fullscreen mode Exit fullscreen mode

3. Install AWS provider

Crossplane can provision infrastructure in many platforms. Let say if you want to deploy to Azure, you can install Azure provider.

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: aws-config
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${AWS_IAM_ROLE_NAME}
spec:
  podSecurityContext:
    fsGroup: 2000
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: crossplane/provider-aws:v0.27.0
  controllerConfigRef:
    name: aws-config
EOF
Enter fullscreen mode Exit fullscreen mode

4. Create IAM role for IRSA

Again it is a Chicken or the egg problem. Crossplane controller needs to have permissions to provision AWS resources. So we need to manually provision a IAM role for once.

SERVICE_ACCOUNT_NAME=$(kubectl get providers.pkg.crossplane.io provider-aws -o jsonpath="{.status.currentRevision}")
OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:crossplane-system:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json

aws iam create-role \
    --role-name "${IAM_ROLE_NAME}" \
    --assume-role-policy-document file://trust.json \
    --description "IAM role for Crossplane provider-aws"

aws iam attach-role-policy --role-name "${IAM_ROLE_NAME}" --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess

rm trust.json

cat <<EOF | kubectl apply -f -
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: aws-provider
spec:
  credentials:
    source: InjectedIdentity
EOF
Enter fullscreen mode Exit fullscreen mode

6. Try it out

Let try to deploy something.

apiVersion: iam.aws.crossplane.io/v1beta1
kind: Role
metadata:
  name: crossplane-sample-role
  annotations:
spec:
  deletionPolicy: Delete
  forProvider:
      description: "A role created by Crossplane"
      assumeRolePolicyDocument: |
        {
          "Version": "2012-10-17",
          "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "Service": "ec2.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
              }
          ]
        }
  providerConfigRef:
    name: aws-provider

---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: Policy
metadata:
  name: crossplane-sample-policy
spec:
  deletionPolicy: Delete
  forProvider:
    name: crossplane-sample-policy
    description: A policy created by Crossplane
    document: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster"
            ],
            "Resource": "*"
          }
        ]
      }
  providerConfigRef:
    name: aws-provider
---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: RolePolicyAttachment
metadata:
  name: crossplane-sample-role-policy-attachment
spec:
  deletionPolicy: Delete
  forProvider:
    roleNameRef:
      name: crossplane-sample-role
    policyArnRef: 
      name: crossplane-sample-policy
  providerConfigRef:
    name: aws-provider
Enter fullscreen mode Exit fullscreen mode

6. Cleanup

Always a good practice to do cleanup.

helm uninstall crossplane --namespace crossplane-system
kubectl delete ns crossplane-system
kubectl get crd -o name | grep crossplane.io | xargs kubectl delete

aws iam detach-role-policy --role-name ${AWS_IAM_ROLE_NAME} --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess
aws iam delete-role --role-name "${AWS_IAM_ROLE_NAME}"
Enter fullscreen mode Exit fullscreen mode

Discussion

Imagine you need to deploy an application that needs an ALB, RDS. Now you can package all the things using k8s manifests. No more Terraform is involved and much less management overheads.

Take a step forward, you can use Crossplane to replace Terraform to provision any infrastructure. Crossplane allows you to write Crossplane version Terraform modules, which are called Configurations.

So do I still recommend people to use Terraform? Yes I do. Developers might not really comfortable with k8s while writing Terraform is like writing a simple program to them. And Terraform has a large community supporting, new features will be supported quicker and bug fixes will be sooner.

Conclusion

In short, if you are going to deploy some infrastructure that are only for your k8s deployments, e.g. load balancers, IAM Roles for IRSA. Consider Crossplane.

If you are going to deploy some shared infrastructure or it is not relevant to your k8s deployments, e.g. Network, EC2 bastions. Use Terraform.

Discussion (5)

Collapse
jason_powell_faa3e31e5a22 profile image
Jason Powell

Have you heard of Upbound? They created Crossplane and are doing very cool things to commercialize and bring Crossplane to the masses.

Collapse
timtsoitt profile image
timtsoitt Author

Yes. If you are going to use Crossplane massively, I also suggest you to explore these commercial solutions.

Collapse
decktecheu profile image
Andrzej Kozlowski

Thanks for sharing. In your example Crossplane is running in EKS cluster with name EKS_CLUSTER_NAME, right ? I cannot use this example if I run Crossplane cluster on Azure AKS or GCP GKE ?

Collapse
timtsoitt profile image
timtsoitt Author • Edited on

Yes and no.

My tutorial is based on AWS EKS and IRSA is a EKS feature. So you can't simply follow my tutorial in Azure AKS or GCP GKE.

However it is possible that you use Azure or GCP K8S to provision AWS infrastructure. You need to provide a static AWS credential instead of using IRSA feature, which you can checkout Crossplane documentation for details.

Collapse
decktecheu profile image
Andrzej Kozlowski

Yes this I know. Are you aware about similar features as IRSA in Azure and GCP ?
I know that you can use eg Hashicorp Vault as mentioned in XP documentation but I like IRSA a lot. It is very simple solution comparing to quite advanced vault.