Table of Contents
- Introduction
- Prerequisites
- Setting Up EKS with Karpenter
- Deploying Airbyte on EKS
- Upgrading Airbyte OSS Version
- Troubleshooting Common Errors
Introduction
Airbyte is an open-source data integration platform that enables you to consolidate your data from various sources to data warehouses, lakes, and databases. Deploying Airbyte on Amazon Elastic Kubernetes Service (EKS) provides scalability and reliability, while Karpenter enhances cluster efficiency by dynamically provisioning nodes. This guide walks you through setting up a self-hosted Airbyte on EKS using Karpenter.
Prerequisites
- AWS Account: An active AWS account with permissions to create EKS clusters and associated resources.
- AWS CLI: Installed and configured with your AWS credentials.
- kubectl: Installed for Kubernetes cluster management.
- Helm: Installed for deploying applications on Kubernetes.
- eksctl: Installed for EKS cluster management.
Setting Up EKS with Karpenter
Creating an EKS Cluster
First, create an EKS cluster using eksctl
.
eksctl create cluster \
--name airbyte-cluster \
--region us-west-2 \
--version 1.25 \
--with-oidc \
--zones us-west-2a,us-west-2b \
--nodegroup-name standard-workers \
--node-type t3.medium \
--nodes 2 \
--nodes-min 1 \
--nodes-max 3 \
--managed
- --with-oidc: Enables IAM roles for service accounts.
- --zones: Specify availability zones.
- --node-type: Instance type for worker nodes.
Installing Karpenter
- Create AWS IAM Resources for Karpenter
Karpenter requires specific IAM roles and permissions.
aws cloudformation deploy \
--stack-name karpenter-iam \
--template-file https://karpenter.sh/docs/getting-started/cloudformation.yaml \
--capabilities CAPABILITY_NAMED_IAM
- Install Karpenter Controller
Add the Helm repository and install Karpenter.
helm repo add karpenter https://charts.karpenter.sh
helm repo update
helm install karpenter karpenter/karpenter \
--namespace karpenter --create-namespace \
--version v0.17.2 \
--set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<AWS_ACCOUNT_ID>:role/KarpenterControllerRole \
--set settings.aws.clusterName=airbyte-cluster \
--set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-airbyte-cluster \
--set settings.aws.interruptionQueueName=KarpenterInterruptionQueue
- Create a Provisioner
Karpenter uses provisioners to manage node provisioning.
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
name: default
spec:
requirements:
- key: "node.kubernetes.io/instance-type"
operator: In
values: ["t3.medium", "t3.large"]
provider:
subnetSelector:
karpenter.sh/discovery: airbyte-cluster
securityGroupSelector:
karpenter.sh/discovery: airbyte-cluster
Apply the provisioner:
kubectl apply -f provisioner.yaml
Deploying Airbyte on EKS
Configuring Airbyte Resources
- Create a Namespace
kubectl create namespace airbyte
- Set Up Persistent Storage
Airbyte requires persistent storage for configurations and data.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: airbyte-pvc
namespace: airbyte
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
Apply the PVC:
kubectl apply -f pvc.yaml
Deploying Airbyte
- Add Airbyte Helm Repository
helm repo add airbyte https://airbytehq.github.io/helm-charts
helm repo update
- Install Airbyte
helm install airbyte airbyte/airbyte \
--namespace airbyte \
--set persistence.enabled=true \
--set persistence.existingClaim=airbyte-pvc
- Verify Deployment
kubectl get pods -n airbyte
Ensure all pods are in the Running
state.
Upgrading Airbyte OSS Version
- Check for New Versions
Visit the Airbyte GitHub Releases page to find the latest version.
- Update Helm Chart
Update the Helm repository and list available chart versions.
helm repo update
helm search repo airbyte -l
- Upgrade Airbyte
helm upgrade airbyte airbyte/airbyte \
--namespace airbyte \
--set image.tag=<NEW_VERSION_TAG>
Replace <NEW_VERSION_TAG>
with the new version number.
- Verify Upgrade
kubectl rollout status deployment/airbyte-server -n airbyte
Troubleshooting Common Errors
Pods Stuck in Pending State
- Cause: Insufficient resources or misconfigured Karpenter provisioner.
- Solution: Check Karpenter logs and ensure the provisioner allows for the required instance types.
kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter
PVC Not Bound
- Cause: Storage class issues or lack of storage resources.
- Solution: Verify the storage class and ensure that the AWS EBS CSI driver is installed.
Airbyte Server CrashLoopBackOff
- Cause: Misconfiguration or insufficient memory.
- Solution: Increase resource limits in the Helm chart values.
resources:
limits:
memory: "2Gi"
requests:
memory: "1Gi"
Karpenter Not Provisioning Nodes
- Cause: IAM permissions or subnet/security group selectors misconfigured.
- Solution: Verify IAM roles and Karpenter's provisioner configurations.
Deploying Airbyte on EKS with Karpenter allows for a scalable and efficient data integration platform. By following this guide, you should have a self-hosted Airbyte instance up and running, with the ability to upgrade and troubleshoot as needed.
Feel free to reach out in the comments below if you have any questions or run into issues!
Top comments (1)
Hello, I am getting the next message:
Could you please suggests?