DEV Community

Cover image for The Ultimate Comprehensive Guide to Setting Up Self-Hosted Airbyte on EKS Using Karpenter

The Ultimate Comprehensive Guide to Setting Up Self-Hosted Airbyte on EKS Using Karpenter

Table of Contents

  1. Introduction
  2. Prerequisites
  3. Setting Up EKS with Karpenter
  4. Deploying Airbyte on EKS
  5. Upgrading Airbyte OSS Version
  6. Troubleshooting Common Errors

Introduction

Airbyte is an open-source data integration platform that enables you to consolidate your data from various sources to data warehouses, lakes, and databases. Deploying Airbyte on Amazon Elastic Kubernetes Service (EKS) provides scalability and reliability, while Karpenter enhances cluster efficiency by dynamically provisioning nodes. This guide walks you through setting up a self-hosted Airbyte on EKS using Karpenter.

Prerequisites

  • AWS Account: An active AWS account with permissions to create EKS clusters and associated resources.
  • AWS CLI: Installed and configured with your AWS credentials.
  • kubectl: Installed for Kubernetes cluster management.
  • Helm: Installed for deploying applications on Kubernetes.
  • eksctl: Installed for EKS cluster management.

Setting Up EKS with Karpenter

Creating an EKS Cluster

First, create an EKS cluster using eksctl.

eksctl create cluster \
  --name airbyte-cluster \
  --region us-west-2 \
  --version 1.25 \
  --with-oidc \
  --zones us-west-2a,us-west-2b \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 2 \
  --nodes-min 1 \
  --nodes-max 3 \
  --managed
Enter fullscreen mode Exit fullscreen mode
  • --with-oidc: Enables IAM roles for service accounts.
  • --zones: Specify availability zones.
  • --node-type: Instance type for worker nodes.

Installing Karpenter

  1. Create AWS IAM Resources for Karpenter

Karpenter requires specific IAM roles and permissions.

   aws cloudformation deploy \
     --stack-name karpenter-iam \
     --template-file https://karpenter.sh/docs/getting-started/cloudformation.yaml \
     --capabilities CAPABILITY_NAMED_IAM
Enter fullscreen mode Exit fullscreen mode
  1. Install Karpenter Controller

Add the Helm repository and install Karpenter.

   helm repo add karpenter https://charts.karpenter.sh
   helm repo update

   helm install karpenter karpenter/karpenter \
     --namespace karpenter --create-namespace \
     --version v0.17.2 \
     --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<AWS_ACCOUNT_ID>:role/KarpenterControllerRole \
     --set settings.aws.clusterName=airbyte-cluster \
     --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-airbyte-cluster \
     --set settings.aws.interruptionQueueName=KarpenterInterruptionQueue
Enter fullscreen mode Exit fullscreen mode
  1. Create a Provisioner

Karpenter uses provisioners to manage node provisioning.

   apiVersion: karpenter.sh/v1alpha5
   kind: Provisioner
   metadata:
     name: default
   spec:
     requirements:
       - key: "node.kubernetes.io/instance-type"
         operator: In
         values: ["t3.medium", "t3.large"]
     provider:
       subnetSelector:
         karpenter.sh/discovery: airbyte-cluster
       securityGroupSelector:
         karpenter.sh/discovery: airbyte-cluster
Enter fullscreen mode Exit fullscreen mode

Apply the provisioner:

   kubectl apply -f provisioner.yaml
Enter fullscreen mode Exit fullscreen mode

Deploying Airbyte on EKS

Configuring Airbyte Resources

  1. Create a Namespace
   kubectl create namespace airbyte
Enter fullscreen mode Exit fullscreen mode
  1. Set Up Persistent Storage

Airbyte requires persistent storage for configurations and data.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: airbyte-pvc
     namespace: airbyte
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi
Enter fullscreen mode Exit fullscreen mode

Apply the PVC:

   kubectl apply -f pvc.yaml
Enter fullscreen mode Exit fullscreen mode

Deploying Airbyte

  1. Add Airbyte Helm Repository
   helm repo add airbyte https://airbytehq.github.io/helm-charts
   helm repo update
Enter fullscreen mode Exit fullscreen mode
  1. Install Airbyte
   helm install airbyte airbyte/airbyte \
     --namespace airbyte \
     --set persistence.enabled=true \
     --set persistence.existingClaim=airbyte-pvc
Enter fullscreen mode Exit fullscreen mode
  1. Verify Deployment
   kubectl get pods -n airbyte
Enter fullscreen mode Exit fullscreen mode

Ensure all pods are in the Running state.

Upgrading Airbyte OSS Version

  1. Check for New Versions

Visit the Airbyte GitHub Releases page to find the latest version.

  1. Update Helm Chart

Update the Helm repository and list available chart versions.

   helm repo update
   helm search repo airbyte -l
Enter fullscreen mode Exit fullscreen mode
  1. Upgrade Airbyte
   helm upgrade airbyte airbyte/airbyte \
     --namespace airbyte \
     --set image.tag=<NEW_VERSION_TAG>
Enter fullscreen mode Exit fullscreen mode

Replace <NEW_VERSION_TAG> with the new version number.

  1. Verify Upgrade
   kubectl rollout status deployment/airbyte-server -n airbyte
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Errors

Pods Stuck in Pending State

  • Cause: Insufficient resources or misconfigured Karpenter provisioner.
  • Solution: Check Karpenter logs and ensure the provisioner allows for the required instance types.
   kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter
Enter fullscreen mode Exit fullscreen mode

PVC Not Bound

  • Cause: Storage class issues or lack of storage resources.
  • Solution: Verify the storage class and ensure that the AWS EBS CSI driver is installed.

Airbyte Server CrashLoopBackOff

  • Cause: Misconfiguration or insufficient memory.
  • Solution: Increase resource limits in the Helm chart values.
   resources:
     limits:
       memory: "2Gi"
     requests:
       memory: "1Gi"
Enter fullscreen mode Exit fullscreen mode

Karpenter Not Provisioning Nodes

  • Cause: IAM permissions or subnet/security group selectors misconfigured.
  • Solution: Verify IAM roles and Karpenter's provisioner configurations.

Deploying Airbyte on EKS with Karpenter allows for a scalable and efficient data integration platform. By following this guide, you should have a self-hosted Airbyte instance up and running, with the ability to upgrade and troubleshoot as needed.


Feel free to reach out in the comments below if you have any questions or run into issues!

Top comments (1)

Collapse
 
xyz_oblivion profile image
oblivion

Hello, I am getting the next message:

Invalid template path https://karpenter.sh/docs/getting-started/cloudformation.yaml
Enter fullscreen mode Exit fullscreen mode

Could you please suggests?