Developing An Internal Developer Platform

In this blog we will be discussing about Internal Developer Platform (IDP), what it is , how to set one up as a lead DevOps Engineer.Present tech landscape implementing an IDP helps to streamline workflows and boost productivity.

What is an Internal Developer Platforming

It is a platform or an application which is built by platform engineecloring team for building golden paths and to facilitate developer self service. It consists of various tools and technologies integrated together so that it lowers cognitive load on developers without abstraction of context and underlying technologies.

Why is there a need to build IDP

It Centralizes and simplifies application deployment and management.

Simplify Deployment:

Helps engineers to deploy applications with minimal manual intervention.

Increase Reliability:

This ensures that deployments are consistent and error free.

Enhance Productivity:'

Reduces time and efforts invested for managing infrastructure

Tools and Technologies Used

Kubernetes

This is a container orchestration tool which is used for automating deployments , scaling and management of applications which are containerized.

Documentation: https://kubernetes.io/docs/home/
Youtube: https://youtu.be/X48VuDVv0do?si=9aPNky0DDPyATiFM
Interview Questions: https://kuberty.io/blog/top-kubernetes-interview-questions/

Cyclops

It is a user friendly UI and streaming tool for managing kuberenetes clusters.

Documentation: https://cyclops-ui.com/docs/about/
Youtube: https://www.youtube.com/live/QQAqHGEYVRM?si=GEgZ3bDyhsCh_cHV
Github Repository: https://github.com/cyclops-ui/cyclops

Docker

It is a tool which is used for easy developing , shipping and running applications inside a container

Documentation: https://docs.docker.com/
Youtube: https://youtu.be/3c-iBn73dDE?si=FI7ZKDi3Kij1MYMo
Interview Questions: https://hackr.io/blog/docker-interview-questions

Helm

It is a package manager for kubernetes which helps in simplification of deployments and services

Documentation: https://helm.sh/docs/
Youtube: https://youtu.be/5_J7RWLLVeQ?si=8DPo9rD5cYf-moJd
Interview Questions: https://www.devopsschool.com/blog/top-50-helm-interview-questions-and-answers/

Prometheus

It is used for monitoring and alert management

Documentation: https://prometheus.io/docs/introduction/overview/
Youtube: https://youtu.be/h4Sl21AKiDg?si=2Heu9OAkG_FkZSNJ
Interview Questions: https://www.devopsschool.com/blog/top-50-prometheus-interview-questions-and-answers/

Graphana

It is a visualization tools for visualizing data

Documentation: https://grafana.com/docs/
Youtube: https://youtu.be/CjABEnRg9NI?si=QrZ3aBVUy_B9LOKH
Interview Questions: https://www.devopsschool.com/blog/top-50-grafana-interview-questions-and-answers/

PostgreSQL and MongoDB

These are popular relative and NoSQL databases respectively and is used for storing application data.

Documentation-PostgreSQL: https://www.postgresql.org/docs/
Youtube-PostgreSQL: https://youtu.be/qw--VYLpxG4?si=rory5ndP3nqGzKjZ
Interview Questions-PostgreSQL: https://www.datacamp.com/blog/top-postgresql-interview-questions-for-all-levels
Documentation-MongoDB: https://www.mongodb.com/docs/
Youtube-MongoDB: https://youtube.com/playlist?list=PL4cUxeGkcC9h77dJ-QJlwGlZlTd4ecZOA&si=N49ED6H2sztBQrFu
Interview Questions: https://hackr.io/blog/mongodb-interview-questions

Istio

It is a service mesh for advanced traffic management , security , observability for microservice applications

Documentation: https://istio.io/latest/docs/
Youtube: https://youtube.com/playlist?list=PLI4xy7phW54nbfjf7ZMnlEx1O5cHneigh&si=bs3RdBT6Tm6SF9Fc
Interview Questions: https://www.devopsschool.com/blog/top-50-istio-interview-questions-and-answers/

Jenkins

It is an open source automation server tool which is used for Continuous Integration (CI) and Continuous Delivery (CD), which automates building, testing and deploying applications via pipelines.

Documentation: https://www.jenkins.io/doc/
Youtube: https://youtube.com/playlist?list=PLy7NrYWoggjw_LIiDK1LXdNN82uYuuuiC&si=yjaMJdICFYJlCFZY
Interview Questions: https://hackr.io/blog/jenkins-interview-questions

Argo CD

it is a gitops CD tool for kubernetes , it is declarative and ensures the desires state of deployed application is matched as defined in git repository

Documentation: https://argo-cd.readthedocs.io/en/stable/
Youtube: https://youtu.be/MeU5_k9ssrs?si=PUzJxpQobUiVcqvm
Interview Questions: https://www.lwplab.com/forum/questions-answers/interview-questions-answers-on-argo-cd

Implementation

Step 1: Setup Docker

sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce

Step 2: Start Docker

sudo systemctl start docker
sudo systemctl enable docker

Step 3: Set up kubernetes

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

Step 4: Set Up Minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube-linux-amd64

Step 5: Start Minikube

minikube start

Step 6: Install Helm

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Step 7: Install Cyclops-UI

kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.8.2/install/cyclops-install.yaml && kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.8.2/install/demo-templates.yaml

Step 8: Expose cyclops and access it

kubectl port forward svc/cyclops-ui 3000:3000 -n cyclops

cyclops is now accesssible in browser by using this url: http://localhost:300

Step 9: Install and Configure Databases

PostgreSQL

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgresql bitnami/postgresql

MongoDB

helm install mongodb bitnami/mongodb

Step 10: Install Monitoring and Logging Tools

Prometheus

helm install prometheus bitnami/prometheus

Grafana

helm install grafana bitnami/graphana

configure grafana to add prometheus as data source, below is example yaml file named as grafana-datasource.yaml

Step 11: Install and implement Service Mesh with Istio

curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH
istioctl install --set-profile=demo -y
kubectl label namespace default istio-injection=enabled

deploy a sample app using istio , below is example yaml file named as sample-app.yaml used for deployment

deploy using kubectl apply -f sample-app.yaml

Step 12: Install ArgoCD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argocd/stable/manifests/install.yaml

• Expose the argocd to get access

kubectl port-forward svc/argocd-server -n argocd 8080:443

• Log in to argocd

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d 

Step 13: Install Jenkins

kubectl create namespace jenkins
helm repo add jenkins https://charts.jenkins.io
helm repo update
helm install jenkins jenkins/jenkins --namespace jenkins

• Expose jenkins to get access

kubectl port-forward svc/jenkins -n jenkins 8080:8080

• Login to jenkins

kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo

Step 14: Set up Jenkins Pipeline;

Access jenkins using the url http://localhost:8080, login using the admin password and username as admin, below is the sample image of jenkins console

now create a sample app in your local directory, it can be in any programming language , below is the sample app structure in node js format

next create a Dockerfile to run docker container , sample Dockerfile is shown below

now create a Jenkinsfile which contains the steps for jenkins pipeline , a sample pipeline is shown below

at last create a deployment.yaml file for kubernetes and other services to run , below is sample file for deployment.yaml

Usecase

Let's consider an E-commerce company which requires frequent updates and deployments and engineers are struggling with their deployments which is manual , inconsistent , lack of visibility. The answer to tackle these issues is building and IDP. Lets see how each tool helps in solving these problems.

Cyclops-Ui

1. Pre Deployment: Cyclops Ui can be accessed to set parameters
   like environment settings and resource allocation.

2. Deployment: the pipeline will be triggered by Cyclops-UI
   which leverages tools like kubernetes , docker and helm and
   ensures consistent and reliable manner of deployment.

Jenkins

1. Automated Builds: The source code repository in github is
   monitored by jenkins and an immediate trigger for new build
   happens when any change is detected in the repository.

2. Automated Tests: Once the changes are applied jenkins also
   tests the changes to ensure no bugs or regressions.

3. Deployment: After tests are successful the application is
   deployed on staging environment by jenkins and after validations
   are done it is deployed to production environment.

Argocd

1. Continuous Delivery: Detects changes in repository and synchronizes the kubernetes cluster to the desires state mentioned in manifest file.

Prometheus

This can be used to collect metrics from application and kubernetes cluster which provides insights into performance and cluster health.

Graphana

This is used for visualizng the metric data taken from prometheus
and create dashboards and monitor the application in real time

PostgreSQL and MongoDB

These are the primary databases of the application which uses transactional and non relational data respectively, it is managed by helm charts for consistency and scalability.

Istio

1. Traffic Management: Istio provides features for managing
   traffic between microservices , for loadbalancing and for
   routing as well.

2. Security: Security policies are enforced by istio to ensure
   secure communication between services.

Benefits

1. Efficient Deployment: Since deployment process is automated
   manual errors are reduced and consistency is ensured.

2. Enhanced Monitoring: With realtime dashboards ,visualization
   performance and issues are being monitored effortlessly.

3. Scalable Infrastructure: With tools like kubernetes , docker
   , helm applications can be scaled and managed easily also
   effective resource allocation is done without any difficulty.

4. Streamlined CI/CD: With jenkins building , testing and
   deployment is automated and consistent as and when changes in
   source repository are detected.

5. GitOps Workflow: With tool like ArgoCD deployed applications
   will always seam lessly match desired states mentioned in
   manifest file in git repository.

Final Thoughts

Building an IDP using cyclops and kubernetes includes setting up a variety of tools whcih ensures smooth and efficient deployment and management process. By using tools like Docker , Helm , ArgoCD, prometheus , graphana, PostgreSQL, MongoDB, Istio , Jenkins we have built a comprehensible and reliable platform which makes deployments and management of projects very easy.

By using a range of DevOps and CI/CD tools we have built a powerful environment which accelerates development and deployments. We have also seen an example usecase of E-Commerce website which shows how efficiently deployments can be managed and how it enhances both productivity and reliability