In this blog we will be discussing about Internal Developer Platform (IDP), what it is , how to set one up as a lead DevOps Engineer.Present tech landscape implementing an IDP helps to streamline workflows and boost productivity.
What is an Internal Developer Platforming
It is a platform or an application which is built by platform engineecloring team for building golden paths and to facilitate developer self service. It consists of various tools and technologies integrated together so that it lowers cognitive load on developers without abstraction of context and underlying technologies.
Why is there a need to build IDP
It Centralizes and simplifies application deployment and management.
Simplify Deployment:
Helps engineers to deploy applications with minimal manual intervention.
Increase Reliability:
This ensures that deployments are consistent and error free.
Enhance Productivity:'
Reduces time and efforts invested for managing infrastructure
Tools and Technologies Used
Kubernetes
This is a container orchestration tool which is used for automating deployments , scaling and management of applications which are containerized.
Documentation: https://kubernetes.io/docs/home/
Youtube: https://youtu.be/X48VuDVv0do?si=9aPNky0DDPyATiFM
Interview Questions: https://kuberty.io/blog/top-kubernetes-interview-questions/
Cyclops
It is a user friendly UI and streaming tool for managing kuberenetes clusters.
Documentation: https://cyclops-ui.com/docs/about/
Youtube: https://www.youtube.com/live/QQAqHGEYVRM?si=GEgZ3bDyhsCh_cHV
Github Repository: https://github.com/cyclops-ui/cyclops
Docker
It is a tool which is used for easy developing , shipping and running applications inside a container
Documentation: https://docs.docker.com/
Youtube: https://youtu.be/3c-iBn73dDE?si=FI7ZKDi3Kij1MYMo
Interview Questions: https://hackr.io/blog/docker-interview-questions
Helm
It is a package manager for kubernetes which helps in simplification of deployments and services
Documentation: https://helm.sh/docs/
Youtube: https://youtu.be/5_J7RWLLVeQ?si=8DPo9rD5cYf-moJd
Interview Questions: https://www.devopsschool.com/blog/top-50-helm-interview-questions-and-answers/
Prometheus
It is used for monitoring and alert management
Documentation: https://prometheus.io/docs/introduction/overview/
Youtube: https://youtu.be/h4Sl21AKiDg?si=2Heu9OAkG_FkZSNJ
Interview Questions: https://www.devopsschool.com/blog/top-50-prometheus-interview-questions-and-answers/
Graphana
It is a visualization tools for visualizing data
Documentation: https://grafana.com/docs/
Youtube: https://youtu.be/CjABEnRg9NI?si=QrZ3aBVUy_B9LOKH
Interview Questions: https://www.devopsschool.com/blog/top-50-grafana-interview-questions-and-answers/
PostgreSQL and MongoDB
These are popular relative and NoSQL databases respectively and is used for storing application data.
Documentation-PostgreSQL: https://www.postgresql.org/docs/
Youtube-PostgreSQL: https://youtu.be/qw--VYLpxG4?si=rory5ndP3nqGzKjZ
Interview Questions-PostgreSQL: https://www.datacamp.com/blog/top-postgresql-interview-questions-for-all-levels
Documentation-MongoDB: https://www.mongodb.com/docs/
Youtube-MongoDB: https://youtube.com/playlist?list=PL4cUxeGkcC9h77dJ-QJlwGlZlTd4ecZOA&si=N49ED6H2sztBQrFu
Interview Questions: https://hackr.io/blog/mongodb-interview-questions
Istio
It is a service mesh for advanced traffic management , security , observability for microservice applications
Documentation: https://istio.io/latest/docs/
Youtube: https://youtube.com/playlist?list=PLI4xy7phW54nbfjf7ZMnlEx1O5cHneigh&si=bs3RdBT6Tm6SF9Fc
Interview Questions: https://www.devopsschool.com/blog/top-50-istio-interview-questions-and-answers/
Jenkins
It is an open source automation server tool which is used for Continuous Integration (CI) and Continuous Delivery (CD), which automates building, testing and deploying applications via pipelines.
Documentation: https://www.jenkins.io/doc/
Youtube: https://youtube.com/playlist?list=PLy7NrYWoggjw_LIiDK1LXdNN82uYuuuiC&si=yjaMJdICFYJlCFZY
Interview Questions: https://hackr.io/blog/jenkins-interview-questions
Argo CD
it is a gitops CD tool for kubernetes , it is declarative and ensures the desires state of deployed application is matched as defined in git repository
Documentation: https://argo-cd.readthedocs.io/en/stable/
Youtube: https://youtu.be/MeU5_k9ssrs?si=PUzJxpQobUiVcqvm
Interview Questions: https://www.lwplab.com/forum/questions-answers/interview-questions-answers-on-argo-cd
Implementation
Step 1: Setup Docker
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce
Step 2: Start Docker
sudo systemctl start docker
sudo systemctl enable docker
Step 3: Set up kubernetes
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl
Step 4: Set Up Minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
chmod +x minikube-linux-amd64
Step 5: Start Minikube
minikube start
Step 6: Install Helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
Step 7: Install Cyclops-UI
kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.8.2/install/cyclops-install.yaml && kubectl apply -f https://raw.githubusercontent.com/cyclops-ui/cyclops/v0.8.2/install/demo-templates.yaml
Step 8: Expose cyclops and access it
kubectl port forward svc/cyclops-ui 3000:3000 -n cyclops
cyclops is now accesssible in browser by using this url: http://localhost:300
Step 9: Install and Configure Databases
PostgreSQL
helm repo add bitnami https://charts.bitnami.com/bitnami
helm install postgresql bitnami/postgresql
MongoDB
helm install mongodb bitnami/mongodb
Step 10: Install Monitoring and Logging Tools
Prometheus
helm install prometheus bitnami/prometheus
Grafana
helm install grafana bitnami/graphana
configure grafana to add prometheus as data source, below is example yaml file named as grafana-datasource.yaml
Step 11: Install and implement Service Mesh with Istio
curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH
istioctl install --set-profile=demo -y
kubectl label namespace default istio-injection=enabled
deploy a sample app using istio , below is example yaml file named as sample-app.yaml used for deployment
deploy using kubectl apply -f sample-app.yaml
Step 12: Install ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argocd/stable/manifests/install.yaml
- Expose the argocd to get access
kubectl port-forward svc/argocd-server -n argocd 8080:443
- Log in to argocd
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
Step 13: Install Jenkins
kubectl create namespace jenkins
helm repo add jenkins https://charts.jenkins.io
helm repo update
helm install jenkins jenkins/jenkins --namespace jenkins
- Expose jenkins to get access
kubectl port-forward svc/jenkins -n jenkins 8080:8080
- Login to jenkins
kubectl exec --namespace jenkins -it svc/jenkins -c jenkins -- /bin/cat /run/secrets/additional/chart-admin-password && echo
Step 14: Set up Jenkins Pipeline;
Access jenkins using the url http://localhost:8080, login using the admin password and username as admin, below is the sample image of jenkins console
now create a sample app in your local directory, it can be in any programming language , below is the sample app structure in node js format
next create a Dockerfile to run docker container , sample Dockerfile is shown below
now create a Jenkinsfile which contains the steps for jenkins pipeline , a sample pipeline is shown below
at last create a deployment.yaml file for kubernetes and other services to run , below is sample file for deployment.yaml
Usecase
Let's consider an E-commerce company which requires frequent updates and deployments and engineers are struggling with their deployments which is manual , inconsistent , lack of visibility. The answer to tackle these issues is building and IDP. Lets see how each tool helps in solving these problems.
Cyclops-Ui
Pre Deployment: Cyclops Ui can be accessed to set parameters
like environment settings and resource allocation.Deployment: the pipeline will be triggered by Cyclops-UI
which leverages tools like kubernetes , docker and helm and
ensures consistent and reliable manner of deployment.
Jenkins
Automated Builds: The source code repository in github is
monitored by jenkins and an immediate trigger for new build
happens when any change is detected in the repository.Automated Tests: Once the changes are applied jenkins also
tests the changes to ensure no bugs or regressions.Deployment: After tests are successful the application is
deployed on staging environment by jenkins and after validations
are done it is deployed to production environment.
Argocd
- Continuous Delivery: Detects changes in repository and synchronizes the kubernetes cluster to the desires state mentioned in manifest file.
Prometheus
This can be used to collect metrics from application and kubernetes cluster which provides insights into performance and cluster health.
Graphana
This is used for visualizng the metric data taken from prometheus
and create dashboards and monitor the application in real time
PostgreSQL and MongoDB
These are the primary databases of the application which uses transactional and non relational data respectively, it is managed by helm charts for consistency and scalability.
Istio
Traffic Management: Istio provides features for managing
traffic between microservices , for loadbalancing and for
routing as well.Security: Security policies are enforced by istio to ensure
secure communication between services.
Benefits
Efficient Deployment: Since deployment process is automated
manual errors are reduced and consistency is ensured.Enhanced Monitoring: With realtime dashboards ,visualization
performance and issues are being monitored effortlessly.Scalable Infrastructure: With tools like kubernetes , docker
, helm applications can be scaled and managed easily also
effective resource allocation is done without any difficulty.Streamlined CI/CD: With jenkins building , testing and
deployment is automated and consistent as and when changes in
source repository are detected.GitOps Workflow: With tool like ArgoCD deployed applications
will always seam lessly match desired states mentioned in
manifest file in git repository.
Final Thoughts
Building an IDP using cyclops and kubernetes includes setting up a variety of tools whcih ensures smooth and efficient deployment and management process. By using tools like Docker , Helm , ArgoCD, prometheus , graphana, PostgreSQL, MongoDB, Istio , Jenkins we have built a comprehensible and reliable platform which makes deployments and management of projects very easy.
By using a range of DevOps and CI/CD tools we have built a powerful environment which accelerates development and deployments. We have also seen an example usecase of E-Commerce website which shows how efficiently deployments can be managed and how it enhances both productivity and reliability
Top comments (0)