In the course of building a product, we make technical decisions every other day. These decisions could be small decisions or big decisions. These could be reversible or irreversible decisions. Though there is no technical decision that is irreversible, it’s the amount of effort that makes it going back rather improbable. Architectural Design Decisions that a tech team makes at the start of product building fall into the category of big irreversible decisions.
Once these decisions are made, any tech lead will keep making smaller reversible decisions every other day with a few occasional big irreversible decisions. Here are some of those big technical decisions that we’ve made at AnalyticsVerse.
☁ AWS over Other Cloud Platforms
One of the major reasons why we chose AWS over other cloud providers was our familiarity with AWS. In addition, the amount of resources available on the internet for AWS is sufficient to get anyone up to speed in a couple of weeks.
☕ Java over Python
It is no secret that Java is much faster than Python. Speed of code execution was one of the biggest advantages of Java over Python. This has helped us provide a much better experience to our customers by generating reports faster.
🔢 SQL over NoSQL
Since AnalyticsVerse generates reports by correlating data across different tools such as Git Repositories and Task Tracking tools, all of this data had an inherent relation between them making SQL the go-to choice for us.
🖥 Server over Serverless
With Serverless, there comes the problem of warming/provisioning lambdas. This option turned out to be costlier as compared to running an instance behind a load balancer with autoscaling rules. In addition, the traffic that we serve is somewhat predictable which made not going serverless more suitable for our use case.
⌛ Airflow over Step Functions
Our first choice for setting up our data pipelines was Step Functions. However, one of the founders proposed we could evaluate Airflow. With airflow, we got an inbuilt scheduler as well in addition to being able to set up DAGs jobs. With airflow, we could easily support on-prem deployments from the get-go. We did face a challenge with scaling airflow a few months later, however that is an individual post in itself.
☁ Terraform over Cloudformation
In terms of the effort, it takes for setting up an Infra using terraform or cloud formation we found there was no difference. Terraform provided more control over state as compared to cloud formation and with terraform, we had an added advantage of being cloud-provider agnostic like most of our choices above.
The scaling challenge that I mentioned about airflow above was one such challenge that forced us to start thinking of alternatives. However, being a big irreversible decision, we were not very keen on moving out of airflow (given the effort it would take us :P) which forced us to think of creative ways to solve the scaling problem without having to increase compute power.
Top comments (0)