I Wrote a Dropbox Clone

I took part in a course on cloud computing at my university last semester. The course is offered with cooperation with IBM. Two of IBMs engineer joins us as in the course and together with the professor, they assigned us the task to build a dropbox like cloud storage solution using the cloud.

We must choose at least two cloud providers to deploy our application on and my team chose Google Cloud Platform (GCP) and Microsoft Azure. We can also use any sort services offered by the cloud, at any level that we prefer. In the first day of the course, we are given the introduction into the abstraction levels offered by cloud providers. These are from the bottom: Host own infrastructure, Infrastructure as a Service, Platform as a Service, Serverless, or Software as a Service.

Background

Before explaining further, maybe it's good to first tell you first about our background. We are a group of 4 Computer Engineering students who, before this, mainly works with programming hardware. I have used Docker, Maven in the past and would consider myself somewhat comfortable on the command line in Linux. Another one does cloud software data connecting using software. That is how far our knowledge about cloud infrastructure goes --- virtually none.

Choosing Cloud Services

Our first task is to choose what type of abstraction we are going for. We are given 3 weeks for this task. After doing ~a bit of research~ discussion with my teammate, we decided to go as low-level as possible but still without putting too much burden on our shoulders on managing the infrastructure itself. As we are planning to deploy on multiple cloud, we also consider the work that are needed to port the application from one cloud provider to another.

As a result, we decided on Kubernetes! Why? Kubernetes as a service is offered by both of our chosen cloud providers. Depending on Kubernetes also means that we don't rely on the API/interface provided by the cloud provider. We can program our application once for Kubernetes, and the ground structure will not change for other cloud provider as we are still in the same environment --- in home k8s cluster.

We also decided on S3 based cloud storage and database services provided by the cloud providers so that we don't have to manage the software ourselves. Just use it.

Architecture

Next comes how we want to structure our application. Being cloud-native is almost synonymous with microservice architecture. So, giving in to the hype, we decided to use it too although we don't really have any concrete reason to prefer this architecture compared to classic monolithic architecture. Hello, distributed monoliths!

We need to do a lot of catchup on the terms used in web development generally e.g. what is URL, endpoints, basic auth, token, JWT but after our first meeting, I would say that we know a little bit more about the world of web development in general.

Hello, Cloud!

After we got all the designs done, now comes the implementing part. We first started with Azure as our first cloud provider. The documentation is okay, with tutorials on using the services and SDKs in the most popular language. The first challenge comes when each of us must work with new tools such as Docker, Maven and Spring Framework.

Regretfully (more on this later), with the deadline encroaching, I decided to take away the difficulties from my friends and be the guy that are responsible for building and packaging the software. I gave them the template generated from Spring Initializr, built the software, and wrote the Dockerfile myself. All they have to do is focus on the software part, using the built-in run button in their IDE of choice to run the software when they need to test if their code is working. This works out fine but I do wish that we took the time to explain and learn the tools together. As time goes by, I am becoming more and more the bottleneck of our team.

At the same time the others start writing the software, I started my journey playing with Kubernetes. There are tonnes of resources available online to learn K8s and the official documentation is also really good. I have to give a shoutout to the CNCF YouTube channel because I spent countless hours in my evening watching videos and learning more about the wonders of Kubernetes.

The Bus Factor

As I've mentioned before, ~we~ I decided to hide the complexities of the tools from my teammates with the intention to give them the smoothest experience writing software for the cloud. Admittedly, that decision is wrong. We should have spent the time learning the tools and concepts needed.

I think none of my teammates understood Kubernetes Pods, Deployments, Services. I never explained it to them. As the project goes on, I am the goto guy if we have to debug what is happening on our cluster. Only I knew how to use kubectl logs to see which exception is thrown. This dependency really did slow our development process. Now, we didn't have the knowledge to solve our own problem. All we did is throw our codes at the wall and hope that it works. If it didn't work, call Aiman!

Conclusion

We ~learned~ crammed a lot in our first three weeks in the project. Amidst all that, there are sure some concepts that we got wrong and designs that are clearly flawed due to our inexperience. We will see that in the next entry in this series. Hope to see you again next time!