The Future of DevOps is Open-Source

Throughout my engineering career, which includes working at Amazon and Facebook, I have reaped the benefits of utilizing open-source software such as elasticsearch, redis, MySQL, and Postgres to build complex platforms.

To say that I and the rest of the team at Plural are bullish on open-source software would be an understatement.

When I founded Plural in 2021, the open-source services market was valued at $21.7 billion with 97% of codebases containing open-source software. A recent report from MarketsandMarkets predicts that the global open-source services market is expected to reach $50 billion by 2026.

Open-source software is growing in demand and is beginning to play a pivotal role in the evolution of the DevOps toolkit.

The Benefits of Using Open-Source Software

As developers, there are plenty of benefits to using open-source software. Before diving into the more nuanced, let’s start with the obvious benefits that most developers enjoy immediately after implementing open-source software into their tech stack.

1. Decrease in Cost

Software can be extremely expensive, especially with variable-priced models that quickly scale to an astronomical monthly bill. Often, this is deprioritized vs. time to market, but optimizing software and infrastructure spend has never been more valuable, especially in a world where financing is less plentiful and burn rates are top-of-mind.

I’m not saying that you should use just open-source software, as there are plenty of scenarios where it might not make sense. But you should seriously consider open-source solutions as alternatives to high-cost software products.

At Plural, we utilize an open-source software first stack, and in return have saved tens of thousands of dollars a year on what would have been high ticket purchase items such as Looker, Amazon RDS, and Fivetran. We can invest that money back into our employees and open up budget headroom for another critical hire, or an increase in marketing spending.

2. Reinventing the Wheel

A mature stack is going to require a lot of subcomponents with perfectly adequate existing implementations. When building your product, you’ll likely need to leverage a feature flag system, a job orchestrator, and of course relational databases. Reimplementing them yourself is a waste of bandwidth, and honestly, you’ll have a hard time outperforming battle-hardened implementations in the open-source ecosystem.

3. Strong Community

High-quality open-source projects have active communities with thousands of developers contributing to the source code, dividing and conquering on immensely labor-intensive tasks. A great example of this is Airbyte, an open-source EL(T) platform that helps you replicate data from disparate sources to your data warehouses, data lakes, and databases.

Airbyte connects to over 170 heterogeneous APIs, and writing extractors and loaders for them is an immensely time-intensive process. The team over at Airbyte has done a great job in creating an active community that is eager to contribute to their platform. Over time they have gradually aggregated all of those integrations.

Another key aspect that comes from the community is being able to gain the consensus of thousands of developers. Some of the most popular open-source projects often have the most usable or featureful APIs.

4. Portable Software

Finally, I think one of the least appreciated attributes of open-source software is that it is built to be self-hosted. That means you can move an application into its appropriate environment for your organization’s security posture or to deal with intractable data gravity problems.

A lot of the applications that our users find most helpful such as Airflow and Airbyte require access to internal databases that cannot be reached from a centrally hosted SaaS solution. That level of portability becomes a superpower when you start knocking against those issues.

Why Open-Source Fits into the Future of DevOps

Earlier I hinted that open-source has a unique value for DevOps, which is worth diving into further. Our friends over at Gitlab define DevOps as a software engineering methodology that aims to integrate the work of software development and software operations teams by facilitating a culture of collaboration and shared responsibility.

At its core, DevOps is a knowledge problem for developers. Your job is to glue together a ton of disparate systems and ensure they’re properly running, which involves knowing all their internals and failure modes.

Realistically, the only way to make this situation traceable is to drive standardization, which we have seen continuously come out of strong open-source communities that drive adoption. If you were to enumerate the typical developer stack, it’s almost all open-source software:

• Linux as the operating system
• Docker/containerd for containerization
• Kubernetes as the orchestration engine
• Git for version control
• Postgres for persistence
• redis/Memcached for caching
• nginx/envoy/HAProxy for load balancing

This open-source stack substantially reduces the surface area an operator needs to cover to understand their system. Additionally, by keeping the source code and development process open to the public, you unlock a wealth of knowledge that comes in handy when troubleshooting, as there is likely some Github issue or forum post that’s tangential to what you are currently experiencing.

Finally, the ability to self-host solutions is a major differentiator for a DevOps tool. Almost any tool that touches the developer lifecycle is a high-security risk, either potentially affecting the availability of your product or leaking valuable IP. I can’t stress enough how important it is to keep all developer-facing tools on secure networks with a vetted infrastructure.

Sure, ease of use and time to market are countervailing concerns, but any engineer nowadays understands the implications a security breach can have on their organization. If you can make your own informed decisions I highly recommend you choose a private instance for most of your developer tooling.

Where Open-Source Falls Short in DevOps

I wouldn't be doing justice to the community if I didn’t quickly explain why some people are still cautious around open-source.

The first one that is commonly thrown around by open-source skeptics is the quality of open-source products. I can attest to this, as I have personally seen some open-source projects that are not properly maintained, and as a result, are a low-quality alternative to a SaaS solution.

I frequently see this in web applications that are developed in open-source, where the design and product polish are nowhere comparable to the closed-source competitor. A great example of this is Apache Superset vs. Looker. We’re happy users of Apache Superset at Plural, but we’d be lying if we said it’s as good as a product as Looker, which is a market leader for a reason.

It’s also important to call out that the cost of ownership is sometimes hidden in open-source products. Sure, the adoption of any open-source product is, in theory, “cheap.” However, there is an unclear cost in engineering overhead to manage the deployment long term.

Most SaaS products have transparent costs that are available on their pricing page, and even if it’s a consumption-based pricing model you can still get a general idea of how much it will cost you a month. If you fail to manage open-source solutions properly, the cost can add up quickly, or even worse turn into an unreliable product.

Lowering the Barrier to Entry for Open-Source

Ultimately, we believe that the quality concern is a corollary of the true problem, which is the cost of ownership. If it is seamless to adopt open-source software, then there would be a demand for quality in open-source solutions. In turn, that demand would be met by viable, funded, open-source solution businesses that provide the necessary recourses to meet the rising demand. However, that demand will continue to stay low if users feel like the cost of getting started is high.

Fortunately, Plural aims to make open-source adoption a matter of two commands and waiting for a K8s cluster to provision, with frictionless long-term operations. Hopefully, that can be the spark for the next wave of open-source software and lower the barrier to entry when it comes to getting started with open-source software.

To learn more about how Plural works and how we are helping engineering teams across the world deploy open-source applications in a cloud production environment, reach out to me and the rest of the team.

Like what we are doing and want to contribute to our code base? Check out our code base

Ready to effortlessly deploy and operate open-source applications in minutes? Get started with Plural today

Join us on our Discord channel for questions, discussions, and to meet the rest of the community.