Choose Your Own Adventure: Open source, proprietary software, self-hosted or managed

We’re among friends here, so I think I can say this: proprietary software is expensive, static, unloved by developers. Open source is a great choice in terms of extensibility and its abundance, but it doesn’t come with support (I’m not talking about StackOverflow!), or SLAs, and in many cases sadly not even Docs.

A managed service provider takes care of upgrading the open source tools you rely on, so that you can enjoy bug fixes and new features. Typically managed services can host multiple components that are already in your stack, simplifying your infrastructure and offering easier integrations with new services.

Sounds good right? That’s because it’s a gross oversimplification. Some open source projects in recent years have changed their license to “cloud restricted” licenses to avoid cloud infra providers and managed service providers from offering their service and making money off of it - especially when they themselves offered paid support - excuse my french… an “enterprise” plan - for it.

Cloud hosted infrastructure doesn’t always have the best level of support either, since they’re spread across the entire technology stack. From storage to kubernetes, from edge computing, to testing frameworks. And you and yours might have opinions about being vendor locked in.

Spoiler alert: I will still try to convince you that managed can be the best of both worlds.

Dependency management

In this modern word we rely on a lot of components to make sure our stuff works and continues to work. I know you know this to be true, but I will still also give you stats.

The 2022 OSSRA (Open Source Security and Risk) report, produced by the Synopsys Cybersecurity Research Center (CyRC), examines the results of more than 2,400 audits of commercial codebases, the audit came back: 97% contained open source.

Four of the 17 industry sectors represented in the 2022 OSSRA report—Computer Hardware and Semiconductors, Cybersecurity, Energy and Clean Tech, and Internet of Things—contained open source in 100% of their audited codebases. The remaining verticals had open source in 93% to 99% of their codebases.

Security concerns

88% of organizations are still behind in keeping open source updated and they’re all worried about security.

The Snyk State of Open Source Security report for 2020:
Opened with stating that Open source ecosystems are experiencing continuous expansion. The leader was npm, with over 33% year-over-year growth and 1.8 million packages as of March 2022. And The majority of OS vulnerabilities continue to be discovered in indirect dependencies (packages).

Official base images tagged as latest often include known vulnerabilities, most notably the official node image which has almost 700 known vulnerabilities.

Open Source security culture is shifting towards developers:

• 85% felt developers were responsible for open source security
• 55% felt security teams were responsible
• 35% felt operations had a role to play

Now obvi this is more than 100% because some people selected more than 1 team, which really is a great development.

According to the survey, 47% of respondents said they expect a vulnerability to be fixed within a week of discovery, and nearly 18% expect a fix within a day.
In actuality, only 35% of vulnerabilities in scanned projects were fixed in under 20 days, while 36% took 70 days or more, within an average fix time of 68 days.

It’s clear organizations need to manage expectations around their risk posture. They need to be aware of SLAs for open source vulnerability fixes, especially when an individual contributor is responsible for maintaining the component they rely on.

How much will it cost to secure open-source software? OpenSSF says $147.9M based on critically scoring.

The State of Enterprise Open Source: A Red Hat report from February 2022, predicts a change in software penetration.

• Proprietary software: Today - 45% / In two years - 37%
• Enterprise open source software: Today - 29% / In two years - 34%
• Community-based open source software: Today-21% / In two years - 24%

This all to show that your organization too uses a mix of custom-built code, commercial off-the-shelf code, and open source components to create applications. Knowing about all the moving parts, means we can simplify handshakes between services, and make sure we’re secure.

If you’re using open source in your stack (spoiler: you are) you will want to know about it. We’re all moving fast and in order to do so we’re relying on a lot of dependencies to give us that commercial edge. In doing so we’re trusting the work of strangers on the internet.

Developers will use open source anyway to get their work done. And they might not even use company issued creds to gain access. They still have their private account, and they forgot to switch, or they forgot the password. Or - and I’m sure this doesn’t happen in your company because you only hire goody-goodies - they try to workaround Compliance.

It’s better if you know about it and monitor its health, or troubleshoot when things go awry.

Software Bill of Materials

Installing an average npm package introduces an implicit trust on 79 third-party packages and 39 maintainers, creating a large attack surface.
A software Bill of Materials (SBOM) is a list of all the open source and third-party components present in a codebase. An SBOM also lists the licenses that govern those components, the versions of the components used in the codebase, and their patch status, which allows security teams to quickly identify any associated security or license risks.

The concept of a software Bill of Materials derives from manufacturing, where a Bill of Materials is an inventory detailing all the items included in a product.

Sounds like a bunch of work? Good thing there are software composition analysis (SCA) tools that can help you do the job. Like Thomas Steenbergen’s OSS Review Toolkit (ORT), which includes Software Package Data Exchange (SPDX), an open standard for Software Bill of Materials (SBOM). SPDX allows the expression of components, licenses, copyrights, security references and other metadata relating to software.

License management

If you install Electron and have to add 87 packages — that means 87 license dependencies. Every single package is likely to have its own dependencies, and therefore, another license you need to comply with. As you can imagine license management can’t be done manually and when done incorrectly can create a technical debt.

License litigation may end up forcing you to release your code under the same license as the package dependency you used.

Other potential problems include:

• Being sued for financial liability by the creator of the component,
• Having to rewrite major part of the product, getting penalties and restrictions on selling your software until the compliance is met,
• Losing reputation and getting negative press coverage, certainly in sensitive industries

There are 300+ OSS licenses, and that list is growing (for a quick cheatsheet: https://brainhub.eu/library/open-source-licenses-to-avoid). However, the good news is that around 20 licenses account for 80% of the OSS commonly used in enterprises. So a deny/allowlist of these licenses together with a scanning tool already provides a very good starting point in managing them.

What can help you to track licenses inside your code, is this License Auditor tool, which sends notifications after spotting a potential problem.

89% of IT leaders believe enterprise open source is as secure or more secure than proprietary software. Maybe you’ve heard the phrase “Given enough eyeballs, all bugs are shallow” aka Linus’ Law - The law was formulated by Eric S. Raymond in his essay and book The Cathedral and the Bazaar from 1999 (not a recommended read, it did not age well), and was named in honor of Linus Torvalds, linux relies on a huge community for its development.

I myself am wary of the many eyes security “strategy” - hope is not strategy, but delayed disappointment. It’s not about avoiding vulnerabilities, they’re inevitable, but reacting to the upcoming ones faster.

To the question “Why security is a benefit of enterprise open source” came back:

• My team can use well-tested open source code for our in-house applications — 55%
• Security patches are well-documented and can be scanned for — 52%
• Vendors make vulnerability patches for enterprise open source available promptly — 51%
• More people have had their eyes on the code than with proprietary software — 44%
• My team can audit the code — 38%

Top benefits of using enterprise Open Source:

• Better security — 32% of respondents said this
• Higher quality software — 32%
• Ability to safely leverage open source tech — 28%
• Designed to work in cloud, cloud-native tech — 26%

Find the right partner in Open Source

I want to argue that when you go for a vendor that offers open source tools as a service, and that vendor being an excellent open source citizen, is possibly the sweet spot. Cloud vendors (offering open source services) typically do not have an unblemished reputation in this regard - the recent adoption of cloud restricted / more restrictive licenses is a direct consequence of their behavior. Not to say that they do all bad. Many of these vendors award funds to open source projects, allow employees to work on open source during working hours, and publish their own tech as open source.

But not-Cloud vendor tech companies and managed services and, when they have one, their OSPO (Open Source Program Office), or OTPO (Open Technology Program Office) often do excellent work.

Postman, the API platform, supports the AsyncAPI core team, contributes to several standards and specifications, and made sure that JSON Schema is well-maintained.

Spotify’s Backstage portal is open source and hosts a whole lot of plugins and tooling. Spotify’s OSPO runs a FOSS Fund, supporting projects the community relies on.

At Percona, the OSPO enables internal folks to contribute to open source.

Open Source at Aiven

I am biased towards how we do open source at Aiven: we employ people to contribute to the upstream projects for the services in our portfolio, like PostgreSQL, Kafka, and OpenSearch. We also publish our own libraries and connectors like Guardian and Karapace under the Apache 2.0 license.

Aiven is heavily invested in driving OpenSearch forward as the open source alternative to Elasticsearch. As is AWS.

To make sure it remains well maintained Aiven recently acquired Kafkawize - now renamed Klaw - an open source data governance toolkit, helping enterprises exercise Apache Kafka® Topic and schema governance.

Then there are individual investments, like in Avro, in an alternative to Akka (incubating as Apache Pekko as we speak), in Linux packaging... Outside of work hours too, where we can get compensated for time spent on open source through a program.

Offering multiple tools in a particular space allows us and companies like Aiven to share learnings across different communities.

If you want to learn more about what OSPOs do, please refer to the excellent resources by the TODO Group. For me the OSPO is the group of people that try their hardest to nurture the health of the components and their nested dependencies in your stack, finding alternatives when a project goes bad.

More people appreciate vendors who have a heart for open source, shows in the aforementioned Red Hat report as well: 82% of IT leaders are more likely to select a vendor who contributes to the open source community (respondents were not aware that Red Hat issued this report).

Top reasons why enterprise open source vendors are preferred:

• They are familiar with open source processes — 49%
• They help sustain healthy open source communities — 49%
• They can influence the development of features that we need — 48% 🤮🤮🤮
• They are going to be more effective if I face technical challenges — 46%

Find a partner that navigates participation in open source well, by abiding by The Principles of Authentic Participation, derived at the Sustain Summit 2020 event in Brussels, Belgium. There, Duane O’Brien and others facilitated discussion groups loosely focused on corporate accountability in the context of open source. I recommend you check those out.

In conclusion

Two weeks ago, at Devopsdays Eindhoven, I suggested an Open Space about how DevSecOps is just a Band-Aid on a bullet wound, after a talk about supply chain security (tools). I opened with, that while we should certainly scan our code for vulnerabilities, we should also invest in mitigating some of the root causes of vulns creeping into our code bases through open source use. Shifting security further left, if you will.

One of the people in the room mentioned just having a mirror of all the components you have in use. Which,

• congratulations, you’re now the maintainer of a bunch of mirrors
• as we learned earlier: sometimes vulnerabilities are in the code for many weeks, months or even years, before anyone notices. New releases bring fixes just as well.

Your organisation uses a mix of custom-built code, proprietary products, and open source components to create applications.

And while of course the real answer is “it depends”, I encourage you to think about managed services, and select a partner that is invested in the ecosystem they’re in. Make it a priority in your due diligence. It saves you a lot of headaches, and you’re ultimately contributing to a healthier ecosystem where we can all thrive.