What spawned Open Distro?

In January 2021, Elastic announced that Elasticsearch and Kibana would move from the OSS Apache 2.0 license to a dual license model. The dual licenses are the paid Elastic License and a free-tier Server Side Public License (SSPL). While SSPL is free for use, it is not classified as open-source by the Open Source Initiative (OSI). Instead, SSPL is a ‘source available’ license meaning that modifications you make must be made public. For more details on this license and its implications with Elastic, see the Coralogix blog’s breakdown.

While not every company is affected by this new agreement, some companies could not conform to the new restrictions. A platform that simply used the Elastic stack without modification has no new restrictions (assuming they never would change the source code). However, some businesses built functionality around the Elastic stack. Most notably, this issue affected AWS’s release of AWS Elasticsearch Service.

AWS launched Open Distro in 2019 as an Elasticsearch Distribution that would be used by AWS and could be used as an alternative to Elastic. Prior to the license change, AWS would both pull in Elastic’s source updates and contribute to the Elastic distribution. As a result of the license change, AWS decided to maintain a fork from the most recently available Elastic distribution under the Apache license. Open Distro still operates under the Apache License and is what now drives the AWS Elasticsearch service. According to the license, users are encouraged but not required to submit code changes, including features and bug fixes.

License Differences between Elastic and Open Distro

Elastic runs under a dual-license model. The free tier is the Server Side Public License which is not technically open source. The license essentially means the software is free to use, but contributors must make modifications publicly available if using the source code in any SaaS platform. There is also a paid tier under the Elastic License, allowing users to access advanced features without access to the source code.

The Open Distro Distribution is completely available under the Apache License Version 2.0. Apache is a genuinely open-source license as defined by the OSI. Modifications may be used privately or made public at the discretion of the contributor.

If you choose to use Elastic, keep in mind there are two different free tiers. The tier labeled ‘Free and Open’ can be under either SSPL or the Elastic License. The one labeled ‘Basic - Free and Open’ is the Elastic License.

Security in Elastic and Open Distro

One reason for creating Open Distro in 2019 was that security features in Elastic are mostly linked to a paid and licensed plugin service. With Elastic Security, you pay for the hardware resources you use. Open Distro uses a security plugin under the same open-source Apache 2.0 license. The two security offerings differ in their feature set as well.

Open Distro Security

Open Distro has enterprise-grade security features connected through a plugin for authentication and access control. A demo setup is available for use in testing and development environments, but a complete setup must be done to use security in production environments. Steps to set up security are clearly laid out on the Open Distro site.

Elastic Security

Elastic Stack Security has some features in the free tier, and others require Gold, Platinum, or Enterprise subscriptions. By default, security settings are disabled with the two free licenses. Security must be enabled using XPack.

Secure settings are available on every license tier of Elastic, but not all features are included in every tier. Features only included on paid tiers include Elasticsearch and Kibana audit logging, AP filtering, user authentication, and Elasticsearch token service. Platinum and Enterprise subscriptions only include single sign-on, attribute-based access control, field, and document level security, custom authentication and authorization, encryption at rest, and FIPS mode.

Security Features Available in Both Distributions

The following features are available on both distributions, with some features being paid services in Elastic Security.

Node-to-Node Encryption

Open Distro encrypts all data flowing between nodes in your cluster. Elastic also provides encrypted communications and also encryption at rest. Encrypted communications are available only in the free and open tier of Elastic. Encryption at rest is only available at the platinum or enterprise subscription level.

HTTP Basic Authentication

Both distributions use basic authentication over HTTP. This authentication uses a user name and password as part of each HTTP request. It is needed in every request since both distributions run as stateless systems.

Industry-Standard User Authentication

Both distributions use existing industry-standard authentication tools to authenticate existing users or create new users. Users are stored in an internal user database. Support includes Kerberos, OpenID Connect, and SAML. Open Distro also includes Active Directory and LDAP.

Role-Based Access

User roles determine what actions users may take in the cluster. This includes what data can be read and written, what cluster settings can be modified, and whether or not they can add users to the cluster. Roles can be reused across different users, and users may have multiple roles as well.

Audit Logging

Logs are printed from both Elasticsearch and Kibana, tracking access to your cluster. Businesses can use these logs to prove regulatory compliance or to analyze cluster exposures after an attack. Users can view and manage logs in Kibana. Events tracked include failed logins, successful authentications, missing or granted privileges, and role change attempts. Elastic only includes audit logging on its paid tiers.

Multi-Level Security

Access can be granted or restricted at multiple data levels. In Open Distro, these include at the index, document, or even field level. In Elastic, field and document level security is available only on Platinum or Enterprise tiers.

Search Features in Elastic and Open Distro

AWS forked Open Distro from Elasticsearch and Kibana 7.10 source code. Features created before this version in Elastic are the same on the two distributions since both are based on the same source code. Here we will discuss some significant diversions between the two distributions to help determine which is better for your needs.

Since Open Distro forked only minor versions back from the current Elastic version, the differences between these two distributions will only grow from this list. Open Distro has had two patch releases since the fork of Elasticsearch and Kibana. There are no significant differences added other than some bug fixes in the distribution.

Elastic has had three minor version releases since removing its open-source license. There have been many enhancements, new features, and bug fixes released in that time. In Elastic 7.11, developers released a beta version of schema on read, or schema discovery. Elastic calls these runtime fields, and they allow for the calculation of data on searching. In Elastic version 7.13.0, developers released a frozen storage tier for inexpensive data storage with a penalty of query time. For a complete list of features released on Elastic, see their release notes.

Alerting in Elastic and Open Distro

Alerting is present in Kibana and is used to send notifications to users on certain predetermined events. These alerts are critical for knowing when there has been an attack on your cluster when your cluster is not behaving optimally or on a fixed interval.

Open Distro allows users to configure monitors and send alerts using their choice of communication. Users can set up triggers for their alerts using visual graphs in Kibana, using an extraction query or the anomaly detector. In each case, the user must provide some custom setup for the trigger.

Elastic alerts run on a schedule and check that certain conditions are met before taking action. Only basic Kibana alerts are available in the free tier version using the Elastic License. All other alerting, such as anomaly detection, is behind the paywall.

Other third-party services are available with alerting capabilities as well as anomaly detection. Coralogix proves user-defined alerts that can be used with either distribution.

Summary

Earlier this year, Elastic shifted to a source-open licensing model angering the open-source community. As a result of this shift, AWS will now maintain an open-source distribution of Elasticsearch and Kibana called Open Distro. From version 7.10, the two distributions are essentially the same except for paid vs. free functionality. The divergence in functionality only started after the license change applied in Elastic, starting with version 7.11.

Only time will tell which distribution will win out in the market for search and analytics functionality. For now, it seems that Elastic will be faster at delivering new features and enhancements. Open Distro, being relatively new on the scene, will take longer to ramp up and start delivering new features. Open Distro has replicated some of the paid features in Elastic in an Open Source version.

If you are starting a new project, consider using Open Distro since the genuine open-source license means more freedom to make source code changes. As well, features are all available for use and not blocked by subscription layers. Elastic will have more features to start, but once AWS ramps up development, they will be able to push out functionality quickly. For features missing from Open Distro or behind a paywall in Elastic, consider third-party tools like Coralogix’s Cloud Security Platform or their Log Analytics Platform.