GreptimeDB v0.4 Officially Launched with New Engine Mito2 Tailored for Time-Series Data Processing

Introduction

Since its open-source launch last November, GreptimeDB has evolved significantly. We progressed from our foundational architecture in v0.1 to introducing a PromQL-compatible standalone version in v0.2, and then expanded to include distributed capabilities by v0.3.

For funtionality, v0.3 has proven to be quite stable. It boasts both standalone and distributed versions with PromQL compatibility and support for various access protocols. We've seen a surge of users giving it a spin and received invaluable feedback and suggestions. Also, we've been using v0.3 extensively in our GreptimeCloud project, a true testament to our belief in the product, and giving feedback to our GreptimeDB team.

A primary feedback from our users centered around query performance. While v0.3 offered stability and could meet most writing performance requirements, it lacked the optimal efficiency for large-scale data queries. In response, v0.4 introduced a comprehensive overhaul of our storage query engine, Mito2, entirely reworking most engine-associated modules.

After testing with the TSBS benchmark suite, there was an average 6x performance improvement. For detailed results, please refer to the test report at the end of this article. And for those with a passion for cars, 'mito' might ring a bell, echoing the Alfa Romeo MiTo. That’s not by chance – we actually have three Alfa Romeo enthusiasts on our team! :)

After rigorous testing, we're thrilled to announce the official release of GreptimeDB v0.4. This isn't just another regular upgraded version; v0.4 represents GreptimeDB's readiness to thrive in production environments, heralding a fresh chapter of exploration and challenges!

GreptimeDB v0.4 Summary

• Performance enhancement: Redesigned to introduce our new engine, Mito2
• Stability optimization: Unified standalone and distributed implementations for DDLs such as CREATE, DROP, ALTER, and introduced the Procedure framework to ensure eventual consistency.
• User experience improvement: Support for Vector, OpenTelemetry, etc.

From version v0.3 to v0.4, the Greptime team has made significant strides. We've had 42 unique contributors from 6+ countries, merging a whopping 726 PRs, leading to changes in 7,659 files, encompassing 271 feature enhancements, 163 bug fixes, 107 code refactors, and a multitude of tests.

Concurrently, we've updated our user manuals and documentation to reflect the latest changes. Additionally, we've rolled out SDKs for C++, Erlang, and JS, and we're excited to announce official support for six programming languages: Golang, Java, Rust, C++, Erlang, and JS.

Excitingly, we're thrilled to introduce our newest Committer: @Niwaka.

GreptimeDB v0.4 Key Updates

• Query and Storage Performance

  • Introduced the completely redesigned engine, Mito2, specifically tailored for time-series data, boasting a performance boost of from 5x to more than 20x compared to v0.3.
  • Inspired by Cassandra, we've incorporated the TWCS (Time Window Compaction Strategy) compaction method.
  • Support new object storage: Google Cloud Storage.

• Stability and Reliability

  • Expanded our 'sqlness' test scenarios.
  • Based on our improved Chaos Testing cases, v0.4 boasts enhanced distributed robustness.
  • Leveraged the Procedure framework to support DDL operations like Create, Alter, and Drop Table, harmonizing both standalone and distributed implementations.
  • Refactored the entire distributed architecture and optimized the heartbeat protocol to ensure a more stable operation in the distributed mode.

• User Experience Improvements

  • Launched an entirely new Dashboard.
  • Enhanced range select grammar, now supporting nested use with regular functions.
  • Refined the DESC TABLE user experience.
  • Support external tables in the Apache ORC format.
  • Support for the gRPC protocol in both row and col format.

• Integration and Others

  • Support GreptimeDB metrics output in Vector.
  • Facilitated the initiation of GreptimeDB clusters using KubeBlocks.
  • Integrated with EMQX.
  • Support for CentOS/Windows/Android.
  • Released official client SDKs for JS, Erlang, and C++.

For detailed release note of GreptimeDB v0.4, please refer to our release note on GitHub

GreptimeDB v0.4 Performance Testing

GreptimeDB's performance tests are conducted based on TSBS(Time Series Benchmark Suite), which is a set of programs mainly written in Go, designed to generate time series datasets and benchmark read/write performance. Originally released by time series database engineers at InfluxDB, it has been continuously improved upon by the TimescaleDB team. This suite is scalable, accommodating different types of data and query formats, and facilitating comparisons across systems.

GreptimeDB has extended this suite for its specific requirements.

The detailed project can be found at: https://github.com/GreptimeTeam/tsbs/

Hardware and Data Preparation

Hardware Parameters

Testing Data (based on TSBS)

Startup Configuration (Default settings with modification for data directory)

[storage]
data_home = "/var/greptimedb/"

Testing Result (Compared to v0.3.2, a smaller value represents the better)

Upgrade Notification

To carry out a thorough optimization from 0.3 to 0.4, there are some breaking changes that necessitate a system downtime for the upgrade. We've developed an upgrade tool to assist in this transition. A detailed upgrade guide will be added to our official documentation shortly.

Here's an overview of the steps:

1. Set up a new 0.4 cluster;
2. Shut down traffic to the old cluster (stop data writing);
3. Use the CLI upgrade tool to export the data;
4. Import the data into the new 0.4 cluster;
5. Switch traffic over to the new 0.4 cluster.

Future Plan

Central to GreptimeDB's ongoing development is our steadfast dedication to crafting a flexible serverlessDB architecture. Recognizing the future trend towards autonomous databases, we're progressively investing more in autonomous database capabilities. By integrating advanced scheduling technology, we aim to steadily achieve secure autopilot for time-series databases.

A Thank You to Our Community

We're profoundly grateful to everyone who has starred GreptimeDB, offered suggestions, and even contributed code. Your unwavering support fortifies our resolve to push forward.

GreptimeDB is a database project built on Rust. Whether you're exploring Rust and seeking a project to get started, a seasoned tech expert, or even a non-technical individual aspiring to contribute to the open-source community, Greptime promises avenues tailor-made for your contribution. As a contributor, you'll join a vibrant community of tech enthusiasts from all over the world, exchanging the latest in tech innovation. And, just a little insider tip: contributors get to enjoy exclusive Greptime SWAG! 🎁

Explore our repo now on GitHub and find something interesting to start with today.