HarperDB Roadmap - Interview with the Product Team

As with any innovative, fast-paced tech startup, the community is often curious about what’s to come. What’s the next feature, product adaptation, project, etc. (As if being the first distributed database with hybrid SQL/NoSQL functionality and a REST API isn’t enough … 😆 jk jk). So, as HarperDB embarks on the most exciting year yet, I decided to interview none other than our VP of Product, Jaxon Repp, to discuss our upcoming roadmap. There’s a lot on this list, so of course some features will happen far sooner than others. We will work to prioritize the roadmap in an order that will best serve both our community users and enterprise customers.
This discussion provides insight into the exciting new features and improvements we’re planning for the future of HarperDB. While the priority of these items may change, this effort at transparency is intended to help you understand where HarperDB is heading. As always, feedback and feature requests are welcome and appreciated: visit feedback.harperdb.io to vote, comment on feature requests, and add new ones of your own!
When I asked Jaxon if he would be comfortable sharing HarperDB’s upcoming roadmap, he responded, “Of course - when we say one of our core values here at HarperDB is transparency, we mean it.”

Well then, let’s get started!

Functionality

Margo:
If you haven’t noticed, our logo is a dog. 🐶 (The company was actually named after our CEO’s border collie, Harper!) So I asked Jaxon what type of dog would HarperDB be in the database world? He provided some interesting insight:
Jaxon:
We’re the Border Collie of databases: intelligent, active, and with a laser focus on situational awareness. But we never forget the broader marketplace, and our place in it. So we work hard to ensure that alongside all our shiny, cool, developer-friendly features, choosing HarperDB never means leaving industry-standard functionality on the table:

• SQL Subselects: Long-awaited and much-requested, the workhorse of data science is coming to HarperDB’s sql operation. In the meantime, HarperDB drivers support subselects, so your BI Tools can subquery in peace.
• Backup & Restore: Explicit operations to backup and restore a HarperDB, useful for creating clustered replicas, exporting data for analysis or testing, and disaster recovery.
• Data Sharding: Distribute your largest datasets across multiple HarperDB instances with the most efficient distribution algorithm available, affording you unlimited scale at without unnecessary cost.
• Authentication Options: Authentication strategies beyond JWT and Basic Auth. The leading candidate is OAUTH support.

Performance

Margo:
Wow, there are some pretty sweet functionality features in the pipeline! Let’s talk performance. Based on our benchmarks (and general customer feedback), I know that HarperDB is already breaking records with crazy fast speeds on both reads and writes. What’s to come? Let me guess… even more speed. 🚀

Jaxon:
HarperDB is dedicated to delivering best-in-class performance. I’m pretty sure we’re going to have to rerun some benchmarks with our 3.0 release scheduled for late Q1/early Q2 2021:

• Query Engine: HarperDB is going to reconfigure our underlying key-value store to standardize the way we create and manage indexes across all data types, which will result in significant performance improvements for all operations.
• HTTP/S API: We’re replacing Express.js with Fastify as our API layer, resulting in significant performance improvement on every single API interaction.
• Internal Processes: Internal communication is being streamlined, speeding up both internal processes and external operations.

Margo:
So, it sounds like we’ll be breaking our own records this time? 😎

Extensibility

Margo:
Since my very first day on the HarperDB team, it was clear that our core objective is to simplify the lives of developers. Really, our goal is to be the most user-friendly (and fastest 🙂) database on the market. So let’s get a little insight on extensibility.

Jaxon:
Our HTTP/S API is a great example of extending HarperDB beyond data persistence to achieve that goal. And while our core operations are a great first step, we’re going to let our users extend it themselves:

• Custom Functions: Create your own serverless functions using Node.js. This will enable developers to create their own API endpoints with custom logic and direct access to HarperDB functions.
• Triggers: Execute one or more cascading operations when something else happens to a specific schema, table, or record.
• GraphQL: Expose GraphQL as a query option within HarperDB so you can reduce the number of calls required to get exactly the data you need in one shot.

Margo:
What I’m hearing is even more flexibility and customization… sounds like a win win!

Clustering

Margo:
Many HarperDB users love the clustering features. Our clustering engine already delivers bi-directional, table-level data replication to ensure you only store the data you need, where you need it. So where do we go from here?

Jaxon:
We’re planning substantial improvements and additions to our flexible data-plane architecture:

• Distributed Queries: Provide users with the ability to execute query operations (both SQL and NoSQL) across a cluster via a single HarperDB Instance.
• Improved Pub/Sub Performance and Durability: We’re replacing our existing clustering engine with the fastest greyhound we could find: NATS.io’s new Jetstream platform, delivering durable subscriptions and a 100x throughput boost.

Administration Studio

Margo:
Ah, one of the greatest features of HarperDB, the Management Studio. The Studio makes it easy to manage your organizations, instances, architectures, and replication. And you can even do it all without writing a line of code (if you don’t want to, that is)! Jaxon, in addition to things like graphs, example code, SDK’s, drivers, and video tutorials, what will we be adding to the Studio?

Jaxon:
We’ve got some big improvements on deck to ensure Studio works across every topology, at any scale.

• Offline Studio: We’re going to upgrade the offline, instance-hosted version of HarperDB Studio to match the feature set of the cloud-hosted version available at studio.harperdb.io. We’ll also allow you to disable it entirely for maximum security.
• Clustering Visualization: Our current clustering configuration is done on an instance-by-instance basis. We’re going to add a single, comprehensive interface for managing and monitoring data flow across your network.
• CSV Export: We’re going to let you export and download the results of a specific query (or entire tables) in CSV format directly from the Studio, rather than requiring that you use our current export operation and fetch the file from the filesystem.

Deployment

Margo:
Last but not least, let’s talk about deployment. The HarperDB team has already gone to great lengths to make HarperDB easy to install locally (via npm), and we built a powerful, simple, hosted option in HarperDB Cloud. (HarperDB can run anywhere!) What’s next?

Jaxon:
You’re right, and we are dedicated to both expanding those offerings and making HarperDB available on all the most popular enterprise-class deployment channels:

• HarperDB Cloud Goes International: Fire up the yacht, we’re going to Monaco 🚤! Well, maybe not, but we do intend to provide users with the ability to deploy hosted HarperDB Cloud both on AWS instances outside of the US, as well as on other cloud providers.
• Kubernetes Deployments: Templates and guidance for automated Kubernetes deployments.
• Optimized Docker Container: Performance optimizations of our existing HarperDB Docker container.
• One Click DigitalOcean Deployments with Kubernetes: Provide DigitalOcean users the ability to create a one click distributed database solution using Kubernetes.

Margo:
Amazing! I’m sure everyone will be excited to hear that HarperDB is going international. 🌏 We already have so many users globally, and this will only improve their performance and accessibility. Kubernetes is definitely a plus as well.

Well there you have it folks, straight from the horse’s mouth! Jaxon and the rest of the HarperDB product and development team take customer feedback and market needs very seriously, and are working every day to ensure that we are providing the highest value to HarperDB users as possible. We’re a lean team, but we’re passionate about what we do and proud of how far we’ve come. It will be interesting to see which features are released first, and which are pushed further down the line, but either way each release is sure to bring something pretty awesome to the table. Thank you to our community for contributing feedback and helping to shape our product roadmap 💜. Onward and upward!