The state of real-time analytics in 2023

This post is the second in a series on real-time analytics. It is an excerpt from Real-time analytics, a definitive guide which can be read in full here.

--

Modern real-time analytics applications power more than just dashboards. While faster dashboards certainly are a tangible byproduct of the modern movement towards real time, they don’t capture the complete state of the art for real-time analytics in 2023.

In addition to powering dashboards, real-time analytics applications are often - and increasingly - directly connected to other user-facing apps.

Today, a new class of real-time analytics platforms enables such use cases.

These platforms are built around three core tenets:

1. High-frequency ingestion of events and dimensions from multiple data sources. Real-time analytics demands a database that can handle writes at hundreds of megabytes per second magnitude and with very low latency. In general, this requires OLAP databases that are optimized for insert performance. ClickHouse, for example, claims insert throughput performance of 50-200 MB/s and up to 1M+ rows per second in common use cases, so it fits the bill for real-time analytics use cases.

2. Real-time data processing and transformation. As a class of data analytics, real-time analytics involves aggregating (almost always), filtering (usually), and enriching (sometimes) data as it streams in. In some real-time architectures, these transformations happen at query time, but in some cases, transformations must happen as data as ingested using materialized views or snapshots so that transformations themselves are persisted into storage.

3. Low-latency, high-concurrency publication layer. Finally, real-time analytics offers an API or query language that exposes analytics metrics to dashboards and user-facing applications. Request latency should be measured in hundreds of milliseconds or less to avoid a subpar user experience.
   Real-time analytics platforms have 3 core tenets: high-frequency database inserts, real-time data transformations, and a low-latency publication layer.

Real-time analytics platforms enable a whole new class of applications and use cases, such as:

• Cybersecurity applications that can intelligently detect patterns in real time and take automated action, such as adjusting DNS or updating firewall deny-lists.
• Personalized travel booking experiences that put the best offer in front of potential hotel patrons based on their current session data.
• Centralized cryptocurrency trading platforms that optimize trades for crypto market makers.
• Conversion-optimized eCommerce stores that display products most likely to be purchased by a specific visitor, and track the immediate performance of single-day flash sales as they’re happening.
• In-product analytics for content creators that shows them up-to-date data on how users are interacting with what they’ve made.
• High-precision, privacy-focused web analytics applications that track user behavior across a website.
• Stock management systems that identify when inventory needs to be diverted to alternative warehouses based on current user purchasing trends.

The benefits of real-time analytics platforms

Here are a handful of the advantages of building real-time analytics applications:

• Faster decision making. Real-time analytics answers complex questions within milliseconds, a feat that batch processing cannot achieve. In doing so, it allows for time-sensitive reactions and interventions (for example, in healthcare, manufacturing, or retail settings) made by humans who can interpret data more quickly to spur faster decisions.
• Automated, intelligent software. Real-time analytics doesn’t just boost human decision making, but increasingly enables automated decisions within software. Software applications and services can interact with the outputs of real-time analytics systems to automate functions based on real-time metrics.
• Improved user experiences. Real-time analytics can provide insights into customer behavior, preferences, and sentiment as they use products and services. Applications can then provide interactive tools that respond to customer usage, share information with customers through transparent in-product visualizations, or personalize their product experience within an active session.
• Better cost and process efficiencies. Real-time analytics can be used to optimize business processes, reducing costs and improving efficiency. This could include identifying and acting on cost-saving opportunities, such as reducing energy consumption in manufacturing processes. Real-time analytics can also help identify performance bottlenecks or identify testing problems early, enabling developers to quickly optimize application performance both before and after moving systems to production.
• Powerful differentiation. Real-time analytics can create a competitive moat for businesses that build it well. It gives a two-pronged speed advantage: Faster time to market and faster response times to customer needs. These two things make real-time analytics a powerful differentiator.

Real-time analytics enables faster human decision making, intelligent software automation, better cost efficiencies, and powerful differentiation.