Exploring Metrics with Thanos Query UI and PromQL

The Thanos Query UI provides robust querying capabilities across multiple Prometheus instances, offering a comprehensive analysis of your metrics. Using the Thanos Interactive example dataset, I experimented with various PromQL queries to explore metrics. Here’s a summary based on frequent use cases:

Available Metrics in the Metrics Explorer:

• continuous_app_metric0
• continuous_app_metric1
• continuous_app_metric2
• continuous_app_metric3
• continuous_app_metric4

PromQL Query Examples:

1. Sum of Metrics: sum(continuous_app_metric0) - Sum across all clusters and replicas.
2. Sum by Cluster: sum(continuous_app_metric0) by (cluster) - Grouped by cluster.
3. Sum by Replica: sum(continuous_app_metric0) by (replica) - Grouped by replica.
4. Average by Cluster: avg(continuous_app_metric0) by (cluster) - Average grouped by cluster.
5. Maximum Value by Replica: max(continuous_app_metric0) by (replica) - Max value grouped by replica.
6. Count of Instances: count(continuous_app_metric0) - Count of reporting instances.
7. Specific Cluster and Replica: continuous_app_metric0{cluster="eu-1", replica="0"} - Current value for a specific cluster and replica.
8. Alerting: continuous_app_metric0 > 100 - Create an alert if the metric exceeds a threshold.

Understanding Empty Query Results:

• Histogram Quantile Query: May return empty if _bucket suffix is missing, required for histogram_quantile.
• Comparison Between Clusters: Returns empty if data points are missing or unmatched in timestamps.

PromQL Info Messages:

• Rate and Increase Queries: Indicate that continuous_app_metric0 might not be a counter as it does not end in _total, _sum, _count, or _bucket.

Metric Types Explained:

• Metric Types in Prometheus
  • Gauge: Tracks current values that can go up or down (e.g., memory usage).
  • Counter: Tracks cumulative totals that only increase (e.g., total HTTP requests).
  • Histogram: Tracks distributions across buckets (e.g., request latencies).
  • Summary: Tracks distributions as quantiles/percentiles (e.g., request latencies).
• Metric Types in Instrumentation
  • Gauge: Use methods like Set(), Inc(), Dec(), Add(), Sub().
  • Counter: Use methods like Inc(), Add().
  • Histogram: Configure buckets, use Observe() to categorize values.
  • Summary: Specify quantiles, use Observe() to track values.