Recording HikariCP Metrics in Clojure

HikariCP is a popular JDBC connection pool, battle-tested and exhibiting good performance. A Clojure wrapper does the things discussed in this post here. However, our project uses HikariCP directly.

Last year I needed to give us some additional insight into our application performance, and this turned out to be a short, fun exercise. So, after I dug into HikariCP, it looked like these two interfaces must be implemented.

• IMetricsTracker
• MetricsTrackerFactory

Implementing the interfaces

Using the incredibly simple clj-statsd library, we can write a small record to implement the IMetricsTracker.

(def pool-metrics-prefix "my-pool")

(defn make-key [db-name key] (clojure.string/join \. [pool-metrics-prefix db-name key]))

(defrecord StatsdMetricsTracker [db-name pool-name pool-stats]
  IMetricsTracker
  (recordConnectionCreatedMillis
    [_ connection-created-millis]
    (statsd/gauge (make-key db-name "wait-millis") connection-created-millis))
  (recordConnectionAcquiredNanos
    [_ elapsed-acquired-nanos]
    (statsd/gauge (make-key db-name "wait-nanos") elapsed-acquired-nanos))
  (recordConnectionUsageMillis
    [_ elaspsed-borrowed-millis]
    (statsd/gauge (make-key db-name "usage") elaspsed-borrowed-millis))
  (recordConnectionTimeout [_]
    (statsd/increment (make-key db-name "timeout"))))

The previous code is the core of the implementation, with the close function omitted. We will come back to that.

The most straightforward implementation of MetricsTrackerFactory would instantiate StatsdMetricsTracker and return it.

(defrecord StatsdMetricsTrackerFactory [db-name]
  MetricsTrackerFactory
  (create [_ pool-name pool-stats]
    (->StatsdMetricsTracker db-name pool-name pool-stats)))

At this point, we have enough code to glue to our application and report the following stats:

• milliseconds required to create a connection
• nanoseconds needed to acquire a connection
• milliseconds a connection has been borrowed
• cumulative number of connections timed out

I will leave the application glue to the reader but feel free to take the following code as a starting point.

(.setMetricsTrackerFactory datasource (->StatsdMetricsTrackerFactory db-name))

Gluing all the code in this section to your app should be sufficient to give you some helpful graphs:

There are more metrics we can grab.

Polling for additional metrics

HikariCP will "push" the metrics discussed in the last section, but there are some additional metrics that you can access. You will have to poll for these, however. Unfortunately, these metrics are less valuable and more challenging to report than I expected at first.

Starting is easy enough. Let's write a function to report these metrics.

(defn log-pool-stats [db-name pool-stats]
  (try
    (statsd/gauge (make-key db-name "total-connnections") (.getTotalConnections pool-stats))
    (statsd/gauge (make-key db-name "idle-connections") (.getIdleConnections pool-stats))
    (statsd/gauge (make-key db-name "active-connections") (.getActiveConnections pool-stats))
    (statsd/gauge (make-key db-name "pending-connections") (.getPendingThreads pool-stats))
    (catch Exception e
      (log/warn e "the database pool closed while we were logging metrics"))))

pool-stats needs to be an instance of StatsdMetricsTrackerFactory.

The most frustrating novelty I discovered about HikariCP is that it does not call create on the metrics tracker factory until it performs a query. Your solution to this problem may be different than mine - I made a promise to force log-pool-stats to wait. It looks something like this.

(def trackers (atom {}))

(defn log-pool-stats
  ([db-name]
  ;; The tracker is wrapped in a promise, so we do a blocking de-ref until it becomes available.
   (when-let [tracker (get @trackers db-name)]
     (when-let [pool-stats (:pool-stats @tracker)]
       (log-pool-stats db-name pool-stats))))
  ([db-name pool-stats]
   ...))

(defrecord StatsdMetricsTrackerFactory [db-name]
  MetricsTrackerFactory
  (create [_ pool-name pool-stats]
    (let [tracker (->StatsdMetricsTracker db-name pool-name pool-stats)]
      (deliver (get @trackers db-name) tracker)
      tracker)))

I have modified the create function to inject the StatsdMetricsTracker instance into trackers. I have again omitted some code for brevity, but clearly, you will need to baby trackers a bit before create is called. That is pretty easy to do with your glue code.

Takeaways

This project took an evening, and I enjoyed working on it. However, there turned out to be some funny state management problems related to closing all the resources cleanly and removing old data so that metrics tracking would play nice with our reloaded workflow.

We will be removing this code shortly because it provides little benefit compared to the APM metrics we get through datadog.

If someone has used tomekw/hikari-cp I would be happy to hear about your experience. It's doubtless a better solution than this home-grown thing anyway.