Pipeline all the things: Redis performance boost at application level

While researching for my latest tutorial at DigitalOcean, I had the chance to play around with Redis and I've run all sorts of benchmark tests on a dozen of servers.

Something that was quite surprising to me was how many more requests per second your Redis server can handle if you just pack some commands together, in what is called pipelining.

This happens because even though Redis is quite fast, a request consumes other resources from the system. It's like buying several single bread slices each wrapped in a package, instead of getting a full loaf.

A regular benchmark with the redis-benchmark tool would go like this:

redis-benchmark -t get,set -q

SET: 95785.44 requests per second
GET: 97370.98 requests per second

Now this will simulate a pipeline of 16 commands:

redis-benchmark -t get,set -q -P 16

SET: 877193.00 requests per second
GET: 1351351.38 requests per second

From 97,370.90 to 1,351,351.38 requests per second!!!

This is an optimization that needs to happen at code level, just so we're clear. You can find more about pipelining with Redis in their official documentation. To give you an idea of how the code looks like, here's an actual Ruby code taken from that documentation, which makes a little benchmark test with and without pipelining:

require 'rubygems'
require 'redis'

def bench(descr)
    start = Time.now
    yield
    puts "#{descr} #{Time.now-start} seconds"
end

def without_pipelining
    r = Redis.new
    10000.times {
        r.ping
    }
end

def with_pipelining
    r = Redis.new
    r.pipelined {
        10000.times {
            r.ping
        }
    }
end

bench("without pipelining") {
    without_pipelining
}
bench("with pipelining") {
    with_pipelining
}

And here is the output (also copied from their docs):

without pipelining 1.185238 seconds
with pipelining 0.250783 seconds

You can also make use of this feature in PHP with the PHP Redis Client library.

Check the Redis clients page for clients supporting pipelining in other languages.

Cheers!