In an effort to better protect our production data, last year we choose to setup a readonly production console. This allows developers to poke around production data without having to worry that they might accidentally change something they shouldn't. In this post, I will break down exactly what we did in order to accomplish this for our Ruby on Rails application.

Before I dive into the specifics for each database, I want to first mention that we use a completely separate server for console access. Using a separate server allows us to tweak application settings in order to achieve readonly access. In order to deploy these changes we use Ansible. When Ansible runs a deploy it looks for the console box tag to know what settings and configs need to be deployed to that particular box.

MySQL

The first thing we did was setup a user with readonly access in MySQL. Then, in order to make our application readonly for MySQL, all we simply had to do was put the readonly user credentials in our database.yml file on our console server box.

production:
  adapter: mysql2
  encoding: utf8
  reconnect: true
  pool: 16
  database: "prod_db"
  username: "readonly"
  password: "you_wish"
  host: "127.0.0.1"
  strict: false

Now any time someone opens up a Rails console on our console server it is automatically using the readonly credentials from the config.

However, there are still times when we want to be able to allow devs to edit data via the console. In order to accomplish this, we setup a bash script which is used to open a Rails console. In this bash script we choose to use a lesser known feature that Rails offers, DATABASE_URL. If you set the DATABASE_URL variable in your environment, Rails will use it to connect to your database rather than reading from your database.yml file. This allows us to override our database configs when we need to. We set it up in our bash script like so:

#!/bin/bash
cd /application_path
if [ "$1" = 'write' ]; then
  export DATABASE_URL="mysql2://write_username:write_password@host/db_name"
fi
RAILS_ENV=production /usr/local/bin/bundler exec rails console

Now if a developer needs to edit data they can simply open up a write console using the command console write.

Redis

To handle setting up Redis as readonly we choose to override our Redis client to explicitly block any write commands. Since we have a Ruby on Rails application we use the redis-rb gem in order to talk to Redis. To block write commands we first collected all the commands that were write based by calling the command method on our Redis client. For reference, Rails.cache.data will simply give you your Redis client.

dev> Rails.cache.data
=> #<Redis client v4.0.3 for redis://127.0.0.1:6379/15>

The command method will return an array of all the commands your Redis instance will respond to along with some additional information about each command.

dev> Rails.cache.data.command.first(3)
=> [["expireat", 3, ["write", "fast"], 1, 1, 1],
 ["setnx", 3, ["write", "denyoom", "fast"], 1, 1, 1],
 ["getrange", 4, ["readonly"], 1, 1, 1]]

To filter out only the write commands we simply checked for the "write" value in the list of command attributes.

WRITE_COMMANDS = Rails.cache.data.command.map { |a| a[0] if a[2].include?('write') }.compact.to_set

Once we had a list of write commands, we overrode the process method in our gem to raise an error if any of those methods were called.

def process(commands)
  if commands.flatten.any? { |c| WRITE_COMMANDS.include?(c.to_s) }
    raise NotImplementedError, "REDIS_ACCESS_MODE is set to 'readonly', disallowing writes"
  end

  # additional method logic
end

Those two pieces allow us to block Redis write commands. But, the question still remains, how do we block those write commands ONLY on our console box? Once again, we turned to our environment variables and our bash console script. In our console script we set our environment variables based on if the console was the default readonly or if it was a write console.

#!/bin/bash
cd /application_path
if [ "$1" = 'write' ]; then
  export DATABASE_URL="mysql2://write_username:write_password@host/db_name"
  export REDIS_ACCESS_MODE=""
else
  export REDIS_ACCESS_MODE="readonly"
fi
RAILS_ENV=production /usr/local/bin/bundler exec rails console

Then, in our redis.rb initializer file in our application, we monkey patched the process method to return an error if a write command was called in readonly access mode.

if ENV['REDIS_ACCESS_MODE'] == 'readonly'
  class Redis
    class Client
      WRITE_COMMANDS = ::Rails.cache.data.command.map { |a| a[0] if a[2].include?('write') }.compact.to_set.freeze

      def process(commands)
        if commands.flatten.any? { |c| WRITE_COMMANDS.include?(c.to_s) }
          raise NotImplementedError, "REDIS_ACCESS_MODE is set to 'readonly', disallowing writes"
        end

        # additional method logic 
      end
    end
  end
end

BOOM! The console box was now readonly for MySQL and for Redis by default. Only one piece of the puzzle was left, Elasticsearch.

Elasticsearch

Elasticsearch is at the cornerstone of our application so we needed that to be readonly as well. To talk to Elasticsearch we use the elasticsearch-ruby gem. Much the same way we did Redis, we found the core method used to make external requests to Elasticsearch, perform_resquest and patched it so that it would raise an error whenever a write method was executed. Since we talk to Elasticsearch using HTTP requests, the methods we wanted to block were PUT, POST, and DELETE.

module Elasticsearch
  module Transport
    class Client
      if ENV['ELASTICSEARCH_ACCESS_MODE'] == 'readonly'
        def perform_request(method, path, params={}, body=nil, headers=nil)
          raise 'Elasticsearch is in readonly mode.' if method.to_s.match?(/PUT|POST|DELETE/)
          method = @send_get_body_as if 'GET' == method && body
          transport.perform_request(method, path, params, body, headers)
        end
      end
    end
  end
end

Once again, we also choose to use an environment variable to determine whether or not we should be patching the perform_request method. We then took that environment variable and added it to our bash script. Our completed bash script looks like this:

#!/bin/bash
cd /application_path
if [ "$1" = 'write' ]; then
  export DATABASE_URL="mysql2://write_username:write_password@host/db_name"
  export REDIS_ACCESS_MODE=""
  export ELASTICSEARCH_ACCESS_MODE=""
else
  export REDIS_ACCESS_MODE="readonly"
  export ELASTICSEARCH_ACCESS_MODE="readonly"
fi
RAILS_ENV=production /usr/local/bin/bundler exec rails console

This script ensures that when a dev or support person is opening a console using the console command, by default it will be readonly. When necessary, they can call console write if they need to update any data. Even though it is very easy to open a write console, the vast majority of the time people are working in readonly consoles. The readonly consoles have proven themselves many times over by saving people from making silly mistakes while browsing production data.

Other Options

There are many other ways to approach data safety when it comes to working with production data. This is just one approach and the one we have chosen to use at Kenna. One other very popular option is to make a replica, or clone, of your data and then allow people to run whatever queries they want against that replica or clone. The downside to this is that getting an up-to-date replica every time you need it can be time consuming depending on the size of your dataset. In addition, cloning a single database such as MySQL is pretty straightforward. However, when you are working with 3 different data stores, such as we do at Kenna, it is a lot more effort to replicate all of that data together.

At Kenna, devs have the ability to clone production data, but it is only available in MySQL at the moment. Normally, a MySQL clone is only used to check high risk migrations or scripts that are going to change a lot of production data at once. Beyond that, we have found our readonly console has worked great for our use case because the majority of the time devs and support simply want to look at up-to-date production data.

Hope you found this post useful! As always, please let me know if you have any questions! 🤗