How I migrated from Flask-Security(-Too) to Devise

#rails #flask #webdev

Recently I started a rewrite of one of my smallish projects (in terms of complexity, but with ~200 users, so I must handle that with care) from Flask to Ruby on Rails.
Why? I'm planning on writing another article about that, so I don't want to go into details here, but in short, I believe it will enable me to write the app both faster and better. But there are some hurdles to overcome to get there.

For some time, I ignored the authentication part of the app. So today, I made my first attempt at making that work.

In Flask, I used the Flask-Security-Too package from the start. I never tried to implement any authentication logic myself (as you probably never should for, you know, security reasons).
I knew that in Rails, I could use the devise gem.
But it means jumping from one black box into another. I knew that it might bring some difficulties.

Warning: I'm no expert on security by any means! I understand really little about this, just enough to make it work. There might be some security concerns I'm not aware of. Read this just as a story, not a tutorial!

There was some little work involved in correctly adding columns to the database to support devise (as some column names are the same in both libraries and some are not), but not that much. The most annoying is that I had a password column, and that caused some problems with Rails/devise. But after some tries with migrations, I got sorted that out.

The main problem, as expected, was being able to validate the same hashes in Rails that Flask-Security created automatically for me.

The Problem - Password hashes

Short aside if this is new for you: what are password hashes?
It would be very unwise to save passwords to the database as-is. I certainly don't want to have that responsibility on me.
So instead password hashes are usually stored in the database, and on login attempt hash is calculated from the provided password and compared to the saved one.
So the app knows the same password was provided as when the user registered, but is none the wiser about the actual value.

When I looked into my database, I had hashes like $2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46 there.

Ok, back on track. The trouble is that there are many different ways to hash passwords.
You can use different algorithms (sha256, bcrypt, argon,..), they can have different configurations, there may be some salt or pepper involved,..

There's basically NO CHANCE that a different library (devise in my case) will hash, and therefore validate, passwords the same way as some other (flask-security for me).

So it was no surprise that after adding devise to my project, logging in with my credentials wasn't working.

The easy way out for me would be the following - just toss out all passwords and send mail to my users that they need to use the Forgotten password feature. Well, not gonna happen, thank you very much.

So I needed to find out way to migrate successfully.

There were two parts to the solution:
1) Slightly changing mechanism for Devise-provided password-validation to validate using its built-in mechanism first, and if that fails, try our own validator, simulating how Flask-Security does validation.
2) Writing the validator

Allowing both Devise and Flask-Security-like validation

The first part is quite straightforward:

# app/models/user.rb
class User < ApplicationRecord
  # Include default devise modules. Others available are:
  # :confirmable, :lockable, :timeoutable, :trackable and :omniauthable
  devise :database_authenticatable, :registerable,
         :recoverable, :rememberable, :validatable

  (...)

  def valid_password?(password)
    if Devise::Encryptor.compare(self.class, encrypted_password, password)
      true
    elsif flask_security_compare(password)
      true
    else
      false
    end
  end

  (...)
end

It's probably pretty self-explanatory, I'm just overriding the default valid_password? method Devise is providing, adding another way to validate the password. Nice, let's move into the second part - how will flask_security_compare work?

Simulating Flask-Security hashing

Now this was a bit more difficult (took me a few hours to solve).

First, I needed to understand what Flask-Security is doing when hashing passwords.

Luckily, I was already using bcrypt as my hashing algorithm (as it's the default for Flask-Security), the same as Devise uses.
I needed to check that it is configured the same way, mainly the number of stretches (technique making hashing slower, thus making brute-force attacks more difficult).

Sidenote:
Remember the hash I showed you before? Here it is again $2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46._
$2a tells us it's bcrypt using 2a variation
$12 is the number of stretches

Now for a tricky part.

When the password is hashed using bcrypt, there's salt provided. That's a way to make pre-calculating hashes nearly impossible (preventing usage of the infamous rainbow tables).

Where does this salt come from?

It's generated randomly.

For a moment I was afraid that it meant I would be unable to simulate this - how can I recreate something random? Of course, that's not the case - bcrypt needs to keep this value around. And it does so directly in the hash string:

in $2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46, etyAllfaDgIDC61ZUS9UB. is the salt (22 characters after the initial part).
Only the rest (peoAIYpZborbU4T/6c8SejvrIhbIL46) is the actual hash!

Nice! And even nicer - I don't need to parse the value manually, I can use the library for that (as suggested by Copilot):

legacy_password_hash = "$2a$12$etyAllfaDgIDC61ZUS9UB.peoAIYpZborbU4T/6c8SejvrIhbIL46"
bcrypt_password_hash = BCrypt::Password.new(legacy_password_hash)
puts bcrypt_password_hash.salt
# $2a$12$etyAllfaDgIDC61ZUS9UB.

Ok, we have a salt, let's hash some passwords!

hashed_password = ::BCrypt::Engine.hash_secret(password, bcrypt_password_hash.salt)

Well, I did get a result, but it was different than what I had in the database...

The reason is that Flask-Security uses one more security enhancement.

It doesn't use the password directly for hashing but encodes it to HMAC using a server-wide secret key (set as SECURITY_PASSWORD_SALT env variable).

I had to look into Flask-Security source code to find how exactly it's done:

def get_hmac(password: SB) -> bytes:
    """Returns a Base64 encoded HMAC+SHA512 of the password signed with
    the salt specified by *SECURITY_PASSWORD_SALT*.

    :param password: The password to sign
    """
    salt = config_value("PASSWORD_SALT")

    if salt is None:
        raise RuntimeError(
            "The configuration value `SECURITY_PASSWORD_SALT` must "
            "not be None when the value of `SECURITY_PASSWORD_HASH` is "
            'set to "%s"' % config_value("PASSWORD_HASH")
        )

    h = hmac.new(encode_string(salt), encode_string(password), hashlib.sha512)
    return base64.b64encode(h.digest())

and then recreated the algorithm in ruby:

def get_hmac(password)
    require 'openssl'
    require 'base64'

    salt = ENV['LEGACY_SECURITY_SALT']

    if salt.nil?
      raise "The configuration value `LEGACY_SECURITY_SALT` must not be None when the value of `SECURITY_PASSWORD_HASH` is set to #{ENV['SECURITY_PASSWORD_HASH']}"
    end

    digest = OpenSSL::Digest.new('sha512')
    hmac = OpenSSL::HMAC.digest(digest, salt, password)
    Base64.encode64(hmac)
  end

and used that to write the flask_security_compare function I need:

def flask_security_compare(password)
    bcrypt_password = BCrypt::Password.new(legacy_password_hash)

    hmaced_password = User.get_hmac(password)
    hashed_password = ::BCrypt::Engine.hash_secret(hmaced_password, bcrypt_password.salt)

    Devise.secure_compare(hashed_password, encrypted_password)
end

And it still didn't work.
I was a little desperate by then, but after some debugging and comparing of strings I found out small problem:
The ruby implementation sometimes added \n symbols into the hmaced_password.
Fast fix is this: Base64.encode64(hmac).chomp.gsub(/\n/, '').

Now I'm just curious if this works with every password there is in my database. Let's hope.
And maybe show a banner to users warning them that it might happen.

Oh, I forgot one more improvement - if the user logs in using the "old mechanism", it can rehash the password to update the hash. This way, I can remove the flask branch later (like in a year or so). I already did something similar when I was migrating from SHA256 to the bcrypt algorithm in my older project.

Now I'm done.

So, I learned a bit about how password hashing works in Flask-Security, which was interesting. And I'm really happy that I can continue with my rewrite to Rails, knowing that users will be able to log in using their old passwords.

Again, this is not a tutorial. Please, if you do some authentication stuff, be careful. And if you find any mistakes here, let me know in the comments!

Update:
I realized I didn't address properly one potential downside to this. As password validation tries two way of hashing, it makes guessing password easier, as there are potentially two strings that produce the desired hash. That's not great, as it's lowering security of our authentication, but as we use otherwise safe way, it shouldn't do much damage (half the time for brute-forcing will still be a lot). We could migitage this by having flask_password_hash and password_hash as different fields, and only allowing one of them to be present.