This post will describe step by step extraction process of a Rails engine from a monolithic application. This is a follow-up to my previous piece, where I explained the pros and cons of Rails engines as a solution for modularizing big monolithic applications. The full source code of application used for this post can be found here
Table Of Contents
Introduction
To set a scene, let's imagine that we are earls (aka chieftains) of Viking communities. As a forward-minded leader, we've already created a simple social media website for our people. It allows townsmen to enlist as warriors or farmers, and browse lists of each profession, to find companions or suppliers for upcoming raids. It also has a separate section where special e-seer gives its daily weather prophecies. It all works together, but now we've decided that it is time to improve architecture, and maybe share some parts of our systems with other Viking communities as an open source. With no further ado let's get to work, and extract seer module into the engine.
Engine preparation
Before We can move any of the existing code from the parent app into the engine, we need to initialize it, set up a unit test framework and configure development and test database connections.
Initialize engine
To initialize engine We can use Rails generator with this command.
$ rails plugin new seer --mountable --dummy-path=spec/dummy --skip-test --databse=postgresql --skip-git
It accepts a number of flags that adjust the generated engine scaffold to our needs. Most important ones are:
-
--mountable
, it instructs the generator that we want a 'mountable' engine, and it should be namespace isolated from the host application. This prevents constants name collision, between the code of the host app and the engine. -
--dummy-path
, sets the location of 'dummy' app generated by Rails for unit test purposes. -
--skip-test
, instructs generator to not scaffold tests, it is useful if you prefer to use another test framework over Minitest. For me preferred one is Rspec (which will be configured later on). -
--database
, configures database adapter which will be used. It is the best to set it, to the same as the one used by the application from which we want to extract an engine -
--skip-git
, prevents new git repository initialization. I prefer to do that at the very end of a whole process when I'm moving the engine from a subdirectory of the parent app into a separate repository. Before that, I work on a separate branch of the main repository and compare changes in the merge (aka pull) request.
After the command is executed, the generator scaffolds the requested engine in a subdirectory of the parent app. As the generator is working we can see typical output upon the terminal. Among standard parts like controllers, models, we can also notice a few additional files has been generated
create lib/seer.rb
create lib/seer/engine.rb
create seer.gemspec
create lib/seer/version.rb
vendor_app spec/dummy
Those files are really interesting for two reasons, one they clearly indicate that engine is a specialized gem, and two each of them has a special role:
-
lib/seer.rb
is an entry point to the engine, this file will be required when the host application of the engine loads its dependencies. It can be used to make engines external dependencies configurable and loadslib/seer/engine.rb
-
lib/seer/engine.rb
informs the parent application that this given gem is an engine. It also can be used to configure engine parameters (similarconfig/application.rb
file is used in a standard Rails app). In fact, the generator already configured one parameter for us:isolate_namespace Seer
, it declares that the whole engine is isolated in the namespace named after the engine. -
seer.gemspec
contains metadata about engine and gems upon which it depends. -
lib/seer/version.rb
this file declares the current version of the engine -
spec/dummy
is previously mentioned 'dummy' Rails app which will be used in unit tests of the engine
Rspec initialization
After the engine is initialized, we should configure a unit test framework. Although there is no limitation on which one we can use, it is the most convenient to choose the same one, as it is used in the parent app. That way, we can simply reuse most of the already existing unit test code. At first, we need to add development dependencies into the gemspec
file.
Gem::Specification.new do |spec|
....
spec.test_files = Dir["spec/**/*"]
spec.add_development_dependency 'rspec-rails'
spec.add_development_dependency 'factory_bot_rails'
spec.add_development_dependency 'dotenv-rails'
end
Then we can proceed lib/seer/engine.rb
and set up generators.
module Seer
class Engine < ::Rails::Engine
isolate_namespace Seer
config.generators do |generators|
generators.test_framework :rspec
generators.fixture_replacement :factory_bot
generators.factory_bot dir: 'spec/factories'
end
end
end
And at least we should install new dependencies and RSpec
developer@macbook engines-directory $ bundle install
developer@macbook engines-directory $ rails g rspec:install
create .rspec
exist spec
create spec/spec_helper.rb
create spec/rails_helper.rb
After rails_helper.rb
file gets created, we have to adjust the environment path in it, to point onto the 'dummy' app.
require File.expand_path('../dummy/config/environment.rb', __FILE__)
And now we are ready for the next step.
Database connection configuration
A first step to configure 'dummy' app database connection is to adjust database.yml
file in 'dummy' db
directory, for that most of the database.yml
file from the parent app can be reused, except for databases names.
default: &default
adapter: postgresql
host: <%= ENV.fetch('POSTGRES_HOST') %>
username: <%= ENV.fetch('POSTGRES_USER') %>
password: <%= ENV.fetch('POSTGRES_PASSWORD') %>
port: <%= ENV.fetch('POSTGRES_PORT') %>
development:
<<: *default
database: <%= ENV.fetch('POSTGRES_DB') %>_development
test:
<<: *default
database: <%= ENV.fetch('POSTGRES_DB') %>_test
I've decided to store sensitive data outside the git repository, inside environment variables. In this case, we should prepare a tool to load locally environment variables for us. I've chosen Dotenv, so I've created a .env
file in 'dummy' app directory and required dotenv
before dummy apps boots.
#seer/spec/dummy/config/application.rb
Bundler.require(*Rails.groups)
require "seer"
require 'dotenv-rails'
module Dummy
class Application < Rails::Application
...
At least we should execute two Rails database-related tasks.
developer@macbook engines-directory $ rails db:setup
developer@macbook engines-directory $ rails db:migrate
If all steps are correctly performed inside the 'dummy' app db
catalog, schema
file should get created.
After that, we are ready to take the next step.
Parent app preparation
With the initialized engine next, we need to plug it into the parent app, so the changes we are going to make Can be iteratively verified as we proceeded with code transfer.
Request test creation
Before we extract any piece of code, it will be very handy to ensure, that the whole system from an end-user perspective has not changed after the process is completed. To assure that, I prefer to add requests test for the extracted module.
require 'rails_helper'
RSpec.describe 'WeatherProphecies', type: :request do
let(:villager) { create(:villager) }
before { sign_in villager }
describe 'GET /weather_prophecies' do
it 'returns current prophecy' do
get weather_prophecies_path
expect(response.body).to include('rain')
end
end
end
Engine installation
To use newly created engine we need to include it in Gemfile
of thr parent app
source 'https://rubygems.org'
...
# engines
gem 'seer', path: 'seer'
And mount it in the routes.
Rails.application.routes.draw do
...
mount Seer::Engine, at: '/seer'
...
end
Code transfer
With the parent app and the engine ready, we can start moving pieces of code from one to another. I prefer to start from the database and continue up the layers until I got to the controllers. That way we have less external dependencies to handle at the time.
Migrations
In the end, the engine and the parent app will use the same database. However for test and development purposes, and also to make the engine truly mountable, even to other apps, we have to copy all migrations related to transferred database tables into the engine. While coping migrations from the parent, we should leave migrations names intact, so when the engine is installed those migrations will not get duplicated.
By default all tables of the engine are prefixed with the name of the engine (seer
in this case) since that could not be the case before the extraction when all existing migrations are copied to the engine, we must add new ones which will rename transferred tables with the correct prefix.
class RenameWeatherPropheciesToSeerWeatherProphecies < ActiveRecord::Migration[6.0]
def change
rename_table :weather_prophecies, :seer_weather_prophecies
end
end
To copy and apply the migrations of the engine to the parent app, we can use the following commands.
developer@macbook host-app-directory $ rails seer:install:migrations
NOTE: Migration 20170809130629_create_weather_prophecies.rb from seer has been skipped. Migration with the same name already exists.
developer@macbook host-app-directory $ rails db:migrate
Models
To transfer models from the parent into the engine, we can simply run the bash command, or use the cut and paste method.
developer@macbook host-app-directory $ mv app/models/weather_prophecy.rb seer/app/models/seer/weather_prophecy.rb
We should be careful to place each file in the correct directory, which represents the namespace of the engine (eg: seer/app/models/seer) otherwise, an autoloader will ignore them and we will get an error NameError: uninitialized constant
.
After each model with the corresponding spec is transferred, we need to wrap their code in the module of the engine.
module Seer
class WeatherProphecy < ApplicationRecord
end
end
Factories
When models and their specs are moved, the one thing we lack to make test back to the green state are factories.
To use FactoryBot at first we need to adjust rails_helper.rb
file and point set a path to a directory which will contain factories definitions
require 'factory_bot'
FactoryBot.definition_file_paths = [File.join(File.dirname(__FILE__), 'factories')]
FactoryBot.find_definitions
RSpec.configure do |config|
config.include FactoryBot::Syntax::Methods
...
end
Then we can move factory definitions just like we've done with the model files.
Rest of Rails parts
The rest of the Rails parts like controllers, jobs, channels can be moved the same way as we've transferred models.
As for the routes, they should be cut out from the routes.rb
file of the parent app and placed in the one that belongs to the engine.
Not standard Rails components
In this scenario I've created an additional services
layer, there is one gotcha with such an approach. To ensure that autoloader loads these parts of the code as well we can do one of two things, either we extend autoloader paths, or we place these additional layers in the app directory of the engine. I've chosen the latter for this scenario.
Besides that, we can move them like the rest.
External dependencies
The last finishing touch is to make external dependencies configurable.
Although while developing the engine, we could refer to parents app modules and classes, by using their qualified names (prefixed with ::
).
module Seer
class ApplicationController < ::ApplicationController
end
end
Unfortunately, this introduces thigh coupling between the engine and the parent app. To address that, and make our engine more flexible we should allow others to configure external dependencies upon engine initialization.
Because of that, we should add a module accessor which would store such dependencies.
# lib/seer.rb
module Seer
mattr_accessor :application_controller_class
def self.application_controller_class
@@application_controller_class.constantize
end
end
To configure them, we need to create an initializer in the parent app.
# parent_app/config/initializers/seer.rb
Seer.application_controller_class = 'ApplicationController'
Inside the engine, we can use them in the following manner.
class ApplicationController < Seer.application_controller_class
After this last step, our engine is ready to be used. We could leave it as a subdirectory of parents app, or move it to stand alone repository and use it freely in a number of apps.
As we've just accomplished this challenging task our system is better structuralized and easier to navigate, a test suite is running faster, it is easier to work in parallel on different parts of the system without worry about merge conflicts. We are now ready to celebrate our glory on a Viking feast.
Top comments (3)
Nicely written. Thanks Mikolaj!
What if there is an exact opposite scenario where you have a full-fledged rails application and want to use it as an engine to another app?
Like I have github.com/chatwoot/chatwoot and I want to build a marketing website on top of it without touching(or modify with minimal changes) the source code of that app(chatwoot)
So the webapp essentially will have Marketing stuff + Chatwoot
Thanks, used as good source for better understanding of Rails engines and as good starting point.
Awesome! A great example of how we can moduliraze our application, thanks for sharing! 🤟