This post originally appeared on Arjun Rajkumar's blog.
This past week I got an error which as it turn out originated because Rails failed with its built-in Active Record validation checkers.
I have this has_many association set up in my database - a Website
has many pages
class Website < ApplicationRecord
has_many :pages, -> { order("created_at DESC")} , dependent: :destroy
end
class Page < ActiveRecord::Base
belongs_to :website, touch: true, counter_cache: true
end
Each page
has a unique URL scoped to a website. This means that pages under a website shouldn't have the same URLs.
I’m using Sidekiq workers to crawl a website in the background, get all it's on-page links and add those link as pages to the website. Sidekiq is multi-threaded - and runs thousands of jobs at the same time. Each of these jobs is creating a page with the crawled link as the URL. So, it may happen that two different jobs creates two pages with the same URL value at the same time, if it discovers two identical links on the website.
And this is exactly what happened.
Here are a few things I tried that didn’t stop Sidekiq from creating duplicate pages under the same website.
These didn't work.
#Active Record's built-in Validation checkers
class Page < ActiveRecord::Base
validates_uniqueness_of :url, :scope => :website_id
belongs_to :website, touch: true, counter_cache: true
end
#Transactions
ActiveRecord::Base.transaction do
website.pages.create(url: new_url)
end
This worked!
What worked instead was pretty simple.
Adding a unique index directly in the database solved this error.
class AddIndexToPageUrls < ActiveRecord::Migration[5.2]
def change
add_index :pages, [:website_id, :url], unique: true
end
end
And this works 100% of the time. Rails has documented this properly in their docs, and I fixed the error quickly once I knew what was causing it. But diagnosing this error initially to understand why this is happening took time.
I hope this helps someone in the future.
Top comments (0)