I know, how dare I suggest running a script in production. I am a Site Reliability Engineer, I should never condone such craziness. But truth is,...
For further actions, you may consider blocking this person and/or reporting abuse
When I do this (which is more often than I'd admit in an interview) I'll usually do something like this:
Excuse my butchered pseudo-ruby, I don't use it so am inferring from yours. What I mean, though, is I display what would be called as a kind of dry-run, and short-circuit the rest of the loop. Then when I'm ready, I uncomment the "real" line.
That is definitely another great way to try it out first before actually doing it. When I "dry run" a script I usually run the script for a single object, check that it looks how I expect and then l let the script loose on the rest.
I recently came across this gem data-migrate which allows you to do data migrations like you do schema migrations.
One reason I am wary about doing data migrations in schema migrations is bc sometimes they can take a large amount of time and if your schema migrations are executed inline with your deploy pipeline that can block a deploy. At DEV we use DataUpdateScripts which is pretty similar but it runs asynchronously in the background.
Set Up Framework For Running Data Update Scripts #6025
What type of PR is this? (check all applicable)
Description
While hooking up our first Elasticsearch model Tags, I realized that in order for search to work I would have to manually reindex all of the tags in our database before the search code went live. This is not a huge deal for us, but it means an extra undocumented step that others who are using this codebase might miss. This framework would give us the ability to run scripts like this the same way we run migrations.
A script like this would be deployed ahead of using the new data. When the app deploys or a local environment is updated the
DataUpdateWorker
would look at all the files in thedata_update_scripts
folder. Any file that has not been run, ie is not in our database, it will create a record for and then call run on that class.We used something like this at my prior company because we had to keep 5+ VPCs data in sync and it worked out really well.
Why not use migrations? The reason we may want to separate this from migrations is so that it is not tied to our deploy process. In the past, I have had scripts that take hours to run bc they touch a lot of data and you don't want that holding up a deploy. This way a deploy goes out, kicks off a worker, and that worker does its thing in the background for however long it needs.
THOUGHTS?!
Added to documentation?
If people are on board with this approach I will add the necessary documentation to this branch as well
Awesome !!!
But how do you keep track of the order in which script ran? Do the script file names get timestamp attached to them similar to regular migrations?
Yep! That is exactly how it works, same as migrations. Here you can see a list of our scripts github.com/thepracticaldev/dev.to/...
If the task takes a good amount of time, you have risks of being disconnected from either the SSH session, your internet provider, etc.
To avoid this, I recommended running these scripts (although, I do not recommend running scripts like this at all) using screen. It's super easy to use:
screen
Ctrl
+a
followed byd
to detach the sessionscreen -r
Screen has a lot of awesome things, make sure to check out the man page.
I actually learned about tmux one time I had to run a script that took hours to finish, total savior <3
Oh man, YES! screen or tmux sessions are a must as well for long-running scripts!
Screen is a must for SSH... or even local if you want to be minimal in your shells.
Hi, ๐, By the way are you talking about
SQL Script or YML Script
. BecauseRuby
is a programming language right ?Programming languages can still be used to write scripts ;)
What kind of script is used in this post. Is there a documentation or tutorial. so I can learn?
The script in this post is written in Ruby. I highly recommend Googling Ruby Tutorial and you will find MANY resources to help you learn Ruby in whatever learning style is best for you.
Nope, I am talking about Ruby. In this sense, I am using "script" to define a small chunk of Ruby code that is run standalone in a console.
I learned all these lessons the hard way by running scripts in production without applying them in the first place. I think I'd still have hair if I'd read this kind of article before. So thank you, it will save lives for sure!
A technique I used recently was to have my script write out two SQL files. One with all the fixes, and the other to revert everything back to how it was. The script took a long time to run but were completely non destructive. The SQL could actually then be examined and run on a copy of the database and the results checked, both the fixing SQL and the reversion (just check that both databases are the same again). Then just run the fix SQL on production.
great article. while this is about a script for changing data, the points are also applicable to writing scripts for transforming code / codemods. thanks for sharing molly.