DEV Community

Blue/Green Node.js Deploys with NGINX

Justin on September 10, 2020

I recently faced a situation where I needed to deploy Node.js apps to my own servers1. When I started this endeavor, I tried to find helpful materi...
Collapse
 
edgarbonet profile image
Edgar Bonet

Thanks for this very instructive article! Here are a couple of bash tricks you may use to slightly simplify your scripts:

  • Instead of using a retry loop, the random numbers can be generated with
  echo $((floor + RANDOM % (range - floor + 1)))
  • A very old trick used to avoid grepping-out grep:
  grep "[n]ginx: worker process"

But note that you may be able to avoid grep if you can ask ps to display only the relevant processes. On Ubuntu, you may select the process that are owned by www-data:

echo $(ps --no-header -u www-data | wc -l)
Collapse
 
jwkicklighter profile image
Jordan Kicklighter

I think there was a great learning experience in all of this, but here are some resources to consider researching if you haven't:

  • Ansible - makes managing remote servers easier, especially the bit about running a script remotely
  • Haproxy - reverse proxy to use instead of nginx, might be better at swapping backends gracefully
  • Kubernetes (potentially too heavy for this use, but handles basically everything you built here)

Also just a note from the outside about the port switching: your solution chooses a random port and checks if it is available. You could apply this logic to the original idea of swapping between 2 ports. Port A is the default, check if Port A is available. If it is, use it. If it isn't, use Port B. No state to keep track of, you're just dynamically choosing whether to use the port or to use an alternate one.

Collapse
 
justincy profile image
Justin • Edited

Thank you. That's helpful. I particularly like your idea about how to get rid of the random port generation. That would remove the need for cleaning up the old code (because I would only ever have two copies of it on the machine, instead of up to 1,000) and would open the path to quick rollbacks.

Collapse
 
jwkicklighter profile image
Jordan Kicklighter

That's a great point about the quick rollbacks. It would make the blue/green nature even better.

Collapse
 
douglasvbarone profile image
douglasvbarone

Did you check CapRover (caprover.com)? It's basically a self-hosted Heroku 🤷‍♂️. I like it. You can even manage a cluster with it.

Collapse
 
justincy profile image
Justin

Holy smokes! I haven't heard of that before. I gotta try it out. Thank you!

Collapse
 
utking profile image
Gennady

Jordan already provided a list of tools that can improve the process. I just want to add about several moments in your process:

  • if there is a daemon manager (service or systemctl), it's better to use it instead of nginx -s SIGNAL because the manager won't know anything about such reloads and won't be able to interact with the running process anymore.
  • checking when to remove the old code directory - since you know the old port, why don't just check if there are open connections to them? No connections - remove the code
  • the 1-second delay - when you do the switch, are you sure the new deploy is ready to be served? Perhaps it makes sense to make a readiness request before the switch? Anyway, you can use tcpdump (or anything similar) to record network traffic, with port/protocol filtering, and figure out to which backend (old or new port) is was made and failed to complete.
  • to run tasks in parallel, you can use parallel-ssh
  • speaking about one failed host - there are basically only two options if your scripts remove the old code and do it in parallel on several hosts. You can re-deploy for all or just for the failed host. Even rollback here is a re-deploy. By the way, you can keep several old release folders to stop worrying about the right time to remove the previous release and to be able to revert your deployment (however you would need to track which is the previous one, but it is solvable)
Collapse
 
justincy profile image
Justin

Thanks for taking the time to read this post and to share your recommendations for improvements. They're helpful.

Collapse
 
joehonton profile image
Joe Honton

Part of your solution could be tightened up with this simple command.

    setcap 'cap_net_bind_service=+ep' /usr/bin/node

Here's the full story. How to Configure Node.js to Use Port 443
"Granting server access to well-known, privileged Linux system ports."

Collapse
 
m4r4v profile image
m4r4v

Great article it looks like you covered everything up.
I do have a question for you, how much time did spent to put all this together?

Collapse
 
justincy profile image
Justin

It took me 3-4 days to figure it out and code it up and then 1 day to write the blog post about it (because I was sick).