Today I wanted to set up a MySQL cluster with a leader and a replica using GTID replication. I copied some code I had for setting up consul clusters and modified the startup scripts thinking everything would continue to work. I was wrong.
For some reason the following line was failing with
dpkg lock issues
apt update && apt install --yes ruby
Some days I just want to throw the computer at the wall but race conditions are a fact of life and I should have been more careful with how I install things to avoid any random lock failures. So I modified the line to the following
export DEBIAN_FRONTEND="noninteractive" while ! (apt update; sleep 2; apt install --yes ruby); do echo "Failed to update and install ruby. Sleeping and retrying." sleep 2 done
Computers are basically Murphy's Law incarnate, if something can fail then it will fail. Design your scripts accordingly and put anything that can potentially fail inside a retry loop with messages to indicate what is going on.