So probably for me the time I tried an update at 4am, got the wrong branch merged and sent 100 text messages to every user of an app. The app was from a bank. So yeah, I woke 30,000 people up at 4am by having their bank ring them. This incident did make it to Twitter. Ugh. Didn't lose the client BTW and the app continued to be successful. Lucky.
Merging PRs should be reviewed by ANYONE. I now keep rabbits, a rabbit would do.
Checklist for "very late at night" deploys including being asked questions by someone, somewhere in the world
I don't know - learn to live with the shame I guess lol
This is a big reason why I always advocate for updates/deployments during normal business hours. I know, I know - a lotta businesses would still balk at such a request. But I'm now a firm believer that, if you've done proper QA/testing/best-practices, you should be able to deploy at almost any time.
I'm not naive. I realize that, even in the best of scenarios, you can still bork a deployment. But even in those cases, would you rather have an emergency when all of the potential support personnel are at work and immediately available to help? Or would you like to be trying to call around the support tree at 4AM trying to find someone who will pick up??
As long as you can do zero downtime I agree! That's often the driver for out of office hours. With this client they mandated an update window between 1am and 6am if there was downtime...
Ahh... yes. I agree. If your update/change/deployment requires downtime, then yeah, there's no way to get that kind of authorization during normal business hours.
Do you remember what you were updating at 4am? I can't even imagine waking up to 100 text messages from my bank. I would immediately assume all of my money was gone.
Well yeah, ummm, I was updating the thing that stopped notification messages going out in quiet hours - at the time there was a problem with 3rd party services that if they would send in a window, but would retry out of hours. We'd written our own. I had been testing it with something that just sent lots of messages to known app installs to prove they didn't make it. I got my wires crossed on the release test, just stupid, and left in the "send test messages" code. I'm a bloody idiot.
Fortunately it was the marketing/voucher app for the bank, so while a bunch of people were very unhappy, and a bunch turned off notifications - it didn't actually suffer as much as I thought it would at around 5am, with a glass of whisky and a lot of cigarettes...
So probably for me the time I tried an update at 4am, got the wrong branch merged and sent 100 text messages to every user of an app. The app was from a bank. So yeah, I woke 30,000 people up at 4am by having their bank ring them. This incident did make it to Twitter. Ugh. Didn't lose the client BTW and the app continued to be successful. Lucky.
This is a big reason why I always advocate for updates/deployments during normal business hours. I know, I know - a lotta businesses would still balk at such a request. But I'm now a firm believer that, if you've done proper QA/testing/best-practices, you should be able to deploy at almost any time.
I'm not naive. I realize that, even in the best of scenarios, you can still bork a deployment. But even in those cases, would you rather have an emergency when all of the potential support personnel are at work and immediately available to help? Or would you like to be trying to call around the support tree at 4AM trying to find someone who will pick up??
As long as you can do zero downtime I agree! That's often the driver for out of office hours. With this client they mandated an update window between 1am and 6am if there was downtime...
Ahh... yes. I agree. If your update/change/deployment requires downtime, then yeah, there's no way to get that kind of authorization during normal business hours.
“A rabbit would do” 🤣🤣🤣
💀
Do you remember what you were updating at 4am? I can't even imagine waking up to 100 text messages from my bank. I would immediately assume all of my money was gone.
Well yeah, ummm, I was updating the thing that stopped notification messages going out in quiet hours - at the time there was a problem with 3rd party services that if they would send in a window, but would retry out of hours. We'd written our own. I had been testing it with something that just sent lots of messages to known app installs to prove they didn't make it. I got my wires crossed on the release test, just stupid, and left in the "send test messages" code. I'm a bloody idiot.
Fortunately it was the marketing/voucher app for the bank, so while a bunch of people were very unhappy, and a bunch turned off notifications - it didn't actually suffer as much as I thought it would at around 5am, with a glass of whisky and a lot of cigarettes...
The irony of the fact that the feature was for not sending after hour texts makes me laugh 😂😂😂