I know several experienced people across different companies that have gotten rid of their staging server and now do CD straight to production. I'm considering the same as it mostly just create deploy bureaucracy.
Obviously, you need good tests. Anytime you have a problem in production that did not break your tests, that's a problem. Goes without saying really.
You need insight into your servers. That means monitoring, hardware telemetry, application telemetry, and logging. These are very different things that get confused a lot because they can all be facilitated with a good logging infrastructure. We use things like metricbeat, filebeat, auditbeat, logstash, and kibana. Great stuff; and mission critical if you don't want to run blind.
Stuff will happen anyway and you need to not get hung up on it and instead just move forward with a fix or at least a change that takes out whatever is broken. Learn from it and fix your tests as well of course. What matters here is the mean time between issues and the mean time to get the fix out. If this happens on a daily/hourly basis, fix it.
No rollbacks though. Don't deploy versions older than what you just deployed. If something breaks, you fix it with a new version. So instead of restoring some git hash, instead just do a git revert or some other commit that rolls you back and let that roll through your CI. This keeps your CD simple: it only rolls forward. Your version is always HEAD. If HEAD sucks, create a new HEAD that doesn't suck any way you can (hint, git revert gets the job done). Your fixes roll out the same way everything else does. No manual overrides.
Some things you shouldn't test for the firs time in production. You need infrastructure to facilitate testing e.g. db schema changes in a sandbox. These are comparatively rare that you don't need a dedicated environment running 24x7 but it sure is nice to spin some sandbox up on demand and test some things out. This requires automation. Most projects I've been on don't have the ability to this at all and rely on staging instead. Staging in those projects is usually a mess.
I would argue that you should be able to test this locally. When we run locally it runs every schema change script on a local containerised DB. When you make a change just TDD the change like normal.
Sure works for toy databases. But that alter table that works fine locally might completely cripple your production DB. Once it does, all the fixes are ugly. Some things just resist closed world assumptions.
I know several experienced people across different companies that have gotten rid of their staging server and now do CD straight to production. I'm considering the same as it mostly just create deploy bureaucracy.
Nice comments
I would argue that you should be able to test this locally. When we run locally it runs every schema change script on a local containerised DB. When you make a change just TDD the change like normal.
Sure works for toy databases. But that alter table that works fine locally might completely cripple your production DB. Once it does, all the fixes are ugly. Some things just resist closed world assumptions.
+1 to this and not just the obvious alter tables.