loading...
Cover image for 🚒 When do you ship? (Technical or Business/Product reasons acceptable)

🚒 When do you ship? (Technical or Business/Product reasons acceptable)

waylonwalker profile image Waylon Walker ・1 min read

Do you have a checklist of requirements? (Technical or Business/Product)

πŸ‘‡ Discuss πŸ‘‡

Discussion

markdown guide
 

At my current employer (Root insurance), we currently deploy our server code at least twice a day and release mobile app once a week (Apple's review process doesn't really allow a faster cadence). In my time there I've seen us move from deploying once in whenever-something-needs-to-get-out, to once a week, to once a day, to now twice a day, and probably will accelerate even more as we grow.

It's an amazing feeling only being a few hours from shipping something and I have huge respect for the engineers who made our deployment process smooth as butter.

 

That is amazing!!

When you do you ship new features? As soon as they are ready? When stakeholder approve? When they pass certain tests?

 

Everything needs to pass automated unit/integration/system tests and be approved by 2 engineers before it can merge into the main branch. Everything on the main branch will be shipped at the next deploy (all engineers have the capability to deploy outside of the regularly scheduled deploys; the schedule is the minimum cadence).

For things that need to be timed (like to coordinate with a third-party or announced launch-time), we'll often use a feature flag controlled by an environment variable (so we can enable it at any time independent of a deploy).

Everything is real-time monitored for health. Rolling a deploy back to a previous "good" state is even easier than deploying. Stakeholders should be included through the development process (as possible); but if they end up not liking something after seeing it live, a git revert and a few hours will take them back to their previous system (and everything going in front of the general public is A/B tested with control over how much of the population gets an experience)

That sounds like a dream setup!

Well, we're hiring at a 120% rate ;) and fully remote until the end of the year at least.

root.engineering/

 

Roughly once a quarter, though we implement individual features such that in theory, we could go to prod whenever without breaking things.

The motivation behind this is purely business. Since it is a very large healthcare organization, we need at least 2 weeks of process and paperwork and approvals to put any code into production. And our client agreements require a week or two heads up before putting out new features so they can get their support and training docs ready for it, too.

Our release checklist is... quite large. About a dozen tasks for QA and product, two dozen for dev, and a dozen for architecture.

 

What kind of things are in the checklist? Are they purely technical or are some more business/product manager focused? Are they things that can be checked by automated testing?

 

It's all documentation.

Are the licenses of dependencies all on the up and up? Did you run security scans? Did you run automated tests? Did you run manual tests? Did you notify internal clients? Did you notify external clients? Are release notes done? Are bugs prioritized for the release done? Has design signed off on the release features? Are the epics in Jira finished for the release? Is the firewall configured correctly? Are the architecture settings correct for the level of data the application has (PHI, PII, PCI, etc)? etc etc etc

Almost every item links further to the Jira ticket to do that thing (Did you dry run the release? <link to Jira issue for deploying to stage>) and/or a link to the Confluence doc that elaborates more (Did you do the internal write up for testing the release candidate? <link to big ol' testing summary doc> <link to Jira issue for regression testing>)

The act of doing the things can be automated (especially the more on the nose stuff like did you run automated e2e or did you audit dependency vulnerabilities) but the write-ups and forms for the results to put in the checklist still take time. We used to joke that regression testing takes an hour while filling out the test summary form with what we did in that hour took 2 days.

 

I our team every PR gets deployed to production, so ~ 5 times a day. We use feature flags, so that doesn't necessarily mean that our users will see the changes right away.

 

Even with the feature flags is it a really good thing to deploy every 5 minutes in the production server. Curious to know how you will manage downtime between the deployment (I mean the service restart)

 

We don't have downtime when we deploy, we use a load balanced deploys with AWS beanstalk.

But yes, you want to deploy often, it will make your team more efficient, and generally it will reduce the number of bugs

 

Never on fridays!! πŸ™‚