Photo by Afif Ramdhasuma on Unsplash

Release Management: is your product ready for success?

You never get a second chance to make a first impression. So how do you make sure it’s a good one? This matter is constantly in the mind of release managers and product owners, because they know that once a product is set loose into the world all bets are off.

Although it may be tempting to release a product or new feature as soon as it is functional, users expect (and deserve) quality and polish. Before even reaching the beta test stage, a lot of non-technical work must be done: documentation must be written, security must be assessed, and business objectives must be established. These are some of the things we have learned about software release at Semaphore.

Project stages

From idea to release, we can break a software project into roughly three stages:

• Design and development: comprises everything from idea to MVP or working feature. We’ve discussed design patterns in other blog posts, so I won’t spend too much time on that at this point. All I will say is that there is no shortage of practices to help us uncover good designs, like writing acceptance criteria and following Test-Driven Development, Behavior-Driven Development, or Domain-Driven Development methodologies.
• Tech preview: the sharp blade of the reality check. We bring a few users to get their early feedback so we can make improvements and reduce uncertainty at the release.
• Release: also known as General Availability (GA). In this last stage, we make the release available to all users.

The stages are separated by checkpoints. Of which we have two kinds Definition of Ready and Definition of Done:

• Definition of Ready (DoR): these are all required tasks before users can access the system or experience the feature.
• Definition of Done (DoD): consists of all the follow-up tasks closing a stage, like gathering user feedback, completing outstanding tickets, or doing a retrospective.

We cannot enter a new project stage or consider it done until all the tasks in DoR and DoD are ticked off.

The ready checkpoint (DoR) dictates if users can be allowed into our system. Thus, the technical preview can begin only once we meet the DoR. It does not end until we’ve completed all the tasks in the DoD. The same thing happens during GA release.

The technical preview

The technical preview is an opportunity to get the user's feedback before the final release. We must bring them into the mix before evaluating whether a feature or product works. During the preview stage, we deploy the latest release in production and use feature flags to control who can see the previewed feature.

We can deploy the latest release in production and use feature flags to select which users access the previewed feature.

Most users understand that they may find some kinks and will be happy to help us sort them out. Therefore, we can start the tech preview even if the application is not perfect (hint: it never will be).

That being said, there are some precautions we need to take before any user is let into our systems. Here's where the DoR for the technical release comes into focus.

Definition of Ready for the technical preview

Let’s say we’re shipping a new feature in our product. Early feedback from real users is essential for its success. We can reach out to users who may benefit from the feature or be interested in testing it out.

How do we know when we’re ready to invite users to check it out? We define a list of tasks and conditions that must happen before we can define our system as ready. These are tasks that are required to be complete for every feature before it can be shown to the users:

• Performance metrics: every new component shipped should be tracked and added to the monitoring dashboard to observe its performance and stability. Semaphore, for instance, monitors the four golden signals (latency, traffic, errors, and saturation).
• Utilization metrics: we need to make sure if the new features are being used and how.
• Audit Logs: shipping anything to production that's not covered in audit logs is a security issue. Every event should be logged to have full audit compliance.
• Security Assessment: we must ensure the new components don’t have any critical vulnerabilities. This is important for two reasons. First, to protect users from security exploits. And second, to comply with any security standards the company follows, such as ISO 27001.
• Deployment to production: the release should be deployed to production. This can be the primary productive environment or a staging copy that's as close as possible to the real thing.
• Preliminary documentation: the users will need instructions about what changed and how to use it. It doesn’t need to be full-blown documentation yet.

The list can be expanded to include more items as needed.

Once we have our DoR defined and fulfilled, we’re ready to start the technical preview. The feature passes to the “in-review” state, and we can begin the feedback-gathering process that may last weeks or months.

💡 Instead of technical previews, the engineering team at Semaphore used to have two kinds of betas: public and private, which had different processes and requirements. Having two sets of similar-but-not-quite-the-same DoRs and DoDs ended up being unnecessarily complicated. So, the processes were streamlined into one. Users are invited to the technical preview based on needs and interest in a given feature.

Definition of Done for the technical preview

The technical preview ends when we've gathered enough feedback and refined the system to the point we consider it ready for release.

Before we can close the technical preview, however, there are some tasks to complete, which will depend on the DoD for our project.

The items in the DoD bucket vary, as they will change from team to team or from company to company. For Semaphore, the list includes:

• Gathering feedback: the whole point of this exercise is to gather actionable feedback from the users. Feedback lets us better incorporate the voice of the customer into the system. It lets us find bugs and improve usability.
• Cleaning up any remaining tasks: address and close outstanding tickets. Ensure all the problems and feedback obtained during the technical preview have either been fixed or have mitigation plans in motion.
• Communicating internally: everyone involved in supporting production should be notified about any problems found during the technical preview (and their solutions).
• Setting up alerts: we need to be aware of a problem in production as soon as it happens. This includes configuring Slack/Teams alerts, setting up pagers, and ensuring that the on-call personnel are on the same page.

General availability release

The moment of truth we’ve prepared for so hard is close at hand. Users will not be as forgiving during a general availability (GA) release as during a tech preview. Problems might disrupt their work. That’s why we went to such lengths to minimize the chance of problems in the release. However, no amount of preparation will guarantee a successful release. So, we need to prepare for any issues that might arise.

Definition of Ready for release

The software might be ready to release, but that doesn't mean we are ready for the release. So, we must define and fulfill the GA release DoR checklist before going into the next step.

As always, every project has different needs. As a guideline, here’re are some of the tasks that are considered necessary at Semaphore:

• Writing the public docs: public docs must be updated with the details of the new feature. In addition, it should be clear which plans have support for it.
• Skilling up internally: everyone in the company, especially support and customer-facing staff, should be briefed on the upcoming changes.
• Setting KPIs: based on technical preview utilization, we can set business objectives and key performance indicators (KPI) for the feature in question. This may include new revenue expected, utilization metrics, or feature adoption.
• Writing a change log: the change log should include everything that has changed since the last release.
• Setting up alerts: we need to be aware of any problem with the new features shipped as soon as it happens. This includes Slack/Teams alerts and pager notifications for the on-call personnel.
• Planning a marketing strategy: the marketing strategy communicates the changes and generates user interest. A good campaign can bring new customers or convert free users into paid ones.

Definition of Done for GA

With some luck, the release went smoothly, and users are happy with the new features. What now? Can we move to the next project? Not yet. This project is not done until all items in the DoD are checked off.

What's typically left at this point is doing some cleanup work and tidying up loose ends:

• Clean leftover tickets: after shipping the new features to GA, there might be some remaining low-priority tasks. They should be addressed or scheduled to be worked on later.
• CLI and API Support: provide support to the new feature via public APIs or the CLI tools, because this makes it easier for users to access resources from the terminal and integrate them into their custom tooling. If this support is integral to the new feature, we can move this item to the DoR checklist.
• Do the project retrospective: The retrospective is one of the most critical phases of a project. It’s a space for the team to discuss what went well and what can be improved for the next project. Also, it allows the team to identify potential pitfalls early and resolve conflict areas.

Conclusion

It's a mistake to think that the hard part of software development is coding. In reality, the real challenge comes when you open the floodgates and users pour in. Unless everything is prepared, issues can compound and overtake any engineering team. Having a formal definition of what constitutes being ready and done serves us to avoid rushing through a release.

Thanks for reading, and happy releasing!