How We Measure Software Product Quality: Scrum Metrics We Use

The article was originally published on Ascendix Tech Blog.

We want to share the ways we measure software product quality with Scrum metrics to keep our project progress and team productivity on track.

So, you will learn about:

• Scrum metrics we apply for quality measurement
• Formulas to calculate each metric
• Examples of metric charts and tables
• Reasons to use Agile metrics.

Let’s get down to business.

1. Sprint Velocity

Sprint velocity is a popular Scrum metrics which enables you to get a historic overview of how much value you have delivered during each sprint.

In order to calculate this metric, you need to sum the story points completed at all sprints and divide them by the number of sprints.

For instance, let’s calculate sprint velocity based on the below data:

• Sprint 2: 5 user stories x 12 story points = 60
• Sprint 3: 6 user stories x 12 story points = 72

• Sprint 4: 7 user stories x 12 story points = 84

• Total: [Sprint 2] + [Sprint 3] + [Sprint 4] = 216

This way, your average sprint velocity is 216 / 3 = 72.

Here you can review an example of a sprint velocity chart:

2. Scope Change (SC)

Scope change is an effective Scrum metric that allows you to analyze the root causes of no meeting commitments.

Specifically, this metric helps you track the number of stories added/removed within the project during a sprint or release.

‘Descope’ and ‘scope increase’ are two indicators that will determine when and why any mistakes or changes were made.

The formulas calculate them are as follows:

• Descope: = (D/C) * 100
• Scope increase = (A/C) * 100 , where D – removed SP, A- added SP, C – committed SP.

Let’s now check the example of committed, added, and descoped stories within a project:

So, the sprints 2, 4, and 5 were modified by adding or de-scoping story points.

Now let’s calculate the scope change variance to see the added/descoped sprints:

In order to analyze and demonstrate these changes more vividly to stakeholders, you can build a scope change chart and see changes from sprint to sprint.

3. Effort Estimation Variance

Effort estimation variance is one of the wide-spread Scrum reporting metrics which allows you to measure under- and overestimation rates.

Put it simply, it is a metric that allows you to track actual efforts made over the estimated hours.

Let’s consider an example of how to calculate effort estimation variance.

Here is a table of sprints, estimated hours, and actual efforts in hours that were spent:

Based on the table data, you can see that only sprint 3 has no over/underestimation which means that no extra/lacking efforts were put into the software development process. Other sprints were either over- or underestimated.

There are two simple formulas to calculate over/underestimation values:

• Overestimate = (E-A)/E * 100
• Underestimate = (A-E)/E * 100

, where A – Actual effort and E – Estimated effort.

So, they allow us to easily calculate the effort estimation variance in percent:

Based on the data above, you can safely assume, that Sprint 2 was excessively over-estimated while Sprint 4 was underestimated.

If you see a negative variance, it is a sign that something is wrong.

Below you can see an example of an effort estimation variance chart:

4. Defect Leakage

As your software product grows, you will definitely find multiple bugs as it’s a default and normal situation dedicated to the human factor.

These errors influence the final estimate, client satisfaction, and team happiness.

If you are wondering whether there is a method or metric to track a bug ratio within your project and improve on, you should use defect leakage.

It is one of the key Agile metrics which allows you to track the summarized number of defects detected when a sprint or release are done.

At Ascendix, we have incorporated a company RAG (red, amber, green) status. It helps us get a high-level overview of a project’s activity, generate a relation coefficient/variance, and measure all our projects by specific upper and lower limits.

Here is an example of the way we categorize possible variance values based on our RAG status:

Now let’s take a look at the example where we summarize the detected sprint defects:

The formula to calculate a defect leakage value is as follows:

• Defect leakage = (Post sprint defect count / defects detected during sprint) * 100

It allows us to measure the variance value for each sprint:

Additionally, you can create a defect leakage chart that indicates deeply the volume of defects within every completed sprint:

How to Make Scrum Metrics Effective?

It’s worth mentioning that metrics are only instruments to help you evaluate specific project activities or progress. NOT ALL Scrum metrics should be implemented to every project you carry out as different project require different measurement.

The value delivered is the key and the only result you should focus on while developing projects. So, if metrics are of great help for you and they provide value measurement, only then it’s a great idea to implement them and start analyzing.

You should keep in mind that METRICS ARE ABOUT VALUE, not about numbers.

At Ascendix, we have built 5 key rules that should be followed to make the most out of the Scrum metrics you use. Let’s take a look at them:

• A metric should be used by the team
• A metric should be based on a conversation
• A metric should answer concrete questions
• A metric should be used together with other metrics
• A metric should be easy to calculate and understand

Bottom Line

We shared with you the Scrum metrics we use when developing software. They can help you better analyze and measure your team’s performance and project progress to improve on.

However, they are NOT a single source of truth. So, try and test them for your project, measure whether they help you evaluate the iterative or final value delivered, and make your own choice.

Want to get more information about our best practices in measuring the project’s progress? Feel free to read the full blog post: How We Measure Software Product Quality: Scrum Metrics We Use.

What ways do you use to measure software quality? Do you use any metrics? Share your experience in the comments below.