The Agile + DevOps paradigm is now the de facto norm in software development. Especially given the software application diversity, complexity and need for speed in the digital boom. It is a fine balance between software quality, speed and the cost-effectiveness. And test Automation is a key driver that helps businesses achieve this ideal. However, it is easy to lose sight of the quality goals set at the test automation velocity that organizations operate.
In order to ensure the balance and maintain the speed, it is necessary to track the right performance indicators and measure quality throughout the DevOps lifecycle. But unless you match them with business goals, a metric is just that – a metric. It is important to measure the right things.
What are some of the measures that help you improve your performance and efficiency? Here’s taking a look:
Agile Metrics
In Agile, the key metrics are leadtime, team velocity, cycle time and open/close rates. Measuring these help planning and decision making for process improvement. Even though they don’t measure the success per se, here’s why.
Leadtime is the time taken from idea to actual software. If you want to be more cost-efficient and competitive, improve your leadtime. You can do this by simplifying decision making and decreasing the wait time.
Team velocity measures the number of units a team completes in a sprint. This is not for comparing team performances but rather a benchmark used to plan iterations. Also, velocity shouldn’t be treated like a success measure because that would mess up the accuracy of estimation and planning.
Open close rates show the number of production issues reported and closed during a time period. Here, the general trend is more important than the specific numbers.
Cycle time is the time that takes you to make a change to the software system and deliver it to production. In the continuous delivery world, teams take as little as minutes or seconds for their cycle time.
It is worth mentioning that numbers are a good indicator of performance, but you can’t assume the root causes from the metrics. These metrics point you to the right direction to probe further into processes that might need your attention.
Production Metrics
As the name indicates, these metrics evaluate the scope of assignments and measure the overall productivity. They may be a good indicator of operation speed if measured correctly.
Mean time between failures
Mean time to recover or repair
Both MTBF and MTTR measure the overall software performance in the existing production environment.
Whereas, Application crash rate – is derived by the number of times an app fails divided by the number of times it was used.
While the numbers for all of these production metrics don’t tell you more about the features or users affected, they are a good indicator of software performance. That is, the smaller the numbers the better it is.
There are several granular measurements beyond MTBF and MTTR that take individual transactions and applications into the picture.
Test Automation Metrics
Requirements coverage helps minimize the risks and ensures that software quality meets the detailed requirements specified by business and project members. Additionally, functional tests and bug density, ratio of pass versus fail tests and number of functional defects found give you an accurate status of the overall software function.
Unit test coverage tracked and prioritized by risk is a great way to improve the overall efficiency of your testing suite. With the right tools, you can not only provide critical feedback earlier, but also show a requirements traceability matrix that gives a comprehensive view of the coverage.
Integration testing metrics
In the DevOps and CI/CD context, the importance of API testing can’t be emphasized enough. The API layer plays an important role in the overall success of your release and hence API testing is necessary. This can include API bug density, requirements covered by API tests, the pass/fail rate and the API functional code coverage.
Regression testing
Defect Detection Effectiveness is a good measure of your regression testing efforts. It is basically the ratio of defects found before and after your release.
Multi location or distributed teams are typically battling with an insurmountable size of regression test suite. This is why it makes sense to use automation to deliver higher ROI. Due to the incremental nature of code changes in regression testing, functional test code coverage is an important measure. Many teams also find it useful the measure the percent of end-to-end automated test cases.
While all of the above-mentioned metrics can have remarkable benefit if used correctly, how they are used and measured is also equally important. Objective software quality metrics are easy to measure and quantify, it is the subjective metrics that matter now for business success in addressing the whys and how’s of delivering successful software.
Top comments (0)