How to use metrics to tell the story of a software development process

In a world run by data, metrics act as our compass. They shed light on our performance, progress, and successes, empowering us to translate abstract goals into actionable data. However, just like any powerful tool, metrics can be a double-edged sword.

Numbers can be deceiving. A seemingly positive metric, like a decrease in Mean Time to Repair, might mask a troubling reality – the team is resorting to quick fixes that create even bigger problems down the line. This is where the context in these metrics becomes crucial.

Even after understanding the context behind the data, how do we effectively communicate it to stakeholders with varying technical backgrounds? Numbers alone can be cold and impersonal. Storytelling breathes life into data, allowing us to explain not just what's happening but why it matters. It lets us explain why we're dedicating resources to a particular problem or feature and how it aligns with the bigger picture.

This article dives into software development metrics, exploring the pitfalls of traditional approaches and highlighting how a nuanced understanding of data, presented with context and compelling narratives, can be the key to building successful software.

The purpose of metrics in software development

Software metrics are quantifiable measurements used to assess the quality and health of our software development process. These metrics must be quantitative, understandable, reproducible, and applicable. They provide data pinpointing improvement areas and guiding decision-making in the development phase.

Although software metrics and their use have evolved over decades, traditionally, they are used for two main purposes in software development:

• Performance evaluation: Metrics was a primary tool for assessing individual developer performance. Lines of code written, number of bugs fixed, and time spent on tasks were common metrics used for this purpose.
• Process improvement: Metrics were employed to identify inefficiency in the development process. Measuring factors like cycle time, defect rate, and code complexity helped teams understand bottlenecks and areas for improvement. This data-driven approach allowed for more informed decision-making when optimizing workflows and tools.

While traditional metrics laid the foundation for everything we know now about metrics, they were not without limitations. These limitations begged for a more nuanced and holistic approach to evaluating performance and driving process improvement used today.

The challenge: Beyond the numbers game

Traditional metrics focused heavily on quantifiable data, a factor contributing to their limitations. While metrics like lines of code and latency provided some insights, they could be misleading and often lacked context. This resulted in an incomplete picture of performance. Here are some examples of these limitations:

• Inability to identify trends: Context helps in understanding trends over time. For example, memory usage might appear high at 90%. Still, if this is typical during certain processing periods and doesn't result in performance issues, it might not be a cause for concern. Understanding trends requires context, such as baseline performance, peak usage times, and the specific applications or tasks running on the server.
• Risk of confirmation bias: When relying solely on raw metrics, there's a risk of confirmation bias—interpreting data to confirm preexisting beliefs or biases. Context helps challenge these biases by providing a more comprehensive view.
• Lack of nuance: Context adds nuance and richness to data. A customer satisfaction score of 80% might seem good at first glance, but if we dig deeper and find out it's due to one outlier region skewing the data, the overall picture changes**.
• Ignoring the "Why" behind the numbers: Consider a scenario where a network monitoring tool reports a sudden spike in network traffic, showing a 200% increase compared to the previous day. This raw metric alone might lead to panic or confusion.

However, without delving into the why, this spike could be due to various factors: perhaps a new software update was released, causing all devices to download it simultaneously. It could also be a distributed denial-of-service (DDoS) attack, artificially inflating the numbers. Without context and further investigation into the cause of the spike, the appropriate response (such as optimizing network traffic for updates or implementing DDoS protection) cannot be determined.

However, metrics can be even more powerful than they already are if we use these data to tell a story about our development process. Rather than presenting these raw data points, we can use these metrics as plot points, transforming abstract figures into a tangible story that illustrates progress setbacks, milestones, the value delivered, and what caused all these at different points in the development process.

By presenting these metrics with the context they live in, teams and stakeholders gain a deeper understanding of the development process, its challenges, successes, and the evolving story of the product's creation.

But how exactly do we tell this story?

Telling the story

Similar to any story, crafting a narrative with metrics is made up of different parts, which we will explore in this section.

Paying attention to the metrics that matter

One of the best parts of a story is the quality of the characters, in this case, the metrics. To make the most out of our metrics and learn as much from our system, we must first recognize which metrics are important to monitor and why. To make that decision, we need to ask ourselves some of these important questions:

• What are our goals and objectives?
• Who are the stakeholders?
• What are the key performance indicators (KPIs)?
• What resources are available for data collection and analysis?
• Are there any regulatory or compliance requirements?
• How will these metrics drive action and improvement?

By addressing these questions, software engineering teams can identify the most important metrics to monitor, ensuring that they align with project goals, stakeholder needs, industry standards, and the project's overall success.

However, selecting the right metrics is just one part. To truly understand our systems, we need to delve deeper. This means uncovering the "backstory" behind the metrics – what trends and patterns influence them and when these trends typically occur. We can anticipate changes and prepare more effectively by understanding these underlying factors.

Identifying trends and patterns

Recognizing the limitations of traditional metrics, many have opted for an artificial intelligence approach to IT operations (AIOps). With 97 percent of IT professionals believing that AIOps will deliver actionable insights from their metrics, AIOps tools like Eyer are pioneering this new wave of intelligent metric collection and analysis.

By leveraging machine learning and artificial intelligence, these tools become intimately familiar with our system – essentially learning from the data they're designed to analyze. By studying historical data, AIOps tools can pinpoint frequently occurring behaviors and cyclical patterns and even identify anomalies that shouldn't happen at all.

For example, a recurring pattern, such as periodic spikes in network traffic during specific hours, might indicate regular batch processing jobs or daily video meetings. AIOps tools, having learned our system's patterns, can differentiate between these normal spikes and those that require attention.

Eyer, an AIOps tool, goes a step further by grouping related metrics. This allows Eyer to identify a single metric signaling trouble and pinpoint other related processes that might be affected.

Highlight areas of success.

Another great selling point of storytelling with metrics is that it lets us craft narratives that resonate with our audiences, whether they are stakeholders, team members, or customers.

By combining quantitative and qualitative data, like deployment frequency metrics with customer satisfaction feedback, we can illustrate the efficiency of delivering new features and the tangible impact on user experience. This approach quantifies success in terms of the number of deployments and qualitatively captures the positive outcomes, such as increased user engagement and satisfaction.

An amazing example of this phenomenon is Netflix opting to use feature flags as an alternative to traditional deployment methods. This decision caused a significant increase in deployment frequency, enabling faster access to new features, improved feature quality, and more, allowing Netflix to respond quickly to users' needs and build a product their users love.

With storytelling, we can paint a comprehensive picture that appeals to stakeholders, highlighting the "what" of success and the "why," ultimately fostering a deeper understanding and appreciation of achievements within the context of the user's journey.

Benefits of storytelling with metrics

Storytelling coupled with AIOps observability metrics are transformative tools for any organization; they offer a myriad of benefits that extend beyond traditional data presentation, some of which are:

• Clear communication: By weaving data into a narrative, storytelling with metrics cuts through information overload, fostering clear communication. Metrics provide the objective foundation, while the story translates complex data points into relatable concepts, ensuring everyone, from technical experts to stakeholders, understands the data, leading to more informed decisions and a shared vision for success.
• Data-driven decision-making: Compelling stories grounded in AIOps data can inform decision-making processes. Stakeholders can see the impact of different approaches, directly connect development efforts to positive user outcomes, and foster a culture of data-backed innovation.
• Proactive problem identification and predictive insights: Considering trends, context, and patterns, we can recognize quickly when a metric points out anomalous behaviors in a process. This recognition makes it easier to identify potential issues before they snowball into bigger resource-intensive issues.

This proactive approach is a key reason AIOps tools are gaining widespread adoption. As Rich Fairbank, CEO of Capital One, succinctly stated, "We expect anomalies to happen, [and] we expect some of our segments to be stressed. So we're on the lookout, and we have machine learning to help us."

• A single source of truth: Using AIOps tools to drive the collation of metrics helps consolidate data and metrics from various sources, eliminating discrepancies and inconsistencies. Additionally, we don't leave the data open to interpretation because we care about the context as much as the data itself.

This lack of ambiguity ensures that stakeholders, whether team members, managers, or clients, have a unified and accurate understanding of the events the metrics represent. Tools like Eyer promote transparency and trust within the organization by establishing a single source of truth.

Conclusion

As highlighted throughout this article, metrics can be more than just random numbers; they can be characters with complex backstories and motivations and even warn us about impending risks in the future. Embracing this shift or journey from traditional metrics to a more nuanced storytelling approach can be the difference between staring at a flat map and navigating through a complex and vibrant landscape.

This richer understanding empowers software engineering teams to make informed decisions, translating to exceptional user software experiences and proactively identifying inefficiencies or potential risks early in development.

In summary, an organization's ability to make sense of its metrics and use them to tell a story could be the true test of its agility and ability to stand the test of time.

To learn more about your processes and gain proactive insights, check out the AIOps observability tool, Eyer, and join the waiting list today.