In the world of software engineering, it seems like nobody can agree on what productivity means. I’ve seen many engineering leaders struggle with its definition, many with conflicting opinions on what engineering productivity means and what metrics to look at.
But understanding productivity is key to unlocking the potential of many engineering organizations, and can ultimately help us ship better products.
Today I’ll go through the reasons why I believe we’ve struggled to define productivity in our industry. I’ll also offer my views on it, who owns it, how to measure it, and how engineering teams can improve it.
Here’s my quick and dirty definition of engineering productivity:
💡 Engineering productivity is the capacity to measure if your engineering teams are doing the right things, with the right investment, to achieve a specific business outcome.
It’s important to note that productivity is often driven by expectations, and its meaning might depend on where you sit in the company, as well as the maturity stage of the company.
For example, the board and CEO are typically more concerned with time to market (velocity), while the CTO knows that there are other important indicators of productivity, such as quality and maintainability of the product and the developer experience.
Understanding The Different Components of Productivity
Before we dive in 👉 Since there seems to be a recurring theme in our industry where different meanings are given to the same word, here are my definitions of some of the terms you’ll see throughout the article:
🛠 Throughput: The nº of items you ship to production.
🛠 Velocity: The speed at which those items go through the pipeline. Velocity helps your to predict the time it takes to deliver something.
🛠 Quality: How satisfying your delivery is based on customer expectations. There are multiple criteria you can have to understand product quality: nº of bugs, ease of use, performance, etc.
🛠 Capacity: The nº of devs available to work on a specific project or initiative.
🛠 Outputs: Activities you’ve been executing. 👉 (e.g. deploying a feature)
🛠 Outcomes: The impact of those activities. 👉 (e.g. I want this feature to increase my user NPS by 20%)
🛠 Investment: The money you want to put into delivering a specific outcome to the market or specific output to the market. 👉 (nº of people/salary) x time to deliver
🛠 Business Value: The value for the business of delivering a specific outcome. Ideally measured in $. (eg. If I get a SOC2 compliance certificate I’m able to open a market that is worth x millions of dollars.)
🛠 Effectiveness: How fast you’re shipping things and how their quality is perceived. In this case, it’s not about technical quality, it’s about having the right impact on the market. Effectiveness combines quality, investment, and business value.
To me, effectiveness is deeply connected to productivity. Understanding this helps to answer: “Am I able to deliver the right thing at the right time, with the right investment?”
Ok, now that we have those definitions out of the way, let’s dive in.
Defining Software Engineering Productivity
Engineering productivity is hard to define because there is no set framework for it. And when you don’t have a framework, meaning will stem from expectations.
Typically in the early days of an organization, it feels like you’re moving fast. But as you scale, the distance between leaders and developers increases. Like zooming out on a map, you lose visibility, and your communication channels, dependencies, and issues increase.
Your team grows, and things start moving slower.
Slower compared to what?
If you are comparing a team with no customers, 10 devs, and no live product, to a team with 100 devs and 2,000 customers, it may seem like things have slowed down. But this is normal. As soon as you have more customers, you have more incidents and responsibilities. You need to care about value delivered to customers, not only about shipping new features, so you slow down.
“But if I added 10 engineers to my teams, shouldn’t I be moving 10x faster?”
Unless you’ve perfectly set up your organization for the smoothest onboarding of new hires, you have the best engineering managers, and your customers love everything about your product, no, you won’t be moving faster.
🚘 You buy a car. It’s a good car, and you’re a good driver, so it takes you 20min to get to the beach. One day, you have a kid, and your kid sits in the backseat as you drive to the beach. Suddenly you’re moving slower. But it’s not the engine, it’s not the car, it’s not that you suddenly became a bad driver: it’s the kid. The kid requires more attention and safety. You still get to the beach, but as you adapt to having a miniature version of you in the back seat, your ride there will take longer than those blissful 20 minutes. Your customers are the kid in the car, they increase your business value, but they also demand attention and caution.
Why & How Productivity Expectations Change
We’ve established that slowing down as you scale is normal - but you have a board to report to, and the pressure coming from the top is all about time to market.
But if you only look at velocity, you’re not capturing productivity. If I’m running around in circles super fast, am I being productive?
As you scale, the CEO gets further away from the CTO, and the CTO gets further away from the engineering teams. And you start to see something across the multiple levels of the organization:
🏎 The higher-up you are, the more productivity is connected to speed and revenue.
💪 The lower-down you are, the more productivity is connected to your effectiveness as a developer (which is inherently linked to developer experience).
TL;DR: The bigger the org, the harder it is to define productivity.
But it gets more complex. The expectations of productivity change as you grow, where you sit in the organization, and where the market is.
If the market has shifted its focus from growing, to cost optimization, the first thing that’s going to come from the board is “how can we make our engineering team more effective?”.
Effectiveness is a synonym for productivity. But because of the current market, the board/CEO will look at effectiveness differently. To them, it’s about having more output with fewer people.
This translates to 👉 “How can I maintain my effectiveness or productivity while cutting costs?”
This is different from the growth mindset of “how can I add more people to the team to do more, even if we go a little bit slower per contributor”.
The first statement is about reducing costs, the second is about improving the overall throughput.
In sum, how you measure productivity also depends a lot on your goals.
How to Measure Engineering Productivity
One of the first things you have to do as an organization is to define your productivity framework - this will allow you to consistently observe increases and decreases in productivity.
Productivity is the throughput of the engineering organization, and it’s measured by how much business value the engineering org can achieve in a given timeframe.
💡 Your framework is the consistent system by which you measure these productivity indicators, so they can be easily compared throughout your reporting periods (i.e., translating all values to dollars and repeating the process every quarter).
Whenever the market changes, the team grows, or the conditions change, you want to have a metric that allows you to compare how you were six months ago, or 16 months ago, from today.
Across industries and organizations, people, contexts, and cultures are completely different. So the way we define and measure productivity has to be too.
Productivity is one of the pulses of the organization. 🫀 Measuring it allows engineering organizations to course correct their processes and organization as they adapt and grow.
But how do you measure it?
The way that I measure productivity is by aligning business outcomes with the product team and attributing value to those outcomes.
This is sometimes called Business Value Framework. Based on this framework, the Product organization estimates the potential value for the company of an outcome/output and engineering estimates how much effort and time it would take to get there.
One key thing is to understand what the organization is prioritizing at the moment, which will be dependent on where the market moves.
A great way to think about it is by looking at the Business Value Matrix:
Once the high-level priority is clear, specific business goals are created.
With that in mind, we can see productivity as “how much effort/investment did I exercise to achieve a specific business goal”.
Initially, you will measure this by looking back, but as you start to gather more knowledge, patterns will arise, and you’ll be able to make predictions on future investments and potential impact. It will always be hard to be precise, but you want to use comparisons and have a common understanding across the org to generate alignment.
Once you’ve defined your metrics for productivity, you need to measure consistently (I suggest doing this quarterly). At that point, you can start planning ahead by asking the right questions, such as:
- What are the outcomes we want to achieve in the next quarter?
- What capacity do the multiple teams have/need to achieve them?
- What is the business value of each of those outcomes?
- What initiatives/activities will lead to these outcomes?
By looking at those four things: Outcomes, capacity, business value, and activities - we have the elements needed to measure productivity and keep an eye on it as the quarter goes by.
Each quarter we can look back and see if our productivity has improved, and adjust accordingly. We can also tweak for velocity and quality since everything is aligned with our business goals.
An Example of Measuring Productivity
To me, the easiest way to measure productivity with the metrics like the ones we have at Athenian is to:
1. Identify the quarterly initiatives or the epics that the teams are working on and their potential value.
2. Understand the overall lead time of your initiatives
3. Dig into the data to understand what impacted your productivity
Here’s a great example of how to do this 👉 How to Use Data to Improve MTTR of customer-facing bugs.
Once you have a clear understanding of all this, you can take action in terms of being effective in keeping lead time under check or reducing lead time for epics.
I like this method because you can measure whether you’re getting better or worse in terms of productivity without impacting quality.
To be clear: Athenian can help you measure the effectiveness of your engineering teams, but to get a complete understanding of productivity you will need to link this to your business goals, get customer feedback, and more.
There is an art to building organizations, and lot’s of luck. When you are able to combine the two, you will have a product that customers love, and that your teams are proud to be building.
In the end as a CTO or VP of Engineering, you’re responsible for bringing this up to the executive team and senior leadership team, to ensure you create alignment on how to measure the impact of your org.
Ready to dig into the data? Find out more about how Athenian can help!
Top comments (0)