Developer Productivity - How to measure it?

Humans are inherently cognitive misers and we have a tendency to solve complex problems in simple and least effortful ways. And that’s what we have been doing with “developer productivity” by taking the easiest available ways to measure it.

First thoughts when you hear “developer productivity”?

I bet it’s something negative, and no doubt this is almost a taboo amongst development teams, as leaders are often scared that having a conversation about this would make the team think they are being micro-managed or that there’s a lack of trust. Two reasons why:

1. The way developer productivity has been misused by engineering leaders and is being misused as we speak.
2. “Developer productivity” is not a formula, and we don’t thrive in uncertainty, so we choose to either take up the easiest path or just stay away from this.

Just think about this, there are billions and billions of dollars spent on development teams every year and if there was a one fits all way to measure developer productivity, would it have been still a mystery? Or would you be even reading this blog?

PS: If your goal from reading this blog is to measure your most productive developers or get that one number that helps you promote and fire developers, please turn away, as you are in for disappointment.

Then should we even try to measure developer productivity?

Yes, absolutely!! Because developer productivity is not only about improving engineering outcomes but also ensuring the satisfaction and well-being of the team. And often productivity indicators will also help uncover bottlenecks in development processes and challenges with the working environment and culture of your team.

One of the most successful basketball coaches Phil Jackson puts this brilliantly as

“The strength of the team is each individual member. The strength of each member is the team.” — Phil Jackson

And in the context of development teams, every team’s success is dependent on the fact that each developer is performing at his / her best and constantly contributing towards the success of the team.

Okay, now how should I measure developer productivity?

Two fundamental pillars to maximize your chances of being successful in measuring developer productivity.

Never reduce productivity to one single metric

Measuring developer productivity is difficult as we are trying to measure humans who are involved with logical and creative tasks. And us being cognitive misers try to reduce productivity to one metric — and let me state the obvious, that this model WILL fail.

Rather, trying to capture productivity across multiple dimensions and making use of something like the SPACE framework (S- satisfaction and well-being, P — performance, A — activity, C — communication and collaboration, E — efficiency and flow) can assist dev teams in measuring developer productivity the right way.

Communicate with the team

There’s a very common myth, that developer productivity is only for managers. This cannot be further from the truth. Research suggests that the perception of developer productivity is significantly different between the developer and the managers. As most developers correlate higher productivity with higher activity, whereas most managers relate productivity to performance and efficiency.

Source: https://arxiv.org/pdf/2111.04302.pdf

This stark difference between the perceptions can only be eliminated when dev teams have a conversation about what productivity means for them and what are their true intentions in tracking it. This helps in bringing clarity about why is this important, and how should we measure it, and also eliminates reservations most dev teams have like, Is this the number that decides our appraisals? Or are we doing this because the leadership doesn’t trust us to get the job done?

Do’s

1. Use the SPACE framework to track developer productivity across multiple dimensions.
2. Communicate the intentions with the entire team.
3. Use productivity measures to identify areas of improvement, and eliminate bottlenecks.

Don't's

1. Reduce productivity to a single metric.
2. Create secretive measures to track productivity.
3. Use a metric as the only indicator to decide on appraisals.

Developer productivity metrics

Now let’s look at some developer productivity metrics to track across the SPACE dimensions.

Satisfaction and well-being

A highly satisfied team is a highly productive team. And one of the biggest indicators of a healthy team and working culture. However, satisfaction is an abstract concept, and forget metrics, if someone asks you “How satisfied are you?” I am sure you will think for minutes before answering this question on what satisfied means for you and how to quantify this. And we understand capturing this aspect with numbers is extremely difficult. Thus what you see here are proxy metrics that try to best capture different aspects of the satisfaction and well-being of a developer.

1. Work completion
   Our brain releases dopamine whenever we complete a task, which causes us to be satisfied and motivated right after a task is completed. Thus having a high work completion percentage compared to what was committed will make a developer feel highly satisfied that he/she is able to timely complete what was committed and contribute to the team’s success.

2. Extra hours worked
   Busyness is not a means to accomplishment, but an obstacle to it
   — Alex Soojung-Kim Pang
   Working extra hours ≠ Higher productivity, actually, it’s the other way, where Working extra hours is one of the biggest factors contributing to developer burnout and hampering their well-being. Tracking extra hours worked, like the number of weekend hours or late-night hours put in can help you in understanding whether a developer is happy and doing well in the current working environment.

3. Disengaged
   The most common indicators of dissatisfaction and burnout are getting disengaged from the team and team activities. One of the ways to measure how disengaged a developer is by measuring a change in the general response times by the developer towards team activities like code reviews, or a decrease in interactions or presence in team meetings.

4. Developer surveys
   Often in figuring out the best productivity metrics we forget the most obvious ways, which is asking your team and understanding the team’s sentiment by running and analyzing developer satisfaction surveys. Asking a question like “How satisfied are you? (Rate 1–5)” is the worst way to understand this, however, there could be other questions that help you capture similar information in different ways and dimensions.

5. Tenure
   A great way to track satisfaction across the entire team, you can look at the average tenure of members in a dev team. Taking into account the nature of developer trends, a decent tenure range could be anywhere between 12–18 months. Anything lower is definitely a cause for concern.

Performance

The best way to measure the performance of developers and dev teams is by measuring outcomes and not output. These metrics help us in capturing the quality aspect of work done by developers. And an ideal outcome for any developer would be “to develop a feature with the least reworks, ensuring timely delivery and maximum customer satisfaction”.

1. Rework
   When a developer needs to correct their pull requests or often a task comes back from QA to the developer for bug fixes it is a clear indication that the quality of work done is not up to the expected standard. And this to and fro eventually leads to extended feature dev cycles. The idea is not to have zero corrections and often rework is also due to changing requirements, however, if a developer is facing this abnormally higher than the rest within the same team constraints, it’s definitely a sign of a performance gap.

2. Timely delivery
   An outcome that every engineering and business leader cares about is having predictability in deliveries, as often a lot of other business decisions, customer communications rely on these delivery dates. And to have predictability in the entire engineering pipeline it’s absolutely important that every developer also imbibes this quality. One way to measure this is by seeing how much was completed from the committed tasks across development sprints/iterations by the developer.

3. Customer satisfaction
   Unanimously this is the most important outcome that drives value for any organization and thus has to be true for development teams as well. Customer satisfaction might mean better reviews on the app store or higher uptime and faster response time on your API service or for a platform team it could be the ease of use and least bugs reported on internal libraries used by other teams. And even though customer satisfaction is not only driven by engineering teams, keeping that as an indicator of performance keeps development teams in connect with the real users of what they are building and helps them focus on the right things.

Activity

Activity dimension alone is widely equated to developer productivity, as it’s the one that’s easiest to track. However, activity alone can never be a true measure of developer productivity. Tracking activity metrics in conjunction with other dimensions across different areas of your SDLC process will help you in identifying true bottlenecks and areas of improvement for developers.

1. Tasks resolved
   Activities in this phase help in identifying how frequently and how much a developer is contributing towards development tasks. And given development tasks are always planned as tasks, user stories, or sub-tasks on a project management tool, looking at the total tasks resolved helps in understanding the involvement of a developer in this part of the development cycle.

2. Pull requests reviewed
   Often the responsibility of reviewing change/pull requests is ONLY with tech leads or the managers of a dev team, which is a clear anti-pattern. Encouraging every developer to review more and more code of their peers helps in eliminating review bottlenecks. And this metric would be a good way to identify if a developer is contributing to the review load of the team.

3. Deployment frequency
   How many times a team deploys changes to production systems, helps you understand the speed aspect of the development process. This is also one of the DORA metrics, and research shows DORA metrics also have a high correlation with customer satisfaction and even the profitability of an organization, making this a great measure to track the activity dimension of a development team’s productivity.

Communication and Collaboration

In any development team, the final outcome be it a feature, service, application, or enhancement is always a result of team effort. And great communication and collaboration are the foundation blocks on which highly effective development teams are built. Including this dimension in measuring developer productivity promotes a culture of transparency and information sharing. Some productivity metrics that help to capture this are:

1. PR wait time and cycle time
   If a dev team has great collaboration, it’s clearly reflected in their review process as potentially this is the most bottlenecked development process, as it depends on a contributor to communicate effectively with the reviewer and vice versa. A metric that helps in tracking how well a developer is collaborating is by measuring how long it takes for this developer to start reviewing a pull request after it’s assigned. And in continuation measuring the median cycle time of pull requests helps in understanding the contributor’s communication skills.

2. Count of members co-worked with
   Development teams very commonly have knowledge silos and groups of developers that interact only with each other and not with the rest of the team, this is another one of the classic anti-patterns. Measuring how many and how well a developer is communicating with other team members is a great way to measure this dimension.

3. Onboarding time for new members
   Whenever a new developer is onboarded to a team, they undergo an initial learning curve and understand the business context, and getting familiar with the tech stack, and often are helped with code walkthroughs as well. The time it takes for a developer to make the first impactful change since they have joined is a great productivity metric to capture the communication aspect of a dev team. As a team with great documentation practices, developers who are willing to spend efforts to help the new joiners will enable the new developer to make an impactful change as soon as possible. A good benchmark to strive for is that the first productive output of a new developer is within the first 30 days.

Efficiency and Flow

This is the dimension that captures “getting into the flow” that developers often talk about. Metrics here help in understanding how many interruptions are there within the development cycles and how smooth is the flow of a task from start to end. Frequent interruptions not only affect the developer’s productivity but also cause increased levels of stress and exhaustion.

1. Uninterrupted development time
   It’s absolutely important for developers to have enough uninterrupted time every day to get into flow and invest time in development activities. And of the biggest blockers to this would be team meetings. Often to encourage higher development times, teams adopt a no-meeting weekday or adopt strict timeslots where team meetings can be scheduled. A higher uninterrupted development time doesn’t necessarily indicate a higher productive developer, however, it’s certain that a developer who doesn’t get enough uninterrupted time wouldn’t be able to achieve the required flow for development.

2. Commit lead time
   More interruptions, more handoffs, and tasks too frequently being re-opened in the development cycle are all indicators of bad efficiency and flow in development tasks. And commit lead time captures this accurately as it measures the total time it takes for a change to make an impact on the end users. A comparatively higher commit lead time (CLT) would definitely mean a drop in the efficiency and flow of the development team. CLT is also one of the DORA metrics. More about it here.

3. Average In progress (WIP) tickets
   Context switching is undoubtedly a productivity killer. Having more to do parallelly would always mean it would take more time to complete all and would also lead to unnecessary mental exhaustion.
   2 parallel tasks — 20% is lost to context switching
   3 parallel tasks — 40% is lost to context switching
   - Gerald M. Weinberg
   And WIP tickets beautifully capture how many tasks a developer is working on parallelly. Tracking this productivity metric and trying to keep it always < three tasks is a great practice to start for your dev team.

Change Involvement

When you look at metrics to improve developer productivity, the action that will help you drive change will be a change in your development processes. And measuring how involved and engaged a developer is with the changes the team is driving helps in understanding the effort that everyone is spending to correct the team’s processes. And every process change can have a metric related to it, that captures how well the process is being followed, and tracking leaderboards across these metrics can help you in gamification and understand which developers are contributing well to the process change initiatives.

That’s all folks!

We saw how developer productivity is very commonly misinterpreted as a single metric or with metrics that are easy to track rather than the actual ones that matter. And it’s absolutely crucial that we track developer productivity and take inspiration from frameworks like SPACE to take a holistic approach to developer productivity. And it’s best to start with only a few metrics, but it’s absolutely important to select these metrics from at least three different dimensions. We have talked about an exhaustive list of dimensions and many metrics in each, and now it’s upon you and your team to figure out the right dimensions and the right metrics in them that would be most impactful.

TL;DR: (thanks to ChatGPT)

Measuring developer productivity requires a holistic approach, focusing on multiple dimensions rather than a single metric. The SPACE framework (Satisfaction and well-being, Performance, Activity, Communication and collaboration, Efficiency, and flow) can assist in capturing productivity accurately. It is important to communicate with the team to align intentions. Metrics such as work completion, rework, customer satisfaction, code impact, PR wait time, uninterrupted development time, and process change involvement can provide insights into developer productivity. Taking a comprehensive approach will improve outcomes, team satisfaction, and process efficiency.

And if you are looking for a platform to track developer productivity that’s built on these ideologies, give AnalyticsVerse a spin here.