loading...
Cover image for Hidden Knowledge In Software Teams

Hidden Knowledge In Software Teams

busypeoples profile image A. Sharif ・6 min read

Introduction

If you’ve been working in Software long enough you probably have seen the following two situations:

a. Started working on a new system

b. Joined an already running system

Systems evolve and change over time. Another trait they commonly have, is that they become complex, not necessarily on a code level, but in terms of interactions between parts of the system. How does A behave in relation to B? etc. To be fluent within a large system, you need an understanding of how everything is interconnected.

Very often a single contributor or a small number of people have an understanding of how the system works in its entirety. And as long as a single person is working on that system, this person can be productive, very productive even, moving with full speed. But at some point, more people will be needed to maintain and grow that system, as it can’t be handled efficiently by a single person anymore. It needs to scale from a contributor perspective.

What happens naturally, is that development speed has to be exchanged with on-boarding time. It’s a trade-off, balancing short-term speed with long-term maintainability.

It means that the most productive member, has to invest time on-boarding the ones joining. This on-boarding process comes in different shapes and forms, but commonly achieved with the help of documentation and tests and other ways to describe an expected behaviour.

Common Approaches

In written form, we document architecture, sub-systems, the relation to each other. We might have an explicit documentation for setting up the project, in form of a readme file f.e. etc.
Further more, information can be manifested in automation scripts, in project management tools etc. As we can see, there is a lot of work needed when trying to make any knowledge, relevant to a system, explicit.

In reality these artifacts are often either mostly outdated, available in parts or completely missing. One or two people starting to build a system, don’t necessarily need this information because there is a common understanding of the details, as they have written it themselves.
They know why something behaves in a certain way, what happens when they change a certain part and what is needed to achieve a certain task. Due to the deep understanding of the systems internals they also might be able to move fast, confidently make changes and continuously ship features.

Now what happens when the system requires more people, due to reaching a certain size or complexity?

Scaling the Team

Let’s think about a possible scenario where new people join the original authors. These creators need to on-board the new members.
The new group needs to understand design decisions, they need to figure out how everything is interconnected, they need to find out about boundaries and so and so forth. Due to the lack of historical context, they also need to figure out why certain implementations are built in a certain way, which might be obvious to the original contributors. There is hidden information that needs to be made explicit, step by step. From the perspective of the original author, it might seem that nothing is too complex, that reading the source code etc. should suffice. The more complex a system gets, the harder it is for an outsider to understand these things.

Things get tricky, when the original contributors think that everyone has the same understanding as they do. Mostly this is not the case. How does such thinking surface to the top? Some examples. “That should take 5 minutes to implement”, “Obvious”, “That should be easy” etc. Knowledge is contextual, it depends on an existing implementation, on understanding decisions made in the past etc.

But what happens when a team skips this part of the transition?

What happens when the original authors assume that there is no need for further explanation, no need for a deeper documentation etc.?

What happens when the burden is shifted to the new team members?

Hidden knowledge hinders team performance, people depend on one or two individuals to move forward. Most probably the rest of the team will try to understand certain parts by making small changes in isolation to prevent anything from breaking. They might try to go through available commits or comments to get a better understanding of what might happen when they change A. Will changing A have an effect on B or C? etc.

So, opposed to the original writers, new members will move cautiously, double checking any changes and trying to gather more information along the way. But what can we do to improve the transition phase?

As mentioned, further above, it’s an explicit trade-off between current and future productivity. Time needs to be shifted away from developing on full speed to knowledge transfer across the team. The further ahead knowledge is documented, the less stressful this phase is for anyone involved.

Actions

What are possible actions that can help in this situation? Just think about it from the perspective of someone joining an existing setup.

Readme

A readme that at least has information on how to setup a system f.e. There is none or is it outdated? This is a good opportunity to write or update that readme file. It might seem like nothing too special, at first glance. But how often have you checked if a readme file exists when having an initial look at a repository?

Documentation

Documentation can manifest itself in a number of forms, the most conservative being something like a wiki, where high level decisions are described and where you can look up on architecture as well as other fundamental information. Also, documentation can be found in a design system, in a style-guide, via user stories etc. Depending on what we are building, a combination of multiple tools might be needed to sufficiently transport the required information. Sometimes these infos are partially available, which is a good opportunity to start updating or completing when new people join.

Tests

Tests is another way to describe expected behaviour. They can also provide confidence to new developers, due to signalling when something breaks, as opposed to finding out later on via a review or in production. Partly tests could even fall into the documentation category.

Pair or Mob Programming

Another strong approach to transferring knowledge is actually working together on an existing problem. It can highlight hidden knowledge, it can help quickly resolve questions and it provides confidence to new members when working on the actual feature, due to very short feedback cycles. This is also good opportunity to fill any gaps in regards to documentation, especially information that seems straightforward to the original contributors but not to anyone joining later on.

Work together on Stories

If pair or mob programming is not an option, then work together on a story, meaning break it up into small enough tasks. This is better than having every developer working on a single story.

Understanding Context

This one is probably the most important of all, and might hinder all other approaches: Is there a common understanding that there is hidden knowledge not available to every member working on that product/system? Everyone needs to watch out for phrases like “easy” and “should take 5 minutes to implement” etc.

Being helpful and understanding

This point is a logical conclusion of the previous one. Once there is an acceptance for the fact, that not everyone has the same level of knowledge regarding the system, there should be supportive environment for the newly joined. Supportive can mean, always ready to answer questions, provide help etc. Also means not to assume people are under-performing or “don’t get it”, just because it takes the original authors half the time to solve a problem. At the end it’s about moving from individual high-performers to a performing team.

Ship less

Deliberately choose to ship less when in a transformation phase. A team can’t keep on shipping at the same rate, while on-boarding parts of the team. Committing to less, gives room for the on-boarding phase. This is an important investment for future iterations.

More

There is more that can be done, but we probably have covered the more common ones, which we can build upon on. Other supportive approaches include scripts for importing-data, setting up local environments etc. All these help to further uncover hidden knowledge.

Summary

We know that if we want to go fast, it’s better to work in solo mode, but if we need long-term maintainability and extensibility it’s better to rely on a team. Sometimes it appears that teams can’t perform, relying on one or two high productive developers doing most of the work. Often the problem is hidden knowledge, meaning that not everyone has the same level of experience with or knowledge of a system.

If you have any questions or feedback please leave a comment here or connect via Twitter: A. Sharif

Posted on by:

busypeoples profile

A. Sharif

@busypeoples

Focusing on quality. Software Development. Product Management. https://twitter.com/sharifsbeat

Discussion

markdown guide